0

I need to limit the amount of memory being used when running a Python script in NiFi, and I found this answer in Stack Overflow. I follow the logic of how the code works, but I'm a bit stumped as to how to actually implement it in a Python script.

For example, if I have a script

import sys
import json

flowFile = sys.stdin.read()
try: 
    j = json.loads(flowFile)
    sys.stdout.write(j)
except:
    sys.stdout.write(flowFile)

would I then implement the memory limit as 

import sys
import json

def memory_limit():
    soft, hard = resource.getrlimit(resource.RLIMIT_AS)
    resource.setrlimit(resource.RLIMIT_AS, (get_memory() * 1024 / 2, hard))

def get_memory():
    with open('/proc/meminfo', 'r') as mem:
        free_memory = 0
        for i in mem:
            sline = i.split()
            if str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'):
                free_memory += int(sline[1])
    return free_memory

if __name__ == '__main__':
    memory_limit() # Limitates maximun memory usage to half
    try:
        main()
    except MemoryError:
        sys.stderr.write('\n\nERROR: Memory Exception\n')
        sys.exit(1)

flowFile = sys.stdin.read()
try: 
    j = json.loads(flowFile)
    sys.stdout.write(j)
except:
    sys.stdout.write(flowFile)

or something else entirely?

carousallie
  • 723
  • 1
  • 6
  • 16
  • Have you tried the code given? Do you have a specific problem with it? As it is, this reads as a duplicate of the pre-existing question you linked (and whose answer is included, in adapted form, in the question). Make sure you're asking something new and different, f/e, about a specific problem you had when applying that answer. – Charles Duffy Apr 10 '20 at 14:50
  • The only thing obviously wrong here is that your own code should be indented under the `if __name__ == '__main__':` – Charles Duffy Apr 10 '20 at 14:54

0 Answers0