Python - Register functions from imported module

Question

I'm trying to determine the best way to implement my application where there will be a main program (main.py) and a separate rules module (rules.py) where an arbitrary number of rules can be written and they will all be applied in main.py to manipulate some data (a dictionary). A user of the application could add their custom rules in rules.py without impacting the logic in main.py.

I was thinking decorators would be useful here to register each function in rules.py in a way that main.py could iterate over them, but I'm not certain of the exact implementation. Here is my skeleton code.

main.py

import rules

modifiers = [] # List of fuctions to modify data

def add_modifier(f):
  modifiers.append(f)
  return f

def invoke_modifiers(data):
  for modifier in modifiers:
    data = modifier(data)
  return data

if __name__ == "__main__":
  data = {'foo': 'bar'}
  print(f"Invoking modifiers on data: {data}")
  data = invoke_modifiers(data)
  print(f"Done invoking modifiers: {data}")

rules.py

from main import add_modifier

@add_modifier
def mod1(data):
  data['foo'] = 'baz'
  return data

@add_modifier
def mod2(data):
  data['quz'] = 'qux'
  return data

But when I execute the code, it doesn't modify my data.

$ python main.py
Invoking modifiers on data: {'foo': 'bar'}
Done invoking modifiers: {'foo': 'bar'}

So my questions are two-fold:

Is this a good way to go about having user-defined functions outside of the main application?
What needs to change to get the data to be modified by mod1 and mod2 in the rules.py module?

EDIT

If I omit the from main import add_modifier in rules.py, I get the following during execution:

Traceback (most recent call last):
  File "main.py", line 3, in <module>
    import rules
  File "/home/telorb/Python/registerTest/rules.py", line 3, in <module>
    @add_modifier
NameError: name 'add_modifier' is not defined

Looks like you have a circular import. This is probably throwing an error in your rules.py. — Hofbr, Aug 25 '20 at 17:09
See my edit for the result of omitting the import statement from rules.py. Is there another way to do it? — Rusty Lemur, Aug 25 '20 at 17:13

score 1 · Answer 1 · answered Aug 25 '20 at 17:58

I'm not really that familiar with decorators, so perhaps someone else may be able to advise on this.

But considering what you're doing, I think creating a class for users to add functions to would the have the functionality you're looking for.

Something this thread: Is there a way to loop through and execute all of the functions in a Python class?

One suggested solution does utilize decorators, so that may clarify how better to build out your structure.

score 0 · Answer 2 · answered Aug 25 '20 at 18:30

For now I'm going with a solution found here: How can I decorate all functions imported from a file?

main.py

import types
import functools

modifiers = [] # List of fuctions to modify data

def decorate_all_in_module(module, decorator):
    for name in dir(module):
        obj = getattr(module, name)
        if isinstance(obj, types.FunctionType):
            setattr(module, name, decorator(obj))

def add_modifier(f):
  print(f"Adding modifier: {f}")
  modifiers.append(f)
  return f

def invoke_modifiers(data):
  print(f'Invoking {len(modifiers)} modifiers')
  for modifier in modifiers:
    print(f'Invoking {modifier}')
    data = modifier(data)
  return data

@add_modifier
def mod_main(data):
  data['xray'] = 'zulu'
  return data

import rules

if __name__ == "__main__":
  decorate_all_in_module(rules, add_modifier)
  data = {'foo': 'bar'}
  print(f"Invoking modifiers on data: {data}")
  data = invoke_modifiers(data)
  print(f"Done invoking modifiers: {data}")

rules.py

def mod1(data):
  data['foo'] = 'baz'
  return data

def mod2(data):
  data['quz'] = 'qux'
  return data

$ python main.py

Adding modifier: <function mod_main at 0x7fe69ce75160>
Adding modifier: <function mod1 at 0x7fe69ce75040>
Adding modifier: <function mod2 at 0x7fe69ce75700>
Invoking modifiers on data: {'foo': 'bar'}
Invoking 3 modifiers
Invoking <function mod_main at 0x7fe69ce75160>
Invoking <function mod1 at 0x7fe69ce75040>
Invoking <function mod2 at 0x7fe69ce75700>
Done invoking modifiers: {'foo': 'baz', 'xray': 'zulu', 'quz': 'qux'}

azelcer · Answer 3 · 2021-11-19T19:37:18.367

Python import machinery is quite complicated, but in this case it is quite simple to explain why the original code does not work.

When main.py is invoked, its module name is "__main__" and not "main". When rules is imported and in turn it imports main, a new module named "main" (no underscores) is created, which has its own instance of modifiers. You can easily check this by putting print("modifiers id = ", id(modifiers)) to add_modifier into the main block.

I think the best solution here is to make a new module that manages modifiers:

main.py

from modifiers import invoke_modifiers
import rules

if __name__ == "__main__":
  data = {'foo': 'bar'}
  print(f"Invoking modifiers on data: {data}")
  data = invoke_modifiers(data)
  print(f"Done invoking modifiers: {data}")

rules.py

from modifiers import add_modifier

@add_modifier
def mod1(data):
  data['foo'] = 'baz'
  return data

@add_modifier
def mod2(data):
  data['quz'] = 'qux'
  return data

modifiers.py

_modifiers = [] # List of fuctions to modify data

def add_modifier(f):
  _modifiers.append(f)
  return f


def invoke_modifiers(data):
  for modifier in _modifiers:
    data = modifier(data)
  return data

Result

azelcer@ZX80:~$ python main.py
Invoking modifiers on data: {'foo': 'bar'}
Done invoking modifiers: {'foo': 'baz', 'quz': 'qux'}

Python - Register functions from imported module

3 Answers3