3

I have a project hierarchy like below, when I run python src/bot/main I got no error. While if I run python -m src.bot.main I got error. Why?

This is my file hierarchy:

MyProject
└── src
    ├── __init__.py
    ├── bot
    │   ├── __init__.py
    │   ├── main.py
    │   └── sib1.py
    └── mod
        ├── __init__.py
        └── module1.py

This is the content of main.py:

import sys
    

if __name__ == "__main__":

    # sys.path will print the parent folder.
    print(sys.path, end="\n\n")
    
    # my problem is on this line.
    import sib1
    sib1.test()

The error:

Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/me/Desktop/test_py/src/bot/main.py", line 16, in <module>
    import sib1
ModuleNotFoundError: No module named 'sib1'

Some conclusion I've made so far:

Since the output of sys.path in both cases include /Users/me/Desktop/MyProject, the reason should not related to scope?

The output of sys.path of both python -m src.bot.main and python src/bot/main:

(test) ✔ me Desktop/test_py % python -m src.bot.main
['/Users/me/Desktop/test_py', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python39.zip', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/lib-dynload', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/site-packages']
VimNing
  • 1,192
  • 4
  • 20
  • 47
  • 1
    Using `python -m src.bot.main` tells Python that `src` is a top-level package. Everything below `src` in the directory structure will be considered submodules/subpackages of `src`. The proper name for `sib1` under that organization is `src.bot.sib1`. No top-level module named `sib1` exists as far as Python is concerned. – Brian Sep 07 '21 at 23:12
  • @Brian: I recently started to think about project structure in Python more deeply. May I ask you that if I just want to build a personal project(I'm not intended to publish it to PyPI) and given that I append an `if __name__=='__main__'` in each file for testing, then which one should I use `python -m ...` or just `python ...` when I run each as a script? And if you will you can put it into an answer I can provide you a bounty. – VimNing Sep 07 '21 at 23:17
  • 1
    Using `python -m` is required if you want to Python to be aware of the package structure. If you only need to run submodules for testing purpose, you could consider using more robust testing tools like [doctest](https://docs.python.org/3/library/doctest.html) (lightweight) or [unittest](https://docs.python.org/3/library/unittest.html) (heavyweight). – Brian Sep 08 '21 at 00:10
  • 1
    You'll also probably want to pick a more informative package name than `src`. Placing you package files under `MyProject/src/MyProject` or just `MyProject/MyProject` would be typical. Just make sure that the directory containing `MyProject` (the package) is on your Python path. You'll then be able to use `import MyProject.bot.sib1` and the like. – Brian Sep 08 '21 at 00:16
  • 1
    Also, you can rename `main.py` to [`__main__.py`](https://docs.python.org/3/library/__main__.html) to make a package directly executable. E.g., `python -m MyProject.bot` would execute `MyProject.bot.__main__`. – Brian Sep 08 '21 at 00:17
  • Some additional helpful reading on structuring packages: https://docs.python.org/3/tutorial/modules.html#packages and https://packaging.python.org/tutorials/packaging-projects/ – Brian Sep 08 '21 at 00:20
  • First, thanks for your kindly links to testing tools! (I've planned to find&learn some of them after I can understand this project structure topic. I'm building a bot for an messaging mobile app btw) Consider `Just make sure that the directory containing MyProject (the package) is on your Python path.` so when I run `python -m ...` this command does this for me? – VimNing Sep 08 '21 at 00:20
  • @Brian: You might put your comments into an answer because indeed you solve my confusion, and this confusion is not easy for me to express in my OP. – VimNing Sep 08 '21 at 00:25
  • 1
    Re: python path: the package you're running via `python -m` needs to be in one of the directories in `sys.path`. Otherwise, Python won't be able to find it. You're current working directory (`~/Desktop/test_py` in your example) is always on your Python path, so if you're in the directory containing `MyProject`, using `python -m MyProject.x.y.z` will work. If you're in any other directory, you'll need to add the package directory to your Python path manually. Many IDEs let you do this in the run configuration settings. Alternatively, you can set the env var `PYTHONPATH`. – Brian Sep 08 '21 at 00:28

1 Answers1

2

I will try my best to clarify each of my confusions in a Q&A form, and organize @Brain's comments and during the way:

Q1: when I run python src/bot/main I got no error.

The sys.path will include the current directory containing the file main.py, i.e. the interpreter will see the file MyProject/src/bot:

import sib1

is (logically) equivalent to:

import /path/to/MyProject/src/bot/sib1.py

Hence, no error.

BUT this is called IMPLICIT-RELATIVE-IMPORT, which should be used only on the main script. See Example.

Q2: While if I run python -m src.bot.main I got error. Why?

Now it's time to quote @Brain's valuable (first) comment:

Using python -m src.bot.main tells Python that src is a top-level package. Everything below src in the directory structure will be considered submodules/subpackages of src. The proper name for sib1 under that organization is src.bot.sib1. No top-level module named sib1 exists as far as Python is concerned.

(emphasis by me)

So:

  • The -m option will provide some context information(details, I recommend reading the first two high-upvoted answers entirely) for the file you're going to run.
  • Now every absolute (import-)path that does not begin with src. will be regarded as built-in or third-party libraries installed in your virtual environment. Since I didn't install any package called sib1, you got the error. (Ignore about the inclusion of the current directory in the sys.path. I guess this is for compatibility reason. Remember that we DONT want IMPLICIT-RELATIVE-IMPORT in python3)
  • You can use relative path BUT you should not go out of the src (since src is the top-level package in the execution) by prepending too many .'s.

For example this will work:

from . import sib1
from ..mod import module1    # The `..` is equivalent to the MyProject/src.

module1.hello()
sib1.test()

Finally, don't test your package by inserting many if __name__ == '__main__'. Do so by professional tools:

If you only need to run submodules for testing purpose, you could consider using more robust testing tools like doctest (lightweight) or unittest (heavyweight).

VimNing
  • 1,192
  • 4
  • 20
  • 47