Rapidly Testing Python Packages As Modules

This is a post I’ll file under “How on Earth did I go so long without knowing this?” when it comes to working with Python packages. The majority of the Python code that I write tends to be backend code that is containerized. As I had alluded to in my post on deploying Groovy applications, containerizing a Python application generally just involves using a multi-stage deployment to install and compile any 3rd party libraries, copy the code into the container, and you’re basically done. This means trying out changes to the code is simply when developing locally since I can just manually execute it.

Recently, though, I ended up writing a CLI application that some members of my team would use for onboarding new customers to our monitoring platform. It can ingest an Excel file with their site information that the customer fills out during the onboarding, a YAML file with various metadata we need, and it builds out the framework of the customer, their locations, their base-level Dashboards, their reports, roles, etc. using the monitoring platform’s REST API. Since other folks need to run this, I figured I would bundle it up as a Python wheel file for ease of installation.

The most immediate difference when using Python like this is that it changes how your local imports need to be written. In the root of my project directory, for example, I have a folder called src that contains my Python files:

├── src/
│   ├── __init__.py
│   ├── main.py
│   └── file_name.py
├── pyproject.toml
└── ... (other project files)

Inside of main.py, I need to import something. Instead of just doing:

from file_name import ClassName

I have to specify the module in order for the bundled package to work:

from src.file_name import ClassName

Doing the imports this way allows them to work properly from the module once the package is been built with something like:

python -m build

However, it means that you cannot directly execute main.py like this:

python ./src/main.py

Naturally, though, when making some changes or troubleshooting a bug, no one wants to have to go through the process of:

1. Build the project
2. Install the wheel
3. Test the change
4. Tweak the code
5. Uninstall the wheel
6. Build the project
7. Install the wheel
8. Test the change

That would get old super fast. Instead, the solution is to simply tell Python to run the code as a module. To test from the CLI, I can simply use the -m parameter:

python -m src.main