Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

include flowdir in syspath while loading file in Runner API #2182

Merged
merged 4 commits into from
Dec 18, 2024

Conversation

madhur-ob
Copy link
Collaborator

@madhur-ob madhur-ob commented Dec 17, 2024

Context:

The cwd argument of the Runner is for selecting the directory under which the subprocess executes in..

But, it doesn't play a role in loading of the flow file itself -- which is the first step of the lower level API to generate the command that will be executed via subprocess..

aka loading of file --> generate command --> run via subprocess (cwd is only relevant here)

Thus, in the directory structure like this:

├── pipelines/
│   ├── training.py
│   └── common.py
└── tests/
    └── test_sample.py

if training.py has from common import packages
and test_sample.py uses the Runner API as follows:

with Runner("pipelines/training.py", environment="conda").run() as running:
        return running.run.data

the following issue is encountered:

>   from common import packages
E   ModuleNotFoundError: No module named 'common'

This PR adds the flow directory into sys.path and also removes it later...

@savingoyal savingoyal merged commit 20406f4 into Netflix:master Dec 18, 2024
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants