chore: devenv environment

[joscha]

Description

This adds a Devenv environment to the repository which allows immediate development w/o have to worry about local installation of dependencies (make, python, poetry, etc.).
Devenv is using nix under the hood, which provides a reproducible environment due to the fact that it controls the whole dependency closure. That means that the files in this PR would ensure that all developers would use the same version of make, python, poetry, etc. for their local development of dlt, which reduces a whole bunch of problems when talking about issues, opening pull requests, etc.

If the system has direnv and devenv installed everything is automatic.
After activating the environment, setup can be run, which calls out to make dev.

Also happy to swap this for a raw nix flake or even a nix-shell if devenv is too bleeding edge.

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	9141ee4
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/673f1e84440fe800097c7c89

[joscha]

this runs the linter before commit. Can be omitted by passing -n, however I am also happy to remove this, if in question.

[sh-rp]

Hey @joscha, since no one has replied here yet, maybe my take on this: Currently we are using poetry where you can work within a shell, so the prerequsites are to have poetry installed and nothing more, so I am not sure we need any other system to manage dependencies or development environment. Are you working within some kind of browser based IDE or what do you need the nix setup for?

[joscha]

Nix is controlling the whole dependency closure, not only poetry. E.g. it also controls:

• What version of Python is used
• What version of make is used
• What version of bash for shell scripts
• Etc.

Basically any dependency (direct AND transitive) in the repository.
The reason this is helpful is manifold, but my top one is:
Whenever I work within dlt I get the blessed python version with all transitive dependencies, independently of what my system version is at the time. I also don't litter my global system with project-specific tooling.

[sh-rp]

Ok, since poetry manages all python dependencies and dlt should work with all currently supported python versions and any make versions I don't really see the need in this project at the moment, but if anyone on the core team wants to review and maintain this that would be fine of course. I just have the suspicion that no one is using / is familiar with nix.

[joscha]

since poetry manages all python dependencies and dlt should work with all currently supported python versions

I am not a poetry pro, it doesn't manage python itself, does it? you probably have something like pyenv installed on your system or so to do that?

I don't really see the need in this project at the moment

The binaries in my original post are just an example.

We also have bash (and some dependent tooling like sed, git and column (probably some more) in

• tools/check-package.sh
• tools/poetry-deps.sh

node and netlify in

• docs/website

and a myriad of external dependencies (ODBC driver, Docker, etc.) in various spaces.

Whenever you have a new member onboarding the dlt project I am sure there is a whole bunch of tooling, etc. to be installed if they want to be successful in the project. The versions they install (and the downstream closure of all transitive dependencies) depends on:

• which system they install it on
• When they install it
• In what order they install it

I think dlt & other projects in the dlthub space have more than enough tools, etc. to warrant controlling that closure.
You can see nix as the poetry.lock file for all your dev tooling, making each state of the repo reproducible.

Example:

If dlt introduces support for another python version, we'd update that lock file. When you then in a year's time after dlt 4.0 has been released have to go back to do a 1.x patch, you can just go back in time in git history and your old python version (and all other dependencies) will be available immediately without having to change all of these dependencies on your computer. You could even do git bisect to automatedly find an issue that was introduced, not having to worry about all the changes to currently unmanaged dependencies.

I just have the suspicion that no one is using / is familiar with nix.

Time to learn something new? But also with direnv enabled, it is an easy start, you have nix and direnv as the only dependencies on your system and the rest goes by itself :)

[sh-rp]

Time to learn something new? But also with direnv enabled, it is an easy start, you have nix and direnv as the only dependencies on your system and the rest goes by itself :)

Sure, but you'll have to let the people contributing here decide for themselves how they need to invest their time. Like I said, if someone wants to take it it might be merged..

[joscha]

you'll have to let the people contributing here decide for themselves

Of course, I am advocating for it, because I am convinced it will actually save time. The PR only contains a scaffold for the python/poetry/make part. If I haven't piqued your interest with the value proposition in the response above, maybe the pain in the aforementioned points hasn't been felt enough, yet. As someone who switches between open source projects regularly with every single one using a different runtime versions / package managers / binaries it becomes more obvious.