Python dataclasses do not have built-in validation, like a Pydantic class. You can still use type hints to define variables, like name: str = None, but it has no actual effect on the dataclass.
You can use the __post_init__(self) method of a dataclass to perform data validation. A few examples below:
My personal knowledgebase. Use the sections along the top (i.e. Programming) to navigate areas of the KB.
\ud83d\udc0d Python
Python standard project filesnoxfile.pyruffpython .gitignoredynaconf\ud83c\udd7f\ufe0f Powershell
Warning
In progress...
"},{"location":"programming/index.html","title":"Programming","text":"Notes & code snippets I want to remember/reference.
Use the sections along the left to read through my notes and see code examples. You are in the Programming section, but might want to check out my Standard project files for Docker (as an example).
Warning
In progress...
Todo
Notes, links, & reference code for Docker/Docker Compose.
Warning
In progress...
Todo
dev vs prodENV vs ARGEXPOSECMD vs ENTRYPOINT vs RUNdocker-compose.ymlversion (required) and volumes (optional))depends_on)You can take advantage of Docker's BuildKit, which caches Docker layers so subsequent rebuilds with docker build are much faster. BuildKit works by keeping a cache of the \"layers\" in your Docker container, rebuilding a layer only if changes have been made. What this means in practice is that you can separate the steps you use to build your container into stages like base, build, and run, and if nothing in your build layer has changed (i.e. no new dependencies added), that layer will not be rebuilt.
In this example, I am building a simple Python app inside a Docker container. The Python code itself does not matter for this example.
To illustrate the differences in a multistage Dockerfile, let's start with a \"flat\" Dockerfile, and modify it with build layers. This is the basic Dockerfile:
Example flat Dockerfile## Start with a Python 3.11 Docker base\nFROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\n## Set the CWD inside the container\nWORKDIR /app\n\n## Copy Python requirements.txt file into container & install dependencies\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\n## Copy project source from host into container\nCOPY ./src .\n\n## Expose port 8000, which we can pretend the Python app uses to serve the application\nEXPOSE 8000\n## Run the Python app\nCMD [\"python\", \"main.py\"]\nIn this example, any changes to the code or dependencies will cause the entire container to rebuild each time. This is slow & inefficient, and leads to a larger container image. We can break these stages into multiple build layers. In the example below, the container is built in 3 \"stages\": base, build, and run:
## Start with the python:3.11-slim Docker image as your \"base\" layer\nFROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\n## Create a \"build\" layer, where you setup your Python environment\nFROM base AS build\n\n## Set the CWD inside the container\nWORKDIR /app\n\n## Copy Python requirements.txt file into container & install dependencies\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\n## Inherit from the build layer\nFROM build AS run\n\n## Set the CWD inside the container\nWORKDIR /app\n\n## Copy project source from host into container\nCOPY ./src .\n\n## Expose port 8000, which we can pretend the Python app uses to serve the application\nEXPOSE 8000\n## Run the Python app\nCMD [\"python\", \"main.py\"]\nLayers:
base: The base layer provides a common environment for the rest of the layers.ENV variables, which persist across layersARG lines can be set per-layer, and will need to be re-set for each new layer. This example does not use any ARG lines, but be aware that build arguments you set with ARG are only present for the layer they are declared in. If you create a new layer and want to access the same argument, you will need to set the ARG value again in the new layerbuild: The build layer is where you install your Python dependencies.apt/apt-getpython:3.11-slim base image is built on Debian. If you are using a different Dockerfile, i.e. python:3.11-alpine, use the appropriate package manager (i.e. apk for Alpine, rpm for Fedora/OpenSuSE, etc) to install packages in the build layerrun: Finally, the run layer executes the code built in the previous base & build steps. It also exposes port 8000 inside the container to the host, which can be mapped with docker run -p 1234:8000, where 1234 is the port on your host you want to map to port 8000 inside the container.Using this method, each time you run docker build after the first, only layers that have changed in some way will trigger a rebuild. For example, if you add a Python dependency with pip install <pkg> and update the requirements.txt file with pip freeze > requirements.txt, the build layer will be rebuilt. If you make changes to your Python application, the run layer will be rebuilt. Each layer that does not need to be rebuilt reduces the overall build time of the container, and only the run layer will be saved as your image, leading to smaller Docker images.
With multistage builds, you can also create a dev and prod layer, which you can target with docker run or a docker-compose.yml file. This allows you to build the development & production version of an application using the same Dockerfile.
Let's modify the multistage Dockerfile example from above to add a dev and prod layer. Modifications to the multistage Dockerfile include adding an ENV variable for storing the app's environment (dev/prod). In my projects, I use Dynaconf to manage app configurations depending on my environment. Dynaconf allows you to set an ENV variable called $ENV_FOR_DYNACONF so you can control app configurations per-environment (Dynaconf environment docs).
FROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\nFROM base AS build\n\nWORKDIR /app\n\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\n## Use target: dev to build this step\nFROM build AS dev\n\n## Set the Dynaconf env to dev\nENV ENV_FOR_DYNACONF=dev\n## Tell Dynaconf to always load from the environment first while in the container\nENV DYNACONF_ALWAYS_LOAD_ENV_VARS=True\n\nWORKDIR /app\nCOPY ./src .\n\n############\n# Optional #\n############\n# Export ports, set an entrypoint/CMD, etc\n# Note: This is normally handled by your orchestrator (docker-compose, Azure Container App, etc).\n# If you are buliding/running the container directly, uncomment the EXPOSE & COMMAND lines below\n\n# EXPOSE 5000\n# CMD [\"python\", \"main.py\"]\n\n## Use target: prod to build this step\nFROM build AS prod\n\n## Set the Dynaconf env to prod\nENV ENV_FOR_DYNACONF=prod\n## Tell Dynaconf to always load from the environment first while in the container\nENV DYNACONF_ALWAYS_LOAD_ENV_VARS=True\n\nWORKDIR /app\nCOPY ./src .\n\n############\n# Optional #\n############\n# Export ports, set an entrypoint/CMD, etc\n# Note: This is normally handled by your orchestrator (docker-compose, Azure Container App, etc)\n# If you are buliding/running the container directly, uncomment the EXPOSE & COMMAND lines below\n\n# EXPOSE 5000\n# CMD [\"python\", \"main.py\"]\nWith this multistage Dockerfile, you can target a specific layer with docker built --target <layer-name> (i.e. docker build --target dev). This will run through the base and build layers, but skip the prod layer.
You can also target a specific layer in a docker-compose.yml file:
version: \"3.8\"\n\nservices:\n\n my_app:\n container_name: my-application\n restart: unless-stopped\n build:\n ## The \"root\" directory for Docker compose. If your Dockerfile/project\n # are in a subdirectory, specify it here.\n context: .\n ## Set the name/path to the Dockerfile, keeping in mind the context you set above\n dockerfile: Dockerfile\n ## Target the \"dev\" layer of the Dockerfile\n target: dev\n ## Set the working directory inside the container to /app\n working_dir: /app\n ## Set the command to run inside the container. Equivalent to CMD in the Dockerfile\n command: python main.py\n volumes:\n ## Mount the project's code directory in the container so changes don't require a rebuild\n - ./src:/app\n ports:\n ## Expose port 8000 in the container, set to port 80 on the host\n - 80:8000\n ...\nThe example docker-compose.yml file above demonstrates targeting the dev layer of the multistage Dockerfile above it. We also set the entrypoint (instead of using CMD in the Dockerfile), and expose port 8000 in the container.
The ENV and ARG commands in a Dockerfile can be used to control how an image is built and how it functions when live. The differences between an ENV and an ARG are outlined below.
Note
This list is not a complete comparison between ENV and ARG. For more information, please check the Docker build documentation guide.
ENV$ENV_VAR_NAME.docker build -e, or the environment: stanza in a docker-compose.yml file.build and run phases when building a container.docker build or docker compose build), ENV variables will always use the value declared in the Dockerfile.docker run or docker compose up), the values can be overridden with docker run -e/--env or the environment: stanza in a docker-compose.yml file.ARGdocker build --build-arg ARG_NAME=value, or the build: args: stanza in a docker-compose.yml fileExample:
ENV vs ARGFROM python:3.11-slim AS base\n\n## This env variable will be available in the build layer\nENV PYTHONDONTWRITEBYTECODE 1\n\n## Define a required ARG, without setting its value. The build will fail if this arg is not passed\nARG SOME_VAR\n## Define an ARG and set a default value\nARG SOME_OTHER_VAR_ARG=1.0\n\n## Set an ENV value, using an ARG's value, to make it available throughout the rest of the build\nENV SOME_OTHER_VAR $SOME_OTHER_VAR_ARG\n\nFROM base AS build\n\n## Re-define SOME_OTHER_VAR_ARG from the SOME_OTHER_VAR ENV variable.\n# The ENV variable carries into the build layer, but the ARG defined\n# in the base layer is not.\nARG SOME_OTHER_VAR_ARG=$SOME_OTHER_VAR\nBuild ARGS are useful for setting things like a software version number, i.e. when downloading a specific software release from Github. You can set a build arg for the release version, i.e. ARG RELEASE_VER, and provide it at buildtime with docker build --build-arg RELEASE_VER=1.2.3, or in a docker-compose.yml file like:
...\n\nservices:\n\n service1:\n build:\n context: .\n args:\n RELEASE_VER: 1.2.3\n\n...\nENV variables, meanwhile, can store things like a database password or some other secret, or configurations for the app.
...\n\nservices:\n\n service1:\n container_name: service1\n restart: unless-stopped\n build:\n ...\n ...\n environment:\n ## Load $RELEASE_VERSION from the host's environment or a .env file\n RELEASE_VER: ${RELEASE_VERSION:-1.2.3}\n\n...\nIn previous examples you have seen the EXPOSE line in a Dockerfile. This command exposes a network port from within the container to the host. This is useful if your containerized application utilizes network ports (i.e. running a web frontend on port 8000), and you are running the container directly with docker run instead of through an orchestrator like Docker Compose or Kubernetes.
Note
When using an orchestrator like docker-compose, kubernetes, hashicorp nomad, etc, it is not necessary (and often counterproductive) to define EXPOSE lines in a Dockerfile. It is better to define port binds between the host and container using the orchestrator's capabilities, i.e. the ports: stanza of a docker-compose.yml file.
When building & running a container image locally or without an orchestrator, you can add these sections to a Dockerfile so when you run the built container image, you can bind ports with docker run -p $HOST_PORT:$CONTAINER_PORT.
Example:
Example EXPOSE syntax...\n\nFROM build AS run\n\n...\n\n## Expose port 8000 in the container to the host running this container image\nEXPOSE 8000\n\n## Start a Uvicorn server inside the container. The web server runs on port 8000 (by default)\nCMD [\"uvicorn\", \"main:app\", \"--reload\"]\nAfter building this container, you can run it and bind to a port on the host (i.e. port 80) with docker run -rm -p 80:8000 ..., or by specifying the port binding in a docker-compose.yml file
Warning
If you are using Docker Compose, comment/remove the EXPOSE and CMD lines in your container and pass the values in through Docker Compose
...\n\nservices:\n\n service1:\n ...\n ## Set the container startup command here, instead of with RUN in the Dockerfile\n command: uvicorn main:app --reload\n ports:\n ## Serve the container application running on port 8000 in the container\n # over port 80 on the host.\n - 80:8000\nRUNRUN is executed on top of the current base image.neovim container inside of a Dockerfile built on top of ubuntu:latest image: RUN apt-get update -y && apt-get install -y neovimRUN show their output in the console as the container is built, but are not executed when the built container image is run with docker run or docker compose upCMDCMD execute when you run the container with docker run or docker compose updocker run myimage cat log.txtcat log.txt command overrides the CMD defined in the containerCMD defined in your image is executed. If you specify more than one, all but the last CMD will execute.CMD command supersedes ENTRYPOINT, and should almost always be used instead of ENTRYPOINT.ENTRYPOINTENTRYPOINT functions almost the same way as CMD, but should be used only when extending an existing image (i.e. nginx, tomcat, etc)ENTRYPOINT to an existing container will change the way that container executes, running the underlying Dockerfile logic with an ad-hoc command you provide.docker run --entrypointCMD and an ENTRYPOINT, the CMD line will be provided as arguments to the ENTRYPOINT, meaning you can do things like cat a file with a CMD, then \"pipe\" the command's output into an ENTRYPOINTCMD in your Dockerfiles, unless you are aware of and fully understand a reason to use ENTRYPOINT instead.Doc pages for specific containers/container stacks I use.
"},{"location":"programming/docker/my_containers/portainer/portainer.html","title":"Portainer","text":"At least 1 Portainer server must be available for agents to connect to. Copy this script to a file, i.e. run-portainer.sh.
Note
Don't forget to set chmod +x run-portainer.sh, or execute the script with bash run-portainer.sh.
#!/bin/bash\n\nWEBUI_PORT=\"9000\"\n## Defaults to 'portainer' if empty\nCONTAINER_NAME=\n## Defaults to a named volume, portainer_data.\n# Note: create this volume with $ docker volume create portainer_data\nDATA_DIR=\n\necho \"\"\necho \"Checking for new image\"\necho \"\"\n\ndocker pull portainer/portainer-ce\n\necho \"\"\necho \"Restarting Portainer\"\necho \"\"\n\ndocker stop portainer && docker rm portainer\n\ndocker run -d \\\n -p 8000:8000 \\\n -p ${WEBUI_PORT:-9000}:9000 \\\n --name=${CONTAINER_NAME:-portainer} \\\n --restart=unless-stopped \\\n -v /var/run/docker.sock:/var/run/docker.sock \\\n -v ${DATA_DIR:-portainer_data}:/data \\\n portainer/portainer-ce\nStart a Portainer in agent mode to allow connection from a Portainer server. This setup is done in the Portainer server's webUI.
Warning
It is probably easier to just download the agent script from the Portainer server when you are adding a connection. It offers a command you can run to simplify setup.
run-portainer_agent.sh#!/bin/bash\n\necho \"\"\necho \"Checking for new container image\"\necho \"\"\n\ndocker pull portainer/agent\n\necho \"\"\necho \"Restarting Portainer\"\necho \"\"\n\ndocker stop portainer-agent && docker rm portainer-agent\n\ndocker run -d \\\n -p 9001:9001 \\\n --name=portainer-agent \\\n --restart=unless-stopped \\\n -v /var/run/docker.sock:/var/run/docker.sock \\\n -v /var/lib/docker/volumes:/var/lib/docker/volumes \\\n portainer/agent:latest\nRun this script to backup the Portainer portainer_data volume. Backup will be placed at ${CWD}/portainer_data_backup
#!/bin/bash\n\n# Name of container containing volume(s) to back up\nCONTAINER_NAME=${1:-portainer}\nTHIS_DIR=${PWD}\nBACKUP_DIR=$THIS_DIR\"/portainer_data_backup\"\n# Directory to back up in container\nCONTAINER_BACKUP_DIR=${2:-/data}\n# Container image to use as temporary backup mount container\nBACKUP_IMAGE=${3:-busybox}\nBACKUP_METHOD=${4:-tar}\nDATA_VOLUME_NAME=${5:-portainer-data}\n\nif [[ ! -d $BACKUP_DIR ]]; then\n echo \"\"\n echo $BACKUP_DIR\" does not exist. Creating.\"\n echo \"\"\n\n mkdir -pv $BACKUP_DIR\nfi\n\nfunction RUN_BACKUP () {\n\n sudo docker run --rm --volumes-from $1 -v $BACKUP_DIR:/backup $BACKUP_IMAGE $2 /backup/backup.tar $CONTAINER_BACKUP_DIR\n\n}\n\nfunction RESTORE_BACKUP () {\n\n echo \"\"\n echo \"The restore function is experimental until this comment is removed.\"\n echo \"\"\n read -p \"Do you want to continue? Y/N: \" choice\n\n case $choice in\n [yY] | [YyEeSs])\n echo \"\"\n echo \"Test print: \"\n echo \"sudo docker create -v $CONTAINER_BACKUP_DIR --name $DATA_VOLUME_NAME\"2\" $BACKUP_IMAGE true\"\n echo \"\"\n echo \"Test print: \"\n echo \"sudo docker run --rm --volumes-from $DATA_VOLUME_NAME\"2\" -v $BACKUP_DIR:/backup $BACKUP_IMAGE tar xvf /backup/backup.tar\"\n echo \"\"\n\n echo \"\"\n echo \"Compare to original container: \"\n echo \"\"\n echo \"Test print: \"\n echo \"sudo docker run --rm --volumes-from $CONTAINER_NAME -v $BACKUP_DIR:/backup $BACKUP_IMAGE ls /data\"\n ;;\n [nN] | [NnOo])\n echo \"\"\n echo \"Ok, nevermind.\"\n echo \"\"\n ;;\n esac\n\n}\n\n# Run a temporary container, mount volume to back up, create backup file\ncase $1 in\n \"-b\" | \"--backup\")\n case $BACKUP_METHOD in\n \"tar\")\n echo \"\"\n echo \"Running \"$BACKUP_METHOD\" backup using image \"$BACKUP_IMAGE\n echo \"\"\n\n RUN_BACKUP $CONTAINER_NAME \"tar cvf\"\n ;;\n esac\n ;;\n \"-r\" | \"--restore\")\n ;;\nesac\nNote
I do not use this method. I find it easier to run Portainer with a shell script.
Portainer docker-compose.ymlversion: \"3\"\n\nservices:\n\n portainer:\n image: portainer/portainer-ce:linux-amd64\n container_name: portainer\n restart: unless-stopped\n volumes:\n - /var/run/docker.sock:/var/run/docker.sock\n - ${PORTAINER_DATA:-./data}:/data\nWarning
In progress...
Todo
Warning
In progress...
Todo
virtualenvcondapdmjupyterlabTodo
When importing Python code into a Jupyter notebook, any changes made in the Python module(s) do not become active in the Jupyter kernel until it's restarted. This is undesirable when working with data that takes a long time up front to prepare.
Add this code to a cell (I usually put it in the very first cell) to automatically reload Python modules on changes, without having to reload the whole notebook kernel.
Automatically reload file on changes## Set notebook to auto reload updated modules\n%load_ext autoreload\n%autoreload 2\nTodo
pre-commit runs and what to do to fix itpre-commitWhen running Jupyter notebooks, it's usually good practice to clear the notebook's output when committing to git. This can be for privacy/security (if you're debugging PII or environment variables in the notebook), or just a tidy repository.
Using pre-commit (check my section on pre-commit) and the nbstripout action, we can automate stripping the notebook each time a git commit is created.
Instructions
pre-commitNote
Warning
If your preferred package manager is not listed below, check the documentation for that package manager for instructions on installing packages.
pip:pip install pre-commitpipx:pipx install pre-commitpdm:pdm add pre-commitTODO:
poetryconda/miniconda/microconda.pre-commit-config.yml with these contents:repos:\n repo: https://github.com/kynan/nbstripout\n rev: 0.6.1\n hooks:\n - id: nbstripout\npre-commit hook with $ pre-commit installNote
If you installed pre-commit in a virtualenv, or with a tool like pdm, make sure the .venv is activated, or you run with $ pdm run pre-commit ...
Now, each time you make a git commit, after writing your commit message, pre-commit will execute nbstripout to strip your notebooks of their output.
Warning
In progress...
Todo
mkdocsmkdocs.ymlsite_namesite_descriptionrepo_namerepo_urlexclude_docstheme configuration for mkdocs-materialmkdocs-material featuressearchnavigationcontentpluginsliterate-navsection-indexmarkdown_extensionsWarning
In progress...
reference page for more in-depth documentation.Todo
Get-AdUserTodo
A Powershell profile is a .ps1 file, which does not exist by default but is expected at C:\\Users\\<username>\\Documents\\WindowsPowerShell\\Microsoft.PowerShell_profile.ps1, is a file that is \"sourced\"/loaded each time a Powershell session is opened. You can use a profile to customize your Powershell session. Functions declared in your profile are accessible to your whole session. You can set variables with default values, customize your session's colors (by editing a function Prompt {} section), split behavior between regular/elevated prompts, and more.
At the top of your Powershell script, below your docstring, you can declare param() to enable passing -Args to your script, our switches.
<#\n Description: Example script with params\n\n Usage:\n ...\n#>\n\nparam(\n ## Enable transcription, like ~/.bashhistory\n [bool]$Transcript = $True,\n [bool]$ClearNewSession = $True\n)\nCode snippets with little-to-no explanation. Some pages may link to a more in-depth article in the Powershell Reference page
Export a list of installed packages discovered by winget to a .json file, then import the list to reinstall everything. Useful as a backup, or to move to a new computer.
Note
The filename in the examples below, C:\\path\\to\\winget-pkgs.json, can be named anything you want, as long as it has a .json file extension.
## Set a path, the file must be a .json\n$ExportfilePath = \"C:\\path\\to\\winget-pkgs.json\"\n\n## Export package list\nwinget export -o \"$($ExportfilePath)\"\n## Set the path to your export .json file\n$ImportfilePath = \"C:\\path\\to\\winget-pkgs.json\"\n\n## Import package list\nwinget import -i \"$($ImportfilePath)\"\n(Get-Date) \u2013 (Get-CimInstance Win32_OperatingSystem).LastBootUpTime\nFunctions in your profile will be executed automatically if you call them within the profile, but they are also available to your entire session. For example, the Edit-Profile function function can be executed in any session that loads a profile with that function declared!
The function below returns $True if the current Powershell session is elevated, otherwise returns $False.
function Get-ElevatedShellStatus {\n ## Check if current user is admin\n $Identity = [Security.Principal.WindowsIdentity]::GetCurrent()\n $Principal = New-Object Security.Principal.WindowsPrincipal $Identity\n $AdminUser = $Principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)\n\n return $AdminUser\n}\n\n## Declare variable for references throughout script.\n# Can be used to prevent script from exiting/crashing.\n$isAdmin = $(Get-ElevatedShellStatus)\nfunction Open-AsAdmin {\n <#\n Run command as admin, or start new admin session if no args are passed\n #>\n if ($args.Count -gt 0) { \n $argList = \"& '\" + $args + \"'\"\n Start-Process \"$psHome\\powershell.exe\" -Verb runAs -ArgumentList $argList\n }\n else {\n Start-Process \"$psHome\\powershell.exe\" -Verb runAs\n }\n}\nfunction Edit-Profile {\n <#\n Open current profile.ps1 in PowerShell ISE\n #>\n If ($host.Name -match \"ise\") {\n ## Edit in PowerShell ISE, if available\n $psISE.CurrentPowerShellTab.Files.Add($profile.CurrentUserAllHosts)\n }\n Else {\n ## Edit in Notepad if no PowerShell ISE found\n notepad $profile.CurrentUserAllHosts\n }\n}\nConda is a Python package manager. It's a very useful utility, but I've found adding it to my $PATH or Powershell profile results in a very slow session load in new tabs/windows. Adding the SOURCE_CONDA function below, and settings an alias to the conda command to call this function instead (Set-Alias conda SOURCE_CONDA), delays the sourcing of the Conda path. The first time you run conda in a new session, you will see a message that Conda has been initialized and you need to re-run your command. You can simply press the up key on your keyboard and run it again; now that Conda is initialized, it will execute, and once a Powershell session is loaded, sourcing Conda is much quicker!
function SOURCE_CONDA {\n <#\n Initialize Conda only when the conda command is run.\n Conda takes a while to initialize, and is not needed in\n every PowerShell session\n #>\n param(\n [String]$CONDA_ROOT = \"%USERPROFILE%\\mambaforge\\Scripts\\conda.exe\"\n )\n\n #region conda initialize\n # !! Contents within this block are managed by 'conda init' !!\n (& \"$CONDA_ROOT\" \"shell.powershell\" \"hook\") | Out-String | Invoke-Expression\n #endregion\n\n Write-Host \"Conda initialized. Run your command again.\"\n\n}\nUnix OSes have a very nice, simple command, uptime, that will simply print the number of days/hours/minutes your machine has been online. The Powershell syntax for this is difficult for me to remember, so my Powershell profile has an uptime function declared.
function uptime {\n ## Print system uptime\n\n If ($PSVersionTable.PSVersion.Major -eq 5 ) {\n Get-WmiObject win32_operatingsystem |\n Select-Object @{EXPRESSION = { $_.ConverttoDateTime($_.lastbootuptime) } } | Format-Table -HideTableHeaders\n }\n Else {\n net statistics workstation | Select-String \"since\" | foreach-object { $_.ToString().Replace('Statistics since ', '') }\n }\n}\nUnix OSes have a simple, easy to remember unzip command. This function tries to emulate that simplicity.
function unzip ($file) {\n ## Extract zip archive to current directory\n\n Write-Output(\"Extracting\", $file, \"to\", $pwd)\n $fullFile = Get-ChildItem -Path $pwd -Filter .\\cove.zip | ForEach-Object { $_.FullName }\n Expand-Archive -Path $fullFile -DestinationPath $pwd\n}\nUnix OSes have a useful utility called touch, which will create an empty file if one doesn't exist at the path you pass it, i.. touch ./example.txt. This function tries to emulate that usefulness and simplicity.
function touch($file) {\n ## Create a blank file at $file path\n\n \"\" | Out-File $file -Encoding ASCII\n}\nAdding this function to your Powershell profile lets you lock your computer's screen by simply running lock-screen in a Powershell session.
function lock-machine {\n ## Set computer state to Locked\n\n try {\n rundll32.exe user32.dll, LockWorkStation\n }\n catch {\n Write-Error \"Unhandled exception locking machine. Details: $($_.Exception.Message)\"\n }\n\n}\nConcepts & code snippets I want to document, ideas I don't want to forget, things I've found useful, etc.
For short/quick references with code examples, check the snippets section.
Notes, links, & reference code for Python programming.
Warning
In progress...
"},{"location":"programming/python/ad-hoc-codecell.html","title":"Ad-Hoc, Jupyter-like code \"cells\"","text":"Note
ipykernel package installed.ipykernel (i.e. JetBrains), from what I've read, but I have not tested it anywhere else.You can create ad-hoc \"cells\" in any .py file (i.e. notebook.py, nb.py, etc) by adding # %% line(s) to the file.
# %%\nYou can also create multiple code cells by adding more # %% lines:
# %%\nmsg = \"This code is executed in the first code cell. It sets the value of 'msg' to this string.\"\n\n# %%\n## Display the message\nmsg\n\n# %%\n150 + 150\nNote
\ud83d\udea7Page is still in progress\ud83d\udea7
"},{"location":"programming/python/dataclasses.html#what-is-a-dataclass","title":"What is adataclass?","text":"A Python data class is a regular Python class that has the @dataclass decorator. It is specifically created to hold data (from python.land).
Dataclasses reduce the boilerplate code when creating a Python class. As an example, below are 2 Python classes: the first is written with Python's standard class syntax, and the second is the simplified dataclass:
Standard class vs dataclassfrom dataclasses import dataclass\n\n## Standard Python class\nclass User:\n def __init__(self, name: str, age: int, enabled: bool):\n self.name = name\n self.age = age\n self.enabled = enabled\n\n\n## Python dataclass\n@dataclass\nclass User:\n user: str\n age: int\n enabled: bool\nWith a regular Python class, you must write an __init__() method, define all your parameters, and assign the values to the self object. The dataclass removes the need for this __init__() method and simplifies writing the class.
This example is so simple, it's hard to see the benefits of using a dataclass over a regular class. Dataclasses are a great way to quickly write a \"data container,\" i.e. if you're passing results back from a function:
Example dataclass function returnfrom dataclasses import dataclass\n\n@dataclass\nclass FunctionResults:\n original_value: int\n new_value: int\n\n\ndef some_function(x: int = 0, _add: int = 15) -> FunctionResults:\n y = x + _add\n\n return FunctionResults(original_value=x, new_value=y)\n\nfunction_results: FunctionResults = some_function(x=15)\n\nprint(function_results.new_value) # = 30\nInstead of returning a dict, returning a dataclass allows for accessing parameters using .dot.notation, like function_results.original_value, instead of `function_results[\"original_value\"].
A mixin class is a pre-defined class you define with certain properties/methods, where any class inheriting from this class will have access to those methods.
For example, the DictMixin dataclass below adds a method .as_dict() to any dataclass that inherits from DictMixin.
Adds a .as_dict() method to any dataclass inheriting from this class. This is an alternative to dataclasses.asdict(_dataclass_instance), but also not as flexible.
from dataclasses import dataclass\nfrom typing import Generic, TypeVar\n\n## Generic type for dataclass classes\nT = TypeVar(\"T\")\n\n\n@dataclass\nclass DictMixin:\n \"\"\"Mixin class to add \"as_dict()\" method to classes. Equivalent to .__dict__.\n\n Adds a `.as_dict()` method to classes that inherit from this mixin. For example,\n to add `.as_dict()` method to a parent class, where all children inherit the .as_dict()\n function, declare parent as:\n\n # ``` py linenums=\"1\"\n # @dataclass\n # class Parent(DictMixin):\n # ...\n # ```\n\n # and call like:\n\n # ```py linenums=\"1\"\n # p = Parent()\n # p_dict = p.as_dict()\n # ```\n \"\"\"\n\n def as_dict(self: Generic[T]):\n \"\"\"Return dict representation of a dataclass instance.\n\n Description:\n self (Generic[T]): Any class that inherits from `DictMixin` will automatically have a method `.as_dict()`.\n There are no extra params.\n\n Returns:\n A Python `dict` representation of a Python `dataclass` class.\n\n \"\"\"\n try:\n return self.__dict__.copy()\n\n except Exception as exc:\n raise Exception(\n f\"Unhandled exception converting class instance to dict. Details: {exc}\"\n )\n\n## Demo inheriting from DictMixin\n@dataclass\nclass ExampleDictClass(DictMixin):\n x: int\n y: int\n z: str\n\n\nexample: ExampleDictclass = ExampleDictClass(x=1, y=2, z=\"Hello, world!\")\nexample_dict: dict = example.as_dict()\n\nprint(example_dict) # {\"x\": 1, \"y\": 2, \"z\": \"Hello, world!\"}\nInherit from the json.JSONEncoder class to allow returning a DataClass as a JSON encode-able dict.
import json\nfrom dataclasses import asdict\n\nclass DataclassEncoder(json.JSONEncoder):\n def default(self, obj):\n if hasattr(obj, '__dict__'):\n return obj.__dict__\n elif hasattr(obj, '__dataclass_fields__'):\n return asdict(obj)\n return super().default(obj)\n\nperson = Person(name=\"Alice\", age=25)\njson.dumps(person, cls=DataclassEncoder) # Returns '{\"name\": \"Alice\", \"age\": 25}'\n...
"},{"location":"programming/python/fastapi.html#common-fixes","title":"Common Fixes","text":"...
"},{"location":"programming/python/fastapi.html#fix-failed-to-load-api-definition-not-found-apiv1openapijson","title":"Fix: \"Failed to load API Definition. Not found /api/v1/openapi.json\"","text":"When running behind a proxy (and in some other circumstances) you will get an error when trying to load the /docs endpoint. The error will say \"Failed to load API definition. Fetch error Not Found /api/v1/openapi.json\":
To fix this, simply modify your app = FastAPI() line, adding `openapi_url=\"/docs/openapi\":
...\n\n# app = FastAPI(openapi_url=\"/docs/openapi\")\napp = FastAPI(openapi_url=\"/docs/openapi\")\nTodo
pdmpdm?pdm solve?pdm cheat sheetapplication and a librarypypi with `pdmToc
pdm cheat sheetpdm init Initialize a new project with pdm The pdm tool will ask a series of questions, and build an environment based on your responses pdm add <package-name> Add a Python package to the project's dependencies. pdm will automatically find all required dependencies and add them to the install command, and will ensure the package installed is compatible with your environment. pdm add -d <package-name> Add a Python package to the project's development dependency group. Use this for packages that should not be built with your Python package. Useful for things like formatters (black, ruff, pyflakes, etc), testing suites (pytest, pytest-xidst), automation tools (nox, tox), etc. pdm add -G standard <package-name> Add a Python package to the project's standard group. In this example, standard is a custom dependency group. You can use whatever name you want, and users can install the dependencies in this group with pip install <project-name>[standard] (or with pdm: pdm add <project-name>[standard]) pdm remove <package-name> Remove a Python dependency from the project. Analagous to pip uninstall <package-name> pdm remove -d <package-name> Remove a Python development dependency from the project. pdm remove -G standard <package-name> Remove a Python dependency from the standard (or whatever group name you're targeting) dependency group. standard is an example name. You can use whatever name you want for dependency groups, as long as they adhere to the naming rules. You will be warned if a name is not compatible. pdm update Update all dependencies in the pdm.lock file. pdm update <package-name> Update a specific dependency. pdm update -d <package-name> Update a specific development dependency. pdm update -G standard <package-name> Update a specific dependency in the \"standard\" dependency group. standard is an example name; make sure you're using the right group name. pdm update -d Update all development dependencies. pdm update -G standard Update all dependencies in the dependency group named \"standard\". pdm update -G \"standard,another_group\" Update multiple dependency groups at the same time. pdm lock Lock dependencies to a pdm.lock file. Helps pdm with reproducible builds. pdm run python src/app_name/main.py Run the main.py file using the pdm-controller Python executable. Prepending Python commands with pdm run ensures you are running them with the pdm Python version instead of the system's Python. You can also activate the .venv environment and drop pdm run from the beginning of the command to accomplish the same thing. pdm run custom-script Execute a script defined in pyproject.toml file named custom-script. PDM scripts"},{"location":"programming/python/pdm.html#what-is-pdm","title":"What is PDM?","text":"\ud83d\udd17 PDM official site
\ud83d\udd17 PDM GitHub
PDM (Python Dependency Manager) is a tool that helps manage Python projects & dependencies. It does many things, but a simple way to describe it is that it replaces virtualenv and pip, and ensures when you install dependencies, they will be compatible with the current Python version and other dependencies you add to the project. Some dependencies require other dependencies, and where pip might give you an error about a ModuleNotFound, pdm is smart enough to see that there are extra dependencies and will automatically add them as it installs the dependency you asked for.
Note
Unlike pip, pdm uses a pyproject.toml file to manage your project and its dependencies. This is a flexible file where you can add metadata to your project, create custom pdm scripts, & more.
If you have already used tools like poetry or conda, you are familiar with the problem pdm solves. In a nutshell, dependency management in Python is painful, frustrating, and difficult. pdm (and other similar tools) try to solve this problem using a \"dependency matrix,\" which ensures the dependencies you add to your project remain compatible with each other. This approach helps avoid \"dependency hell,\" keeps your packages isolated to the project they were installed for, and skips the ModuleNotFound error you will see in pip when a dependency requires other dependencies to install.
pdm is also useful for sharing code, as anyone else who uses pdm can install the exact same set of dependencies that you installed on your machine. This is accomplished using a \"lockfile,\" which is a special pdm.lock file the pdm tool creates any time you install a dependency in your project. Unless you manually update the package, any time you run pdm install, it will install the exact same set of dependencies, down to the minor release number, keeping surprise errors at bay if you update a package that does not \"fit\" in your current environment by ensuring a specific, compatible version of that package is installed.
pdm can also help you build your projects and publish them to pypi, and can install your development tools (like black, ruff, pytest, etc) outside of your project's environment, separating development and production dependencies. pdm also makes it easy to create dependency \"groups,\" where a user can install your package with syntax like package_name[group]. You may have seen this syntax in the pandas or uvicorn packages, where the documentation will tell you to install like pandas[excel] or uvicorn[standard]. The maintainers of these packages have created different dependency \"groups,\" where if you only need the Excel functionality included in pandas, for example, it will only install the dependencies required for that group, instead of all of the dependencies the pandas package requires.
Lastly, pdm can make you more efficient by providing functionality for scripting. With pdm, you can create scripts like start or start-dev to control executing commands you would manually re-type each time you wanted to run them.
Note
Please refer to the Official PDM installation instructions for instructions. I personally use the pipx method (pipx install pdm), but there are multiple sets of instructions that are updated occasionally, and it is best to check PDM's official documentation for installation/setup instructions.
When you initialize a new project with pdm init, you will be asked a series of questions that will help pdm determine the type of environment to set up. One of the questions asks if your project is a library or application:
Is the project a library that is installable?\nIf yes, we will need to ask a few more questions to include the project name and build backend [y/n] (n):\nWarning
This choice does matter! It affects how Python structures your project, and how the project executes. You will need to answer a couple of extra questions for an \"application,\" but there is a correct way to answer this question when pdm asks it.
Continue reading for more information.
For a more in-depth description of when to choose a \"library\" vs an \"application\", please read the Official PDM documentation.
As a rule of thumb, modules you import into other scripts/apps and do not have a CLI tool (like the pydantic or requests modules) are a \"library.\" A module that includes a CLI and can be executed from the command line, but is built with Python (like black, ruff, pytest, mypy, uvicorn, etc) are an \"application.\"
Todo
src/ layoutsrc/ projectToc
Jump right to a section:
If you're just here for a quick reference, use the sections below for quick pyenv setup & usage instructions. Any sections linked here will have a button that returns you to this TL;DR section, so if you're curious about one of the steps and follow a link you can get right back to it when you're done reading.
This is what the button will look like: \u2934\ufe0f back to TL;DR
See the Install pyenv section for OS-specific installation instructions.
Warning
If you skip the installation instructions, make sure you add pyenv to your PATH. If you need help setting PATH variables, please read the Install pyenv section.
~/.bashrc, add this to the bottom of the file (if in a terminal environment, use an editor like $ nano ~.bashrc):export PATH=\"$HOME/.pyenv/bin:$PATH\"\neval \"$(pyenv init --path)\"\neval \"$(pyenv virtualenv-init -)\"\nPATH variable, add these 2 paths:%USERPROFILE%\\\\.pyenv\\\\pyenv-win\\\\bin\n%USERPROFILE%\\\\.pyenv\\\\pyenv-win\\\\shims\nShow versions available for install. Update the list with $ pyenv update if you don't see the version you want.
$ pyenv install -l\n# 3.xx.x\n# 3.xx.x-xxx\n# ...\n\n$ pyenv install 3.12.1\nWarning
The 3.12.1 version string is used as an example throughout the documentation. Make sure you're using a more recent version, if one is available.
You can check by running $ pyenv update && pyenv install -l.
Examples commands
global version to 3.12.1: $ pyenv global 3.12.1global to multiple versions, 3.12.1 + 3.11.8: $ pyenv global 3.12.1 3.11.8Version-order
The order of the versions you specify does matter. For example, in the command above,3.12.1 is the \"primary\" Python, but 3.11.8 will also be available for tools like pytest, which can run tests against multiple versions of Python, provided they are available in the PATH. local version to 3.11.8, creating a .python-version file in the process: $ pyenv local 3.11.8local version will override the global settinglocal to multiple versions, 3.12.1 + 3.11.8: $ pyenv local 3.12.1 3.11.8shell version to `3.pyenv commands List available pyenv commands Useful as a quick reference if you forget a command name. Does not include help/man text. To see available help for a command, run it without any parameters, i.e. pyenv whence pyenv update Refresh pyenv's repository Updates the pyenv utility, fetches new available Python versions, etc. Run this command occasionally to keep everything up to date. pyenv install 3.xx.xx Install a version of Python If you don't see the version you want, run pyenv update pyenv uninstall 3.xx.xx Uninstall a version of Python that was installed with pyenv. You can uninstall multiple versions at a time with pyenv uninstall 3.11.2 3.11.4 pyenv global 3.xx.xx Set the global/default Python version to use. You can set multiple versions like pyenv global 3.11.4 3.11.6 3.12.2. Run without any args to print the current global Python interpreter(s). pyenv local 3.xx.xx Set the local Python interpreter. Creates a file in the current directory .python-version, which pyenv will detect and will set your interpreter to the version specified. Like pyenv global, you can set multiple versions with pyenv local 3.11.4 3.11.6 3.12.2. Run without any args to print the current global Python interpreter(s). pyenv shell 3.xx.xx Set the Python interpreter for the current shell session. Resets when session exits. Like pyenv global, you can set multiple versions with pyenv shell 3.11.4 3.11.6 3.12.2. Run without any args to print the current global Python interpreter(s). pyenv versions List versions of Python installed with Pyenv & available for use. Versions will be printed as a list of Python version numbers, and may include interpreter paths. pyenv which <executable> Print the path to a pyenv-installed executable, i.e. pip Helps with troubleshooting unexpected behavior. If you suspect pyenv is using the wrong interpreter, check the path with pyenv which python, for example. pyenv rehash Rehash pyenv shims. Run this after switching Python versions with pyenv if you're getting unexpected outputs."},{"location":"programming/python/pyenv.html#what-is-pyenv","title":"What is pyenv","text":"pyenv is a tool for managing versions of Python. It handles downloading the version archive, extracting & installing, and can isolate different versions of Python into their own individual environments.
pyenv can also handle running multiple versions of Python at the same time. For example, when using nox, you can declare a list of Python versions the session should run on, like @nox.session(python=[\"3.11.2\", \"3.11.4\", \"3.12.1\"], name=\"session-name\"). As long as one of the pyenv scopes (shell, local, or global) has all of these versions of Python, nox will run the session multiple times, once for each version declared in python=[].
Note
For more information on the problem pyenv solves, read the \"Why use pyenv?\" section below.
If you just want to see installation & setup instructions, you can skip to the \"install pyenv\" section, or to \"using pyenv\" to see notes on using the tool.
For more detailed (and likely more up-to-date) documentation on pyenv installation & usage, check the pyenv gihub's README.
With pyenv, you can separate your user Python installation from the system Python. This is generally a good practice, specifically on Linux machines; the version of Python included in some distributions is very old, and you are not meant to install packages with pip using the version of Python that came installed on your machine (especially without at least using a .venv).
The system Python is there for system packages you install that have some form of Python scripting included. Packages built for specific Linux distributions, like Debian, Ubuntu, Fedora, OpenSuSE, etc, are able to target the system version of Python, ensuring a stable and predictable installation. When you add pip packages to your system's Python environment, it not only adds complexity for other packages on your system to work around, you also increase your chance of landing yourself in dependency hell, where 2 pip packages require the same package but different versions of that package.
For more reading on why it's a good idea to install a separate version of Python, and leave the version that came installed with your machine untouched, check this RealPython article.
You also have the option of installing Python yourself, either compiling it from source or downloading, building, and installing a package from python.org. This option is perfectly fine, but can be difficult for beginners, and involves more steps and room for error when trying to automate.
"},{"location":"programming/python/pyenv.html#pyenv-scopes","title":"pyenv scopes","text":"\u2934\ufe0f back to TL;DR
pyenv has 3 \"scopes\":
shellpyenv shell x.xx.xx sets the Python interpreter for the current shellexec $SHELL or source ~/.bashrc), this value is also resetlocalpyenv local x.xx.xx creates a file called .python-version at the current pathpyenv uses this file to automatically set the version of Python while in this directoryglobalpyenv global x.xx.xx sets the default, global Python versionpython -m pip install ...) will use this global version, if no local or shell version is specifiedlocal and shell override this valuepyenv, in the event you have not set a pyenv local or pyenv shell version of Python.The global scope is essentially the \"default\" scope. When no other version is specified, the global Python version will be used.
pyenv shell overrides > pyenv local overrides > pyenv global
Warning
Make sure to pay attention to your current pyenv version. If you are getting unexpected results when running Python scripts, check the version with python3 --version. This will help if you are expecting 3.11.4 to be the version of Python for your session, for example, but have set pyenv shell 3.12.1. Because shell overrides local, the Python version in .python-version will be ignored until the session is exited.
Installing pyenv varies between OSes. On Windows, you use the pyenv-win package, for example. Below are installation instructions for Windows and Linux.
\u2934\ufe0f back to TL;DR
sudo apt-get install -y \\\n git \\\n gcc \\\n make \\\n openssl \\\n libssl-dev \\\n libbz2-dev \\\n libreadline-dev \\\n libsqlite3-dev \\\n zlib1g-dev \\\n libncursesw5-dev \\\n libgdbm-dev \\\n libc6-dev \\\n zlib1g-dev \\\n libsqlite3-dev \\\n tk-dev \\\n libssl-dev \\\n openssl \\\n libffi-dev\nsudo dnf install -y \\\n git \\\n gcc \\\n make \\\n openssl \\\n openssl-devel \\\n bzip2-devel \\\n zlib-devel \\\n readline-devel \\\n soci-sqlite3-devel \\\n ncurses-devel \\\n gdbm \\\n glibc-devel \\\n tk-devel \\\n libffi-devel\npyenv with the convenience scriptcurl https://pyenv.run | bash\npyenv variables to your ~/.bashrc...\n\n## Pyenv\nexport PATH=\"$HOME/.pyenv/bin:$PATH\"\neval \"$(pyenv init -)\"\neval \"$(pyenv virtualenv-init -)\"\n\u2934\ufe0f back to TL;DR
pyenv-win with PowershellInvoke-WebRequest -UseBasicParsing -Uri \"https://raw.githubusercontent.com/pyenv-win/pyenv-win/master/pyenv-win/install-pyenv-win.ps1\" -OutFile \"./install-pyenv-win.ps1\"; &\"./install-pyenv-win.ps1\"\nPATH%USERPROFILE%\\\\.pyenv\\\\pyenv-win\\\\bin\n%USERPROFILE%\\\\.pyenv\\\\pyenv-win\\\\shims\n[CmdletBinding()]\nparam (\n [Parameter(Mandatory = $true)]\n [string] $PythonVersion\n)\n\n$ErrorActionPreference = \"Stop\"\n$ProgressPreference = \"SilentlyContinue\"\nSet-StrictMode -Version Latest\n\nfunction _runCommand {\n [CmdletBinding()]\n param (\n [Parameter(Mandatory = $true, Position = 0)]\n [string] $Command,\n [switch] $PassThru\n )\n\n try {\n if ( $PassThru ) {\n $res = Invoke-Expression $Command\n }\n else {\n Invoke-Expression $Command\n }\n\n if ( $LASTEXITCODE -ne 0 ) {\n $errorMessage = \"'$Command' reported a non-zero status code [$LASTEXITCODE].\"\n if ($PassThru) {\n $errorMessage += \"`nOutput:`n$res\"\n }\n throw $errorMessage\n }\n\n if ( $PassThru ) {\n return $res\n }\n }\n catch {\n $PSCmdlet.WriteError( $_ )\n }\n}\n\nfunction _addToUserPath {\n [CmdletBinding()]\n param (\n [Parameter(Mandatory = $true, Position = 0)]\n [string] $AppName,\n [Parameter(Mandatory = $true, Position = 1)]\n [string[]] $PathsToAdd\n )\n\n $pathEntries = [System.Environment]::GetEnvironmentVariable(\"PATH\", [System.EnvironmentVariableTarget]::User) -split \";\"\n\n $pathUpdated = $false\n foreach ( $pathToAdd in $PathsToAdd ) {\n if ( $pathToAdd -NotIn $pathEntries ) {\n $pathEntries += $pathToAdd\n $pathUpdated = $true\n }\n }\n if ( $pathUpdated ) {\n Write-Host \"$($AppName): Updating %PATH%...\" -f Green\n # Remove any duplicate or blank entries\n $cleanPaths = $pathEntries | Select-Object -Unique | Where-Object { -Not [string]::IsNullOrEmpty($_) }\n\n # Update the user-scoped PATH environment variable\n [System.Environment]::SetEnvironmentVariable(\"PATH\", ($cleanPaths -join \";\").TrimEnd(\";\"), [System.EnvironmentVariableTarget]::User)\n\n # Reload PATH in the current session, so we don't need to restart the console\n $env:PATH = [System.Environment]::GetEnvironmentVariable(\"PATH\", [System.EnvironmentVariableTarget]::User)\n }\n else {\n Write-Host \"$($AppName): PATH already setup.\" -f Cyan\n }\n}\n\n# Install pyenv\nif ( -Not ( Test-Path $HOME/.pyenv ) ) {\n if ( $IsWindows ) {\n Write-Host \"pyenv: Installing for Windows...\" -f Green\n & git clone https://github.com/pyenv-win/pyenv-win.git $HOME/.pyenv\n if ($LASTEXITCODE -ne 0) {\n Write-Error \"git reported a non-zero status code [$LASTEXITCODE] - check previous output.\"\n }\n }\n else {\n Write-Error \"This script currently only supports Windows.\"\n }\n}\nelse {\n Write-Host \"pyenv: Already installed.\" -f Cyan\n}\n\n# Add pyenv to PATH\n_addToUserPath \"pyenv\" @(\n \"$HOME\\.pyenv\\pyenv-win\\bin\"\n \"$HOME\\.pyenv\\pyenv-win\\shims\"\n)\n\n# Install default pyenv python version\n$pyenvVersions = _runCommand \"pyenv versions\" -PassThru | Select-String $PythonVersion\nif ( -Not ( $pyenvVersions ) ) {\n Write-Host \"pyenv: Installing python version $PythonVersion...\" -f Green\n _runCommand \"pyenv install $PythonVersion\"\n}\nelse {\n Write-Host \"pyenv: Python version $PythonVersion already installed.\" -f Cyan\n}\n\n# Set pyenv global version\n$globalPythonVersion = _runCommand \"pyenv global\" -PassThru\nif ( $globalPythonVersion -ne $PythonVersion ) {\n Write-Host \"pyenv: Setting global python version: $PythonVersion\" -f Green\n _runCommand \"pyenv global $PythonVersion\"\n}\nelse {\n Write-Host \"pyenv: Global python version already set: $globalPythonVersion\" -f Cyan\n}\n\n# Update pip\n_runCommand \"python -m pip install --upgrade pip\"\n\n# Install pipx, pdm, black, cookiecutter\n_runCommand \"pip install pipx\"\n_runCommand \"pip install pdm\"\n_runCommand \"pip install black\"\n_runCommand \"pip install cookiecutter\"\n\u2934\ufe0f back to TL;DR
The sections below will detail how to install a version (or multiple versions!) of Python using pyenv, switching between them, and configuring your shell to make multiple versions of Python available to tools like nox and pytest.
Note
To see a list of the commands you can run, execute $ pyenv without any commands.
Updating pyenv is as simple as typing $ pyenv update in your terminal. The pyenv update command will update pyenv itself, as well as the listing of available Python distributions.
You can ask pyenv to show you a list of all the versions of Python available for you to install with the tool by running: $ pyenv install -l (or $ pyenv install --list). You will see a long list of version numbers, which you can install with pyenv.
Note
Some releases will indicate a specific CPU architecture, like -win32, -arm64, etc. Make sure you're installing the correct version for your CPU type!
To be safe, you can simply use a Python version string, omitting any CPU specification, like 3.12.1 instead of 3.12.1-arm.
Once you have decided on a version of Python to install (we will use 3.12.1, a recent release as of 2/14/2024), install it with: $ pyenv install 3.12.1 (or whatever version you want to install).
You can see which versions of Python are available to pyenv by running $ pyenv versions. The list will grow as you install more versions of Python with $ pyenv install x.x.x.
Note
To learn more about the global, local, and shell scopes, check the pyenv scopes section.
## Set global/default Python version to 3.12.1\n$ pyenv global 3.12.1\n\n## Make multiple versions available to tools like pytest, nox, etc\n$ pyenv global 3.11.8 3.11.6 3.12.1\n\n## Create a file `.python-version` in the local directory.\n# Python commands called from this directory will use\n# the version(s) specified this way\n$ pyenv local 3.12.1\n\n## Multiple versions in `.python-version`\n$ pyenv local 3.11.8 3.12.1\n\n## Set python version(s) for current shell session.\n# This value is cleared on logout, exit, or shutdown/restart\n$ pyenv shell 3.12.1\n\n# Set newer version first to prioritize it, make older version available\n$ pyenv shell 3.12.1 3.11.8\nTo uninstall a version of Python that you installed with pyenv, use $ pyenv uninstall x.x.x. For example, to uninstall version 3.12.1: pyenv uninstall 3.12.1.
The rich package helps make console/terminal output look nicer. It has colorization, animations, and more.
rich Github repositoryrich docsrich pypiThis example uses the rich.console.Console and rich.spinner.Spinner classes. The first context manager, get_console(), yields a rich.Console() object, which can be used with the rich.Spinner() class to display a spinner on the command line.
from contextlib import contextmanager\nimport typing as t\nfrom rich.console import Console\n\n@contextmanager\ndef get_console() -> t.Generator[Console, t.Any, None]:\n \"\"\"Yield a `rich.Console`.\n\n Usage:\n `with get_console() as console:`\n\n \"\"\"\n try:\n console: Console = Console()\n\n yield console\n\n except Exception as exc:\n msg = Exception(f\"Unhandled exception getting rich Console. Details: {exc}\")\n log.error(msg)\n\n raise exc\nThe simple_spinner() context manager yields a rich.Spinner() instance. Wrap a function in with simple_spinner(msg=\"Some message\") as spinner: to show a spinner while the function executes.
from contextlib import contextmanager\nfrom rich.spinner import Spinner\n\n@contextmanager\ndef simple_spinner(text: str = \"Processing... \\n\", animation: str = \"dots\"):\n if not text:\n text: str = \"Processing... \\n\"\n assert isinstance(text, str), TypeError(\n f\"Expected spinner text to be a string. Got type: ({type(text)})\"\n )\n\n if not text.endswith(\"\\n\"):\n text += \" \\n\"\n\n if not animation:\n animation: str = \"dots\"\n assert isinstance(animation, str), TypeError(\n f\"Expected spinner animation to be a string. Got type: ({type(text)})\"\n )\n\n try:\n _spinner = Spinner(animation, text=text)\n except Exception as exc:\n msg = Exception(f\"Unhandled exception getting console spinner. Details: {exc}\")\n log.error(msg)\n\n raise exc\n\n ## Display spinner\n try:\n with get_console() as console:\n with console.status(text, spinner=animation):\n yield console\n except Exception as exc:\n msg = Exception(\n f\"Unhandled exception yielding spinner. Continuing without animation. Details: {exc}\"\n )\n log.error(msg)\n\n pass\nPutting both of these functions in the same file allows you to import just the simple_spinner method, which calls get_console() for you. You can also get a console using with get_console() as console:, and write custom spinner/rich logic.
from some_modules import simple_spinner\n\nwith simple_spinner(text=\"Thinking... \"):\n ## Some long-running code/function\n ...\nTodo
Toc
Jump right to a section:
virtualenv .venv Create a new virtual environment This command creates a new directory called .venv/ at the path where you ran the command. (Windows) .\\.venv\\Scripts\\activate Activate a virtual environment on Windows. Your shell should change to show (.venv), indicating you are in a virtual environment (Linux/Mac) ./venv/bin/activate Activate a virtual environment on Linux. Your shell should change to show (.venv), indicating you are in a virtual environment deactivate Exit/deactivate a virtual environment. You can also simply close your shell session, which exists the environment. This command only works once a virtual environment is activated; deactivate will give an error saying the command is not found if you do not have an active virtual environment."},{"location":"programming/python/virtualenv.html#what-is-virtualenv","title":"What is virtualenv?","text":"RealPython: Virtualenv Primer
virtualenv is a tool for creating virtual Python environments. The virtualenv tool is sometimes installed with the \"system\" Python, but you may need to install it yourself (see the warning below).
These virtual environments are stored in a directory, usually ./.venv, and can be \"activated\" when you are developing a Python project. Virtual environments can also be used as Jupyter kernels if you install the ipykernel package.
Warning
If you ever see see an error saying the virtualenv command cannot be found (or something along those lines), simply install virtualenv with:
$ pip install virtualenv\nvirtualenv helps developers avoid \"dependency hell\". Without virtualenv, when you insall a package with pip, it is installed \"globally.\" This becomes an issue when 2 different Python projects use the same dependency, but different versions of that dependency. This leads to a \"broken environment,\" where the only fix is to essentially uninstall all dependencies and start from scratch.
With virtualenv, you start each project by running $ virtualenv .venv, which will create a directory called .venv/ at the path where you ran the command (i.e. inside of a Python project directory).
Note
.venv/ is the convention, but you can name this directory whatever you want. If you run virtualenv using a different path, like virtualenv VirtualEnvironment (which would create a directory called VirtualEnvironment/ at the local path), make sure you use that name throughout this guide where you see .venv.
Once a virtual environment is created, you need to \"activate\" it. Activating a virtual environment will isolate your current shell/session from the global Python, allowing you to install dependencies specific to the current Python project without interfering with the global Python state.
Activating a virtual environment is as simple as:
$ ./.venv/Scripts/activate$ ./.venv/bin/activateAfter activating an environment, your shell will change to indicate that you are within a virtual environment. Example:
activating virtualenv## Before .venv activation, using the global Python. Depdendency will\n# be installed to the global Python environment\n$ pip install pandas\n\n## Activate the virtualenv\n$ ./.venv/Scripts/activate\n\n## The shell will change to indicate you're in a venv. The \"(.venv)\" below\n# indicates a virtual environment has been activated. The pip install command\n# installs the dependency within the virtual environment\n(.venv) $ pip install pandas\nThis method of \"dependency isolation\" ensures a clean environment for each Python project you start, and keeps the \"system\"/global Python version clean of dependency errors.
"},{"location":"programming/python/virtualenv.html#exportingimporting-requirements","title":"Exporting/importing requirements","text":"Once a virtual environment is activated, commands like pip will run much the same as they do without a virtual environment, but the outputs will be contained to the .venv directory in the project you ran the commands from.
To export pip dependencies:
## Make sure you've activated your virtual environment first\n$ .\\\\venv\\\\Scripts\\\\activate # Windows\n$ ./venv/bin/activate # Linux/Mac\n\n## Export pip requirements\n(.venv) $ pip freeze > requirements.txt\nTo import/install pip dependencies from an existing requirements.txt file:
## Make sure you've activated your virtual environment first\n$ .\\\\venv\\\\Scripts\\\\activate # Windows\n$ ./venv/bin/activate # Linux/Mac\n\n## Install dependencies from a requirements.txt file\n(.venv) $ pip install -r requirements.txt\nTodo
.venv environmentTodo
virtualenvvirtualenvpoetrypdmTodo
with open() as f:with sqlalchemy.Session() as sess:@contextmanager__enter__() and __exit__() methodsCode I copy/paste into most of my projects.
Note
There is also a companion repository for this section, which may be more up to date than the examples in this section.
I use Docker a lot. I'm frequently copying/pasting Dockerfiles from previous projects to modify for my current project. Here, I will add some of my more common Dockerfiles, with notes & code snippets for reference.
This section covers some of the Dockerfiles I frequently re-use.
","tags":["standard-project-files","docker"]},{"location":"programming/standard-project-files/Docker/python_docker.html","title":"Python Dockerfiles","text":"Building/running a Python app in a Dockerfile can be accomplished many different ways. The example(s) below are my personal approach, meant to serve as a starting point/example of Docker's capabilities.
One thing you will commonly see in my Python Dockerfiles are a set of ENV variables in the base layer. Below is a list of the ENV variables I commonly set in Dockerfiles, and what they do:
Note
In some Dockerfiles, you will see ENV variables declared with an equal sign, and others without. These are equivalent, you can declare/set these variables either way and they will produce the same result. For example, the following 2 ENV variable declarations are equivalent:
PYTHONUNBUFFERED 1
PYTHONUNBUFFERED=1
PYTHONDONTWRITEBYTECODE=1.pyc files.pyc files are essentially simplified bytecode versions of your scripts that are created when a specific .py file is executed the first time..pyc files are the instructions/ingredients written like code on the side of your cup that tell the barista what to make. They don't need to know the full recipe, just the ingredients. A .pyc file is the \"ingredients\" for a .py file, meant to speed up subsequent executions of the same script..py file as a .py file each time for reproducibility.PYTHONUNBUFFERED=1stdout and stderr messages directly instead of buffering, ensuring realtime output to the container's sdtdout/stderrPIP_NO_CACHE_DIR=offpip not to cache dependencies. This would affect reproducibility in the container, and also needlessly takes up space.pip installs as a Dockerfile layer with buildkitPIP_DISABLE_PIP_VERSION_CHECK=onpip being out of date in a container environment. Also for reproducibility.PIP_DEFAULT_TIMEOUT=100pip's timeout value to a higher number to give it more time to finish downloading/installing a dependency.Use this container for small/simple Python projects, or as a starting point for a multistage build.
Python simple DockerfileFROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\nWORKDIR /app\n\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\nCOPY ./src .\nThis Dockerfile is a multi-stage build, which means it uses \"layers.\" These layers are cached by the Docker buildkit, meaning if nothing has changed in a given layer between builds, Docker will speed up the total buildtime by using a cached layer.
For example, the build layer below installs dependencies from the requirements.txt file. If no new dependencies are added between docker build commands, this layer will be re-used, \"skipping\" the pip install command.
FROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\nFROM base AS build\n\nWORKDIR /app\n\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\n## Use target: dev to build this step\nFROM build AS dev\n\nENV ENV_FOR_DYNACONF=dev\n## Tell Dynaconf to always load from the environment first while in the container\nENV DYNACONF_ALWAYS_LOAD_ENV_VARS=True\n\nWORKDIR /app\nCOPY ./src .\n\n############\n# Optional #\n############\n# Export ports, set an entrypoint/CMD, etc\n# Note: This is normally handled by your orchestrator (docker-compose, Azure Container App, etc)\n\n# EXPOSE 5000\n# CMD [\"python\", \"main.py\"]\n\n## Use target: prod to build this step\nFROM build AS prod\n\nENV ENV_FOR_DYNACONF=prod\n## Tell Dynaconf to always load from the environment first while in the container\nENV DYNACONF_ALWAYS_LOAD_ENV_VARS=True\n\nWORKDIR /app\nCOPY ./src .\n\n############\n# Optional #\n############\n# Export ports, set an entrypoint/CMD, etc\n# Note: This is normally handled by your orchestrator (docker-compose, Azure Container App, etc)\n\n# EXPOSE 5000\n# CMD [\"python\", \"main.py\"]\nSome pre-commit hooks I use frequently.
repos:\n\n- repo: https://gitlab.com/vojko.pribudic/pre-commit-update\n rev: v0.1.1\n hooks:\n - id: pre-commit-update\n\n- repo: https://github.com/kynan/nbstripout\n rev: 0.6.1\n hooks:\n - id: nbstripout\nThis hook will update the revisions for all installed hooks each time pre-commit runs.
- repo: https://gitlab.com/vojko.pribudic/pre-commit-update\n rev: v0.1.1\n hooks:\n - id: pre-commit-update\n args: [--dry-run --exclude black --keep isort]\nThis hook will scan for jupyter notebooks (.ipynb) and clear any cell output before committing.
- repo: https://github.com/kynan/nbstripout\n rev: 0.6.1\n hooks:\n - id: nbstripout\nCopy/paste-able code, or snippets for files like pyproject.toml and noxfile.py
# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\n# lib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n*.py,cover\n.hypothesis/\n.pytest_cache/\ncover/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\n*.log.*\nlocal_settings.py\n\n## Database\ndb.sqlite3\ndb.sqlite3-journal\n*.sqlite\n*.sqlite3\n*.db\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\n.pybuilder/\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n# For a library or package, you might want to ignore these files since the code is\n# intended to run in multiple environments; otherwise, check them in:\n# .python-version\n\n# pipenv\n# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n# However, in case of collaboration, if having platform-specific dependencies or dependencies\n# having no cross-platform support, pipenv may install dependencies that don't work, or not\n# install all needed dependencies.\n#Pipfile.lock\n\n# poetry\n# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.\n# This is especially recommended for binary packages to ensure reproducibility, and is more\n# commonly ignored for libraries.\n# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control\n#poetry.lock\n\n# pdm\n# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.\n#pdm.lock\n# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it\n# in version control.\n# https://pdm.fming.dev/#use-with-ide\n.pdm.toml\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm\n__pypackages__/\n\n# Celery stuff\ncelerybeat-schedule\ncelerybeat.pid\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n*.env\n*.*.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n## Allow Environment patterns\n!*example*\n!*example*.*\n!*.*example*\n!*.*example*.*\n!*.*.*example*\n!*.*.*example*.*\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n\n# pytype static type analyzer\n.pytype/\n\n# Cython debug symbols\ncython_debug/\n\n# PyCharm\n# JetBrains specific template is maintained in a separate JetBrains.gitignore that can\n# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore\n# and can be added to the global gitignore or merged into this file. For a more nuclear\n# option (not recommended) you can uncomment the following to ignore the entire idea folder.\n#.idea/\n\n## PDM\n.pdm-python\n\n## Pyenv\n.python-version\n\n## Dynaconf\n**/.secrets.toml\n**/*.local.toml\n\n!**/*.example.toml\n!**/.*.example.toml\n\n## Jupyter\n# Uncomment to ignore Jupyter notebooks, i.e. if\n# pre-hook is not configured and you don't want to\n# commit notebook with output.\n\n# *.ipynb\n\n## Ignore mkdocs builds\nsite/\n~/.pypirc600<pypi-test-token>/<pypi-token> with your pypi/pypi-test publish token.## ~/.pypirc\n# chmod: 600\n[distutils]\nindex-servers=\n pypi\n testpypi\n\n## Example of a local, private Python package index\n# [local]\n# repository = http://127.0.0.1:8080\n# username = test \n# password = test\n\n[testpypi]\nusername = __token__ \npassword = <pypi-test-token>\n\n[pypi]\nrepository = https://upload.pypi.org/legacy/\nusername = __token__\npassword = <pypi-token>\nI use Dynaconf frequently to manage loading my project's settings from a local file (config/settings.local.toml) during development, and environment variables when running in a container. Dynaconf allows for overriding configurations by setting environment variables.
To load configurations from the environment, you can:
envvar_prefix value of a Dynaconf() instanceLOG_LEVEL: export DYNACONF_LOG_LEVEL=...config/settings.local.toml fileconfig/settings.toml file should not be edited, nor should it contain any real valuesconfig/settings.local.toml during local developmentconfig/settings.local.tomlNote
The Database section is commented below because not all projects will start with a database. This file can still be copy/pasted to config/settings.toml/config/settings.local.toml as a base/starting point.
##\n# My standard Dynaconf settings.toml file.\n#\n# I normally put this file in a directory like src/config/settings.toml, then update my config.py, adding\n# root_path=\"config\" to the Dynaconf instance.\n##\n\n[default]\n\nenv = \"prod\"\ncontainer_env = false\nlog_level = \"INFO\"\n\n############\n# Database #\n############\n\n# db_type = \"sqlite\"\n# db_drivername = \"sqlite+pysqlite\"\n# db_username = \"\"\n# # Set in .secrets.toml\n# db_password = \"\"\n# db_host = \"\"\n# db_port = \"\"\n# db_database = \".data/app.sqlite\"\n# db_echo = false\n\n[dev]\n\nenv = \"dev\"\nlog_level = \"DEBUG\"\n\n############\n# Database #\n############\n\n# db_type = \"sqlite\"\n# db_drivername = \"sqlite+pysqlite\"\n# db_username = \"\"\n# # Set in .secrets.toml\n# db_password = \"\"\n# db_host = \"\"\n# db_port = \"\"\n# db_database = \".data/app-dev.sqlite\"\n# db_echo = true\n\n[prod]\n\n############\n# Database #\n############\n\n# db_type = \"sqlite\"\n# db_drivername = \"sqlite+pysqlite\"\n# db_username = \"\"\n# # Set in .secrets.toml\n# db_password = \"\"\n# db_host = \"\"\n# db_port = \"\"\n# db_database = \".data/app.sqlite\"\n# db_echo = false\nNote
The Database section is commented below because not all projects will start with a database. This file can still be copy/pasted to config/.secrets.toml as a base/starting point.
##\n# Any secret values, like an API key or database password, or Azure connection string.\n##\n\n[default]\n\n############\n# Database #\n############\n\n# db_password = \"\"\n\n[dev]\n\n############\n# Database #\n############\n\n# db_password = \"\"\n\n[prod]\n\n############\n# Database #\n############\n\n# db_password = \"\"\nWarning
This code is highly specific to the way I structure my apps. Make sure to understand what it's doing so you can customize it to your environment, if you're using this code as a basis for your own config.py file
Note
Notes:
The following imports/vars/classes start out commented, in case the project is not using them or they require additional setup:
valid_db_types list (used to validate DBSettings.type)DYNACONF_DB_SETTINGS Dynaconf settings objectDBSettings class definitionIf the project is using a database and SQLAlchemy as the ORM, uncomment these values and modify your config/settings.local.toml & config/.secrets.toml accordingly.
from __future__ import annotations\n\nfrom typing import Union\n\nfrom dynaconf import Dynaconf\nfrom pydantic import Field, ValidationError, field_validator\nfrom pydantic_settings import BaseSettings\n\n## Uncomment if adding a database config\n# import sqlalchemy as sa\n# import sqlalchemy.orm as so\n\nDYNACONF_SETTINGS: Dynaconf = Dynaconf(\n environments=True,\n envvar_prefix=\"DYNACONF\",\n settings_files=[\"settings.toml\", \".secrets.toml\"],\n)\n\n## Uncomment if adding a database config\n# valid_db_types: list[str] = [\"sqlite\", \"postgres\", \"mssql\"]\n\n## Uncomment to load database settings from environment\n# DYNACONF_DB_SETTINGS: Dynaconf = Dynaconf(\n# environments=True,\n# envvar_prefix=\"DB\",\n# settings_files=[\"settings.toml\", \".secrets.toml\"],\n# )\n\n\nclass AppSettings(BaseSettings):\n env: str = Field(default=DYNACONF_SETTINGS.ENV, env=\"ENV\")\n container_env: bool = Field(\n default=DYNACONF_SETTINGS.CONTAINER_ENV, env=\"CONTAINER_ENV\"\n )\n log_level: str = Field(default=DYNACONF_SETTINGS.LOG_LEVEL, env=\"LOG_LEVEL\")\n\n\n## Uncomment if you're configuring a database for the app\n# class DBSettings(BaseSettings):\n# type: str = Field(default=DYNACONF_SETTINGS.DB_TYPE, env=\"DB_TYPE\")\n# drivername: str = Field(\n# default=DYNACONF_DB_SETTINGS.DB_DRIVERNAME, env=\"DB_DRIVERNAME\"\n# )\n# user: str | None = Field(\n# default=DYNACONF_DB_SETTINGS.DB_USERNAME, env=\"DB_USERNAME\"\n# )\n# password: str | None = Field(\n# default=DYNACONF_DB_SETTINGS.DB_PASSWORD, env=\"DB_PASSWORD\", repr=False\n# )\n# host: str | None = Field(default=DYNACONF_DB_SETTINGS.DB_HOST, env=\"DB_HOST\")\n# port: Union[str, int, None] = Field(\n# default=DYNACONF_DB_SETTINGS.DB_PORT, env=\"DB_PORT\"\n# )\n# database: str = Field(default=DYNACONF_DB_SETTINGS.DB_DATABASE, env=\"DB_DATABASE\")\n# echo: bool = Field(default=DYNACONF_DB_SETTINGS.DB_ECHO, env=\"DB_ECHO\")\n\n# @field_validator(\"port\")\n# def validate_db_port(cls, v) -> int:\n# if v is None or v == \"\":\n# return None\n# elif isinstance(v, int):\n# return v\n# elif isinstance(v, str):\n# return int(v)\n# else:\n# raise ValidationError\n\n# def get_db_uri(self) -> sa.URL:\n# try:\n# _uri: sa.URL = sa.URL.create(\n# drivername=self.drivername,\n# username=self.user,\n# password=self.password,\n# host=self.host,\n# port=self.port,\n# database=self.database,\n# )\n\n# return _uri\n\n# except Exception as exc:\n# msg = Exception(\n# f\"Unhandled exception getting SQLAlchemy database URL. Details: {exc}\"\n# )\n# raise msg\n\n# def get_engine(self) -> sa.Engine:\n# assert self.get_db_uri() is not None, ValueError(\"db_uri is not None\")\n# assert isinstance(self.get_db_uri(), sa.URL), TypeError(\n# f\"db_uri must be of type sqlalchemy.URL. Got type: ({type(self.db_uri)})\"\n# )\n\n# try:\n# engine: sa.Engine = sa.create_engine(\n# url=self.get_db_uri().render_as_string(hide_password=False),\n# echo=self.echo,\n# )\n\n# return engine\n# except Exception as exc:\n# msg = Exception(\n# f\"Unhandled exception getting database engine. Details: {exc}\"\n# )\n\n# raise msg\n\n# def get_session_pool(self) -> so.sessionmaker[so.Session]:\n# engine: sa.Engine = self.get_engine()\n# assert engine is not None, ValueError(\"engine cannot be None\")\n# assert isinstance(engine, sa.Engine), TypeError(\n# f\"engine must be of type sqlalchemy.Engine. Got type: ({type(engine)})\"\n# )\n\n# session_pool: so.sessionmaker[so.Session] = so.sessionmaker(bind=engine)\n\n# return session_pool\n\n\nsettings: AppSettings = AppSettings()\n## Uncomment if you're configuring a database for the app\n# db_settings: DBSettings = DBSettings()\nalembic with pip install alembic (or some equivalent package install command)alembicsrc directory, i.e. src/app, initialize alembic at src/app/alembicalembic init src/app/alembicIf using a \"flat\" repository, simply run alembic init alembic
Note
You can use any name for the directory instead of \"alembic.\" For instance, another common convention is to initialize alembic with alembic init migrations
Whatever directory name you choose, use that throughout these instructions where you see references to the alembic init path
Edit the alembic.ini file
script_location to the path you set for alembic, i.e. src/app/alembicprepend_sys_path, set to src/appalembic can do things like importing your app's config, loading the SQLAlchemy Base object, etc## alembic.ini\n\n[alembic]\n\nscript_location = src/app/alembic\n\n...\n\nprepend_system_path = src/app\n\n...\nalembic env.py file, which should be in src/app/alembic (or whatever path you initialized alembic in)sqlalchemy.URL), you can import it in env.py, or you can create a new one:## src/app/alembic.py\n\n...\n\nimport sqlalchemy as sa\n\n## Using a SQLite example\nDB_URI: sa.URL = sa.URL.create(\n drivername=\"sqlite+pysqlite\",\n username=None,\n password=None,\n host=None,\n port=None,\n database=\"database.sqlite\"\n)\n\n...\n\n## Set config's sqlalchemy.url value, after \"if config.config_filename is not None:\"\nconfig.set_main_option(\n \"sqlalchemy.url\",\n ## Use .render_as_string(hide_password=False) with sqlalchemy.URL objects,\n # otherwise database password will be \"**\"\n DB_URI.render_as_string(hide_password=False)\n)\n\n...\n\n# add your model's MetaData object here\n# for 'autogenerate' support\n# from myapp import mymodel\n# target_metadata = mymodel.Base.metadata\nfrom app.module.models import SomeModel # Import project's SQLAlchemy table classes\nfrom app.module.database import Base # Import project's SQLAlchemy Base object\n\ntarget_metadata = Base().metadata # Tell Alembic to use the project's Base() object\nalembic revision --autogenerate -m \"initial migration\"alembic revision will create an empty revision, with no changes.alembic upgrade +1+1 is how many migration levels to apply at once. If you have multiple migrations that have not been committed, you can use +2, +3, etc)alembic upgrade headalembic downgrade -1-1 is how many migration levels to revert, can also be -2, -3, etc)Some changes, like renaming a column, are not possible for Alembic to accurately track. In these cases, you will need to create an Alembic migration, then edit the new file in alembic/versions/{revision-hash}.py.
In the def upgrade() section, comment the innacurate op.add_column()/op.drop_column(), then add something like this (example uses the User class, with a renamed column .username -> .user_name):
# alembic/versions/{revision-hash}.py\n\n...\n\ndef upgrade() -> None:\n ...\n\n ## Comment the inaccurate changes\n # op.add_column(\"users\", sa.Column(\"user_name\", sa.VARCHAR(length=255), nullable=True))\n # op.drop_column(\"users\", \"username)\n\n ## Manually add a change of column type that wasn't detected by alembic\n op.alter_column(\"products\", \"description\", type_=sa.VARCHAR(length=3000))\n\n ## Manually describe column rename\n op.alter_column(\"users\", \"username\", new_column_name=\"user_name\")\nAlso edit the def downgrade() function to describe the changes that should be reverted when using alembic downgrade:
# alembic/versions/{revision-hash}.py\n\n...\n\ndef downgrade() -> None:\n ## Comment the inaccurate changes\n # op.add_column(\"users\", sa.Column(\"user_name\", sa.VARCHAR(length=255), nullable=True))\n # op.drop_column(\"users\", \"username)\n\n ## Manually describe changes to reverse if downgrading\n op.alter_column(\"users\", \"user_name\", new_column_name=\"username\")\n op.drop_column(\"products\", \"price\")\nAfter describing manual changes in an Alembic version file, you need to run alembic upgrade head to push the changes from the revision to the database.
After initializing alembic (i.e. alembic init alembic), a file alembic/env.py will be created. This file can be edited to include your project's models and SQLAlchemy Base.
Below are snippets of custom code I add to my alembic env.py file.
Provide database connection URL to alembic.
Todo
dynaconf to load database connection values from environment.env.pyDB_URI: sa.URL = sa.URL.create(\n drivername=\"sqlite+pysqlite\",\n username=None,\n password=None,\n host=None,\n port=None,\n database=\"database.sqlite\"\n)\nInstead of hardcording the database connection string in alembic.ini, load it from DB_URI.
Todo
env.py## Set config's sqlalchemy.url value, after \"if config.config_filename is not None:\"\nconfig.set_main_option(\n \"sqlalchemy.url\",\n ## Use .render_as_string(hide_password=False) with sqlalchemy.URL objects,\n # otherwise database password will be \"**\"\n DB_URI.render_as_string(hide_password=False)\n)\nImporting your project's SQLAlchemy Base and table classes that inherit from the Base object into alembic allows for automatic metadata creation.
Note
When using alembic to create the Base metadata, you do not need to run Base.metadata.create(bind=engine) in your code. Alembic handles the metadata creation for you. Just make sure to run alembic when first creating the database, or when cloning if using a local database like SQLite.
# add your model's MetaData object here\n# for 'autogenerate' support\n# from myapp import mymodel\n# target_metadata = mymodel.Base.metadata\n\n# Import project's SQLAlchemy table classes\nfrom app.module.models import SomeModel\n# Import project's SQLAlchemy Base object\nfrom app.core.database import Base\n\n# Tell Alembic to use the project's Base() object\ntarget_metadata = Base().metadata\nI am still deciding between tox and nox as my preferred task runner, but I've been leaning more towards nox for the simple reason that it's nice to be able to write Python code for things like try/except and creating directories that don't exist yet.
The basis for most/all of my projects' noxfile.py.
Note
The following are commented because they require additional setup, or may not be used in every project:
INIT_COPY_FILESsrc and a dest to copy it toconfig/.secrets.example.toml -> config/.secrets.tomlinit-setup Nox sessionINIT_COPY_FILES variableNote
If running all sessions with $ nox, only the sessions defined in nox.sessions will be executed. The list of sessions is conservative to start in order to maintain as generic a nox environment as possible.
Enabled sessions:
from __future__ import annotations\n\nfrom pathlib import Path\nimport platform\nimport shutil\n\nimport nox\n\nnox.options.default_venv_backend = \"venv\"\nnox.options.reuse_existing_virtualenvs = True\nnox.options.error_on_external_run = False\nnox.options.error_on_missing_interpreters = False\n# nox.options.report = True\n\n## Define sessions to run when no session is specified\nnox.sessions = [\"lint\", \"export\", \"tests\"]\n\n# INIT_COPY_FILES: list[dict[str, str]] = [\n# {\"src\": \"config/.secrets.example.toml\", \"dest\": \"config/.secrets.toml\"},\n# {\"src\": \"config/settings.toml\", \"dest\": \"config/settings.local.toml\"},\n# ]\n## Define versions to test\nPY_VERSIONS: list[str] = [\"3.12\", \"3.11\"]\n## Set PDM version to install throughout\nPDM_VER: str = \"2.11.2\"\n## Set paths to lint with the lint session\nLINT_PATHS: list[str] = [\"src\", \"tests\", \"./noxfile.py\"]\n\n## Get tuple of Python ver ('maj', 'min', 'mic')\nPY_VER_TUPLE = platform.python_version_tuple()\n## Dynamically set Python version\nDEFAULT_PYTHON: str = f\"{PY_VER_TUPLE[0]}.{PY_VER_TUPLE[1]}\"\n\n## Set directory for requirements.txt file output\nREQUIREMENTS_OUTPUT_DIR: Path = Path(\"./requirements\")\n## Ensure REQUIREMENTS_OUTPUT_DIR path exists\nif not REQUIREMENTS_OUTPUT_DIR.exists():\n try:\n REQUIREMENTS_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n except Exception as exc:\n msg = Exception(\n f\"Unable to create requirements export directory: '{REQUIREMENTS_OUTPUT_DIR}'. Details: {exc}\"\n )\n print(msg)\n\n REQUIREMENTS_OUTPUT_DIR: Path = Path(\".\")\n\n\n@nox.session(python=PY_VERSIONS, name=\"build-env\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef setup_base_testenv(session: nox.Session, pdm_ver: str):\n print(f\"Default Python: {DEFAULT_PYTHON}\")\n session.install(f\"pdm>={pdm_ver}\")\n\n print(\"Installing dependencies with PDM\")\n session.run(\"pdm\", \"sync\")\n session.run(\"pdm\", \"install\")\n\n\n@nox.session(python=[DEFAULT_PYTHON], name=\"lint\")\ndef run_linter(session: nox.Session):\n session.install(\"ruff\", \"black\")\n\n for d in LINT_PATHS:\n if not Path(d).exists():\n print(f\"Skipping lint path '{d}', could not find path\")\n pass\n else:\n lint_path: Path = Path(d)\n print(f\"Running ruff imports sort on '{d}'\")\n session.run(\n \"ruff\",\n \"--select\",\n \"I\",\n \"--fix\",\n lint_path,\n )\n\n print(f\"Formatting '{d}' with Black\")\n session.run(\n \"black\",\n lint_path,\n )\n\n print(f\"Running ruff checks on '{d}' with --fix\")\n session.run(\n \"ruff\",\n \"check\",\n \"--config\",\n \"ruff.ci.toml\",\n lint_path,\n \"--fix\",\n )\n\n\n@nox.session(python=[DEFAULT_PYTHON], name=\"export\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef export_requirements(session: nox.Session, pdm_ver: str):\n session.install(f\"pdm>={pdm_ver}\")\n\n print(\"Exporting production requirements\")\n session.run(\n \"pdm\",\n \"export\",\n \"--prod\",\n \"-o\",\n f\"{REQUIREMENTS_OUTPUT_DIR}/requirements.txt\",\n \"--without-hashes\",\n )\n\n print(\"Exporting development requirements\")\n session.run(\n \"pdm\",\n \"export\",\n \"-d\",\n \"-o\",\n f\"{REQUIREMENTS_OUTPUT_DIR}/requirements.dev.txt\",\n \"--without-hashes\",\n )\n\n # print(\"Exporting CI requirements\")\n # session.run(\n # \"pdm\",\n # \"export\",\n # \"--group\",\n # \"ci\",\n # \"-o\",\n # f\"{REQUIREMENTS_OUTPUT_DIR}/requirements.ci.txt\",\n # \"--without-hashes\",\n # )\n\n\n@nox.session(python=PY_VERSIONS, name=\"tests\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef run_tests(session: nox.Session, pdm_ver: str):\n session.install(f\"pdm>={pdm_ver}\")\n session.run(\"pdm\", \"install\")\n\n print(\"Running Pytest tests\")\n session.run(\n \"pdm\",\n \"run\",\n \"pytest\",\n \"-n\",\n \"auto\",\n \"--tb=auto\",\n \"-v\",\n \"-rsXxfP\",\n )\n\n\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-all\")\ndef run_pre_commit_all(session: nox.Session):\n session.install(\"pre-commit\")\n session.run(\"pre-commit\")\n\n print(\"Running all pre-commit hooks\")\n session.run(\"pre-commit\", \"run\")\n\n\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-update\")\ndef run_pre_commit_autoupdate(session: nox.Session):\n session.install(f\"pre-commit\")\n\n print(\"Running pre-commit autoupdate\")\n session.run(\"pre-commit\", \"autoupdate\")\n\n\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-nbstripout\")\ndef run_pre_commit_nbstripout(session: nox.Session):\n session.install(f\"pre-commit\")\n\n print(\"Running nbstripout pre-commit hook\")\n session.run(\"pre-commit\", \"run\", \"nbstripout\")\n\n\n# @nox.session(python=[PY_VER_TUPLE], name=\"init-setup\")\n# def run_initial_setup(session: nox.Session):\n# if INIT_COPY_FILES is None:\n# print(f\"INIT_COPY_FILES is empty. Skipping\")\n# pass\n\n# else:\n\n# for pair_dict in INIT_COPY_FILES:\n# src = Path(pair_dict[\"src\"])\n# dest = Path(pair_dict[\"dest\"])\n# if not dest.exists():\n# print(f\"Copying {src} to {dest}\")\n# try:\n# shutil.copy(src, dest)\n# except Exception as exc:\n# msg = Exception(\n# f\"Unhandled exception copying file from '{src}' to '{dest}'. Details: {exc}\"\n# )\n# print(f\"[ERROR] {msg}\")\nCode snippets for nox sessions
## Run all pre-commit hooks\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-all\")\ndef run_pre_commit_all(session: nox.Session):\n session.install(\"pre-commit\")\n session.run(\"pre-commit\")\n\n print(\"Running all pre-commit hooks\")\n session.run(\"pre-commit\", \"run\")\n## Automatically update pre-commit hooks on new revisions\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-update\")\ndef run_pre_commit_autoupdate(session: nox.Session):\n session.install(f\"pre-commit\")\n\n print(\"Running pre-commit update hook\")\n session.run(\"pre-commit\", \"run\", \"pre-commit-update\")\npytest-xdist runs tests concurrently, significantly improving test execution speed.
## Run pytest with xdist, allowing concurrent tests\n@nox.session(python=PY_VERSIONS, name=\"tests\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef run_tests(session: nox.Session, pdm_ver: str):\n session.install(f\"pdm>={pdm_ver}\")\n session.run(\"pdm\", \"install\")\n\n print(\"Running Pytest tests\")\n session.run(\n \"pdm\",\n \"run\",\n \"pytest\",\n \"-n\",\n \"auto\",\n \"--tb=auto\",\n \"-v\",\n \"-rsXxfP\",\n )\n## Run pytest\n@nox.session(python=PY_VERSIONS, name=\"tests\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef run_tests(session: nox.Session, pdm_ver: str):\n session.install(f\"pdm>={pdm_ver}\")\n session.run(\"pdm\", \"install\")\n\n print(\"Running Pytest tests\")\n session.run(\n \"pdm\",\n \"run\",\n \"pytest\",\n \"--tb=auto\",\n \"-v\",\n \"-rsXxfP\",\n )\nI use pdm to manage most of my Python projects. It's a fantastic tool for managing environments, dependencies, builds, and package publishing. PDM is similar in functionality to poetry, which I have also used and liked. My main reasons for preferring pdm over poetry are:
[tool.pdm.scripts] section in my pyproject.toml, and being able to script long/repeated things like alembic commands or project execution scripts is so handy.poetry (reference)Most of my notes and code snippets will assume pdm is the dependency manager for a given project.
In your pyproject.toml file, add a section [tool.pdm.scripts], then copy/paste whichever scripts you want to add to your project.
Warning
Check the code for some of the scripts, like the start script, which assumes your project code is at ./src/app/main.py. If your code is in a different path, or named something other than app, make sure to change this and any other similar lines.
[tool.pdm.scripts]\n\n###############\n# Format/Lint #\n###############\n\n# Lint with black & ruff\nlint = { shell = \"pdm run ruff check . --fix && pdm run black .\" }\n## With nox\n# lint = { cmd = \"nox -s lint\"}\n# Check only, don't fix\ncheck = { cmd = \"black .\" }\n# Check and fix\nformat = { cmd = \"ruff check . --fix\" }\n\n########################\n# Start/Launch Scripts #\n########################\n\n# Run main app or script. Launches from app/\nstart = { shell = \"cd app && pdm run python main.py\" }\n\n## Example Dynaconf start\nstart-dev = { cmd = \"python src/app/main.py\", env = { ENV_FOR_DYNACONF = \"dev\" } }\n\n######################\n# Export Requirement #\n######################\n\n# Export production requirements\nexport = { cmd = \"pdm export --prod -o requirements/requirements.txt --without-hashes\" }\n# Export only development requirements\nexport-dev = { cmd = \"pdm export -d -o requirements/requirements.dev.txt --without-hashes\" }\n## Uncomment if/when using a CI group\n# export-ci = { cmd = \"pdm export -G ci -o requirements/requirements.ci.txt --without-hashes\" }\n## Uncomment if using mkdocs or sphinx\n# export-docs = { cmd = \"pdm export -G docs --no-default -o docs/requirements.txt --without-hashes\" }\n\n###########\n# Alembic #\n###########\n\n## Create initial commit\nalembic-init = { cmd = \"alembic revision -m 'Initial commit.'\" }\n\n## Upgrade Alembic head after making model changes\nalembic-upgrade = { cmd = \"alembic upgrade head\" }\n\n## Run migrations\n# Prompts for a commit message\nalembic-migrate = { shell = \"read -p 'Commit message: ' commit_msg && pdm run alembic revision --autogenerate -m '${commit_msg}'\" }\n\n## Run full migration, upgrade - commit - revision\nmigrations = { shell = \"pdm run alembic upgrade head && read -p 'Commit message: ' commit_msg && pdm run alembic revision --autogenerate -m '${commit_msg}'\" }\nTodo
pytest for my projectsPut the pytest.ini file in the root of the project to configure pytest executions
[pytest]\n# Filter unregistered marks. Suppresses all UserWarning\n# messages, and converts all other errors/warnings to errors.\nfilterwarnings =\n error\n ignore::UserWarning\ntestpaths = tests\nNote
These tests don't really do anything, but they are the basis for writing pytest tests.
from __future__ import annotations\n\nfrom pytest import mark, xfail\n\n@mark.hello\ndef test_say_hello(dummy_hello_str: str):\n assert isinstance(\n dummy_hello_str, str\n ), f\"Invalid test output type: ({type(dummy_hello_str)}). Should be of type str\"\n assert (\n dummy_hello_str == \"world\"\n ), f\"String should have been 'world', not '{dummy_hello_str}'\"\n\n print(f\"Hello, {dummy_hello_str}!\")\n\n\n@mark.always_pass\ndef test_pass():\n assert True, \"test_pass() should have been True\"\n\n\n@mark.xfail\ndef test_fail():\n test_pass = False\n assert test_pass, \"This test is designed to fail\"\nPut conftest.py inside your tests/ directory. This file configures pytest, like providing test fixture paths so they can be accessed by tests.
import pytest\n\n## Add fixtures as plugins\npytest_plugins = [\n \"tests.fixtures.dummy_fixtures\"\n]\nSome templates/example of pytest fixtures
from pytest import fixture\n\n\n@fixture\ndef dummy_hello_str() -> str:\n \"\"\"A dummy str fixture for pytests.\"\"\"\n return \"hello, world\"\nRuff...is...awesome! It's hard to summarize, Ruff describes itself as \"An extremely fast Python linter and code formatter, written in Rust.\" It has replaced isort, flake8, and is on its way to replacing black in my environments.
I use 2 ruff.toml files normally:
ruff.tomlruff command and VSCoderuff.ci.tomlnoxfile.py, I have a lint section.ruff.toml to avoid pipeline errors for things that don't matter like docstring warnings and other rules I find overly strict.## Set assumed Python version\ntarget-version = \"py311\"\n\n## Same as Black.\nline-length = 88\n\n## Enable pycodestyle (\"E\") and Pyflakes (\"F\") codes by default\n# # Docs: https://beta.ruff.rs/docs/rules/\nlint.select = [\n \"D\", ## pydocstyle\n \"E\", ## pycodestyle\n \"F401\", ## remove unused imports\n \"I\", ## isort\n \"I001\", ## Unused imports\n]\n\n## Ignore specific checks.\n# Comment lines in list below to \"un-ignore.\"\n# This is counterintuitive, but commenting a\n# line in ignore list will stop Ruff from\n# ignoring that check. When the line is\n# uncommented, the check will be run.\nlint.ignore = [\n \"D100\", ## missing-docstring-in-public-module\n \"D101\", ## missing-docstring-in-public-class\n \"D102\", ## missing-docstring-in-public-method\n \"D103\", ## Missing docstring in public function\n \"D106\", ## missing-docstring-in-public-nested-class\n \"D203\", ## one-blank-line-before-class\n \"D213\", ## multi-line-summary-second-line\n \"D406\", ## Section name should end with a newline\n \"D407\", ## Missing dashed underline after section\n \"E501\", ## Line too long\n \"E402\", ## Module level import not at top of file\n \"F401\", ## imported but unused\n]\n\n## Allow autofix for all enabled rules (when \"--fix\") is provided.\n# NOTE: Leaving these commented until I know what they do\n# Docs: https://beta.ruff.rs/docs/rules/\nlint.fixable = [\n # \"A\", ## flake8-builtins\n # \"B\", ## flake8-bugbear\n \"C\",\n \"D\", ## pydocstyle\n \"E\", ## pycodestyle-error\n \"E402\", ## Module level import not at top of file\n \"E501\", ## Line too long\n # \"F\", ## pyflakes\n \"F401\", ## unused imports\n # \"G\", ## flake8-logging-format\n \"I\", ## isort\n \"N\", ## pep8-naming\n # \"Q\", ## flake8-quotas\n # \"S\", ## flake8-bandit\n \"T\",\n \"W\", ## pycodestyle-warning\n # \"ANN\", ## flake8-annotations\n # \"ARG\", ## flake8-unused-arguments\n # \"BLE\", ## flake8-blind-except\n # \"COM\", ## flake8-commas\n # \"DJ\", ## flake8-django\n # \"DTZ\", ## flake8-datetimez\n # \"EM\", ## flake8-errmsg\n \"ERA\", ## eradicate\n # \"EXE\", ## flake8-executables\n # \"FBT\", ## flake8-boolean-trap\n # \"ICN\", ## flake8-imort-conventions\n # \"INP\", ## flake8-no-pep420\n # \"ISC\", ## flake8-implicit-str-concat\n # \"NPY\", ## NumPy-specific rules\n # \"PD\", ## pandas-vet\n # \"PGH\", ## pygrep-hooks\n # \"PIE\", ## flake8-pie\n \"PL\", ## pylint\n # \"PT\", ## flake8-pytest-style\n # \"PTH\", ## flake8-use-pathlib\n # \"PYI\", ## flake8-pyi\n # \"RET\", ## flake8-return\n # \"RSE\", ## flake8-raise\n \"RUF\", ## ruf-specific rules\n # \"SIM\", ## flake8-simplify\n # \"SLF\", ## flake8-self\n # \"TCH\", ## flake8-type-checking\n \"TID\", ## flake8-tidy-imports\n \"TRY\", ## tryceratops\n \"UP\", ## pyupgrade\n # \"YTT\" ## flake8-2020\n]\n# unfixable = []\n\n# Exclude a variety of commonly ignored directories.\nexclude = [\n \".bzr\",\n \".direnv\",\n \".eggs\",\n \".git\",\n \".ruff_cache\",\n \".venv\",\n \"__pypackages__\",\n \"__pycache__\",\n \"*.pyc\",\n]\n\n[lint.per-file-ignores]\n\"__init__.py\" = [\"D104\"]\n\n## Allow unused variables when underscore-prefixed.\n# dummy-variable-rgx = \"^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$\"\n\n[lint.mccabe]\nmax-complexity = 10\n\n[lint.isort]\ncombine-as-imports = true\nforce-sort-within-sections = true\nforce-wrap-aliases = true\n## Use a single line after each import block.\nlines-after-imports = 1\n## Use a single line between direct and from import\nlines-between-types = 1\n## Order imports by type, which is determined by case,\n# in addition to alphabetically.\norder-by-type = true\nrelative-imports-order = \"closest-to-furthest\"\n## Automatically add imports below to top of files\nrequired-imports = [\"from __future__ import annotations\"]\n## Define isort section priority\nsection-order = [\n \"future\",\n \"standard-library\",\n \"first-party\",\n \"local-folder\",\n \"third-party\",\n]\n## Set assumed Python version\ntarget-version = \"py311\"\n\n## Same as Black.\nline-length = 88\n\n## Enable pycodestyle (\"E\") and Pyflakes (\"F\") codes by default\n# # Docs: https://beta.ruff.rs/docs/rules/\nlint.select = [\n \"D\", ## pydocstyle\n \"E\", ## pycodestyle\n \"F401\", ## remove unused imports\n \"I\", ## isort\n \"I001\", ## Unused imports\n]\n\n## Ignore specific checks.\n# Comment lines in list below to \"un-ignore.\"\n# This is counterintuitive, but commenting a\n# line in ignore list will stop Ruff from\n# ignoring that check. When the line is\n# uncommented, the check will be run.\nlint.ignore = [\n \"D100\", ## missing-docstring-in-public-module\n \"D101\", ## missing-docstring-in-public-class\n \"D102\", ## missing-docstring-in-public-method\n \"D103\", ## Missing docstring in public function\n \"D105\", ## Missing docstring in magic method\n \"D106\", ## missing-docstring-in-public-nested-class\n \"D107\", ## Missing docstring in __init__\n \"D200\", ## One-line docstring should fit on one line\n \"D203\", ## one-blank-line-before-class\n \"D205\", ## 1 blank line required between summary line and description\n \"D213\", ## multi-line-summary-second-line\n \"D401\", ## First line of docstring should be in imperative mood\n \"E402\", ## Module level import not at top of file\n \"D406\", ## Section name should end with a newline\n \"D407\", ## Missing dashed underline after section\n \"D414\", ## Section has no content\n \"D417\", ## Missing argument descriptions in the docstring for [variables]\n \"E501\", ## Line too long\n \"E722\", ## Do note use bare `except`\n \"F401\", ## imported but unused\n]\n\n## Allow autofix for all enabled rules (when \"--fix\") is provided.\n# NOTE: Leaving these commented until I know what they do\n# Docs: https://beta.ruff.rs/docs/rules/\nlint.fixable = [\n # \"A\", ## flake8-builtins\n # \"B\", ## flake8-bugbear\n \"C\",\n \"D\", ## pydocstyle\n \"E\", ## pycodestyle-error\n \"E402\", ## Module level import not at top of file\n # \"F\", ## pyflakes\n \"F401\", ## unused imports\n # \"G\", ## flake8-logging-format\n \"I\", ## isort\n \"N\", ## pep8-naming\n # \"Q\", ## flake8-quotas\n # \"S\", ## flake8-bandit\n \"T\",\n \"W\", ## pycodestyle-warning\n # \"ANN\", ## flake8-annotations\n # \"ARG\", ## flake8-unused-arguments\n # \"BLE\", ## flake8-blind-except\n # \"COM\", ## flake8-commas\n # \"DJ\", ## flake8-django\n # \"DTZ\", ## flake8-datetimez\n # \"EM\", ## flake8-errmsg\n \"ERA\", ## eradicate\n # \"EXE\", ## flake8-executables\n # \"FBT\", ## flake8-boolean-trap\n # \"ICN\", ## flake8-imort-conventions\n # \"INP\", ## flake8-no-pep420\n # \"ISC\", ## flake8-implicit-str-concat\n # \"NPY\", ## NumPy-specific rules\n # \"PD\", ## pandas-vet\n # \"PGH\", ## pygrep-hooks\n # \"PIE\", ## flake8-pie\n \"PL\", ## pylint\n # \"PT\", ## flake8-pytest-style\n # \"PTH\", ## flake8-use-pathlib\n # \"PYI\", ## flake8-pyi\n # \"RET\", ## flake8-return\n # \"RSE\", ## flake8-raise\n \"RUF\", ## ruf-specific rules\n # \"SIM\", ## flake8-simplify\n # \"SLF\", ## flake8-self\n # \"TCH\", ## flake8-type-checking\n \"TID\", ## flake8-tidy-imports\n \"TRY\", ## tryceratops\n \"UP\", ## pyupgrade\n # \"YTT\" ## flake8-2020\n]\n# unfixable = []\n\n# Exclude a variety of commonly ignored directories.\nexclude = [\n \".bzr\",\n \".direnv\",\n \".eggs\",\n \".git\",\n \".ruff_cache\",\n \".venv\",\n \"__pypackages__\",\n \"__pycache__\",\n \"*.pyc\",\n]\n\n[lint.per-file-ignores]\n\"__init__.py\" = [\"D104\"]\n\n## Allow unused variables when underscore-prefixed.\n# dummy-variable-rgx = \"^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$\"\n\n[lint.mccabe]\nmax-complexity = 10\n\n[lint.isort]\ncombine-as-imports = true\nforce-sort-within-sections = true\nforce-wrap-aliases = true\n## Use a single line after each import block.\nlines-after-imports = 1\n## Use a single line between direct and from import\nlines-between-types = 1\n## Order imports by type, which is determined by case,\n# in addition to alphabetically.\norder-by-type = true\nrelative-imports-order = \"closest-to-furthest\"\n## Automatically add imports below to top of files\nrequired-imports = [\"from __future__ import annotations\"]\n## Define isort section priority\nsection-order = [\n \"future\",\n \"standard-library\",\n \"first-party\",\n \"local-folder\",\n \"third-party\",\n]\nTodo
See my full database directory (meant to be copied to either core/database or modules/database):
core/database on Github
","tags":["standard-project-files","python","sqlalchemy"]},{"location":"programming/standard-project-files/python/sqlalchemy/database_dir.html","title":"app/core/database","text":"My standard SQLAlchemy base setup. The files in the core/database directory of my projects provides a database config from a dataclass (default values create a SQLite database at the project root), a SQLAlchemy Base, and methods for getting SQLAlchemy Engine and Session.
from __future__ import annotations\n\nfrom .annotated import INT_PK, STR_10, STR_255\nfrom .base import Base\nfrom .db_config import DBSettings\nfrom .methods import get_db_uri, get_engine, get_session_pool\nfrom .mixins import TableNameMixin, TimestampMixin\nCustom annotations live in database/annotated
from __future__ import annotations\n\nfrom .annotated_columns import INT_PK, STR_10, STR_255\nfrom __future__ import annotations\n\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\nfrom typing_extensions import Annotated\n\n## Annotated auto-incrementing integer primary key column\nINT_PK = Annotated[\n int, so.mapped_column(sa.INTEGER, primary_key=True, autoincrement=True, unique=True)\n]\n\n## SQLAlchemy VARCHAR(10)\nSTR_10 = Annotated[str, so.mapped_column(sa.VARCHAR(10))]\n## SQLAlchemy VARCHAR(255)\nSTR_255 = Annotated[str, so.mapped_column(sa.VARCHAR(255))]\nMixin classes can be used with classes that inherit from the SQLAlchemy Base class to add extra functionality.
Example
Automatically add a created_at and updated_at column by inheriting from TimestampMixin
...\n\nclass ExampleModel(Base, TimestampMixin):\n \"\"\"Class will have a created_at and modified_at timestamp applied automatically.\"\"\"\n __tablename__ = \"example\"\n\n id: Mapped[int] = mapped_column(sa.INTEGER, primary_key=True, autoincrement=True, unique=True)\n\n ...\nfrom __future__ import annotations\n\nfrom .classes import TableNameMixin, TimestampMixin\nfrom __future__ import annotations\n\nimport pendulum\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\n\nclass TimestampMixin:\n \"\"\"Add a created_at & updated_at column to records.\n\n Add to class declaration to automatically create these columns on\n records.\n\n Usage:\n\n ``` py linenums=1\n class Record(Base, TimestampMixin):\n __tablename__ = ...\n\n ...\n ```\n \"\"\"\n\n created_at: so.Mapped[pendulum.DateTime] = so.mapped_column(\n sa.TIMESTAMP, server_default=sa.func.now()\n )\n updated_at: so.Mapped[pendulum.DateTime] = so.mapped_column(\n sa.TIMESTAMP, server_default=sa.func.now(), onupdate=sa.func.now()\n )\n\n\nclass TableNameMixin:\n \"\"\"Mixing to automatically name tables based on class name.\n\n Generates a `__tablename__` for classes inheriting from this mixin.\n \"\"\"\n\n @so.declared_attr.directive\n def __tablename__(cls) -> str:\n return cls.__name__.lower() + \"s\"\nfrom __future__ import annotations\n\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\n\nREGISTRY: so.registry = so.registry()\nMETADATA: sa.MetaData = sa.MetaData()\n\n\nclass Base(so.DeclarativeBase):\n registry = REGISTRY\n metadata = METADATA\nfrom __future__ import annotations\n\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\n\nfrom dataclasses import dataclass, field\n\n\n@dataclass\nclass DBSettings:\n drivername: str = field(default=\"sqlite+pysqlite\")\n user: str | None = field(default=None)\n password: str | None = field(default=None)\n host: str | None = field(default=None)\n port: str | None = field(default=None)\n database: str = field(default=\"app.sqlite\")\n echo: bool = field(default=False)\n\n def __post_init__(self):\n assert self.drivername is not None, ValueError(\"drivername cannot be None\")\n assert isinstance(self.drivername, str), TypeError(\n f\"drivername must be of type str. Got type: ({type(self.drivername)})\"\n )\n assert isinstance(self.echo, bool), TypeError(\n f\"echo must be a bool. Got type: ({type(self.echo)})\"\n )\n if self.user:\n assert isinstance(self.user, str), TypeError(\n f\"user must be of type str. Got type: ({type(self.user)})\"\n )\n if self.password:\n assert isinstance(self.password, str), TypeError(\n f\"password must be of type str. Got type: ({type(self.password)})\"\n )\n if self.host:\n assert isinstance(self.host, str), TypeError(\n f\"host must be of type str. Got type: ({type(self.host)})\"\n )\n if self.port:\n assert isinstance(self.port, int), TypeError(\n f\"port must be of type int. Got type: ({type(self.port)})\"\n )\n assert self.port > 0 and self.port <= 65535, ValueError(\n f\"port must be an integer between 1 and 65535\"\n )\n\n def get_db_uri(self) -> sa.URL:\n try:\n _uri: sa.URL = sa.URL.create(\n drivername=self.drivername,\n username=self.user,\n password=self.password,\n host=self.host,\n port=self.port,\n database=self.database,\n )\n\n return _uri\n\n except Exception as exc:\n msg = Exception(\n f\"Unhandled exception getting SQLAlchemy database URL. Details: {exc}\"\n )\n raise msg\n\n def get_engine(self) -> sa.Engine:\n assert self.get_db_uri() is not None, ValueError(\"db_uri is not None\")\n assert isinstance(self.get_db_uri(), sa.URL), TypeError(\n f\"db_uri must be of type sqlalchemy.URL. Got type: ({type(self.db_uri)})\"\n )\n\n try:\n engine: sa.Engine = sa.create_engine(\n url=self.get_db_uri().render_as_string(hide_password=False),\n echo=self.echo,\n )\n\n return engine\n except Exception as exc:\n msg = Exception(\n f\"Unhandled exception getting database engine. Details: {exc}\"\n )\n\n raise msg\n\n def get_session_pool(self) -> so.sessionmaker[so.Session]:\n engine: sa.Engine = self.get_engine()\n assert engine is not None, ValueError(\"engine cannot be None\")\n assert isinstance(engine, sa.Engine), TypeError(\n f\"engine must be of type sqlalchemy.Engine. Got type: ({type(engine)})\"\n )\n\n session_pool: so.sessionmaker[so.Session] = so.sessionmaker(bind=engine)\n\n return session_pool\nfrom __future__ import annotations\n\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\n\ndef get_db_uri(\n drivername: str = \"sqlite+pysqlite\",\n username: str | None = None,\n password: str | None = None,\n host: str | None = None,\n port: int | None = None,\n database: str = \"demo.sqlite\",\n) -> sa.URL:\n assert drivername is not None, ValueError(\"drivername cannot be None\")\n assert isinstance(drivername, str), TypeError(\n f\"drivername must be of type str. Got type: ({type(drivername)})\"\n )\n if username is not None:\n assert isinstance(username, str), TypeError(\n f\"username must be of type str. Got type: ({type(username)})\"\n )\n if password is not None:\n assert isinstance(password, str), TypeError(\n f\"password must be of type str. Got type: ({type(password)})\"\n )\n if host is not None:\n assert isinstance(host, str), TypeError(\n f\"host must be of type str. Got type: ({type(host)})\"\n )\n if port is not None:\n assert isinstance(port, int), TypeError(\n f\"port must be of type int. Got type: ({type(port)})\"\n )\n assert database is not None, ValueError(\"database cannot be None\")\n assert isinstance(database, str), TypeError(\n f\"database must be of type str. Got type: ({type(database)})\"\n )\n\n try:\n db_uri: sa.URL = sa.URL.create(\n drivername=drivername,\n username=username,\n password=password,\n host=host,\n port=port,\n database=database,\n )\n\n return db_uri\n except Exception as exc:\n msg = Exception(\n f\"Unhandled exception creating SQLAlchemy URL from inputs. Details: {exc}\"\n )\n\n raise msg\n\n\ndef get_engine(db_uri: sa.URL = None, echo: bool = False) -> sa.Engine:\n assert db_uri is not None, ValueError(\"db_uri is not None\")\n assert isinstance(db_uri, sa.URL), TypeError(\n f\"db_uri must be of type sqlalchemy.URL. Got type: ({type(db_uri)})\"\n )\n\n try:\n engine: sa.Engine = sa.create_engine(url=db_uri, echo=echo)\n\n return engine\n except Exception as exc:\n msg = Exception(f\"Unhandled exception getting database engine. Details: {exc}\")\n\n raise msg\n\n\ndef get_session_pool(engine: sa.Engine = None) -> so.sessionmaker[so.Session]:\n assert engine is not None, ValueError(\"engine cannot be None\")\n assert isinstance(engine, sa.Engine), TypeError(\n f\"engine must be of type sqlalchemy.Engine. Got type: ({type(engine)})\"\n )\n\n session_pool: so.sessionmaker[so.Session] = so.sessionmaker(bind=engine)\n\n return session_pool\nUseful software utilities I reach for often enough to document
Warning
In progress
"},{"location":"utilities/ssh/index.html","title":"Secure Shell (SSH)","text":"Warning
In progress
Todo
I had a hard time understanding what to do with my private/public keys when I was learning SSH. I don't know why it was a difficult concept for me, but I have worked with enough other people who were confused in the same way I was that I think it's worth it to just spell out what to do with each key.
Your private key (default name is id_rsa) should NEVER leave the server it was created on, and should not be accessible to any other user (chmod 600). There are exceptions to this, such as when uploading a keypair to an Azure or Hashicorp vault, or providing to a Docker container. But in general, when creating SSH tunnels between machines, the private key is meant to stay on the machine it was created on.
Your public key is like your swipe card; when using the ssh command with -f /path/to/id_rsa.pub and the correct user@server combo, you will not need to enter a password to authenticate.
Your public key can also be used for SFTP.
Todo
.ppk file with PuTTYYou can create a keypair using the ssh-keygen utility. This is installed with SSH (openssh-server on Linux, see installation instructions for Windows), and is available cross-platform.
Note
You can run $ ssh-keygen --help on any platform to see a list of available commands. --help is not a valid flag, so you will see a warning unknown option -- -, but the purpose of this is to show available commands.
If you run ssh-keygen without any arguments, you will be guided through a series of prompts, after which 2 files will be created (assuming you chose the defaults): id_rsa (your private key) and id_rsa.pub (your public key).
You can also pass some parameters to automatically answer certain prompts:
-f parameter specifies the output file. This will skip the prompt Enter file in which to save the key$ ssh-keygen -f /path/to/<ssh_key_filename>-f, a private and public (.pub) key will be created-t parameter allows you to specify a key type$ ssh-keygen -t rsadsaecdsaecdsa-sked25519ed25519-sk-b option allows you to specify the number of bits. For rsa keys, the minimum is 1024 and the default is 3072.4096 with: $ ssh-keygen -b 4096Example ssh-keygen commands
ssh-keygen commands## Generate a 4096-bit RSA key named example_key\n$ ssh-keygen -t rsa -b 4096 -f ~/.ssh/example_key\n\n## Generate another 4096b RSA key, this time skipping the password prompt\n# Note: This works better on Linux, I'm not sure what the equivalent is\n# on Windows\n$ ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N \"\"\nYou can (and should) use your SSH key to authenticate as a user on a remote system. There are 2 ways of adding keys for this type of authentication.
Note
Whatever method you use to add your public key to a remote machine, make sure you edit ~/.ssh/config (create the file if it does not exist). The ssh command can use this file to configure connections with SSH keys so you don't have to specify $ ssh -i /path/to/key each time you connect to a remote you've already copied a key to.
## You can set $Host to whatever you want.\n# You will connect with: ssh $Host\nHost example.com\n## The actual FQDN/IP address of the server\nHostName example.com\n## If the remote SSH server is running on a\n# port other than the default 22, set here\n# Port 222\n## The remote user your key is paired to\nUser test\n## The public key exists on the remote.\n# You provide the private key to complete the pair\nIdentityFile ~/.ssh/<your_ssh_key>\n\n## On Windows, set \"ForwardAgent yes\" for VSCode remote editing.\n# Uncomment the line below if on Windows\n# ForwardAgent yes\nOnce you've copied your key, you can simply run ssh $Host, where $Host is the value you set for Host in the ~/.ssh/config file. The SSH client will find the matching Host entry and use the options you specified.
## Note: If you get a message about trusting the host, hit yes.\n# You will need to type your password the first time\n$ ssh-copy-id -i ~/.ssh/your_key.pub test@example\nYou can also manually copy your public keyfile (.pub) to a remote host and cat the contents into ~/.ssh/authorized_keys. The most straightforward way of accomplishing this is to use scp to copy the keyfile to your remote host, typing the password to authenticate, then following up by logging in directly with ssh.
Instructions:
.pub keyfile$ ssh-copy-id -i /path/to/id_rsa.pub test@example.com:/home/test$ ssh test@example.com$ ssh example.com~/.ssh/config file, using the instruction in the note in \"Install an SSH key on a Remote Machine for passwordless login\".pub keyfile from the user's home into .ssh.ssh directory does not exist, create it with mkdir .ssh$ mv id_rsa.pub .ssh.ssh and cat the contents of id_rsa.pub into authorized_keys$ cd .ssh && cat id_rsa.pub authorized_keysid_rsa.pub key. Now that it's in authorized_keys, you don't need the keyfile on the remote machine anymore.It is crucial your chmod permissions are set properly on the ~/.ssh directory. Invalid permissions will lead to errors when trying to ssh into remote machines.
Check the table below for the chmod values you should use. To set a value (for example on the .ssh directory itself and the keypair):
$ chmod 700 ~/.ssh\n$ chmod 644 ~/.ssh/id_rsa{.pub}\n~/.ssh/ There is no general requirement to keep the entire contents of this directory secret, but the recommended permissions are read/write/execute for the user, and not accessible by others. 700 ~/.ssh/authorized_keys This file is not highly sensitive, but the recommended permissions are read/write for the user, and not accessible by others| 600 ~/.ssh/config Because of the potential for abuse, this file must have strict permissions: read/write for the user, and not writable by others 600 ~/.ssh/identity ~/.ssh/id_dsa ~/.ssh/id_rsa These files contain sensitive data and should be readable by the user but not accessible by others (read/write/execute) 600 ~/.ssh/identity.pub ~/.ssh/id_dsa.pub ~/.ssh/id_rsa.pub Contains the public key for authentication. These files are not sensitive and can (but need not) be readable by anyone. 644 *(table data source: Superuser.com answer)
","tags":["utilities","ssh"]},{"location":"tags.html","title":"tags","text":""},{"location":"tags.html#alembic","title":"alembic","text":"My personal knowledgebase. Use the sections along the top (i.e. Programming) to navigate areas of the KB.
\ud83d\udc0d Python
Python standard project filesnoxfile.pyruffpython .gitignoredynaconf\ud83c\udd7f\ufe0f Powershell
Warning
In progress...
"},{"location":"programming/index.html","title":"Programming","text":"Notes & code snippets I want to remember/reference.
Use the sections along the left to read through my notes and see code examples. You are in the Programming section, but might want to check out my Standard project files for Docker (as an example).
Warning
In progress...
Todo
Notes, links, & reference code for Docker/Docker Compose.
Warning
In progress...
Todo
dev vs prodENV vs ARGEXPOSECMD vs ENTRYPOINT vs RUNdocker-compose.ymlversion (required) and volumes (optional))depends_on)You can take advantage of Docker's BuildKit, which caches Docker layers so subsequent rebuilds with docker build are much faster. BuildKit works by keeping a cache of the \"layers\" in your Docker container, rebuilding a layer only if changes have been made. What this means in practice is that you can separate the steps you use to build your container into stages like base, build, and run, and if nothing in your build layer has changed (i.e. no new dependencies added), that layer will not be rebuilt.
In this example, I am building a simple Python app inside a Docker container. The Python code itself does not matter for this example.
To illustrate the differences in a multistage Dockerfile, let's start with a \"flat\" Dockerfile, and modify it with build layers. This is the basic Dockerfile:
Example flat Dockerfile## Start with a Python 3.11 Docker base\nFROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\n## Set the CWD inside the container\nWORKDIR /app\n\n## Copy Python requirements.txt file into container & install dependencies\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\n## Copy project source from host into container\nCOPY ./src .\n\n## Expose port 8000, which we can pretend the Python app uses to serve the application\nEXPOSE 8000\n## Run the Python app\nCMD [\"python\", \"main.py\"]\nIn this example, any changes to the code or dependencies will cause the entire container to rebuild each time. This is slow & inefficient, and leads to a larger container image. We can break these stages into multiple build layers. In the example below, the container is built in 3 \"stages\": base, build, and run:
## Start with the python:3.11-slim Docker image as your \"base\" layer\nFROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\n## Create a \"build\" layer, where you setup your Python environment\nFROM base AS build\n\n## Set the CWD inside the container\nWORKDIR /app\n\n## Copy Python requirements.txt file into container & install dependencies\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\n## Inherit from the build layer\nFROM build AS run\n\n## Set the CWD inside the container\nWORKDIR /app\n\n## Copy project source from host into container\nCOPY ./src .\n\n## Expose port 8000, which we can pretend the Python app uses to serve the application\nEXPOSE 8000\n## Run the Python app\nCMD [\"python\", \"main.py\"]\nLayers:
base: The base layer provides a common environment for the rest of the layers.ENV variables, which persist across layersARG lines can be set per-layer, and will need to be re-set for each new layer. This example does not use any ARG lines, but be aware that build arguments you set with ARG are only present for the layer they are declared in. If you create a new layer and want to access the same argument, you will need to set the ARG value again in the new layerbuild: The build layer is where you install your Python dependencies.apt/apt-getpython:3.11-slim base image is built on Debian. If you are using a different Dockerfile, i.e. python:3.11-alpine, use the appropriate package manager (i.e. apk for Alpine, rpm for Fedora/OpenSuSE, etc) to install packages in the build layerrun: Finally, the run layer executes the code built in the previous base & build steps. It also exposes port 8000 inside the container to the host, which can be mapped with docker run -p 1234:8000, where 1234 is the port on your host you want to map to port 8000 inside the container.Using this method, each time you run docker build after the first, only layers that have changed in some way will trigger a rebuild. For example, if you add a Python dependency with pip install <pkg> and update the requirements.txt file with pip freeze > requirements.txt, the build layer will be rebuilt. If you make changes to your Python application, the run layer will be rebuilt. Each layer that does not need to be rebuilt reduces the overall build time of the container, and only the run layer will be saved as your image, leading to smaller Docker images.
With multistage builds, you can also create a dev and prod layer, which you can target with docker run or a docker-compose.yml file. This allows you to build the development & production version of an application using the same Dockerfile.
Let's modify the multistage Dockerfile example from above to add a dev and prod layer. Modifications to the multistage Dockerfile include adding an ENV variable for storing the app's environment (dev/prod). In my projects, I use Dynaconf to manage app configurations depending on my environment. Dynaconf allows you to set an ENV variable called $ENV_FOR_DYNACONF so you can control app configurations per-environment (Dynaconf environment docs).
FROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\nFROM base AS build\n\nWORKDIR /app\n\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\n## Use target: dev to build this step\nFROM build AS dev\n\n## Set the Dynaconf env to dev\nENV ENV_FOR_DYNACONF=dev\n## Tell Dynaconf to always load from the environment first while in the container\nENV DYNACONF_ALWAYS_LOAD_ENV_VARS=True\n\nWORKDIR /app\nCOPY ./src .\n\n############\n# Optional #\n############\n# Export ports, set an entrypoint/CMD, etc\n# Note: This is normally handled by your orchestrator (docker-compose, Azure Container App, etc).\n# If you are buliding/running the container directly, uncomment the EXPOSE & COMMAND lines below\n\n# EXPOSE 5000\n# CMD [\"python\", \"main.py\"]\n\n## Use target: prod to build this step\nFROM build AS prod\n\n## Set the Dynaconf env to prod\nENV ENV_FOR_DYNACONF=prod\n## Tell Dynaconf to always load from the environment first while in the container\nENV DYNACONF_ALWAYS_LOAD_ENV_VARS=True\n\nWORKDIR /app\nCOPY ./src .\n\n############\n# Optional #\n############\n# Export ports, set an entrypoint/CMD, etc\n# Note: This is normally handled by your orchestrator (docker-compose, Azure Container App, etc)\n# If you are buliding/running the container directly, uncomment the EXPOSE & COMMAND lines below\n\n# EXPOSE 5000\n# CMD [\"python\", \"main.py\"]\nWith this multistage Dockerfile, you can target a specific layer with docker built --target <layer-name> (i.e. docker build --target dev). This will run through the base and build layers, but skip the prod layer.
You can also target a specific layer in a docker-compose.yml file:
version: \"3.8\"\n\nservices:\n\n my_app:\n container_name: my-application\n restart: unless-stopped\n build:\n ## The \"root\" directory for Docker compose. If your Dockerfile/project\n # are in a subdirectory, specify it here.\n context: .\n ## Set the name/path to the Dockerfile, keeping in mind the context you set above\n dockerfile: Dockerfile\n ## Target the \"dev\" layer of the Dockerfile\n target: dev\n ## Set the working directory inside the container to /app\n working_dir: /app\n ## Set the command to run inside the container. Equivalent to CMD in the Dockerfile\n command: python main.py\n volumes:\n ## Mount the project's code directory in the container so changes don't require a rebuild\n - ./src:/app\n ports:\n ## Expose port 8000 in the container, set to port 80 on the host\n - 80:8000\n ...\nThe example docker-compose.yml file above demonstrates targeting the dev layer of the multistage Dockerfile above it. We also set the entrypoint (instead of using CMD in the Dockerfile), and expose port 8000 in the container.
The ENV and ARG commands in a Dockerfile can be used to control how an image is built and how it functions when live. The differences between an ENV and an ARG are outlined below.
Note
This list is not a complete comparison between ENV and ARG. For more information, please check the Docker build documentation guide.
ENV$ENV_VAR_NAME.docker build -e, or the environment: stanza in a docker-compose.yml file.build and run phases when building a container.docker build or docker compose build), ENV variables will always use the value declared in the Dockerfile.docker run or docker compose up), the values can be overridden with docker run -e/--env or the environment: stanza in a docker-compose.yml file.ARGdocker build --build-arg ARG_NAME=value, or the build: args: stanza in a docker-compose.yml fileExample:
ENV vs ARGFROM python:3.11-slim AS base\n\n## This env variable will be available in the build layer\nENV PYTHONDONTWRITEBYTECODE 1\n\n## Define a required ARG, without setting its value. The build will fail if this arg is not passed\nARG SOME_VAR\n## Define an ARG and set a default value\nARG SOME_OTHER_VAR_ARG=1.0\n\n## Set an ENV value, using an ARG's value, to make it available throughout the rest of the build\nENV SOME_OTHER_VAR $SOME_OTHER_VAR_ARG\n\nFROM base AS build\n\n## Re-define SOME_OTHER_VAR_ARG from the SOME_OTHER_VAR ENV variable.\n# The ENV variable carries into the build layer, but the ARG defined\n# in the base layer is not.\nARG SOME_OTHER_VAR_ARG=$SOME_OTHER_VAR\nBuild ARGS are useful for setting things like a software version number, i.e. when downloading a specific software release from Github. You can set a build arg for the release version, i.e. ARG RELEASE_VER, and provide it at buildtime with docker build --build-arg RELEASE_VER=1.2.3, or in a docker-compose.yml file like:
...\n\nservices:\n\n service1:\n build:\n context: .\n args:\n RELEASE_VER: 1.2.3\n\n...\nENV variables, meanwhile, can store things like a database password or some other secret, or configurations for the app.
...\n\nservices:\n\n service1:\n container_name: service1\n restart: unless-stopped\n build:\n ...\n ...\n environment:\n ## Load $RELEASE_VERSION from the host's environment or a .env file\n RELEASE_VER: ${RELEASE_VERSION:-1.2.3}\n\n...\nIn previous examples you have seen the EXPOSE line in a Dockerfile. This command exposes a network port from within the container to the host. This is useful if your containerized application utilizes network ports (i.e. running a web frontend on port 8000), and you are running the container directly with docker run instead of through an orchestrator like Docker Compose or Kubernetes.
Note
When using an orchestrator like docker-compose, kubernetes, hashicorp nomad, etc, it is not necessary (and often counterproductive) to define EXPOSE lines in a Dockerfile. It is better to define port binds between the host and container using the orchestrator's capabilities, i.e. the ports: stanza of a docker-compose.yml file.
When building & running a container image locally or without an orchestrator, you can add these sections to a Dockerfile so when you run the built container image, you can bind ports with docker run -p $HOST_PORT:$CONTAINER_PORT.
Example:
Example EXPOSE syntax...\n\nFROM build AS run\n\n...\n\n## Expose port 8000 in the container to the host running this container image\nEXPOSE 8000\n\n## Start a Uvicorn server inside the container. The web server runs on port 8000 (by default)\nCMD [\"uvicorn\", \"main:app\", \"--reload\"]\nAfter building this container, you can run it and bind to a port on the host (i.e. port 80) with docker run -rm -p 80:8000 ..., or by specifying the port binding in a docker-compose.yml file
Warning
If you are using Docker Compose, comment/remove the EXPOSE and CMD lines in your container and pass the values in through Docker Compose
...\n\nservices:\n\n service1:\n ...\n ## Set the container startup command here, instead of with RUN in the Dockerfile\n command: uvicorn main:app --reload\n ports:\n ## Serve the container application running on port 8000 in the container\n # over port 80 on the host.\n - 80:8000\nRUNRUN is executed on top of the current base image.neovim container inside of a Dockerfile built on top of ubuntu:latest image: RUN apt-get update -y && apt-get install -y neovimRUN show their output in the console as the container is built, but are not executed when the built container image is run with docker run or docker compose upCMDCMD execute when you run the container with docker run or docker compose updocker run myimage cat log.txtcat log.txt command overrides the CMD defined in the containerCMD defined in your image is executed. If you specify more than one, all but the last CMD will execute.CMD command supersedes ENTRYPOINT, and should almost always be used instead of ENTRYPOINT.ENTRYPOINTENTRYPOINT functions almost the same way as CMD, but should be used only when extending an existing image (i.e. nginx, tomcat, etc)ENTRYPOINT to an existing container will change the way that container executes, running the underlying Dockerfile logic with an ad-hoc command you provide.docker run --entrypointCMD and an ENTRYPOINT, the CMD line will be provided as arguments to the ENTRYPOINT, meaning you can do things like cat a file with a CMD, then \"pipe\" the command's output into an ENTRYPOINTCMD in your Dockerfiles, unless you are aware of and fully understand a reason to use ENTRYPOINT instead.Doc pages for specific containers/container stacks I use.
"},{"location":"programming/docker/my_containers/portainer/portainer.html","title":"Portainer","text":"At least 1 Portainer server must be available for agents to connect to. Copy this script to a file, i.e. run-portainer.sh.
Note
Don't forget to set chmod +x run-portainer.sh, or execute the script with bash run-portainer.sh.
#!/bin/bash\n\nWEBUI_PORT=\"9000\"\n## Defaults to 'portainer' if empty\nCONTAINER_NAME=\n## Defaults to a named volume, portainer_data.\n# Note: create this volume with $ docker volume create portainer_data\nDATA_DIR=\n\necho \"\"\necho \"Checking for new image\"\necho \"\"\n\ndocker pull portainer/portainer-ce\n\necho \"\"\necho \"Restarting Portainer\"\necho \"\"\n\ndocker stop portainer && docker rm portainer\n\ndocker run -d \\\n -p 8000:8000 \\\n -p ${WEBUI_PORT:-9000}:9000 \\\n --name=${CONTAINER_NAME:-portainer} \\\n --restart=unless-stopped \\\n -v /var/run/docker.sock:/var/run/docker.sock \\\n -v ${DATA_DIR:-portainer_data}:/data \\\n portainer/portainer-ce\nStart a Portainer in agent mode to allow connection from a Portainer server. This setup is done in the Portainer server's webUI.
Warning
It is probably easier to just download the agent script from the Portainer server when you are adding a connection. It offers a command you can run to simplify setup.
run-portainer_agent.sh#!/bin/bash\n\necho \"\"\necho \"Checking for new container image\"\necho \"\"\n\ndocker pull portainer/agent\n\necho \"\"\necho \"Restarting Portainer\"\necho \"\"\n\ndocker stop portainer-agent && docker rm portainer-agent\n\ndocker run -d \\\n -p 9001:9001 \\\n --name=portainer-agent \\\n --restart=unless-stopped \\\n -v /var/run/docker.sock:/var/run/docker.sock \\\n -v /var/lib/docker/volumes:/var/lib/docker/volumes \\\n portainer/agent:latest\nRun this script to backup the Portainer portainer_data volume. Backup will be placed at ${CWD}/portainer_data_backup
#!/bin/bash\n\n# Name of container containing volume(s) to back up\nCONTAINER_NAME=${1:-portainer}\nTHIS_DIR=${PWD}\nBACKUP_DIR=$THIS_DIR\"/portainer_data_backup\"\n# Directory to back up in container\nCONTAINER_BACKUP_DIR=${2:-/data}\n# Container image to use as temporary backup mount container\nBACKUP_IMAGE=${3:-busybox}\nBACKUP_METHOD=${4:-tar}\nDATA_VOLUME_NAME=${5:-portainer-data}\n\nif [[ ! -d $BACKUP_DIR ]]; then\n echo \"\"\n echo $BACKUP_DIR\" does not exist. Creating.\"\n echo \"\"\n\n mkdir -pv $BACKUP_DIR\nfi\n\nfunction RUN_BACKUP () {\n\n sudo docker run --rm --volumes-from $1 -v $BACKUP_DIR:/backup $BACKUP_IMAGE $2 /backup/backup.tar $CONTAINER_BACKUP_DIR\n\n}\n\nfunction RESTORE_BACKUP () {\n\n echo \"\"\n echo \"The restore function is experimental until this comment is removed.\"\n echo \"\"\n read -p \"Do you want to continue? Y/N: \" choice\n\n case $choice in\n [yY] | [YyEeSs])\n echo \"\"\n echo \"Test print: \"\n echo \"sudo docker create -v $CONTAINER_BACKUP_DIR --name $DATA_VOLUME_NAME\"2\" $BACKUP_IMAGE true\"\n echo \"\"\n echo \"Test print: \"\n echo \"sudo docker run --rm --volumes-from $DATA_VOLUME_NAME\"2\" -v $BACKUP_DIR:/backup $BACKUP_IMAGE tar xvf /backup/backup.tar\"\n echo \"\"\n\n echo \"\"\n echo \"Compare to original container: \"\n echo \"\"\n echo \"Test print: \"\n echo \"sudo docker run --rm --volumes-from $CONTAINER_NAME -v $BACKUP_DIR:/backup $BACKUP_IMAGE ls /data\"\n ;;\n [nN] | [NnOo])\n echo \"\"\n echo \"Ok, nevermind.\"\n echo \"\"\n ;;\n esac\n\n}\n\n# Run a temporary container, mount volume to back up, create backup file\ncase $1 in\n \"-b\" | \"--backup\")\n case $BACKUP_METHOD in\n \"tar\")\n echo \"\"\n echo \"Running \"$BACKUP_METHOD\" backup using image \"$BACKUP_IMAGE\n echo \"\"\n\n RUN_BACKUP $CONTAINER_NAME \"tar cvf\"\n ;;\n esac\n ;;\n \"-r\" | \"--restore\")\n ;;\nesac\nNote
I do not use this method. I find it easier to run Portainer with a shell script.
Portainer docker-compose.ymlversion: \"3\"\n\nservices:\n\n portainer:\n image: portainer/portainer-ce:linux-amd64\n container_name: portainer\n restart: unless-stopped\n volumes:\n - /var/run/docker.sock:/var/run/docker.sock\n - ${PORTAINER_DATA:-./data}:/data\nWarning
In progress...
Todo
Warning
In progress...
Todo
virtualenvcondapdmjupyterlabTodo
When importing Python code into a Jupyter notebook, any changes made in the Python module(s) do not become active in the Jupyter kernel until it's restarted. This is undesirable when working with data that takes a long time up front to prepare.
Add this code to a cell (I usually put it in the very first cell) to automatically reload Python modules on changes, without having to reload the whole notebook kernel.
Automatically reload file on changes## Set notebook to auto reload updated modules\n%load_ext autoreload\n%autoreload 2\nTodo
pre-commit runs and what to do to fix itpre-commitWhen running Jupyter notebooks, it's usually good practice to clear the notebook's output when committing to git. This can be for privacy/security (if you're debugging PII or environment variables in the notebook), or just a tidy repository.
Using pre-commit (check my section on pre-commit) and the nbstripout action, we can automate stripping the notebook each time a git commit is created.
Instructions
pre-commitNote
Warning
If your preferred package manager is not listed below, check the documentation for that package manager for instructions on installing packages.
pip:pip install pre-commitpipx:pipx install pre-commitpdm:pdm add pre-commitTODO:
poetryconda/miniconda/microconda.pre-commit-config.yml with these contents:repos:\n repo: https://github.com/kynan/nbstripout\n rev: 0.6.1\n hooks:\n - id: nbstripout\npre-commit hook with $ pre-commit installNote
If you installed pre-commit in a virtualenv, or with a tool like pdm, make sure the .venv is activated, or you run with $ pdm run pre-commit ...
Now, each time you make a git commit, after writing your commit message, pre-commit will execute nbstripout to strip your notebooks of their output.
Warning
In progress...
Todo
mkdocsmkdocs.ymlsite_namesite_descriptionrepo_namerepo_urlexclude_docstheme configuration for mkdocs-materialmkdocs-material featuressearchnavigationcontentpluginsliterate-navsection-indexmarkdown_extensionsWarning
In progress...
reference page for more in-depth documentation.Todo
Get-AdUserTodo
A Powershell profile is a .ps1 file, which does not exist by default but is expected at C:\\Users\\<username>\\Documents\\WindowsPowerShell\\Microsoft.PowerShell_profile.ps1, is a file that is \"sourced\"/loaded each time a Powershell session is opened. You can use a profile to customize your Powershell session. Functions declared in your profile are accessible to your whole session. You can set variables with default values, customize your session's colors (by editing a function Prompt {} section), split behavior between regular/elevated prompts, and more.
At the top of your Powershell script, below your docstring, you can declare param() to enable passing -Args to your script, our switches.
<#\n Description: Example script with params\n\n Usage:\n ...\n#>\n\nparam(\n ## Enable transcription, like ~/.bashhistory\n [bool]$Transcript = $True,\n [bool]$ClearNewSession = $True\n)\nCode snippets with little-to-no explanation. Some pages may link to a more in-depth article in the Powershell Reference page
Export a list of installed packages discovered by winget to a .json file, then import the list to reinstall everything. Useful as a backup, or to move to a new computer.
Note
The filename in the examples below, C:\\path\\to\\winget-pkgs.json, can be named anything you want, as long as it has a .json file extension.
## Set a path, the file must be a .json\n$ExportfilePath = \"C:\\path\\to\\winget-pkgs.json\"\n\n## Export package list\nwinget export -o \"$($ExportfilePath)\"\n## Set the path to your export .json file\n$ImportfilePath = \"C:\\path\\to\\winget-pkgs.json\"\n\n## Import package list\nwinget import -i \"$($ImportfilePath)\"\n(Get-Date) \u2013 (Get-CimInstance Win32_OperatingSystem).LastBootUpTime\nFunctions in your profile will be executed automatically if you call them within the profile, but they are also available to your entire session. For example, the Edit-Profile function function can be executed in any session that loads a profile with that function declared!
The function below returns $True if the current Powershell session is elevated, otherwise returns $False.
function Get-ElevatedShellStatus {\n ## Check if current user is admin\n $Identity = [Security.Principal.WindowsIdentity]::GetCurrent()\n $Principal = New-Object Security.Principal.WindowsPrincipal $Identity\n $AdminUser = $Principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)\n\n return $AdminUser\n}\n\n## Declare variable for references throughout script.\n# Can be used to prevent script from exiting/crashing.\n$isAdmin = $(Get-ElevatedShellStatus)\nfunction Open-AsAdmin {\n <#\n Run command as admin, or start new admin session if no args are passed\n #>\n if ($args.Count -gt 0) { \n $argList = \"& '\" + $args + \"'\"\n Start-Process \"$psHome\\powershell.exe\" -Verb runAs -ArgumentList $argList\n }\n else {\n Start-Process \"$psHome\\powershell.exe\" -Verb runAs\n }\n}\nfunction Edit-Profile {\n <#\n Open current profile.ps1 in PowerShell ISE\n #>\n If ($host.Name -match \"ise\") {\n ## Edit in PowerShell ISE, if available\n $psISE.CurrentPowerShellTab.Files.Add($profile.CurrentUserAllHosts)\n }\n Else {\n ## Edit in Notepad if no PowerShell ISE found\n notepad $profile.CurrentUserAllHosts\n }\n}\nConda is a Python package manager. It's a very useful utility, but I've found adding it to my $PATH or Powershell profile results in a very slow session load in new tabs/windows. Adding the SOURCE_CONDA function below, and settings an alias to the conda command to call this function instead (Set-Alias conda SOURCE_CONDA), delays the sourcing of the Conda path. The first time you run conda in a new session, you will see a message that Conda has been initialized and you need to re-run your command. You can simply press the up key on your keyboard and run it again; now that Conda is initialized, it will execute, and once a Powershell session is loaded, sourcing Conda is much quicker!
function SOURCE_CONDA {\n <#\n Initialize Conda only when the conda command is run.\n Conda takes a while to initialize, and is not needed in\n every PowerShell session\n #>\n param(\n [String]$CONDA_ROOT = \"%USERPROFILE%\\mambaforge\\Scripts\\conda.exe\"\n )\n\n #region conda initialize\n # !! Contents within this block are managed by 'conda init' !!\n (& \"$CONDA_ROOT\" \"shell.powershell\" \"hook\") | Out-String | Invoke-Expression\n #endregion\n\n Write-Host \"Conda initialized. Run your command again.\"\n\n}\nUnix OSes have a very nice, simple command, uptime, that will simply print the number of days/hours/minutes your machine has been online. The Powershell syntax for this is difficult for me to remember, so my Powershell profile has an uptime function declared.
function uptime {\n ## Print system uptime\n\n If ($PSVersionTable.PSVersion.Major -eq 5 ) {\n Get-WmiObject win32_operatingsystem |\n Select-Object @{EXPRESSION = { $_.ConverttoDateTime($_.lastbootuptime) } } | Format-Table -HideTableHeaders\n }\n Else {\n net statistics workstation | Select-String \"since\" | foreach-object { $_.ToString().Replace('Statistics since ', '') }\n }\n}\nUnix OSes have a simple, easy to remember unzip command. This function tries to emulate that simplicity.
function unzip ($file) {\n ## Extract zip archive to current directory\n\n Write-Output(\"Extracting\", $file, \"to\", $pwd)\n $fullFile = Get-ChildItem -Path $pwd -Filter .\\cove.zip | ForEach-Object { $_.FullName }\n Expand-Archive -Path $fullFile -DestinationPath $pwd\n}\nUnix OSes have a useful utility called touch, which will create an empty file if one doesn't exist at the path you pass it, i.. touch ./example.txt. This function tries to emulate that usefulness and simplicity.
function touch($file) {\n ## Create a blank file at $file path\n\n \"\" | Out-File $file -Encoding ASCII\n}\nAdding this function to your Powershell profile lets you lock your computer's screen by simply running lock-screen in a Powershell session.
function lock-machine {\n ## Set computer state to Locked\n\n try {\n rundll32.exe user32.dll, LockWorkStation\n }\n catch {\n Write-Error \"Unhandled exception locking machine. Details: $($_.Exception.Message)\"\n }\n\n}\nConcepts & code snippets I want to document, ideas I don't want to forget, things I've found useful, etc.
For short/quick references with code examples, check the snippets section.
Notes, links, & reference code for Python programming.
Warning
In progress...
"},{"location":"programming/python/ad-hoc-codecell.html","title":"Ad-Hoc, Jupyter-like code \"cells\"","text":"Note
ipykernel package installed.ipykernel (i.e. JetBrains), from what I've read, but I have not tested it anywhere else.You can create ad-hoc \"cells\" in any .py file (i.e. notebook.py, nb.py, etc) by adding # %% line(s) to the file.
# %%\nYou can also create multiple code cells by adding more # %% lines:
# %%\nmsg = \"This code is executed in the first code cell. It sets the value of 'msg' to this string.\"\n\n# %%\n## Display the message\nmsg\n\n# %%\n150 + 150\nNote
\ud83d\udea7Page is still in progress\ud83d\udea7
"},{"location":"programming/python/dataclasses.html#what-is-a-dataclass","title":"What is adataclass?","text":"A Python data class is a regular Python class that has the @dataclass decorator. It is specifically created to hold data (from python.land).
Dataclasses reduce the boilerplate code when creating a Python class. As an example, below are 2 Python classes: the first is written with Python's standard class syntax, and the second is the simplified dataclass:
Standard class vs dataclassfrom dataclasses import dataclass\n\n## Standard Python class\nclass User:\n def __init__(self, name: str, age: int, enabled: bool):\n self.name = name\n self.age = age\n self.enabled = enabled\n\n\n## Python dataclass\n@dataclass\nclass User:\n user: str\n age: int\n enabled: bool\nWith a regular Python class, you must write an __init__() method, define all your parameters, and assign the values to the self object. The dataclass removes the need for this __init__() method and simplifies writing the class.
This example is so simple, it's hard to see the benefits of using a dataclass over a regular class. Dataclasses are a great way to quickly write a \"data container,\" i.e. if you're passing results back from a function:
Example dataclass function returnfrom dataclasses import dataclass\n\n@dataclass\nclass FunctionResults:\n original_value: int\n new_value: int\n\n\ndef some_function(x: int = 0, _add: int = 15) -> FunctionResults:\n y = x + _add\n\n return FunctionResults(original_value=x, new_value=y)\n\nfunction_results: FunctionResults = some_function(x=15)\n\nprint(function_results.new_value) # = 30\nInstead of returning a dict, returning a dataclass allows for accessing parameters using .dot.notation, like function_results.original_value, instead of `function_results[\"original_value\"].
A mixin class is a pre-defined class you define with certain properties/methods, where any class inheriting from this class will have access to those methods.
For example, the DictMixin dataclass below adds a method .as_dict() to any dataclass that inherits from DictMixin.
Adds a .as_dict() method to any dataclass inheriting from this class. This is an alternative to dataclasses.asdict(_dataclass_instance), but also not as flexible.
from dataclasses import dataclass\nfrom typing import Generic, TypeVar\n\n## Generic type for dataclass classes\nT = TypeVar(\"T\")\n\n\n@dataclass\nclass DictMixin:\n \"\"\"Mixin class to add \"as_dict()\" method to classes. Equivalent to .__dict__.\n\n Adds a `.as_dict()` method to classes that inherit from this mixin. For example,\n to add `.as_dict()` method to a parent class, where all children inherit the .as_dict()\n function, declare parent as:\n\n # ``` py linenums=\"1\"\n # @dataclass\n # class Parent(DictMixin):\n # ...\n # ```\n\n # and call like:\n\n # ```py linenums=\"1\"\n # p = Parent()\n # p_dict = p.as_dict()\n # ```\n \"\"\"\n\n def as_dict(self: Generic[T]):\n \"\"\"Return dict representation of a dataclass instance.\n\n Description:\n self (Generic[T]): Any class that inherits from `DictMixin` will automatically have a method `.as_dict()`.\n There are no extra params.\n\n Returns:\n A Python `dict` representation of a Python `dataclass` class.\n\n \"\"\"\n try:\n return self.__dict__.copy()\n\n except Exception as exc:\n raise Exception(\n f\"Unhandled exception converting class instance to dict. Details: {exc}\"\n )\n\n## Demo inheriting from DictMixin\n@dataclass\nclass ExampleDictClass(DictMixin):\n x: int\n y: int\n z: str\n\n\nexample: ExampleDictclass = ExampleDictClass(x=1, y=2, z=\"Hello, world!\")\nexample_dict: dict = example.as_dict()\n\nprint(example_dict) # {\"x\": 1, \"y\": 2, \"z\": \"Hello, world!\"}\nInherit from the json.JSONEncoder class to allow returning a DataClass as a JSON encode-able dict.
import json\nfrom dataclasses import asdict\n\nclass DataclassEncoder(json.JSONEncoder):\n def default(self, obj):\n if hasattr(obj, '__dict__'):\n return obj.__dict__\n elif hasattr(obj, '__dataclass_fields__'):\n return asdict(obj)\n return super().default(obj)\n\nperson = Person(name=\"Alice\", age=25)\njson.dumps(person, cls=DataclassEncoder) # Returns '{\"name\": \"Alice\", \"age\": 25}'\nPython dataclasses do not have built-in validation, like a Pydantic class. You can still use type hints to define variables, like name: str = None, but it has no actual effect on the dataclass.
You can use the __post_init__(self) method of a dataclass to perform data validation. A few examples below:
from dataclasses import dataclass\nimport typing as t\nfrom pathlib import Path\n\n\ndef validate_path(p: t.Union[str, Path] = None) -> Path:\n assert p, ValueError(\"Missing an input path to validate.\")\n assert isinstance(p, str) or isinstance(p, Path), TypeError(f\"p must be a str or Path. Got type: ({type(p)})\")\n\n p: Path = Path(f\"{p}\")\n if \"~\" in f\"{p}\":\n p = p.expanduser()\n\n return p\n\n\n@dataclass\nclass ComputerDirectory:\n ## Use | None in the annotation to denote an optional value\n dir_name: str | None = None\n dir_path: t.Union[str, Path] = None\n\n def __post_init__(self):\n if self.dir_name is None:\n ## self.dir_name is allowed to be None\n pass\n else:\n if not isinstance(self.dir_name, str):\n raise TypeError(\"dir_name should be a string.\")\n\n if self.dir_path is None:\n raise ValueError(\"Missing required parameter: dir_path\")\n else:\n ## Validate self.dir_path with the validate_path() function\n self.dir_path = validate_path(p=self.dir_path)\n...
"},{"location":"programming/python/fastapi.html#common-fixes","title":"Common Fixes","text":"...
"},{"location":"programming/python/fastapi.html#fix-failed-to-load-api-definition-not-found-apiv1openapijson","title":"Fix: \"Failed to load API Definition. Not found /api/v1/openapi.json\"","text":"When running behind a proxy (and in some other circumstances) you will get an error when trying to load the /docs endpoint. The error will say \"Failed to load API definition. Fetch error Not Found /api/v1/openapi.json\":
To fix this, simply modify your app = FastAPI() line, adding `openapi_url=\"/docs/openapi\":
...\n\n# app = FastAPI(openapi_url=\"/docs/openapi\")\napp = FastAPI(openapi_url=\"/docs/openapi\")\nTodo
pdmpdm?pdm solve?pdm cheat sheetapplication and a librarypypi with `pdmToc
pdm cheat sheetpdm init Initialize a new project with pdm The pdm tool will ask a series of questions, and build an environment based on your responses pdm add <package-name> Add a Python package to the project's dependencies. pdm will automatically find all required dependencies and add them to the install command, and will ensure the package installed is compatible with your environment. pdm add -d <package-name> Add a Python package to the project's development dependency group. Use this for packages that should not be built with your Python package. Useful for things like formatters (black, ruff, pyflakes, etc), testing suites (pytest, pytest-xidst), automation tools (nox, tox), etc. pdm add -G standard <package-name> Add a Python package to the project's standard group. In this example, standard is a custom dependency group. You can use whatever name you want, and users can install the dependencies in this group with pip install <project-name>[standard] (or with pdm: pdm add <project-name>[standard]) pdm remove <package-name> Remove a Python dependency from the project. Analagous to pip uninstall <package-name> pdm remove -d <package-name> Remove a Python development dependency from the project. pdm remove -G standard <package-name> Remove a Python dependency from the standard (or whatever group name you're targeting) dependency group. standard is an example name. You can use whatever name you want for dependency groups, as long as they adhere to the naming rules. You will be warned if a name is not compatible. pdm update Update all dependencies in the pdm.lock file. pdm update <package-name> Update a specific dependency. pdm update -d <package-name> Update a specific development dependency. pdm update -G standard <package-name> Update a specific dependency in the \"standard\" dependency group. standard is an example name; make sure you're using the right group name. pdm update -d Update all development dependencies. pdm update -G standard Update all dependencies in the dependency group named \"standard\". pdm update -G \"standard,another_group\" Update multiple dependency groups at the same time. pdm lock Lock dependencies to a pdm.lock file. Helps pdm with reproducible builds. pdm run python src/app_name/main.py Run the main.py file using the pdm-controller Python executable. Prepending Python commands with pdm run ensures you are running them with the pdm Python version instead of the system's Python. You can also activate the .venv environment and drop pdm run from the beginning of the command to accomplish the same thing. pdm run custom-script Execute a script defined in pyproject.toml file named custom-script. PDM scripts"},{"location":"programming/python/pdm.html#what-is-pdm","title":"What is PDM?","text":"\ud83d\udd17 PDM official site
\ud83d\udd17 PDM GitHub
PDM (Python Dependency Manager) is a tool that helps manage Python projects & dependencies. It does many things, but a simple way to describe it is that it replaces virtualenv and pip, and ensures when you install dependencies, they will be compatible with the current Python version and other dependencies you add to the project. Some dependencies require other dependencies, and where pip might give you an error about a ModuleNotFound, pdm is smart enough to see that there are extra dependencies and will automatically add them as it installs the dependency you asked for.
Note
Unlike pip, pdm uses a pyproject.toml file to manage your project and its dependencies. This is a flexible file where you can add metadata to your project, create custom pdm scripts, & more.
If you have already used tools like poetry or conda, you are familiar with the problem pdm solves. In a nutshell, dependency management in Python is painful, frustrating, and difficult. pdm (and other similar tools) try to solve this problem using a \"dependency matrix,\" which ensures the dependencies you add to your project remain compatible with each other. This approach helps avoid \"dependency hell,\" keeps your packages isolated to the project they were installed for, and skips the ModuleNotFound error you will see in pip when a dependency requires other dependencies to install.
pdm is also useful for sharing code, as anyone else who uses pdm can install the exact same set of dependencies that you installed on your machine. This is accomplished using a \"lockfile,\" which is a special pdm.lock file the pdm tool creates any time you install a dependency in your project. Unless you manually update the package, any time you run pdm install, it will install the exact same set of dependencies, down to the minor release number, keeping surprise errors at bay if you update a package that does not \"fit\" in your current environment by ensuring a specific, compatible version of that package is installed.
pdm can also help you build your projects and publish them to pypi, and can install your development tools (like black, ruff, pytest, etc) outside of your project's environment, separating development and production dependencies. pdm also makes it easy to create dependency \"groups,\" where a user can install your package with syntax like package_name[group]. You may have seen this syntax in the pandas or uvicorn packages, where the documentation will tell you to install like pandas[excel] or uvicorn[standard]. The maintainers of these packages have created different dependency \"groups,\" where if you only need the Excel functionality included in pandas, for example, it will only install the dependencies required for that group, instead of all of the dependencies the pandas package requires.
Lastly, pdm can make you more efficient by providing functionality for scripting. With pdm, you can create scripts like start or start-dev to control executing commands you would manually re-type each time you wanted to run them.
Note
Please refer to the Official PDM installation instructions for instructions. I personally use the pipx method (pipx install pdm), but there are multiple sets of instructions that are updated occasionally, and it is best to check PDM's official documentation for installation/setup instructions.
When you initialize a new project with pdm init, you will be asked a series of questions that will help pdm determine the type of environment to set up. One of the questions asks if your project is a library or application:
Is the project a library that is installable?\nIf yes, we will need to ask a few more questions to include the project name and build backend [y/n] (n):\nWarning
This choice does matter! It affects how Python structures your project, and how the project executes. You will need to answer a couple of extra questions for an \"application,\" but there is a correct way to answer this question when pdm asks it.
Continue reading for more information.
For a more in-depth description of when to choose a \"library\" vs an \"application\", please read the Official PDM documentation.
As a rule of thumb, modules you import into other scripts/apps and do not have a CLI tool (like the pydantic or requests modules) are a \"library.\" A module that includes a CLI and can be executed from the command line, but is built with Python (like black, ruff, pytest, mypy, uvicorn, etc) are an \"application.\"
Todo
src/ layoutsrc/ projectToc
Jump right to a section:
If you're just here for a quick reference, use the sections below for quick pyenv setup & usage instructions. Any sections linked here will have a button that returns you to this TL;DR section, so if you're curious about one of the steps and follow a link you can get right back to it when you're done reading.
This is what the button will look like: \u2934\ufe0f back to TL;DR
See the Install pyenv section for OS-specific installation instructions.
Warning
If you skip the installation instructions, make sure you add pyenv to your PATH. If you need help setting PATH variables, please read the Install pyenv section.
~/.bashrc, add this to the bottom of the file (if in a terminal environment, use an editor like $ nano ~.bashrc):export PATH=\"$HOME/.pyenv/bin:$PATH\"\neval \"$(pyenv init --path)\"\neval \"$(pyenv virtualenv-init -)\"\nPATH variable, add these 2 paths:%USERPROFILE%\\\\.pyenv\\\\pyenv-win\\\\bin\n%USERPROFILE%\\\\.pyenv\\\\pyenv-win\\\\shims\nShow versions available for install. Update the list with $ pyenv update if you don't see the version you want.
$ pyenv install -l\n# 3.xx.x\n# 3.xx.x-xxx\n# ...\n\n$ pyenv install 3.12.1\nWarning
The 3.12.1 version string is used as an example throughout the documentation. Make sure you're using a more recent version, if one is available.
You can check by running $ pyenv update && pyenv install -l.
Examples commands
global version to 3.12.1: $ pyenv global 3.12.1global to multiple versions, 3.12.1 + 3.11.8: $ pyenv global 3.12.1 3.11.8Version-order
The order of the versions you specify does matter. For example, in the command above,3.12.1 is the \"primary\" Python, but 3.11.8 will also be available for tools like pytest, which can run tests against multiple versions of Python, provided they are available in the PATH. local version to 3.11.8, creating a .python-version file in the process: $ pyenv local 3.11.8local version will override the global settinglocal to multiple versions, 3.12.1 + 3.11.8: $ pyenv local 3.12.1 3.11.8shell version to `3.pyenv commands List available pyenv commands Useful as a quick reference if you forget a command name. Does not include help/man text. To see available help for a command, run it without any parameters, i.e. pyenv whence pyenv update Refresh pyenv's repository Updates the pyenv utility, fetches new available Python versions, etc. Run this command occasionally to keep everything up to date. pyenv install 3.xx.xx Install a version of Python If you don't see the version you want, run pyenv update pyenv uninstall 3.xx.xx Uninstall a version of Python that was installed with pyenv. You can uninstall multiple versions at a time with pyenv uninstall 3.11.2 3.11.4 pyenv global 3.xx.xx Set the global/default Python version to use. You can set multiple versions like pyenv global 3.11.4 3.11.6 3.12.2. Run without any args to print the current global Python interpreter(s). pyenv local 3.xx.xx Set the local Python interpreter. Creates a file in the current directory .python-version, which pyenv will detect and will set your interpreter to the version specified. Like pyenv global, you can set multiple versions with pyenv local 3.11.4 3.11.6 3.12.2. Run without any args to print the current global Python interpreter(s). pyenv shell 3.xx.xx Set the Python interpreter for the current shell session. Resets when session exits. Like pyenv global, you can set multiple versions with pyenv shell 3.11.4 3.11.6 3.12.2. Run without any args to print the current global Python interpreter(s). pyenv versions List versions of Python installed with Pyenv & available for use. Versions will be printed as a list of Python version numbers, and may include interpreter paths. pyenv which <executable> Print the path to a pyenv-installed executable, i.e. pip Helps with troubleshooting unexpected behavior. If you suspect pyenv is using the wrong interpreter, check the path with pyenv which python, for example. pyenv rehash Rehash pyenv shims. Run this after switching Python versions with pyenv if you're getting unexpected outputs."},{"location":"programming/python/pyenv.html#what-is-pyenv","title":"What is pyenv","text":"pyenv is a tool for managing versions of Python. It handles downloading the version archive, extracting & installing, and can isolate different versions of Python into their own individual environments.
pyenv can also handle running multiple versions of Python at the same time. For example, when using nox, you can declare a list of Python versions the session should run on, like @nox.session(python=[\"3.11.2\", \"3.11.4\", \"3.12.1\"], name=\"session-name\"). As long as one of the pyenv scopes (shell, local, or global) has all of these versions of Python, nox will run the session multiple times, once for each version declared in python=[].
Note
For more information on the problem pyenv solves, read the \"Why use pyenv?\" section below.
If you just want to see installation & setup instructions, you can skip to the \"install pyenv\" section, or to \"using pyenv\" to see notes on using the tool.
For more detailed (and likely more up-to-date) documentation on pyenv installation & usage, check the pyenv gihub's README.
With pyenv, you can separate your user Python installation from the system Python. This is generally a good practice, specifically on Linux machines; the version of Python included in some distributions is very old, and you are not meant to install packages with pip using the version of Python that came installed on your machine (especially without at least using a .venv).
The system Python is there for system packages you install that have some form of Python scripting included. Packages built for specific Linux distributions, like Debian, Ubuntu, Fedora, OpenSuSE, etc, are able to target the system version of Python, ensuring a stable and predictable installation. When you add pip packages to your system's Python environment, it not only adds complexity for other packages on your system to work around, you also increase your chance of landing yourself in dependency hell, where 2 pip packages require the same package but different versions of that package.
For more reading on why it's a good idea to install a separate version of Python, and leave the version that came installed with your machine untouched, check this RealPython article.
You also have the option of installing Python yourself, either compiling it from source or downloading, building, and installing a package from python.org. This option is perfectly fine, but can be difficult for beginners, and involves more steps and room for error when trying to automate.
"},{"location":"programming/python/pyenv.html#pyenv-scopes","title":"pyenv scopes","text":"\u2934\ufe0f back to TL;DR
pyenv has 3 \"scopes\":
shellpyenv shell x.xx.xx sets the Python interpreter for the current shellexec $SHELL or source ~/.bashrc), this value is also resetlocalpyenv local x.xx.xx creates a file called .python-version at the current pathpyenv uses this file to automatically set the version of Python while in this directoryglobalpyenv global x.xx.xx sets the default, global Python versionpython -m pip install ...) will use this global version, if no local or shell version is specifiedlocal and shell override this valuepyenv, in the event you have not set a pyenv local or pyenv shell version of Python.The global scope is essentially the \"default\" scope. When no other version is specified, the global Python version will be used.
pyenv shell overrides > pyenv local overrides > pyenv global
Warning
Make sure to pay attention to your current pyenv version. If you are getting unexpected results when running Python scripts, check the version with python3 --version. This will help if you are expecting 3.11.4 to be the version of Python for your session, for example, but have set pyenv shell 3.12.1. Because shell overrides local, the Python version in .python-version will be ignored until the session is exited.
Installing pyenv varies between OSes. On Windows, you use the pyenv-win package, for example. Below are installation instructions for Windows and Linux.
\u2934\ufe0f back to TL;DR
sudo apt-get install -y \\\n git \\\n gcc \\\n make \\\n openssl \\\n libssl-dev \\\n libbz2-dev \\\n libreadline-dev \\\n libsqlite3-dev \\\n zlib1g-dev \\\n libncursesw5-dev \\\n libgdbm-dev \\\n libc6-dev \\\n zlib1g-dev \\\n libsqlite3-dev \\\n tk-dev \\\n libssl-dev \\\n openssl \\\n libffi-dev\nsudo dnf install -y \\\n git \\\n gcc \\\n make \\\n openssl \\\n openssl-devel \\\n bzip2-devel \\\n zlib-devel \\\n readline-devel \\\n soci-sqlite3-devel \\\n ncurses-devel \\\n gdbm \\\n glibc-devel \\\n tk-devel \\\n libffi-devel\npyenv with the convenience scriptcurl https://pyenv.run | bash\npyenv variables to your ~/.bashrc...\n\n## Pyenv\nexport PATH=\"$HOME/.pyenv/bin:$PATH\"\neval \"$(pyenv init -)\"\neval \"$(pyenv virtualenv-init -)\"\n\u2934\ufe0f back to TL;DR
pyenv-win with PowershellInvoke-WebRequest -UseBasicParsing -Uri \"https://raw.githubusercontent.com/pyenv-win/pyenv-win/master/pyenv-win/install-pyenv-win.ps1\" -OutFile \"./install-pyenv-win.ps1\"; &\"./install-pyenv-win.ps1\"\nPATH%USERPROFILE%\\\\.pyenv\\\\pyenv-win\\\\bin\n%USERPROFILE%\\\\.pyenv\\\\pyenv-win\\\\shims\n[CmdletBinding()]\nparam (\n [Parameter(Mandatory = $true)]\n [string] $PythonVersion\n)\n\n$ErrorActionPreference = \"Stop\"\n$ProgressPreference = \"SilentlyContinue\"\nSet-StrictMode -Version Latest\n\nfunction _runCommand {\n [CmdletBinding()]\n param (\n [Parameter(Mandatory = $true, Position = 0)]\n [string] $Command,\n [switch] $PassThru\n )\n\n try {\n if ( $PassThru ) {\n $res = Invoke-Expression $Command\n }\n else {\n Invoke-Expression $Command\n }\n\n if ( $LASTEXITCODE -ne 0 ) {\n $errorMessage = \"'$Command' reported a non-zero status code [$LASTEXITCODE].\"\n if ($PassThru) {\n $errorMessage += \"`nOutput:`n$res\"\n }\n throw $errorMessage\n }\n\n if ( $PassThru ) {\n return $res\n }\n }\n catch {\n $PSCmdlet.WriteError( $_ )\n }\n}\n\nfunction _addToUserPath {\n [CmdletBinding()]\n param (\n [Parameter(Mandatory = $true, Position = 0)]\n [string] $AppName,\n [Parameter(Mandatory = $true, Position = 1)]\n [string[]] $PathsToAdd\n )\n\n $pathEntries = [System.Environment]::GetEnvironmentVariable(\"PATH\", [System.EnvironmentVariableTarget]::User) -split \";\"\n\n $pathUpdated = $false\n foreach ( $pathToAdd in $PathsToAdd ) {\n if ( $pathToAdd -NotIn $pathEntries ) {\n $pathEntries += $pathToAdd\n $pathUpdated = $true\n }\n }\n if ( $pathUpdated ) {\n Write-Host \"$($AppName): Updating %PATH%...\" -f Green\n # Remove any duplicate or blank entries\n $cleanPaths = $pathEntries | Select-Object -Unique | Where-Object { -Not [string]::IsNullOrEmpty($_) }\n\n # Update the user-scoped PATH environment variable\n [System.Environment]::SetEnvironmentVariable(\"PATH\", ($cleanPaths -join \";\").TrimEnd(\";\"), [System.EnvironmentVariableTarget]::User)\n\n # Reload PATH in the current session, so we don't need to restart the console\n $env:PATH = [System.Environment]::GetEnvironmentVariable(\"PATH\", [System.EnvironmentVariableTarget]::User)\n }\n else {\n Write-Host \"$($AppName): PATH already setup.\" -f Cyan\n }\n}\n\n# Install pyenv\nif ( -Not ( Test-Path $HOME/.pyenv ) ) {\n if ( $IsWindows ) {\n Write-Host \"pyenv: Installing for Windows...\" -f Green\n & git clone https://github.com/pyenv-win/pyenv-win.git $HOME/.pyenv\n if ($LASTEXITCODE -ne 0) {\n Write-Error \"git reported a non-zero status code [$LASTEXITCODE] - check previous output.\"\n }\n }\n else {\n Write-Error \"This script currently only supports Windows.\"\n }\n}\nelse {\n Write-Host \"pyenv: Already installed.\" -f Cyan\n}\n\n# Add pyenv to PATH\n_addToUserPath \"pyenv\" @(\n \"$HOME\\.pyenv\\pyenv-win\\bin\"\n \"$HOME\\.pyenv\\pyenv-win\\shims\"\n)\n\n# Install default pyenv python version\n$pyenvVersions = _runCommand \"pyenv versions\" -PassThru | Select-String $PythonVersion\nif ( -Not ( $pyenvVersions ) ) {\n Write-Host \"pyenv: Installing python version $PythonVersion...\" -f Green\n _runCommand \"pyenv install $PythonVersion\"\n}\nelse {\n Write-Host \"pyenv: Python version $PythonVersion already installed.\" -f Cyan\n}\n\n# Set pyenv global version\n$globalPythonVersion = _runCommand \"pyenv global\" -PassThru\nif ( $globalPythonVersion -ne $PythonVersion ) {\n Write-Host \"pyenv: Setting global python version: $PythonVersion\" -f Green\n _runCommand \"pyenv global $PythonVersion\"\n}\nelse {\n Write-Host \"pyenv: Global python version already set: $globalPythonVersion\" -f Cyan\n}\n\n# Update pip\n_runCommand \"python -m pip install --upgrade pip\"\n\n# Install pipx, pdm, black, cookiecutter\n_runCommand \"pip install pipx\"\n_runCommand \"pip install pdm\"\n_runCommand \"pip install black\"\n_runCommand \"pip install cookiecutter\"\n\u2934\ufe0f back to TL;DR
The sections below will detail how to install a version (or multiple versions!) of Python using pyenv, switching between them, and configuring your shell to make multiple versions of Python available to tools like nox and pytest.
Note
To see a list of the commands you can run, execute $ pyenv without any commands.
Updating pyenv is as simple as typing $ pyenv update in your terminal. The pyenv update command will update pyenv itself, as well as the listing of available Python distributions.
You can ask pyenv to show you a list of all the versions of Python available for you to install with the tool by running: $ pyenv install -l (or $ pyenv install --list). You will see a long list of version numbers, which you can install with pyenv.
Note
Some releases will indicate a specific CPU architecture, like -win32, -arm64, etc. Make sure you're installing the correct version for your CPU type!
To be safe, you can simply use a Python version string, omitting any CPU specification, like 3.12.1 instead of 3.12.1-arm.
Once you have decided on a version of Python to install (we will use 3.12.1, a recent release as of 2/14/2024), install it with: $ pyenv install 3.12.1 (or whatever version you want to install).
You can see which versions of Python are available to pyenv by running $ pyenv versions. The list will grow as you install more versions of Python with $ pyenv install x.x.x.
Note
To learn more about the global, local, and shell scopes, check the pyenv scopes section.
## Set global/default Python version to 3.12.1\n$ pyenv global 3.12.1\n\n## Make multiple versions available to tools like pytest, nox, etc\n$ pyenv global 3.11.8 3.11.6 3.12.1\n\n## Create a file `.python-version` in the local directory.\n# Python commands called from this directory will use\n# the version(s) specified this way\n$ pyenv local 3.12.1\n\n## Multiple versions in `.python-version`\n$ pyenv local 3.11.8 3.12.1\n\n## Set python version(s) for current shell session.\n# This value is cleared on logout, exit, or shutdown/restart\n$ pyenv shell 3.12.1\n\n# Set newer version first to prioritize it, make older version available\n$ pyenv shell 3.12.1 3.11.8\nTo uninstall a version of Python that you installed with pyenv, use $ pyenv uninstall x.x.x. For example, to uninstall version 3.12.1: pyenv uninstall 3.12.1.
The rich package helps make console/terminal output look nicer. It has colorization, animations, and more.
rich Github repositoryrich docsrich pypiThis example uses the rich.console.Console and rich.spinner.Spinner classes. The first context manager, get_console(), yields a rich.Console() object, which can be used with the rich.Spinner() class to display a spinner on the command line.
from contextlib import contextmanager\nimport typing as t\nfrom rich.console import Console\n\n@contextmanager\ndef get_console() -> t.Generator[Console, t.Any, None]:\n \"\"\"Yield a `rich.Console`.\n\n Usage:\n `with get_console() as console:`\n\n \"\"\"\n try:\n console: Console = Console()\n\n yield console\n\n except Exception as exc:\n msg = Exception(f\"Unhandled exception getting rich Console. Details: {exc}\")\n log.error(msg)\n\n raise exc\nThe simple_spinner() context manager yields a rich.Spinner() instance. Wrap a function in with simple_spinner(msg=\"Some message\") as spinner: to show a spinner while the function executes.
from contextlib import contextmanager\nfrom rich.spinner import Spinner\n\n@contextmanager\ndef simple_spinner(text: str = \"Processing... \\n\", animation: str = \"dots\"):\n if not text:\n text: str = \"Processing... \\n\"\n assert isinstance(text, str), TypeError(\n f\"Expected spinner text to be a string. Got type: ({type(text)})\"\n )\n\n if not text.endswith(\"\\n\"):\n text += \" \\n\"\n\n if not animation:\n animation: str = \"dots\"\n assert isinstance(animation, str), TypeError(\n f\"Expected spinner animation to be a string. Got type: ({type(text)})\"\n )\n\n try:\n _spinner = Spinner(animation, text=text)\n except Exception as exc:\n msg = Exception(f\"Unhandled exception getting console spinner. Details: {exc}\")\n log.error(msg)\n\n raise exc\n\n ## Display spinner\n try:\n with get_console() as console:\n with console.status(text, spinner=animation):\n yield console\n except Exception as exc:\n msg = Exception(\n f\"Unhandled exception yielding spinner. Continuing without animation. Details: {exc}\"\n )\n log.error(msg)\n\n pass\nPutting both of these functions in the same file allows you to import just the simple_spinner method, which calls get_console() for you. You can also get a console using with get_console() as console:, and write custom spinner/rich logic.
from some_modules import simple_spinner\n\nwith simple_spinner(text=\"Thinking... \"):\n ## Some long-running code/function\n ...\nTodo
Toc
Jump right to a section:
virtualenv .venv Create a new virtual environment This command creates a new directory called .venv/ at the path where you ran the command. (Windows) .\\.venv\\Scripts\\activate Activate a virtual environment on Windows. Your shell should change to show (.venv), indicating you are in a virtual environment (Linux/Mac) ./venv/bin/activate Activate a virtual environment on Linux. Your shell should change to show (.venv), indicating you are in a virtual environment deactivate Exit/deactivate a virtual environment. You can also simply close your shell session, which exists the environment. This command only works once a virtual environment is activated; deactivate will give an error saying the command is not found if you do not have an active virtual environment."},{"location":"programming/python/virtualenv.html#what-is-virtualenv","title":"What is virtualenv?","text":"RealPython: Virtualenv Primer
virtualenv is a tool for creating virtual Python environments. The virtualenv tool is sometimes installed with the \"system\" Python, but you may need to install it yourself (see the warning below).
These virtual environments are stored in a directory, usually ./.venv, and can be \"activated\" when you are developing a Python project. Virtual environments can also be used as Jupyter kernels if you install the ipykernel package.
Warning
If you ever see see an error saying the virtualenv command cannot be found (or something along those lines), simply install virtualenv with:
$ pip install virtualenv\nvirtualenv helps developers avoid \"dependency hell\". Without virtualenv, when you insall a package with pip, it is installed \"globally.\" This becomes an issue when 2 different Python projects use the same dependency, but different versions of that dependency. This leads to a \"broken environment,\" where the only fix is to essentially uninstall all dependencies and start from scratch.
With virtualenv, you start each project by running $ virtualenv .venv, which will create a directory called .venv/ at the path where you ran the command (i.e. inside of a Python project directory).
Note
.venv/ is the convention, but you can name this directory whatever you want. If you run virtualenv using a different path, like virtualenv VirtualEnvironment (which would create a directory called VirtualEnvironment/ at the local path), make sure you use that name throughout this guide where you see .venv.
Once a virtual environment is created, you need to \"activate\" it. Activating a virtual environment will isolate your current shell/session from the global Python, allowing you to install dependencies specific to the current Python project without interfering with the global Python state.
Activating a virtual environment is as simple as:
$ ./.venv/Scripts/activate$ ./.venv/bin/activateAfter activating an environment, your shell will change to indicate that you are within a virtual environment. Example:
activating virtualenv## Before .venv activation, using the global Python. Depdendency will\n# be installed to the global Python environment\n$ pip install pandas\n\n## Activate the virtualenv\n$ ./.venv/Scripts/activate\n\n## The shell will change to indicate you're in a venv. The \"(.venv)\" below\n# indicates a virtual environment has been activated. The pip install command\n# installs the dependency within the virtual environment\n(.venv) $ pip install pandas\nThis method of \"dependency isolation\" ensures a clean environment for each Python project you start, and keeps the \"system\"/global Python version clean of dependency errors.
"},{"location":"programming/python/virtualenv.html#exportingimporting-requirements","title":"Exporting/importing requirements","text":"Once a virtual environment is activated, commands like pip will run much the same as they do without a virtual environment, but the outputs will be contained to the .venv directory in the project you ran the commands from.
To export pip dependencies:
## Make sure you've activated your virtual environment first\n$ .\\\\venv\\\\Scripts\\\\activate # Windows\n$ ./venv/bin/activate # Linux/Mac\n\n## Export pip requirements\n(.venv) $ pip freeze > requirements.txt\nTo import/install pip dependencies from an existing requirements.txt file:
## Make sure you've activated your virtual environment first\n$ .\\\\venv\\\\Scripts\\\\activate # Windows\n$ ./venv/bin/activate # Linux/Mac\n\n## Install dependencies from a requirements.txt file\n(.venv) $ pip install -r requirements.txt\nTodo
.venv environmentTodo
virtualenvvirtualenvpoetrypdmTodo
with open() as f:with sqlalchemy.Session() as sess:@contextmanager__enter__() and __exit__() methodsCode I copy/paste into most of my projects.
Note
There is also a companion repository for this section, which may be more up to date than the examples in this section.
I use Docker a lot. I'm frequently copying/pasting Dockerfiles from previous projects to modify for my current project. Here, I will add some of my more common Dockerfiles, with notes & code snippets for reference.
This section covers some of the Dockerfiles I frequently re-use.
","tags":["standard-project-files","docker"]},{"location":"programming/standard-project-files/Docker/python_docker.html","title":"Python Dockerfiles","text":"Building/running a Python app in a Dockerfile can be accomplished many different ways. The example(s) below are my personal approach, meant to serve as a starting point/example of Docker's capabilities.
One thing you will commonly see in my Python Dockerfiles are a set of ENV variables in the base layer. Below is a list of the ENV variables I commonly set in Dockerfiles, and what they do:
Note
In some Dockerfiles, you will see ENV variables declared with an equal sign, and others without. These are equivalent, you can declare/set these variables either way and they will produce the same result. For example, the following 2 ENV variable declarations are equivalent:
PYTHONUNBUFFERED 1
PYTHONUNBUFFERED=1
PYTHONDONTWRITEBYTECODE=1.pyc files.pyc files are essentially simplified bytecode versions of your scripts that are created when a specific .py file is executed the first time..pyc files are the instructions/ingredients written like code on the side of your cup that tell the barista what to make. They don't need to know the full recipe, just the ingredients. A .pyc file is the \"ingredients\" for a .py file, meant to speed up subsequent executions of the same script..py file as a .py file each time for reproducibility.PYTHONUNBUFFERED=1stdout and stderr messages directly instead of buffering, ensuring realtime output to the container's sdtdout/stderrPIP_NO_CACHE_DIR=offpip not to cache dependencies. This would affect reproducibility in the container, and also needlessly takes up space.pip installs as a Dockerfile layer with buildkitPIP_DISABLE_PIP_VERSION_CHECK=onpip being out of date in a container environment. Also for reproducibility.PIP_DEFAULT_TIMEOUT=100pip's timeout value to a higher number to give it more time to finish downloading/installing a dependency.Use this container for small/simple Python projects, or as a starting point for a multistage build.
Python simple DockerfileFROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\nWORKDIR /app\n\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\nCOPY ./src .\nThis Dockerfile is a multi-stage build, which means it uses \"layers.\" These layers are cached by the Docker buildkit, meaning if nothing has changed in a given layer between builds, Docker will speed up the total buildtime by using a cached layer.
For example, the build layer below installs dependencies from the requirements.txt file. If no new dependencies are added between docker build commands, this layer will be re-used, \"skipping\" the pip install command.
FROM python:3.11-slim as base\n\n## Set ENV variables to control Python/pip behavior inside container\nENV PYTHONDONTWRITEBYTECODE 1 \\\n PYTHONUNBUFFERED 1 \\\n ## Pip\n PIP_NO_CACHE_DIR=off \\\n PIP_DISABLE_PIP_VERSION_CHECK=on \\\n PIP_DEFAULT_TIMEOUT=100\n\nFROM base AS build\n\nWORKDIR /app\n\nCOPY requirements.txt requirements.txt\nRUN pip install -r requirements.txt\n\n## Use target: dev to build this step\nFROM build AS dev\n\nENV ENV_FOR_DYNACONF=dev\n## Tell Dynaconf to always load from the environment first while in the container\nENV DYNACONF_ALWAYS_LOAD_ENV_VARS=True\n\nWORKDIR /app\nCOPY ./src .\n\n############\n# Optional #\n############\n# Export ports, set an entrypoint/CMD, etc\n# Note: This is normally handled by your orchestrator (docker-compose, Azure Container App, etc)\n\n# EXPOSE 5000\n# CMD [\"python\", \"main.py\"]\n\n## Use target: prod to build this step\nFROM build AS prod\n\nENV ENV_FOR_DYNACONF=prod\n## Tell Dynaconf to always load from the environment first while in the container\nENV DYNACONF_ALWAYS_LOAD_ENV_VARS=True\n\nWORKDIR /app\nCOPY ./src .\n\n############\n# Optional #\n############\n# Export ports, set an entrypoint/CMD, etc\n# Note: This is normally handled by your orchestrator (docker-compose, Azure Container App, etc)\n\n# EXPOSE 5000\n# CMD [\"python\", \"main.py\"]\nSome pre-commit hooks I use frequently.
repos:\n\n- repo: https://gitlab.com/vojko.pribudic/pre-commit-update\n rev: v0.1.1\n hooks:\n - id: pre-commit-update\n\n- repo: https://github.com/kynan/nbstripout\n rev: 0.6.1\n hooks:\n - id: nbstripout\nThis hook will update the revisions for all installed hooks each time pre-commit runs.
- repo: https://gitlab.com/vojko.pribudic/pre-commit-update\n rev: v0.1.1\n hooks:\n - id: pre-commit-update\n args: [--dry-run --exclude black --keep isort]\nThis hook will scan for jupyter notebooks (.ipynb) and clear any cell output before committing.
- repo: https://github.com/kynan/nbstripout\n rev: 0.6.1\n hooks:\n - id: nbstripout\nCopy/paste-able code, or snippets for files like pyproject.toml and noxfile.py
# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\n# lib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n*.py,cover\n.hypothesis/\n.pytest_cache/\ncover/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\n*.log.*\nlocal_settings.py\n\n## Database\ndb.sqlite3\ndb.sqlite3-journal\n*.sqlite\n*.sqlite3\n*.db\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\n.pybuilder/\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n# For a library or package, you might want to ignore these files since the code is\n# intended to run in multiple environments; otherwise, check them in:\n# .python-version\n\n# pipenv\n# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n# However, in case of collaboration, if having platform-specific dependencies or dependencies\n# having no cross-platform support, pipenv may install dependencies that don't work, or not\n# install all needed dependencies.\n#Pipfile.lock\n\n# poetry\n# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.\n# This is especially recommended for binary packages to ensure reproducibility, and is more\n# commonly ignored for libraries.\n# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control\n#poetry.lock\n\n# pdm\n# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.\n#pdm.lock\n# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it\n# in version control.\n# https://pdm.fming.dev/#use-with-ide\n.pdm.toml\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm\n__pypackages__/\n\n# Celery stuff\ncelerybeat-schedule\ncelerybeat.pid\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n*.env\n*.*.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n## Allow Environment patterns\n!*example*\n!*example*.*\n!*.*example*\n!*.*example*.*\n!*.*.*example*\n!*.*.*example*.*\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n\n# pytype static type analyzer\n.pytype/\n\n# Cython debug symbols\ncython_debug/\n\n# PyCharm\n# JetBrains specific template is maintained in a separate JetBrains.gitignore that can\n# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore\n# and can be added to the global gitignore or merged into this file. For a more nuclear\n# option (not recommended) you can uncomment the following to ignore the entire idea folder.\n#.idea/\n\n## PDM\n.pdm-python\n\n## Pyenv\n.python-version\n\n## Dynaconf\n**/.secrets.toml\n**/*.local.toml\n\n!**/*.example.toml\n!**/.*.example.toml\n\n## Jupyter\n# Uncomment to ignore Jupyter notebooks, i.e. if\n# pre-hook is not configured and you don't want to\n# commit notebook with output.\n\n# *.ipynb\n\n## Ignore mkdocs builds\nsite/\n~/.pypirc600<pypi-test-token>/<pypi-token> with your pypi/pypi-test publish token.## ~/.pypirc\n# chmod: 600\n[distutils]\nindex-servers=\n pypi\n testpypi\n\n## Example of a local, private Python package index\n# [local]\n# repository = http://127.0.0.1:8080\n# username = test \n# password = test\n\n[testpypi]\nusername = __token__ \npassword = <pypi-test-token>\n\n[pypi]\nrepository = https://upload.pypi.org/legacy/\nusername = __token__\npassword = <pypi-token>\nI use Dynaconf frequently to manage loading my project's settings from a local file (config/settings.local.toml) during development, and environment variables when running in a container. Dynaconf allows for overriding configurations by setting environment variables.
To load configurations from the environment, you can:
envvar_prefix value of a Dynaconf() instanceLOG_LEVEL: export DYNACONF_LOG_LEVEL=...config/settings.local.toml fileconfig/settings.toml file should not be edited, nor should it contain any real valuesconfig/settings.local.toml during local developmentconfig/settings.local.tomlNote
The Database section is commented below because not all projects will start with a database. This file can still be copy/pasted to config/settings.toml/config/settings.local.toml as a base/starting point.
##\n# My standard Dynaconf settings.toml file.\n#\n# I normally put this file in a directory like src/config/settings.toml, then update my config.py, adding\n# root_path=\"config\" to the Dynaconf instance.\n##\n\n[default]\n\nenv = \"prod\"\ncontainer_env = false\nlog_level = \"INFO\"\n\n############\n# Database #\n############\n\n# db_type = \"sqlite\"\n# db_drivername = \"sqlite+pysqlite\"\n# db_username = \"\"\n# # Set in .secrets.toml\n# db_password = \"\"\n# db_host = \"\"\n# db_port = \"\"\n# db_database = \".data/app.sqlite\"\n# db_echo = false\n\n[dev]\n\nenv = \"dev\"\nlog_level = \"DEBUG\"\n\n############\n# Database #\n############\n\n# db_type = \"sqlite\"\n# db_drivername = \"sqlite+pysqlite\"\n# db_username = \"\"\n# # Set in .secrets.toml\n# db_password = \"\"\n# db_host = \"\"\n# db_port = \"\"\n# db_database = \".data/app-dev.sqlite\"\n# db_echo = true\n\n[prod]\n\n############\n# Database #\n############\n\n# db_type = \"sqlite\"\n# db_drivername = \"sqlite+pysqlite\"\n# db_username = \"\"\n# # Set in .secrets.toml\n# db_password = \"\"\n# db_host = \"\"\n# db_port = \"\"\n# db_database = \".data/app.sqlite\"\n# db_echo = false\nNote
The Database section is commented below because not all projects will start with a database. This file can still be copy/pasted to config/.secrets.toml as a base/starting point.
##\n# Any secret values, like an API key or database password, or Azure connection string.\n##\n\n[default]\n\n############\n# Database #\n############\n\n# db_password = \"\"\n\n[dev]\n\n############\n# Database #\n############\n\n# db_password = \"\"\n\n[prod]\n\n############\n# Database #\n############\n\n# db_password = \"\"\nWarning
This code is highly specific to the way I structure my apps. Make sure to understand what it's doing so you can customize it to your environment, if you're using this code as a basis for your own config.py file
Note
Notes:
The following imports/vars/classes start out commented, in case the project is not using them or they require additional setup:
valid_db_types list (used to validate DBSettings.type)DYNACONF_DB_SETTINGS Dynaconf settings objectDBSettings class definitionIf the project is using a database and SQLAlchemy as the ORM, uncomment these values and modify your config/settings.local.toml & config/.secrets.toml accordingly.
from __future__ import annotations\n\nfrom typing import Union\n\nfrom dynaconf import Dynaconf\nfrom pydantic import Field, ValidationError, field_validator\nfrom pydantic_settings import BaseSettings\n\n## Uncomment if adding a database config\n# import sqlalchemy as sa\n# import sqlalchemy.orm as so\n\nDYNACONF_SETTINGS: Dynaconf = Dynaconf(\n environments=True,\n envvar_prefix=\"DYNACONF\",\n settings_files=[\"settings.toml\", \".secrets.toml\"],\n)\n\n## Uncomment if adding a database config\n# valid_db_types: list[str] = [\"sqlite\", \"postgres\", \"mssql\"]\n\n## Uncomment to load database settings from environment\n# DYNACONF_DB_SETTINGS: Dynaconf = Dynaconf(\n# environments=True,\n# envvar_prefix=\"DB\",\n# settings_files=[\"settings.toml\", \".secrets.toml\"],\n# )\n\n\nclass AppSettings(BaseSettings):\n env: str = Field(default=DYNACONF_SETTINGS.ENV, env=\"ENV\")\n container_env: bool = Field(\n default=DYNACONF_SETTINGS.CONTAINER_ENV, env=\"CONTAINER_ENV\"\n )\n log_level: str = Field(default=DYNACONF_SETTINGS.LOG_LEVEL, env=\"LOG_LEVEL\")\n\n\n## Uncomment if you're configuring a database for the app\n# class DBSettings(BaseSettings):\n# type: str = Field(default=DYNACONF_SETTINGS.DB_TYPE, env=\"DB_TYPE\")\n# drivername: str = Field(\n# default=DYNACONF_DB_SETTINGS.DB_DRIVERNAME, env=\"DB_DRIVERNAME\"\n# )\n# user: str | None = Field(\n# default=DYNACONF_DB_SETTINGS.DB_USERNAME, env=\"DB_USERNAME\"\n# )\n# password: str | None = Field(\n# default=DYNACONF_DB_SETTINGS.DB_PASSWORD, env=\"DB_PASSWORD\", repr=False\n# )\n# host: str | None = Field(default=DYNACONF_DB_SETTINGS.DB_HOST, env=\"DB_HOST\")\n# port: Union[str, int, None] = Field(\n# default=DYNACONF_DB_SETTINGS.DB_PORT, env=\"DB_PORT\"\n# )\n# database: str = Field(default=DYNACONF_DB_SETTINGS.DB_DATABASE, env=\"DB_DATABASE\")\n# echo: bool = Field(default=DYNACONF_DB_SETTINGS.DB_ECHO, env=\"DB_ECHO\")\n\n# @field_validator(\"port\")\n# def validate_db_port(cls, v) -> int:\n# if v is None or v == \"\":\n# return None\n# elif isinstance(v, int):\n# return v\n# elif isinstance(v, str):\n# return int(v)\n# else:\n# raise ValidationError\n\n# def get_db_uri(self) -> sa.URL:\n# try:\n# _uri: sa.URL = sa.URL.create(\n# drivername=self.drivername,\n# username=self.user,\n# password=self.password,\n# host=self.host,\n# port=self.port,\n# database=self.database,\n# )\n\n# return _uri\n\n# except Exception as exc:\n# msg = Exception(\n# f\"Unhandled exception getting SQLAlchemy database URL. Details: {exc}\"\n# )\n# raise msg\n\n# def get_engine(self) -> sa.Engine:\n# assert self.get_db_uri() is not None, ValueError(\"db_uri is not None\")\n# assert isinstance(self.get_db_uri(), sa.URL), TypeError(\n# f\"db_uri must be of type sqlalchemy.URL. Got type: ({type(self.db_uri)})\"\n# )\n\n# try:\n# engine: sa.Engine = sa.create_engine(\n# url=self.get_db_uri().render_as_string(hide_password=False),\n# echo=self.echo,\n# )\n\n# return engine\n# except Exception as exc:\n# msg = Exception(\n# f\"Unhandled exception getting database engine. Details: {exc}\"\n# )\n\n# raise msg\n\n# def get_session_pool(self) -> so.sessionmaker[so.Session]:\n# engine: sa.Engine = self.get_engine()\n# assert engine is not None, ValueError(\"engine cannot be None\")\n# assert isinstance(engine, sa.Engine), TypeError(\n# f\"engine must be of type sqlalchemy.Engine. Got type: ({type(engine)})\"\n# )\n\n# session_pool: so.sessionmaker[so.Session] = so.sessionmaker(bind=engine)\n\n# return session_pool\n\n\nsettings: AppSettings = AppSettings()\n## Uncomment if you're configuring a database for the app\n# db_settings: DBSettings = DBSettings()\nalembic with pip install alembic (or some equivalent package install command)alembicsrc directory, i.e. src/app, initialize alembic at src/app/alembicalembic init src/app/alembicIf using a \"flat\" repository, simply run alembic init alembic
Note
You can use any name for the directory instead of \"alembic.\" For instance, another common convention is to initialize alembic with alembic init migrations
Whatever directory name you choose, use that throughout these instructions where you see references to the alembic init path
Edit the alembic.ini file
script_location to the path you set for alembic, i.e. src/app/alembicprepend_sys_path, set to src/appalembic can do things like importing your app's config, loading the SQLAlchemy Base object, etc## alembic.ini\n\n[alembic]\n\nscript_location = src/app/alembic\n\n...\n\nprepend_system_path = src/app\n\n...\nalembic env.py file, which should be in src/app/alembic (or whatever path you initialized alembic in)sqlalchemy.URL), you can import it in env.py, or you can create a new one:## src/app/alembic.py\n\n...\n\nimport sqlalchemy as sa\n\n## Using a SQLite example\nDB_URI: sa.URL = sa.URL.create(\n drivername=\"sqlite+pysqlite\",\n username=None,\n password=None,\n host=None,\n port=None,\n database=\"database.sqlite\"\n)\n\n...\n\n## Set config's sqlalchemy.url value, after \"if config.config_filename is not None:\"\nconfig.set_main_option(\n \"sqlalchemy.url\",\n ## Use .render_as_string(hide_password=False) with sqlalchemy.URL objects,\n # otherwise database password will be \"**\"\n DB_URI.render_as_string(hide_password=False)\n)\n\n...\n\n# add your model's MetaData object here\n# for 'autogenerate' support\n# from myapp import mymodel\n# target_metadata = mymodel.Base.metadata\nfrom app.module.models import SomeModel # Import project's SQLAlchemy table classes\nfrom app.module.database import Base # Import project's SQLAlchemy Base object\n\ntarget_metadata = Base().metadata # Tell Alembic to use the project's Base() object\nalembic revision --autogenerate -m \"initial migration\"alembic revision will create an empty revision, with no changes.alembic upgrade +1+1 is how many migration levels to apply at once. If you have multiple migrations that have not been committed, you can use +2, +3, etc)alembic upgrade headalembic downgrade -1-1 is how many migration levels to revert, can also be -2, -3, etc)Some changes, like renaming a column, are not possible for Alembic to accurately track. In these cases, you will need to create an Alembic migration, then edit the new file in alembic/versions/{revision-hash}.py.
In the def upgrade() section, comment the innacurate op.add_column()/op.drop_column(), then add something like this (example uses the User class, with a renamed column .username -> .user_name):
# alembic/versions/{revision-hash}.py\n\n...\n\ndef upgrade() -> None:\n ...\n\n ## Comment the inaccurate changes\n # op.add_column(\"users\", sa.Column(\"user_name\", sa.VARCHAR(length=255), nullable=True))\n # op.drop_column(\"users\", \"username)\n\n ## Manually add a change of column type that wasn't detected by alembic\n op.alter_column(\"products\", \"description\", type_=sa.VARCHAR(length=3000))\n\n ## Manually describe column rename\n op.alter_column(\"users\", \"username\", new_column_name=\"user_name\")\nAlso edit the def downgrade() function to describe the changes that should be reverted when using alembic downgrade:
# alembic/versions/{revision-hash}.py\n\n...\n\ndef downgrade() -> None:\n ## Comment the inaccurate changes\n # op.add_column(\"users\", sa.Column(\"user_name\", sa.VARCHAR(length=255), nullable=True))\n # op.drop_column(\"users\", \"username)\n\n ## Manually describe changes to reverse if downgrading\n op.alter_column(\"users\", \"user_name\", new_column_name=\"username\")\n op.drop_column(\"products\", \"price\")\nAfter describing manual changes in an Alembic version file, you need to run alembic upgrade head to push the changes from the revision to the database.
After initializing alembic (i.e. alembic init alembic), a file alembic/env.py will be created. This file can be edited to include your project's models and SQLAlchemy Base.
Below are snippets of custom code I add to my alembic env.py file.
Provide database connection URL to alembic.
Todo
dynaconf to load database connection values from environment.env.pyDB_URI: sa.URL = sa.URL.create(\n drivername=\"sqlite+pysqlite\",\n username=None,\n password=None,\n host=None,\n port=None,\n database=\"database.sqlite\"\n)\nInstead of hardcording the database connection string in alembic.ini, load it from DB_URI.
Todo
env.py## Set config's sqlalchemy.url value, after \"if config.config_filename is not None:\"\nconfig.set_main_option(\n \"sqlalchemy.url\",\n ## Use .render_as_string(hide_password=False) with sqlalchemy.URL objects,\n # otherwise database password will be \"**\"\n DB_URI.render_as_string(hide_password=False)\n)\nImporting your project's SQLAlchemy Base and table classes that inherit from the Base object into alembic allows for automatic metadata creation.
Note
When using alembic to create the Base metadata, you do not need to run Base.metadata.create(bind=engine) in your code. Alembic handles the metadata creation for you. Just make sure to run alembic when first creating the database, or when cloning if using a local database like SQLite.
# add your model's MetaData object here\n# for 'autogenerate' support\n# from myapp import mymodel\n# target_metadata = mymodel.Base.metadata\n\n# Import project's SQLAlchemy table classes\nfrom app.module.models import SomeModel\n# Import project's SQLAlchemy Base object\nfrom app.core.database import Base\n\n# Tell Alembic to use the project's Base() object\ntarget_metadata = Base().metadata\nI am still deciding between tox and nox as my preferred task runner, but I've been leaning more towards nox for the simple reason that it's nice to be able to write Python code for things like try/except and creating directories that don't exist yet.
The basis for most/all of my projects' noxfile.py.
Note
The following are commented because they require additional setup, or may not be used in every project:
INIT_COPY_FILESsrc and a dest to copy it toconfig/.secrets.example.toml -> config/.secrets.tomlinit-setup Nox sessionINIT_COPY_FILES variableNote
If running all sessions with $ nox, only the sessions defined in nox.sessions will be executed. The list of sessions is conservative to start in order to maintain as generic a nox environment as possible.
Enabled sessions:
from __future__ import annotations\n\nfrom pathlib import Path\nimport platform\nimport shutil\n\nimport nox\n\nnox.options.default_venv_backend = \"venv\"\nnox.options.reuse_existing_virtualenvs = True\nnox.options.error_on_external_run = False\nnox.options.error_on_missing_interpreters = False\n# nox.options.report = True\n\n## Define sessions to run when no session is specified\nnox.sessions = [\"lint\", \"export\", \"tests\"]\n\n# INIT_COPY_FILES: list[dict[str, str]] = [\n# {\"src\": \"config/.secrets.example.toml\", \"dest\": \"config/.secrets.toml\"},\n# {\"src\": \"config/settings.toml\", \"dest\": \"config/settings.local.toml\"},\n# ]\n## Define versions to test\nPY_VERSIONS: list[str] = [\"3.12\", \"3.11\"]\n## Set PDM version to install throughout\nPDM_VER: str = \"2.11.2\"\n## Set paths to lint with the lint session\nLINT_PATHS: list[str] = [\"src\", \"tests\", \"./noxfile.py\"]\n\n## Get tuple of Python ver ('maj', 'min', 'mic')\nPY_VER_TUPLE = platform.python_version_tuple()\n## Dynamically set Python version\nDEFAULT_PYTHON: str = f\"{PY_VER_TUPLE[0]}.{PY_VER_TUPLE[1]}\"\n\n## Set directory for requirements.txt file output\nREQUIREMENTS_OUTPUT_DIR: Path = Path(\"./requirements\")\n## Ensure REQUIREMENTS_OUTPUT_DIR path exists\nif not REQUIREMENTS_OUTPUT_DIR.exists():\n try:\n REQUIREMENTS_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n except Exception as exc:\n msg = Exception(\n f\"Unable to create requirements export directory: '{REQUIREMENTS_OUTPUT_DIR}'. Details: {exc}\"\n )\n print(msg)\n\n REQUIREMENTS_OUTPUT_DIR: Path = Path(\".\")\n\n\n@nox.session(python=PY_VERSIONS, name=\"build-env\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef setup_base_testenv(session: nox.Session, pdm_ver: str):\n print(f\"Default Python: {DEFAULT_PYTHON}\")\n session.install(f\"pdm>={pdm_ver}\")\n\n print(\"Installing dependencies with PDM\")\n session.run(\"pdm\", \"sync\")\n session.run(\"pdm\", \"install\")\n\n\n@nox.session(python=[DEFAULT_PYTHON], name=\"lint\")\ndef run_linter(session: nox.Session):\n session.install(\"ruff\", \"black\")\n\n for d in LINT_PATHS:\n if not Path(d).exists():\n print(f\"Skipping lint path '{d}', could not find path\")\n pass\n else:\n lint_path: Path = Path(d)\n print(f\"Running ruff imports sort on '{d}'\")\n session.run(\n \"ruff\",\n \"--select\",\n \"I\",\n \"--fix\",\n lint_path,\n )\n\n print(f\"Formatting '{d}' with Black\")\n session.run(\n \"black\",\n lint_path,\n )\n\n print(f\"Running ruff checks on '{d}' with --fix\")\n session.run(\n \"ruff\",\n \"check\",\n \"--config\",\n \"ruff.ci.toml\",\n lint_path,\n \"--fix\",\n )\n\n\n@nox.session(python=[DEFAULT_PYTHON], name=\"export\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef export_requirements(session: nox.Session, pdm_ver: str):\n session.install(f\"pdm>={pdm_ver}\")\n\n print(\"Exporting production requirements\")\n session.run(\n \"pdm\",\n \"export\",\n \"--prod\",\n \"-o\",\n f\"{REQUIREMENTS_OUTPUT_DIR}/requirements.txt\",\n \"--without-hashes\",\n )\n\n print(\"Exporting development requirements\")\n session.run(\n \"pdm\",\n \"export\",\n \"-d\",\n \"-o\",\n f\"{REQUIREMENTS_OUTPUT_DIR}/requirements.dev.txt\",\n \"--without-hashes\",\n )\n\n # print(\"Exporting CI requirements\")\n # session.run(\n # \"pdm\",\n # \"export\",\n # \"--group\",\n # \"ci\",\n # \"-o\",\n # f\"{REQUIREMENTS_OUTPUT_DIR}/requirements.ci.txt\",\n # \"--without-hashes\",\n # )\n\n\n@nox.session(python=PY_VERSIONS, name=\"tests\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef run_tests(session: nox.Session, pdm_ver: str):\n session.install(f\"pdm>={pdm_ver}\")\n session.run(\"pdm\", \"install\")\n\n print(\"Running Pytest tests\")\n session.run(\n \"pdm\",\n \"run\",\n \"pytest\",\n \"-n\",\n \"auto\",\n \"--tb=auto\",\n \"-v\",\n \"-rsXxfP\",\n )\n\n\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-all\")\ndef run_pre_commit_all(session: nox.Session):\n session.install(\"pre-commit\")\n session.run(\"pre-commit\")\n\n print(\"Running all pre-commit hooks\")\n session.run(\"pre-commit\", \"run\")\n\n\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-update\")\ndef run_pre_commit_autoupdate(session: nox.Session):\n session.install(f\"pre-commit\")\n\n print(\"Running pre-commit autoupdate\")\n session.run(\"pre-commit\", \"autoupdate\")\n\n\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-nbstripout\")\ndef run_pre_commit_nbstripout(session: nox.Session):\n session.install(f\"pre-commit\")\n\n print(\"Running nbstripout pre-commit hook\")\n session.run(\"pre-commit\", \"run\", \"nbstripout\")\n\n\n# @nox.session(python=[PY_VER_TUPLE], name=\"init-setup\")\n# def run_initial_setup(session: nox.Session):\n# if INIT_COPY_FILES is None:\n# print(f\"INIT_COPY_FILES is empty. Skipping\")\n# pass\n\n# else:\n\n# for pair_dict in INIT_COPY_FILES:\n# src = Path(pair_dict[\"src\"])\n# dest = Path(pair_dict[\"dest\"])\n# if not dest.exists():\n# print(f\"Copying {src} to {dest}\")\n# try:\n# shutil.copy(src, dest)\n# except Exception as exc:\n# msg = Exception(\n# f\"Unhandled exception copying file from '{src}' to '{dest}'. Details: {exc}\"\n# )\n# print(f\"[ERROR] {msg}\")\nCode snippets for nox sessions
## Run all pre-commit hooks\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-all\")\ndef run_pre_commit_all(session: nox.Session):\n session.install(\"pre-commit\")\n session.run(\"pre-commit\")\n\n print(\"Running all pre-commit hooks\")\n session.run(\"pre-commit\", \"run\")\n## Automatically update pre-commit hooks on new revisions\n@nox.session(python=PY_VERSIONS, name=\"pre-commit-update\")\ndef run_pre_commit_autoupdate(session: nox.Session):\n session.install(f\"pre-commit\")\n\n print(\"Running pre-commit update hook\")\n session.run(\"pre-commit\", \"run\", \"pre-commit-update\")\npytest-xdist runs tests concurrently, significantly improving test execution speed.
## Run pytest with xdist, allowing concurrent tests\n@nox.session(python=PY_VERSIONS, name=\"tests\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef run_tests(session: nox.Session, pdm_ver: str):\n session.install(f\"pdm>={pdm_ver}\")\n session.run(\"pdm\", \"install\")\n\n print(\"Running Pytest tests\")\n session.run(\n \"pdm\",\n \"run\",\n \"pytest\",\n \"-n\",\n \"auto\",\n \"--tb=auto\",\n \"-v\",\n \"-rsXxfP\",\n )\n## Run pytest\n@nox.session(python=PY_VERSIONS, name=\"tests\")\n@nox.parametrize(\"pdm_ver\", [PDM_VER])\ndef run_tests(session: nox.Session, pdm_ver: str):\n session.install(f\"pdm>={pdm_ver}\")\n session.run(\"pdm\", \"install\")\n\n print(\"Running Pytest tests\")\n session.run(\n \"pdm\",\n \"run\",\n \"pytest\",\n \"--tb=auto\",\n \"-v\",\n \"-rsXxfP\",\n )\nI use pdm to manage most of my Python projects. It's a fantastic tool for managing environments, dependencies, builds, and package publishing. PDM is similar in functionality to poetry, which I have also used and liked. My main reasons for preferring pdm over poetry are:
[tool.pdm.scripts] section in my pyproject.toml, and being able to script long/repeated things like alembic commands or project execution scripts is so handy.poetry (reference)Most of my notes and code snippets will assume pdm is the dependency manager for a given project.
In your pyproject.toml file, add a section [tool.pdm.scripts], then copy/paste whichever scripts you want to add to your project.
Warning
Check the code for some of the scripts, like the start script, which assumes your project code is at ./src/app/main.py. If your code is in a different path, or named something other than app, make sure to change this and any other similar lines.
[tool.pdm.scripts]\n\n###############\n# Format/Lint #\n###############\n\n# Lint with black & ruff\nlint = { shell = \"pdm run ruff check . --fix && pdm run black .\" }\n## With nox\n# lint = { cmd = \"nox -s lint\"}\n# Check only, don't fix\ncheck = { cmd = \"black .\" }\n# Check and fix\nformat = { cmd = \"ruff check . --fix\" }\n\n########################\n# Start/Launch Scripts #\n########################\n\n# Run main app or script. Launches from app/\nstart = { shell = \"cd app && pdm run python main.py\" }\n\n## Example Dynaconf start\nstart-dev = { cmd = \"python src/app/main.py\", env = { ENV_FOR_DYNACONF = \"dev\" } }\n\n######################\n# Export Requirement #\n######################\n\n# Export production requirements\nexport = { cmd = \"pdm export --prod -o requirements/requirements.txt --without-hashes\" }\n# Export only development requirements\nexport-dev = { cmd = \"pdm export -d -o requirements/requirements.dev.txt --without-hashes\" }\n## Uncomment if/when using a CI group\n# export-ci = { cmd = \"pdm export -G ci -o requirements/requirements.ci.txt --without-hashes\" }\n## Uncomment if using mkdocs or sphinx\n# export-docs = { cmd = \"pdm export -G docs --no-default -o docs/requirements.txt --without-hashes\" }\n\n###########\n# Alembic #\n###########\n\n## Create initial commit\nalembic-init = { cmd = \"alembic revision -m 'Initial commit.'\" }\n\n## Upgrade Alembic head after making model changes\nalembic-upgrade = { cmd = \"alembic upgrade head\" }\n\n## Run migrations\n# Prompts for a commit message\nalembic-migrate = { shell = \"read -p 'Commit message: ' commit_msg && pdm run alembic revision --autogenerate -m '${commit_msg}'\" }\n\n## Run full migration, upgrade - commit - revision\nmigrations = { shell = \"pdm run alembic upgrade head && read -p 'Commit message: ' commit_msg && pdm run alembic revision --autogenerate -m '${commit_msg}'\" }\nTodo
pytest for my projectsPut the pytest.ini file in the root of the project to configure pytest executions
[pytest]\n# Filter unregistered marks. Suppresses all UserWarning\n# messages, and converts all other errors/warnings to errors.\nfilterwarnings =\n error\n ignore::UserWarning\ntestpaths = tests\nNote
These tests don't really do anything, but they are the basis for writing pytest tests.
from __future__ import annotations\n\nfrom pytest import mark, xfail\n\n@mark.hello\ndef test_say_hello(dummy_hello_str: str):\n assert isinstance(\n dummy_hello_str, str\n ), f\"Invalid test output type: ({type(dummy_hello_str)}). Should be of type str\"\n assert (\n dummy_hello_str == \"world\"\n ), f\"String should have been 'world', not '{dummy_hello_str}'\"\n\n print(f\"Hello, {dummy_hello_str}!\")\n\n\n@mark.always_pass\ndef test_pass():\n assert True, \"test_pass() should have been True\"\n\n\n@mark.xfail\ndef test_fail():\n test_pass = False\n assert test_pass, \"This test is designed to fail\"\nPut conftest.py inside your tests/ directory. This file configures pytest, like providing test fixture paths so they can be accessed by tests.
import pytest\n\n## Add fixtures as plugins\npytest_plugins = [\n \"tests.fixtures.dummy_fixtures\"\n]\nSome templates/example of pytest fixtures
from pytest import fixture\n\n\n@fixture\ndef dummy_hello_str() -> str:\n \"\"\"A dummy str fixture for pytests.\"\"\"\n return \"hello, world\"\nRuff...is...awesome! It's hard to summarize, Ruff describes itself as \"An extremely fast Python linter and code formatter, written in Rust.\" It has replaced isort, flake8, and is on its way to replacing black in my environments.
I use 2 ruff.toml files normally:
ruff.tomlruff command and VSCoderuff.ci.tomlnoxfile.py, I have a lint section.ruff.toml to avoid pipeline errors for things that don't matter like docstring warnings and other rules I find overly strict.## Set assumed Python version\ntarget-version = \"py311\"\n\n## Same as Black.\nline-length = 88\n\n## Enable pycodestyle (\"E\") and Pyflakes (\"F\") codes by default\n# # Docs: https://beta.ruff.rs/docs/rules/\nlint.select = [\n \"D\", ## pydocstyle\n \"E\", ## pycodestyle\n \"F401\", ## remove unused imports\n \"I\", ## isort\n \"I001\", ## Unused imports\n]\n\n## Ignore specific checks.\n# Comment lines in list below to \"un-ignore.\"\n# This is counterintuitive, but commenting a\n# line in ignore list will stop Ruff from\n# ignoring that check. When the line is\n# uncommented, the check will be run.\nlint.ignore = [\n \"D100\", ## missing-docstring-in-public-module\n \"D101\", ## missing-docstring-in-public-class\n \"D102\", ## missing-docstring-in-public-method\n \"D103\", ## Missing docstring in public function\n \"D106\", ## missing-docstring-in-public-nested-class\n \"D203\", ## one-blank-line-before-class\n \"D213\", ## multi-line-summary-second-line\n \"D406\", ## Section name should end with a newline\n \"D407\", ## Missing dashed underline after section\n \"E501\", ## Line too long\n \"E402\", ## Module level import not at top of file\n \"F401\", ## imported but unused\n]\n\n## Allow autofix for all enabled rules (when \"--fix\") is provided.\n# NOTE: Leaving these commented until I know what they do\n# Docs: https://beta.ruff.rs/docs/rules/\nlint.fixable = [\n # \"A\", ## flake8-builtins\n # \"B\", ## flake8-bugbear\n \"C\",\n \"D\", ## pydocstyle\n \"E\", ## pycodestyle-error\n \"E402\", ## Module level import not at top of file\n \"E501\", ## Line too long\n # \"F\", ## pyflakes\n \"F401\", ## unused imports\n # \"G\", ## flake8-logging-format\n \"I\", ## isort\n \"N\", ## pep8-naming\n # \"Q\", ## flake8-quotas\n # \"S\", ## flake8-bandit\n \"T\",\n \"W\", ## pycodestyle-warning\n # \"ANN\", ## flake8-annotations\n # \"ARG\", ## flake8-unused-arguments\n # \"BLE\", ## flake8-blind-except\n # \"COM\", ## flake8-commas\n # \"DJ\", ## flake8-django\n # \"DTZ\", ## flake8-datetimez\n # \"EM\", ## flake8-errmsg\n \"ERA\", ## eradicate\n # \"EXE\", ## flake8-executables\n # \"FBT\", ## flake8-boolean-trap\n # \"ICN\", ## flake8-imort-conventions\n # \"INP\", ## flake8-no-pep420\n # \"ISC\", ## flake8-implicit-str-concat\n # \"NPY\", ## NumPy-specific rules\n # \"PD\", ## pandas-vet\n # \"PGH\", ## pygrep-hooks\n # \"PIE\", ## flake8-pie\n \"PL\", ## pylint\n # \"PT\", ## flake8-pytest-style\n # \"PTH\", ## flake8-use-pathlib\n # \"PYI\", ## flake8-pyi\n # \"RET\", ## flake8-return\n # \"RSE\", ## flake8-raise\n \"RUF\", ## ruf-specific rules\n # \"SIM\", ## flake8-simplify\n # \"SLF\", ## flake8-self\n # \"TCH\", ## flake8-type-checking\n \"TID\", ## flake8-tidy-imports\n \"TRY\", ## tryceratops\n \"UP\", ## pyupgrade\n # \"YTT\" ## flake8-2020\n]\n# unfixable = []\n\n# Exclude a variety of commonly ignored directories.\nexclude = [\n \".bzr\",\n \".direnv\",\n \".eggs\",\n \".git\",\n \".ruff_cache\",\n \".venv\",\n \"__pypackages__\",\n \"__pycache__\",\n \"*.pyc\",\n]\n\n[lint.per-file-ignores]\n\"__init__.py\" = [\"D104\"]\n\n## Allow unused variables when underscore-prefixed.\n# dummy-variable-rgx = \"^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$\"\n\n[lint.mccabe]\nmax-complexity = 10\n\n[lint.isort]\ncombine-as-imports = true\nforce-sort-within-sections = true\nforce-wrap-aliases = true\n## Use a single line after each import block.\nlines-after-imports = 1\n## Use a single line between direct and from import\nlines-between-types = 1\n## Order imports by type, which is determined by case,\n# in addition to alphabetically.\norder-by-type = true\nrelative-imports-order = \"closest-to-furthest\"\n## Automatically add imports below to top of files\nrequired-imports = [\"from __future__ import annotations\"]\n## Define isort section priority\nsection-order = [\n \"future\",\n \"standard-library\",\n \"first-party\",\n \"local-folder\",\n \"third-party\",\n]\n## Set assumed Python version\ntarget-version = \"py311\"\n\n## Same as Black.\nline-length = 88\n\n## Enable pycodestyle (\"E\") and Pyflakes (\"F\") codes by default\n# # Docs: https://beta.ruff.rs/docs/rules/\nlint.select = [\n \"D\", ## pydocstyle\n \"E\", ## pycodestyle\n \"F401\", ## remove unused imports\n \"I\", ## isort\n \"I001\", ## Unused imports\n]\n\n## Ignore specific checks.\n# Comment lines in list below to \"un-ignore.\"\n# This is counterintuitive, but commenting a\n# line in ignore list will stop Ruff from\n# ignoring that check. When the line is\n# uncommented, the check will be run.\nlint.ignore = [\n \"D100\", ## missing-docstring-in-public-module\n \"D101\", ## missing-docstring-in-public-class\n \"D102\", ## missing-docstring-in-public-method\n \"D103\", ## Missing docstring in public function\n \"D105\", ## Missing docstring in magic method\n \"D106\", ## missing-docstring-in-public-nested-class\n \"D107\", ## Missing docstring in __init__\n \"D200\", ## One-line docstring should fit on one line\n \"D203\", ## one-blank-line-before-class\n \"D205\", ## 1 blank line required between summary line and description\n \"D213\", ## multi-line-summary-second-line\n \"D401\", ## First line of docstring should be in imperative mood\n \"E402\", ## Module level import not at top of file\n \"D406\", ## Section name should end with a newline\n \"D407\", ## Missing dashed underline after section\n \"D414\", ## Section has no content\n \"D417\", ## Missing argument descriptions in the docstring for [variables]\n \"E501\", ## Line too long\n \"E722\", ## Do note use bare `except`\n \"F401\", ## imported but unused\n]\n\n## Allow autofix for all enabled rules (when \"--fix\") is provided.\n# NOTE: Leaving these commented until I know what they do\n# Docs: https://beta.ruff.rs/docs/rules/\nlint.fixable = [\n # \"A\", ## flake8-builtins\n # \"B\", ## flake8-bugbear\n \"C\",\n \"D\", ## pydocstyle\n \"E\", ## pycodestyle-error\n \"E402\", ## Module level import not at top of file\n # \"F\", ## pyflakes\n \"F401\", ## unused imports\n # \"G\", ## flake8-logging-format\n \"I\", ## isort\n \"N\", ## pep8-naming\n # \"Q\", ## flake8-quotas\n # \"S\", ## flake8-bandit\n \"T\",\n \"W\", ## pycodestyle-warning\n # \"ANN\", ## flake8-annotations\n # \"ARG\", ## flake8-unused-arguments\n # \"BLE\", ## flake8-blind-except\n # \"COM\", ## flake8-commas\n # \"DJ\", ## flake8-django\n # \"DTZ\", ## flake8-datetimez\n # \"EM\", ## flake8-errmsg\n \"ERA\", ## eradicate\n # \"EXE\", ## flake8-executables\n # \"FBT\", ## flake8-boolean-trap\n # \"ICN\", ## flake8-imort-conventions\n # \"INP\", ## flake8-no-pep420\n # \"ISC\", ## flake8-implicit-str-concat\n # \"NPY\", ## NumPy-specific rules\n # \"PD\", ## pandas-vet\n # \"PGH\", ## pygrep-hooks\n # \"PIE\", ## flake8-pie\n \"PL\", ## pylint\n # \"PT\", ## flake8-pytest-style\n # \"PTH\", ## flake8-use-pathlib\n # \"PYI\", ## flake8-pyi\n # \"RET\", ## flake8-return\n # \"RSE\", ## flake8-raise\n \"RUF\", ## ruf-specific rules\n # \"SIM\", ## flake8-simplify\n # \"SLF\", ## flake8-self\n # \"TCH\", ## flake8-type-checking\n \"TID\", ## flake8-tidy-imports\n \"TRY\", ## tryceratops\n \"UP\", ## pyupgrade\n # \"YTT\" ## flake8-2020\n]\n# unfixable = []\n\n# Exclude a variety of commonly ignored directories.\nexclude = [\n \".bzr\",\n \".direnv\",\n \".eggs\",\n \".git\",\n \".ruff_cache\",\n \".venv\",\n \"__pypackages__\",\n \"__pycache__\",\n \"*.pyc\",\n]\n\n[lint.per-file-ignores]\n\"__init__.py\" = [\"D104\"]\n\n## Allow unused variables when underscore-prefixed.\n# dummy-variable-rgx = \"^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$\"\n\n[lint.mccabe]\nmax-complexity = 10\n\n[lint.isort]\ncombine-as-imports = true\nforce-sort-within-sections = true\nforce-wrap-aliases = true\n## Use a single line after each import block.\nlines-after-imports = 1\n## Use a single line between direct and from import\nlines-between-types = 1\n## Order imports by type, which is determined by case,\n# in addition to alphabetically.\norder-by-type = true\nrelative-imports-order = \"closest-to-furthest\"\n## Automatically add imports below to top of files\nrequired-imports = [\"from __future__ import annotations\"]\n## Define isort section priority\nsection-order = [\n \"future\",\n \"standard-library\",\n \"first-party\",\n \"local-folder\",\n \"third-party\",\n]\nTodo
See my full database directory (meant to be copied to either core/database or modules/database):
core/database on Github
","tags":["standard-project-files","python","sqlalchemy"]},{"location":"programming/standard-project-files/python/sqlalchemy/database_dir.html","title":"app/core/database","text":"My standard SQLAlchemy base setup. The files in the core/database directory of my projects provides a database config from a dataclass (default values create a SQLite database at the project root), a SQLAlchemy Base, and methods for getting SQLAlchemy Engine and Session.
from __future__ import annotations\n\nfrom .annotated import INT_PK, STR_10, STR_255\nfrom .base import Base\nfrom .db_config import DBSettings\nfrom .methods import get_db_uri, get_engine, get_session_pool\nfrom .mixins import TableNameMixin, TimestampMixin\nCustom annotations live in database/annotated
from __future__ import annotations\n\nfrom .annotated_columns import INT_PK, STR_10, STR_255\nfrom __future__ import annotations\n\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\nfrom typing_extensions import Annotated\n\n## Annotated auto-incrementing integer primary key column\nINT_PK = Annotated[\n int, so.mapped_column(sa.INTEGER, primary_key=True, autoincrement=True, unique=True)\n]\n\n## SQLAlchemy VARCHAR(10)\nSTR_10 = Annotated[str, so.mapped_column(sa.VARCHAR(10))]\n## SQLAlchemy VARCHAR(255)\nSTR_255 = Annotated[str, so.mapped_column(sa.VARCHAR(255))]\nMixin classes can be used with classes that inherit from the SQLAlchemy Base class to add extra functionality.
Example
Automatically add a created_at and updated_at column by inheriting from TimestampMixin
...\n\nclass ExampleModel(Base, TimestampMixin):\n \"\"\"Class will have a created_at and modified_at timestamp applied automatically.\"\"\"\n __tablename__ = \"example\"\n\n id: Mapped[int] = mapped_column(sa.INTEGER, primary_key=True, autoincrement=True, unique=True)\n\n ...\nfrom __future__ import annotations\n\nfrom .classes import TableNameMixin, TimestampMixin\nfrom __future__ import annotations\n\nimport pendulum\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\n\nclass TimestampMixin:\n \"\"\"Add a created_at & updated_at column to records.\n\n Add to class declaration to automatically create these columns on\n records.\n\n Usage:\n\n ``` py linenums=1\n class Record(Base, TimestampMixin):\n __tablename__ = ...\n\n ...\n ```\n \"\"\"\n\n created_at: so.Mapped[pendulum.DateTime] = so.mapped_column(\n sa.TIMESTAMP, server_default=sa.func.now()\n )\n updated_at: so.Mapped[pendulum.DateTime] = so.mapped_column(\n sa.TIMESTAMP, server_default=sa.func.now(), onupdate=sa.func.now()\n )\n\n\nclass TableNameMixin:\n \"\"\"Mixing to automatically name tables based on class name.\n\n Generates a `__tablename__` for classes inheriting from this mixin.\n \"\"\"\n\n @so.declared_attr.directive\n def __tablename__(cls) -> str:\n return cls.__name__.lower() + \"s\"\nfrom __future__ import annotations\n\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\n\nREGISTRY: so.registry = so.registry()\nMETADATA: sa.MetaData = sa.MetaData()\n\n\nclass Base(so.DeclarativeBase):\n registry = REGISTRY\n metadata = METADATA\nfrom __future__ import annotations\n\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\n\nfrom dataclasses import dataclass, field\n\n\n@dataclass\nclass DBSettings:\n drivername: str = field(default=\"sqlite+pysqlite\")\n user: str | None = field(default=None)\n password: str | None = field(default=None)\n host: str | None = field(default=None)\n port: str | None = field(default=None)\n database: str = field(default=\"app.sqlite\")\n echo: bool = field(default=False)\n\n def __post_init__(self):\n assert self.drivername is not None, ValueError(\"drivername cannot be None\")\n assert isinstance(self.drivername, str), TypeError(\n f\"drivername must be of type str. Got type: ({type(self.drivername)})\"\n )\n assert isinstance(self.echo, bool), TypeError(\n f\"echo must be a bool. Got type: ({type(self.echo)})\"\n )\n if self.user:\n assert isinstance(self.user, str), TypeError(\n f\"user must be of type str. Got type: ({type(self.user)})\"\n )\n if self.password:\n assert isinstance(self.password, str), TypeError(\n f\"password must be of type str. Got type: ({type(self.password)})\"\n )\n if self.host:\n assert isinstance(self.host, str), TypeError(\n f\"host must be of type str. Got type: ({type(self.host)})\"\n )\n if self.port:\n assert isinstance(self.port, int), TypeError(\n f\"port must be of type int. Got type: ({type(self.port)})\"\n )\n assert self.port > 0 and self.port <= 65535, ValueError(\n f\"port must be an integer between 1 and 65535\"\n )\n\n def get_db_uri(self) -> sa.URL:\n try:\n _uri: sa.URL = sa.URL.create(\n drivername=self.drivername,\n username=self.user,\n password=self.password,\n host=self.host,\n port=self.port,\n database=self.database,\n )\n\n return _uri\n\n except Exception as exc:\n msg = Exception(\n f\"Unhandled exception getting SQLAlchemy database URL. Details: {exc}\"\n )\n raise msg\n\n def get_engine(self) -> sa.Engine:\n assert self.get_db_uri() is not None, ValueError(\"db_uri is not None\")\n assert isinstance(self.get_db_uri(), sa.URL), TypeError(\n f\"db_uri must be of type sqlalchemy.URL. Got type: ({type(self.db_uri)})\"\n )\n\n try:\n engine: sa.Engine = sa.create_engine(\n url=self.get_db_uri().render_as_string(hide_password=False),\n echo=self.echo,\n )\n\n return engine\n except Exception as exc:\n msg = Exception(\n f\"Unhandled exception getting database engine. Details: {exc}\"\n )\n\n raise msg\n\n def get_session_pool(self) -> so.sessionmaker[so.Session]:\n engine: sa.Engine = self.get_engine()\n assert engine is not None, ValueError(\"engine cannot be None\")\n assert isinstance(engine, sa.Engine), TypeError(\n f\"engine must be of type sqlalchemy.Engine. Got type: ({type(engine)})\"\n )\n\n session_pool: so.sessionmaker[so.Session] = so.sessionmaker(bind=engine)\n\n return session_pool\nfrom __future__ import annotations\n\nimport sqlalchemy as sa\nimport sqlalchemy.orm as so\n\ndef get_db_uri(\n drivername: str = \"sqlite+pysqlite\",\n username: str | None = None,\n password: str | None = None,\n host: str | None = None,\n port: int | None = None,\n database: str = \"demo.sqlite\",\n) -> sa.URL:\n assert drivername is not None, ValueError(\"drivername cannot be None\")\n assert isinstance(drivername, str), TypeError(\n f\"drivername must be of type str. Got type: ({type(drivername)})\"\n )\n if username is not None:\n assert isinstance(username, str), TypeError(\n f\"username must be of type str. Got type: ({type(username)})\"\n )\n if password is not None:\n assert isinstance(password, str), TypeError(\n f\"password must be of type str. Got type: ({type(password)})\"\n )\n if host is not None:\n assert isinstance(host, str), TypeError(\n f\"host must be of type str. Got type: ({type(host)})\"\n )\n if port is not None:\n assert isinstance(port, int), TypeError(\n f\"port must be of type int. Got type: ({type(port)})\"\n )\n assert database is not None, ValueError(\"database cannot be None\")\n assert isinstance(database, str), TypeError(\n f\"database must be of type str. Got type: ({type(database)})\"\n )\n\n try:\n db_uri: sa.URL = sa.URL.create(\n drivername=drivername,\n username=username,\n password=password,\n host=host,\n port=port,\n database=database,\n )\n\n return db_uri\n except Exception as exc:\n msg = Exception(\n f\"Unhandled exception creating SQLAlchemy URL from inputs. Details: {exc}\"\n )\n\n raise msg\n\n\ndef get_engine(db_uri: sa.URL = None, echo: bool = False) -> sa.Engine:\n assert db_uri is not None, ValueError(\"db_uri is not None\")\n assert isinstance(db_uri, sa.URL), TypeError(\n f\"db_uri must be of type sqlalchemy.URL. Got type: ({type(db_uri)})\"\n )\n\n try:\n engine: sa.Engine = sa.create_engine(url=db_uri, echo=echo)\n\n return engine\n except Exception as exc:\n msg = Exception(f\"Unhandled exception getting database engine. Details: {exc}\")\n\n raise msg\n\n\ndef get_session_pool(engine: sa.Engine = None) -> so.sessionmaker[so.Session]:\n assert engine is not None, ValueError(\"engine cannot be None\")\n assert isinstance(engine, sa.Engine), TypeError(\n f\"engine must be of type sqlalchemy.Engine. Got type: ({type(engine)})\"\n )\n\n session_pool: so.sessionmaker[so.Session] = so.sessionmaker(bind=engine)\n\n return session_pool\nUseful software utilities I reach for often enough to document
Warning
In progress
"},{"location":"utilities/ssh/index.html","title":"Secure Shell (SSH)","text":"Warning
In progress
Todo
I had a hard time understanding what to do with my private/public keys when I was learning SSH. I don't know why it was a difficult concept for me, but I have worked with enough other people who were confused in the same way I was that I think it's worth it to just spell out what to do with each key.
Your private key (default name is id_rsa) should NEVER leave the server it was created on, and should not be accessible to any other user (chmod 600). There are exceptions to this, such as when uploading a keypair to an Azure or Hashicorp vault, or providing to a Docker container. But in general, when creating SSH tunnels between machines, the private key is meant to stay on the machine it was created on.
Your public key is like your swipe card; when using the ssh command with -f /path/to/id_rsa.pub and the correct user@server combo, you will not need to enter a password to authenticate.
Your public key can also be used for SFTP.
Todo
.ppk file with PuTTYYou can create a keypair using the ssh-keygen utility. This is installed with SSH (openssh-server on Linux, see installation instructions for Windows), and is available cross-platform.
Note
You can run $ ssh-keygen --help on any platform to see a list of available commands. --help is not a valid flag, so you will see a warning unknown option -- -, but the purpose of this is to show available commands.
If you run ssh-keygen without any arguments, you will be guided through a series of prompts, after which 2 files will be created (assuming you chose the defaults): id_rsa (your private key) and id_rsa.pub (your public key).
You can also pass some parameters to automatically answer certain prompts:
-f parameter specifies the output file. This will skip the prompt Enter file in which to save the key$ ssh-keygen -f /path/to/<ssh_key_filename>-f, a private and public (.pub) key will be created-t parameter allows you to specify a key type$ ssh-keygen -t rsadsaecdsaecdsa-sked25519ed25519-sk-b option allows you to specify the number of bits. For rsa keys, the minimum is 1024 and the default is 3072.4096 with: $ ssh-keygen -b 4096Example ssh-keygen commands
ssh-keygen commands## Generate a 4096-bit RSA key named example_key\n$ ssh-keygen -t rsa -b 4096 -f ~/.ssh/example_key\n\n## Generate another 4096b RSA key, this time skipping the password prompt\n# Note: This works better on Linux, I'm not sure what the equivalent is\n# on Windows\n$ ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N \"\"\nYou can (and should) use your SSH key to authenticate as a user on a remote system. There are 2 ways of adding keys for this type of authentication.
Note
Whatever method you use to add your public key to a remote machine, make sure you edit ~/.ssh/config (create the file if it does not exist). The ssh command can use this file to configure connections with SSH keys so you don't have to specify $ ssh -i /path/to/key each time you connect to a remote you've already copied a key to.
## You can set $Host to whatever you want.\n# You will connect with: ssh $Host\nHost example.com\n## The actual FQDN/IP address of the server\nHostName example.com\n## If the remote SSH server is running on a\n# port other than the default 22, set here\n# Port 222\n## The remote user your key is paired to\nUser test\n## The public key exists on the remote.\n# You provide the private key to complete the pair\nIdentityFile ~/.ssh/<your_ssh_key>\n\n## On Windows, set \"ForwardAgent yes\" for VSCode remote editing.\n# Uncomment the line below if on Windows\n# ForwardAgent yes\nOnce you've copied your key, you can simply run ssh $Host, where $Host is the value you set for Host in the ~/.ssh/config file. The SSH client will find the matching Host entry and use the options you specified.
## Note: If you get a message about trusting the host, hit yes.\n# You will need to type your password the first time\n$ ssh-copy-id -i ~/.ssh/your_key.pub test@example\nYou can also manually copy your public keyfile (.pub) to a remote host and cat the contents into ~/.ssh/authorized_keys. The most straightforward way of accomplishing this is to use scp to copy the keyfile to your remote host, typing the password to authenticate, then following up by logging in directly with ssh.
Instructions:
.pub keyfile$ ssh-copy-id -i /path/to/id_rsa.pub test@example.com:/home/test$ ssh test@example.com$ ssh example.com~/.ssh/config file, using the instruction in the note in \"Install an SSH key on a Remote Machine for passwordless login\".pub keyfile from the user's home into .ssh.ssh directory does not exist, create it with mkdir .ssh$ mv id_rsa.pub .ssh.ssh and cat the contents of id_rsa.pub into authorized_keys$ cd .ssh && cat id_rsa.pub authorized_keysid_rsa.pub key. Now that it's in authorized_keys, you don't need the keyfile on the remote machine anymore.It is crucial your chmod permissions are set properly on the ~/.ssh directory. Invalid permissions will lead to errors when trying to ssh into remote machines.
Check the table below for the chmod values you should use. To set a value (for example on the .ssh directory itself and the keypair):
$ chmod 700 ~/.ssh\n$ chmod 644 ~/.ssh/id_rsa{.pub}\n~/.ssh/ There is no general requirement to keep the entire contents of this directory secret, but the recommended permissions are read/write/execute for the user, and not accessible by others. 700 ~/.ssh/authorized_keys This file is not highly sensitive, but the recommended permissions are read/write for the user, and not accessible by others| 600 ~/.ssh/config Because of the potential for abuse, this file must have strict permissions: read/write for the user, and not writable by others 600 ~/.ssh/identity ~/.ssh/id_dsa ~/.ssh/id_rsa These files contain sensitive data and should be readable by the user but not accessible by others (read/write/execute) 600 ~/.ssh/identity.pub ~/.ssh/id_dsa.pub ~/.ssh/id_rsa.pub Contains the public key for authentication. These files are not sensitive and can (but need not) be readable by anyone. 644 *(table data source: Superuser.com answer)
","tags":["utilities","ssh"]},{"location":"tags.html","title":"tags","text":""},{"location":"tags.html#alembic","title":"alembic","text":"