perf: Quick prefix validation check (prefix-dev#2400)

This pr will add a hash of the lock file to the `EnvironmentFile`
(`.pixi/envs/default/conda-meta/pixi`)

It will look like this:
```json
{
  "manifest_path": "/home/rarts/dev/pixi/pixi.toml",
  "environment_name": "default",
  "pixi_version": "0.34.0",
  "environment_lock_file_hash": "4f36ee620f10329d"
}
```
And that hash will be compared with the current lockfile on `pixi run`
`pixi shell` `pixi shell-hook`.

### Profile result

```shell
❯ hyperfine "pixi run echo" "old-pixi run echo"
Benchmark 1: pixi run echo
  Time (mean ± σ):     381.6 ms ±  22.1 ms    [User: 193.8 ms, System: 246.3 ms]
  Range (min … max):   344.5 ms … 414.3 ms    10 runs
 
Benchmark 2: old-pixi run echo
  Time (mean ± σ):     868.2 ms ±  58.2 ms    [User: 480.0 ms, System: 557.0 ms]
  Range (min … max):   791.1 ms … 950.8 ms    10 runs
 
Summary
  pixi run echo ran
    2.28 ± 0.20 times faster than old-pixi run echo
```
> [!NOTE]  
> The remaining `381ms` is the activation which is fixed by
prefix-dev#2367

### UX
- It's turned on by default
- You can request a re-validate on `pixi run/shell/shell-hook` with
`--revalidate`
- All commands designed to update the lock file or `pixi install` will
always re-validate.

### TODO:

- [x] : Add tests: chosen python integration tests as I was to fed up
with the extreme amount of time spent on writing tests in Rust
- [x] : use Enum instead of booleans. Using `UpdateMode::QuickValidate`
and `UpdateMode::Revalidate`
- [x] : Document behavior
- [x] : Extend logic to cli. `--revalidate`

---------

Co-authored-by: Hofer-Julian <[email protected]>

docs/advanced/explain_info_command.md

@@ -57,7 +57,7 @@ In that case, if pixi cannot find the `__cuda` virtual package on your machine t
 ### Cache dir
 
 The directory where pixi stores its cache.
-Checkout the [cache documentation](../features/environment.md#caching) for more information.
+Checkout the [cache documentation](../features/environment.md#caching-packages) for more information.
 
 ### Auth storage
 

docs/features/environment.md

@@ -40,6 +40,31 @@ These directories are conda environments, and you can use them as such, but you
 Pixi will always make sure the environment is in sync with the `pixi.lock` file.
 If this is not the case then all the commands that use the environment will automatically update the environment, e.g. `pixi run`, `pixi shell`.
 
+### Environment Installation Metadata
+On environment installation, pixi will write a small file to the environment that contains some metadata about installation.
+This file is called `pixi` and is located in the `conda-meta` folder of the environment.
+This file contains the following information:
+- `manifest_path`: The path to the manifest file that describes the project used to create this environment
+- `environment_name`: The name of the environment
+- `pixi_version`: The version of pixi that was used to create this environment
+- `environment_lock_file_hash`: The hash of the `pixi.lock` file that was used to create this environment
+
+```json
+{
+  "manifest_path": "/home/user/dev/pixi/pixi.toml",
+  "environment_name": "default",
+  "pixi_version": "0.34.0",
+  "environment_lock_file_hash": "4f36ee620f10329d"
+}
+```
+
+The `environment_lock_file_hash` is used to check if the environment is in sync with the `pixi.lock` file.
+If the hash of the `pixi.lock` file is different from the hash in the `pixi` file, pixi will update the environment.
+
+This is used to speedup activation, in order to trigger a full revalidation pass `--revalidate` to the `pixi run` or `pixi shell` command.
+A broken environment would typically not be found with a hash comparison, but a revalidation would reinstall the environment.
+By default, all lock file modifying commands will always use the revalidation and on `pixi install` it always revalidates.
+
 ### Cleaning up
 
 If you want to clean up the environments, you can simply delete the `.pixi/envs` directory, and pixi will recreate the environments when needed.
@@ -186,7 +211,7 @@ For the `[pypi-dependencies]`, `uv` implements `sdist` building to retrieve the
 For this building step, `pixi` requires to first install `python` in the (conda)`[dependencies]` section of the `pixi.toml` file.
 This will always be slower than the pure conda solves. So for the best pixi experience you should stay within the `[dependencies]` section of the `pixi.toml` file.
 
-## Caching
+## Caching packages
 
 Pixi caches all previously downloaded packages in a cache folder.
 This cache folder is shared between all pixi projects and globally installed tools.

docs/reference/cli.md

@@ -208,6 +208,8 @@ You cannot run `pixi run source setup.bash` as `source` is not available in the
 - `--locked`: only install if the `pixi.lock` is up-to-date with the [manifest file](project_configuration.md)[^1]. It can also be controlled by the `PIXI_LOCKED` environment variable (example: `PIXI_LOCKED=true`). Conflicts with `--frozen`.
 - `--environment <ENVIRONMENT> (-e)`: The environment to run the task in, if none are provided the default environment will be used or a selector will be given to select the right environment.
 - `--clean-env`: Run the task in a clean environment, this will remove all environment variables of the shell environment except for the ones pixi sets. THIS DOESN't WORK ON `Windows`.
+- `--revalidate`: Revalidate the full environment, instead of checking the lock file hash. [more info](../features/environment.md#environment-installation-metadata)
+
 ```shell
 pixi run python
 pixi run cowpy "Hey pixi user"
@@ -636,6 +638,7 @@ To exit the pixi shell, simply run `exit`.
 - `--frozen`: install the environment as defined in the lock file, doesn't update `pixi.lock` if it isn't up-to-date with [manifest file](project_configuration.md). It can also be controlled by the `PIXI_FROZEN` environment variable (example: `PIXI_FROZEN=true`).
 - `--locked`: only install if the `pixi.lock` is up-to-date with the [manifest file](project_configuration.md)[^1]. It can also be controlled by the `PIXI_LOCKED` environment variable (example: `PIXI_LOCKED=true`). Conflicts with `--frozen`.
 - `--environment <ENVIRONMENT> (-e)`: The environment to activate the shell in, if none are provided the default environment will be used or a selector will be given to select the right environment.
+- `--revalidate`: Revalidate the full environment, instead of checking lock file hash. [more info](../features/environment.md#environment-installation-metadata)
 
 ```shell
 pixi shell
@@ -664,6 +667,7 @@ This command prints the activation script of an environment.
 - `--environment <ENVIRONMENT> (-e)`: The environment to activate, if none are provided the default environment will be used or a selector will be given to select the right environment.
 - `--json`: Print all environment variables that are exported by running the activation script as JSON. When specifying
   this option, `--shell` is ignored.
+- `--revalidate`: Revalidate the full environment, instead of checking lock file hash. [more info](../features/environment.md#environment-installation-metadata)
 
 ```shell
 pixi shell-hook

src/cli/add.rs

@@ -22,7 +22,7 @@ use crate::{
     cli::cli_config::{DependencyConfig, PrefixUpdateConfig, ProjectConfig},
     environment::verify_prefix_location_unchanged,
     load_lock_file,
-    lock_file::{filter_lock_file, LockFileDerivedData, UpdateContext},
+    lock_file::{filter_lock_file, LockFileDerivedData, UpdateContext, UpdateMode},
     project::{grouped_environment::GroupedEnvironment, DependencyType, Project},
 };
 
@@ -294,7 +294,7 @@ pub async fn execute(args: Args) -> miette::Result<()> {
         && default_environment_is_affected
     {
         updated_lock_file
-            .prefix(&project.default_environment())
+            .prefix(&project.default_environment(), UpdateMode::Revalidate)
             .await?;
     }
 

src/cli/cli_config.rs

@@ -1,5 +1,6 @@
 use crate::cli::has_specs::HasSpecs;
 use crate::environment::LockFileUsage;
+use crate::lock_file::UpdateMode;
 use crate::DependencyType;
 use crate::Project;
 use clap::Parser;
@@ -98,6 +99,10 @@ pub struct PrefixUpdateConfig {
 
     #[clap(flatten)]
     pub config: ConfigCli,
+
+    /// Run the complete environment validation. This will reinstall a broken environment.
+    #[arg(long)]
+    pub revalidate: bool,
 }
 impl PrefixUpdateConfig {
     pub fn lock_file_usage(&self) -> LockFileUsage {
@@ -114,6 +119,15 @@ impl PrefixUpdateConfig {
     pub(crate) fn no_install(&self) -> bool {
         self.no_install || self.no_lockfile_update
     }
+
+    /// Which `[UpdateMode]` to use
+    pub(crate) fn update_mode(&self) -> UpdateMode {
+        if self.revalidate {
+            UpdateMode::Revalidate
+        } else {
+            UpdateMode::QuickValidate
+        }
+    }
 }
 #[derive(Parser, Debug, Default)]
 pub struct DependencyConfig {

src/cli/install.rs

@@ -1,5 +1,6 @@
 use crate::cli::cli_config::ProjectConfig;
 use crate::environment::update_prefix;
+use crate::lock_file::UpdateMode;
 use crate::Project;
 use clap::Parser;
 use fancy_display::FancyDisplay;
@@ -52,7 +53,13 @@ pub async fn execute(args: Args) -> miette::Result<()> {
         let environment = project.environment_from_name_or_env_var(Some(env))?;
 
         // Update the prefix by installing all packages
-        update_prefix(&environment, args.lock_file_usage.into(), false).await?;
+        update_prefix(
+            &environment,
+            args.lock_file_usage.into(),
+            false,
+            UpdateMode::Revalidate,
+        )
+        .await?;
 
         installed_envs.push(environment.name().clone());
     }

src/cli/project/channel/add.rs

@@ -1,5 +1,6 @@
 use crate::{
     environment::{update_prefix, LockFileUsage},
+    lock_file::UpdateMode,
     Project,
 };
 
@@ -16,6 +17,7 @@ pub async fn execute(mut project: Project, args: AddRemoveArgs) -> miette::Resul
         &project.default_environment(),
         LockFileUsage::Update,
         args.no_install,
+        UpdateMode::Revalidate,
     )
     .await?;
     project.save()?;

src/cli/project/channel/remove.rs

@@ -1,3 +1,4 @@
+use crate::lock_file::UpdateMode;
 use crate::{
     environment::{update_prefix, LockFileUsage},
     Project,
@@ -16,6 +17,7 @@ pub async fn execute(mut project: Project, args: AddRemoveArgs) -> miette::Resul
         &project.default_environment(),
         LockFileUsage::Update,
         args.no_install,
+        UpdateMode::Revalidate,
     )
     .await?;
     project.save()?;

src/cli/project/platform/add.rs

@@ -2,6 +2,7 @@ use std::str::FromStr;
 
 use crate::{
     environment::{update_prefix, LockFileUsage},
+    lock_file::UpdateMode,
     Project,
 };
 use clap::Parser;
@@ -48,6 +49,7 @@ pub async fn execute(mut project: Project, args: Args) -> miette::Result<()> {
         &project.default_environment(),
         LockFileUsage::Update,
         args.no_install,
+        UpdateMode::Revalidate,
     )
     .await?;
     project.save()?;

src/cli/project/platform/remove.rs

@@ -1,5 +1,6 @@
 use std::str::FromStr;
 
+use crate::lock_file::UpdateMode;
 use crate::{
     environment::{update_prefix, LockFileUsage},
     Project,
@@ -47,6 +48,7 @@ pub async fn execute(mut project: Project, args: Args) -> miette::Result<()> {
         &project.default_environment(),
         LockFileUsage::Update,
         args.no_install,
+        UpdateMode::Revalidate,
     )
     .await?;
     project.save()?;

src/cli/remove.rs

@@ -6,6 +6,7 @@ use crate::DependencyType;
 use crate::Project;
 
 use crate::cli::cli_config::{DependencyConfig, PrefixUpdateConfig, ProjectConfig};
+use crate::lock_file::UpdateMode;
 
 use super::has_specs::HasSpecs;
 
@@ -82,6 +83,7 @@ pub async fn execute(args: Args) -> miette::Result<()> {
             &project.default_environment(),
             prefix_update_config.lock_file_usage(),
             prefix_update_config.no_install,
+            UpdateMode::Revalidate,
         )
         .await?;
     }

src/cli/run.rs

@@ -1,12 +1,11 @@
-use std::collections::hash_map::Entry;
-use std::collections::HashSet;
-use std::convert::identity;
-use std::{collections::HashMap, string::String};
-
 use clap::Parser;
 use dialoguer::theme::ColorfulTheme;
 use itertools::Itertools;
 use miette::{Diagnostic, IntoDiagnostic};
+use std::collections::hash_map::Entry;
+use std::collections::HashSet;
+use std::convert::identity;
+use std::{collections::HashMap, string::String};
 
 use crate::cli::cli_config::{PrefixUpdateConfig, ProjectConfig};
 use crate::environment::verify_prefix_location_unchanged;
@@ -169,7 +168,12 @@ pub async fn execute(args: Args) -> miette::Result<()> {
             Entry::Occupied(env) => env.into_mut(),
             Entry::Vacant(entry) => {
                 // Ensure there is a valid prefix
-                lock_file.prefix(&executable_task.run_environment).await?;
+                lock_file
+                    .prefix(
+                        &executable_task.run_environment,
+                        args.prefix_update_config.update_mode(),
+                    )
+                    .await?;
 
                 let command_env = get_task_env(
                     &executable_task.run_environment,

src/cli/shell.rs

@@ -9,6 +9,7 @@ use rattler_shell::{
 };
 
 use crate::cli::cli_config::{PrefixUpdateConfig, ProjectConfig};
+use crate::lock_file::UpdateMode;
 use crate::{
     activation::CurrentEnvVarBehavior, environment::update_prefix,
     project::virtual_packages::verify_current_platform_has_required_virtual_packages, prompt,
@@ -241,6 +242,7 @@ pub async fn execute(args: Args) -> miette::Result<()> {
         &environment,
         args.prefix_update_config.lock_file_usage(),
         false,
+        UpdateMode::QuickValidate,
     )
     .await?;
 

src/cli/shell_hook.rs

@@ -10,12 +10,12 @@ use rattler_shell::{
 use serde::Serialize;
 use serde_json;
 
-use crate::cli::cli_config::{PrefixUpdateConfig, ProjectConfig};
-use crate::project::HasProjectRef;
+use crate::activation::CurrentEnvVarBehavior;
+use crate::environment::update_prefix;
 use crate::{
-    activation::{get_activator, CurrentEnvVarBehavior},
-    environment::update_prefix,
-    project::Environment,
+    activation::get_activator,
+    cli::cli_config::{PrefixUpdateConfig, ProjectConfig},
+    project::{Environment, HasProjectRef},
     Project,
 };
 
@@ -113,6 +113,7 @@ pub async fn execute(args: Args) -> miette::Result<()> {
         &environment,
         args.prefix_update_config.lock_file_usage(),
         false,
+        args.prefix_update_config.update_mode(),
     )
     .await?;