Abstract class for an analyzer that computes overall variance metrics for subgroups.
+base_model
+Base model for stability measuring
+base_model_name (str)
+Model name like 'HoeffdingTreeClassifier' or 'LogisticRegression'
+bootstrap_fraction (float)
+[0-1], fraction from train_pd_dataset for fitting an ensemble of base models
+X_train (pandas.core.frame.DataFrame)
+Processed features train set
+y_train (pandas.core.frame.DataFrame)
+Targets train set
+X_test (pandas.core.frame.DataFrame)
+Processed features test set
+y_test (pandas.core.frame.DataFrame)
+Targets test set
+dataset_name (str)
+Name of dataset, used for correct results naming
+n_estimators (int)
+Number of estimators in ensemble to measure base_model stability
+random_state (int) – defaults to None
[Optional] Controls the randomness of the bootstrap approach for model arbitrariness evaluation
+with_predict_proba (bool) – defaults to True
[Optional] A flag if model can return probabilities for its predictions. If no, only metrics based on labels (not labels and probabilities) will be computed.
+notebook_logs_stdout (bool) – defaults to False
[Optional] True, if this interface was execute in a Jupyter notebook, False, otherwise.
+verbose (int) – defaults to 0
[Optional] Level of logs printing. The greater level provides more logs. As for now, 0, 1, 2 levels are supported.
+Quantifying uncertainty of the base model by constructing an ensemble from bootstrapped samples.
+Return a dictionary where keys are models indexes, and values are lists of correspondent model predictions for X_test set.
+Parameters
+True Measure metrics for the base model. Save results to a .csv file.
+Parameters
+True True Abstract class for a subgroup analyzer to compute metrics for subgroups.
+X_test (pandas.core.frame.DataFrame)
+Processed features test set
+y_test (pandas.core.frame.DataFrame)
+Targets test set
+sensitive_attributes_dct (dict)
+A dictionary where keys are sensitive attributes names (including attributes intersections), and values are privilege values for these attributes
+test_protected_groups (dict)
+A dictionary where keys are sensitive attributes, and values input dataset rows that are correspondent to these sensitive attributes
+computation_mode (str) – defaults to None
A mode to compute metrics. It can have two values 'error_analysis' and default (None).
+Compute metrics for each subgroup in self.test_protected_groups using _compute_metrics method.
+Return a dictionary where keys are subgroup names, and values are subgroup metrics.
+Parameters
+None None Parameters
+Analyzer to compute subgroup variance metrics for batch learning models.
+base_model
+Base model for stability measuring
+base_model_name (str)
+Model name like 'DecisionTreeClassifier' or 'LogisticRegression'
+bootstrap_fraction (float)
+[0-1], fraction from train_pd_dataset for fitting an ensemble of base models
+X_train (pandas.core.frame.DataFrame)
+Processed features train set
+y_train (pandas.core.frame.DataFrame)
+Targets train set
+X_test (pandas.core.frame.DataFrame)
+Processed features test set
+y_test (pandas.core.frame.DataFrame)
+Targets test set
+target_column (str)
+Name of the target column
+dataset_name (str)
+Name of dataset, used for correct results naming
+n_estimators (int)
+Number of estimators in ensemble to measure base_model stability
+random_state (int) – defaults to None
[Optional] Controls the randomness of the bootstrap approach for model arbitrariness evaluation
+with_predict_proba (bool) – defaults to True
[Optional] A flag if model can return probabilities for its predictions. If no, only metrics based on labels (not labels and probabilities) will be computed.
+notebook_logs_stdout (bool) – defaults to False
[Optional] True, if this interface was execute in a Jupyter notebook, False, otherwise.
+verbose (int) – defaults to 0
[Optional] Level of logs printing. The greater level provides more logs. As for now, 0, 1, 2 levels are supported.
+Quantifying uncertainty of the base model by constructing an ensemble from bootstrapped samples.
+Return a dictionary where keys are models indexes, and values are lists of correspondent model predictions for X_test set.
+Parameters
+True Measure metrics for the base model. Save results to a .csv file.
+Parameters
+True True Analyzer to compute subgroup variance metrics using the defined post-processor.
+postprocessor
+One of postprocessors from aif360 (https://aif360.readthedocs.io/en/stable/modules/algorithms.html#module-aif360.algorithms.postprocessing)
+sensitive_attribute (str)
+A sensitive attribute to use for post-processing
+base_model
+Base model for stability measuring
+base_model_name (str)
+Model name like 'DecisionTreeClassifier' or 'LogisticRegression'
+bootstrap_fraction (float)
+[0-1], fraction from train_pd_dataset for fitting an ensemble of base models
+X_train (pandas.core.frame.DataFrame)
+Processed features train set
+y_train (pandas.core.frame.DataFrame)
+Targets train set
+X_test (pandas.core.frame.DataFrame)
+Processed features test set
+y_test (pandas.core.frame.DataFrame)
+Targets test set
+target_column (str)
+Name of the target column
+dataset_name (str)
+Name of dataset, used for correct results naming
+n_estimators (int)
+Number of estimators in ensemble to measure base_model stability
+random_state (int) – defaults to None
[Optional] Controls the randomness of the bootstrap approach for model arbitrariness evaluation
+with_predict_proba (bool) – defaults to True
[Optional] A flag if model can return probabilities for its predictions. If no, only metrics based on labels (not labels and probabilities) will be computed.
+notebook_logs_stdout (bool) – defaults to False
[Optional] True, if this interface was execute in a Jupyter notebook, False, otherwise.
+verbose (int) – defaults to 0
[Optional] Level of logs printing. The greater level provides more logs. As for now, 0, 1, 2 levels are supported.
+Quantifying uncertainty of the base model by constructing an ensemble from bootstrapped samples and applying postprocessing intervention.
+Return a dictionary where keys are models indexes, and values are lists of correspondent model predictions for X_test set.
+Parameters
+True Measure metrics for the base model. Save results to a .csv file.
+Parameters
+True True Analyzer to compute error metrics for subgroups.
+X_test (pandas.core.frame.DataFrame)
+Processed features test set
+y_test (pandas.core.frame.DataFrame)
+Targets test set
+sensitive_attributes_dct (dict)
+A dictionary where keys are sensitive attributes names (including attributes intersections), and values are privilege values for these subgroups
+test_protected_groups (dict) – defaults to None
A dictionary where keys are sensitive attributes, and values input dataset rows that are correspondent to these sensitive attributes.
+computation_mode (str) – defaults to None
[Optional] A non-default mode for metrics computation. Should be included in the ComputationMode enum.
+Compute metrics for each subgroup in self.test_protected_groups using _compute_metrics method.
+Return a dictionary where keys are subgroup names, and values are subgroup metrics.
+Parameters
+None None Parameters
+Analyzer to compute variance metrics for subgroups.
+model_setting (metrics.ModelSetting)
+Model learning type; a constant from virny.configs.constants.ModelSetting
+n_estimators (int)
+Number of estimators for bootstrap
+base_model
+Initialized base model to analyze
+base_model_name (str)
+Model name
+bootstrap_fraction (float)
+[0-1], fraction from train_pd_dataset for fitting an ensemble of base models
+dataset (custom_classes.BaseFlowDataset)
+Initialized object of GenericPipeline class
+dataset_name (str)
+Name of dataset, used for correct results naming
+sensitive_attributes_dct (dict)
+A dictionary where keys are sensitive attribute names (including attributes intersections), and values are privilege values for these attributes
+test_protected_groups (dict)
+A dictionary of protected groups where keys are subgroup names, and values are X_test row indexes correspondent to this subgroup.
+postprocessor – defaults to None
One of postprocessors from aif360 (https://aif360.readthedocs.io/en/stable/modules/algorithms.html#module-aif360.algorithms.postprocessing)
+postprocessing_sensitive_attribute (str) – defaults to None
A sensitive attribute to use for post-processing
+random_state (int) – defaults to None
[Optional] Controls the randomness of the bootstrap approach for model arbitrariness evaluation
+computation_mode (str) – defaults to None
[Optional] A non-default mode for metrics computation. Should be included in the ComputationMode enum.
+with_predict_proba (bool) – defaults to True
[Optional] True, if models in models_config have a predict_proba method and can return probabilities for predictions, False, otherwise. Note that if it is set to False, only metrics based on labels (not labels and probabilities) will be computed. Ignored when a postprocessor is not None, and set to False in this case.
+notebook_logs_stdout (bool) – defaults to False
[Optional] True, if this interface was execute in a Jupyter notebook, False, otherwise.
+verbose (int) – defaults to 0
[Optional] Level of logs printing. The greater level provides more logs. As for now, 0, 1, 2 levels are supported.
+Measure variance metrics for subgroups for the base model. Display variance plots for analysis if needed. Save results to a .csv file if needed.
+Return averaged bootstrap predictions and a pandas dataframe of variance metrics for subgroups.
+Parameters
+None None True Calculator that calculates variance metrics for subgroups.
+X_test (pandas.core.frame.DataFrame)
+Processed features test set
+y_test (pandas.core.frame.DataFrame)
+Targets test set
+sensitive_attributes_dct (dict)
+A dictionary where keys are sensitive attributes names (including attributes intersections), and values are privilege values for these subgroups
+test_protected_groups – defaults to None
A dictionary where keys are sensitive attributes, and values input dataset rows that are correspondent to these sensitive attributes.
+computation_mode (str) – defaults to None
[Optional] A non-default mode for metrics computation. Should be included in the ComputationMode enum.
+with_predict_proba (bool) – defaults to True
[Optional] A flag if model can return probabilities for its predictions. If no, only metrics based on labels (not labels and probabilities) will be computed.
+Compute variance metrics for subgroups.
+Return a dict of dicts where key is 'overall' or a subgroup name, and value is a dict of metrics for this subgroup.
+Parameters
+None None Parameters
+Dataset class with custom train and test splits that is used as input for metrics computation interfaces. Create your dataset class based on this one to use it for metrics computation interfaces.
+init_sensitive_attrs_df (pandas.core.frame.DataFrame)
+Full train + test non-preprocessed dataset of sensitive attributes with initial indexes. It is used for creating test groups.
+X_train_val (pandas.core.frame.DataFrame)
+Train dataframe of features
+X_test (pandas.core.frame.DataFrame)
+Test dataframe of features
+y_train_val (pandas.core.frame.DataFrame)
+Train dataframe with a target column
+y_test (pandas.core.frame.DataFrame)
+Test dataframe with a target column
+target (str)
+Name of the target column name
+numerical_columns (list)
+List of numerical column names
+categorical_columns (list)
+List of categorical column names
+Metric Composer class that combines different subgroup metrics to create disparity metrics such as 'Disparate_Impact' or 'Accuracy_Difference'.
+Definitions of the disparity metrics could be observed in the init method of the Metric Composer: https://github.com/DataResponsibly/Virny/blob/main/virny/custom_classes/metrics_composer.py
+models_metrics_dct (dict)
+Dictionary where keys are model names and values are dataframes of subgroups metrics for each model
+sensitive_attributes_dct (dict)
+A dictionary where keys are sensitive attribute names (including attributes intersections), and values are privilege values for these attributes
+Compose subgroup metrics from self.model_metrics_df.
+Return a dictionary of composed metrics.
+Class to create an interactive web app based on models metrics.
+X_data (pandas.core.frame.DataFrame)
+An original features dataframe
+y_data (pandas.core.frame.DataFrame)
+An original target column pandas series
+model_metrics
+A dictionary or a dataframe where keys are model names and values are dataframes of subgroup metrics for each model
+sensitive_attributes_dct (dict)
+A dictionary where keys are sensitive attributes names (including attributes intersections), and values are privilege values for these attributes
+Build an interactive web application.
+Parameters
+True Class to create useful visualizations of models metrics.
+models_metrics_dct (dict)
+Dictionary where keys are model names and values are dataframes of subgroup metrics for each model
+models_composed_metrics_df (pandas.core.frame.DataFrame)
+Dataframe of all model composed metrics
+dataset_name (str)
+Name of a dataset that was included in metric filenames and was used for the metrics computation
+model_names (list)
+Metrics for what model names to visualize
+sensitive_attributes_dct (dict)
+A dictionary where keys are sensitive attributes names (including attributes intersections), and values are privilege values for these attributes
+This boxes and whiskers plot is based on overall subgroup error and stability metrics for all defined models and results after all runs. Using it, you can see combined information on one plot that includes different models, subgroup metrics, and results after multiple runs.
+Parameters
+Create a heatmap for disparity metrics.
+Parameters
+0.001 (0.7, 0.5) -3 Create a heatmap for overall metrics.
+Parameters
+0.001 (0.7, 0.5) -3 This bar chart includes all defined models and all overall subgroup error and stability metrics, which are averaged across multiple runs. Using it, you can compare all models for each subgroup error or stability metric. This comparison also includes reversed metrics, in which values closer to zero are better since straight and reversed metrics in this plot are converted to the same format -- values closer to one are better.
+Parameters
+Overall Metrics Dataset class for the employment task from the folktables dataset. Target: binary classification, predict if a person is employed. Source of the dataset: https://github.com/socialfoundations/folktables
+state
+State in the US for which to get the data. All states in the US are available.
+year
+Year for which to get the data. Five different years of data collection are available: 2014–2018 inclusive.
+root_dir – defaults to None
Path to the root directory where to store the extracted dataset or where it is stored.
+with_nulls – defaults to False
Whether to keep nulls in the dataset or replace them on the new categorical class. Default: False.
+with_filter – defaults to True
Whether to use a folktables filter for this task. Default: True.
+optimize – defaults to True
Whether to optimize the dataset size by downcasting categorical columns. Default: True.
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset.
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas.
+To save simulated nulls
+Parameters
+Dataset class for the income task from the folktables dataset. Target: binary classification, predict if a person has an annual income > $50,000. Source of the dataset: https://github.com/socialfoundations/folktables
+state
+State in the US for which to get the data. All states in the US are available.
+year
+Year for which to get the data. Five different years of data collection are available: 2014–2018 inclusive.
+root_dir – defaults to None
Path to the root directory where to store the extracted dataset or where it is stored.
+with_nulls – defaults to False
Whether to keep nulls in the dataset or replace them on the new categorical class. Default: False.
+with_filter – defaults to True
Whether to use a folktables filter for this task. Default: True.
+optimize – defaults to True
Whether to optimize the dataset size by downcasting categorical columns. Default: True.
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset.
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas.
+To save simulated nulls
+Parameters
+Dataset class for the mobility task from the folktables dataset. Target: binary classification, predict whether a young adult moved addresses in the last year. Source of the dataset: https://github.com/socialfoundations/folktables
+state
+State in the US for which to get the data. All states in the US are available.
+year
+Year for which to get the data. Five different years of data collection are available: 2014–2018 inclusive.
+root_dir – defaults to None
Path to the root directory where to store the extracted dataset or where it is stored.
+with_nulls – defaults to False
Whether to keep nulls in the dataset or replace them on the new categorical class. Default: False.
+To save simulated nulls
+Parameters
+Dataset class for the public coverage task from the folktables dataset. Target: binary classification, predict whether a low-income individual, not eligible for Medicare, has coverage from public health insurance. Source of the dataset: https://github.com/socialfoundations/folktables
+state
+State in the US for which to get the data. All states in the US are available.
+year
+Year for which to get the data. Five different years of data collection are available: 2014–2018 inclusive.
+root_dir – defaults to None
Path to the root directory where to store the extracted dataset or where it is stored.
+with_nulls – defaults to False
Whether to keep nulls in the dataset or replace them on the new categorical class. Default: False.
+with_filter – defaults to True
Whether to use a folktables filter for this task. Default: True.
+optimize – defaults to True
Whether to optimize the dataset size by downcasting categorical columns. Default: True.
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset.
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas.
+To save simulated nulls
+Parameters
+Dataset class for the travel time task from the folktables dataset. Target: binary classification, predict whether a working adult has a travel time to work of greater than 20 minutes. Source of the dataset: https://github.com/socialfoundations/folktables
+state
+State in the US for which to get the data. All states in the US are available.
+year
+Year for which to get the data. Five different years of data collection are available: 2014–2018 inclusive.
+root_dir – defaults to None
Path to the root directory where to store the extracted dataset or where it is stored.
+with_nulls – defaults to False
Whether to keep nulls in the dataset or replace them on the new categorical class. Default: False.
+To save simulated nulls
+Parameters
+Dataset class for the Bank Marketing dataset that contains sensitive attributes among feature columns. Source: https://github.com/tailequy/fairness_dataset/blob/main/experiments/data/bank-full.csv General description and analysis: https://arxiv.org/pdf/2110.00530.pdf (Section 3.1.5) Broad description: https://archive.ics.uci.edu/dataset/222/bank+marketing
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas
+Dataset class for the Cardiovascular Disease dataset that contains sensitive attributes among feature columns. Source and broad description: https://www.kaggle.com/datasets/sulianova/cardiovascular-disease-dataset
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas
+Dataset class for the COMPAS dataset that contains sensitive attributes among feature columns.
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas
+dataset_path – defaults to None
[Optional] Path to a file with the data
+Dataset class for the COMPAS dataset that does not contain sensitive attributes among feature columns to test blind classifiers
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas
+dataset_path – defaults to None
[Optional] Path to a file with the data
+Dataset class for the Diabetes 2019 dataset that contains sensitive attributes among feature columns. Source and broad description: https://www.kaggle.com/datasets/tigganeha4/diabetes-dataset-2019/data
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas
+with_nulls (bool) – defaults to True
Whether to keep nulls in the dataset or drop rows with any nulls. Default: True.
+Dataset class for the German Credit dataset that contains sensitive attributes among feature columns. Source: https://github.com/tailequy/fairness_dataset/blob/main/experiments/data/german_data_credit.csv General description and analysis: https://arxiv.org/pdf/2110.00530.pdf (Section 3.1.3) Broad description: https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas
+Dataset class for the Law School dataset that contains sensitive attributes among feature columns. Source: https://github.com/tailequy/fairness_dataset/blob/main/experiments/data/law_school_clean.csv Description: https://arxiv.org/pdf/2110.00530.pdf
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas
+dataset_path – defaults to None
[Optional] Path to a file with the data
+Dataset class for the Ricci dataset that contains sensitive attributes among feature columns. Source: https://github.com/tailequy/fairness_dataset/blob/main/experiments/data/ricci_race.csv Description: https://arxiv.org/pdf/2110.00530.pdf
+dataset_path – defaults to None
[Optional] Path to a file with the data
+Dataset class for the Student Performance Portuguese dataset that contains sensitive attributes among feature columns. Source: https://github.com/tailequy/fairness_dataset/blob/main/experiments/data/student_por_new.csv Description: https://arxiv.org/pdf/2110.00530.pdf (Section 3.4.1)
+subsample_size (int) – defaults to None
Subsample size to create based on the input dataset
+subsample_seed (int) – defaults to None
Seed for sampling using the sample() method from pandas
+Compute aleatoric uncertainty as average predictive entropy.
+y_true (pandas.core.frame.DataFrame)
+A pandas dataframe of true labels. Is not used in this function, required for consistency.
+uq_predict_probas (pandas.core.frame.DataFrame)
+A pandas dataframe of predictions (probabilities) from all estimators in the bootstrap.
+Compute inter-quantile range (IQR) of predictive variance.
+y_true (pandas.core.frame.DataFrame)
+A pandas dataframe of true labels. Is not used in this function, required for consistency.
+uq_predict_probas (pandas.core.frame.DataFrame)
+A pandas dataframe of predictions (probabilities) from all estimators in the bootstrap
+Jitter is a stability metric that shows how the base model predictions fluctuate. Values closer to 0 -- perfect stability, values closer to 1 -- extremely bad stability.
+y_true (pandas.core.frame.DataFrame)
+A pandas dataframe of true labels. Is not used in this function, required for consistency.
+uq_labels (pandas.core.frame.DataFrame)
+uq_labels variable from count_prediction_metrics()
Compute per-sample accuracy for each model predictions.
+Return per_sample_accuracy and label_stability (refer to https://www.osti.gov/servlets/purl/1527311)
+y_true (pandas.core.frame.DataFrame)
+y test dataset
+uq_labels (pandas.core.frame.DataFrame)
+uq_labels variable from count_prediction_metrics()
Compute mean predictions of all estimators in the boostrap.
+y_true (pandas.core.frame.DataFrame)
+A pandas dataframe of true labels. Is not used in this function, required for consistency.
+uq_predict_probas (pandas.core.frame.DataFrame)
+A pandas dataframe of predictions (probabilities) from all estimators in the bootstrap
+Compute overall uncertainty as predictive entropy.
+y_true (pandas.core.frame.DataFrame)
+A pandas dataframe of true labels. Is not used in this function, required for consistency.
+uq_predict_probas (pandas.core.frame.DataFrame)
+A pandas dataframe of predictions (probabilities) from all estimators in the bootstrap.
+Compute statistical bias.
+y_true (pandas.core.frame.DataFrame)
+A pandas dataframe of true labels. Is not used in this function, required for consistency.
+uq_predict_probas (pandas.core.frame.DataFrame)
+A pandas dataframe of predictions (probabilities) from all estimators in the bootstrap
+Compute standard deviation of predictive variance.
+y_true (pandas.core.frame.DataFrame)
+A pandas dataframe of true labels. Is not used in this function, required for consistency.
+uq_predict_probas (pandas.core.frame.DataFrame)
+A pandas dataframe of predictions (probabilities) from all estimators in the bootstrap
+Subgroup Error and Variance Analyzers.
+This module contains fairness and stability analysing methods for defined subgroups. +The purpose of an analyzer is to analyse defined metrics for defined subgroups.
+Configs amd constants for the source code logic.
+This module contains custom classes for metrics computation interfaces. +The purpose is to split metrics computation and visualization pipeline on components +that are highly customizable for future library features.
+ +This module contains sample datasets and data loaders. +The purpose is to provide sample datasets for functionality testing and show examples of data loaders (aka dataset classes).
+This module contains functions for computing subgroup variance and error metrics.
+Preprocessing techniques.
+This module contains function for input dataset preprocessing.
+ +User interfaces.
+This module contains user interfaces for metrics computation.
+Common helpers and utils.
+ + + + + + + + + + + + + + + + +Return a dataset made by one-hot encoding for categorical columns and concatenate with numerical columns.
+data (pandas.core.frame.DataFrame)
+Dataframe for one-hot encoding
+categorical_columns (list)
+List of categorical column names
+numerical_columns (list)
+List of numerical column names
+Return preprocessed train and test feature dataframes after one-hot encoding and standard scaling.
+X_train (pandas.core.frame.DataFrame)
+X_test (pandas.core.frame.DataFrame)
+dataset (custom_classes.BaseFlowDataset)
+Preprocess an input dataset using sklearn ColumnTransformer. Split the dataset on train and test using test_set_fraction. Create an instance of BaseFlowDataset.
+data_loader (virny.datasets.base.BaseDataLoader)
+Instance of BaseDataLoader that contains a target, numerical, and categorical columns.
+column_transformer (sklearn.compose._column_transformer.ColumnTransformer)
+Instance of sklearn ColumnTransformer to preprocess categorical and numerical columns.
+sensitive_attributes_dct (dict)
+Dictionary of sensitive attribute names and their disadvantaged values.
+test_set_fraction (float)
+Fraction from 0 to 1. Used to split the input dataset on the train and test sets.
+dataset_split_seed (int)
+Seed for dataset splitting.
+Compute stability and accuracy metrics for each model in models_config. Arguments are defined as an input config object. Save results in save_results_dir_path folder.
Return a dictionary where keys are model names, and values are metrics for sensitive attributes defined in config.
+dataset (custom_classes.BaseFlowDataset)
+BaseFlowDataset object that contains all needed attributes like target, features, numerical_columns etc.
+config
+Object that contains bootstrap_fraction, dataset_name, n_estimators, sensitive_attributes_dct attributes
+models_config (dict)
+Dictionary where keys are model names, and values are initialized models
+save_results_dir_path (str)
+Location where to save result files with metrics
+postprocessor – defaults to None
[Optional] Postprocessor object to apply to model predictions before metrics computation
+with_predict_proba (bool) – defaults to True
[Optional] True, if models in models_config have a predict_proba method and can return probabilities for predictions, False, otherwise. Note that if it is set to False, only metrics based on labels (not labels and probabilities) will be computed. Ignored when a postprocessor is not None, and set to False in this case.
+notebook_logs_stdout (bool) – defaults to False
[Optional] True, if this interface was execute in a Jupyter notebook, False, otherwise.
+verbose (int) – defaults to 0
[Optional] Level of logs printing. The greater level provides more logs. As for now, 0, 1, 2 levels are supported. Currently, verbose works only with notebook_logs_stdout = False.
+Compute stability and accuracy metrics for each model in models_config. Arguments are defined as an input config object. Save results to a database after each run appending fields and value from custom_tbl_fields_dct and using db_writer_func.
+Return a dictionary where keys are model names, and values are metrics for sensitive attributes defined in config.
+dataset (custom_classes.BaseFlowDataset)
+BaseFlowDataset object that contains all needed attributes like target, features, numerical_columns etc.
+config
+Object that contains bootstrap_fraction, dataset_name, n_estimators, sensitive_attributes_dct attributes
+models_config (dict)
+Dictionary where keys are model names, and values are initialized models
+custom_tbl_fields_dct (dict)
+Dictionary where keys are column names and values to add to inserted metrics during saving results to a database
+db_writer_func
+Python function object has one argument (run_models_metrics_df) and save this metrics df to a target database
+postprocessor – defaults to None
[Optional] Postprocessor object to apply to model predictions before metrics computation
+with_predict_proba (bool) – defaults to True
[Optional] True, if models in models_config have a predict_proba method and can return probabilities for predictions, False, otherwise. Note that if it is set to False, only metrics based on labels (not labels and probabilities) will be computed. Ignored when a postprocessor is not None, and set to False in this case.
+notebook_logs_stdout (bool) – defaults to False
[Optional] True, if this interface was execute in a Jupyter notebook, False, otherwise.
+verbose (int) – defaults to 0
[Optional] Level of logs printing. The greater level provides more logs. As for now, 0, 1, 2 levels are supported. Currently, verbose works only with notebook_logs_stdout = False.
+Compute stability and accuracy metrics for each model in models_config based on dataset.X_test and each extra test set in extra_test_sets_lst. Arguments are defined as an input config object. Save results to a database after each run appending fields and value from custom_tbl_fields_dct and using db_writer_func. Index of each test set is also added as a separate column in out final records in the database (0 index -- for dataset.X_test, 1 and greater -- for each extra test set in extra_test_sets_lst, keeping the original sequence).
+dataset (custom_classes.BaseFlowDataset)
+BaseFlowDataset object that contains all needed attributes like target, features, numerical_columns etc.
+extra_test_sets_lst
+List of extra test sets like [(X_test1, y_test1), (X_test2, y_test2), ...] to compute metrics that are not equal to original dataset.X_test and dataset.y_test
+config
+Object that contains bootstrap_fraction, dataset_name, n_estimators, sensitive_attributes_dct attributes
+models_config (dict)
+Dictionary where keys are model names, and values are initialized models
+custom_tbl_fields_dct (dict)
+Dictionary where keys are column names and values to add to inserted metrics during saving results to a database
+db_writer_func
+Python function object has one argument (run_models_metrics_df) and save this metrics df to a target database
+with_predict_proba (bool) – defaults to True
[Optional] True, if models in models_config have a predict_proba method and can return probabilities for predictions, False, otherwise. Note that if it is set to False, only metrics based on labels (not labels and probabilities) will be computed. Ignored when a postprocessor is not None, and set to False in this case.
+notebook_logs_stdout (bool) – defaults to False
[Optional] True, if this interface was execute in a Jupyter notebook, False, otherwise.
+verbose (int) – defaults to 0
[Optional] Level of logs printing. The greater level provides more logs. As for now, 0, 1, 2 levels are supported. Currently, verbose works only with notebook_logs_stdout = False.
+Compute means, stds, iqr, entropy, jitter, label stability, and transform predictions to pd.Dataframe.
+Return a 1D numpy array of predictions, 2D array of each model prediction for y_test, a data structure of metrics.
+y_true
+True labels
+uq_results
+2D array of prediction proba for the zero value label by each model
+with_predict_proba (bool) – defaults to True
[Optional] A flag if model can return probabilities for its predictions. If no, only metrics based on labels (not labels and probabilities) will be computed.
+Create protected groups based on a test feature set. Use a disadvantaged group as a reference group.
+Return a dictionary where keys are subgroup names, and values are X_test row indexes correspondent to this subgroup.
+X_test (pandas.core.frame.DataFrame)
+Test feature set
+init_sensitive_attrs_df (pandas.core.frame.DataFrame)
+Initial full dataset of sensitive attributes without preprocessing
+sensitive_attributes_dct (dict)
+A dictionary where keys are sensitive attribute names (including attributes intersections), and values are disadvantaged values for these attributes
+Tune each model on a validation set with GridSearchCV.
+Return each model with its best hyperparameters that have the highest F1 score and Accuracy. results_df is a dataframe with metrics and tuned parameters; models_config is a dict with model tuned params for the metrics computation stage
+models_params_for_tuning (dict)
+A dictionary, where keys are model names and values are a dictionary of hyperparameters and value ranges to tune.
+base_flow_dataset (custom_classes.BaseFlowDataset)
+An instance of BaseFlowDataset object. Its train and test sets are used for training and tuning.
+dataset_name (str)
+A name of the dataset. Used to save tuned hyperparameters to a csv file with an appropriate filename.
+n_folds (int) – defaults to 3
The number of folds for k-fold cross validation.
+Validate parameters types and values in config yaml file.
+Extra details: * config_obj.model_setting is an optional argument that defines a type of models to use to compute fairness and stability metrics. Default: 'batch'.
+config_obj
+Object with parameters defined in a yaml file
+