Ensemble Integration (EI): Integrating multimodal data through interpretable heterogeneous ensembles

The latest version of EI fully written in python is implemented here, or you may install it by pip install ensemble-integration with full documentation.

Ensemble Integration (EI) is a customizable pipeline for generating diverse ensembles of heterogeneous classifiers, as well as the accompanying metadata needed for ensemble learning approaches utilizing ensemble diversity for improved performance. It also fairly evaluates the performance of several ensemble learning methods including ensemble selection [Caruana2004], and stacked generalization (stacking) [Wolpert1992]. Though other tools exist, we are unaware of a similarly modular, scalable pipeline designed for large-scale ensemble learning. EI was developed to support research by Yan Chak Li, Linhua Wang, and Gaurav Pandey.

EI is designed for generating extremely large ensembles (taking days or weeks to generate) and thus consists of an initial data generation phase tuned for multicore and distributed computing environments. The output is a set of compressed CSV files containing the class distribution produced by each classifier that serves as input to a later ensemble learning phase.

More details of EI can be found in our Biorxiv preprint:

Full citation:

Yan Chak Li, Linhua Wang, Jeffrey N Law, T M Murali, Gaurav Pandey, Integrating multimodal data through interpretable heterogeneous ensembles, Bioinformatics Advances, Volume 2, Issue 1, 2022, vbac065, https://doi.org/10.1093/bioadv/vbac065

This repository is protected by CC BY-NC 4.0.

Configurations

Install Java and groovy.

This can be done using sdkman (https://sdkman.io/).

Install python libraries:

python==3.7.4
scikit-learn==0.22
xgboost==1.2.0
numpy==1.19.5
pandas==0.25.3
argparse==1.1
scipy==1.3.1

Download weka.jar from github/or the link below:

curl -O -L https://prdownloads.sourceforge.net/weka/weka-3-8-5-azul-zulu-linux.zip

Data

Under the data path, 2 files and a list of feature folders are expected:

1. classifiers.txt This file specifies the list of base classifiers. See the sample_data/classifiers.txt as an example.

2. weka.properties This file specifies the list of weka properties that are parsed to the training/testing of base classifiers. See the sample_data/weka.properties as an example.

3. Folders for feature sets This is a list of folders under the main data path. Each of them originally contains only one file named as data.arff. The .arff files are the input feature matrices and labels for training/testing Weka base classifiers. Indices and labels of .arff files should be aligned across all feature sets.

sample_folder of this repository is an example for reference.

Sample data

We uploaded the sample data used in the paper to zenodo.

The compressed zip files PFP.zip contains the input data used for EI.

For PFP, since the raw data is very large (around 2139 * 2GB), we uploaded 5 samples of the GO terms which have been transformed into the format for EI. The remaining terms can be generated by the STRING DB (PFP/STRING_csv) & GO annotation files (GO_annotation.tsv) using generate_data.py

For example, you may generate the input data for predicting GO:0000166 by the following command:

python processing_scripts/generate_data.py --outcome GO:0000166 

Due to IRB constraints, we are currently unable to publicly share the COVID-19 EHR dataset used in our study. However, we shared the model built based on the dataset for application in covid19-model-built.zip which can load by using load_models.py (more detail here).

Evaluate/Model Selection of EI models by nested CV

Train base classifiers

Arguments of train_base.py:

--path, -P: Path of the multimodal data
--queue, -Q: LSF queue to submit the job
--node, -N: number of node requested to HPC
--time, -T: number of hours requested to HPC
--memory, -M: memory requsted in MB to HPC
--classpath, -CP: Path of 'weka.jar' (default:'./weka.jar')
--hpc: use HPC cluster or not
--fold, -F: number of cross-validation fold

Option 1: Without access to Minerva, EI can be run sequentially.

python train_base.py --path [data path] --hpc False

Option 2: Run the pipeline in parallel on Minerva HPC

python train_base.py --path [data path] --node [#node] --queue [queue] --time [hour:min] --memory [memory]

Train and evaluate EI

Arguments of ensemble.py:

--path, -P: Path of the multimodal data
--fold, -F: cross-validation fold

Run the following command:

python ensemble.py --path [data path]

F-max scores of these models will be printed and written in the performance.csv file and saved to the analysis folder under the data path.

The prediction scores by the ensemble methods will be saved in predictions.csv file in analysis folder under the data path.

Model interpretation by EI

Similar to the above step, we will run train_base.py and ensemble.py again, with option --rank True, to train the EI by the whole dataset. All these results will be created in path/model_built folder.

We first generate the local feature ranks (LFR) by the following:

python train_base.py --path [path] --rank True

This step will generate a new folder feature_rank under the data path, which contains a dataset merged with a pseudo test set only for interpretation purposes.

From the path/analysis/performance.csv generated before (--rank=False), we may determine the performance of the ensembles by the Nested-CV setup. We suggest using the best-performing ensemble for EI, eg S.LR, CES, Mean etc. So we may generate the local model rank (LMR) by the following:

python ensemble.py --path [path] --rank True --ens [ensemble algorithm] 

After these two steps for calculating LFR and LMR, we may run the ensemble feature ranking by the following:

python ensemble_ranking.py --path [path] --ens [ensemble algorithm]

Saving and loading EI models

Saving local & ensemble models of EI

We may save both local models and EI models for further inference by setting --writeModel True for both train_base.py and ensemble.py Local models were saved by:

python train_base.py --path [path] --writeModel True

By default, the following command saves all the ensemble models of EI. We may save the specific ensemble model only (e.g. the best-performing ensemble for EI) by specifying --ens option:

python ensemble.py --path [path] --writeModel True --ens [ensemble algorithm, default:all ensemble algorithms] 

Loading local models and make base prediction to new dataset (the model_path would be the path/model_built):

python load_models.py --data_path [new dataset path] --model_path [model path] --local_predictor True

We suggest using the best-performing ensemble for EI (eg S.LR, CES, Mean etc.) known from Nested-CV setup. We can use the saved ensemble model to perform integrative prediction, after obtaining the base prediction of new dataset:

python load_models.py --data_path [new dataset path] --model_path [model path] --ens [ensemble model]

After this step, prediction_scores.csv containing predictions of new dataset is generated in data_path/analysis folder.

More information about the implementation of EI

We used 10 standard binary classification algorithms, such as support vector machine (SVM), random forest (RF) and logistic regression (LR), as implemented in Weka to derive local predictive models from each individual data modality.

Here are the base classifier included in classifier.txt, which are used in train_base.py.

Base Classifier Name	Weka Class Name
AdaBoost	weka.classifiers.meta.AdaBoostM1
Decision Tree	weka.classifiers.trees.J48
Gradient Boosting	weka.classifiers.meta.LogitBoost
K-nearest Neighbors	weka.classifiers.lazy.IBk
Logistic Regression	weka.classifiers.functions.Logistic -M 100
Voted Perceptron	weka.classifiers.functions.VotedPerceptron
Naive Bayes	weka.classifiers.bayes.NaiveBayes
Random Forest	weka.classifiers.trees.RandomForest
Support Vector Machine	weka.classifiers.functions.SMO -C 1.0E-3
Rule-based classification	weka.classifiers.rules.PART

We then applied the mean aggregation, ensemble selection method, and stacking to these local models to generate the final EI model.

Here are the meta-classifiers used in stacking, which are used in ensemble.py.

Meta-classifier Name	Python Class Name	Short Name
AdaBoost	sklearn.ensemble.AdaBoostClassifier	S.AB
Decision Tree	sklearn.tree.DecisionTreeClassifier	S.DT
Gradient Boosting	sklearn.ensemble.GradientBoostingClassifier	S.GB
K-nearest Neighbors	sklearn.neighbors.KNeighborsClassifier	S.KNN
Logistic Regression	sklearn.linear_model.LogisticRegression	S.LR
Naive Bayes	sklearn.naive_bayes.GaussianNB	S.NB
Random Forest	sklearn.ensemble.RandomForestClassifier	S.RF
Support Vector Machine	sklearn.svm.SVC(kernel='linear')	S.SVM
XGBoost	xgboost.XGBClassifier	S.XGB