This repository contains the implementation of RankingSHAP, discussed in our paper titled "RankingSHAP – Listwise Feature Attribution Explanations for Ranking Models". With this code we demonstrate the ability of RankingSHAP to generate feature attribution explanations for ranking models. We compare to several baselines, showing that the only approach that can accurately approximate feature attribution, as defined by marginal contributions and shapley values for the ranking application is RankingSHAP.
A list of dependencies can be found in the requirements.txt file. The code is tested on python 3.11.
First clone this repository to your local machine:
git clone [email protected]:MariaHeuss/RankingShap.git
Navigate to the cloned repository directory:
cd RankingShap
Create a virtual environment on your machine and install the dependencies
pip install -r requirements.txt
Before running the scripts, ensure you have collected all the necessary data. For our paper we have tested two datasets from LETOR4.0. MQ2008 and MSLR-WEB10K. You can download those datasets by following these steps:
- Create a folder data
- Download the dataset from
https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval/letor-4-0/. Follow the provided link to OneDrive and download the file namedMQ2008.rar. - Unpack the data and place the dataset in a folder called MQ2008 in the data directory within the project.
- The folder structure should be for example
data/MQ2008/Fold1/test.txt.
In the same way download the MSLR-WEB10K data from https://www.microsoft.com/en-us/research/project/mslr/ and store it
in a folder called MSLR-WEB10K in the data folder.
To run the main scripts of the project, execute the following commands in the terminal:
run_rankingshap.bash
This script will execute a number of scripts that sequentially train a model, generate explanations, generate ground truth attribution labels and evaluate the explanations with those for the MQ2008 data. You can test the code first by running
run_rankingshap_test.bash
which executes the same code but for only one query.
If you use this code or our results in your research, please cite our paper as follows:
@article{heuss2024,
title={RankingSHAP – Listwise Feature Attribution Explanations
for Ranking Models},
author={Heuss,Maria and deRijke, Maarten and Anand, Avishek},
year={2024}
}
This repository is published under the terms of the GNU General Public License version 3. For more information, see the file LICENSE.
RankingSHAP – Listwise Feature Attribution Explanations
for Ranking Models
Copyright (C) 2024 Maria Heuss
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>
For any queries, please reach out to m dot c dot heuss at uva dot nl.