Protein function prediction with GO - Part 3

[aditya0by0]

• PR for the Issue Protein function prediction with GO #36

Note: The above issue will be implemented in 3 PRs:

• Protein function prediction with GO #39 (Merged)
• Protein function prediction with GO - Part 2 #57 (Merged)
• Protein function prediction with GO - Part 3 #64

Changes to be done in this PR

evaluation: Evaluate using the same metrics as DeepGO for comparing the models

From comment #36 (comment)

• on a new branch: metrics for evaluation (I talked to Martin about the Fmax score: Although it has some methodological issues, we should include it in our evaluation to do a comparison with DeepGO)
• DeepGO-SE (paper): use these results as a baseline, integrate their data into our pipeline (there is a link to the dataset on their github page

[aditya0by0]

I have made the suggested changes for migration. Please check.

Config for DeepGO1:

class_path: chebai.preprocessing.datasets.go_uniprot.DeepGO1MigratedData
init_args:
  go_branch: "MF"
  max_sequence_length: 1002
  reader_kwargs: {n_gram: 3}

Config for DeepGO2:

class_path: chebai.preprocessing.datasets.go_uniprot.DeepGO2MigratedData
init_args:
  go_branch: "MF"
  max_sequence_length: 1000
  reader_kwargs: {n_gram: 3}

[sfluegel05]

I ran the migration script for DeepGO2 and tried training a model with the data. I noticed three issues:

• The max_sequence_length parameter seems to have no effect (I set it to 1,000, but the processed dataset contains feature-entries that have a length of more than 4,000)
• The labels are empty lists in the processed data (although a classes_deep_go2.txt file does get generated)
• The dataset contains "invalid" amino acids (O and U for pyrrolysine and selenocysteine) - as you pointed out, DeepGO does not include them as valid, but as far as I can tell, that is not enforced. Either way, currently, when processing the migrated data, chebai raises an error and stops when reaching either an O or U in the dataset. This is not very useful. We should either raise a warning and skip the protein or include O and U as valid (I would suggest to do both, i.e., adding O and U as safe and raising a warning for other letters that are definitely invalid).

@aditya0by0 Can you have a look at that?

Edit: I used the following commands:
Migration:

python chebai/preprocessing/migration/deep_go_migrate_deep_go_2_data.py migrate --data_dir=data/deepgo2-train-data --go_branch=mf

Run:

python -m fit --trainer=configs/training/default_trainer.yml --trainer.min_epochs=10 --trainer.max_epochs=10 --model=configs/model/electra.yml --model.train_metrics=configs/metrics/micro-macro-f1.yml --model.test_metrics=configs/metrics/micro-macro-f1.yml --model.val_metrics=configs/metrics/micro-macro-f1.yml --data=configs/data/deepGO2.yml --trainer.logger=configs/training/csv_logger.yml --model.out_dim=898 --model.criterion=configs/loss/bce.yml --data.init_args.batch_size=10 --data.init_args.num_workers=10 --model.pass_loss_kwargs=false --trainer.logger.init_args.name=DeepGO2_MF --data.init_args.splits_file_path=data/GO_UniProt/GO_MF_1000/processed/splits_deep_go2.csv

[aditya0by0]

• DeepGO does not include them as valid, but as far as I can tell, that is not enforced.

• We should either raise a warning and skip the protein or include O and U as valid (I would suggest to do both, i.e., adding O and U as safe and raising a warning for other letters that are definitely invalid).

I agree that invalid amino acids like "O" and "U" are not explicitly enforced as invalid by DeepGO2, but they are also not explicitly treated as valid in their data pipeline.

To elaborate, DeepGO2 includes the amino acid notation "X" in its set of valid amino acids, where "X" represents any/unknown amino acids (as per [Wikipedia].

In their pipeline, when invalid amino acids such as "U", "O", "B", "Z", "J", or "*" are encountered in the protein sequences, they are effectively mapped to "X". This is evident in their implementation, where the index of "X" is used for any amino acid that doesn't belong to the valid set.

You can see this behavior in their code here:
[DeepGO2 Amino Acid relevant code].

So, while "O" and "U" are not explicitly handled, the use of "X" as a catch-all ensures that any invalid amino acids are safely represented.

If we want to follow the approach mentioned above, we would need to replace every invalid amino acid in the sequence with "X" as part of a pre-processing step before tokenization. This to avoid inconsistencies in the n-gram tokenization process.

Please let me know how we want to proceed with it.

[sfluegel05]

Ok, so if DeepGO2 replaces every not explicitly valid amino acid with X, then we should do the same. I think that is the easiest solution.