ProteomicsLFQ memory consumed

[daichengxin]

Description of the bug

The errors are reported when i ran PXD001819 LFQ datasets (about 10G mzML files). It looks like it's running out of memory? But the available memory is 120G. So I'm not sure if this is normal or not.

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Command used and terminal output

ProteomicsLFQ       -threads 2       -in UPS1_12500amol_R1.mzML UPS1_12500amol_R2.mzML UPS1_12500amol_R3.mzML UPS1_125amol_R1.mzML UPS1_125amol_R2.mzML UPS1_125amol_R3.mzML UPS1_25000amol_R1.mzML UPS1_25000amol_R2.mzML UPS1_25000amol_R3.mzML UPS1_2500amol_R1.mzML UPS1_2500amol_R2.mzML UPS1_2500amol_R3.mzML UPS1_250amol_R1.mzML UPS1_250amol_R2.mzML UPS1_250amol_R3.mzML UPS1_50000amol_R1.mzML UPS1_50000amol_R2.mzML UPS1_50000amol_R3.mzML UPS1_5000amol_R1.mzML UPS1_5000amol_R2.mzML UPS1_5000amol_R3.mzML UPS1_500amol_R1.mzML UPS1_500amol_R2.mzML UPS1_500amol_R3.mzML UPS1_50amol_R1.mzML UPS1_50amol_R2.mzML UPS1_50amol_R3.mzML       -ids UPS1_12500amol_R1_comet_feat_perc_pep_filter.idXML UPS1_12500amol_R2_comet_feat_perc_pep_filter.idXML UPS1_12500amol_R3_comet_feat_perc_pep_filter.idXML UPS1_125amol_R1_comet_feat_perc_pep_filter.idXML UPS1_125amol_R2_comet_feat_perc_pep_filter.idXML UPS1_125amol_R3_comet_feat_perc_pep_filter.idXML UPS1_25000amol_R1_comet_feat_perc_pep_filter.idXML UPS1_25000amol_R2_comet_feat_perc_pep_filter.idXML UPS1_25000amol_R3_comet_feat_perc_pep_filter.idXML UPS1_2500amol_R1_comet_feat_perc_pep_filter.idXML UPS1_2500amol_R2_comet_feat_perc_pep_filter.idXML UPS1_2500amol_R3_comet_feat_perc_pep_filter.idXML UPS1_250amol_R1_comet_feat_perc_pep_filter.idXML UPS1_250amol_R2_comet_feat_perc_pep_filter.idXML UPS1_250amol_R3_comet_feat_perc_pep_filter.idXML UPS1_50000amol_R1_comet_feat_perc_pep_filter.idXML UPS1_50000amol_R2_comet_feat_perc_pep_filter.idXML UPS1_50000amol_R3_comet_feat_perc_pep_filter.idXML UPS1_5000amol_R1_comet_feat_perc_pep_filter.idXML UPS1_5000amol_R2_comet_feat_perc_pep_filter.idXML UPS1_5000amol_R3_comet_feat_perc_pep_filter.idXML UPS1_500amol_R1_comet_feat_perc_pep_filter.idXML   UPS1_500amol_R2_comet_feat_perc_pep_filter.idXML UPS1_500amol_R3_comet_feat_perc_pep_filter.idXML UPS1_50amol_R1_comet_feat_perc_pep_filter.idXML UPS1_50amol_R2_comet_feat_perc_pep_filter.idXML UPS1_50amol_R3_comet_feat_perc_pep_filter.idXML       -design PXD001819.sdrf_openms_design.tsv       -fasta uniprot_yeast_ups_decoy.fasta       -protein_inference aggregation       -quantification_method feature_intensity       -targeted_only false       -feature_with_id_min_score 0.10       -feature_without_id_min_score 0.75       -mass_recalibration false       -Seeding:intThreshold 1000       -protein_quantification unique_peptides       -alignment_order star              -psmFDR 0.01       -proteinFDR 0.01       -picked_proteinFDR true       -out_cxml PXD001819.sdrf_openms_design_openms.consensusXML       -out PXD001819.sdrf_openms_design_openms.mzTab       -out_msstats PXD001819.sdrf_openms_design_msstats_in.csv    -PeptideQuantification:extract:batch_size 1000          -debug 0       2>&1 | tee proteomicslfq.log

Relevant files

log file: proteomicslfq.log

System information

quantms 1.3.0

[jpfeuffer]

I have analysed this dataset many times. Never had issues.

[jpfeuffer]

You can trace the memory consumption while it's running.

[ypriverol]

Me too, but the last release of quantms 1.3.0 uses OpenMS 3.2.0. We have other ongoing problems with this version and mzTab export in ProteinQuantifier. @timosachsenberg Can you help us here?

[timosachsenberg]

log does not indicate that it is the export. is there a way we can find out where/when this regression was introduced?

[daichengxin]

I retried openms 3.2.0 and traced the memory. It does exceed the memory. Why is it out of memory? mzML only has 10G. I haven't encountered this before either

[timosachsenberg]

Thanks for checking. This it is really suspicious. Can you reproduce this e.g., for one or two files?

[daichengxin]

I can reproduce this in two files. But a single file are work. Test files: https://www.dropbox.com/scl/fi/jgbw0pvnm18cga1kwgy54/proteomicslfq.zip?rlkey=6igoyec9ffztk9p8f4uriukct&st=osldx9cp&dl=0

[timosachsenberg]

I can confirm that it uses 400gb for two small files during feature extraction.
My first guess would be that something inside e.g. the OpenSWATH code might have changed.

[ypriverol]

These are the two files in PRIDE:

https://ftp.pride.ebi.ac.uk/pride/data/archive/2015/12/PXD001819/UPS1_125amol_R1.raw
https://ftp.pride.ebi.ac.uk/pride/data/archive/2015/12/PXD001819/UPS1_125amol_R2.raw

[timosachsenberg]

Likely related to a different conversion using TRFP.
The file works with ProteoWizard msconvert

[timosachsenberg]

is this fixed?

[ypriverol]

We reverted to the previous version of ThermoRawfileparser which is not ideal. We will explore more in detail with the new version of ThermoRawFileParser.

To be honest I have lost track of this.