2024.10

Deployed on October 14th, 2024

• Added update_resource_allocation_redis and companion code, so resource allocations summaries are available for review. Thank you @Gossty!
• Now is possible to have default workflows with only one step.
• qiita_client.update_job_step now accepts an ignore_error optional parameter. Thank you @charles-cowart!
• Initial changes in qiita_client to have more accurate variable names: QIITA_SERVER_CERT -> QIITA_ROOTCA_CERT. Thank you @charles-cowart!
• Added get_artifact_html_summary to qiita_client to retrieve the summary file of an artifact.
• Re-added github actions to https://github.com/qiita-spots/qiita_client.
• SortMeRNA v4.3.7 superseded Sortmerna v2.1b, which relies on Silva 138 and now produced even mates. Thank you @ekopylova and @biocodz for the support.
• Remove SynDNA reads superseded SynDNA Woltka, which now generates even mates.
• Woltka v0.1.7, paired-end superseded Woltka v0.1.6 in qp-woltka; more information. Thank you to @qiyunzhu for the benchmarks!
• Other general fixes, like #3424, #3425, #3439, #3440.
• General SPP improvements, like: NuQC modified to preserve metadata in fastq files, use squeue instead of sacct, , job aborts if Qiita study contains sample metadata columns reserved for prep-infos, metapool generates OverrideCycles value.
• We updated the available parameters for Filter features against reference [filter_features], Non V4 16S sequence assessment [non_v4_16s] and all the phylogenetic analytical commands so they can use Greengenes2 2024.09.

2024.07

Deployed on July 16th, 2024

• On June 14th, 2024 we modified the SPP to use "fastp & minimap2 against GRCh38.p14 + Phi X 174 + T2T-CHM13v2.0, then Movi against GRCh38.p14, T2T-CHM13v2.0 + Human Pangenome Reference Consortium release 2023" to filter human-reads.
• Full refactor of the DB patching system to make sure that a new production deployment has a fully empty database.
• Fully removed Qiimp from Qiita.
• Users can now add ORCID, ResearchGate and/or GoogleScholar information to their profile and the creation (registration) timestamp is kept in the database. Thank you @jlab.
• Admins can now track and purge non-confirmed users from the database via the GUI (/admin/purge_users/). Thank you @jlab.
• Added qiita.slurm_resource_allocations to store general job resource usage, which can be populated by qiita_db.util.update_resource_allocation_table.
• Added qiita_db.util.resource_allocation_plot to generate different models to allocate resources from a given software command based on previous jobs, thank you @Gossty !
• The stats page map can be centered via the configuration file; additionally, the Help and Admin emails are defined also via the configuration files, thank you @jlab !
• Sequel IIe, Revio, and Onso are now valid instruments for the PacBio_SMRT platform.
• Added current_human_filtering to the prep-information and human_reads_filter_method to the artifact to keep track of the method that it was used to human reads filter the raw artifact and know if it's up to date with what is expected via the best practices.
• Added reprocess_job_id to the prep-information so we keep track if a preparation has been reprocessed with another job.
• Other general fixes, like #3385, #3397, #3399, #3400, #3409, #3410.

2024.02

Deployed on February 27th, 2024

• Default workflows now accept commands with multiple inputs.
• The loading time of the main study page was improved #3350.
• SPP improvements - mainly @charles-cowart, thank you! Errors are now show to the user in the GUI #127, admins can restart jobs #129, adapter-trimmer files now are stored and their sequence counts are part of the prep-info #126, and support for per instrument/data-type configuration #123.
• The internal Sequence Processing Pipeline is now using the https://www.gencodegenes.org human transcripts v44 for Metatranscriptomic data - additional to the human pan-genome reference, with the GRCh38 genome + PhiX and T2T-CHM13v2.0 genome - for human host filtering.
• Added a command to qp-woltka: 'Calculate RNA Copy Counts'.
• Other fixes - mainly by @sjanssen2, thank you!: #3345,#3224, #3357, #3358, #3359, #3362, #3364.

2023.12

Version 2023.12

Deployed on January 8th, 2024

• The sample and preparation information pages will display the timestamp of their last update.
• Added a ProcessingJob.complete_processing_job method to retrieve the job that is completing the current job.
• Added a ProcessingJob.trace method to trace all the jobs of a processing_job.
• Analyses now accept SLURM reservation's via the GUI; this will be helpful for workshops or classes.
• Admins can now add per-user-level SLURM submission parameters via the DB; this is helpful to prioritize wet-lab and admin jobs.
• Workflow definitions can now use sample or preparation information columns/values to differentiate between them.
• Updated the Adapter and host filtering plugin (qp-fastp-minimap2) to v2023.12 addressing a bug in adapter filtering; more information.
• Other fixes: 3334, 3338. Thank you @sjanssen2.
• The internal Sequence Processing Pipeline is now using the human pan-genome reference, together with the GRCh38 genome + PhiX and CHM13 genome for human host filtering.

2023.10

• Added a new notebooks folder to the repository to allocate the resource-allocation code and plotting.
• Allowed dynamic resource allocations for time (before it was only for memory), and allowed complex formulas (via eval).
• Deployed the new resource allocation formulas. This included changes on how the ProcesingJob.shape method calculates its variables, which include: (a) build_analysis_files: only use the biom file (vs. all of them); and (b) the SPP counts the input-file number of lines know the number of samples.
• qp-woltka generates a new file: coverates.tgz, which contains: (a) artifact.cov: the coverages of this artifact, can be used with other coverages to generate combined coverages; (b) coverage_percentage.txt: the total percentage coverage per-genome of this artifact; and (c) coverages folder: the per-sample coverage.
• The SPP now supports replicates within the same lane/run and creates multiple preparations as needed.
• The SPP automatically adds and executes the default pipeline after it is done generating and linking the raw data.
• Other fixes: #3300, metapool #135, metapool #136, qtp-sequencing #47.

2023.06

• EBI-ENA submissions require a sample metadata column: geographic location (country and/or sea), Qiita will automatically rename country to satisfy this requirement.
• The qp-knight-lab-processing was refactored for better performance and testing; mainly contributed by @charles-cowart.
• Admin Sample Validation now uses tube_id; mainly contributed by @CatFish47.
• Registered users cannot start sample deletion jobs within public studies.
• The qp-knight-lab-processing now links the SPP job to the preparation it creates and links the created files to that preparation. Note that no files are being moved to the upload folder.
• Fixed the following issues #3227, #3279, #3227.
• Added ProcessingJob.external_id to status-report messages generated, and added sending an email to our system-admin when a (wet-lab) admin job errors.

2023.05

NOTE: Human-associated shotgun samples have been determined to leak human DNA, even with state of the art human read filtering mechanisms in place. To address this critical privacy concern as soon as possible, we are limiting per-sample FASTQ downloads to admins and study owners for human-associated shotgun samples, and in parallel we are (a) working with EBI-ENA to address submitted data and (b) updating Qiita to only allow download of sequence data that positively matches to a microbial specific database.

• Added "Artifact.has_human" to check if an artifact has samples that can potentially contain human sequences.

2023.03

• Added "Sample Validation" for all Admin levels.
• Tools and Plugins can add preparation information via the endpoint: /qiita_db/prep_template/.
• Users can now access specific preparations via the URL by adding it as a parameter: /study/description/<study_id>?prep_id=<prep_id>.
• The internal Sequence Processing Pipeline now automatically inserts new BLANKs to the sample information file, adds the preparation information file to Qiita, and keeps a record of which preparations were inserted to which studies.
• Added the possibility of processing BIOM artifacts in the processing pipeline.
• Added qiime2.2023.2 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization.
• Added q2-greengeenes2 to the system, specifically actions: filter_features and non_v4_16s. This is the first processing (vs. analysis) plugin added to the system.
• Fixed the diversity pcoa_biplot functionality and added the emperor biplot action.
• The system can now validate QIIME 2 FeatureData[Sequences] via qtp-diversity.
• Fixed the following issues: #3249, #3264, #3245, #3243.

2023.02

• Added "Software and Data Licensing" to the Qiita FAQ page.
• External resources can now add new sample metadata categories to a study via
  the /api/v1/study/ endpoint.
• Added preparation-id to the GUI list of artifacts used in an analysis.
• Added automatic lower-casing to INSDC null values #3246.
• ArtifactHandler now returns the file size and full path of the files available in
  an Artifact. This change had two consequences: (1) the plugins now can control
  their behavior based on the file sizes, and (2) all plugins had to be updated to use this new feature.
• Added qiita_client.artifact_and_preparation_files
  to help plugins filter per_sample_FASTQ based on size and ignore small file sizes.
• Added qiime2.2022.11 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization.
• Added q2-mislabeled and composition's ancom to the qiime2.2022.11 deployment.
• Added support for Amplicon data to the internal Sequence Processing Pipeline.

2022.11

• Adding new Woltka v0.1.4 and WoLr2 - WoLr1 and RS210 (updated version of rep200) are available but not default.
• Update qp-woltka so it runs faster, contribution with @wasade. Add newest improvements for woltka processing so it runs faster.
• Fix qp-qiime2, "filter features" bug that expects a "feature metadata" value from the dropdown.
• Avoid having multiple git version commands running in the system - this in theory should fix some of the slow downs.
• Fix sample_name in per preparation sample information files.
• Now the sequencing artifacts (like per-sample-FASTQ, FASTA, etc) accept log files.
• SortMeRNA via qp-meta will generate processing logs and will be added to their artifacts - note that this will only be available in newer runs AKA you can delete the outputs of past runs and rerun it.
• Added new "Admin Sample Validation" to validate a list of sample names against what's available in a study, contributed by @sarayupai.