2021.01

• Moved the qiita repo from biocore to qiita-spots.
• Created the Qiita portal for the Cancer Microbiome.
• The EBI-ENA code now verifies that the sample information file has a description column; this wasn't previously required because it was automatically prefilled by the QIIME 1 mapping file.
• Now it is possible to download the per preparation sample information file and the sample-preparation summary.
• Added a faster metagenomic/metatranscriptomic adaptor and host removal step based on fastp and minimap2. The previous version, using atropos and bowtie2 for QC host filtering, is now deprecated.
• Added qiime2.2020.11 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization.
• Added WoL tree for phylogenetic analyses (/projects/wol/release/databases/qiime2/phylogeny.qza) with per-genome WoL artifacts.
• Fixed the following issues: #3060, #3049, and #2751.

2020.11

• Deprecated the automatic creation of the per template QIIME1 mapping file. If you want to merge a preparation and a BIOM table you must first create a meta-analysis.
• Added a new autoloaded boolean flag to the Study object so we can control if a study was autoloaded via an automatic EBI-ENA or SRA loading job.
• We stopped adding the CRC32 information from mod_zip, which should remove the warnings about CRC incorrect checksums.
• Removed the show/hide button from an analysis while the analysis was being built to avoid possible confusion.
• A new per-sample, fast, bowtie2 and Woltka plugin for WGS and Metatrascriptomics processing has been added, deprecating the Shogun plugin and moving the rest of the commands to a new qp-meta plugin.
• Added the possibility for plugins to submit and control their own jobs vs. Qiita automatically submitting for them. For specifics of this new functionality, (visit #3040)
• We increased the number of workers in qiita.ucsd.edu for the web interface (from eight to twenty), redbiom (from eight to ten), and the plugin interactions (from eight to twenty). This should speed up responses and improve general performance.
• For the qp-qiime2 plugin, we removed some unsupported alpha rarefaction metrics from the options, following QIIME2 guidelines.
• For the qp-qiime2 plugin, if the user selects a tree but it doesn't exist, it will not try to parse and skip it.

2020.09

Version 2020.09

• Added a new endpoint to inject artifacts to existing preparations or jobs: /qiita_db/artifact/
• Outdated commands with the exact same name than newer commands will be marked as not outdated. This is helpful for cases where the commands haven't changed between version
• Added the add_ebi_accessions to the to_dataframe() method of the information files so it can be used to redbiom. This will allow searching via sample or experiment accessions
• Added the release_validator_job method to ProcessingJob method to easily retrieve the release_validator job of a processing_job
• Re-added STUDY_TYPE to the EBI-ENA submission as they are required but deprecated so just adding as Other
• Added qiime2.2020.08 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization
• Shogun processing using Woltka will now produce 2 extra artifacts: a per genome and per gene artifacts

2020.07

The last "official" release was done back on October 31st, 2017; however, code and deployments to the main Qiita site have been happening around every 2 months since then. For all the changes since the last "official" release visit the CHANGELOG.md.

Version 2020.07

• Added per preparation LIBRARY_STRATEGY and removed the study wide STUDY_TYPE values for EBI-ENA submissions to comply with newer metadata standards
• Changed Ion Torrent to Ion_Torrent as described by EBI-ENA
• Added a VALIDATOR job_type to be able to specify job validator resources
• Added a job.shape method that returns the number of columns, samples and input size of each job based its input artifacts
• Added the possibility of requesting memory resources for a job based on the input size, number of samples and/or columns
• Warnings from commands will only use the message part of the warning/errors (#2898)
• Fixed error when deleting multiple artifacts with summaries and support_files
• Button now will be disabled when submitting a workflow via GUI to avoid double clicking from users
• Jobs will now display their "external job id" to users, in practice their barnacle job id
• Fixed bug that prevented delete of full analyses when the processing tree had multiple paths
• Added initial script for nightly auto-processing of workflows
• Removed legacy future dependencies from Python2.7
• Users can see the available system plugins, their commands and resource allocations: https://qiita.ucsd.edu/software/
• Added qiime2.2020.06 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization
• Shogun v1.0.8 for Metagenomic and Metatrascriptomics processing; this new version includes bowtie2 v2.4.1 as aligner and Web of Life and rep200

October 31st, 2017

In this release:

• Taxonomy information is kept in the BIOM tables after filtering samples.
• User-friendly command and parameter names.
• When uploading sample/prep information, the system prefixes with the study id only those samples that have not been prefixed before.
• A warning for the "Downloads" dropdown has been added to the user is aware that he is going to download a big file.
• When uploading per-sample FASTQ's now the system provides more information about the missing (or extra) files.
• The validation process of per-sample FASTQ's now properly checks for the run prefix, rather than exact filename.
• A button to download the mapping file has been added in the analysis pipeline.
• In the study list, now the artifacts are grouped by processing command and parameters.
• RedBiom results now shows the number of unique samples, rather than the number of samples in all artifacts
• When processing an artifact, it now gets auto-selected when entering in the processing page.
• A bug preventing the Sample Summary page from loading in some cases has been fixed.
• The ID's of the jobs validating the output of a previous job have been added to the parent job status message.
• Fixed a bug that embedded the login page in a div if a query to the system failed.
• Sample Summary and Sample Information is now under the same menu item
• A bug in the bulk study deletion that prevented to delete studies without Sample Information has been fixed
• A log of files uploaded to the sample/prep information pages is shown.
• Update to use QIIME2 2017.9
• Update to use deblur 1.0.3
• Artifact parameters and commands have been normalized to latest command available.

October 13th, 2017 (Bugfix from September 27th)

In this release:

• Redbiom integration
• Removal of the moi/ipython cluster dependency
• UTF8 characters are now correctly checked at upload time (rather than just printable)
• Processing job list is shown as a modal rather than a list on the right
• Download of directories is now available
• User-defined artifact name is now honored.
• New artifacts are named after the command output name
• Admins now have access to analysis to support users
• Bulk study deletion
• Input textbox for renaming artifacts automatically focused
• Add BIOMs directly to the analysis from the study processing pipeline.
• EBI submission bug and error catching fixed.

August 23rd, 2017

Functionality added:

• Non-UTF8 errors in the metadata: when found, the system shows a message changing the offending character by 🐾 and with the (row, column) numbers
• The study list now shows the number of samples present in the prep information of the BIOM table
• The study list now shows the owner of the study
• Artifact creation timestamp is displayed in the artifact page
• Prep summary page now has the values collapsed similarly to the sample summary page
• Studies are not searchable by alias

Issues fixed:

• The warning messages from the metadata template are now correctly reseted when the metadata is updated
• The race condition when completing jobs (e.g. deblur jobs) has been removed. All jobs should be finishing smoothly

Qiita 0.2.0: alpha release

The Qiita development team is pleased to announce the second alpha release of Qiita.

In this release we closed more than 80 issues, some of these are:

• Portals: Within the same hardware you can host as many portals as you want simply by changing the configuration file that is being used. This will allow admins to generate specific presentations of data without having to duplicate the information. For example, http://qiita.microbio.me has all available studies and http://sloan_microbe.microbio.me is a front end only for the Sloan related studies. User accounts will work in any portal within the same system i.e. if you already have an account in the main Qiita deploy you don’t need to create a new account to access the Sloan portal.
• Download analysis biom and mapping file: Now you can download the biom and mapping file of any analysis generated within your account.
• Notification system: Qiita has a new system allowing users to receive messages about the system, possible updates, and improvements.
• Now you can upload and process SFF.gz files. This should represent an improvement on the size of files you need to upload.
• Ability to process sequence data that has already been demultiplexed, note that your sequences need not to include any technical reads (barcodes or primers).
• New documentation hosted as part of every system, see an example here, these documents include information about the file formats, pipelines and use cases that Qiita tries to account for. Make sure to add an issue if there’s information that you think may be missing here.

Miscellaneous

• Transactions: To avoid database collisions all modifications to the DB are now run within transactions, which allow complete rollbacks on any failure, including file creation.

As always we encourage you to participate in the development of Qiita by:

• Reporting a bug or feature request in our issue tracker.
• Submitting a new feature (if you are curious please review our development guidelines).
• Adding your study to the system, which should not only facilitate its analysis but will allow you to combine it with other public studies or even share it with other users.

Qiita 0.1.0: alpha release

We are pleased to announce the first alpha release of Qiita (:feet: try it out now 🐾), thanks to our contributors and active users of the system. We’ve come a long way since our pre-alpha release, the following is a quick summary of the major changes that occurred during this development cycle:

Metadata constraints have now been relaxed

When you upload a sample and a prep template, you are required to include only the minimal information needed for processing. This allows users to quickly get from sequence to preliminary results. As always, we recommend that you standardize your metadata to be compliant with the GSC ontologies where possible. Processing of some tasks is disabled if needed information is not supplied by the user, so pay close attention to messages from the system regarding your metadata. For example, raw sequences cannot be demultiplexed if your prep template does not contain a “barcode” column. Please see a full description here.

Improved analysis pipeline

Finding the samples you want to use in a new analysis is now more intuitive. You can browse your studies in a single view, search by sample or study metadata, and add samples to analyses using a familiar “shopping cart”-like interface (see the animated GIF below):

Revamped study management interface

You can now update metadata columns and values or even add new samples to an existing sample template. We’ve also added the ability to delete existing studies and their different components (processed data, preprocessed data, sample template, etc.).

We have improved study privacy settings

You can now change the privacy status of your study's files on a per-processed data basis (a processed data for a 16S analysis is typically an OTU table). A processed data (and the study, by extension) can have one of four possible states: sandboxed, awaiting approval, private, and public. A sandboxed processed data is only visible to the owner and other users with whom the processed data is shared but can be combined with other data available to those users in analyses as usual. Once the user is satisfied with the state of the processed data, he or she can ask for approval from the Qiita admins (QA) to make it private, in which case the processed data changes to the awaiting approval state. To move to a private state, the QA will check that all metadata is MiMARKs compliant and that it processed correctly. In this state the sequence data can be submitted to EBI. The processed data (and hence the study) becomes public when the user explicitly makes the processed data public.

Miscellaneous

• Based on your feedback we have improved the error messages shown on screen so they are more informative.
• Browser compatibility has been improved.
• Quality control, demultiplexing, and OTU picking are now parameterized so you can select from a variety of parameters. Make sure you let us know if you would like to see new parameters.
• Qiita is powered by QIIME v1.9.0.
• The deployment of Qiita maintained by the Knight lab has now officially moved to the University of California, San Diego. Our new DNS is: http://qiita.microbio.me and you should use this URL if you want to reference the central system.
• As always we encourage you to participate in the development of Qiita by:
  • Reporting a bug or feature request in our issue tracker.
  • Submitting a new feature (if you are curious please review our development guidelines).
  • Adding your study to the system, which should not only facilitate its analysis but will allow you to combine it with other public studies or even share it with other users.

Qiita 0.0.1: pre-alpha release

The qiita-dev-team is delighted to let you know that qiita.colorado.edu is up for public initial testing. The goal of this pre-alpha release is to get feedback from users, and attract new developers.

In the current system, you will be able to create new accounts and manage them. Additionally, you will be able to create studies, add sample and prep templates, and upload raw sequences. Note that the templates must follow the MIMARKS standard, and that we currently only support raw (without pre-demultiplexing) FASTQ sequences as initial input.

Qiita relies on the concept of studies, where each study has one sample template (metadata about the samples), and multiple raw data files (e.g. different sequencing runs) where each run has a different preparation (prep) templates (run information). This structure allows us to support diverse sequencing strategies, like having different preparations (e.g. ITS and 16S on the same samples) in a single sequencing run.

Once a sample and prep template are uploaded and added to the database, you can visualize summaries of each one of them, redownload them, and get a valid QIIME mapping file for further analysis. Additionally, you can download the post-demultiplexing raw sequence files generated by the system. For those studies, where we have preprocessed these sequences using the new tools that will be available in QIIME 1.9.0, more below, you will be able to also download the processed data, also know as the close reference biom table.

A test subset of the Earth Microbiome Project studies (around 100) have been processed using the new pipelines that will be available in QIIME 1.9.0, release due on December 3rd, 2014. Meta-analyses (beta, alpha, and taxonomy analysis of single or combined studies) can be tested on the live system. Note that the demultiplexed sequences for these studies are not available yet, but should be available within the next couple of weeks. Also, the system still does not support automatic creation of processed data (e.g. otu picking), although again this will be done soon.

This new system is capable of automatically submitting pre-processed sequences, for example those produced after quality control and demultiplexing, to the European Bioinformatics Institute. During this pre-alpha phase, we only submit sequences processed within the system and by requesting an admin to do it. The submission is per prep template.

Another new feature in the system is that you can share your study with other registered users. This new feature allows collaboration and addresses cases where the raw files exist at a different site.

As noted in the text qiita.colorado.edu will reflect the current status of our master development branch, meaning that as improvements come along they will be proactively deployed. To ensure no data is lost throughout this development phase we have setup an internal replication system of the database.

If you want to try out Qiita in your own system see our installation notes.

Finally, remember that this is a pre-release, so please be patient and if you have some spare time, we need help both on finding issues and on fixing existing ones. If you find new issues with the live system, have suggestions or possible improvements, add them to the issue tracker. In the next week we will be adding more documentation so local installs and testing is easier.