Skip to content

Commit

Permalink
Merge pull request #36 from qbicsoftware/testing
Browse files Browse the repository at this point in the history 
Prepare release 0.1.3
  • Loading branch information
@sven1103
sven1103 authored Jul 2, 2018
2 parents 770efad + beb79e1 commit 56904ce
Show file tree
Hide file tree
Showing 7 changed files with 260 additions and 195 deletions.
135 changes: 114 additions & 21 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
[![Build Status](https://qbic-intranet.am10.uni-tuebingen.de/jenkins/job/qPostMan-development/badge/icon)](https://qbic-intranet.am10.uni-tuebingen.de/jenkins/job/qPostMan-development/)
[![Build Status](https://qbic-intranet.am10.uni-tuebingen.de/jenkins/job/postman-development/badge/icon)](https://qbic-intranet.am10.uni-tuebingen.de/jenkins/job/postman-development/)

# postman

**Current stable version: 0.1.2.3**

# qPostMan
A client software written in Java for dataset downloads from QBiC's data management system openBIS (https://wiki-bsse.ethz.ch/display/bis/Home).

We are making use of the V3 API of openBIS (https://wiki-bsse.ethz.ch/display/openBISDoc1605/openBIS+V3+API) in order to interact with the data management system from command line, in order to provide a quick data retreaval on server or cluster resources, where the download via the qPortal is impractical.

## Download
You can download qPostMan from our repositoy with i.e. `wget`:
You can download postman from our repositoy with i.e. `wget`:
```bash
VERSION=0.1.2.3
wget https://qbic-repo.am10.uni-tuebingen.de/repository/maven-releases/life/qbic/qpostman/$VERSION/qpostman-$VERSION-jar-with-dependencies.jar
wget https://qbic-repo.am10.uni-tuebingen.de/repository/maven-releases/life/qbic/qpostman/$VERSION/qpostman-$VERSION-jar-with-dependencies.jar.md5
wget https://qbic-repo.am10.uni-tuebingen.de/repository/maven-releases/life/qbic/postman/$VERSION/postman-$VERSION-jar-with-dependencies.jar
wget https://qbic-repo.am10.uni-tuebingen.de/repository/maven-releases/life/qbic/postman/$VERSION/postman-$VERSION-jar-with-dependencies.jar.md5
```
Please compare the md5 checksum after the download.

Expand All @@ -19,30 +22,32 @@ You need to have **Java JRE** or **JDK** installed (**openJDK** is fine), at lea

## Usage
### Options
Just execute qPostMan with `java -jar qpostman.jar` or `java -jar qpostman.jar -h` to get an overview of the options:
Just execute postman with `java -jar postman.jar` or `java -jar postman.jar -h` to get an overview of the options:
```bash

~$ java -jar qpostman.jar
usage: qPostMan [-b <arg>] [-f <arg>] [-h] [-i <arg>] [-u <arg>]
-b,--buffer-size <arg> A integer muliple of 1024 bytes (default). Only
change this if you know what you are doing.
-f,--file <arg> File containing openBis sample IDs (one per
line) [mutually exclusive]
-h,--help Print this help
-i,--identifier <arg> openBis sample ID [mutually exclusive]
-u,--user-name <arg> openBIS user name
~$ java -jar postman.jar
Usage: <main class> [-h] [-b=<bufferMultiplier>] [-f=<filePath>]
[-t=<datasetType>] -u=<user> [SAMPLE_ID]...
[SAMPLE_ID]... one or more QBiC sample ids
-b, --buffer-size=<bufferMultiplier>
a integer muliple of 1024 bytes (default). Only
change this if you know what you are doing.
-f, --file=<filePath> a file with line-separated list of QBiC sample ids
-h, --help display a help message
-t, --type=<datasetType> filter for a given openBIS dataset type
-u, --user=<user> openBIS user name
```
### Provide a QBiC ID
The simplest scenario is, that you want to download a dataset/datasets from a sample. Just provide the QBiC ID for that sample and your username (same as the one you use for the qPortal):
```bash
~$ java -jar qpostman.jar -i <QBiC ID> -u <your_qbic_username>
~$ java -jar postman.jar -i <QBiC ID> -u <your_qbic_username>
```
qPostMan will prompt you for your password, which is the password from your QBiC user account.
postman will prompt you for your password, which is the password from your QBiC user account.
After you have provided your password and authenticate successfully, qPostMan tries to download all datasets that are registered for that given sample ID and downloads them to the current working directory:
After you have provided your password and authenticate successfully, postman tries to download all datasets that are registered for that given sample ID and downloads them to the current working directory:
```bash
[bbbfs01@u-003-ncmu03 ~]$ qpostman-0.1.2.1 -i QMFKD003AG -u bbbfs01
[bbbfs01@u-003-ncmu03 ~]$ java -jar postman.jar -u bbbfs01 QMFKD003AG
Provide password for user 'bbbfs01':

12:32:02.038 [main] INFO life.qbic.App - OpenBis login returned with 0
Expand All @@ -54,13 +59,101 @@ Provide password for user 'bbbfs01':
QMFKD003AG_SRR099967_1.filt.fastq.gz [### ] 0.38/7.94 Gb
```
### Filter for dataset type
You can filter for dataset types, using the `-t` option and one of the following openBIS dataset types we are currently using:
```bash
ARR
AUDIT
CEL
CSV
EXPERIMENTAL_DESIGN
EXPRESSION_MATRIX
FASTQ
FEATUREXML
GZ
IDXML
JPG
MAT
MZML
PDF
PNG
Q_BMI_IMAGING_DATA
Q_DOCUMENT
Q_EXT_MS_QUALITYCONTROL_RESULTS
Q_EXT_NGS_QUALITYCONTROL_RESULTS
Q_FASTA_DATA
Q_HT_QPCR_DATA
Q_MA_AGILENT_DATA
Q_MA_CHIP_IMAGE
Q_MA_RAW_DATA
Q_MS_MZML_DATA
Q_MS_RAW_DATA
Q_MTB_ARCHIVE
Q_NGS_HLATYPING_DATA
Q_NGS_IMMUNE_MONITORING_DATA
Q_NGS_IONTORRENT_DATA
Q_NGS_MAPPING_DATA
Q_NGS_MTB_DATA
Q_NGS_RAW_DATA
Q_NGS_READ_MATCH_ARCHIVE
Q_NGS_VARIANT_CALLING_DATA
Q_PEPTIDE_DATA
Q_PROJECT_DATA
Q_TEST
Q_VACCINE_CONSTRUCT_DATA
Q_WF_EDDA_BENCHMARK_LOGS
Q_WF_EDDA_BENCHMARK_RESULTS
Q_WF_MA_QUALITYCONTROL_LOGS
Q_WF_MA_QUALITYCONTROL_RESULTS
Q_WF_MS_INDIVIDUALIZED_PROTEOME_LOGS
Q_WF_MS_INDIVIDUALIZED_PROTEOME_RESULTS
Q_WF_MS_LIGANDOMICS_ID_LOGS
Q_WF_MS_LIGANDOMICS_ID_RESULTS
Q_WF_MS_LIGANDOMICS_QC_LOGS
Q_WF_MS_LIGANDOMICS_QC_RESULTS
Q_WF_MS_MAXQUANT_LOGS
Q_WF_MS_MAXQUANT_ORIGINAL_OUT
Q_WF_MS_MAXQUANT_RESULTS
Q_WF_MS_PEAKPICKING_LOGS
Q_WF_MS_PEPTIDEID_LOGS
Q_WF_MS_PEPTIDEID_RESULTS
Q_WF_MS_QUALITYCONTROL_LOGS
Q_WF_MS_QUALITYCONTROL_RESULTS
Q_WF_NGS_16S_TAXONOMIC_PROFILING_LOGS
Q_WF_NGS_EPITOPE_PREDICTION_LOGS
Q_WF_NGS_EPITOPE_PREDICTION_RESULTS
Q_WF_NGS_HLATYPING_LOGS
Q_WF_NGS_HLATYPING_RESULTS
Q_WF_NGS_MAPPING_LOGS
Q_WF_NGS_MAPPING_RESULTS
Q_WF_NGS_QUALITYCONTROL_LOGS
Q_WF_NGS_QUALITYCONTROL_RESULTS
Q_WF_NGS_RNAEXPRESSIONANALYSIS_LOGS
Q_WF_NGS_RNAEXPRESSIONANALYSIS_RESULTS
Q_WF_NGS_SHRNA_COUNTING_LOGS
Q_WF_NGS_SHRNA_COUNTING_RESULTS
Q_WF_NGS_VARIANT_ANNOTATION_LOGS
Q_WF_NGS_VARIANT_ANNOTATION_RESULTS
Q_WF_NGS_VARIANT_CALLING_LOGS
Q_WF_NGS_VARIANT_CALLING_RESULTS
RAW
SHA256SUM
TAR
UNKNOWN
VCF
```
### Provide a file with several QBiC IDs
In order to download datasets from several samples at once, you can provide a simple text file with multiple, line-separated, QBiC IDs and hand it to qPostMan with the `-i` option.
In order to download datasets from several samples at once, you can provide a simple text file with multiple, line-separated, QBiC IDs and hand it to postman with the `-f` option.
qPostMan will automatically iterate over the IDs and try to download them.
postman will automatically iterate over the IDs and try to download them.
### Performance issues
We discovered, that a default buffer size of 1024 bytes seems not always to get all out of the performance that is possible for the dataset download. Therefore, we allow you to enter a multipler Integer value that increases the buffer size. For example a multipler of 2 will result in 2x1024 = 2048 bytes and so on.
Just use the `-b` option for that. The default buffer size remains 1024 bytes, if you don't specify this value.
136 changes: 75 additions & 61 deletions pom.xml
View file
Original file line number Diff line number Diff line change
@@ -1,48 +1,61 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>life.qbic</groupId>
<artifactId>qpostman</artifactId>
<groupId>life.qbic.cli</groupId>
<artifactId>postman-cli</artifactId>
<packaging>jar</packaging>
<version>0.1.2.3</version>
<name>qpostman</name>
<version>0.1.3</version>
<name>postman</name>
<url>http://maven.apache.org</url>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.maven.plugins/maven-dependency-plugin -->
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.1.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>life.qbic.openbis</groupId>
<artifactId>openbis_api</artifactId>
<version>3-S253.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.5</version>
</dependency>

<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>life.qbic.openbis</groupId>
<artifactId>openbis_api</artifactId>
<version>3-S253.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/commons-cli/commons-cli -->
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.4</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-core -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.9.1</version>
</dependency>
<dependency>
<groupId>jline</groupId>
<artifactId>jline</artifactId>
<version>2.14.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-core -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.9.1</version>
</dependency>
<dependency>
<groupId>jline</groupId>
<artifactId>jline</artifactId>
<version>2.14.6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/info.picocli/picocli -->
<dependency>
<groupId>info.picocli</groupId>
<artifactId>picocli</artifactId>
<version>2.3.0</version>
</dependency>
</dependencies>

</dependencies>
<build>
<plugins>
<plugin>
Expand All @@ -56,36 +69,37 @@
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<appendAssemblyId>false</appendAssemblyId>
</configuration>
</plugin>
</plugins>
</build>

<repositories>
<repository>
<id>nexus-snap</id>
<name>qbic snapshots</name>
<url>https://qbic-repo.am10.uni-tuebingen.de/repository/maven-snapshots</url>
</repository>
<repository>
<id>nexus-release</id>
<name>qbic release</name>
<url>https://qbic-repo.am10.uni-tuebingen.de/repository/maven-releases</url>
</repository>
</repositories>
<distributionManagement>
<repository>
<uniqueVersion>true</uniqueVersion>
<id>nexus-release</id>
<name>QBiC Releases</name>
<url>https://qbic-repo.am10.uni-tuebingen.de/repository/maven-releases</url>
</repository>
<snapshotRepository>
<uniqueVersion>false</uniqueVersion>
<id>nexus-snap</id>
<name>QBiC Snapshots</name>
<url>https://qbic-repo.am10.uni-tuebingen.de/repository/maven-snapshots</url>
</snapshotRepository>
<repositories>
<repository>
<id>nexus-snap</id>
<name>qbic snapshots</name>
<url>https://qbic-repo.am10.uni-tuebingen.de/repository/maven-snapshots</url>
</repository>
<repository>
<id>nexus-release</id>
<name>qbic release</name>
<url>https://qbic-repo.am10.uni-tuebingen.de/repository/maven-releases</url>
</repository>
</repositories>
<distributionManagement>
<repository>
<uniqueVersion>true</uniqueVersion>
<id>nexus-release</id>
<name>QBiC Releases</name>
<url>https://qbic-repo.am10.uni-tuebingen.de/repository/maven-releases</url>
</repository>
<snapshotRepository>
<uniqueVersion>false</uniqueVersion>
<id>nexus-snap</id>
<name>QBiC Snapshots</name>
<url>https://qbic-repo.am10.uni-tuebingen.de/repository/maven-snapshots</url>
</snapshotRepository>

</distributionManagement>
</project>
</distributionManagement>
</project>
Loading

0 comments on commit 56904ce

Please sign in to comment.