Skip to content

Commit

Permalink
Merge branch 'main' into adding-mgikit-plugin
Browse files Browse the repository at this point in the history
  • Loading branch information
ziadbkh authored Dec 12, 2024
2 parents aa90ab6 + fa8f476 commit fb15563
Show file tree
Hide file tree
Showing 46 changed files with 1,936 additions and 859 deletions.
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,4 @@ demo
# Other things
.idea
.vscode
docs
test-data
11 changes: 8 additions & 3 deletions .github/RELEASE_CHECKLIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@ This checklist is for my own reference, as I forget the steps every time.

1. Check that everything is up-to-date and ready to go
2. Update version numbers in `pyproject.toml`
3. Generate a new changelog section stub:
3. Generate a new changelog section stub (make sure to export a GitHub token to avoid rate limits):

```bash
export GITHUB_TOKEN=<your token>
python scripts/print_changelog.py
```

Expand All @@ -18,17 +19,20 @@ This checklist is for my own reference, as I forget the steps every time.
python scripts/make_module_docs.py
```

5. Install the package again in the `install` mode:
5. Install the package again in the `install` mode. Make sure to remove the build directory and egg info that might cache outdated metadata such as entry point names:

```bash
rm -rf build/ *.egg-info
pip install .
```

This removes the commit hash from the version number when MultiQC runs.

6. Run using test data

- Check for any command line or javascript errors
- Check version numbers are printed correctly

7. Create new demo reports for the website

- Comment out any config in `~/.multiqc_config.yaml`
Expand All @@ -37,9 +41,10 @@ This checklist is for my own reference, as I forget the steps every time.
mv ~/.multiqc_config.yml ~/.multiqc_config.yml.bkup
```

- Generate reports in the multiqc/website repo.
- Generate reports in the seqeralabs/web repo.

```bash
cd ../seqeralabs/web/packages/website
bash update_examples.sh
```

Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ on:
release:
# Build release
types: [published]
workflow_dispatch:

jobs:
build:
Expand All @@ -35,20 +36,22 @@ jobs:

- name: "Log in to Docker Hub"
uses: docker/login-action@v2
if: github.event_name != 'pull_request'
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: "Login to GitHub Container Registry"
uses: docker/login-action@v2
if: github.event_name != 'pull_request'
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: "Build dev"
uses: docker/build-push-action@v3
if: github.event_name != 'release'
if: github.event_name != 'pull_request' && github.event_name != 'release'
with:
# All available with python:3.X-slim are:
# platforms: linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x
Expand All @@ -62,7 +65,7 @@ jobs:
- name: "Build release"
uses: docker/build-push-action@v3
if: github.event_name == 'release'
if: github.event_name != 'pull_request' && github.event_name == 'release'
with:
# All available with python:3.X-slim are:
# platforms: linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x
Expand Down
91 changes: 91 additions & 0 deletions .github/workflows/seqera_docs_changelog.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
import re
from pathlib import Path
import subprocess
from typing import TypedDict


def extract_first_second_level_section(markdown) -> str:
# Match the first second-level section and capture its content
pattern = r"(##\s+.*?)(?=(\n##\s+|$))"
match = re.search(pattern, markdown, re.DOTALL)

if not match:
raise ValueError("Could not find first second-level section in changelog")

# Return the matched section
return match.group(1).strip()


class ChangelogData(TypedDict):
version: str
url: str
date: str
summary: str
the_rest: str


def extract_latest_version(changelog_content) -> ChangelogData:
"""Extract the latest version section from the changelog."""
# Skip the first header line
last_version_changelog = extract_first_second_level_section(changelog_content)

# Find the first version section and extract the version, url and date
matches = re.search(r"^## \[MultiQC v([\d.]+)\]\((.*?)\) - ([\d-]+)", last_version_changelog)
if not matches:
raise ValueError("Could not find version information in changelog")
version, url, date = matches.groups()

# Remove the first line
last_version_changelog = "\n".join(last_version_changelog.splitlines()[1:]).strip()
# Get data until the third-level header
next_header_index = last_version_changelog.find("###")
if next_header_index == -1:
raise ValueError("Could not find next header in changelog")
summary = last_version_changelog[:next_header_index].strip()
the_rest = last_version_changelog[next_header_index:].strip()

return {"version": version, "date": date, "url": url, "summary": summary, "the_rest": the_rest}


# Read CHANGELOG.md
changelog_path = Path("CHANGELOG.md")
if not changelog_path.exists():
raise FileNotFoundError("CHANGELOG.md not found")

# Create seqeralabs-docs directory if it doesn't exist
docs_dir = Path("seqeralabs-docs")
# Clone seqeralabs-docs repo if it doesn't exist
if not docs_dir.exists():
print("Cloning seqeralabs-docs repository...")
repo_url = "https://github.com/seqeralabs/docs.git"
try:
subprocess.run(["git", "clone", repo_url, str(docs_dir)], check=True)
except subprocess.CalledProcessError as e:
raise RuntimeError(f"Failed to clone repository: {e}")

# Extract latest version
with open(changelog_path) as f:
changelog_data: ChangelogData = extract_latest_version(f.read())

# Create output directory
output_dir = docs_dir / "changelog" / "multiqc"
output_dir.mkdir(parents=True, exist_ok=True)

# Create output file
mdx_content: str = f"""---
title: MultiQC v{changelog_data['version']}
date: {changelog_data['date']}
tags: [multiqc]
---
{changelog_data['summary']}
{{/* truncate */}}
{changelog_data['the_rest']}
"""
with open(output_path := output_dir / f"v{changelog_data['version']}.mdx", "w") as f:
f.write(mdx_content)

# Print version for GitHub Actions
print(f"::set-output name=version::{changelog_data['version']}")
42 changes: 42 additions & 0 deletions .github/workflows/seqera_docs_changelog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: Push changelog to Seqera Docs

on:
release:
types: [published]
workflow_dispatch:

jobs:
update-docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Clone seqeralabs/docs
run: |
git clone https://github.com/seqeralabs/docs.git seqeralabs-docs
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.x"

- name: Convert changelog
id: convert
run: python ${GITHUB_WORKSPACE}/.github/workflows/seqera_docs_changelog.py

- name: Create Pull Request
uses: peter-evans/create-pull-request@v5
with:
token: ${{ secrets.SEQERALABS_DOCS }}
path: seqeralabs-docs
commit-message: "Changelog: MultiQC ${{ steps.convert.outputs.version }}"
title: "Changelog: MultiQC ${{ steps.convert.outputs.version }}"
body: |
This PR adds the changelog for MultiQC ${{ steps.convert.outputs.version }} to the Seqera documentation.
This is an automated PR created from the MultiQC repository.
branch: changelog-multiqc-${{ steps.convert.outputs.version }}
base: master
delete-branch: true
5 changes: 0 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,6 @@ multiqc_config.yaml
multiqc_report.html
multiqc_data/
multiqc_plots/
/examples/
/result
/benchmarks
/tmp
/playground

# CI tooling
.ruff_cache
Expand Down
50 changes: 50 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,55 @@
# MultiQC Version History

## [MultiQC v1.25.2](https://github.com/MultiQC/MultiQC/releases/tag/v1.25.2) - 2024-11-20

Multiple bug fixes and minor updates.

### Feature updates and improvements

- Add natural sort for sample sorting ([#2959](https://github.com/MultiQC/MultiQC/pull/2959))
- Custom content: for `plot_type: image`, support `custom_data` config with section name and description. Fix misleading logging ([#2939](https://github.com/MultiQC/MultiQC/pull/2939))
- Config validation improvements (group messages, cast types, validate column headers) ([#2899](https://github.com/MultiQC/MultiQC/pull/2899))

### Fixes

- Workaround for displaying sample grouping in Safari because of missing `visibility: collapse` ([#2941](https://github.com/MultiQC/MultiQC/pull/2941))
- Fix table CSV export where a title contains a comma ([#2911](https://github.com/MultiQC/MultiQC/pull/2911))
- Showing table in notebooks: respect `col1_header` ([#2914](https://github.com/MultiQC/MultiQC/pull/2914))
- Customizing `custom_table_header_config`: fix docs, support both the old and the new ways ([#2955](https://github.com/MultiQC/MultiQC/pull/2955))
- Table scatter mini-plots: fix rounding and range ([#2956](https://github.com/MultiQC/MultiQC/pull/2956))
- File line block iterator: fix reading long lines that do not fit one block ([#2935](https://github.com/MultiQC/MultiQC/pull/2935))
- Fix `cond_formatting_rules` type hint to avoid validation error ([#2922](https://github.com/MultiQC/MultiQC/pull/2922))
- Fix `config.prepend_dirs` or `-d -dd 1` ([#2913](https://github.com/MultiQC/MultiQC/pull/2913))
- Sample grouping fixes ([#2920](https://github.com/MultiQC/MultiQC/pull/2920)):
- Keep sample name column fix width to avoid jumping
- Fix hiding columns through the modal
- Custom content fixes:
- Avoid showing `section_comment` both for module and section when they have the same ID ([#2954](https://github.com/MultiQC/MultiQC/pull/2954))
- Address issue of sections without search patterns and headers in files ([#2921](https://github.com/MultiQC/MultiQC/pull/2921))
- Fix duplicated custom content sections in the report ([#2921](https://github.com/MultiQC/MultiQC/pull/2921))
- Fix support for `plot_type: violin` ([#2957](https://github.com/MultiQC/MultiQC/pull/2957))

### Module updates

- ngsbits: add submodule samplegender ([#2854](https://github.com/MultiQC/MultiQC/pull/2854))
- nanoq: change lineplots for barplots ([#2934](https://github.com/MultiQC/MultiQC/pull/2934))
- Qualimap: clarify the direction of the transcript in coverage plot ([#2946](https://github.com/MultiQC/MultiQC/pull/2946))
- picard: add table with all metrics to VariantCallingMetrics section ([#2885](https://github.com/MultiQC/MultiQC/pull/2885))
- Nanostat: add general stats columns ([#2961](https://github.com/MultiQC/MultiQC/pull/2961))
- Samtools: add insert size to general stats table ([#2905](https://github.com/MultiQC/MultiQC/pull/2905))

### Module fixes

- bcl2fastq: fix missing `R1_*`/`R2_*` metrics ([#2965](https://github.com/MultiQC/MultiQC/pull/2965))
- Cutadapt: fix for null values from r2 data ([#2936](https://github.com/MultiQC/MultiQC/pull/2936))
- Qualimap: fix parsing ∞ value ([#2937](https://github.com/MultiQC/MultiQC/pull/2937))
- bclconvert: fix undetermined barcodes plot ([#2976](https://github.com/MultiQC/MultiQC/pull/2976))
- featurecounts: fix missing section name and anchor ([#2967](https://github.com/MultiQC/MultiQC/pull/2967))

### Infrastructure

- Pin kaleido to 0.2.1 (new 0.4.1 does not embed a browser and thus not portable) ([#2963](https://github.com/MultiQC/MultiQC/pull/2963))

## [MultiQC v1.25.1](https://github.com/MultiQC/MultiQC/releases/tag/v1.25.1) - 2024-09-30

Python 3.13, bugs fixed, improved sample grouping UI, and handling freezes in containers with incompatible architectures.
Expand Down
2 changes: 1 addition & 1 deletion docs/markdown/custom_content/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -448,4 +448,4 @@ The MultiQC automated testing runs with a
which you can look through for inspiration.

For example, to see a file which generates a table in a report by itself, you can
have a look at `embedded_config/table_headers_mqc.txt` ([link](https://github.com/MultiQC/test-data/blob/main/data/custom_content/embedded_config/table_headers_mqc.txt)).
have a look at `embedded_config/table_headers_txt_mqc.txt` ([link](https://github.com/MultiQC/test-data/blob/main/data/custom_content/embedded_config/table_headers_txt_mqc.txt)).
1 change: 1 addition & 0 deletions docs/markdown/development/plots.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ config = {
"xsuffix": "%", # Suffix for the X-axis values and labels. Parsed from tt_label by default
"tt_label": "{x}: {y:.2f}%", # Customise tooltip label, e.g. '{point.x} base pairs'
"stacking": "relative", # Set to "group" to have category bars side by side
"sort_samples": True, # Sort samples by name
"tt_decimals": 0, # Number of decimal places to use in the tooltip number
"tt_suffix": "", # Suffix to add after tooltip number
"height": 500 # The default height of the plot, in pixels
Expand Down
1 change: 0 additions & 1 deletion docs/markdown/getting_started/quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ Arguably, the easiest way to do this is with Conda
3. Restart your terminal shell.
4. [Configure your conda channels](https://bioconda.github.io/#usage) to work with BioConda:
```bash
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
conda config --set channel_priority strict
Expand Down
10 changes: 9 additions & 1 deletion docs/markdown/modules.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This file is autogenerated. Do not edit the markdown, it will be overwritten.
~~~~~~~~~~~~~~~~~~~~~~~
-->

MultiQC currently has modules to support 153 bioinformatics tools, listed below.
MultiQC currently has modules to support 154 bioinformatics tools, listed below.

Click the tool name to go to the MultiQC documentation for that tool.

Expand Down Expand Up @@ -282,6 +282,14 @@ import MultiqcModules from "@site/src/components/MultiqcModules";
},
},
{ id: "modules/gopeaks", data: { name: "GoPeaks", summary: "Calls peaks in CUT&TAG/CUT&RUN datasets" } },
{
id: "modules/haplocheck",
data: {
name: "Haplocheck",
summary:
"Haplocheck detects in-sample contamination in mtDNA or WGS sequencing studies by analyzing the mitchondrial content",
},
},
{
id: "modules/happy",
data: { name: "hap.py", summary: "Benchmarks variant calls against gold standard truth datasets" },
Expand Down
30 changes: 30 additions & 0 deletions docs/markdown/modules/haplocheck.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
title: Haplocheck
displayed_sidebar: multiqcSidebar
description: >
Haplocheck detects in-sample contamination in mtDNA or WGS sequencing studies by analyzing the mitchondrial content
---

<!--
~~~~~ DO NOT EDIT ~~~~~
This file is autogenerated from the MultiQC module python docstring.
Do not edit the markdown, it will be overwritten.
File path for the source of this content: multiqc/modules/haplocheck/haplocheck.py
~~~~~~~~~~~~~~~~~~~~~~~
-->

:::note
Haplocheck detects in-sample contamination in mtDNA or WGS sequencing studies by analyzing the mitchondrial content

[https://github.com/genepi/haplocheck/](https://github.com/genepi/haplocheck/)
:::

### File search patterns

```yaml
haplocheck:
contents: "\"Sample\"\t\"Contamination Status\"\t\"Contamination Level\"\t\"Distance\"\
\t\"Sample Coverage\""
num_lines: 10
```
13 changes: 0 additions & 13 deletions docs/markdown/reports/customisation.md
Original file line number Diff line number Diff line change
Expand Up @@ -656,19 +656,6 @@ custom_plot_config:
color: "#c3e6c3"
```

As of version 1.8, this also works for customising the config of bargraph categories:

```yaml
custom_plot_config:
bowtie1_alignment:
reads_aligned:
color: "#d84e2f"
multimapped:
color: "#f2e63f"
not_aligned:
color: "#8bbc21"
```

## Customising tables

Much like with the custom plot config above, you can override almost any configuration options for tables.
Expand Down
Loading

0 comments on commit fb15563

Please sign in to comment.