V2

.github/workflows/ci.yaml

@@ -16,12 +16,12 @@ jobs:
     strategy:
       matrix:
         os: [ ubuntu-latest ]
-        python-version: [ 3.8, 3.9, "3.10", 3.11 ]
+        python-version: [ 3.9, "3.10", 3.11 ]
     steps:
       - uses: actions/checkout@v3
       - uses: mamba-org/setup-micromamba@v1
         with:
-          micromamba-version: '1.4.5-0'
+          micromamba-version: '1.5.8-0'
           environment-name: test-env
           condarc: |
             channels:
@@ -30,9 +30,15 @@ jobs:
               - defaults
           create-args: >-
             python=${{ matrix.python-version }}
-            snakemake
-            mamba
+            sourmash 
             pandas
+            numpy
+            intervaltree
+            mummer=3.23
+            glpk=5.0
+            snakemake=7.32.4
+            plasnet=0.6.0
+            dingII
           init-shell: bash
           cache-environment: true
           post-cleanup: 'all'
@@ -45,4 +51,5 @@ jobs:
         shell: bash -el {0}
         run: |
           micromamba activate test-env
+          python -m pip install .
           PYTHONPATH="." python -m unittest discover -s tests -t .

.readthedocs.yaml

@@ -0,0 +1,32 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.12"
+    # You can also specify other tool versions:
+    # nodejs: "19"
+    # rust: "1.64"
+    # golang: "1.19"
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+# Optionally build your docs in additional formats such as PDF and ePub
+# formats:
+#    - pdf
+#    - epub
+
+# Optional but recommended, declare the Python requirements required
+# to build your documentation
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python:
+  install:
+   - requirements: docs/requirements.txt

MANIFEST.in

@@ -0,0 +1,6 @@
+include pling/resources.tsv
+include pling/align_snakemake/Snakefile
+include pling/batching/Snakefile
+include pling/dcj_snakemake/Snakefile
+include pling/common_rules/common_rules.smk
+include pling/jac_network_snakemake/Snakefile

README.rst

@@ -0,0 +1,5 @@
+Pling!
+======
+Pling is a software workflow for plasmid analysis using rearrangement distances, specifically the Double Cut and Join Indel (DCJ-Indel) distance. By intelligently combining containment distance (shared content as fraction of the smaller) and DCJ-indel distance ("how far apart evolutionarily" in a structural sense), and by preventing shared mobile elements from clouding the issue, it infers clusters of related plasmids.
+
+For installation instructions and documentation, please go `here <https://pling.readthedocs.io>`_.

docs/basic.rst

@@ -0,0 +1,16 @@
+Basic Usage
+===========
+
+Required input is a text file of a list of paths to fasta files ``genomes_list`` and a path to an output directory ``output_dir``. All the genomes must be circular and complete. If you have all your genomes in one directory, you can navigate to that directory and generate ``genomes_list`` by running
+
+.. code-block:: console
+
+    ls -d -1 $PWD/*.fasta > input.txt
+
+Then usage is
+
+.. code-block:: console
+
+    pling input.txt output_dir align
+
+

docs/citation.rst

@@ -0,0 +1,4 @@
+Citation
+========
+
+Preprint: https://doi.org/10.1101/2024.06.12.598623 

docs/conf.py

@@ -0,0 +1,64 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = "pling"
+copyright = "2024, Daria Frolova"
+author = "Daria Frolova"
+
+release = '2.0'
+version = '2.0.0'
+
+# -- General configuration ---------------------------------------------------
+# -- General configuration
+
+extensions = [
+    "sphinx.ext.duration",
+    "sphinx.ext.doctest",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.intersphinx",
+]
+
+intersphinx_mapping = {
+    "rtd": ("https://docs.readthedocs.io/en/stable/", None),
+    "python": ("https://docs.python.org/3/", None),
+    "sphinx": ("https://www.sphinx-doc.org/en/master/", None),
+}
+intersphinx_disabled_domains = ["std"]
+
+templates_path = ["_templates"]
+
+# -- Options for EPUB output
+epub_show_urls = "footnote"
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = "sphinx_book_theme"
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["_static"]

docs/index.rst

@@ -0,0 +1,14 @@
+Pling!
+======
+
+Pling is a software workflow for plasmid analysis using rearrangement distances, specifically the Double Cut and Join Indel (DCJ-Indel) distance. By intelligently combining containment distance (shared content as fraction of the smaller) and DCJ-indel distance ("how far apart evolutionarily" in a structural sense), and by preventing shared mobile elements from clouding the issue, it infers clusters of related plasmids.
+
+.. toctree::
+
+   installation.rst
+   basic.rst
+   tutorial.rst
+   output.rst
+   advanced.rst
+   tips.rst
+   citation.rst

docs/installation.rst

@@ -0,0 +1,37 @@
+Installation
+============
+
+Conda
+-----
+
+To install with conda, just run::
+
+	conda config --add channels bioconda
+	conda config --add channels conda-forge
+	conda install pling
+
+
+From source
+-----------
+
+Start with::
+
+	git clone https://github.com/iqbal-lab-org/pling.git && cd pling
+
+From here you can install dependancies using conda with the ``env.yaml`` file, or whichever method you prefer. To install dependancies with conda, do::
+
+	conda env create -f env.yaml
+	conda activate pling
+
+and then finally install pling with::
+
+	python -m pip install .
+
+
+Installing optional dependancies
+--------------------------------
+
+By default, the DCJ-Indel distances are calculated with the open source solver GLPK. Optionally, pling can use Gurobi for its calculations, but this dependancy is not included in the usual pling installation. This is because Gurobi is a commercial software which requires a license, which is free for academic purposes. Information on obtaining this license can be found here: https://www.gurobi.com/academia/academic-program-and-licenses/. Gurobi can then be installed via conda as follows::
+
+	conda config --add channels https://conda.anaconda.org/gurobi
+	conda install gurobi=10.0.1

docs/output.rst

@@ -0,0 +1,11 @@
+Description and Output
+======================
+
+Pling runs via the python script `run_pling.py`, which creates a config file and runs three snakemake workflows in succession. It starts by calculating containment distances and transforming nucleotide sequences into integer sequences (integerisation, details below), then calculates DCJ-Indel distances, and finally uses these to build a network and cluster on it. The outputs are:
+
+- **Containment communities:** Pling defines broad plasmid communities by building a containment network. In this network, nodes represent plasmids, and an edge is drawn if two plasmids fulfil the containment distance threshold. If plasmid *A* is smaller than plasmid *B*, then the containment distance between the two plasmids is the percentage of plasmid *A* that is *not* contained in plasmid *B*. A plasmid community corresponds exactly to a connected component in this network. Distances and plasmid to community assignments can be found in the folder `containment`.
+- **Hub plasmids:** Pling identifies "hub plasmids", which are plasmids that are densely connected on the containment network, but their neighbours are not interconnected. In practice, these are usually relatively small plasmids that consist mostly of a large mobile genetic element, which has spread across many diverse, unrelated plasmids. They are listed in ``dcj_thresh_4_graph/objects/hub_plasmids.csv``
+- **Integer sequences:** Pling outputs the integer sequences used to calculate DCJ-Indel distances. These are outputted in UniMoG-format (see https://bibiserv.cebitec.uni-bielefeld.de/dcj?id=dcj_manual for description), and also a mapping of integer to sequence coordinates, or a mapping of integer to gene name, is provided. The integer sequences are calculated in batches, so there is a unimog and a map file per batch, all found in the folder ``unimogs``.
+- **DCJ-Indel subcommunities:** Pling identifies plasmid subcommunities by constructing a DCJ-Indel network, and then clustering on this network. The DCJ-Indel network is a subnetwork of the containment network initially built. Edges are kept if a pair of plasmids fulfil the DCJ-Indel distance threshold. Hub plasmids are isolated in the DCJ-Indel network, with no edges connecting to them (even if they fulfil the DCJ-Indel threshold). On this network pling clusters using asynchronous label propagation community detection algorithm. Plasmid clusters are labelled by containment community and DCJ-Indel subcommunity, e.g. ``community_2_subcommunity_13``, and plasmid to subcommunity assignments are found in ``dcj_thresh_4_graph/objects/typing.tsv``. The DCJ-Indel distances are in file ``all_plasmid_distances.tsv`` (note that the DCJ-Indel distance is only calculated between plasmids which fulfil the containment threshold, so not all pairs of plasmids will be in the file).
+- **Visualisations:** Alongside the distances and clustering, pling outputs network visualisations to aid in further analysis. These can be useful to look if you want to spot interesting relationships between plasmids, e.g. plasmid fusions. They include visualisations of the full containment network and each containment network individually, found under ``containment/containment_communities/visualisations``. All nodes are coloured black, but the edges are labelled by containment distance. Additionally, under ``dcj_thresh_4_graph/communities`` are visualisations of each plasmid community, where nodes are coloured by subcommunity assignment and edges are labelled with both containment distance and DCJ-Indel distance. Finally, under ``dcj_thresh_4_graph/subcommunities`` are visualisations of each plasmid subcommunity, where nodes all have the same colour, and edges are labelled by containment distance and DCJ-Indel distance. These subcommunity visualisations don't include edges that don't fulfil the DCJ-Indel threshold, but the others do. You can also optionally output json files of the networks, see Advanced Usage and General Advice for more information.
+

docs/requirements.txt

@@ -0,0 +1,2 @@
+sphinx==7.1.2
+sphinx-book-theme

docs/tips.rst

@@ -0,0 +1,65 @@
+General Advice
+==============
+
+Changing thresholds
+-------------------
+
+Although the default thresholds often work pretty well, unfortunately they are not one size fits all. In pling there are two thresholds you may want to change: containment distance threshold and DCJ-Indel distance threshold.
+If you are working on a small evolutionary time scale, you may want to decrease the containment threshold to 0.3. It is not in our experience so far necessary to go any lower than 0.15, especially since typically datasets on a small time scale have a high degree of sequence similarity anyway.
+For the DCJ-Indel threshold, you can often get an idea of what is a suitable threshold by studying the community network visualisations, especially if the dataset isn't too big. For example, consider the network below:
+
+.. image:: images/example_1.png
+   :alt:  pling network example
+   :align: center
+
+This example is clustered at DCJ-Indel threshold 4, but the lilac, turquoise and green subcommunities only have distances of 5-7 seperating them, while those three subcommunities have distances of at least 10 to the remaining plasmids. Arguably you could join those three subcommunities together into one by setting the threshold to 7, especially if you're looking at longer evolutionary time scales.
+
+When experimenting with DCJ-Indel thresholds, you can rerun pling with the same output directory -- a seperate folder will produced for the DCJ-Indel network and subcommunities for each threshold, and the previous integerisations and DCJ-Indel caluclations will be reused, which reduces runtimes significantly. However, if you are changing the containment threshold, make sure to use a new output folder, otherwise your results will be overwritten.
+
+A good way to develop a feeling for how much DCJ-Indel is "little" or "a lot" is by visualising alignments between plasmids, and comparing to their distance. You can for example use this script from Martin Hunt: https://github.com/martinghunt/bioinf-scripts/blob/master/python/multi_act_cartoon.py.
+
+Interpreting pling networks
+---------------------------
+
+In addition to the clustering, studying and interpreting the pling networks can indicate interesting evolutionary events. Here are collected a couple of examples from real data to showcase what you might want to look out for.
+
+**Example 1: Cointegrate bridging subcommunities**
+
+.. image:: images/russian_dolls_fusion.png
+   :alt:  fusion bridging two subcommunities
+   :align: center
+
+This example was also discussed in figure 4 of the pling paper. The central plasmid bridges two otherwise completely distinct subcommunities, and the distances between that plasmid and the remaining plasmids are low both in terms of containment and DCJ-Indel. This makes it a likely candidate for a cointegrate plasmid, and checking additional data such as plasmid length and Inc types, as well looking at visualisations of alignments between these plasmids, confirms this.
+
+**Example 2: Cointegrates and parental plasmids in a single subcommunity**
+
+.. image:: images/plasmid_cointegrates.png
+   :alt:  cointegrates and parents
+   :align: center
+
+In this example, cointegrates and their parental plasmids form one subcommunity. The five leftmost plasmids are IncR plasmids, while the five rightmost are IncFIB-FII. The three in the middle are IncFII-FIB-R plasmids, which are three cointegrate plasmids formed from the fusion of the IncR and IncFIB-FII plasmids. Like in the previous example, the cointegrates bridge the parental plasmids together. In this case, rather than being split into two subcommunities, they all end up in one -- this is because there aren't enough samples from the parents and cointegrate to create distinct enough structures on the network for the clustering algorithm to pick up on them being different. This is generally something worth looking out for, if e.g. you find a pling subcommunity has no core, check Inc types or length distributions. If you find both single and multireplicon plasmids, or have a very spread out or bimodal length distribution, that may indicate the subcommunity contains both cointegrates and their parental plasmids.
+(This example and figure is courtesy of Sandra Reuters)
+
+**Example 3: Hub plasmids**
+
+.. image:: images/russian_dolls_network.png
+   :alt:  hub plasmids
+   :align: center
+
+This example is the full community from example 1. Centrally, there is a plasmid denoted by a black star -- this is a hub plasmid. This plasmid is dominated by a transposable element, which is also present in all it's neighbours. This is typical of most hub plasmids, but sometimes hub plasmids can be very large plasmids, that are either fusions of many smaller plasmids, or have accumulated many different transposable elements. If you are interested in hubs, checking length and annotating transposons can explain why a hub plasmid has connected to so many distinct plasmids.
+
+Note that there is currently a bug in visualisations -- hubs are supposed to be denoted by a black star, but sometimes are just visualised as black circles, e.g. the two black circles in the upper left corner are both hubs (they are discussed in figure 5 of the paper). You can double check for hubs in the ``dcj_thresh_4_graph/objects/hub_plasmids.csv`` file.
+
+Editing pling visualisations
+----------------------------
+
+The default html visualisations pling produces are often a good starting point, but you may want to modify them for your specific needs. There are several ways you can do this:
+
+- **Run pling with option to output Cytoscape formatted json files.** As of version 2, you can optionally output json files that can be loaded directly into Cytoscape. There is a style file at https://raw.githubusercontent.com/leoisl/plasnet/main/plasnet/ext/templates/plasnet_style.xml that is similar to the visualisation format pling uses by default, which you can load into Cytoscape. You can also use the json files with the python library networkx (see https://networkx.org/documentation/stable/reference/readwrite/generated/networkx.readwrite.json_graph.cytoscape_graph.html). It appears there are also packages in R that will load Cytoscape formatted json files, e.g. https://rdrr.io/github/BodenmillerGroup/bbRtools/man/load_json_graph.html.
+- **Use the distance files and typing files to construct your own network in your language of choice.** You can find an example of this in issue #67 for R.
+- **Directly edit the html files.** This is generally a bit finnicky though.
+
+Bacterial and other non-plasmid genomes
+---------------------------------------
+
+Pling was created with plasmids in mind, but in principle it can be used with other types of genomes (as long as they are complete and single chromosome). The only issue you may run into is with computational resources. The run time bottleneck for pling is generally the integerisation step, which requires doing pairwise alignment. This is reasonably cheap for plasmids as they have small genomes, but may be more of an issue with larger genomes like bacteria. If you think doing pairwise alignment is viable for your dataset, then pling will likely run just fine.