Skip to content

Commit

Permalink
adds relative abundance plot; updates table functionality
Browse files Browse the repository at this point in the history
  • Loading branch information
@brwnj
brwnj committed Jun 16, 2018
1 parent 8e1615f commit 5d22d5d
Show file tree
Hide file tree
Showing 3 changed files with 229 additions and 61 deletions.
124 changes: 92 additions & 32 deletions hundo/Snakefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -258,8 +258,10 @@ REFFASTA = [os.path.join(config.get("database_dir", "."), i) for i in REF if ".f
REFMAP = [os.path.join(config.get("database_dir", "."), i) for i in REF if i.endswith(".map")][0]
REFTRE = [os.path.join(config.get("database_dir", "."), i) for i in REF if i.endswith(".tre")][0]
STYLE = """
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.1.1/css/bootstrap.css"/>
<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.18/css/dataTables.bootstrap4.min.css"/>
<style type="text/css">
body{font-family:Helvetica,arial,sans-serif;font-size:14px;line-height:1.6;padding-bottom:10px;background-color:#fff;color:#333;margin:0}body>div .section::before{content:"";display:block;height:80px;margin:-80px 0 0}#summary::before{margin:0}.topic-title{font-size:18pt}body>div>.section{margin-left:22%;margin-bottom:3em}div.section{margin-right:20px}#contents>p{display:none}button,li p.first{display:inline-block}#contents{margin-top:80px;padding-left:0;width:20%;background-color:#f1f1f1;height:100%;position:fixed;overflow:auto}#contents ul{list-style-type:none}#contents ul>li{font-size:14pt}#contents ul>li a:hover{color:#151d26}button,h1.title{color:#fff;background-color:#151d26}#contents ul>li>ul>li{font-size:12pt}h1.title{margin-top:0;position:fixed;z-index:10;padding:20px;width:100%}code,table tr:nth-child(2n),tt{background-color:#f8f8f8}.one-col{min-width:310px;height:500px;margin:0 auto}.two-col-left{height:300px;width:49%;float:left}.two-col-right{height:300px;width:49%;float:right}button{margin:0 5px 0 0;padding:5px 25px;font-size:18px;line-height:1.8;appearance:none;box-shadow:none;border-radius:3px;border:none}button:focus{outline:0}button:hover{background-color:#4183C4}button:active{background-color:#27496d}.legend-rect{width:20px;height:20px;margin-right:8px;margin-left:20px;float:left;-webkit-border-radius:2px;border-radius:2px}a{color:#4183C4;text-decoration:none}a.absent{color:#c00}a.anchor{padding-left:30px;margin-left:-30px;cursor:pointer;position:absolute;top:0;left:0;bottom:0}dl,dl dt,dl dt:first-child,hr,table,table tr{padding:0}table tr td,table tr th{border:1px solid #ccc;text-align:left;padding:6px 13px}h1,h2,h3,h4,h5,h6{margin:20px 0 10px;padding:0;font-weight:700;-webkit-font-smoothing:antialiased;cursor:text;position:relative}h1:hover a.anchor,h2:hover a.anchor,h3:hover a.anchor,h4:hover a.anchor,h5:hover a.anchor,h6:hover a.anchor{text-decoration:none}h1 code,h1 tt,h2 code,h2 tt,h3 code,h3 tt,h4 code,h4 tt,h5 code,h5 tt,h6 code,h6 tt{font-size:inherit}h1{font-size:28px;color:#151d26;border-bottom:1px solid #ccc}h2{font-size:24px;color:#000}h3{font-size:18px}h4{font-size:16px}dl dt,h5,h6{font-size:14px}h6{color:#777}blockquote,dl,li,ol,p,pre,table,ul{margin:15px 0}hr{background:url(http://tinyurl.com/bq5kskr) repeat-x;border:0;color:#ccc;height:4px}a:first-child h1,a:first-child h2,a:first-child h3,a:first-child h4,a:first-child h5,a:first-child h6{margin-top:0;padding-top:0}h1 p,h2 p,h3 p,h4 p,h5 p,h6 p{margin-top:0}dl dt{font-weight:700;font-style:italic;margin:15px 0 5px}blockquote>:first-child,dl dd>:first-child,dl dt>:first-child,table tr td :first-child,table tr th :first-child{margin-top:0}blockquote>:last-child,dl dd>:last-child,dl dt>:last-child{margin-bottom:0}dl dd{margin:0 0 15px;padding:0 15px}blockquote{border-left:4px solid #ddd;padding:0 15px;color:#777}table{border-spacing:0;border-collapse:collapse}table tr{border-top:1px solid #ccc;background-color:#fff;margin:0}table tr th{font-weight:700;margin:0}table tr td{margin:0}table tr td :last-child,table tr th :last-child{margin-bottom:0}img{max-width:100%}span.frame{display:block;overflow:hidden}span.frame>span{border:1px solid #ddd;display:block;float:left;overflow:hidden;margin:13px 0 0;padding:7px;width:auto}span.frame span img{display:block;float:left}span.frame span span{clear:both;color:#333;display:block;padding:5px 0 0}span.align-center{display:block;overflow:hidden;clear:both}span.align-center>span{display:block;overflow:hidden;margin:13px auto 0;text-align:center}span.align-center span img{margin:0 auto;text-align:center}span.align-right{display:block;overflow:hidden;clear:both}span.align-right>span{display:block;overflow:hidden;margin:13px 0 0;text-align:right}span.align-right span img{margin:0;text-align:right}span.float-left{display:block;margin-right:13px;overflow:hidden;float:left}span.float-left span{margin:13px 0 0}span.float-right{display:block;margin-left:13px;overflow:hidden;float:right}span.float-right>span{display:block;overflow:hidden;margin:13px auto 0;text-align:right}code,tt{margin:0 2px;padding:0 5px;white-space:nowrap;border:1px solid #eaeaea;border-radius:3px}pre code{margin:0;padding:0;white-space:pre;background:0 0}.highlight pre,pre{background-color:#f8f8f8;border:1px solid #ccc;font-size:13px;line-height:19px;overflow:auto;padding:6px 10px;border-radius:3px}pre code,pre tt{background-color:transparent;border:none}div#metadata{text-align:right}
body{font-family:Helvetica,arial,sans-serif;font-size:14px;line-height:1.6;padding-bottom:10px;background-color:#fff;color:#333;margin:0}body>div .section::before{content:"";display:block;height:80px;margin:-80px 0 0}#summary::before{margin:0}.topic-title{font-size:18pt}body>div>.section{margin-left:22%;margin-bottom:3em}div.section{margin-right:20px}#contents>p{display:none}button,li p.first{display:inline-block}#contents{margin-top:80px;padding-left:0;width:20%;background-color:#f1f1f1;height:100%;position:fixed;overflow:auto}#contents ul{list-style-type:none}#contents ul>li{font-size:14pt}#contents ul>li a:hover{color:#151d26}button,h1.title{color:#fff;background-color:#151d26}#contents ul>li>ul>li{font-size:12pt}h1.title{margin-top:0;position:fixed;z-index:10;padding:20px;width:100%}code,table tr:nth-child(2n),tt{background-color:#f8f8f8}.one-col{min-width:310px;height:500px;margin:0 auto}.two-col-left{height:300px;width:49%;float:left}.two-col-right{height:300px;width:49%;float:right}button{margin:0 5px 0 0;padding:5px 25px;font-size:18px;line-height:1.8;appearance:none;box-shadow:none;border-radius:3px;border:none}button:focus{outline:0}button:hover{background-color:#4183C4}button:active{background-color:#27496d}.legend-rect{width:20px;height:20px;margin-right:8px;margin-left:20px;float:left;-webkit-border-radius:2px;border-radius:2px}a{color:#4183C4;text-decoration:none}a.absent{color:#c00}a.anchor{padding-left:30px;margin-left:-30px;cursor:pointer;position:absolute;top:0;left:0;bottom:0}dl,dl dt,dl dt:first-child,hr,table,table tr{padding:0}table tr td,table tr th{border:1px solid #ccc;text-align:left;padding:6px 13px}h1,h2,h3,h4,h5,h6{margin:20px 0 10px;padding:0;font-weight:700;-webkit-font-smoothing:antialiased;cursor:text;position:relative}h1:hover a.anchor,h2:hover a.anchor,h3:hover a.anchor,h4:hover a.anchor,h5:hover a.anchor,h6:hover a.anchor{text-decoration:none}h1 code,h1 tt,h2 code,h2 tt,h3 code,h3 tt,h4 code,h4 tt,h5 code,h5 tt,h6 code,h6 tt{font-size:inherit}h1{font-size:28px;color:#151d26;border-bottom:1px solid #ccc}h2{font-size:24px;color:#000}h3{font-size:18px}h4{font-size:16px}dl dt,h5,h6{font-size:14px}h6{color:#777}blockquote,dl,li,ol,p,pre,table,ul{margin:15px 0}hr{background:url(http://tinyurl.com/bq5kskr) repeat-x;border:0;color:#ccc;height:4px}a:first-child h1,a:first-child h2,a:first-child h3,a:first-child h4,a:first-child h5,a:first-child h6{margin-top:0;padding-top:0}h1 p,h2 p,h3 p,h4 p,h5 p,h6 p{margin-top:0}dl dt{font-weight:700;font-style:italic;margin:15px 0 5px}blockquote>:first-child,dl dd>:first-child,dl dt>:first-child,table tr td :first-child,table tr th :first-child{margin-top:0}blockquote>:last-child,dl dd>:last-child,dl dt>:last-child{margin-bottom:0}dl dd{margin:0 0 15px;padding:0 15px}blockquote{border-left:4px solid #ddd;padding:0 15px;color:#777}table{border-spacing:0;border-collapse:collapse}table tr{border-top:1px solid #ccc;background-color:#fff;margin:0}table tr th{font-weight:700;margin:0}table tr td{margin:0}table tr td :last-child,table tr th :last-child{margin-bottom:0}img{max-width:100%}span.frame{display:block;overflow:hidden}span.frame>span{border:1px solid #ddd;display:block;float:left;overflow:hidden;margin:13px 0 0;padding:7px;width:auto}span.frame span img{display:block;float:left}span.frame span span{clear:both;color:#333;display:block;padding:5px 0 0}span.align-center{display:block;overflow:hidden;clear:both}span.align-center>span{display:block;overflow:hidden;margin:13px auto 0;text-align:center}span.align-center span img{margin:0 auto;text-align:center}span.align-right{display:block;overflow:hidden;clear:both}span.align-right>span{display:block;overflow:hidden;margin:13px 0 0;text-align:right}span.align-right span img{margin:0;text-align:right}span.float-left{display:block;margin-right:13px;overflow:hidden;float:left}span.float-left span{margin:13px 0 0}span.float-right{display:block;margin-left:13px;overflow:hidden;float:right}span.float-right>span{display:block;overflow:hidden;margin:13px auto 0;text-align:right}code,tt{margin:0 2px;padding:0 5px;white-space:nowrap;border:1px solid #eaeaea;border-radius:3px}pre code{margin:0;padding:0;white-space:pre;background:0 0}.highlight pre,pre{background-color:#f8f8f8;border:1px solid #ccc;font-size:13px;line-height:19px;overflow:auto;padding:6px 10px;border-radius:3px}pre code,pre tt{background-color:transparent;border:none}div#metadata{text-align:right}h1{line-height:1.6}
</style>
"""

Expand Down Expand Up @@ -889,20 +891,22 @@ rule report:
s_idx.sort_values(inplace=True)
# sample order for plots
sample_order = s_idx.index.tolist()
mat = [
otu_df = [
["Samples", "OTUs", "OTU Total Count", "OTU Table Density"],
[bt.length("sample"), bt.length("observation"), bt.sum(), '{:f}'.format(bt.get_table_density())]
]
colorscale = [[0, "#151d26"], [.5, "white"], [1, "white"]]
fig = ff.create_table(mat, colorscale)
fig["data"][0]["opacity"] = 100
fig["layout"]["margin"]["b"] = 30
fig["layout"]["margin"]["r"] = 20
# fig["layout"]["font"]["size"] = 13
summary_table = offline.plot(
fig,
**PLOTLY_PARAMS
otu_df = pd.DataFrame(otu_df)
# make the first row the column labels
otu_df.rename(columns=otu_df.iloc[0], inplace=True)
# drop the first row
otu_df = otu_df.reindex(otu_df.index.drop(0))
# build the table
summary_table = otu_df.to_html(
index=False, bold_rows=False, classes=["table", "table-bordered"], table_id="otuTable"
)
# fix for pandoc
summary_table = summary_table.replace("\n", "\n" + 10 * " ")
del(otu_df)

# quality plot
raw_qual_stats = defaultdict(lambda: defaultdict(list))
Expand Down Expand Up @@ -988,9 +992,11 @@ rule report:
# sample_cols = [i for i in df.columns if i not in ["#OTU ID", "taxonomy"]]
df[["kingdom", "phylum", "class", "order", "family", "genus", "species"]] = df["taxonomy"].str.split(",", expand=True)
tax_dataframes = []
tax_dataframes_rel = []
levels = ["phylum", "class", "order"]
for level in levels:
sub = df[[level] + sample_order]
sub = df[[level] + sample_order].copy()
# cols = sub[level].tolist()
sub[level] = sub[level].str.replace("%s__" % level[0], "")
# sum same assignments over rows
sub = sub.groupby([level]).sum().reset_index()
Expand All @@ -999,7 +1005,16 @@ rule report:
sub.sort_values("TotalCount", ascending=False, inplace=True)
# redefine columns after sort
sub = sub[[level] + sample_order]
sub = sub.T
# relative abundance dataframe
sub_p = sub.copy()
sub_p[sample_order] = sub_p[sample_order].div(sub_p[sample_order].sum())
sub_p[sample_order] = sub_p[sample_order] * 100
sub_p = sub_p.transpose()
sub_p.columns = sub_p.loc[level]
sub_p.drop([level], inplace=True)
tax_dataframes_rel.append(sub_p)
# absolute abundance dataframe
sub = sub.transpose()
sub.columns = sub.loc[level]
sub.drop([level], inplace=True)
tax_dataframes.append(sub)
Expand All @@ -1010,6 +1025,8 @@ rule report:
data += [go.Bar(x=subset.index,
y=subset[tax],
name=tax,
text=tax,
hoverinfo="text+y",
visible=True if i == 0 else False
) for tax in subset.columns]
lengths.append(len(subset.columns))
Expand Down Expand Up @@ -1058,6 +1075,22 @@ rule report:
fig,
**PLOTLY_PARAMS
)
# make the relative abundance taxonomy plot
data = []
for i, subset in enumerate(tax_dataframes_rel):
data += [go.Bar(x=subset.index,
y=subset[tax],
name=tax,
text=tax,
hoverinfo="text+y",
visible=True if i == 0 else False
) for tax in subset.columns]
layout["yaxis"]["title"] = "Relative Abundance"
fig = go.Figure(data=data, layout=layout)
relative_abundance_taxonomy_plot = offline.plot(
fig,
**PLOTLY_PARAMS
)

# merged reads quality plot
ee_stats = defaultdict(list)
Expand Down Expand Up @@ -1116,16 +1149,10 @@ rule report:
sd.append('{:f}'.format(s_idx.loc[sample]))
count_data.append(sd)
ss = pd.DataFrame(count_data, columns=["Sample", "Raw", "Filtered", "Merged", "In OTUs", "Shannon"])
colorscale = [[0, '#151d26'], [.5, 'white'], [1, 'white']]
fig = ff.create_table(ss, colorscale=colorscale)
fig["data"][0]["opacity"] = 100
fig["layout"]["margin"]["b"] = 30
fig["layout"]["margin"]["r"] = 20
# fig["layout"]["font"]["size"] = 13
sample_summary_table = offline.plot(
fig,
**PLOTLY_PARAMS
sample_summary_table = ss.to_html(
index=False, bold_rows=False, classes=["table", "table-bordered"], table_id="summaryTable"
)
sample_summary_table = sample_summary_table.replace("\n", "\n" + 10 * " ")

# sample summary counts plot
# remove omitted samples from df
Expand Down Expand Up @@ -1170,25 +1197,42 @@ rule report:
.. raw:: html
{STYLE}
<script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
<script src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
<script src="https://cdn.datatables.net/1.10.18/js/jquery.dataTables.min.js"></script>
<script src="https://cdn.datatables.net/1.10.18/js/dataTables.bootstrap4.min.js"></script>
<script>
$(document).ready( function () {{
$('#summaryTable').DataTable();
$('#otuTable').DataTable( {{
"searching": false,
"paging": false,
"ordering": false,
"info": false,
}} );
}} );
</script>
=============================================================
hundo_ - Experiment Summary
=============================================================
.. contents::
:backlinks: none
:depth: 2
Summary
-------
Sequence Counts
***************
Samples are sorted by diversity (least to most).
In the plots, samples are ordered by diversity (least to most).
.. raw:: html
<div style="overflow-x:auto;">
{sample_summary_table}
</div>
{sample_counts_plot}
Sequence Quality
Expand All @@ -1204,15 +1248,24 @@ rule report:
.. raw:: html
<div style="overflow-x:auto;">
{summary_table}
</div>
Taxonomy
********
Taxonomy by Count
*****************
.. raw:: html
{taxonomy_plot}
Taxonomy by Percent
*******************
.. raw:: html
{relative_abundance_taxonomy_plot}
Methods
-------
Expand Down Expand Up @@ -1312,21 +1365,25 @@ rule report:
The zip archive contains the following files:
**OTU.biom**
OTU.biom
````````
Biom table with raw counts per sample and their associated
taxonomic assignment formatted to be compatible with downstream tools
like phyloseq_.
**OTU.fasta**
OTU.fasta
`````````
Representative DNA sequences of each OTU.
**OTU.tree**
OTU.tree
````````
Newick tree representation of aligned OTU sequences.
**OTU.txt**
OTU.txt
```````
Tab-delimited text table with columns OTU ID, a column for each sample,
and taxonomy assignment in the final column as a comma delimited list.
Expand All @@ -1338,17 +1395,20 @@ rule report:
Other files that may be needed, but that are not included in the
attached archive include:
**OTU_aligned.fasta**
OTU_aligned.fasta
`````````````````
OTU sequences after alignment using MAFFT.
**all-sequences.fasta**
all-sequences.fasta
```````````````````
Quality-controlled, dereplicated DNA sequences of all samples. The
header of each record identifies the sample of origin and the count
resulting from dereplication.
**blast-hits.txt**
blast-hits.txt
``````````````
The BLAST assignments per OTU sequence.
Expand Down
2 changes: 1 addition & 1 deletion hundo/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "1.1.13"
__version__ = "1.1.14"
Loading

0 comments on commit 5d22d5d

Please sign in to comment.