-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
1006 lines (880 loc) · 32.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2021-07-21 Wed 15:18 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>LS4GAN's Toy Zero</title>
<meta name="generator" content="Org mode" />
<meta name="author" content="BV" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center;
margin-bottom: .2em; }
.subtitle { text-align: center;
font-size: medium;
font-weight: bold;
margin-top:0; }
.todo { font-family: monospace; color: red; }
.done { font-family: monospace; color: green; }
.priority { font-family: monospace; color: orange; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.org-right { margin-left: auto; margin-right: 0px; text-align: right; }
.org-left { margin-left: 0px; margin-right: auto; text-align: left; }
.org-center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: auto;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline; margin-top: 14px;}
/* Languages per Org manual */
pre.src-asymptote:before { content: 'Asymptote'; }
pre.src-awk:before { content: 'Awk'; }
pre.src-C:before { content: 'C'; }
/* pre.src-C++ doesn't work in CSS */
pre.src-clojure:before { content: 'Clojure'; }
pre.src-css:before { content: 'CSS'; }
pre.src-D:before { content: 'D'; }
pre.src-ditaa:before { content: 'ditaa'; }
pre.src-dot:before { content: 'Graphviz'; }
pre.src-calc:before { content: 'Emacs Calc'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-fortran:before { content: 'Fortran'; }
pre.src-gnuplot:before { content: 'gnuplot'; }
pre.src-haskell:before { content: 'Haskell'; }
pre.src-hledger:before { content: 'hledger'; }
pre.src-java:before { content: 'Java'; }
pre.src-js:before { content: 'Javascript'; }
pre.src-latex:before { content: 'LaTeX'; }
pre.src-ledger:before { content: 'Ledger'; }
pre.src-lisp:before { content: 'Lisp'; }
pre.src-lilypond:before { content: 'Lilypond'; }
pre.src-lua:before { content: 'Lua'; }
pre.src-matlab:before { content: 'MATLAB'; }
pre.src-mscgen:before { content: 'Mscgen'; }
pre.src-ocaml:before { content: 'Objective Caml'; }
pre.src-octave:before { content: 'Octave'; }
pre.src-org:before { content: 'Org mode'; }
pre.src-oz:before { content: 'OZ'; }
pre.src-plantuml:before { content: 'Plantuml'; }
pre.src-processing:before { content: 'Processing.js'; }
pre.src-python:before { content: 'Python'; }
pre.src-R:before { content: 'R'; }
pre.src-ruby:before { content: 'Ruby'; }
pre.src-sass:before { content: 'Sass'; }
pre.src-scheme:before { content: 'Scheme'; }
pre.src-screen:before { content: 'Gnu Screen'; }
pre.src-sed:before { content: 'Sed'; }
pre.src-sh:before { content: 'shell'; }
pre.src-sql:before { content: 'SQL'; }
pre.src-sqlite:before { content: 'SQLite'; }
/* additional languages in org.el's org-babel-load-languages alist */
pre.src-forth:before { content: 'Forth'; }
pre.src-io:before { content: 'IO'; }
pre.src-J:before { content: 'J'; }
pre.src-makefile:before { content: 'Makefile'; }
pre.src-maxima:before { content: 'Maxima'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-picolisp:before { content: 'Pico Lisp'; }
pre.src-scala:before { content: 'Scala'; }
pre.src-shell:before { content: 'Shell Script'; }
pre.src-ebnf2ps:before { content: 'ebfn2ps'; }
/* additional language identifiers per "defun org-babel-execute"
in ob-*.el */
pre.src-cpp:before { content: 'C++'; }
pre.src-abc:before { content: 'ABC'; }
pre.src-coq:before { content: 'Coq'; }
pre.src-groovy:before { content: 'Groovy'; }
/* additional language identifiers from org-babel-shell-names in
ob-shell.el: ob-shell is the only babel language using a lambda to put
the execution function name together. */
pre.src-bash:before { content: 'bash'; }
pre.src-csh:before { content: 'csh'; }
pre.src-ash:before { content: 'ash'; }
pre.src-dash:before { content: 'dash'; }
pre.src-ksh:before { content: 'ksh'; }
pre.src-mksh:before { content: 'mksh'; }
pre.src-posh:before { content: 'posh'; }
/* Additional Emacs modes also supported by the LaTeX listings package */
pre.src-ada:before { content: 'Ada'; }
pre.src-asm:before { content: 'Assembler'; }
pre.src-caml:before { content: 'Caml'; }
pre.src-delphi:before { content: 'Delphi'; }
pre.src-html:before { content: 'HTML'; }
pre.src-idl:before { content: 'IDL'; }
pre.src-mercury:before { content: 'Mercury'; }
pre.src-metapost:before { content: 'MetaPost'; }
pre.src-modula-2:before { content: 'Modula-2'; }
pre.src-pascal:before { content: 'Pascal'; }
pre.src-ps:before { content: 'PostScript'; }
pre.src-prolog:before { content: 'Prolog'; }
pre.src-simula:before { content: 'Simula'; }
pre.src-tcl:before { content: 'tcl'; }
pre.src-tex:before { content: 'TeX'; }
pre.src-plain-tex:before { content: 'Plain TeX'; }
pre.src-verilog:before { content: 'Verilog'; }
pre.src-vhdl:before { content: 'VHDL'; }
pre.src-xml:before { content: 'XML'; }
pre.src-nxml:before { content: 'XML'; }
/* add a generic configuration mode; LaTeX export needs an additional
(add-to-list 'org-latex-listings-langs '(conf " ")) in .emacs */
pre.src-conf:before { content: 'Configuration File'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.org-right { text-align: center; }
th.org-left { text-align: center; }
th.org-center { text-align: center; }
td.org-right { text-align: right; }
td.org-left { text-align: left; }
td.org-center { text-align: center; }
dt { font-weight: bold; }
.footpara { display: inline; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.equation-container {
display: table;
text-align: center;
width: 100%;
}
.equation {
vertical-align: middle;
}
.equation-label {
display: table-cell;
text-align: right;
vertical-align: middle;
}
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
.org-svg { width: 90%; }
/*]]>*/-->
</style>
<link rel="stylesheet" type="text/css" href="https://ls4gan.github.io/other/darksun/css/htmlize.css"/>
<link rel="stylesheet" type="text/css" href="https://ls4gan.github.io/other/darksun/css/darksun.css"/>
<script type="text/javascript">
// @license magnet:?xt=urn:btih:e95b018ef3580986a04669f1b5879592219e2a7a&dn=public-domain.txt Public Domain
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.classList.add("code-highlighted");
target.classList.add("code-highlighted");
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.classList.remove("code-highlighted");
target.classList.remove("code-highlighted");
}
}
/*]]>*///-->
// @license-end
</script>
</head>
<body>
<div id="content">
<h1 class="title">LS4GAN's Toy Zero</h1>
<div id="table-of-contents">
<h2>Table of Contents</h2>
<div id="text-table-of-contents">
<ul>
<li><a href="#goals">1. Goals</a></li>
<li><a href="#prepare">2. Prepare</a>
<ul>
<li><a href="#env">2.1. Environment</a></li>
<li><a href="#checks">2.2. Check Wire-Cell Toolkit</a></li>
<li><a href="#python">2.3. Python</a></li>
</ul>
</li>
<li><a href="#usage">3. Usage</a></li>
<li><a href="#details">4. Details</a>
<ul>
<li><a href="#resp">4.1. resp</a></li>
<li><a href="#wires">4.2. wires</a></li>
<li><a href="#depos">4.3. depos</a></li>
<li><a href="#frames">4.4. frames</a></li>
<li><a href="#images">4.5. images</a></li>
</ul>
</li>
<li><a href="#plotting">5. Plotting</a>
<ul>
<li><a href="#imgfmt">5.1. Image formats</a></li>
<li><a href="#cmap">5.2. Color maps</a></li>
<li><a href="#realVfake">5.3. Real vs Fake</a></li>
</ul>
</li>
<li><a href="#config">6. Configuration</a></li>
<li><a href="#snake">7. Snakemake</a></li>
<li><a href="#hack">8. Hacking</a></li>
</ul>
</div>
</div>
<p>
The toyzero package provides first steps toward <a href="https://ls4gan.github.io/">LS4GAN</a>. See also <a href="next-steps.html">next-steps.html</a>
</p>
<div id="outline-container-goals" class="outline-2">
<h2 id="goals"><span class="section-number-2">1</span> Goals</h2>
<div class="outline-text-2" id="text-goals">
<p>
After some pre-requirements are met, this package automates the
preparation for and running of various Wire-Cell Toolkit jobs in order
to produce some initial datasets for initial exploration of the LS4GAN
concept.
</p>
<p>
The final results are in the form raw ADC (Z-axis) waveform arrays in
per-channel rows (Y-axis) of time-sampled columns (X-axis). Each
array spans one of three wire planes from one of six anode plane
assemblies (APA) of the ProtoDUNE-SP detector.
</p>
<p>
There are two domains named <b>real</b> and <b>fake</b> (both simulated in toyzero).
The <b>real</b> domain uses the full, best known 2D response model. The <b>fake</b>
domain uses an intentionally "more wrong" quasi-1D model which is
derived from the 2D model by zeroing all responses except those
covering the central wire of interest. (Note that this quasi-1D model
is not the same of the "yet more wronger" 1D model as q1D includes the
variation within the central wire region.)
</p>
<p>
For this initial development we accept a number of simplifying
"cheats" which we will rectify as we gain experience and develop the
next phase. They are at least the following:
</p>
<ul class="org-ul">
<li>Use of simulation for both "real" and "fake". Eventually, "real"
will be detector data and "fake" will from our best available
simulation (currently 2D Wire Cell). We may use paired data for
evaluating the technique in this stage but can not rely on it going
forward.</li>
<li>Our initial <i>data tier</i> is that of <b>ADC waveforms</b> and those from only
ionization electrons from <b>ideal tracks</b> and in particular <b>excluding
noise</b> effects. We eventually will use realistic track simulation
(Geant4) and will include noise. When adding noise we will switch
to a <i>data tier</i> of <b>reconstructed signal waveforms</b>. To produce them
we will include <b>noise models</b> in the simulation and apply <b>noise
filtering</b> to both the fake simulation and the real detector data
just prior to applying <b>signal processing</b>. We may stop there or go
one step further and apply Wire-Cell <b>3D tomographic imaging</b> to form
our data tier. Each step brings "real" and "fake" closer and thus
allows LS4GAN to expend its deep neural nodes to find ever more
subtle differences.</li>
</ul>
</div>
</div>
<div id="outline-container-prepare" class="outline-2">
<h2 id="prepare"><span class="section-number-2">2</span> Prepare</h2>
<div class="outline-text-2" id="text-prepare">
<p>
This package requires a few user environment settings to support use
of Wire-Cell Toolkit and various Python packages. We will assume
<a href="https://direnv.net/">direnv</a> is used to manage the environment. To use <b>another environment
management system</b>, translate the contents of the example <a href="dot.envrc">dot.envrc</a>.
</p>
</div>
<div id="outline-container-env" class="outline-3">
<h3 id="env"><span class="section-number-3">2.1</span> Environment</h3>
<div class="outline-text-3" id="text-env">
<p>
A one time setup of direnv:
</p>
<pre class="example" id="orgd2bd834">
$ cd toyzero
$ cp dot.envrc .envrc
$ emacs .envrc # edit to taste
$ direnv allow
</pre>
</div>
</div>
<div id="outline-container-checks" class="outline-3">
<h3 id="checks"><span class="section-number-3">2.2</span> Check Wire-Cell Toolkit</h3>
<div class="outline-text-3" id="text-checks">
<p>
The Wire-Cell Toolkit program <code>wire-cell</code> and its libraries are
required. See <a href="https://wirecell.bnl.gov/">https://wirecell.bnl.gov/</a> for installation information.
Check if WCT is available with these commands:
</p>
<pre class="example" id="org9bc48ed">
$ wire-cell --help
$ wcsonnet wirecell.jsonnet
$ echo $WIRECELL_PATH
</pre>
<p>
The <code>WIRECELL_PATH</code> should include at least the <code>cfg/</code> directory provided
by WCT. This <code>toyzero</code> package provides additional configuration in
<a href="cfg/">./cfg</a> which will found automatically.
</p>
</div>
</div>
<div id="outline-container-python" class="outline-3">
<h3 id="python"><span class="section-number-3">2.3</span> Python</h3>
<div class="outline-text-3" id="text-python">
<p>
The stand-alone <code>wire-cell-python</code> package, the <code>snakemake</code> program and
various other Python 3 packages are required.
</p>
<pre class="example" id="org2020345">
$ cd toyzero
$ pip install -r requirements.txt
</pre>
<p>
Some checks:
</p>
<pre class="example" id="orgddb5fb1">
$ wirecell-<TAB>
$ wirecell-util --help
$ snakemake --help
</pre>
<p>
See also section <a href="#hack">8</a> for info on providing a developer version of
<code>wire-cell-python</code> code.
</p>
</div>
</div>
</div>
<div id="outline-container-usage" class="outline-2">
<h2 id="usage"><span class="section-number-2">3</span> Usage</h2>
<div class="outline-text-2" id="text-usage">
<p>
Exercising toyzero will automatically download various files from
Wire-Cell GitHub and run various programs. In principle, the user
need run only:
</p>
<pre class="example" id="org1274abe">
$ snakemake -jall all
</pre>
<p>
This can take ten minutes or so depending on how fast your CPU is.
To see what is produced:
</p>
<pre class="example" id="orgf1517d3">
$ tree data plots
</pre>
<p>
To generate the minimum needed for the "images" data tier (see below)
you can do:
</p>
<pre class="example" id="org8363d3c">
$ snakemake -jall just_images
</pre>
<p>
By default, the above does not retain intermediate data files from the
"depos", "wires", "resp" and "frames" data tier files (see below for
descriptions), only keeping "images". To keep the intermediates you
may run as:
</p>
<pre class="example" id="org57e844a">
$ snakemake -jall just_images --notemp
</pre>
<p>
A one "event" run produce about 2MB of "images" and 360 MB if
intermediates are kept.
</p>
<p>
More information on the use of <code>snakemake</code> in toyzero is below and in
sections <a href="#config">6</a> and <a href="#snake">7</a>.
</p>
</div>
</div>
<div id="outline-container-details" class="outline-2">
<h2 id="details"><span class="section-number-2">4</span> Details</h2>
<div class="outline-text-2" id="text-details">
<p>
Looking into the <a href="Snakefile">Snakefile</a> one will see the <code>all</code> target aggregates a
number of <code>all_*</code> sub-targets. You may run each ordered, piece-wise
manner:
</p>
<pre class="example" id="orgb545a74">
$ snakemake -jall all_resp
$ snakemake -jall all_wires
$ snakemake -jall all_depos
$ snakemake -jall all_frames
$ snakemake -jall all_images
</pre>
<p>
Each of these steps are described more in the next subsections.
</p>
</div>
<div id="outline-container-resp" class="outline-3">
<h3 id="resp"><span class="section-number-3">4.1</span> resp</h3>
<div class="outline-text-3" id="text-resp">
<p>
The <code>all_resp</code> target downloads 2D WCT "response file" and from it
derives the quasi-1D response file. then various diagnostic plots are
made for each.
</p>
<p>
Visualization of q1D and 2D responses are generated and shown below.
First "fake" q1D and "real" 2D. These show the instantaneous induced
current (Z-axis) on a "wire of interest" over the drift path of an
electron. One row gives the response for an electron path that begins
at the transverse (aka "pitch") location relative to the wire center
and as given on the Y-axis. Along that path/row, the Z/color value of
each pixel gives the instantaneous current at a moment of time
(X-axis).
</p>
<p>
Note, the actual paths follow some trajectory in space but this
information is not important for applying the response (though is very
important for calculating the responses). Also note, the current is
in a "signed log10" unit with an arbitrary scaling in order to see
both fine and gross features.
</p>
<ul class="org-ul">
<li>Fake/q1D field response for ProtoDUNE-SP</li>
</ul>
<div id="org936dd21" class="figure">
<p><img src="plots/fake-resps-diagnostic.png" alt="fake-resps-diagnostic.png" width="90%" />
</p>
</div>
<ul class="org-ul">
<li>Real/2D field response for ProtoDUNE-SP</li>
</ul>
<div id="org08e9049" class="figure">
<p><img src="plots/real-resps-diagnostic.png" alt="real-resps-diagnostic.png" width="90%" />
</p>
</div>
</div>
</div>
<div id="outline-container-wires" class="outline-3">
<h3 id="wires"><span class="section-number-3">4.2</span> wires</h3>
<div class="outline-text-3" id="text-wires">
<p>
The <code>all_wires</code> target downloads WCT "wires file" and makes a multipage
PDF file with diagnostic plots at <a href="plots/wires-diagnostic.pdf">plots/wires-diagnostic.pdf</a>.
</p>
<p>
Most of these pages may show content that is too esoteric for most
users and they exhaustively cover all six protodune APAs. However,
the pages showing wire segments may be instructive. For the most
part, they can be ignored as "wires files" are typically well tested
prior to making them available.
</p>
</div>
</div>
<div id="outline-container-depos" class="outline-3">
<h3 id="depos"><span class="section-number-3">4.3</span> depos</h3>
<div class="outline-text-3" id="text-depos">
<p>
The <code>all_depos</code> target generates sets of ionization point depositions
for input to the WCT simulation. It produces a file:
</p>
<pre class="example" id="org7036a73">
data/depos/depos.npz
</pre>
<p>
The file is in Numpy format with three types of arrays named like:
</p>
<pre class="example" id="org6259c6d">
depo_data_<N>
depo_info_<N>
track_info_<N>
</pre>
<p>
We will ignore the <code>info</code> arrays here.
</p>
<p>
The <code><N></code> counts the set of depos which are generated together (eg, an
"event"). The <code>data</code> arrays are 2D of shape <code>(n, 7)</code> where <code>n</code> is the
number of depositions over all tracks in the "event" and is
O(100k-1M). Each depo is a 7-tuple with elements:
</p>
<ol class="org-ol">
<li>time</li>
<li>number of electrons</li>
<li>X position</li>
<li>Y position</li>
<li>Z position</li>
<li>longitudinal extent (zero here)</li>
<li>transverse extent (zero here)</li>
</ol>
<p>
A simple summary diagnostic plot for the first "event" is generated
showing distribution of depo times and the depo locations.
</p>
<div id="org49e2005" class="figure">
<p><img src="plots/depos-diagnostic.png" alt="depos-diagnostic.png" width="90%" />
</p>
</div>
<p>
The <code>track_info_<N></code> arrays are <a href="https://numpy.org/doc/stable/user/basics.rec.html">numpy structured arrays</a> with these
"columns":
</p>
<dl class="org-dl">
<dt>pmin</dt><dd>3-vector holding track start point in Cartesian coordinates (mm)</dd>
<dt>pmax</dt><dd>3-vector holding track end point in Cartesian coordinates (mm)</dd>
<dt>tmin</dt><dd>time at start point (ns)</dd>
<dt>tmax</dt><dd>time at end point (ns)</dd>
<dt>step</dt><dd>distance between steps along track (mm)</dd>
<dt>eper</dt><dd>number of ionization electrons at each step</dd>
</dl>
<p>
Each of these columns have one row per "track" (line source) in the
"event" number <code><N></code>. By default their length is 10.
</p>
</div>
</div>
<div id="outline-container-frames" class="outline-3">
<h3 id="frames"><span class="section-number-3">4.4</span> frames</h3>
<div class="outline-text-3" id="text-frames">
<p>
The <code>all_frames</code> target generates "frame" data from depos by running the
Wire-Cell simulation. The "frame" file format is described elsewhere.
For here, we treat it as a temporary.
</p>
<p>
The simulation is internally structured as a DAG as shown:
</p>
<div id="org47fb3ba" class="figure">
<p><img src="plots/dots/depos-sim-adc/dag.png" alt="dag.png" width="90%" />
</p>
</div>
</div>
</div>
<div id="outline-container-images" class="outline-3">
<h3 id="images"><span class="section-number-3">4.5</span> images</h3>
<div class="outline-text-3" id="text-images">
<p>
The <code>all_images</code> target processes each "frame" data to produce one 2D
image for each of three wire planes of each of six anode plane
assemblies. The file name and name of the single array in the file
match. For example:
</p>
<pre class="example" id="org6958ce7">
❯ wirecell-util npzls data/images/real/protodune-orig-0-1-W.npz
protodune-orig-0-1-W (960, 6000)
</pre>
<p>
These image arrays have shape <code>(nchan, ntick)</code>. That is each row is the
waveform from one channel and has <code>ntick=6000</code> samples (3ms). When
visualized with matplotlib's <code>imshow()</code> you will see channels as Y-axis,
tics as X-axis.
</p>
<p>
This "W" file holds one array of 960 channels and 6000 sample time
"ticks" and is from index=0 from the "real" data, APA ID 1 and plane
"W" (collection plane aka plane 2 counting from 0). "U" and "V" are
induction planes and each will have 800 channels and the
contemporaneous 6000 ticks.
</p>
</div>
</div>
</div>
<div id="outline-container-plotting" class="outline-2">
<h2 id="plotting"><span class="section-number-2">5</span> Plotting</h2>
<div class="outline-text-2" id="text-plotting">
<p>
Diagnostic plots for each APA of the first event are made in PDF, PNG
and SVG with different color maps to explore how best to display the
information. These image pose problems as they are relatively high
resolution and high dynamic range but are also sparse. As such we
must take care that some visualization will produce very <b>misleading
artifacts</b>.
</p>
</div>
<div id="outline-container-imgfmt" class="outline-3">
<h3 id="imgfmt"><span class="section-number-3">5.1</span> Image formats</h3>
<div class="outline-text-3" id="text-imgfmt">
<dl class="org-dl">
<dt>png</dt><dd>At <code>matplotlib</code> default DPI expect loss of visual data due to
the resolution being low and potentially due to low values falling
into the "center" color of the color map. Increasing DPI can help
at the cost of larger files. Exploring different color maps follows.</dd>
<dt>pdf</dt><dd>Likely best format to use for accuracy. However <b>beware of
antialiasing</b> that many PDF viewers apply to make images and text
look "prettier". It can obscure features and add artifacts.</dd>
<dt>svg</dt><dd>Much of the benefits of PDF but can be inlined in HTML.</dd>
</dl>
<p>
As an example of DPI problems, compare the two images below. The
first is a high-DPI PNG rendering (by NETPBM) from the original PDF
made by <code>matplotlib</code> and the second is a PNG directly from <code>matplotlib</code> at
its default DPI.
</p>
<div id="org6fd9174" class="figure">
<p><img src="plots/hidpi/frames-fake-apa0.png" alt="frames-fake-apa0.png" width="90%" />
</p>
</div>
<p>
to the same data as PNG at default DPI from matplotlib:
</p>
<div id="orgf4222fb" class="figure">
<p><img src="plots/frames-fake-apa0.png" alt="frames-fake-apa0.png" width="90%" />
</p>
</div>
<p>
If you need more zoom from your <code>evince</code> PDF viewer try
</p>
<pre class="example" id="orgf9b086e">
❯ gsettings set org.gnome.Evince page-cache-size 2014
</pre>
</div>
</div>
<div id="outline-container-cmap" class="outline-3">
<h3 id="cmap"><span class="section-number-3">5.2</span> Color maps</h3>
<div class="outline-text-3" id="text-cmap">
<p>
Selecting a good <a href="https://matplotlib.org/stable/gallery/color/colormap_reference.html">color map</a> can have a big impact. Some guidance:
</p>
<ul class="org-ul">
<li>For bipolar data of large dynamic range consider using "seismic" and
symmetric ranges (<code>vmin/vmax</code>).</li>
<li>For bipolar data with low dynamic range, select a symmetric color
map such as "coolwarm" and use a masked array to remove the 0 value.</li>
<li>Avoid "rainbow" color maps with a large central region of a nearly
common color.</li>
</ul>
<p>
The <code>wire-cell-python</code> package provides a command to easily visualize a
2D Numpy image array in a file with many plotting parameters exposed
as command line options:
</p>
<pre class="example" id="org5b6e093">
❯ wirecell-util npz-to-img --help
Usage: wirecell-util npz-to-img [OPTIONS] NPZFILE
Make an image from an array in an numpy file.
Options:
-o, --output TEXT Output image file
-a, --array TEXT Array to plot
-c, --cmap TEXT Color map
-b, --baseline TEXT Apply median baseline subtraction or if number,
subtract directly
-m, --mask TEXT Value or range to mask, range is lower-edge inclusive
--vmin TEXT Minimum value (z/color axis)
--vmax TEXT Maximum value (z/color axis)
--dpi TEXT The dots-per-inch resolution
-z, --zoom TEXT A zoom range as 'rmin:rmax,cmin:cmax'
--help Show this message and exit.
❯ wirecell-util npz-to-img --cmap Spectral \
--zoom 300:500,0:1000 --mask 0 --vmin -50 --vmax 50 \
--dpi 600 --baseline=median \
-o junk.png data/images/real/protodune-orig-0-2-U.npz
</pre>
<p>
A few color map examples with command like tha bove. Note, 0 value is
masked to white regardless of color map.
</p>
<ul class="org-ul">
<li>"coolwarm" (real)</li>
</ul>
<div id="org8423d06" class="figure">
<p><img src="plots/images/real/coolwarm/protodune-orig-0-2-U.png" alt="protodune-orig-0-2-U.png" width="90%" />
</p>
</div>
<ul class="org-ul">
<li>"seismic" (real)</li>
</ul>
<div id="org2ce85bf" class="figure">
<p><img src="plots/images/real/seismic/protodune-orig-0-2-U.png" alt="protodune-orig-0-2-U.png" width="90%" />
</p>
</div>
<ul class="org-ul">
<li>"Spectral" (real)</li>
</ul>
<div id="org8de0377" class="figure">
<p><img src="plots/images/real/Spectral/protodune-orig-0-2-U.png" alt="protodune-orig-0-2-U.png" width="90%" />
</p>
</div>
<ul class="org-ul">
<li>"viridis" (real)</li>
</ul>
<div id="orgc363d5a" class="figure">
<p><img src="plots/images/real/viridis/protodune-orig-0-2-U.png" alt="protodune-orig-0-2-U.png" width="90%" />
</p>
</div>
</div>
</div>
<div id="outline-container-realVfake" class="outline-3">
<h3 id="realVfake"><span class="section-number-3">5.3</span> Real vs Fake</h3>
<div class="outline-text-3" id="text-realVfake">
<p>
We select one event region (U plane) to show "real" 2D vs "fake" q1D,
both as SVG. In particular, not how there is "long range induction"
effects in 2D which are lacking in q1D.
</p>
<ul class="org-ul">
<li>Real:</li>
</ul>
<div id="org8191e53" class="figure">
<p><object type="image/svg+xml" data="plots/images/real/seismic/protodune-orig-0-2-U.svg" class="org-svg" width="90%">
Sorry, your browser does not support SVG.</object>
</p>
</div>
<ul class="org-ul">
<li>Fake:</li>
</ul>
<div id="orge15e600" class="figure">
<p><object type="image/svg+xml" data="plots/images/fake/seismic/protodune-orig-0-2-U.svg" class="org-svg" width="90%">
Sorry, your browser does not support SVG.</object>
</p>
</div>
</div>
</div>
</div>
<div id="outline-container-config" class="outline-2">
<h2 id="config"><span class="section-number-2">6</span> Configuration</h2>
<div class="outline-text-2" id="text-config">
<p>
By default, <code>toyzero</code> operates as a generator. That is, every time a
particular version is run it should always produce essentially an
identical result.
</p>
<p>
Some high-level parameters may be changed to produce different
results. These can be controlled by a configuration file. For
example the default configuration is held in:
</p>
<div class="org-src-container">
<pre class="src src-yaml">---
# DO NOT EDIT, make a copy, edit that.
# This is the default toyzero config file.
# Use like:
# snakemake --configfile mycfg.yaml -jall all
outdir: "."
datadir: "data"
plotdir: "plots"
seed: "1,2,3,4"
ntracks: 10
nevents: 10
# how noisy wire-cell should be. "debug" is more, "error" is less
wcloglvl: info
</pre>
</div>
<p>
To customize,
</p>
<pre class="example" id="orgc2a6455">
$ cp toyzero.yaml mycfg.yaml
$ emacs mycfg.yaml
$ snakemake -jall --configfile mycfg.yaml all
</pre>
<p>
Files carry identical names between the different variant runs. To
avoid collision be sure to set <code>datadir</code> and <code>plotdir</code> to locations not
yet used.
</p>
<p>
You may also customize specific parameters from the command line:
</p>
<pre class="example" id="orgc1a15ef">
$ snakemake -jall --config outdir=junk ntracks=10 all_depos
$ eom junk/plots/depos-diagnostic.png
</pre>
<p>
A <code>--config</code> wins over an entry in a <code>--configfile</code>. Note, a single
<code>--config</code> flag should be used to indicate all parameter overrides.
Adding another <code>--config</code> will negate all prior.
</p>
</div>
</div>
<div id="outline-container-snake" class="outline-2">
<h2 id="snake"><span class="section-number-2">7</span> Snakemake</h2>
<div class="outline-text-2" id="text-snake">
<p>
As already introduced, this "toyzero" package is organized around a
<code>snakemake</code> <a href="Snakefile">Snakefile</a> which provides automated running of a
<i>directed, acyclic graph</i> (DAG). A node of the DAG is a process and
directional edges are formed through input or output files.
</p>
<p>
It is instructive to visualize the overall DAG and <code>snakemake</code> provides
this by running:
</p>
<div class="org-src-container">
<pre class="src src-shell">snakemake --dag all just_images > toyzero-all-dag.dot
dot -Tsvg -o toyzero-all-dag.svg toyzero-all-dag.dot
dot -Tpdf -o toyzero-all-dag.pdf toyzero-all-dag.dot
</pre>
</div>
<p>
Below is the result as SVG or download the <a href="toyzero-all-dag.pdf">toyzero-all-dag.pdf</a> to
zoom around.
</p>
<div id="orgeeb4163" class="figure">
<p><object type="image/svg+xml" data="toyzero-all-dag.svg" class="org-svg" width="90%">
Sorry, your browser does not support SVG.</object>
</p>
</div>
<p>
The "real" documentation on toyzero <b>is</b> the Snakefile as it describes
exactly what is run. In principle, with the Snakefile one can always
reproduce its results. In practice changes in the <code>wire-cell</code> and
<code>wirecell-*</code> programs, or external data can lead to unexpected changes.
</p>
</div>
</div>
<div id="outline-container-hack" class="outline-2">
<h2 id="hack"><span class="section-number-2">8</span> Hacking</h2>
<div class="outline-text-2" id="text-hack">
<p>
Much of the <code>toyzero</code> package relies on the <code>wire-cell-python</code> package.
You can develop that code locally (instead of push to GitHub +
reinstall). We can leverage the <code>direnv</code> environment for <code>toyzero</code> to do
so like:
</p>
<pre class="example" id="org4d06fd2">
$ git clone [email protected]:wirecell/wire-cell-python.git
$ cd wire-cell-python/
$ pip install -U -e .
</pre>
<p>
You can now hack on wire-cell-python and your changes are immediately
made available. If the changes are generally useful, please consider
making a PR!
</p>
</div>
</div>
</div>
<div id="postamble" class="status">