tweak documentation, create doc folder

docs/parallelism_design.md

@@ -0,0 +1,22 @@
+### our paralellism strategy:
+
+*subject to change*
+
+use nextflow processes (dataflow) to use multinode+multiprocessing parallelise concurrently across samples/files
+
+we work to enable multinode, but for now we just use 1 node
+
+for now, we avoid splitting files for more parallelism
+
+instead we use tool-level threading
+
+so we run multiple nf-processes
+
+each nf-process runs a single tool process 
+
+each tool process runs the tool with the tools parallelism option
+
+
+with the current filesize to count ratio: 1GB to 5 GB per file and about 5-50 files per workflow on ~100 cores this works pretty well
+
+and most importantly it is quite simple to reason about

readme.md

@@ -63,7 +63,7 @@ cd $NEXTFLOW_CALLDIR && nextflow run $NEXTFLOW_MODULES/ukb_main_workflow/main.nf
 
 ## configuration
 
-see the environment variables and nextflow-configs like modules/ukb_main_workflow/user.config
+see the environment variables and nextflow-configs like `modules/ukb_main_workflow/user.config`
 
 
 ## development
@@ -146,26 +146,3 @@ export NXF_LOG_FILE='/path/nextflow_logs'
 export NXF_PLUGINS_DIR='/path/nextflow_plugins'
 export NEXTFLOW_MODULES="$(pwd)/modules"
 ```
-
-### our paralellism strategy:
-
-*subject to change*
-
-use nextflow processes (dataflow) to use multinode+multiprocessing parallelise concurrently across samples/files
-
-we work to enable multinode, but for now we just use 1 node
-
-for now, we avoid splitting files for more parallelism
-
-instead we use tool-level threading
-
-so we run multiple nf-processes
-
-each nf-process runs a single tool process 
-
-each tool process runs the tool with the tools parallelism option
-
-
-with the current filesize to count ratio: 1GB to 5 GB per file and about 5-50 files per workflow on ~100 cores this works pretty well
-
-and most importantly it is quite simple to reason about