update with notes

case-os.qmd

@@ -6,10 +6,38 @@ Chairs: Satish Murphy & Becca Krouse
 
 What is stopping people and companies from contributions in OS? Can we define the case for contributing to OS?
 
-::: {.callout-warning}
+# Main concerns
 
-## Missing notes
+## Licencing
 
-Content is still coming, an email will be shared once the site is complete.
+- R's core is released on a copyleft GPL license, so there are concerns around any additions we make being required to publish back to the community. This is a concern for companies that are not in the business of selling software, but rather use software as a tool to do their business.
+- Python is seen as less of risk, as it is released via permissive licence
+- Is there really a legal risk for copy-left licences, and if so under what circumstances? (e.g. using it internally for a plot vs making and selling an app or algorithm that uses the copyleft dependency)
+- Can we understand better how IP is protected in OS?
 
-:::
+## Process
+
+- When using an internal package in a filing, we can package it directly as part of the study code and give to the FDA - which is seen as less of a risk than publishing it as OS where the global community can view the code.
+- Folks often want to contribute - but there are some limitations both professionally (e.g. internal process to contribute) and personally
+- If someone works on a project in their own time, there is a concern that treated is a company asset even though they are doing it outside of working hours. What is the actual boundary between work and personal contributions?
+- People making the policies need to actually understand the topic of OS
+
+## Liability
+
+- We know the liability is similar between OS and proprietary software, but there is a perception that all OS is more risky
+
+## People
+
+- OS contributions are seen as nice to have - how can this be prioritised vs project work?
+- Be involved in projects like Oak, NEST and admiral brings recognition to the contributors 
+- Often projects are mostly driven by a handful, or even a couple, of people. What if someone leaves? Is OS actually a benefit here as same developer could promote and lead to use of the package at their new company?
+- Write up a short post / article titled “Here’s why you should allow your people to contribute to open source”
+- Write a blog post and short PDF that can be shared internally at the leadership level
+
+## Resources
+
+- [PHUSE Open Source guidance](https://phuse-org.github.io/E2E-OS-Guidance/slides/eu23)
+
+## What can we do?
+
+- Create a framework to help articulate the benefit, and help to tackle the concerns/process that gets in the way

definitions.qmd

@@ -2,5 +2,7 @@
 
 | **Abbreviation** | **Description** |
 | ---------------- | ----------------|
+| CSR | Clinical Study Report |
 | SCE | Statistical/Scientific Computing Environment |
+| OS | Open Source |
 

index.qmd

@@ -37,4 +37,6 @@ following topics:
 
 [CAMIS - comparing differences based on tool used](https://psiaims.github.io/CAMIS/)
 
+[PHUSE Open Source guidance](https://phuse-org.github.io/E2E-OS-Guidance/)
+
 :::

modern-sce.qmd

@@ -27,7 +27,7 @@ What are our goals for a modern clinical reporting workflow, on a modern SCE? Wh
     - Split across companies in the group, with some companies having a single platform for both, and others having seperate platforms.
     - With initiatives like the digital protocol coming, we don't know what the impact will be on routine clinical reporting, and what impacts this will have on the types of people and tasks needed to execute a CSR.
     - Pain points merging:
-        - Validation (CSV) is a long and high cost process in most companaies, which can impact ability to support exploratory work.
+        - Validation (CSV) is a long and high cost process in most companies, which can impact ability to support exploratory work.
         - Needs are different. E.g. clinical reporting is low compute, while design and biomarker work is often heavy in memory and data.
 - Is data part of the SCE? Traditionally yes, but some but not all companies are de-coupling data from compute.
 - Whether it's in the SCE or not, tracebility is extra important in our domain of regulatory reporting.
@@ -40,10 +40,10 @@ What are our goals for a modern clinical reporting workflow, on a modern SCE? Wh
 ## Change-management into a modern SCE
 
 - What are we actually building? A general data science platform? A platform optimised for clinical reporting?
-    - These are not the same platform, and which you pick has an impact. e.g. should statisticals programmers learn git, or should we give a simple GUI for pushing code through QC and to Prod?
+    - These are not the same platform, and which you pick has an impact. e.g. should statistical programmers learn git, or should we give a simple GUI for pushing code through QC and to Prod?
     - There is not a consensus about this for next-gen, with only a handful of companies expecting statistical programmers to work in the same way as general data scientists.
 - Historically we depended on SAS, it's data formats, and filesystems. How to build a modern SCE that doesn't?
-    - 1) Do we enable legacy workflows to work in the new SCE? 2) Only new ways, or 3) how do we find a balance to ensure business continuity while enabling innovation?
+    - Do we enable legacy workflows to work in the new SCE?; Only new ways; or  how do we find a balance to ensure business continuity while enabling innovation?
     - The human and process change management piece is massive, and SCE POs must work in tandem with statistical programming leadership.
     - Agreement the biggest pain point is the dependency on file-based network share drives for data and insight outputs. One company mentioned they have millions of directories in their legacy SCE.
 - Most companies have carried over having the outputs server be a network share drive, but would a more 'publishing' type model be more robust?

os-depends.qmd

@@ -12,4 +12,6 @@ How much risk is there in depending on external packages, and can we foster a cl
 
 Content is still coming, an email will be shared once the site is complete.
 
+In the interim - the [PHUSE Open Source guidance includes a chapter on  depending on Open Source](https://phuse-org.github.io/E2E-OS-Guidance/using.html).
+
 :::

shiny-csr.qmd

@@ -6,10 +6,39 @@ Chairs: Ning Leng and Phil Bowsher
 
 If we assume it's technically possible to transfer - what would it mean to give an interactive CSR? How would primary, secondary and ad-hoc analysis be viewed? Would views change depending on role (e.g. sponsor, statistical reviewer, clinical reviewer)?
 
-::: {.callout-warning}
+# Why?
 
-## Missing notes
+The current process of CSR generation and review involves generating large amounts of tables and plots. In such scenarios, we believe an interactive application would provide a more efficient method to explore both primary and secondary analysis results. 
+
+# Barriers
+
+- Internally
+    - Most companies have a highly process driven to medical writing, where any change is difficult to implement
+- External / FDA
+    - Concern of encouraging data fishing, if controls on for instance subgroup analysis are not in place?
+    - Need cross industry harmonization as HA is highly unlikely to accept each pharma company submitting an interactive CSR based on a different framework
+
+# Requirements
+
+- Can an interactive CSR by default be limited to only analyses defined in the SAP, should/can exploring a wider scope be allowed with clear and explicit intent by the app user?
+  - How is the ROI of interactivity in a CSR impacted if it's limited to produce only pre-specified anlayses? Would different reviewers have different access? (e.g. statisticla vs clinical) 
+- There is a gap in patient profile modules and patient narratives in shiny apps
+
+# Path to the CSR
+
+There are many places where this tooling can be applied before the CSR:
+
+- Monitoring
+- Decision making - data in clinical science hands
+- Efficiency boost (e.g. at unblinding app is on hand)
+- CSR
+
+Can we define the tangible benefits? What is the ROI over different use cases? Could this allow health authorities to make less requests? How does this lead to faster reviews in tangible terms?
+
+How would access control and security work? How would this be audited? Would this be validated? How would results be archived?
+
+Can commenting be possible (as you can markup a PDF/work doc)?
+
+Existing frameworks exist - e.g. `teal` from NEST. ARDs may mean additional results can be pre-generated in a controlled way - but ARDs can't further process data.
 
-Content is still coming, an email will be shared once the site is complete.
 
-:::

validate-shiny.qmd

@@ -15,6 +15,34 @@ We have a path to R package validation - but what about shiny apps? In what cont
     - Is the results going directly from the app into a submission?
     - Don’t validate a shiny app - validate the static functions in the R packages. CSV may not be relevant for UIs (vs static R packages)
 
+## What are we Testing and Why?
+
+There is a clear difference of opinion throughout the industry, often led by quality groups.  Some companies validate shiny apps as if they were distinct pieces of software, using their internal software validation procedures.  These processes are often outdated and unsuitable, requiring timestamped user-testing and screen captures.
+
+Other companies solely consider packages, not even validating shiny apps, but validating just the logic. The group discussed a preferred way of working – separating the logic and the UI.
+
+This brings up the question – **do we really need to validate shiny apps?  Can we just validate the logic?**
+
+## Who Does the Testing?
+
+Again, there is some difference between companies in who does the testing.  Generally, the developer writes the tests but tests are performed either by the business or by the quality group.
+
+## Use of Automation
+
+Question posed to people present around the table: Does your company’s validation system allow for automation?  Answers from the table: 8 companies = yes, 2 companies = no.  Another 4 companies = no (offered by consultant who works with Pharma companies).  Clearly a range of capabilities across the industry.
+
+From an automation perspective, the Pharma industry is very far behind the technology industry.  Technology codebases tend to be far more complex but they are also automated.  Can we learn from their platforms and apply their processes to validating shiny apps?
+Tools such as {shinytest2} are daunting to use.  Can they be made more user friendly?  There have been some steps to help automate these tasks – eg {shinyvalidator} but more work is needed in this area.
+
+It’s very challenging to validate a reactive graph.  Automated processes have the ability to detect changes in a single pixel – is this desirable or undesirable?
+
+## Types of Testing
+
+There is a clear difference across companies in opinions as to the amount of unit testing vs UAT and end-user testing.  Unit tests are easy to write but are do not demonstrate how an app works.  {shinytest2} can be used for end-user testing but, as mentioned above, may be daunting to use, may not be acceptable within a quality organization and may not fit in current work practices.
+
+Unit tests are generally written as code is written.  They are fast to write and fast to execute.  End-to-end tests, however, are written once code is complete and tend to be slow to execute.
+
+
 ## Robust UIs?
 
 - Good to have unit tests - often manual testing. Automated can easily get messed up as the code evolves.
@@ -26,13 +54,31 @@ We have a path to R package validation - but what about shiny apps? In what cont
 - How to handle risk of UI problems if our focus is on the static code - e.g. misnamed reactive values so wrong values being shown, even if static R packages giving correct results.
 - Risk based is really important - e.g. for something like dark mode breaking, we need to know what requirements are high risk (e.g. table is correct) vs low risk (e.g. dark mode button)
 
+# Ideas to improve the process 
+
+- Validation tests as text files (ATDD/BDD from software engineering).  
+- Frame in [Gherkin format](https://www.guru99.com/gherkin-test-cucumber.html) plus package of fixtures
+- Contribute test code to public packages
+- When companies write extra tests, make a PR to add them to the actual package test suite and get others in the community to review and comment
+- Extend to more than tests – documentation, etc.
+- We need clarity around packages used in submissions.
+- Would big Pharma be willing to list all packages that pass their internal risk-assessment and share?  Also share why they pass/fail a risk-assessment?
+- Validating shiny apps – can we share some cross-industry experience?
+- QA vs validation.  At what stage should I worry about validation?
+    - Can we talk to QA departments / QA senior leadership to get them to write up their thoughts / requirements?  Ask “How can we make your job easier?”  
+- Should we include QA and more IT at next year’s summit?
+
 # Actions
 
+- Can we share some common high level guidance on stratifying risk in shiny shared across companies? (Pfizer has written this already internally). 
+- Discuss if we should have an extension of R package whitepaper to cover shiny?
+
 ::: {.callout-note icon=false}
 
 # {{< fa list >}} 
 
-- Can we share some common high level guidance on stratifying risk in shiny shared across companies? (Pfizer has written this already internally). 
-- Discuss if we should have an extension of R package whitepaper to cover shiny?
+- Establish a CSR working group (first talk to Shiny submissions working group to establish overlap?)
 
 :::
+
+