slides.Rmd

---
title: "R Data Packages"
subtitle: "Denver R User Group"
author: "Peter E. DeWitt, Ph.D.<br><peter.dewitt@cuanschutz.edu>"
date: 22 October 2019
output:
 ioslides_presentation:
   keep_md: false
   widescreen: true
   logo: drug-logo.jpeg
   css: style.css
   self_contained: true
---


```{r label = "setup", include = FALSE}
knitr::opts_chunk$set(collapse = TRUE)
```


# Data Packages

## Data Packages are Common
* Common for data sets to be included in packages
    * Useful for examples and documentation
    * Some R packages are just data sets, nothing else.

```{r size = "tiny"}
head(data(package = "ggplot2")$results, 2)

head(data(package = "datasets")$results, 2)
```



# Problem to solve

## Reasons to build and use R Data Packages

1. Commonly used data, e.g., mtcars
    * Good for active work
    * Archiving data sets
2. Limited Memory (RAM and or disk space)
3. Collaborating on data which cannot be hosted on a central location
    * Cannot afford the hardware and admin costs
    * Sensitive data which needs to stay off line

* All scenarios can support version control of data

# My Motivating Problem

## More data than disk space

| Data Set                | Compressed Size     | Uncompressed Size  |
| :---------              | ------------------: | -----------------: |
| Data Set A              |             59.0 GB |            568 GB  |
| Data Set B              |              4.3 GB |             23 GB  |
| Data Set C              |             41.0 GB |            144 GB  |
| Data Set D              |             36.0 GB |             81 GB  |
| Additional common files |           *trivial* |         *trivial*  |
|                         | **total:** 140.3 GB | **total:** 816 GB  |

* Computing Resources: one laptop
 * Intel core i9 8th Gen / 916 GB hard drive space / 32 GB RAM (+32 GB SWAP)

## More data than disk space

* **956 GB of data (compressed & decompressed) > 916 GB hard drive**

* Need space for OS, applications, other projects
* Version control the data?  Back ups?

* Data use agreements:
    * All data stays on Institution owned machines.
    * Cloud storage/computing resources are not an option.

* Operating budget: $0

* Self Imposed Requirement: *Reproducible Reports*
    * Fully documented -- literate programming and dynamic documents
    * Easy to reproduce -- rely on GNU make

* Common solution for each of the major data sets

## Possible Solutions | Compressed data in the package

* Version controlled (compressed data is versioned)
* Decompress to temp directory
* Import data into spark cluster without caching the data

## Possible Solutions | Compressed data in the package

* Pros
    * temp directory will be unlinked with R session exits without error,
    * temp directory will be unlinked on system restart
    * keeps disk open for other data sets and general work.
    * easy to part an analysis into parts
    * scalable

* Cons
    * slow (compared to a DB, server, etc.)

* Slow is better than impossible
* This was the solution I used initially.

## Structure

<div class="box13">
Package Structure
<iframe src="tree1.txt.html"></iframe>
</div>
<div class="box23">
import.R
<iframe src="import.R.html"></iframe>
</div>

## Possible Solutions | Tools for verifying data and importing

I am working on a public project

* R package for downloading and installing the Physician/Supplier Procedure
Summary Data provided by the Centers for Medicare and Medicaid

* Focus on tools for importing the data via data.table or spark

* Let's look at
     * github.com/dewittpe/cms.psps and
     * github.com/dewittpe/cms.program.statistics

# Another Data Problem

## Archive and Publicly Release Data

* A prospective observational research study on pediatric injuries needs to
be:

    1. Protected Health Information (PHI) needs to be scrubbed from the data
    before release.
    2. Secondary potential identifiers need to be scrubbed before release.
    3. Data needs to be documented and published publicly.

* Archived data will be easy to distribute.
* Easy to make sure everyone has the save version of the data.

# Version Control

## Use the data package version number

* Public packages will require well formed and "short" version numbers

```{r }
packageVersion("cms.psps")
packageVersion("cms.psps") == "0.0.0.9000"
packageVersion("cms.psps") >  "0.0.1"
packageVersion("cms.psps") >  "0.0.0.1"
```


* Non public packages can use longer verison numbers, i.e., YYYY.MM.DD.HHMM


```{r }
# packageDescription("<package.name>") == "2019.10.22.1743"
```


# Thank you for your time

<img width=25% src="drug-logo.jpeg" align='left'> </img>

**_Questions?_**

**_Comments?_**

**_Other Solutions?_**