Skip to content

Commit

Permalink
update readme for Mac ARM install
Browse files Browse the repository at this point in the history
  • Loading branch information
traversc committed Oct 13, 2024
1 parent 2bfe0c3 commit 301238c
Show file tree
Hide file tree
Showing 3 changed files with 37 additions and 23 deletions.
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,17 @@ package. It is not compatible with the original `qs` format.
install.packages("qs2")
```

On Mac or Linux, you can enable multi-threading by compiling from
On x64 Mac or Linux, you can enable multi-threading by compiling from
source. It is enabled by default on Windows.

``` r
remotes::install_cran("qs2", type = "source", configure.args = " --with-TBB --with-simd=AVX2")
remotes::install_cran("qs2", type = "source", configure.args = "--with-TBB --with-simd=AVX2")
```

On non-x64 systems (e.g. Mac ARM) remove the AVX2 flag.

``` r
remotes::install_cran("qs2", type = "source", configure.args = "--with-TBB")
```

Multi-threading in `qs2` uses the `Intel Thread Building Blocks`
Expand Down
40 changes: 21 additions & 19 deletions vignettes/vignette.html
Original file line number Diff line number Diff line change
Expand Up @@ -365,40 +365,42 @@ <h1 class="title toc-ignore">qs2</h1>
<div id="installation" class="section level2">
<h2>Installation</h2>
<div class="sourceCode" id="cb2"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true"></a><span class="kw">install.packages</span>(<span class="st">&quot;qs2&quot;</span>)</span></code></pre></div>
<p>On Mac or Linux, you can enable multi-threading by compiling from source. It is enabled by default on Windows.</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true"></a>remotes<span class="op">::</span><span class="kw">install_cran</span>(<span class="st">&quot;qs2&quot;</span>, <span class="dt">type =</span> <span class="st">&quot;source&quot;</span>, <span class="dt">configure.args =</span> <span class="st">&quot; --with-TBB --with-simd=AVX2&quot;</span>)</span></code></pre></div>
<p>On x64 Mac or Linux, you can enable multi-threading by compiling from source. It is enabled by default on Windows.</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true"></a>remotes<span class="op">::</span><span class="kw">install_cran</span>(<span class="st">&quot;qs2&quot;</span>, <span class="dt">type =</span> <span class="st">&quot;source&quot;</span>, <span class="dt">configure.args =</span> <span class="st">&quot;--with-TBB --with-simd=AVX2&quot;</span>)</span></code></pre></div>
<p>On non-x64 systems (e.g. Mac ARM) remove the AVX2 flag.</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true"></a>remotes<span class="op">::</span><span class="kw">install_cran</span>(<span class="st">&quot;qs2&quot;</span>, <span class="dt">type =</span> <span class="st">&quot;source&quot;</span>, <span class="dt">configure.args =</span> <span class="st">&quot;--with-TBB&quot;</span>)</span></code></pre></div>
<p>Multi-threading in <code>qs2</code> uses the <code>Intel Thread Building Blocks</code> framework via the <code>RcppParallel</code> package.</p>
</div>
<div id="converting-qs2-to-rds" class="section level2">
<h2>Converting qs2 to RDS</h2>
<p>Because the <code>qs2</code> format directly uses R serialization, you can convert it to RDS and vice versa.</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true"></a>file_qs2 &lt;-<span class="st"> </span><span class="kw">tempfile</span>(<span class="dt">fileext =</span> <span class="st">&quot;.qs2&quot;</span>)</span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true"></a>file_rds &lt;-<span class="st"> </span><span class="kw">tempfile</span>(<span class="dt">fileext =</span> <span class="st">&quot;.RDS&quot;</span>)</span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true"></a>x &lt;-<span class="st"> </span><span class="kw">runif</span>(<span class="fl">1e6</span>)</span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true"></a></span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true"></a><span class="co"># save `x` with qs_save</span></span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true"></a><span class="kw">qs_save</span>(x, file_qs2)</span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true"></a></span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true"></a><span class="co"># convert the file to RDS</span></span>
<span id="cb4-9"><a href="#cb4-9" aria-hidden="true"></a><span class="kw">qs_to_rds</span>(<span class="dt">input_file =</span> file_qs2, <span class="dt">output_file =</span> file_rds)</span>
<span id="cb4-10"><a href="#cb4-10" aria-hidden="true"></a></span>
<span id="cb4-11"><a href="#cb4-11" aria-hidden="true"></a><span class="co"># read `x` back in with `readRDS`</span></span>
<span id="cb4-12"><a href="#cb4-12" aria-hidden="true"></a>xrds &lt;-<span class="st"> </span><span class="kw">readRDS</span>(file_rds)</span>
<span id="cb4-13"><a href="#cb4-13" aria-hidden="true"></a><span class="kw">stopifnot</span>(<span class="kw">identical</span>(x, xrds))</span></code></pre></div>
<div class="sourceCode" id="cb5"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true"></a>file_qs2 &lt;-<span class="st"> </span><span class="kw">tempfile</span>(<span class="dt">fileext =</span> <span class="st">&quot;.qs2&quot;</span>)</span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true"></a>file_rds &lt;-<span class="st"> </span><span class="kw">tempfile</span>(<span class="dt">fileext =</span> <span class="st">&quot;.RDS&quot;</span>)</span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true"></a>x &lt;-<span class="st"> </span><span class="kw">runif</span>(<span class="fl">1e6</span>)</span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true"></a></span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true"></a><span class="co"># save `x` with qs_save</span></span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true"></a><span class="kw">qs_save</span>(x, file_qs2)</span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true"></a></span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true"></a><span class="co"># convert the file to RDS</span></span>
<span id="cb5-9"><a href="#cb5-9" aria-hidden="true"></a><span class="kw">qs_to_rds</span>(<span class="dt">input_file =</span> file_qs2, <span class="dt">output_file =</span> file_rds)</span>
<span id="cb5-10"><a href="#cb5-10" aria-hidden="true"></a></span>
<span id="cb5-11"><a href="#cb5-11" aria-hidden="true"></a><span class="co"># read `x` back in with `readRDS`</span></span>
<span id="cb5-12"><a href="#cb5-12" aria-hidden="true"></a>xrds &lt;-<span class="st"> </span><span class="kw">readRDS</span>(file_rds)</span>
<span id="cb5-13"><a href="#cb5-13" aria-hidden="true"></a><span class="kw">stopifnot</span>(<span class="kw">identical</span>(x, xrds))</span></code></pre></div>
</div>
<div id="validating-file-integrity" class="section level2">
<h2>Validating file integrity</h2>
<p>The <code>qs2</code> format saves an internal checksum. This can be used to test for file corruption before deserialization via the <code>validate_checksum</code> parameter, but has a minor performance penalty.</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true"></a><span class="kw">qs_save</span>(data, <span class="st">&quot;myfile.qs2&quot;</span>)</span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true"></a>data &lt;-<span class="st"> </span><span class="kw">qs_read</span>(<span class="st">&quot;myfile.qs2&quot;</span>, <span class="dt">validate_checksum =</span> <span class="ot">TRUE</span>)</span></code></pre></div>
<div class="sourceCode" id="cb6"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true"></a><span class="kw">qs_save</span>(data, <span class="st">&quot;myfile.qs2&quot;</span>)</span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true"></a>data &lt;-<span class="st"> </span><span class="kw">qs_read</span>(<span class="st">&quot;myfile.qs2&quot;</span>, <span class="dt">validate_checksum =</span> <span class="ot">TRUE</span>)</span></code></pre></div>
</div>
<div id="the-qdata-format" class="section level1">
<h1>The qdata format</h1>
<p>The package also introduces the <code>qdata</code> format which has its own serialization layout and works with only data types (vectors, lists, data frames, matrices).</p>
<p>It will replace internal types (functions, promises, external pointers, environments, objects) with NULL. The <code>qdata</code> format differs from the <code>qs2</code> format in that it is NOT a general.</p>
<p>The eventual goal of <code>qdata</code> is to also have interoperability with other languages, particularly <code>Python</code>.</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true"></a><span class="kw">qd_save</span>(data, <span class="st">&quot;myfile.qs2&quot;</span>)</span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true"></a>data &lt;-<span class="st"> </span><span class="kw">qd_read</span>(<span class="st">&quot;myfile.qs2&quot;</span>)</span></code></pre></div>
<div class="sourceCode" id="cb7"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true"></a><span class="kw">qd_save</span>(data, <span class="st">&quot;myfile.qs2&quot;</span>)</span>
<span id="cb7-2"><a href="#cb7-2" aria-hidden="true"></a>data &lt;-<span class="st"> </span><span class="kw">qd_read</span>(<span class="st">&quot;myfile.qs2&quot;</span>)</span></code></pre></div>
<div id="benchmarks" class="section level2">
<h2>Benchmarks</h2>
<p>A summary across 4 datasets is presented below.</p>
Expand Down
10 changes: 8 additions & 2 deletions vignettes/vignette.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,16 @@ Use the file extension `qs2` to distinguish it from the original `qs` package. I
install.packages("qs2")
```

On Mac or Linux, you can enable multi-threading by compiling from source. It is enabled by default on Windows.
On x64 Mac or Linux, you can enable multi-threading by compiling from source. It is enabled by default on Windows.

```{r eval=FALSE}
remotes::install_cran("qs2", type = "source", configure.args = " --with-TBB --with-simd=AVX2")
remotes::install_cran("qs2", type = "source", configure.args = "--with-TBB --with-simd=AVX2")
```

On non-x64 systems (e.g. Mac ARM) remove the AVX2 flag.

```{r eval=FALSE}
remotes::install_cran("qs2", type = "source", configure.args = "--with-TBB")
```

Multi-threading in `qs2` uses the `Intel Thread Building Blocks` framework via the `RcppParallel` package.
Expand Down

0 comments on commit 301238c

Please sign in to comment.