Ugh, this should be a draft

Signed-off-by: Richard Si <[email protected]>

content/blog/2025/ddrive/index.md

@@ -6,7 +6,8 @@ description: &desc >-
   By simply moving all temporary file I/O onto the D: drive, pip's
   Windows CI times decreased by up to 30%.
 summary: *desc
-date: 2025-01-05
+date: 2025-01-08
+draft: true
 tags: [pip]
 ---
 
@@ -24,12 +25,14 @@ my attention on something entirely different while waiting for CI to pass. That
 over the pain, but it got me wondering, can pip's tests (and CI in particular)
 be faster?
 
+[auto-merging on the repository]: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/automatically-merging-a-pull-request
+
 It turns out for the Windows CI jobs, the answer was: **yes, they could be _much_ faster.**
 
 ## How slow was pip's Windows CI?
 
 [trimmed redundant work]: https://github.com/psf/black/pull/2205
-[added `pytest-xdist` for parallelized testing ]: https://github.com/psf/black/pull/2196
+[added pytest-xdist for parallelized testing ]: https://github.com/psf/black/pull/2196
 
 At the time of writing, pip's Github Actions CI workflow consists of 28 jobs. 23
 run the test suite across Ubuntu, macOS, Windows, and CPython 3.8 through 3.13.
@@ -42,15 +45,18 @@ run the test suite across Ubuntu, macOS, Windows, and CPython 3.8 through 3.13.
 The extra Ubuntu job runs the functional test suite against a [zipapp] version
 of pip.
 
-On the other hand, the Windows jobs have been so slow historically[citation-needed]
+[zipapp]: https://pip.pypa.io/en/stable/installation/#standalone-zip-application
+
+On the other hand, the Windows jobs have been so slow historically
 that we don't even test pip across the entire supported Python version matrix.
 We only test the oldest and newest version, 3.8 and 3.13, on Windows. Plus, the
 test suite is [sharded over two jobs per version] because—as you can probably guess—the Windows jobs are
-that much slower. 😔
+that much slower 🐌.
+
+[sharded over two jobs per version]: https://github.com/pypa/pip/blob/ffbf6f0ce61170d6437ad5ff3a90086200ba9e2a/.github/workflows/ci.yml#L183-L185
 
 I wrote a script that queries the GitHub API and calculates the average job duration
-over the last X successful runs. Before the changes discussed later, this is what pip's
-CI looked like:
+over the last X successful runs. As a baseline, this is what pip's CI looked like:
 
 ```text { hl_lines=[15, 17, 25, 27] }
                     Last 50 CI runs on main
@@ -85,13 +91,11 @@ CI looked like:
 Among the slowest three jobs, the Windows 3.8 jobs are 3rd and
 1st slowest. Yikes!
 
-Let's see if I can make that faster...
-
 ### Windows file I/O is poor on GitHub Actions
 
-Now, I am no Windows expert. I stopped using the OS on a daily basis
-many years ago. Either way, [it's a **known issue that file I/O on
-the Windows runners is noticeably slower**][slow-windows-io] than the Ubuntu and macOS runners.
+[**It's a known issue that file I/O on the Windows runners is noticeably slower**][slow-windows-io] than the Ubuntu and macOS runners.
+
+[slow-windows-io]: https://github.com/actions/runner-images/issues/8755
 
 As a package installer, **pip's test suite is especially I/O heavy**. Essentially
 every functional test requires at least setting a scratch environment and
@@ -106,9 +110,10 @@ faster D: drive for working space**.
 For many projects, this is _probably_ fine. The repository is still checked out on the D: drive.
 Any files you create in the checkout will be on the D: drive. For pip, however, all of the
 temporary files are created under the system temporary file path, i.e. the `TEMP` environment
-variable. And where is this path set to by default? On the C: drive.[^we-set-it-explicitly]
+variable. And where is this path set to by default? On the C: drive at `C:/Users/runneradmin/AppData/Local/Temp/`.[^we-set-it-explicitly]
 
-Thus, by setting `TEMP` to `D:\\Temp`, the average Windows CI times decreased by up to 30%. 🎉
+Thus, pip's CI was being slowed down by reading and writing primarily to the non-performant disk.
+By setting it to `D:/Temp`, the average Windows CI times decreased by up to 30%. 🎉
 
 | Job | Before | After | % decrease  |
 |---|---|---|---|
@@ -117,7 +122,7 @@ Thus, by setting `TEMP` to `D:\\Temp`, the average Windows CI times decreased by
 | Python 3.13 (1) | 11m 13s | 9m 53s | 11% |
 | Python 3.13 (2) | 10m 39s | 9m 32s | 10% |
 
-Or in graph form (**note**: that the data used for this plot was collected essentially yesterday).
+Or in graph form (**note**: the data shown here is slightly more recent than the statistics).
 
 ![image](windows-ci-times.png)
 
@@ -126,7 +131,10 @@ Or in graph form (**note**: that the data used for this plot was collected essen
 
 ## What about ReFS or Dev Drives?
 
-When I was investigating
+When I was investigating ways to improve file I/O performance on Windows, I'd discovered
+Dev Drives. Now, I am no Windows expert. I stopped using the OS on a daily basis
+many years ago, but the TL;DR Dev Drives are a new kind of storage volume, built on the
+modern ReFS file system and designed with developer workloads in mind.
 
 ## Conclusion
 
@@ -143,7 +151,13 @@ clean-up (which the test suite in dire need of, honestly).
   \
   I did notice that Black's test suite was slow, taking a good few minutes
   on the slow laptop I had at that time. I took a look at the test logic,
-  [trimmed redundant work], and [added `pytest-xdist` for parallelized testing].
+  [trimmed redundant work], and [added pytest-xdist for parallelized testing].
   Afterwards, local test suite runs took less a minute!\
   \
   I was spoiled by Black's fast tests and CI, admittedly.
+
+[^we-set-it-explicitly]: We've actually set it to `C:/Temp` to work around [filename
+  too long errors]. Also, the exact default system temporary directory may vary
+  from worker to worker. I don't feel like double checking this.
+
+[filename too long errors]: https://github.com/pypa/pip/actions/runs/12450425946/job/34757228568/#step:8:2982