A memory efficient implementation of the .mtx reading function

[gjeuken]

• Closes #
• Tests included or not required because: test_datasets.py already implemented

• Release notes not necessary because: This is a backend change

Pandas read_csv function is very memory intensive, and this makes loading data (especially large datasets from EBI Single Cell Expression Atlas) impossible on computers with 16gb of ram or less. The subsequent analysis of such datasets with scanpy, however, works well on such computers.

Loading the data into chunks, using the same pandas function, solves this problem.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.41%. Comparing base (9741ca6) to head (73c2a21).
Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3389      +/-   ##
==========================================
- Coverage   75.44%   75.41%   -0.04%     
==========================================
  Files         113      113              
  Lines       13250    13266      +16     
==========================================
+ Hits         9997    10005       +8     
- Misses       3253     3261       +8     

Files with missing lines	Coverage Δ
src/scanpy/datasets/_ebi_expression_atlas.py	94.44% <100.00%> (+0.46%)	⬆️

... and 3 files with indirect coverage changes

[flying-sheep]

Good idea! some small notes:

[gjeuken]

Please note that the selection of the chunk size of 1e7 is rather arbitrary.
This value solves the loading issue for computers with 8 or 16gb of RAM without looping through too many chunks.

[flying-sheep]

Thank you! Since you don’t seem interested in appearing in a release note:

… I’ll merge this as-is. If you want a release note after all, please comment, and I’ll add one!