Draft SpatialData.filter()

[aeisenbarth]

(In reference to #620)

This PR imlements an more advanced filtering options than subset, allowing to create a new SpatialData object that contains only specific tables, layers, obs keys, var keys.

Use cases

• From a concatenated SpatialData, one can extract parts of it.
• When testing an operation that adds elements or table columns, one can extract from an expected reference dataset the input data and pass it to the operation, then compare the processed data against the reference.
• …

Closes #280
Closes #284
Closes #556

[codecov]

Codecov Report

Attention: Patch coverage is 10.71429% with 25 lines in your changes missing coverage. Please review.

Project coverage is 91.59%. Comparing base (95d69ff) to head (d9c1e0e).
Report is 49 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_core/spatialdata.py	10.71%	25 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #626      +/-   ##
==========================================
- Coverage   91.93%   91.59%   -0.35%     
==========================================
  Files          44       44              
  Lines        6661     6688      +27     
==========================================
+ Hits         6124     6126       +2     
- Misses        537      562      +25     

Files with missing lines	Coverage Δ
src/spatialdata/_core/spatialdata.py	88.51% <10.71%> (-2.47%)	⬇️

[aeisenbarth]

In the current state, it does not yet complete the issues that were aimed to resolve.

• Subset spatialdata by list of cell ids #556:
  • A parameter instances could be added. If provided, rows of these instances will be selected in the table, if not provided, all instances are returned. But it should only allow a single region/element (otherwise it gets complicated).
  • Shapes/points elements should also be filtered (easy)
  • Still unanswered is what effect it should have on the labels. Shall we create a new labels image with the instances eliminated (0 = background), or leave the labels unchanged and just have no reference to them in the table?
• Filter spatialData  #280:
  • This involves a condition. In my opinion, implementing a parameter with a condition as function or query expression is out of scope due to its complexity. I would do this in two steps, users use Pandas to get a list of instances, then pass the instances to the SpatialData filter function.
  • The user also asked about adjusting shapes to match the filtered instances in the table.
• Feature request: spatial cropping from select table rows #284:
  • Filtering labels/shapes/points elements

[LucaMarconato]

Thanks @aeisenbarth, after discussing with @melonora, we are going to first turn the code #627 into an internal function, merge, and then continue working on your PR. The idea is to provide a single entry point for filtering filter() and use for instance subset() or the function from Wouter internally.

[owenwilkins]

is there somewhere is the domentation that now describes how to filter a spatialdata object by cell IDs? this is valuable for several reasons, e.g. filtering cells removed by QC in analysis using other libraries

[LucaMarconato]

I went back to this and to #627 today and realized that we maybe do not need to add a new API, since all the points covered by this PR and by the linked PR, including all the points listed in this message here: #626 (comment) are essentially covered by the example below, which uses the currently available APIs:

##
# constructing the example data
from spatialdata.datasets import blobs_annotating_element
from spatialdata import concatenate
from spatialdata import join_spatialelement_table
from spatialdata import SpatialData

sdata1 = blobs_annotating_element("blobs_polygons")
sdata2 = blobs_annotating_element("blobs_polygons")

sdata = concatenate({"sdata1": sdata1, "sdata2": sdata2}, concatenate_tables=True)
print(sdata)

##
# filtering the data
table_name = "table"
filtered_table = sdata[table_name][sdata[table_name].obs.instance_id < 3]
annotated_regions = sdata.get_annotated_regions(sdata[table_name])
elements, table = join_spatialelement_table(
    sdata, spatial_element_names=annotated_regions, table=filtered_table, how="inner"
)
sdata_filtered = SpatialData.init_from_elements(elements | {table_name: table})
print(sdata_filtered)

Explicitly, the code above first filters the table with standard pandas/anndata operations (and thus is very general), and then reuses the join operations to filter the SpatialData object (which again are very general and allow for several cases). In doing this:

• we limit code redundancy
• we can use any query/condition to filter the object, including:
  • we can filter by a threshold
  • we can filter by instances
  • we can manually subset certain obs/var/layers of the table
• shapes and points are filtered to match the table
• if a table annotates multiple elements, they are all filtered
• filtering labels is delegated to the join operations (currently not supported, but if in the future it will, it would be also supported here)

I think we could proceed by choosing one of the following strategies:

1. we do not add any new API, and put the example above visible in the docs
2. we add a very minimalistic API that essentially reproduces the example above. So convenient for most cases, and for more general cases the user can modify the code
3. we add a feature complete API (similar to this PR); we use the code above internally so we don't have code redundancy.

Any preference?

[aeisenbarth]

This PR indeed overlaps with existing APIs.

I agree that redundancy should be avoided. But I think an API should help to minimize gluing code. The above example solves the task, but contains two filtering steps and two intermediate function calls.

So from my side, I would favor extending the existing API to be more feature-complete. But I would not see it high priority.

[LucaMarconato]

I agree that redundancy should be avoided. But I think an API should help to minimize gluing code. The above example solves the task, but contains two filtering steps and two intermediate function calls.

You are right, that approach is not ergonomic enough. I thought about this and now in the linked PR #627 I introduce a new API match_sdata_to_table() that bundles the code (@aeisenbarth could you please review the PR?).

All the use cases mentioned in your message are included in the tests. I wonder if now the function is ergonomic enough or if we should still add a filter() function.

Here are my thoughts on this:

Ergonomics limitation of the match_sdata_to_table()

• you need to import the function and pass the sdata argument to it; a method in SpatialData could be added to fix this.
• while its usage is powerful and simple, as the tests linked above show, it may not be immediate how to use it without looking at the tests. We could add some examples in the docstring for this.
• the name tells exactly what the function does, but it's not what people would look for when trying to subset a SpatialData object by a threshold on obs/var.

Limitations of the SpatialData.filter()

• filter may suggest a threshold-based filtering, which we probably do not want to implement as the same result can be achieved via pandas/anndata APIs + match_sdata_to_table(), as shown in this test .
• code redundancy

my preferred take

I think my preferred approach would be to gather feedback from the users, and then actually provide a new function (what would be the best name filter()? query()? subset()? match()?) that simplifies the usage of match_sdata_to_table(). This is in line to what reported here: #225. I'd consider strategizing this by closing the current PR and reworking the code when we are ready into a new function, that eventually covers also the bounding box and polygon query cases. But happy to hear what you think @aeisenbarth @melonora. In particular if you think that it would be much more intuitive to have already a filter()/query()/subset() function that mimics what this PR does and that calls match_sdata_to_table() internally, then we could do this directly in the context of this PR.