From a30c621bdfda961f73e137e447015821c5512574 Mon Sep 17 00:00:00 2001 From: cmalinmayor Date: Wed, 21 Aug 2024 19:14:04 +0000 Subject: [PATCH] Commit from GitHub Actions (Build Notebooks) --- exercise.ipynb | 298 ++++++++++++++++++++++++-------------------- solution.ipynb | 329 ++++++++++++++++++++++++++++--------------------- 2 files changed, 353 insertions(+), 274 deletions(-) diff --git a/exercise.ipynb b/exercise.ipynb index 3da2e6c..74ab35e 100644 --- a/exercise.ipynb +++ b/exercise.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "dc64efee", + "id": "fb5945ac", "metadata": {}, "source": [ "# Exercise 9: Tracking-by-detection with an integer linear program (ILP)\n", @@ -45,7 +45,7 @@ }, { "cell_type": "markdown", - "id": "5482c789", + "id": "d7ac7bdb", "metadata": {}, "source": [ "Visualizations on a remote machine\n", @@ -71,7 +71,7 @@ { "cell_type": "code", "execution_count": null, - "id": "6f68cfef", + "id": "72c670b4", "metadata": {}, "outputs": [], "source": [ @@ -81,7 +81,7 @@ }, { "cell_type": "markdown", - "id": "3d91fa7a", + "id": "e7f67965", "metadata": {}, "source": [ "## Import packages" @@ -90,31 +90,24 @@ { "cell_type": "code", "execution_count": null, - "id": "329a3e85", + "id": "e291f63b", "metadata": {}, "outputs": [], "source": [ - "import time\n", - "from pathlib import Path\n", - "\n", "import skimage\n", "import numpy as np\n", "import napari\n", "import networkx as nx\n", "import scipy\n", "\n", - "\n", "import motile\n", "\n", "import zarr\n", - "from motile_toolbox.visualization import to_napari_tracks_layer\n", "from motile_toolbox.candidate_graph import graph_to_nx\n", "from motile_toolbox.visualization.napari_utils import assign_tracklet_ids\n", "import motile_plugin.widgets as plugin_widgets\n", "from motile_plugin.backend.motile_run import MotileRun\n", - "from napari.layers import Tracks\n", "import traccuracy\n", - "from traccuracy import run_metrics\n", "from traccuracy.metrics import CTCMetrics, DivisionMetrics\n", "from traccuracy.matchers import IOUMatcher\n", "from csv import DictReader\n", @@ -126,7 +119,7 @@ }, { "cell_type": "markdown", - "id": "c7d07864", + "id": "d2385ec1", "metadata": {}, "source": [ "## Load the dataset and inspect it in napari" @@ -134,7 +127,7 @@ }, { "cell_type": "markdown", - "id": "22519ef0", + "id": "ab1dede1", "metadata": {}, "source": [ "For this exercise we will be working with a fluorescence microscopy time-lapse of breast cancer cells with stained nuclei (SiR-DNA). It is similar to the dataset at https://zenodo.org/record/4034976#.YwZRCJPP1qt. The raw data, pre-computed segmentations, and detection probabilities are saved in a zarr, and the ground truth tracks are saved in a csv. The segmentation was generated with a pre-trained StartDist model, so there may be some segmentation errors which can affect the tracking process. The detection probabilities also come from StarDist, and are downsampled in x and y by 2 compared to the detections and raw data." @@ -142,7 +135,7 @@ }, { "cell_type": "markdown", - "id": "e2436d52", + "id": "22a39a79", "metadata": {}, "source": [ "Here we load the raw image data, segmentation, and probabilities from the zarr, and view them in napari." @@ -151,7 +144,7 @@ { "cell_type": "code", "execution_count": null, - "id": "19e4e26c", + "id": "bdcb6a05", "metadata": {}, "outputs": [], "source": [ @@ -164,30 +157,30 @@ }, { "cell_type": "markdown", - "id": "dcdf11b1", + "id": "587c87ec", "metadata": {}, "source": [ - "Let's use [napari](https://napari.org/tutorials/fundamentals/getting_started.html) to visualize the data. Napari is a wonderful viewer for imaging data that you can interact with in python, even directly out of jupyter notebooks. If you've never used napari, you might want to take a few minutes to go through [this tutorial](https://napari.org/stable/tutorials/fundamentals/viewer.html)." + "Let's use [napari](https://napari.org/tutorials/fundamentals/getting_started.html) to visualize the data. Napari is a wonderful viewer for imaging data that you can interact with in python, even directly out of jupyter notebooks. If you've never used napari, you might want to take a few minutes to go through [this tutorial](https://napari.org/stable/tutorials/fundamentals/viewer.html). Here we visualize the raw data, the predicted segmentations, and the predicted probabilities as separate layers. You can toggle each layer on and off in the layers list on the left." ] }, { "cell_type": "code", "execution_count": null, - "id": "d80fe285", + "id": "158079c2", "metadata": { "lines_to_next_cell": 1 }, "outputs": [], "source": [ "viewer = napari.Viewer()\n", + "viewer.add_image(probabilities, name=\"probs\", scale=(1, 2, 2))\n", "viewer.add_image(image_data, name=\"raw\")\n", - "viewer.add_labels(segmentation, name=\"seg\")\n", - "viewer.add_image(probabilities, name=\"probs\", scale=(1, 2, 2))" + "viewer.add_labels(segmentation, name=\"seg\")" ] }, { "cell_type": "markdown", - "id": "30616779", + "id": "36c2b808", "metadata": {}, "source": [ "After running the previous cell, open NoMachine and check for an open napari window." @@ -195,7 +188,7 @@ }, { "cell_type": "markdown", - "id": "8f6b0cb2", + "id": "8d1590df", "metadata": {}, "source": [ "## Read in the ground truth graph\n", @@ -213,7 +206,7 @@ }, { "cell_type": "markdown", - "id": "c291b3d4", + "id": "e6bf078f", "metadata": {}, "source": [ "\n", @@ -237,7 +230,7 @@ { "cell_type": "code", "execution_count": null, - "id": "dcaf52fe", + "id": "6a3e2d1d", "metadata": { "tags": [ "task" @@ -256,7 +249,7 @@ { "cell_type": "code", "execution_count": null, - "id": "fdd6d897", + "id": "b7be8448", "metadata": {}, "outputs": [], "source": [ @@ -276,7 +269,7 @@ }, { "cell_type": "markdown", - "id": "eae9240c", + "id": "0a1d207d", "metadata": {}, "source": [ "Here we set up a napari widget for visualizing the tracking results. This is part of the motile napari plugin, not part of core napari.\n", @@ -286,7 +279,7 @@ { "cell_type": "code", "execution_count": null, - "id": "fb62c799", + "id": "8d293cd5", "metadata": {}, "outputs": [], "source": [ @@ -296,7 +289,7 @@ }, { "cell_type": "markdown", - "id": "f92d5028", + "id": "2b6d08c2", "metadata": {}, "source": [ "Here we add a \"MotileRun\" to the napari tracking visualization widget (the \"view_controller\"). A MotileRun includes a name, a set of tracks, and a segmentation. The tracking visualization widget will add:\n", @@ -313,10 +306,11 @@ { "cell_type": "code", "execution_count": null, - "id": "3d8f029a", + "id": "03266dab", "metadata": {}, "outputs": [], "source": [ + "assign_tracklet_ids(gt_tracks)\n", "ground_truth_run = MotileRun(\n", " run_name=\"ground_truth\",\n", " tracks=gt_tracks,\n", @@ -327,7 +321,7 @@ }, { "cell_type": "markdown", - "id": "5d6a8a94", + "id": "fd3ccda5", "metadata": { "lines_to_next_cell": 2 }, @@ -344,7 +338,7 @@ }, { "cell_type": "markdown", - "id": "e0294f57", + "id": "f98caf86", "metadata": {}, "source": [ "

Task 2: Extract candidate nodes from the predicted segmentations

\n", @@ -358,6 +352,7 @@ "
  • The node id is the label of the detection
    • \n", "
  • Each node has an integer \"t\" attribute, based on the index into the first dimension of the input segmentation array
    • \n", "
  • Each node has float \"x\" and \"y\" attributes containing the \"x\" and \"y\" values from the centroid of the detection region
    • \n", + "
  • Each node has a \"score\" attribute containing the probability score output from StarDist. The probability map is at half resolution, so you will need to divide the centroid by 2 before indexing into the probability score.
    • \n", "
  • The graph has no edges (yet!)
    • \n", "\n", "
    " @@ -366,7 +361,7 @@ { "cell_type": "code", "execution_count": null, - "id": "758e94a6", + "id": "24e44ee4", "metadata": { "tags": [ "task" @@ -400,7 +395,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7661a9a3", + "id": "54b0cc4b", "metadata": {}, "outputs": [], "source": [ @@ -415,12 +410,14 @@ " assert type(data[\"x\"]) == float, f\"'x' attribute has type {type(data['x'])}, expected 'float'\"\n", " assert \"y\" in data, f\"'y' attribute missing for node {node}\"\n", " assert type(data[\"y\"]) == float, f\"'y' attribute has type {type(data['y'])}, expected 'float'\"\n", + " assert \"score\" in data, f\"'score' attribute missing for node {node}\"\n", + " assert type(data[\"score\"]) == float, f\"'score' attribute has type {type(data['score'])}, expected 'float'\"\n", "print(\"Your candidate graph passed all the tests!\")" ] }, { "cell_type": "markdown", - "id": "be5c6e1e", + "id": "fcb62447", "metadata": {}, "source": [ "We can visualize our candidate points using the napari Points layer. You should see one point in the center of each segmentation when we display it using the below cell." @@ -429,7 +426,7 @@ { "cell_type": "code", "execution_count": null, - "id": "1e1dad5c", + "id": "1287eca2", "metadata": {}, "outputs": [], "source": [ @@ -440,7 +437,7 @@ }, { "cell_type": "markdown", - "id": "90e26972", + "id": "a551fcb6", "metadata": {}, "source": [ "### Adding Candidate Edges\n", @@ -453,7 +450,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d6580bd6", + "id": "5d9fbeff", "metadata": {}, "outputs": [], "source": [ @@ -522,7 +519,7 @@ }, { "cell_type": "markdown", - "id": "64c81b92", + "id": "4a7e2a4a", "metadata": {}, "source": [ "Visualizing the candidate edges in napari is, unfortunately, not yet possible. However, we can print out the number of candidate nodes and edges, and compare it to the ground truth nodes and edgesedges. We should see that we have a few more candidate nodes than ground truth (due to false positive detections) and many more candidate edges than ground truth - our next step will be to use optimization to pick a subset of the candidate nodes and edges to generate our solution tracks." @@ -531,7 +528,7 @@ { "cell_type": "code", "execution_count": null, - "id": "6d7e2713", + "id": "16399ec0", "metadata": {}, "outputs": [], "source": [ @@ -541,7 +538,7 @@ }, { "cell_type": "markdown", - "id": "3219ba78", + "id": "2c07a3d7", "metadata": {}, "source": [ "## Checkpoint 1\n", @@ -553,7 +550,7 @@ }, { "cell_type": "markdown", - "id": "75535445", + "id": "fd408dd1", "metadata": {}, "source": [ "## Setting Up the Tracking Optimization Problem" @@ -561,7 +558,7 @@ }, { "cell_type": "markdown", - "id": "43caaf3d", + "id": "83b7cdf2", "metadata": {}, "source": [ "As hinted earlier, our goal is to prune the candidate graph. More formally we want to find a graph $\\tilde{G}=(\\tilde{V}, \\tilde{E})$ whose vertices $\\tilde{V}$ are a subset of the candidate graph vertices $V$ and whose edges $\\tilde{E}$ are a subset of the candidate graph edges $E$.\n", @@ -571,14 +568,12 @@ "\n", "A set of linear constraints ensures that the solution will be a feasible cell tracking graph. For example, if an edge is part of $\\tilde{G}$, both its incident nodes have to be part of $\\tilde{G}$ as well.\n", "\n", - "`motile` ([docs here](https://funkelab.github.io/motile/)), makes it easy to link with an ILP in python by implementing common linking constraints and costs.\n", - "\n", - "TODO: delete this?" + "`motile` ([docs here](https://funkelab.github.io/motile/)), makes it easy to link with an ILP in python by implementing common linking constraints and costs." ] }, { "cell_type": "markdown", - "id": "0593e8bc", + "id": "ab7ca92f", "metadata": {}, "source": [ "## Task 3 - Basic tracking with motile\n", @@ -587,11 +582,10 @@ "\n", "Here are some key similarities and differences between the quickstart and our task:\n", "\n", "\n", "Once you have set up the basic motile optimization task in the function below, you will probably need to adjust the weight and constant values on your costs until you get a solution that looks reasonable.\n", @@ -603,7 +597,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e8c691a3", + "id": "c7945f5f", "metadata": { "tags": [ "task" @@ -631,7 +625,7 @@ }, { "cell_type": "markdown", - "id": "4a0df98b", + "id": "09fa7e2f", "metadata": {}, "source": [ "Here is a utility function to gauge some statistics of a solution." @@ -640,7 +634,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4ad62178", + "id": "4fd1e9e6", "metadata": {}, "outputs": [], "source": [ @@ -650,7 +644,7 @@ }, { "cell_type": "markdown", - "id": "96e7bc54", + "id": "e1bf88cc", "metadata": {}, "source": [ "Here we actually run the optimization, and compare the found solution to the ground truth.\n", @@ -666,7 +660,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4eeaab6c", + "id": "28013a2f", "metadata": {}, "outputs": [], "source": [ @@ -675,13 +669,12 @@ "\n", "# then print some statistics about the solution compared to the ground truth\n", "print_graph_stats(solution_graph, \"solution\")\n", - "print_graph_stats(gt_tracks, \"gt tracks\")\n", - "\n" + "print_graph_stats(gt_tracks, \"gt tracks\")" ] }, { "cell_type": "markdown", - "id": "0d7ad23c", + "id": "43337027", "metadata": {}, "source": [ "If you haven't selected any nodes or edges in your solution, try adjusting your weight and/or constant values. Make sure you have some negative costs or selecting nothing will always be the best solution!" @@ -689,7 +682,7 @@ }, { "cell_type": "markdown", - "id": "c9dd1919", + "id": "05f8c8c2", "metadata": {}, "source": [ "

    Question 1: Interpret your results based on statistics

    \n", @@ -701,7 +694,7 @@ }, { "cell_type": "markdown", - "id": "03aa81e4", + "id": "ba1fff25", "metadata": {}, "source": [ "

    Checkpoint 2

    \n", @@ -711,7 +704,7 @@ }, { "cell_type": "markdown", - "id": "c34042f4", + "id": "81ee890b", "metadata": {}, "source": [ "## Visualize the Result\n", @@ -727,7 +720,7 @@ { "cell_type": "code", "execution_count": null, - "id": "299a5feb", + "id": "8c2a5ab5", "metadata": {}, "outputs": [], "source": [ @@ -764,7 +757,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2ca6c490", + "id": "bc1dcf26", "metadata": {}, "outputs": [], "source": [ @@ -779,7 +772,7 @@ }, { "cell_type": "markdown", - "id": "f72c1ea3", + "id": "2031a93f", "metadata": {}, "source": [ "

    Question 2: Interpret your results based on visualization

    \n", @@ -791,7 +784,7 @@ }, { "cell_type": "markdown", - "id": "6dd4b89d", + "id": "c3aaf750", "metadata": { "lines_to_next_cell": 2 }, @@ -810,7 +803,7 @@ }, { "cell_type": "markdown", - "id": "bb594625", + "id": "02540f98", "metadata": {}, "source": [ "The metrics we want to compute require a ground truth segmentation. Since we do not have a ground truth segmentation, we can make one by drawing a circle around each ground truth detection. While not perfect, it will be good enough to match ground truth to predicted detections in order to compute metrics." @@ -819,7 +812,7 @@ { "cell_type": "code", "execution_count": null, - "id": "3706d21e", + "id": "8185c96b", "metadata": {}, "outputs": [], "source": [ @@ -842,7 +835,7 @@ { "cell_type": "code", "execution_count": null, - "id": "50bf91cd", + "id": "4e1184aa", "metadata": {}, "outputs": [], "source": [ @@ -876,7 +869,7 @@ " segmentation=np.squeeze(run.output_segmentation),\n", " )\n", "\n", - " results = run_metrics(\n", + " results = traccuracy.run_metrics(\n", " gt_data=gt_graph,\n", " pred_data=pred_graph,\n", " matcher=IOUMatcher(iou_threshold=0.3, one_to_one=True),\n", @@ -900,7 +893,7 @@ { "cell_type": "code", "execution_count": null, - "id": "11f29870", + "id": "0644933e", "metadata": {}, "outputs": [], "source": [ @@ -911,7 +904,7 @@ }, { "cell_type": "markdown", - "id": "2d14706c", + "id": "b0357a23", "metadata": {}, "source": [ "

    Question 3: Interpret your results based on metrics

    \n", @@ -923,17 +916,64 @@ }, { "cell_type": "markdown", - "id": "965296f6", + "id": "e7415810", "metadata": {}, "source": [ "

    Checkpoint 3

    \n", - "If you reach this checkpoint with extra time, think about what kinds of improvements you could make to the costs and constraints to fix the issues that you are seeing. You can try tuning your weights and constants, or adding or removing motile Costs and Constraints, and seeing how that changes the output. See how good you can make the results!\n", + "If you reach this checkpoint with extra time, think about what kinds of improvements you could make to the costs and constraints to fix the issues that you are seeing. You can try tuning your weights and constants, or adding or removing motile Costs and Constraints, and seeing how that changes the output. We have added a convenience function in the box below where you can copy your solution from above, adapt it, and run the whole pipeline including visualizaiton and metrics computation.\n", + "\n", + "Do not get frustrated if you cannot get good results yet! Try to think about why and what custom costs we might add.\n", "
    " ] }, + { + "cell_type": "code", + "execution_count": null, + "id": "8832c89a", + "metadata": { + "tags": [ + "task" + ] + }, + "outputs": [], + "source": [ + "def adapt_basic_optimization(cand_graph):\n", + " \"\"\"Set up and solve the network flow problem.\n", + "\n", + " Args:\n", + " graph (nx.DiGraph): The candidate graph.\n", + "\n", + " Returns:\n", + " nx.DiGraph: The networkx digraph with the selected solution tracks\n", + " \"\"\"\n", + " cand_trackgraph = motile.TrackGraph(cand_graph, frame_attribute=\"t\")\n", + " solver = motile.Solver(cand_trackgraph)\n", + " ### YOUR CODE HERE ###\n", + " solver.solve()\n", + " solution_graph = graph_to_nx(solver.get_selected_subgraph())\n", + "\n", + " return solution_graph\n", + "\n", + "def run_pipeline(cand_graph, run_name, results_df):\n", + " solution_graph = adapt_basic_optimization(cand_graph)\n", + " solution_seg = relabel_segmentation(solution_graph, segmentation)\n", + " run = MotileRun(\n", + " run_name=run_name,\n", + " tracks=solution_graph,\n", + " output_segmentation=np.expand_dims(solution_seg, axis=1) # need to add a dummy dimension to fit API\n", + " )\n", + " widget.view_controller.update_napari_layers(run, time_attr=\"t\", pos_attr=(\"x\", \"y\"))\n", + " results_df = get_metrics(gt_tracks, gt_dets, run, results_df)\n", + " return results_df\n", + "\n", + "# Don't forget to rename your run below, so you can tell them apart in the results table\n", + "results_df = run_pipeline(cand_graph, \"basic_solution_2\", results_df)\n", + "results_df" + ] + }, { "cell_type": "markdown", - "id": "88548aca", + "id": "249a49f9", "metadata": {}, "source": [ "## Customizing the Tracking Task\n", @@ -948,7 +988,7 @@ }, { "cell_type": "markdown", - "id": "ad0380aa", + "id": "4793ae62", "metadata": {}, "source": [ "## Task 4 - Incorporating Known Direction of Motion\n", @@ -959,7 +999,7 @@ }, { "cell_type": "markdown", - "id": "e0a4b766", + "id": "5b2634db", "metadata": {}, "source": [ "

    Task 4a: Add a drift distance attribute

    \n", @@ -970,7 +1010,7 @@ { "cell_type": "code", "execution_count": null, - "id": "76de218b", + "id": "31ba3bf8", "metadata": { "tags": [ "task" @@ -993,7 +1033,7 @@ }, { "cell_type": "markdown", - "id": "6620734b", + "id": "7f862c0f", "metadata": {}, "source": [ "

    Task 4b: Add a drift distance attribute

    \n", @@ -1005,7 +1045,7 @@ { "cell_type": "code", "execution_count": null, - "id": "25b1191a", + "id": "3bb1d410", "metadata": { "tags": [ "task" @@ -1032,50 +1072,36 @@ " solution_graph = graph_to_nx(solver.get_selected_subgraph())\n", " return solution_graph\n", "\n", - "solution_graph = solve_drift_optimization(cand_graph)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "e2628365", - "metadata": {}, - "outputs": [], - "source": [ - "solution_graph = solve_drift_optimization(cand_graph)\n", - "solution_seg = relabel_segmentation(solution_graph, segmentation)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c11b2c25", - "metadata": {}, - "outputs": [], - "source": [ - "drift_run = MotileRun(\n", - " run_name=\"drift_solution\",\n", - " tracks=solution_graph,\n", - " output_segmentation=np.expand_dims(solution_seg, axis=1) # need to add a dummy dimension to fit API\n", - ")\n", "\n", - "widget.view_controller.update_napari_layers(drift_run, time_attr=\"t\", pos_attr=(\"x\", \"y\"))" + "def run_pipeline(cand_graph, run_name, results_df):\n", + " solution_graph = solve_drift_optimization(cand_graph)\n", + " solution_seg = relabel_segmentation(solution_graph, segmentation)\n", + " run = MotileRun(\n", + " run_name=run_name,\n", + " tracks=solution_graph,\n", + " output_segmentation=np.expand_dims(solution_seg, axis=1) # need to add a dummy dimension to fit API\n", + " )\n", + " widget.view_controller.update_napari_layers(run, time_attr=\"t\", pos_attr=(\"x\", \"y\"))\n", + " results_df = get_metrics(gt_tracks, gt_dets, run, results_df)\n", + " return results_df\n", + "\n", + "# Don't forget to rename your run if you re-run this cell!\n", + "results_df = run_pipeline(cand_graph, \"drift_dist\", results_df)\n", + "results_df" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "0d7efb94", + "cell_type": "markdown", + "id": "f55cdb17", "metadata": {}, - "outputs": [], "source": [ - "results_df = get_metrics(gt_tracks, gt_dets, drift_run, results_df)\n", - "results_df" + "Feel free to tinker with the weights and constants manually to try and improve the results.\n", + "You should be able to get something decent now, but this dataset is quite difficult! There are still many custom costs that could be added to improve the results - we will discuss some ideas together shortly." ] }, { "cell_type": "markdown", - "id": "e26c5ca5", + "id": "7e0477cf", "metadata": {}, "source": [ "

    Checkpoint 4

    \n", @@ -1085,7 +1111,7 @@ }, { "cell_type": "markdown", - "id": "562591e7", + "id": "a7e969d5", "metadata": {}, "source": [ "## Bonus: Learning the Weights" @@ -1093,7 +1119,7 @@ }, { "cell_type": "markdown", - "id": "87cd0be7", + "id": "251f80ea", "metadata": {}, "source": [ "Motile also provides the option to learn the best weights and constants using a [Structured Support Vector Machine](https://en.wikipedia.org/wiki/Structured_support_vector_machine). There is a tutorial on the motile documentation [here](https://funkelab.github.io/motile/learning.html), but we will also walk you through an example below.\n", @@ -1104,7 +1130,7 @@ { "cell_type": "code", "execution_count": null, - "id": "395f3330", + "id": "03ac0e3e", "metadata": {}, "outputs": [], "source": [ @@ -1133,7 +1159,7 @@ }, { "cell_type": "markdown", - "id": "b3201631", + "id": "8acbce61", "metadata": {}, "source": [ "The SSVM does not need dense ground truth - providing only some annotations frequently is sufficient to learn good weights, and is efficient for both computation time and annotation time. Below, we create a validation graph that spans the first three time frames, and annotate it with our ground truth." @@ -1142,7 +1168,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a29ca169", + "id": "86559b41", "metadata": { "lines_to_next_cell": 2 }, @@ -1158,7 +1184,7 @@ }, { "cell_type": "markdown", - "id": "94765d7c", + "id": "b03db1ea", "metadata": {}, "source": [ "Here we print the number of nodes and edges that have been annotated with True and False ground truth. It is important to provide negative/False annotations, as well as positive/True annotations, or the SSVM will try and select weights to pick everything." @@ -1167,7 +1193,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4d88ced9", + "id": "fcfa224a", "metadata": {}, "outputs": [], "source": [ @@ -1182,7 +1208,7 @@ }, { "cell_type": "markdown", - "id": "46e98c19", + "id": "ac765329", "metadata": {}, "source": [ "

    Bonus task: Add your best solver parameters

    \n", @@ -1193,7 +1219,7 @@ { "cell_type": "code", "execution_count": null, - "id": "49817bb6", + "id": "24128d7b", "metadata": { "tags": [ "task" @@ -1213,7 +1239,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e5c35382", + "id": "03b220a0", "metadata": {}, "outputs": [], "source": [ @@ -1221,7 +1247,9 @@ "\n", " cand_trackgraph = motile.TrackGraph(cand_graph, frame_attribute=\"t\")\n", " solver = motile.Solver(cand_trackgraph)\n", - "\n", + " solver.add_cost(\n", + " motile.costs.NodeSelection(weight=-1.0, attribute='score')\n", + " )\n", " solver.add_cost(\n", " motile.costs.EdgeSelection(weight=1.0, constant=-30, attribute=\"drift_dist\")\n", " )\n", @@ -1234,7 +1262,7 @@ }, { "cell_type": "markdown", - "id": "e512dbc4", + "id": "4b12996e", "metadata": {}, "source": [ "To fit the best weights, the solver will solve the ILP many times and slowly converge to the best set of weights in a structured manner. Running the cell below may take some time - we recommend getting a Gurobi license if you want to use this technique in your research, as it speeds up solving quite a bit.\n", @@ -1245,7 +1273,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7a6b8807", + "id": "a7e6b1e3", "metadata": {}, "outputs": [], "source": [ @@ -1257,7 +1285,7 @@ }, { "cell_type": "markdown", - "id": "0a4061c7", + "id": "b9b01fd6", "metadata": {}, "source": [ "After we have our optimal weights, we need to solve with them on the full candidate graph." @@ -1266,7 +1294,7 @@ { "cell_type": "code", "execution_count": null, - "id": "bf4dfa08", + "id": "724d7916", "metadata": { "lines_to_next_cell": 2 }, @@ -1284,7 +1312,7 @@ }, { "cell_type": "markdown", - "id": "91fe340f", + "id": "d90e6db0", "metadata": {}, "source": [ "Finally, we can visualize and compute metrics on the solution found using the weights discovered by the SSVM." @@ -1293,7 +1321,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c9bafa02", + "id": "5550cebf", "metadata": {}, "outputs": [], "source": [ @@ -1303,7 +1331,7 @@ { "cell_type": "code", "execution_count": null, - "id": "83160a74", + "id": "ec2c4d44", "metadata": {}, "outputs": [], "source": [ @@ -1319,7 +1347,7 @@ { "cell_type": "code", "execution_count": null, - "id": "bafe5ef4", + "id": "099dcf01", "metadata": {}, "outputs": [], "source": [ @@ -1330,7 +1358,7 @@ }, { "cell_type": "markdown", - "id": "b3b2b125", + "id": "8337fe65", "metadata": {}, "source": [ "

    Bonus Question: Interpret SSVM results

    \n", diff --git a/solution.ipynb b/solution.ipynb index c5a033b..e497a9f 100644 --- a/solution.ipynb +++ b/solution.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "dc64efee", + "id": "fb5945ac", "metadata": {}, "source": [ "# Exercise 9: Tracking-by-detection with an integer linear program (ILP)\n", @@ -45,7 +45,7 @@ }, { "cell_type": "markdown", - "id": "5482c789", + "id": "d7ac7bdb", "metadata": {}, "source": [ "Visualizations on a remote machine\n", @@ -71,7 +71,7 @@ { "cell_type": "code", "execution_count": null, - "id": "6f68cfef", + "id": "72c670b4", "metadata": {}, "outputs": [], "source": [ @@ -81,7 +81,7 @@ }, { "cell_type": "markdown", - "id": "3d91fa7a", + "id": "e7f67965", "metadata": {}, "source": [ "## Import packages" @@ -90,31 +90,24 @@ { "cell_type": "code", "execution_count": null, - "id": "329a3e85", + "id": "e291f63b", "metadata": {}, "outputs": [], "source": [ - "import time\n", - "from pathlib import Path\n", - "\n", "import skimage\n", "import numpy as np\n", "import napari\n", "import networkx as nx\n", "import scipy\n", "\n", - "\n", "import motile\n", "\n", "import zarr\n", - "from motile_toolbox.visualization import to_napari_tracks_layer\n", "from motile_toolbox.candidate_graph import graph_to_nx\n", "from motile_toolbox.visualization.napari_utils import assign_tracklet_ids\n", "import motile_plugin.widgets as plugin_widgets\n", "from motile_plugin.backend.motile_run import MotileRun\n", - "from napari.layers import Tracks\n", "import traccuracy\n", - "from traccuracy import run_metrics\n", "from traccuracy.metrics import CTCMetrics, DivisionMetrics\n", "from traccuracy.matchers import IOUMatcher\n", "from csv import DictReader\n", @@ -126,7 +119,7 @@ }, { "cell_type": "markdown", - "id": "c7d07864", + "id": "d2385ec1", "metadata": {}, "source": [ "## Load the dataset and inspect it in napari" @@ -134,7 +127,7 @@ }, { "cell_type": "markdown", - "id": "22519ef0", + "id": "ab1dede1", "metadata": {}, "source": [ "For this exercise we will be working with a fluorescence microscopy time-lapse of breast cancer cells with stained nuclei (SiR-DNA). It is similar to the dataset at https://zenodo.org/record/4034976#.YwZRCJPP1qt. The raw data, pre-computed segmentations, and detection probabilities are saved in a zarr, and the ground truth tracks are saved in a csv. The segmentation was generated with a pre-trained StartDist model, so there may be some segmentation errors which can affect the tracking process. The detection probabilities also come from StarDist, and are downsampled in x and y by 2 compared to the detections and raw data." @@ -142,7 +135,7 @@ }, { "cell_type": "markdown", - "id": "e2436d52", + "id": "22a39a79", "metadata": {}, "source": [ "Here we load the raw image data, segmentation, and probabilities from the zarr, and view them in napari." @@ -151,7 +144,7 @@ { "cell_type": "code", "execution_count": null, - "id": "19e4e26c", + "id": "bdcb6a05", "metadata": {}, "outputs": [], "source": [ @@ -164,30 +157,30 @@ }, { "cell_type": "markdown", - "id": "dcdf11b1", + "id": "587c87ec", "metadata": {}, "source": [ - "Let's use [napari](https://napari.org/tutorials/fundamentals/getting_started.html) to visualize the data. Napari is a wonderful viewer for imaging data that you can interact with in python, even directly out of jupyter notebooks. If you've never used napari, you might want to take a few minutes to go through [this tutorial](https://napari.org/stable/tutorials/fundamentals/viewer.html)." + "Let's use [napari](https://napari.org/tutorials/fundamentals/getting_started.html) to visualize the data. Napari is a wonderful viewer for imaging data that you can interact with in python, even directly out of jupyter notebooks. If you've never used napari, you might want to take a few minutes to go through [this tutorial](https://napari.org/stable/tutorials/fundamentals/viewer.html). Here we visualize the raw data, the predicted segmentations, and the predicted probabilities as separate layers. You can toggle each layer on and off in the layers list on the left." ] }, { "cell_type": "code", "execution_count": null, - "id": "d80fe285", + "id": "158079c2", "metadata": { "lines_to_next_cell": 1 }, "outputs": [], "source": [ "viewer = napari.Viewer()\n", + "viewer.add_image(probabilities, name=\"probs\", scale=(1, 2, 2))\n", "viewer.add_image(image_data, name=\"raw\")\n", - "viewer.add_labels(segmentation, name=\"seg\")\n", - "viewer.add_image(probabilities, name=\"probs\", scale=(1, 2, 2))" + "viewer.add_labels(segmentation, name=\"seg\")" ] }, { "cell_type": "markdown", - "id": "30616779", + "id": "36c2b808", "metadata": {}, "source": [ "After running the previous cell, open NoMachine and check for an open napari window." @@ -195,7 +188,7 @@ }, { "cell_type": "markdown", - "id": "8f6b0cb2", + "id": "8d1590df", "metadata": {}, "source": [ "## Read in the ground truth graph\n", @@ -213,7 +206,7 @@ }, { "cell_type": "markdown", - "id": "c291b3d4", + "id": "e6bf078f", "metadata": {}, "source": [ "\n", @@ -237,7 +230,7 @@ { "cell_type": "code", "execution_count": null, - "id": "740d1755", + "id": "ef985889", "metadata": { "tags": [ "solution" @@ -260,6 +253,7 @@ " gt_tracks.add_node(_id, **attrs)\n", " if parent_id != -1:\n", " gt_tracks.add_edge(parent_id, _id)\n", + "\n", " return gt_tracks\n", "\n", "gt_tracks = read_gt_tracks()" @@ -268,7 +262,7 @@ { "cell_type": "code", "execution_count": null, - "id": "fdd6d897", + "id": "b7be8448", "metadata": {}, "outputs": [], "source": [ @@ -288,7 +282,7 @@ }, { "cell_type": "markdown", - "id": "eae9240c", + "id": "0a1d207d", "metadata": {}, "source": [ "Here we set up a napari widget for visualizing the tracking results. This is part of the motile napari plugin, not part of core napari.\n", @@ -298,7 +292,7 @@ { "cell_type": "code", "execution_count": null, - "id": "fb62c799", + "id": "8d293cd5", "metadata": {}, "outputs": [], "source": [ @@ -308,7 +302,7 @@ }, { "cell_type": "markdown", - "id": "f92d5028", + "id": "2b6d08c2", "metadata": {}, "source": [ "Here we add a \"MotileRun\" to the napari tracking visualization widget (the \"view_controller\"). A MotileRun includes a name, a set of tracks, and a segmentation. The tracking visualization widget will add:\n", @@ -325,10 +319,11 @@ { "cell_type": "code", "execution_count": null, - "id": "3d8f029a", + "id": "03266dab", "metadata": {}, "outputs": [], "source": [ + "assign_tracklet_ids(gt_tracks)\n", "ground_truth_run = MotileRun(\n", " run_name=\"ground_truth\",\n", " tracks=gt_tracks,\n", @@ -339,7 +334,7 @@ }, { "cell_type": "markdown", - "id": "5d6a8a94", + "id": "fd3ccda5", "metadata": { "lines_to_next_cell": 2 }, @@ -356,7 +351,7 @@ }, { "cell_type": "markdown", - "id": "e0294f57", + "id": "f98caf86", "metadata": {}, "source": [ "

    Task 2: Extract candidate nodes from the predicted segmentations

    \n", @@ -370,6 +365,7 @@ "
  • The node id is the label of the detection
    • \n", "
  • Each node has an integer \"t\" attribute, based on the index into the first dimension of the input segmentation array
    • \n", "
  • Each node has float \"x\" and \"y\" attributes containing the \"x\" and \"y\" values from the centroid of the detection region
    • \n", + "
  • Each node has a \"score\" attribute containing the probability score output from StarDist. The probability map is at half resolution, so you will need to divide the centroid by 2 before indexing into the probability score.
    • \n", "
  • The graph has no edges (yet!)
    • \n", "\n", "
    " @@ -378,7 +374,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2d55e5c3", + "id": "adb4f101", "metadata": { "tags": [ "solution" @@ -403,10 +399,13 @@ " props = skimage.measure.regionprops(seg_frame)\n", " for regionprop in props:\n", " node_id = regionprop.label\n", + " x = float(regionprop.centroid[0])\n", + " y = float(regionprop.centroid[1])\n", " attrs = {\n", " \"t\": t,\n", - " \"x\": float(regionprop.centroid[0]),\n", - " \"y\": float(regionprop.centroid[1]),\n", + " \"x\": x,\n", + " \"y\": y,\n", + " \"score\": float(probabilities[t, int(x // 2), int(y // 2)]),\n", " }\n", " assert node_id not in cand_graph.nodes\n", " cand_graph.add_node(node_id, **attrs)\n", @@ -418,7 +417,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7661a9a3", + "id": "54b0cc4b", "metadata": {}, "outputs": [], "source": [ @@ -433,12 +432,14 @@ " assert type(data[\"x\"]) == float, f\"'x' attribute has type {type(data['x'])}, expected 'float'\"\n", " assert \"y\" in data, f\"'y' attribute missing for node {node}\"\n", " assert type(data[\"y\"]) == float, f\"'y' attribute has type {type(data['y'])}, expected 'float'\"\n", + " assert \"score\" in data, f\"'score' attribute missing for node {node}\"\n", + " assert type(data[\"score\"]) == float, f\"'score' attribute has type {type(data['score'])}, expected 'float'\"\n", "print(\"Your candidate graph passed all the tests!\")" ] }, { "cell_type": "markdown", - "id": "be5c6e1e", + "id": "fcb62447", "metadata": {}, "source": [ "We can visualize our candidate points using the napari Points layer. You should see one point in the center of each segmentation when we display it using the below cell." @@ -447,7 +448,7 @@ { "cell_type": "code", "execution_count": null, - "id": "1e1dad5c", + "id": "1287eca2", "metadata": {}, "outputs": [], "source": [ @@ -458,7 +459,7 @@ }, { "cell_type": "markdown", - "id": "90e26972", + "id": "a551fcb6", "metadata": {}, "source": [ "### Adding Candidate Edges\n", @@ -471,7 +472,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d6580bd6", + "id": "5d9fbeff", "metadata": {}, "outputs": [], "source": [ @@ -540,7 +541,7 @@ }, { "cell_type": "markdown", - "id": "64c81b92", + "id": "4a7e2a4a", "metadata": {}, "source": [ "Visualizing the candidate edges in napari is, unfortunately, not yet possible. However, we can print out the number of candidate nodes and edges, and compare it to the ground truth nodes and edgesedges. We should see that we have a few more candidate nodes than ground truth (due to false positive detections) and many more candidate edges than ground truth - our next step will be to use optimization to pick a subset of the candidate nodes and edges to generate our solution tracks." @@ -549,7 +550,7 @@ { "cell_type": "code", "execution_count": null, - "id": "6d7e2713", + "id": "16399ec0", "metadata": {}, "outputs": [], "source": [ @@ -559,7 +560,7 @@ }, { "cell_type": "markdown", - "id": "3219ba78", + "id": "2c07a3d7", "metadata": {}, "source": [ "## Checkpoint 1\n", @@ -571,7 +572,7 @@ }, { "cell_type": "markdown", - "id": "75535445", + "id": "fd408dd1", "metadata": {}, "source": [ "## Setting Up the Tracking Optimization Problem" @@ -579,7 +580,7 @@ }, { "cell_type": "markdown", - "id": "43caaf3d", + "id": "83b7cdf2", "metadata": {}, "source": [ "As hinted earlier, our goal is to prune the candidate graph. More formally we want to find a graph $\\tilde{G}=(\\tilde{V}, \\tilde{E})$ whose vertices $\\tilde{V}$ are a subset of the candidate graph vertices $V$ and whose edges $\\tilde{E}$ are a subset of the candidate graph edges $E$.\n", @@ -589,14 +590,12 @@ "\n", "A set of linear constraints ensures that the solution will be a feasible cell tracking graph. For example, if an edge is part of $\\tilde{G}$, both its incident nodes have to be part of $\\tilde{G}$ as well.\n", "\n", - "`motile` ([docs here](https://funkelab.github.io/motile/)), makes it easy to link with an ILP in python by implementing common linking constraints and costs.\n", - "\n", - "TODO: delete this?" + "`motile` ([docs here](https://funkelab.github.io/motile/)), makes it easy to link with an ILP in python by implementing common linking constraints and costs." ] }, { "cell_type": "markdown", - "id": "0593e8bc", + "id": "ab7ca92f", "metadata": {}, "source": [ "## Task 3 - Basic tracking with motile\n", @@ -605,11 +604,10 @@ "\n", "Here are some key similarities and differences between the quickstart and our task:\n", "
      \n", - "
    • We do not have scores on our nodes. This means we do not need to include a `NodeSelection` cost.
      • \n", - "
    • We also do not have scores on our edges. However, we can use the edge distance as a cost, so that longer edges are more costly than shorter edges. Instead of using the `EdgeSelection` cost, we can use the `EdgeDistance` cost with `position_attribute=\"pos\"`. You will want a positive weight, since higher distances should be more costly, unlike in the example when higher scores were good and so we inverted them with a negative weight.
      • \n", - "
    • Because distance is always positive, and you want a positive weight, you will want to include a negative constant on the `EdgeDistance` cost. If there are no negative selection costs, the ILP will always select nothing, because the cost of selecting nothing is zero.
      • \n", - "
    • We want to allow divisions. So, we should pass in 2 to our `MaxChildren` constraint. The `MaxParents` constraint should have 1, the same as the quickstart, because neither task allows merging.
      • \n", - "
    • You should include an Appear cost similar to the one in the quickstart.
      • \n", + "
    • We do not have scores on our edges. However, we can use the edge distance as a cost, so that longer edges are more costly than shorter edges. Instead of using the EdgeSelection cost, we can use the EdgeDistance cost with position_attribute=\"pos\". You will want a positive weight, since higher distances should be more costly, unlike in the example when higher scores were good and so we inverted them with a negative weight.
      • \n", + "
    • Because distance is always positive, and you want a positive weight, you will want to include a negative constant on the EdgeDistance cost. If there are no negative selection costs, the ILP will always select nothing, because the cost of selecting nothing is zero.
      • \n", + "
    • We want to allow divisions. So, we should pass in 2 to our MaxChildren constraint. The MaxParents constraint should have 1, the same as the quickstart, because neither task allows merging.
      • \n", + "
    • You should include an Appear cost and a NodeSelection cost similar to the one in the quickstart.
      • \n", "
    \n", "\n", "Once you have set up the basic motile optimization task in the function below, you will probably need to adjust the weight and constant values on your costs until you get a solution that looks reasonable.\n", @@ -621,7 +619,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7dc617a6", + "id": "01071ec3", "metadata": { "tags": [ "solution" @@ -640,11 +638,14 @@ " \"\"\"\n", " cand_trackgraph = motile.TrackGraph(cand_graph, frame_attribute=\"t\")\n", " solver = motile.Solver(cand_trackgraph)\n", - "\n", + " solver.add_cost(\n", + " motile.costs.NodeSelection(weight=-1.0, attribute=\"score\")\n", + " )\n", " solver.add_cost(\n", " motile.costs.EdgeDistance(weight=1, constant=-20, position_attribute=(\"x\", \"y\"))\n", " )\n", - " solver.add_cost(motile.costs.Appear(constant=1.0))\n", + " solver.add_cost(motile.costs.Appear(constant=2.0))\n", + " solver.add_cost(motile.costs.Split(constant=1.0))\n", "\n", " solver.add_constraint(motile.constraints.MaxParents(1))\n", " solver.add_constraint(motile.constraints.MaxChildren(2))\n", @@ -656,7 +657,7 @@ }, { "cell_type": "markdown", - "id": "4a0df98b", + "id": "09fa7e2f", "metadata": {}, "source": [ "Here is a utility function to gauge some statistics of a solution." @@ -665,7 +666,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4ad62178", + "id": "4fd1e9e6", "metadata": {}, "outputs": [], "source": [ @@ -675,7 +676,7 @@ }, { "cell_type": "markdown", - "id": "96e7bc54", + "id": "e1bf88cc", "metadata": {}, "source": [ "Here we actually run the optimization, and compare the found solution to the ground truth.\n", @@ -691,7 +692,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4eeaab6c", + "id": "28013a2f", "metadata": {}, "outputs": [], "source": [ @@ -700,13 +701,12 @@ "\n", "# then print some statistics about the solution compared to the ground truth\n", "print_graph_stats(solution_graph, \"solution\")\n", - "print_graph_stats(gt_tracks, \"gt tracks\")\n", - "\n" + "print_graph_stats(gt_tracks, \"gt tracks\")" ] }, { "cell_type": "markdown", - "id": "0d7ad23c", + "id": "43337027", "metadata": {}, "source": [ "If you haven't selected any nodes or edges in your solution, try adjusting your weight and/or constant values. Make sure you have some negative costs or selecting nothing will always be the best solution!" @@ -714,7 +714,7 @@ }, { "cell_type": "markdown", - "id": "c9dd1919", + "id": "05f8c8c2", "metadata": {}, "source": [ "

    Question 1: Interpret your results based on statistics

    \n", @@ -726,7 +726,7 @@ }, { "cell_type": "markdown", - "id": "03aa81e4", + "id": "ba1fff25", "metadata": {}, "source": [ "

    Checkpoint 2

    \n", @@ -736,7 +736,7 @@ }, { "cell_type": "markdown", - "id": "c34042f4", + "id": "81ee890b", "metadata": {}, "source": [ "## Visualize the Result\n", @@ -752,7 +752,7 @@ { "cell_type": "code", "execution_count": null, - "id": "299a5feb", + "id": "8c2a5ab5", "metadata": {}, "outputs": [], "source": [ @@ -789,7 +789,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2ca6c490", + "id": "bc1dcf26", "metadata": {}, "outputs": [], "source": [ @@ -804,7 +804,7 @@ }, { "cell_type": "markdown", - "id": "f72c1ea3", + "id": "2031a93f", "metadata": {}, "source": [ "

    Question 2: Interpret your results based on visualization

    \n", @@ -816,7 +816,7 @@ }, { "cell_type": "markdown", - "id": "6dd4b89d", + "id": "c3aaf750", "metadata": { "lines_to_next_cell": 2 }, @@ -835,7 +835,7 @@ }, { "cell_type": "markdown", - "id": "bb594625", + "id": "02540f98", "metadata": {}, "source": [ "The metrics we want to compute require a ground truth segmentation. Since we do not have a ground truth segmentation, we can make one by drawing a circle around each ground truth detection. While not perfect, it will be good enough to match ground truth to predicted detections in order to compute metrics." @@ -844,7 +844,7 @@ { "cell_type": "code", "execution_count": null, - "id": "3706d21e", + "id": "8185c96b", "metadata": {}, "outputs": [], "source": [ @@ -867,7 +867,7 @@ { "cell_type": "code", "execution_count": null, - "id": "50bf91cd", + "id": "4e1184aa", "metadata": {}, "outputs": [], "source": [ @@ -901,7 +901,7 @@ " segmentation=np.squeeze(run.output_segmentation),\n", " )\n", "\n", - " results = run_metrics(\n", + " results = traccuracy.run_metrics(\n", " gt_data=gt_graph,\n", " pred_data=pred_graph,\n", " matcher=IOUMatcher(iou_threshold=0.3, one_to_one=True),\n", @@ -925,7 +925,7 @@ { "cell_type": "code", "execution_count": null, - "id": "11f29870", + "id": "0644933e", "metadata": {}, "outputs": [], "source": [ @@ -936,7 +936,7 @@ }, { "cell_type": "markdown", - "id": "2d14706c", + "id": "b0357a23", "metadata": {}, "source": [ "

    Question 3: Interpret your results based on metrics

    \n", @@ -948,17 +948,74 @@ }, { "cell_type": "markdown", - "id": "965296f6", + "id": "e7415810", "metadata": {}, "source": [ "

    Checkpoint 3

    \n", - "If you reach this checkpoint with extra time, think about what kinds of improvements you could make to the costs and constraints to fix the issues that you are seeing. You can try tuning your weights and constants, or adding or removing motile Costs and Constraints, and seeing how that changes the output. See how good you can make the results!\n", + "If you reach this checkpoint with extra time, think about what kinds of improvements you could make to the costs and constraints to fix the issues that you are seeing. You can try tuning your weights and constants, or adding or removing motile Costs and Constraints, and seeing how that changes the output. We have added a convenience function in the box below where you can copy your solution from above, adapt it, and run the whole pipeline including visualizaiton and metrics computation.\n", + "\n", + "Do not get frustrated if you cannot get good results yet! Try to think about why and what custom costs we might add.\n", "
    " ] }, + { + "cell_type": "code", + "execution_count": null, + "id": "ae8c12a4", + "metadata": { + "lines_to_next_cell": 2, + "tags": [ + "solution" + ] + }, + "outputs": [], + "source": [ + "def adapt_basic_optimization(cand_graph):\n", + " \"\"\"Set up and solve the network flow problem.\n", + "\n", + " Args:\n", + " graph (nx.DiGraph): The candidate graph.\n", + "\n", + " Returns:\n", + " nx.DiGraph: The networkx digraph with the selected solution tracks\n", + " \"\"\"\n", + " cand_trackgraph = motile.TrackGraph(cand_graph, frame_attribute=\"t\")\n", + " solver = motile.Solver(cand_trackgraph)\n", + " solver.add_cost(\n", + " motile.costs.NodeSelection(weight=-5.0, constant=2.5, attribute=\"score\")\n", + " )\n", + " solver.add_cost(\n", + " motile.costs.EdgeDistance(weight=1, constant=-20, position_attribute=(\"x\", \"y\"))\n", + " )\n", + " solver.add_cost(motile.costs.Appear(constant=20.0))\n", + " solver.add_cost(motile.costs.Split(constant=15.0))\n", + "\n", + " solver.add_constraint(motile.constraints.MaxParents(1))\n", + " solver.add_constraint(motile.constraints.MaxChildren(2))\n", + " solver.solve()\n", + " solution_graph = graph_to_nx(solver.get_selected_subgraph())\n", + "\n", + " return solution_graph\n", + "\n", + "def run_pipeline(cand_graph, run_name, results_df):\n", + " solution_graph = adapt_basic_optimization(cand_graph)\n", + " solution_seg = relabel_segmentation(solution_graph, segmentation)\n", + " run = MotileRun(\n", + " run_name=run_name,\n", + " tracks=solution_graph,\n", + " output_segmentation=np.expand_dims(solution_seg, axis=1) # need to add a dummy dimension to fit API\n", + " )\n", + " widget.view_controller.update_napari_layers(run, time_attr=\"t\", pos_attr=(\"x\", \"y\"))\n", + " results_df = get_metrics(gt_tracks, gt_dets, run, results_df)\n", + " return results_df\n", + "\n", + "results_df = run_pipeline(cand_graph, \"basic_solution_2\", results_df)\n", + "results_df" + ] + }, { "cell_type": "markdown", - "id": "88548aca", + "id": "249a49f9", "metadata": {}, "source": [ "## Customizing the Tracking Task\n", @@ -973,7 +1030,7 @@ }, { "cell_type": "markdown", - "id": "ad0380aa", + "id": "4793ae62", "metadata": {}, "source": [ "## Task 4 - Incorporating Known Direction of Motion\n", @@ -984,7 +1041,7 @@ }, { "cell_type": "markdown", - "id": "e0a4b766", + "id": "5b2634db", "metadata": {}, "source": [ "

    Task 4a: Add a drift distance attribute

    \n", @@ -995,7 +1052,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a2b55e6d", + "id": "f302f541", "metadata": { "tags": [ "solution" @@ -1021,7 +1078,7 @@ }, { "cell_type": "markdown", - "id": "6620734b", + "id": "7f862c0f", "metadata": {}, "source": [ "

    Task 4b: Add a drift distance attribute

    \n", @@ -1033,7 +1090,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b17e7962", + "id": "34aae535", "metadata": { "tags": [ "solution" @@ -1053,60 +1110,52 @@ "\n", " cand_trackgraph = motile.TrackGraph(cand_graph, frame_attribute=\"t\")\n", " solver = motile.Solver(cand_trackgraph)\n", - "\n", + " solver.add_cost(\n", + " motile.costs.NodeSelection(weight=-100, constant=75, attribute=\"score\")\n", + " )\n", " solver.add_cost(\n", " motile.costs.EdgeSelection(weight=1.0, constant=-30, attribute=\"drift_dist\")\n", " )\n", + " solver.add_cost(motile.costs.Appear(constant=40.0))\n", + " solver.add_cost(motile.costs.Split(constant=45.0))\n", "\n", " solver.add_constraint(motile.constraints.MaxParents(1))\n", " solver.add_constraint(motile.constraints.MaxChildren(2))\n", "\n", " solver.solve()\n", " solution_graph = graph_to_nx(solver.get_selected_subgraph())\n", - " return solution_graph" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "e2628365", - "metadata": {}, - "outputs": [], - "source": [ - "solution_graph = solve_drift_optimization(cand_graph)\n", - "solution_seg = relabel_segmentation(solution_graph, segmentation)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c11b2c25", - "metadata": {}, - "outputs": [], - "source": [ - "drift_run = MotileRun(\n", - " run_name=\"drift_solution\",\n", - " tracks=solution_graph,\n", - " output_segmentation=np.expand_dims(solution_seg, axis=1) # need to add a dummy dimension to fit API\n", - ")\n", + " return solution_graph\n", "\n", - "widget.view_controller.update_napari_layers(drift_run, time_attr=\"t\", pos_attr=(\"x\", \"y\"))" + "\n", + "def run_pipeline(cand_graph, run_name, results_df):\n", + " solution_graph = solve_drift_optimization(cand_graph)\n", + " solution_seg = relabel_segmentation(solution_graph, segmentation)\n", + " run = MotileRun(\n", + " run_name=run_name,\n", + " tracks=solution_graph,\n", + " output_segmentation=np.expand_dims(solution_seg, axis=1) # need to add a dummy dimension to fit API\n", + " )\n", + " widget.view_controller.update_napari_layers(run, time_attr=\"t\", pos_attr=(\"x\", \"y\"))\n", + " results_df = get_metrics(gt_tracks, gt_dets, run, results_df)\n", + " return results_df\n", + "\n", + "# Don't forget to rename your run if you re-run this cell!\n", + "results_df = run_pipeline(cand_graph, \"node_const_75\", results_df)\n", + "results_df" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "0d7efb94", + "cell_type": "markdown", + "id": "f55cdb17", "metadata": {}, - "outputs": [], "source": [ - "results_df = get_metrics(gt_tracks, gt_dets, drift_run, results_df)\n", - "results_df" + "Feel free to tinker with the weights and constants manually to try and improve the results.\n", + "You should be able to get something decent now, but this dataset is quite difficult! There are still many custom costs that could be added to improve the results - we will discuss some ideas together shortly." ] }, { "cell_type": "markdown", - "id": "e26c5ca5", + "id": "7e0477cf", "metadata": {}, "source": [ "

    Checkpoint 4

    \n", @@ -1116,7 +1165,7 @@ }, { "cell_type": "markdown", - "id": "562591e7", + "id": "a7e969d5", "metadata": {}, "source": [ "## Bonus: Learning the Weights" @@ -1124,7 +1173,7 @@ }, { "cell_type": "markdown", - "id": "87cd0be7", + "id": "251f80ea", "metadata": {}, "source": [ "Motile also provides the option to learn the best weights and constants using a [Structured Support Vector Machine](https://en.wikipedia.org/wiki/Structured_support_vector_machine). There is a tutorial on the motile documentation [here](https://funkelab.github.io/motile/learning.html), but we will also walk you through an example below.\n", @@ -1135,7 +1184,7 @@ { "cell_type": "code", "execution_count": null, - "id": "395f3330", + "id": "03ac0e3e", "metadata": {}, "outputs": [], "source": [ @@ -1164,7 +1213,7 @@ }, { "cell_type": "markdown", - "id": "b3201631", + "id": "8acbce61", "metadata": {}, "source": [ "The SSVM does not need dense ground truth - providing only some annotations frequently is sufficient to learn good weights, and is efficient for both computation time and annotation time. Below, we create a validation graph that spans the first three time frames, and annotate it with our ground truth." @@ -1173,7 +1222,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a29ca169", + "id": "86559b41", "metadata": { "lines_to_next_cell": 2 }, @@ -1189,7 +1238,7 @@ }, { "cell_type": "markdown", - "id": "94765d7c", + "id": "b03db1ea", "metadata": {}, "source": [ "Here we print the number of nodes and edges that have been annotated with True and False ground truth. It is important to provide negative/False annotations, as well as positive/True annotations, or the SSVM will try and select weights to pick everything." @@ -1198,7 +1247,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4d88ced9", + "id": "fcfa224a", "metadata": {}, "outputs": [], "source": [ @@ -1213,7 +1262,7 @@ }, { "cell_type": "markdown", - "id": "46e98c19", + "id": "ac765329", "metadata": {}, "source": [ "

    Bonus task: Add your best solver parameters

    \n", @@ -1224,7 +1273,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e5c35382", + "id": "03b220a0", "metadata": {}, "outputs": [], "source": [ @@ -1232,7 +1281,9 @@ "\n", " cand_trackgraph = motile.TrackGraph(cand_graph, frame_attribute=\"t\")\n", " solver = motile.Solver(cand_trackgraph)\n", - "\n", + " solver.add_cost(\n", + " motile.costs.NodeSelection(weight=-1.0, attribute='score')\n", + " )\n", " solver.add_cost(\n", " motile.costs.EdgeSelection(weight=1.0, constant=-30, attribute=\"drift_dist\")\n", " )\n", @@ -1245,7 +1296,7 @@ }, { "cell_type": "markdown", - "id": "e512dbc4", + "id": "4b12996e", "metadata": {}, "source": [ "To fit the best weights, the solver will solve the ILP many times and slowly converge to the best set of weights in a structured manner. Running the cell below may take some time - we recommend getting a Gurobi license if you want to use this technique in your research, as it speeds up solving quite a bit.\n", @@ -1256,7 +1307,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7a6b8807", + "id": "a7e6b1e3", "metadata": {}, "outputs": [], "source": [ @@ -1268,7 +1319,7 @@ }, { "cell_type": "markdown", - "id": "0a4061c7", + "id": "b9b01fd6", "metadata": {}, "source": [ "After we have our optimal weights, we need to solve with them on the full candidate graph." @@ -1277,7 +1328,7 @@ { "cell_type": "code", "execution_count": null, - "id": "bf4dfa08", + "id": "724d7916", "metadata": { "lines_to_next_cell": 2 }, @@ -1295,7 +1346,7 @@ }, { "cell_type": "markdown", - "id": "91fe340f", + "id": "d90e6db0", "metadata": {}, "source": [ "Finally, we can visualize and compute metrics on the solution found using the weights discovered by the SSVM." @@ -1304,7 +1355,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c9bafa02", + "id": "5550cebf", "metadata": {}, "outputs": [], "source": [ @@ -1314,7 +1365,7 @@ { "cell_type": "code", "execution_count": null, - "id": "83160a74", + "id": "ec2c4d44", "metadata": {}, "outputs": [], "source": [ @@ -1330,7 +1381,7 @@ { "cell_type": "code", "execution_count": null, - "id": "bafe5ef4", + "id": "099dcf01", "metadata": {}, "outputs": [], "source": [ @@ -1341,7 +1392,7 @@ }, { "cell_type": "markdown", - "id": "b3b2b125", + "id": "8337fe65", "metadata": {}, "source": [ "

    Bonus Question: Interpret SSVM results

    \n",