From c1b43268b4b71b37259ea2572b85b67e10c3df0d Mon Sep 17 00:00:00 2001 From: David Shupe Date: Thu, 17 Aug 2017 13:53:36 -0700 Subject: [PATCH 1/3] add firefly displays to wip notebook --- tutorial-firefly.ipynb | 2002 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 2002 insertions(+) create mode 100644 tutorial-firefly.ipynb diff --git a/tutorial-firefly.ipynb b/tutorial-firefly.ipynb new file mode 100644 index 0000000..ac2b855 --- /dev/null +++ b/tutorial-firefly.ipynb @@ -0,0 +1,2002 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Using the LSST DM Stack in Python\n", + "\n", + "This tutorial focuses on using the DM stack in Python. Some of the things we'll be doing are more commonly done on the command-line, via executable scripts the stack also provides. A complete tutorial for the command-line functionality can be found in [DM Tech Note 23](https://dmtn-023.lsst.io/).\n", + "\n", + "More notebook examples can be found here: https://github.com/RobertLuptonTheGood/notebooks/tree/master/Demos" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Data Repository Setup\n", + "\n", + "Instead of operating directly on files and directories, we interact with on-disk data products via an abstraction layer called the *data butler*. The butler operates on *data repositories*, and our first task is to set up a repository with some raw data, master calibration files, and an external reference catalog. All of these are from a self-contained test dataset we call [ci_hsc](https://github.com/lsst/ci_hsc). The full ci_hsc dataset includes just enough data to run the full (current) LSST pipeline, which extends through processing coadds from multiple bands together. In this tutorial we'll focus on processing an individual image, and that's all this particular subset will support. We also won't go into the details of how to build master calibration files or reference catalogs here.\n", + "\n", + "These first few steps to set up a data repository are best performed on the command-line, but use a Jupyter trick to do that within the notebook. You're also welcome to copy and paste these lines (minus the \"`%%script bash`\" line, of course) into a JupyterLab terminal window and run them individually instead if you want to pay close attention to what we're doing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "%%script bash\n", + "export DATA_DIR=$HOME/data\n", + "export CI_HSC_DIR=$DATA_DIR/ci_hsc_small\n", + "mkdir -p $DATA_DIR\n", + "cd $DATA_DIR\n", + "if ! [ -d $CI_HSC_DIR ]; then\n", + " curl -O http://lsst-web.ncsa.illinois.edu/~krughoff/data/small_demo.tar.gz\n", + " tar zxvf small_demo.tar.gz\n", + "fi\n", + "export WORK_DIR=$HOME/WORK\n", + "if ! [ -d $WORK_DIR ]; then\n", + " mkdir -p $WORK_DIR\n", + " echo \"lsst.obs.hsc.HscMapper\" > $WORK_DIR/_mapper\n", + " ingestImages.py $WORK_DIR $CI_HSC_DIR/raw/*.fits --mode=link\n", + " cd $WORK_DIR\n", + " ln -s $CI_HSC_DIR/CALIB .\n", + " mkdir ref_cats\n", + " cd ref_cats\n", + " ln -s $CI_HSC_DIR/ps1_pv3_3pi_20170110 .\n", + "fi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "import os\n", + "DATA_DIR = os.path.join(os.environ['HOME'], \"data\")\n", + "CI_HSC_DIR = os.path.join(DATA_DIR, \"ci_hsc_small\")\n", + "WORK_DIR = os.path.join(os.environ['HOME'], \"WORK\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Instrument Signature Removal and Command-Line Tasks\n", + "\n", + "Before we can start doing *interesting* things, we need some minimally processed images (i.e. flat-fielded, bias-corrected, etc). Because the HSC team has spent a lot of time characterizing the instrument, we really want to run this step with the default configuration they've provided. That's also much actually easier to do from the command-line, and while we *could* do it from Python, that'd involve a lot of little irrelevant workarounds we'd rather not get bogged down in.\n", + "\n", + "ISR is implemented as a subclass of `lsst.pipe.base.Task`. Nearly all of our high-level algorithms are implemented as `Task`s, which are essentially just callable objects that can be composed (a high-level `Task` can hold one or more lower-level \"subtasks\", to which it can delegate work) and configured (every task takes an instance of a configuration class that controls what it does in detail). ISR is actually a `CmdLineTask`, a special kind of task that can be run from the command-line and use the data butler for all of its inputs and outputs (regular `Task`s generally do not use the butler directly). Unlike virtually every other algorithm, there is a different ISR `Task` for each major camera (though there's also a simple default one), reflecting the specialized processing that's needed at this level.\n", + "\n", + "But (for uninteresting, historical reasons), it's not currently possible to run `IsrTask` from the command-line. Instead, what we can do is run a parent `CmdLineTask`, `lsst.pipe.tasks.ProcessCcdTask`, which will run `IsrTask` as well as a few other steps. By default, it doesn't actually save the image directly after ISR is run - it performs a few more operations first, and then saves that image. But we can tell it to do so by modifying the tasks' configuration when we.\n", + "\n", + "The full command-line for running `ProcessCcdTask` is below. Note that `$HOME/WORK` is just the `WORK_DIR` variable we've defined above, but we have to redefine it here because evironment variables in one `%%script` environment don't propagate to the next. If you've been running these from a terminal tab instead, you can just use `$WORK_DIR` instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%script bash\n", + "processCcd.py $HOME/WORK --rerun isr --id visit=903334 ccd=16 --config isr.doWrite=True" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are a few features of this command-line that bear explaining:\n", + " - We run `processCcd.py`, not `ProcessCcdTask`. There's a similar driver script for all `CmdLineTask`s, with the name formed by making the first word lowercase and removing the `Task` suffix. These are added to your `PATH` when you set up the LSST package in which they're defined (which happens automatically in the JupyterLab environment).\n", + " - The first argument to any command-line task is the path an input data repository.\n", + " - We've used the `--rerun` argument to set the location of the output repository, in this case `$HOME/WORK/rerun/isr`. You can also use `--output` to set the path more directly, but we recommend `--rerun` because it enforces a nice convention for where to put outputs that helps with discoverability.\n", + " - The `--id` argument sets the *data ID(s)* to be processed, in this case a single CCD from a single visit. All `CmdLineTasks` share a fairly sophisticated syntax for expressions that match multiple data IDs, which you can learn more about by running any `CmdLineTask` with `--help`.\n", + " - We've overridden a configuration value with the `--config` option, in this case to make sure the just-after-ISR image file is written. Running a `CmdLineTask` automatically also includes applying configuation overrides that customize the task for the kind of data you're processing (i.e. which camera it comes from), and that's how the task knows to run the custom ISR task for HSC, rather than the generic default. You can see all of the config options for a `CmdLineTask` by running with `--show config`, though the results can be a bit overwhelming.\n", + " \n", + "The rest of this tutorial is focused on using LSST software as a Python library, so this will be the last thing we run from the command-line. Again, for more information about how to run LSST's existing processing scripts from the command-line, check out [DM Tech Note 23](https://dmtn-023.lsst.io/)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Data Access with Butler\n", + "\n", + "The outputs of `CmdLineTasks`, like their inputs, are organized into data repositories, which are managed by an object called `Butler`. To retrieve a dataset from the `Butler`, we start by constructing one pointing to the output repository from the processing run (which is now an input repository for this `Butler`, which won't have an output repository since we won't be writing any more files):" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.daf.persistence import Butler\n", + "butler = Butler(inputs=os.path.join(WORK_DIR, \"rerun/isr\"))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can then call `get` with the name and data ID of the dataset. The name of the image that's saved directly after ISR is `postISRCCD`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Image, Boxes, and (Crude) Image Display\n", + "\n", + "A full 2k x 4k HSC CCD is a pretty big image to display when you don't have specialized display code. The DM stack does have specialized display code, but it either requires DS9 (which requires some ssh tunnels to use with data living on a server) or a Firefly server installation. For this tutorial, we'll just throw together a naive matplotlib display function, and create a view to a subimage that we'll display instead of the full image.\n", + "\n", + "This section features a few of our most important class objects:\n", + "\n", + "- `lsst.afw.image.Exposure` is an image object that actually holds three image planes: the science image (`Exposure.image`), an image of variance in every pixel (`Exposure.variance`), an integer bit mask (`Exposure.mask`). It also holds a lot of more complex objects that characterize the image, such as a point-spread function (`lsst.afw.detection.Psf`) and world-coordinate system (`lsst.afw.image.Wcs`). Most of these objects aren't filled in yet, because all we've run so far is ISR. It doesn't generally make sense to perform mathematical operations (i.e. addition) on `Exposure`s, because those operations aren't always well-defined on the more complex objects. You can get a `MaskedImage` object with the same image, mask, and variance planes that does support mathematical operations but doesn't contain `Psf`s and `Wcs`s (etc) with `Exposure.maskedImage`.\n", + "\n", + "- The `Exposure.image` and `Exposure.variance` properties return `lsst.afw.image.Image` objects. These have a `.array` property that returns a `numpy.ndarray` view to the `Image`'s pixels. Conceptually, you should think of an `Image` as just a `numpy.ndarray` with a possibly nonzero origin.\n", + "\n", + "- The `Exposure.mask` property returns a `lsst.afw.image.Mask` object, which behaves like an `Image` with a dictionary-like object that relates string labels to bit numbers.\n", + "\n", + "- All of these image-like objects have a `getBBox()` method, which returns a `lsst.afw.geom.Box2I`. The minimum and maximum points of a `Box2I` are specified in integers that correspond to the *centers* of the lower-left and upper-right pixels in the box, but the box conceptually contains the entirety of those pixels. To get a box with a floating-point representation of the same boundary for the `extent` argument to `imshow` below, we construct a `Box2D` from the `Box2I`.\n", + "\n", + "- `Point2I` and `Extent2I` are used to represent absolute positions and offsets between positions as integers (respectively). These have floating-point counterparts `Point2D` and `Extent2D`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.afw.geom import Box2D, Box2I, Point2I, Extent2I\n", + "from lsst.afw.image import Exposure" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Execute this cell (and the one below) to re-load the post-ISR Exposure from disk after\n", + "# modifying it.\n", + "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "bbox = exposure.getBBox()\n", + "bbox.grow(-bbox.getDimensions()//3) # box containing the central third (in each dimension)\n", + "bbox.grow(-Extent2I(0, 400)) # make it a bit smaller in x\n", + "# exposure[bbox] would also work here because exposure.getXY0() == (0, 0),\n", + "# but it's dangerous in general because it ignores that origin.\n", + "sub = Exposure(exposure, bbox=bbox, dtype=exposure.dtype, deep=False)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import matplotlib\n", + "%matplotlib inline\n", + "matplotlib.rcParams[\"figure.figsize\"] = (8, 6)\n", + "matplotlib.rcParams[\"font.size\"] = 12" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "def display(image, mask=None, colors=None, alpha=0.40, **kwds):\n", + " box = Box2D(image.getBBox())\n", + " extent = (box.getMinX(), box.getMaxX(), box.getMinY(), box.getMaxY())\n", + " kwds.setdefault(\"extent\", extent)\n", + " kwds.setdefault(\"origin\", \"lower\")\n", + " kwds.setdefault(\"interpolation\", \"nearest\")\n", + " matplotlib.pyplot.imshow(image.array, **kwds)\n", + " kwds.pop(\"vmin\", None)\n", + " kwds.pop(\"vmax\", None)\n", + " kwds.pop(\"norm\", None)\n", + " kwds.pop(\"cmap\", None)\n", + " if mask is not None:\n", + " for plane, color in colors.items():\n", + " array = np.zeros(mask.array.shape + (4,), dtype=float)\n", + " rgba = np.array(matplotlib.colors.hex2color(matplotlib.colors.cnames[color]) + (alpha, ),\n", + " dtype=float)\n", + " np.multiply.outer((mask.array & mask.getPlaneBitMask(plane)).astype(bool), rgba, out=array)\n", + " matplotlib.pyplot.imshow(array, **kwds)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And now here's the (cutout) of the detrended image. I've cheated in setting the scale by looking at the background level in advance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display(sub.image, vmin=175, vmax=300, cmap=matplotlib.cm.gray)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Display the whole image with Firefly" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "import lsst.afw.display as afw_display\n", + "afw_display.setDefaultBackend('firefly')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In the next cell, change the `name` parameter to your own unique identifier. If you chose, for example, `my_name`, you would open a browser window to http://lsst-demo.ncsa.illinois.edu/firefly/slate.html;wsch=my_name" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display1 = afw_display.Display(frame=1, \n", + " host='lsst-demo.ncsa.illinois.edu', port=80,\n", + " name='stargaser')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display1.mtv(exposure)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To define additional displays, just specify the frame number and backend. You can use only one Firefly host and port configuration in a given Python session." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display2 = afw_display.getDisplay(frame=2)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display2.mtv(sub)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To allow additional functions provided by firefly_client, an instance is defined here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "import firefly_client\n", + "ffclient = firefly_client.FireflyClient('lsst-demo.ncsa.illinois.edu:80',\n", + " channel='stargaser',\n", + " html_file='slate.html')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Background Subtraction and Task Configuration" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The next step we usually take is to estimate and subtract the background, using `lsst.meas.algorithms.SubtractBackgroundTask`. This is a regular `Task`, not a `CmdLineTask`, and hence we'll just pass it our `Exposure` object (it operates in-place) instead of a `Butler`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.meas.algorithms import SubtractBackgroundTask" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "bkgConfig = SubtractBackgroundTask.ConfigClass()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Execute this cell to get fun & terrible results!\n", + "bkgConfig.useApprox = False\n", + "bkgConfig.binSize = 20" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The pattern for configuration here is the same as it was for `SubaruIsrTask`, but here we're setting values directly instead of loading a configuration file from the `obs_subaru` camera-specialization package. The `config` object here is an instance of a class that inherits from `lsst.pex.config.Config` that contains a set of `lsst.pex.config.Field` objects that define the options that can be modified. Each `Field` behaves more or less like a Python `property`, and you can get information on all of the fields in a config object by either using `help`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "help(bkgConfig)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "SubtractBackgroundTask.ConfigClass.algorithm?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "bkgTask = SubtractBackgroundTask(config=bkgConfig)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "bkgResult = bkgTask.run(exposure)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display(sub.image, vmin=-0.5, vmax=100, cmap=matplotlib.cm.gray)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Firefly display" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display3 = afw_display.getDisplay(frame=3)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display3.mtv(sub.image)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If you've run through all of these steps after executing the cell that warns about terrible results, you should notice that the galaxy in the upper right has been oversubtracted.\n", + "\n", + "**EXERCISE**: Before continuing on, re-load the exposure from disk, reset the configuration and `Task` instances, and re-run without executing the cell that applies bad values to the config, all by just re-executing the right cells above. You should end up an image in which the upper-right galaxy looks essentially the same as it does in the image before we subtracted the background." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**ANSWER:**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", + "sub = Exposure(exposure, bbox=bbox, dtype=exposure.dtype, deep=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We could construct a new instance of `SubtractBackgroundTask.ConfigClass` and pass it unmodified to the constructor (and that's what you'd get by strictly copying existing cells), but it also works to just construct the task with no arguments." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "bkgTask = SubtractBackgroundTask()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "bkgResult = bkgTask.run(exposure)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Installing an Initial-Guess PSF" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Most later processing steps require a PSF model, which is represented by a `Psf` object that's attached to the `Exposure`. For now, we'll just make a Gaussian PSF with some guess at the seeing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.meas.algorithms import SingleGaussianPsf" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "FWHM_TO_SIGMA = 1.0/(2*np.sqrt(2*np.log(2)))\n", + "PIXEL_SCALE = 0.168 # arcsec/pixel\n", + "SEEING = 0.7 # FWHM in arcsec\n", + "sigma = FWHM_TO_SIGMA*SEEING/PIXEL_SCALE\n", + "width = int(sigma*3)*2 + 1\n", + "psf = SingleGaussianPsf(width, width, sigma=sigma)\n", + "exposure.setPsf(psf)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "A `Psf` object can basically just do one thing: it can return an image of itself at a point. `SingleGaussianPsf` represents a constant PSF, so it always returns the same image, regardless of the point you give it.\n", + "\n", + "But there are two ways to evaluate a `Psf` at a point. If you want an image centered on the middle pixel, and that middle pixel to be the origin - what you'd usually want if you're going to convolve the PSF with another model - use `computeKernelImage(point)`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from lsst.afw.geom import Point2D\n", + "display(psf.computeKernelImage(Point2D(60.5, 7.2)), cmap=matplotlib.cm.viridis)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Firefly display" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display4 = afw_display.Display(frame=4)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display4.mtv(psf.computeKernelImage(Point2D(60.5, 7.2)))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In the next two code cells, approximate the result of the matplotlib viridis colormap." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ffclient.dispatch_remote_action(ffclient.channel, \n", + " action_type='ImagePlotCntlr.ColorChange', \n", + " payload = {'plotId':'4', 'cbarId': '11'})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "A current limitation of Firefly is that the image display must complete before the zoom command in the next cell is executed." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display4.scale('linear', 0.0, 0.2)\n", + "display4.zoom(30)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If you want to compare the PSF to a star at the exact same position, use `computeImage(point)`. That will shift the image returned by `computeKernelImage(point)` to the right sub-pixel offset, and update the origin of the image to take care of the rest, so you end up with a postage stamp in the same coordinate system as the original image where the star is." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display(psf.computeImage(Point2D(60.5, 7.2)), cmap=matplotlib.cm.viridis)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Firefly display" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display5 = afw_display.Display(frame=5)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display5.mtv(psf.computeImage(Point2D(60.5, 7.2)))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In the next two code cells, approximate the result of the matplotlib viridis colormap." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ffclient.dispatch_remote_action(ffclient.channel, \n", + " action_type='ImagePlotCntlr.ColorChange', \n", + " payload = {'plotId':'5', 'cbarId': '11'})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "A current limitation of Firefly is that the image display must complete before the zoom command in the next cell is executed." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display5.scale('linear', 0.0, 0.2)\n", + "display5.zoom(30)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Removing Cosmic Rays\n", + "\n", + "Cosmic rays are detected and interpolated by `RepairTask`, which also sets mask planes to indicate where the cosmic rays were (\"CR\") and which pixels were interpolated (\"INTERP\"; this may happen due to saturation or bad pixels as well). Because we're just using the default configuration, we can skip creating a config object and just construct the `Task` with no arguments." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.pipe.tasks.repair import RepairTask" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "repairTask = RepairTask()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "repairTask.run(exposure)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display(sub.image, mask=sub.mask, colors={\"CR\": \"yellow\"},\n", + " vmin=-0.5, vmax=100, alpha=0.8, cmap=matplotlib.cm.gray)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Firefly display" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display6 = afw_display.getDisplay(frame=6)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display6.setMaskPlaneColor('CR', afw_display.YELLOW)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display6.mtv(sub)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display6.zoom(0.5)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Detecting Sources\n", + "\n", + "Unlike the other `Task`s we've dealt with so far, `SourceDetectionTask` creates a `SourceCatalog` in addition to updating the image (all it does to the image is add a \"DETECTED\" mask plane). All `Task`s that work with catalogs need to be initialized with a `lsst.afw.table.Schema` object, to which the `Task` will add the fields necessary to store its outputs. A `SourceCatalog`'s `Schema` cannot be modified after the `SourceCatalog` has been constructed, which means it's necessary to construct all `Schema`-using `Task`s before actually running any of them.\n", + "\n", + "Each record in the catalog returned by `SourceDetectionTask` has a `Footprint` object attached to it. A `Footprint` represents the approximate region covered by a source in a run-length encoding data structure. It also contains a list of peaks found within that region. The \"DETECTED\" mask plane is set to exactly the pixels covered by any `Footprint` in the returned catalog." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.meas.algorithms import SourceDetectionTask\n", + "from lsst.afw.table import SourceTable, SourceCatalog" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "schema = SourceTable.makeMinimalSchema()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "detectTask = SourceDetectionTask(schema=schema)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# A SourceTable is really just a factory object for records; don't confuse it with SourceCatalog, which is\n", + "# usually what you want. But a SourceTable *is* what SourceDetectionTask wants here.\n", + "table = SourceTable.make(schema)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "detectResult = detectTask.run(table, exposure)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display(sub.image, mask=sub.mask, colors={\"DETECTED\": \"blue\"}, vmin=-0.5, vmax=100, cmap=matplotlib.cm.gray)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Firefly display" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display7 = afw_display.getDisplay(frame=7)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display7.mtv(exposure)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "display7.setMaskPlaneColor('DETECTED', afw_display.BLUE)\n", + "display7.setMaskTransparency(20)\n", + "display7.zoom(0.5)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Deblending\n", + "\n", + "Deblending attempts to separate detections with multiple peaks into separate objects. We keep all of the original sources in the `SourceCatalog` (called `parent`s) when we deblend, but for each `parent` source that contains more than one peak, we create a new record (called a `child`) for each of those peaks. The `Footprint`s attached to the `child` objects are instances of a subclass called `HeavyFootprint`, which include new deblended pixel values as well as the region description. These can be used by calling `insert` to replace an `Image`'s pixels with the `HeavyFootprint`'s pixels.\n", + "\n", + "**EXERCISE**: This section will not run if the cells are executed naively in order. At some point you'll have to go re-execute one or more cells in the previous section to get the right behavior. Which one(s)? Why? Copy those cells here (in the right places) when you figure it out." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**ANSWER:**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.meas.deblender import SourceDeblendTask" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It isn't necessary to re-load the `Exposure` from disk (and re-do background subtraction) to get this exercise to run, but not doing so will result in re-subtracting the residual background repeatedly every time `SourceDetectionTask` is run, which means the results will change slightly each time." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", + "sub = Exposure(exposure, bbox=bbox, dtype=exposure.dtype, deep=False)\n", + "exposure.setPsf(psf)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It isn't necessary to construct all of these tasks before running any of them (just the ones that take and modify a schema), but it's easier to get things right if you follow that principle." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "schema = SourceTable.makeMinimalSchema()\n", + "bkgTask = SubtractBackgroundTask()\n", + "repairTask = RepairTask()\n", + "detectTask = SourceDetectionTask(schema=schema)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "deblendTask = SourceDeblendTask(schema=schema)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we run the tasks from the previous sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "bkgResult = bkgTask.run(exposure)\n", + "repairTask.run(exposure)\n", + "table = SourceTable.make(schema)\n", + "detectResult = detectTask.run(table, exposure)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And, finally, we can run the task introduced in this section." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "catalog = detectResult.sources" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "deblendTask.run(exposure, catalog)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To inspect some deblender outputs, we'll start by finding some parent objects that were deblended into multiple children, by looking at the `deblend_nChild` field (which was added to the `Schema` when we constructed the `SourceDeblendTask`, and populated when we called `run`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Find some blended sources inside the subimage:\n", + "blendParents = []\n", + "for record in catalog:\n", + " if record.get(\"deblend_nChild\") > 0 and bbox.contains(record.getFootprint().getBBox()):\n", + " blendParents.append(record)\n", + "# Sort by peak brightness so we can look at something with decent S/N\n", + "blendParents.sort(key=lambda r: -r.getFootprint().getPeaks()[0].getPeakValue())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.afw.image import Image" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The image of the parent object is just the original image, but we'll cut out just the region inside its `Footprint`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "blendParentImage = Image(exposure.image, bbox=blendParents[0].getFootprint().getBBox(),\n", + " deep=True, dtype=np.float32)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we'll insert the deblended child pixels into blank images of the same size:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "blendChildImages = []\n", + "for blendChild in catalog.getChildren(blendParents[0].getId()):\n", + " image = Image(blendParentImage.getBBox(), dtype=np.float32)\n", + " blendChild.getFootprint().insert(image)\n", + " blendChildImages.append(image)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "nSubPlots = len(blendChildImages) + 1\n", + "nCols = 3\n", + "nRows = nSubPlots//nCols + 1\n", + "matplotlib.pyplot.subplot(nRows, nCols, 1)\n", + "display(blendParentImage, vmin=-0.5, vmax=100, cmap=matplotlib.cm.gray)\n", + "for n, image in enumerate(blendChildImages):\n", + " matplotlib.pyplot.subplot(nRows, nCols, n + 2)\n", + " display(image, vmin=-0.5, vmax=100, cmap=matplotlib.cm.gray)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Firefly display" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Reinitialize the image viewer" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "import time" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ffclient.reinit_viewer()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "afw_display.incrDefaultFrame()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "mydisplay = afw_display.getDisplay(frame=afw_display.incrDefaultFrame)\n", + "display0.mtv(blendParentImage)\n", + "time.sleep(2)\n", + "display0.scale('linear', -0.5, 100)\n", + "ffclient.dispatch_remote_action(ffclient.channel, \n", + " action_type='ImagePlotCntlr.ZoomImage', \n", + " payload = {'plotId':'0', 'userZoomType': 'FIT'})\n", + "for n, image in enumerate(blendChildImages):\n", + " mydisplay = afw_display.getDisplay(frame=n+1)\n", + " mydisplay.mtv(image)\n", + " time.sleep(2)\n", + " mydisplay.scale('linear', -0.5, 100)\n", + " ffclient.dispatch_remote_action(ffclient.channel, \n", + " action_type='ImagePlotCntlr.ZoomImage', \n", + " payload = {'plotId':'{}'.format(n+1),\n", + " 'userZoomType': 'FIT'})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Measurement\n", + "\n", + "`SingleFrameMeasurementTask` is typically responsible for adding most fields to a `SourceCatalog`. It runs a series of plugins that make different measurements (you can configure them with the `.plugins` dictionary-like field on its config object, and control which are run with `.names`). If the deblender has been run first, it will measure child objects using their deblended pixels.\n", + "\n", + "**EXERCISE**: Like the Deblending section, you'll have to re-execute some previous cells somewhere in this section to get the right behavior. Copy those cells into the right places here once you've gotten it working." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**ANSWER:** As in the Deblending section exercise, we reload the exposure, create all of the tasks, and then run all of the tasks." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.meas.base import SingleFrameMeasurementTask" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", + "sub = Exposure(exposure, bbox=bbox, dtype=exposure.dtype, deep=False)\n", + "exposure.setPsf(psf)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "schema = SourceTable.makeMinimalSchema()\n", + "bkgTask = SubtractBackgroundTask()\n", + "repairTask = RepairTask()\n", + "detectTask = SourceDetectionTask(schema=schema)\n", + "deblendTask = SourceDeblendTask(schema=schema)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "measureConfig = SingleFrameMeasurementTask.ConfigClass()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# What measurements are configured to run\n", + "print(measureConfig.plugins.names)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Import an extension module that adds a new measurement\n", + "import lsst.meas.extensions.photometryKron" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# What measurements *could* be configured to run\n", + "print(list(measureConfig.plugins.keys()))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Configure the new measurement to run\n", + "measureConfig.plugins.names.add(\"ext_photometryKron_KronFlux\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "measureTask = SingleFrameMeasurementTask(schema=schema, config=measureConfig)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "bkgResult = bkgTask.run(exposure)\n", + "repairTask.run(exposure)\n", + "table = SourceTable.make(schema)\n", + "detectResult = detectTask.run(table, exposure)\n", + "catalog = detectResult.sources\n", + "deblendTask.run(exposure, catalog)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "measureTask.run(catalog, exposure)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We'll show some of the results of measurement by overlaying the measured ellipses on the image.\n", + "\n", + "The shapes and centroids we use here (by calling `record.getX()`, `record.getY()`, `record.getShape()`) are aliases (called \"slots\") to fields with longer names that are our recommended measurements for these quantities. You can see the set of aliases by printing the schema (see next section)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.afw.geom.ellipses import Axes" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display(sub.image, mask=sub.mask, colors={\"DETECTED\": \"blue\"}, vmin=-0.5, vmax=100, cmap=matplotlib.cm.gray)\n", + "\n", + "for record in catalog:\n", + " if record.get(\"deblend_nChild\") != 0:\n", + " continue\n", + " axes = Axes(record.getShape()) # convert to A, B, THETA parameterization\n", + " axes.scale(2.0) # matplotlib uses diameters, not radii\n", + " patch = matplotlib.patches.Ellipse((record.getX(), record.getY()),\n", + " axes.getA(), axes.getB(), axes.getTheta() * 180.0 / np.pi,\n", + " fill=False, edgecolor=\"green\")\n", + " matplotlib.pyplot.gca().add_patch(patch)\n", + "matplotlib.pyplot.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Working With Catalogs" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Print the schema:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "print(catalog.getSchema())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Get arrays of columns (requires the catalog to be continguous in memory, which we can guarantee with a deep copy):" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "catalog = catalog.copy(deep=True)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "psfFlux = catalog[\"base_PsfFlux_flux\"]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that boolean values are stored in `Flag` columns, which are packed into bits. Unlike other column types, when you get an array of a `Flag` column, you get a copy, not a view." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Use `Key` objects instead of strings to do fast repeated access to fields when iterating over records:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "key = catalog.getSchema().find(\"deblend_nChild\").key\n", + "deblended = [record for record in catalog if record.get(key) == 0]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can also get `dict` version of a subset of a `Schema`, a `Catalog`, or a `Record` by calling either `extract` methods with a glob:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "catalog[0].extract(\"base_PsfFlux_*\") # or regex='...'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For `Record`s, the dict values are just the values of the fields, and for `Catalogs`, they're `numpy.ndarray` columns. For `Schema`s they're `SchemaItem`s, which behave liked a named tuple containing a `Key` and a `Field`, which contains more descriptive information." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Get an Astropy view of the catalog (from which you can make a Pandas view):" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "table = catalog.asAstropy()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can find some reference documentation for the catalog library [here](http://doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/afw_table.html)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Exercises\n", + "\n", + "1) Make a scatter plot of Kron Flux vs. Psf Flux." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It's not a pretty plot, with so few points and no calibration, but it's pretty simple:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "matplotlib.pyplot.scatter(catalog[\"slot_PsfFlux_flux\"], catalog[\"ext_photometryKron_KronFlux_flux\"], alpha=0.5)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "2) Write a single function that performs all of the above steps on a post-ISR `Exposure` object, modifying the `Exposure` in-place and returning a new `SourceCatalog` with a complete set of measurements." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "def processExposure1(exposure):\n", + " schema = SourceTable.makeMinimalSchema()\n", + " bkgTask = SubtractBackgroundTask()\n", + " repairTask = RepairTask()\n", + " detectTask = SourceDetectionTask(schema=schema)\n", + " deblendTask = SourceDeblendTask(schema=schema)\n", + " measureConfig = SingleFrameMeasurementTask.ConfigClass()\n", + " measureConfig.plugins.names.add(\"ext_photometryKron_KronFlux\")\n", + " measureTask = SingleFrameMeasurementTask(schema=schema, config=measureConfig)\n", + " bkgResult = bkgTask.run(exposure)\n", + " repairTask.run(exposure)\n", + " table = SourceTable.make(schema)\n", + " detectResult = detectTask.run(table, exposure)\n", + " catalog = detectResult.sources\n", + " deblendTask.run(exposure, catalog)\n", + " measureTask.run(catalog, exposure)\n", + " return catalog" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", + "sub = Exposure(exposure, bbox=bbox, dtype=exposure.dtype, deep=False)\n", + "exposure.setPsf(psf)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "catalog = processExposure1(exposure)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "3) Add PSF modeling to the end of that function, delegating most of the work to `lsst.pipe.tasks.MeasurePsfTask`. You may want to use a higher threshold (e.g. 50-sigma) for detection, since PSF modeling should only use bright stars. Hint: `MeasurePsfTask` requires the catalog you pass it to be contiguous; if it isn't you'll (unfortunately) get a very confusing error message." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.pipe.tasks.measurePsf import MeasurePsfTask\n", + "\n", + "def processExposure2(exposure):\n", + " schema = SourceTable.makeMinimalSchema()\n", + " bkgTask = SubtractBackgroundTask()\n", + " repairTask = RepairTask()\n", + " detectConfig = SourceDetectionTask.ConfigClass()\n", + " detectConfig.thresholdValue = 50.0\n", + " detectTask = SourceDetectionTask(config=detectConfig, schema=schema)\n", + " deblendTask = SourceDeblendTask(schema=schema)\n", + " measureConfig = SingleFrameMeasurementTask.ConfigClass()\n", + " measureConfig.plugins.names.add(\"ext_photometryKron_KronFlux\")\n", + " measureTask = SingleFrameMeasurementTask(schema=schema, config=measureConfig)\n", + " psfTask = MeasurePsfTask(schema=schema)\n", + " bkgResult = bkgTask.run(exposure)\n", + " repairTask.run(exposure)\n", + " table = SourceTable.make(schema)\n", + " detectResult = detectTask.run(table, exposure)\n", + " catalog = detectResult.sources\n", + " deblendTask.run(exposure, catalog)\n", + " catalog = catalog.copy(deep=True)\n", + " measureTask.run(catalog, exposure)\n", + " psfTask.run(exposure, catalog)\n", + " return catalog" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", + "sub = Exposure(exposure, bbox=bbox, dtype=exposure.dtype, deep=False)\n", + "exposure.setPsf(psf)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "catalog = processExposure2(exposure)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "4) Make images of the PSF stars, the PSF model at the position of those stars, and the difference between them." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.afw.image import PARENT\n", + "\n", + "stamps = []\n", + "for record in catalog[catalog[\"calib_psfUsed\"]]:\n", + " # Get the PSF, convert to a single-precision image (yes, ugly syntax)\n", + " psfImage = exposure.getPsf().computeImage(record.getCentroid()).convertF()\n", + " # Make a subimage (yes, even uglier syntax). Last boolean is whether to make a deep copy.\n", + " starImage = exposure.image.Factory(exposure.image, psfImage.getBBox(), PARENT, True)\n", + " # Divide the star image by the PSF flux (note that this wouldn't work as well if we had applied aperture corrections)\n", + " starImage /= record.get(\"slot_PsfFlux_flux\")\n", + " # Make a deep copy, and subtract to get the residuals\n", + " residualImage = starImage.clone()\n", + " residualImage -= psfImage\n", + " stamps.append((psfImage, starImage, residualImage))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "limit = 4\n", + "nCols = 3\n", + "nRows = min(len(stamps), limit)\n", + "index = 1\n", + "for psfImage, starImage, residualImage in stamps[:nRows]:\n", + " vmin = min(psfImage.array.min(), starImage.array.min(), residualImage.array.min())\n", + " vmax = max(psfImage.array.max(), starImage.array.max(), residualImage.array.max())\n", + " matplotlib.pyplot.subplot(nRows, nCols, index)\n", + " display(starImage, vmin=vmin, vmax=vmax)\n", + " index += 1\n", + " matplotlib.pyplot.subplot(nRows, nCols, index)\n", + " display(psfImage, vmin=vmin, vmax=vmax)\n", + " index += 1\n", + " matplotlib.pyplot.subplot(nRows, nCols, index)\n", + " display(residualImage, vmin=vmin, vmax=vmax)\n", + " index += 1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Clearly the PSF model we get without tuning any of the configuration for `MeasurePsfTask` isn't terribly good." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "5) Add another detect-deblend-measure sequence after PSF modeling at a deeper threshold." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from lsst.pipe.tasks.measurePsf import MeasurePsfTask\n", + "\n", + "def processExposure3(exposure):\n", + " # Construct tasks and schema for the first pass to do PSF estimation\n", + " schema1 = SourceTable.makeMinimalSchema()\n", + " bkgTask = SubtractBackgroundTask()\n", + " repairTask = RepairTask()\n", + " detectConfig1 = SourceDetectionTask.ConfigClass()\n", + " detectConfig1.thresholdValue = 50.0\n", + " detectTask1 = SourceDetectionTask(config=detectConfig1, schema=schema1)\n", + " deblendTask1 = SourceDeblendTask(schema=schema1)\n", + " measureTask1 = SingleFrameMeasurementTask(schema=schema1)\n", + " psfTask = MeasurePsfTask(schema=schema1)\n", + " # Construct tasks and schema for the second pass\n", + " schema2 = SourceTable.makeMinimalSchema()\n", + " detectTask2 = SourceDetectionTask(schema=schema2)\n", + " deblendTask2 = SourceDeblendTask(schema=schema2)\n", + " measureConfig2 = SingleFrameMeasurementTask.ConfigClass()\n", + " measureConfig2.plugins.names.add(\"ext_photometryKron_KronFlux\")\n", + " measureTask2 = SingleFrameMeasurementTask(schema=schema2, config=measureConfig2)\n", + " # Run the first pass, and do PSF estimation.\n", + " bkgResult = bkgTask.run(exposure)\n", + " repairTask.run(exposure)\n", + " table1 = SourceTable.make(schema1)\n", + " detectResult1 = detectTask1.run(table1, exposure)\n", + " catalog1 = detectResult1.sources\n", + " deblendTask1.run(exposure, catalog1)\n", + " catalog1 = catalog1.copy(deep=True)\n", + " measureTask1.run(catalog1, exposure)\n", + " # Run the second pass\n", + " table2 = SourceTable.make(schema2)\n", + " detectResult2 = detectTask2.run(table2, exposure)\n", + " catalog2 = detectResult2.sources\n", + " deblendTask2.run(exposure, catalog2)\n", + " measureTask2.run(catalog2, exposure)\n", + " return catalog1, catalog2" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", + "sub = Exposure(exposure, bbox=bbox, dtype=exposure.dtype, deep=False)\n", + "exposure.setPsf(psf)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "catalog1, catalog2 = processExposure3(exposure)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "6) Rewrite the function as a class that constructs all of the `Task`s that it use in `__init__` and processes a single `Exposure` with you call its `run` method. Make sure it will behave properly if `run` is called multiple times wiht different `Exposure` objects." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "class ProcessExposure(object):\n", + " \n", + " def __init__(self):\n", + " # Construct tasks and schema for the first pass to do PSF estimation\n", + " self.schema1 = SourceTable.makeMinimalSchema()\n", + " self.bkgTask = SubtractBackgroundTask()\n", + " self.repairTask = RepairTask()\n", + " detectConfig1 = SourceDetectionTask.ConfigClass()\n", + " detectConfig1.thresholdValue = 50.0\n", + " self.detectTask1 = SourceDetectionTask(config=detectConfig1, schema=self.schema1)\n", + " self.deblendTask1 = SourceDeblendTask(schema=self.schema1)\n", + " self.measureTask1 = SingleFrameMeasurementTask(schema=self.schema1)\n", + " self.psfTask = MeasurePsfTask(schema=self.schema1)\n", + " # Construct tasks and schema for the second pass\n", + " self.schema2 = SourceTable.makeMinimalSchema()\n", + " self.detectTask2 = SourceDetectionTask(schema=self.schema2)\n", + " self.deblendTask2 = SourceDeblendTask(schema=self.schema2)\n", + " measureConfig2 = SingleFrameMeasurementTask.ConfigClass()\n", + " measureConfig2.plugins.names.add(\"ext_photometryKron_KronFlux\")\n", + " self.measureTask2 = SingleFrameMeasurementTask(schema=self.schema2, config=measureConfig2)\n", + " \n", + " def run(self, exposure):\n", + " # Run the first pass, and do PSF estimation.\n", + " bkgResult = self.bkgTask.run(exposure)\n", + " self.repairTask.run(exposure)\n", + " table1 = SourceTable.make(self.schema1)\n", + " detectResult1 = self.detectTask1.run(table1, exposure)\n", + " catalog1 = detectResult1.sources\n", + " self.deblendTask1.run(exposure, catalog1)\n", + " catalog1 = catalog1.copy(deep=True)\n", + " self.measureTask1.run(catalog1, exposure)\n", + " # Run the second pass\n", + " table2 = SourceTable.make(self.schema2)\n", + " detectResult2 = self.detectTask2.run(table2, exposure)\n", + " catalog2 = detectResult2.sources\n", + " self.deblendTask2.run(exposure, catalog2)\n", + " self.measureTask2.run(catalog2, exposure)\n", + " return catalog1, catalog2" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", + "sub = Exposure(exposure, bbox=bbox, dtype=exposure.dtype, deep=False)\n", + "exposure.setPsf(psf)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "task = ProcessExposure()\n", + "catalog1, catalog2 = task.run(exposure)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "- " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.2" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 1bb012814e7640a5bf54241076dcee3f65b2c43e Mon Sep 17 00:00:00 2001 From: David Shupe Date: Thu, 14 Dec 2017 15:20:59 -0800 Subject: [PATCH 2/3] flesh out all displays and charts --- tutorial-firefly.ipynb | 819 ++++++++++++++++++++++++----------------- 1 file changed, 480 insertions(+), 339 deletions(-) diff --git a/tutorial-firefly.ipynb b/tutorial-firefly.ipynb index ac2b855..9ba035d 100644 --- a/tutorial-firefly.ipynb +++ b/tutorial-firefly.ipynb @@ -11,6 +11,27 @@ "More notebook examples can be found here: https://github.com/RobertLuptonTheGood/notebooks/tree/master/Demos" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Firefly displays" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook is a version of the answers notebook, including both the original matplotlib displays and adding Firefly equivalents." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The directories in the next sections have been modified to use mounted directories in a Docker container." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -25,13 +46,12 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "%%script bash\n", - "export DATA_DIR=$HOME/data\n", + "#export DATA_DIR=$HOME/data\n", + "export DATA_DIR=/home/vagrant/mnt/data\n", "export CI_HSC_DIR=$DATA_DIR/ci_hsc_small\n", "mkdir -p $DATA_DIR\n", "cd $DATA_DIR\n", @@ -39,7 +59,8 @@ " curl -O http://lsst-web.ncsa.illinois.edu/~krughoff/data/small_demo.tar.gz\n", " tar zxvf small_demo.tar.gz\n", "fi\n", - "export WORK_DIR=$HOME/WORK\n", + "#export WORK_DIR=$HOME/WORK\n", + "export WORK_DIR=/home/vagrant/mnt/work\n", "if ! [ -d $WORK_DIR ]; then\n", " mkdir -p $WORK_DIR\n", " echo \"lsst.obs.hsc.HscMapper\" > $WORK_DIR/_mapper\n", @@ -55,15 +76,13 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "import os\n", - "DATA_DIR = os.path.join(os.environ['HOME'], \"data\")\n", + "DATA_DIR = os.path.join('/home/vagrant/mnt', \"data\")\n", "CI_HSC_DIR = os.path.join(DATA_DIR, \"ci_hsc_small\")\n", - "WORK_DIR = os.path.join(os.environ['HOME'], \"WORK\")" + "WORK_DIR = os.path.join('/home/vagrant/mnt', \"WORK\")" ] }, { @@ -88,7 +107,8 @@ "outputs": [], "source": [ "%%script bash\n", - "processCcd.py $HOME/WORK --rerun isr --id visit=903334 ccd=16 --config isr.doWrite=True" + "#processCcd.py $HOME/WORK --rerun isr --id visit=903334 ccd=16 --config isr.doWrite=True\n", + "processCcd.py /home/vagrant/mnt/work --rerun isr --id visit=903334 ccd=16 --config isr.doWrite=True" ] }, { @@ -117,12 +137,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.daf.persistence import Butler\n", + "#butler = Butler(inputs=os.path.join(WORK_DIR, \"rerun/isr\"))\n", "butler = Butler(inputs=os.path.join(WORK_DIR, \"rerun/isr\"))" ] }, @@ -136,9 +155,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)" @@ -168,9 +185,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.afw.geom import Box2D, Box2I, Point2I, Extent2I\n", @@ -180,9 +195,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "# Execute this cell (and the one below) to re-load the post-ISR Exposure from disk after\n", @@ -193,9 +206,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "bbox = exposure.getBBox()\n", @@ -209,9 +220,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -224,9 +233,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "def display(image, mask=None, colors=None, alpha=0.40, **kwds):\n", @@ -262,22 +269,27 @@ "metadata": {}, "outputs": [], "source": [ - "display(sub.image, vmin=175, vmax=300, cmap=matplotlib.cm.gray)" + "display(sub.image, vmin=100, vmax=300, cmap=matplotlib.cm.gray)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Display the subimage with Firefly" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Display the whole image with Firefly" + "As much as possible, we will use the lsst.afw.display with the Firefly backend." ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "import lsst.afw.display as afw_display\n", @@ -288,81 +300,71 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In the next cell, change the `name` parameter to your own unique identifier. If you chose, for example, `my_name`, you would open a browser window to http://lsst-demo.ncsa.illinois.edu/firefly/slate.html;wsch=my_name" + "Choose a unique name for your Firefly channel." ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display1 = afw_display.Display(frame=1, \n", - " host='lsst-demo.ncsa.illinois.edu', port=80,\n", - " name='stargaser')" + "# Change to a unique string like your Github username\n", + "my_channel = 'stargaser'" ] }, { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, - "outputs": [], + "cell_type": "markdown", + "metadata": {}, "source": [ - "display1.mtv(exposure)" + "Choose a Firefly server. Here we use a public server at NCSA." ] }, { - "cell_type": "markdown", + "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ - "To define additional displays, just specify the frame number and backend. You can use only one Firefly host and port configuration in a given Python session." + "my_server = 'lsst-demo.ncsa.illinois.edu'" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display2 = afw_display.getDisplay(frame=2)" + "print('Open a browser window to http://{}/firefly/slate.html?__wsch={}'.format(my_server, my_channel))" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display2.mtv(sub)" + "display1 = afw_display.getDisplay(frame=1, \n", + " host=my_server, port=80,\n", + " name=my_channel)" ] }, { - "cell_type": "markdown", + "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ - "To allow additional functions provided by firefly_client, an instance is defined here." + "display1.mtv(sub)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "import firefly_client\n", - "ffclient = firefly_client.FireflyClient('lsst-demo.ncsa.illinois.edu:80',\n", - " channel='stargaser',\n", - " html_file='slate.html')" + "display1.scale('linear', min=100, max=300)" ] }, { @@ -382,9 +384,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.meas.algorithms import SubtractBackgroundTask" @@ -393,9 +393,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "bkgConfig = SubtractBackgroundTask.ConfigClass()" @@ -404,9 +402,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "# Execute this cell to get fun & terrible results!\n", @@ -424,9 +420,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "help(bkgConfig)" @@ -435,9 +429,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "SubtractBackgroundTask.ConfigClass.algorithm?" @@ -446,9 +438,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "bkgTask = SubtractBackgroundTask(config=bkgConfig)" @@ -457,9 +447,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "bkgResult = bkgTask.run(exposure)" @@ -471,7 +459,7 @@ "metadata": {}, "outputs": [], "source": [ - "display(sub.image, vmin=-0.5, vmax=100, cmap=matplotlib.cm.gray)" + "display(sub.image, vmin=-75, vmax=125, cmap=matplotlib.cm.gray)" ] }, { @@ -481,26 +469,38 @@ "### Firefly display" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To define additional displays, just specify the frame number and backend. You can use only one Firefly host and port configuration in a given Python session." + ] + }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display3 = afw_display.getDisplay(frame=3)" + "display2 = afw_display.getDisplay(frame=2)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, + "outputs": [], + "source": [ + "display2.mtv(sub.image)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, "outputs": [], "source": [ - "display3.mtv(sub.image)" + "display2.scale('linear', min=-75, max=125)" ] }, { @@ -522,9 +522,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", @@ -541,9 +539,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "bkgTask = SubtractBackgroundTask()" @@ -552,9 +548,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "bkgResult = bkgTask.run(exposure)" @@ -577,9 +571,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.meas.algorithms import SingleGaussianPsf" @@ -588,9 +580,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "FWHM_TO_SIGMA = 1.0/(2*np.sqrt(2*np.log(2)))\n", @@ -631,23 +621,45 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display4 = afw_display.Display(frame=4)" + "display3 = afw_display.Display(frame=3)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, + "outputs": [], + "source": [ + "display3.mtv(psf.computeKernelImage(Point2D(60.5, 7.2)))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that `afw_display` does not provide any method for changing a colormap." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To enable additional functionality provided by firefly_client, an instance is defined here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, "outputs": [], "source": [ - "display4.mtv(psf.computeKernelImage(Point2D(60.5, 7.2)))" + "import firefly_client\n", + "ffclient = firefly_client.FireflyClient('lsst-demo.ncsa.illinois.edu:80',\n", + " channel='stargaser',\n", + " html_file='slate.html')" ] }, { @@ -665,7 +677,7 @@ "source": [ "ffclient.dispatch_remote_action(ffclient.channel, \n", " action_type='ImagePlotCntlr.ColorChange', \n", - " payload = {'plotId':'4', 'cbarId': '11'})" + " payload = {'plotId':'3', 'cbarId': '11'})" ] }, { @@ -678,13 +690,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display4.scale('linear', 0.0, 0.2)\n", - "display4.zoom(30)" + "display3.scale('linear', 0.0, 0.2)\n", + "display3.zoom(30)" ] }, { @@ -713,23 +723,19 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display5 = afw_display.Display(frame=5)" + "display4 = afw_display.Display(frame=4)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display5.mtv(psf.computeImage(Point2D(60.5, 7.2)))" + "display4.mtv(psf.computeImage(Point2D(60.5, 7.2)))" ] }, { @@ -747,7 +753,7 @@ "source": [ "ffclient.dispatch_remote_action(ffclient.channel, \n", " action_type='ImagePlotCntlr.ColorChange', \n", - " payload = {'plotId':'5', 'cbarId': '11'})" + " payload = {'plotId':'4', 'cbarId': '11'})" ] }, { @@ -760,13 +766,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display5.scale('linear', 0.0, 0.2)\n", - "display5.zoom(30)" + "display4.scale('linear', 0.0, 0.2)\n", + "display4.zoom(30)" ] }, { @@ -781,9 +785,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.pipe.tasks.repair import RepairTask" @@ -792,9 +794,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "repairTask = RepairTask()" @@ -803,9 +803,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "repairTask.run(exposure)" @@ -818,7 +816,7 @@ "outputs": [], "source": [ "display(sub.image, mask=sub.mask, colors={\"CR\": \"yellow\"},\n", - " vmin=-0.5, vmax=100, alpha=0.8, cmap=matplotlib.cm.gray)" + " vmin=-75, vmax=125, alpha=0.8, cmap=matplotlib.cm.gray)" ] }, { @@ -831,45 +829,46 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display6 = afw_display.getDisplay(frame=6)" + "display5 = afw_display.getDisplay(frame=5)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display6.setMaskPlaneColor('CR', afw_display.YELLOW)" + "display5.setMaskPlaneColor('CR', afw_display.YELLOW)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display6.mtv(sub)" + "display5.mtv(sub)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, + "outputs": [], + "source": [ + "display5.zoom(0.5)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, "outputs": [], "source": [ - "display6.zoom(0.5)" + "display5.scale('linear', min=-75, max=125)" ] }, { @@ -886,9 +885,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.meas.algorithms import SourceDetectionTask\n", @@ -898,9 +895,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "schema = SourceTable.makeMinimalSchema()" @@ -909,9 +904,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "detectTask = SourceDetectionTask(schema=schema)" @@ -920,9 +913,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "# A SourceTable is really just a factory object for records; don't confuse it with SourceCatalog, which is\n", @@ -933,9 +924,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "detectResult = detectTask.run(table, exposure)" @@ -947,7 +936,7 @@ "metadata": {}, "outputs": [], "source": [ - "display(sub.image, mask=sub.mask, colors={\"DETECTED\": \"blue\"}, vmin=-0.5, vmax=100, cmap=matplotlib.cm.gray)" + "display(sub.image, mask=sub.mask, colors={\"DETECTED\": \"blue\"}, vmin=-75, vmax=125, cmap=matplotlib.cm.gray)" ] }, { @@ -960,36 +949,31 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display7 = afw_display.getDisplay(frame=7)" + "display6 = afw_display.getDisplay(frame=6)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display7.mtv(exposure)" + "display6.mtv(sub)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "display7.setMaskPlaneColor('DETECTED', afw_display.BLUE)\n", - "display7.setMaskTransparency(20)\n", - "display7.zoom(0.5)\n" + "display6.setMaskPlaneColor('DETECTED', afw_display.BLUE)\n", + "display6.setMaskTransparency(40)\n", + "display6.zoom(0.5)\n", + "display6.scale('linear', min=-75, max=120)" ] }, { @@ -1013,9 +997,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.meas.deblender import SourceDeblendTask" @@ -1031,9 +1013,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", @@ -1051,9 +1031,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "schema = SourceTable.makeMinimalSchema()\n", @@ -1065,9 +1043,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "deblendTask = SourceDeblendTask(schema=schema)" @@ -1083,9 +1059,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "bkgResult = bkgTask.run(exposure)\n", @@ -1104,9 +1078,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "catalog = detectResult.sources" @@ -1115,9 +1087,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "deblendTask.run(exposure, catalog)" @@ -1133,9 +1103,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "# Find some blended sources inside the subimage:\n", @@ -1150,9 +1118,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.afw.image import Image" @@ -1168,9 +1134,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "blendParentImage = Image(exposure.image, bbox=blendParents[0].getFootprint().getBBox(),\n", @@ -1187,9 +1151,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "blendChildImages = []\n", @@ -1222,6 +1184,15 @@ "### Firefly display" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import time" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1232,12 +1203,10 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "import time" + "ffclient.reinit_viewer()" ] }, { @@ -1246,7 +1215,8 @@ "metadata": {}, "outputs": [], "source": [ - "ffclient.reinit_viewer()" + "display0 = afw_display.getDisplay(frame=0)\n", + "display0.mtv(blendParentImage)" ] }, { @@ -1255,33 +1225,26 @@ "metadata": {}, "outputs": [], "source": [ - "afw_display.incrDefaultFrame()" + "zoom_fac = 8\n", + "for n, image in enumerate(blendChildImages):\n", + " mydisplay = afw_display.getDisplay(frame=n+1)\n", + " mydisplay.mtv(image)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ - "mydisplay = afw_display.getDisplay(frame=afw_display.incrDefaultFrame)\n", - "display0.mtv(blendParentImage)\n", - "time.sleep(2)\n", - "display0.scale('linear', -0.5, 100)\n", - "ffclient.dispatch_remote_action(ffclient.channel, \n", - " action_type='ImagePlotCntlr.ZoomImage', \n", - " payload = {'plotId':'0', 'userZoomType': 'FIT'})\n", - "for n, image in enumerate(blendChildImages):\n", - " mydisplay = afw_display.getDisplay(frame=n+1)\n", - " mydisplay.mtv(image)\n", - " time.sleep(2)\n", - " mydisplay.scale('linear', -0.5, 100)\n", + "for i in range(len(blendChildImages) + 1):\n", + " mydisplay = afw_display.getDisplay(frame=i)\n", + " mydisplay.scale('linear', min=-0.5, max=100)\n", + " mydisplay.zoom(zoom_fac)\n", + " time.sleep(0.5)\n", " ffclient.dispatch_remote_action(ffclient.channel, \n", - " action_type='ImagePlotCntlr.ZoomImage', \n", - " payload = {'plotId':'{}'.format(n+1),\n", - " 'userZoomType': 'FIT'})" + " action_type='ImagePlotCntlr.recenter', \n", + " payload = dict(plotId=str(i), centerOnImage=True))" ] }, { @@ -1305,9 +1268,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.meas.base import SingleFrameMeasurementTask" @@ -1316,9 +1277,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", @@ -1329,9 +1288,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "schema = SourceTable.makeMinimalSchema()\n", @@ -1344,9 +1301,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "measureConfig = SingleFrameMeasurementTask.ConfigClass()" @@ -1365,9 +1320,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "# Import an extension module that adds a new measurement\n", @@ -1387,9 +1340,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "# Configure the new measurement to run\n", @@ -1399,9 +1350,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "measureTask = SingleFrameMeasurementTask(schema=schema, config=measureConfig)" @@ -1410,9 +1359,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "bkgResult = bkgTask.run(exposure)\n", @@ -1426,9 +1373,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "measureTask.run(catalog, exposure)" @@ -1446,9 +1391,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.afw.geom.ellipses import Axes" @@ -1460,7 +1403,7 @@ "metadata": {}, "outputs": [], "source": [ - "display(sub.image, mask=sub.mask, colors={\"DETECTED\": \"blue\"}, vmin=-0.5, vmax=100, cmap=matplotlib.cm.gray)\n", + "display(sub.image, mask=sub.mask, colors={\"DETECTED\": \"blue\"}, vmin=-75, vmax=125, cmap=matplotlib.cm.gray)\n", "\n", "for record in catalog:\n", " if record.get(\"deblend_nChild\") != 0:\n", @@ -1474,6 +1417,77 @@ "matplotlib.pyplot.show()" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Firefly display" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Reinitialize the viewer" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ffclient.reinit_viewer()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display1 = afw_display.getDisplay(frame=1)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display1.mtv(sub)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display1.scale('linear', -75, 125)\n", + "display1.setMaskPlaneColor('DETECTED', afw_display.BLUE)\n", + "display1.setMaskTransparency(30)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "xy0 = sub.getXY0()\n", + "dim = sub.getDimensions()\n", + "width = sub.getWidth\n", + "for record in catalog:\n", + " if record.get(\"deblend_nChild\") != 0:\n", + " continue\n", + " xpix = record.getX()\n", + " ypix = record.getY()\n", + " if ((xpix > xy0[0]) and (xpix < xy0[0] + dim[0]) and (ypix > xy0[1]) and (ypix < xy0[1] + dim[1])):\n", + " display1.dot('@:{},{},{}'.format(record.getIxx(), record.getIxy(), record.getIyy()),\n", + " record.getX(), record.getY(), ctype=afw_display.GREEN)\n" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1492,7 +1506,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "collapsed": true + "scrolled": true }, "outputs": [], "source": [ @@ -1509,9 +1523,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "catalog = catalog.copy(deep=True)" @@ -1520,9 +1532,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "psfFlux = catalog[\"base_PsfFlux_flux\"]" @@ -1545,9 +1555,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "key = catalog.getSchema().find(\"deblend_nChild\").key\n", @@ -1587,9 +1595,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "table = catalog.asAstropy()" @@ -1627,6 +1633,93 @@ "matplotlib.pyplot.scatter(catalog[\"slot_PsfFlux_flux\"], catalog[\"ext_photometryKron_KronFlux_flux\"], alpha=0.5)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Firefly charting" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Upload the table" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ffclient.reinit_viewer()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ffclient.add_cell(row=0, col=1, width=1, height=1, element_type='tables', cell_id='main')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import tempfile\n", + "\n", + "with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as fh:\n", + " catalog.asAstropy().write(fh.name, format='csv', overwrite=True)\n", + "ffclient.show_table(ffclient.upload_file(fh.name), tbl_id='catalog')\n", + "\n", + "os.remove(fh.name)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note the table points overlay the subimage displayed earlier." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once the table has been uploaded, a chart can be made" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ffclient.add_cell(row=0, col=0, width=1, height=1, element_type='xyPlots', cell_id='chart')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ffclient.show_chart(group_id='chart',\n", + " data=[dict(type='scatter', mode='markers',\n", + " marker=dict(size=4),\n", + " tbl_id='catalog', x='tables::slot_PsfFlux_flux', \n", + " y='tables::ext_photometryKron_KronFlux_flux' )],\n", + " layout=dict(title='Kron Flux vs PSF Flux',\n", + " xaxis=dict(title='Psf Flux'),\n", + " yaxis=dict(title='Kron Flux')))" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1637,9 +1730,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "def processExposure1(exposure):\n", @@ -1664,9 +1755,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", @@ -1677,9 +1766,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "catalog = processExposure1(exposure)" @@ -1695,9 +1782,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.pipe.tasks.measurePsf import MeasurePsfTask\n", @@ -1729,9 +1814,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", @@ -1742,9 +1825,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "catalog = processExposure2(exposure)" @@ -1760,9 +1841,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.afw.image import PARENT\n", @@ -1812,6 +1891,80 @@ "Clearly the PSF model we get without tuning any of the configuration for `MeasurePsfTask` isn't terribly good." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Firefly display" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ffclient.reinit_viewer()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "index = 0\n", + "mins = {}\n", + "maxs = {}\n", + "for psfImage, starImage, residualImage in stamps[:nRows]:\n", + " vmin = min(psfImage.array.min(), starImage.array.min(), residualImage.array.min())\n", + " vmax = max(psfImage.array.max(), starImage.array.max(), residualImage.array.max())\n", + " mins[index] = vmin\n", + " maxs[index] = vmax\n", + " ffdisplay = afw_display.getDisplay(frame=index)\n", + " ffdisplay.mtv(starImage)\n", + " index += 1\n", + " mins[index] = vmin\n", + " maxs[index] = vmax\n", + " ffdisplay = afw_display.getDisplay(frame=index)\n", + " ffdisplay.mtv(psfImage)\n", + " index += 1\n", + " mins[index] = vmin\n", + " maxs[index] = vmax\n", + " ffdisplay = afw_display.getDisplay(frame=index)\n", + " ffdisplay.mtv(residualImage)\n", + " index += 1\n", + " # dummy display\n", + " zeroim = residualImage.clone()\n", + " zeroim.scaledMinus(1.0, residualImage)\n", + " mins[index] = vmin\n", + " maxs[index] = vmax\n", + " ffdisplay = afw_display.getDisplay(frame=index)\n", + " ffdisplay.mtv(zeroim)\n", + " index += 1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "for i in range(index):\n", + " ffdisplay = afw_display.getDisplay(frame=i)\n", + " ffclient.dispatch_remote_action(ffclient.channel, \n", + " action_type='ImagePlotCntlr.ColorChange', \n", + " payload = dict(plotId=str(i), cbarId='11'))\n", + "\n", + " ffdisplay.scale('linear', min=mins[i], max=maxs[i])\n", + " ffdisplay.zoom(5)\n", + " time.sleep(0.5)\n", + " ffclient.dispatch_remote_action(ffclient.channel, \n", + " action_type='ImagePlotCntlr.recenter', \n", + " payload = dict(plotId=str(i), centerOnImage=True))" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1822,9 +1975,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "from lsst.pipe.tasks.measurePsf import MeasurePsfTask\n", @@ -1868,9 +2019,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", @@ -1881,9 +2030,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "catalog1, catalog2 = processExposure3(exposure)" @@ -1899,9 +2046,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "class ProcessExposure(object):\n", @@ -1947,9 +2092,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "exposure = butler.get(\"postISRCCD\", visit=903334, ccd=16)\n", @@ -1960,9 +2103,7 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "collapsed": true - }, + "metadata": {}, "outputs": [], "source": [ "task = ProcessExposure()\n", From 0001c36b15c310e880e92977d5ace58e5bf7e6ab Mon Sep 17 00:00:00 2001 From: David Shupe Date: Thu, 14 Dec 2017 15:32:59 -0800 Subject: [PATCH 3/3] add a README for Firefly --- README_firefly.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 README_firefly.md diff --git a/README_firefly.md b/README_firefly.md new file mode 100644 index 0000000..32d372d --- /dev/null +++ b/README_firefly.md @@ -0,0 +1,33 @@ +# Using the LSST DM Stack from Python - Firefly edition + +The Jupyter Notebook in this repository is a self-guided tutorial that walks the reader through writing a simple processing script using LSST Data Management Python libraries. This tutorial was originally set up for the LSST2017 Project +and Community Workshop. + +The `tutorial-firefly.ipynb` notebook is a modified form of the notebook that gives Firefly equivalents to the +matplotlib displays used in the original version. This notebook is based on the "answers" version of the tutorial, +in which the solutions to the exercises have been included. + +As of mid-December 2017, the necessary Firefly packages are included in `lsst_distrib`. It is possible to +run the notebook in a Docker container, following the +[developer instructions for Docker](https://pipelines.lsst.io/install/docker.html#docker-tags). + +Here are Docker commands used to test this notebook: + +``` +docker run -itd -p 9745:9745 -v `pwd`:/home/vagrant/mnt --name lsst2 lsstsqre/centos:7-stack-lsst_distrib-d_2017_12_14 + +docker exec -it lsst2 /bin/bash +``` + +Then in the shell inside the container: +``` +source loadLSST.bash + +conda install jupyter notebook ipython + +cd /home/vagrant/mnt/lsst2017 + +jupyter notebook --ip 0.0.0.0 --port 9745 +``` + +Then on the host machine, open a browser to [http://localhost:9745](http://localhost:9745).