Release Notes

For changes in the Github repository since the last release see here.

2024/03/14: Beta-release, version 0.19.100.1

Overview:

In this release we have continued to evolve the support for writing mixed-reality applications using \psi and made further refinements and updates to Platform for Situated Intelligence Studio, the \psi runtime, batch processing infrastructure, as well as other updates and bug fixes across various \psi components.

In addition, we have released SIGMA, standing for Situated Interactive Guidance Management and Assistance, as a testbed application build on \psi that enables research at the intersection of mixed reality and AI. SIGMA is a mixed reality system that leverages the HoloLens 2 device, LLMs, and vision models to guide people through procedural tasks step-by-step. The system is intended to support further research at the intersection of AI and mixed reality. To learn more about SIGMA, please see the corresponding ReadMe.md file. More information about the system is available in this [arxiv paper](add link) and the corresponding extended abstract published at IEEE VR.

Changes to Mixed Reality

Several improvements and extensions have been made to the set of existing mixed reality components:

• The PhotoVideoCamera component can now be configured to turn off continuous auto-focus. Additionally, the camera can be configured in triggered mode (via VideoStreamUsesTriggeredOutputs and PreviewStreamUsesTriggeredOutputs in corresponding configuration). When in triggered mode, the camera will output messages on a cadence specified via the 'TriggeredOutputFrequency' configuration parameter and aligned (in a nearest-message sense) with the first input received on the TriggerInput receiver.

• Added the option in PhotoVideoCamera to output either NV-12 encoded images or BGRA images (thanks @austinbhale).

• When exporting HoloLens data, camera intrinsics are now exported for every frame, since they may periodically change due to continuous auto-focus.

• The Microphone component now supports configurable MediaCategory and AudioProcessing modes. Note that if not specified, the MediaCategory defaults to MediaCategory.Speech whereas previously it defaulted to MediaCategory.Other.

• The WinRT.GazeSensor component now outputs a stream of tuples of eye gaze and head pose.

• The SpatialSound component now has an output stream representing the audio buffers that were played through the speakers, with originating times corresponding to actual playback time.

• The Handle.cs renderer component uses the backend OpenXR time when posting its pose.A few new components have also been added:

• Added a generic ModelRenderer component for rendering any StereoKit Model.

• Added a new StereoKit rendering component, OpenXRHandRenderer, which renders OpenXR Hand objects from any input data stream.

Several other infrastructural updates include:

• The approach for setting and handling world transforms and spatial anchors has been refactored and improved. Local as well as persistent Azure Spatial anchors are supported via the LocalSpatialAnchorProvider (previously named SpatialAnchorHelper - thanks @austinbhale for past contributions to this) and AzureSpatialAnchorProvider classes respectively. Both providers can be used to set the world coordinate system via the MixedReality.SetWorldCoordinateSystem method.

• A static method for loading 3D models stored in a HoloLens folder as a StereoKit Model is now available.

• Serialization of AudioBuffer in HoloLensCaptureInterop now includes the entire WaveFormat header instead of assuming a fixed format.

• Operators in HoloLensCaptureExporter have been moved to HoloLensCaptureInterop.

Changes to PsiStudio and Visualization

Several updates have been made to the set of existing visualizers, and new infrastructure has been added to support visualizers that render a temporal segment of stream data. A detailed list of these changes is below:

• Updated 2D and 3D visualization infrastructure with support for stream interval visualizers. This is based on a newly introduced XYIntervalVisualizationObject and XYZIntervalVisualizationObject types (deriving from StreamIntervalVisualizationObject) that allows the user to specify which portion of the stream is used to construct the visualization data (currently All, Selection, View, and CursorEpsilonare implemented). As an example, theLabeledPointIntervalVisualizationObjectandPoint3DIntervalAsPointCloudVisualizationObject` allows for visualizing a scatter plot of points from a stream with 2D or 3D point messages. These visualizers provide a basis for trajectory visualizations.

• Added new 2D visualizers for Line2D and a corresponding stream interval visualizer.

• Added new 2D visualizer for Object2DDetectionResults

• Changed PseudoColorize method used by depth image visualizer to not render pixels outside the specified range.

• Expanded the set of diagnostic visualization options for streams to include message dropped and processed percentages, both total and withing the averaging window.

Several other improvements have been made to PsiStudio and the visualization infrastructure:

• Added Help window in PsiStudio (accessible from toolbar button) that displays a list of useful mouse and keyboard shortcuts.

• Separated out the most recently used lists for datasets and stores.

• Added a setting to prompt/auto-save changes made to the current layout.

• Added support for copying time interval annotation boundaries from other visualized annotation streams.

• Added support for deleting partitions from all sessions.

• Added support for navigating to a specified session and time by using a <session_name>@<time> syntax in the Go-To-Time dialog box.

• Added several key shortcuts: Space shortcut for play/pause, Shift+Left/Right Arrow shortcut for moving the selection to the next annotation.

• Added menu item for "Open Session Folder in Explorer" on sessions and "Open Partition Folder in Explorer" on partitions.

• Settings file is now in a machine-specific folder, under Documents\PsiStudio\Settings\{machine_name}.

• Simplified running of UI actions within a progress window with new static methods in ProgressWindow.

• Fixed an issue with the alignment of audio playback in PsiStudio: now PsiStudio correctly plays back the audio based on originating times.

• Fixed audio channel selection in PsiStudio and added a setting to control whether to playback only the selected channel.

Changes to Runtime

Several fixes and improvements have been made to the \psi runtime:

• Added method OpenStreamOrDefault<T> for Importer and SessionImporter. The method opens the specified stream if the stream exists or returns null otherwise.

• Added DeliveryPolicySpec enum type to enable specifying pipeline-level delivery policies during batch task processing.

• Added support for optional progress reporting in synchronous Pipeline.Run() methods.

• Improved the memory allocation strategy in BufferWriter.

• Fixed an issue with the importer initializing KnownSerializers based on the evolving set of default serializers. Moving forward, the runtime still allows for constructing a KnownSerializers instance that uses an existing set of known serializers as a starting point (by passing them as a second parameter in the constructor)

• Fixed an issue where custom serializers based on BackCompatSerializer sometimes failed to deserialize data from stores.

• Fixed an issue where IndexEntry metadata was incorrectly interpreted when reading from stores created with different .NET versions.

• Fixed an issue in InfiniteFileWriter where the file handle is not disposed when out of disk space (thanks @austinbhale).

• Fixed an issue enumerating stores with names containing whitespace (thanks @austinbhale).

• Fixed an issue in pipeline diagnostics capture for dropped messages that in some cases led to incorrect statistics being displayed.

• Added Interval<>.Intersection() method for computing the intersection of a set of intervals.

• ConfigurationHelper is now a static class with new static methods ReadFromFileOrDefault<T> and ReadFromStream<T> to read a configuration from a file or stream, respectively.

• Introduced mechanism for accessing platform resources (such as image encoders/decoders) via the PlatformResources.Get<> method in the runtime. Resources have to be registered viaPlatformResources.Register<>().

• An optional maxSpan parameter was added to the AdjacentValuesInterpolator allowing users to control the maximum timespan between adjacent values for which the interpolator will produce a result.

Changes to Microsoft.Psi.Data

Several improvements and extensions have been made to the batch task processing infrastructure:

• The method-based approach to defining batch tasks has been deprecated, and moving forward batch tasks will be defined via the classes derived from BatchProcessingTask<TBatchProcessingTaskConfiguration>. The batch task infrastructure was extended with handlers that are called prior to and after running on each session and dataset, and on encountering exceptions or cancellation, to enable initialization and clean-up code. Batch tasks can now be run programmatically on Dataset and Session objects.

• Added StreamCollection class to support a pattern for hierarchical compositing of batch processing tasks.

• Removed the need for declaring the [ExpandableObject] attribute on batch task configuration classes

• Batch tasks can now be more easily implemented as part of .NET standard libraries and they can leverage platform specific resources via the PlatformResources class methods.

• An overloaded version of the ToTimeIntervalAnnotation() stream operator has been added to support conversions from a 3oolean stream into a time interval annotation stream.

• Several methods on Dataset and Session have been renamed and/or made async.

Changes to Imaging

• Added a method for computing an interpolated pixel in a depth image. This is useful when intersecting a 3D ray with a depth map that might have holes (pixel values of 0).

Changes to Spatial.Euclidean

• Extended Mesh3D to include an (optional) overall pose, and added algorithms for determining intersection with Ray3D and for determining the closest point on the mesh to a specified Point3D

• Static methods for computing triangle planes and testing whether a triangle encloses a point.

• Added a method on Box3D for computing a closest point.

• Added operator for interpolating between two Ray3Ds and determining whether a triangle contains a point.

Changes to Media

• Added a new Microsoft.Psi.Media library defining the platform-independent interface Impeg4Writer.

Changes to Audio

• Added a constructor that creates an empty AudioBuffer with a specified TimeSpan and format.

• Added a concatenation (+) operator for AudioBuffer.

• Added a SelectChannel operator for AudioBuffer.

Changes to CognitiveServices.Speech

• Added SpeechSynthesizer component providing access to Azure text-to-speech services.

• Removed obsolete BingSpeechRecognizer component which has been replaced by AzureSpeechRecognizer.

• Added an operator for converting a stream of SpeechSynthesisProgress messages to temporal annotations.

Changes to CognitiveServices.Face

• Due to deprecation of the method-based approach to defining batch tasks, the method-based PersonGroupTasks have been removed.

Changes to Interop

• Added a new IInteropSerializable interface, which, together with a new set of operators for reading and writing IInteropSerializable types from/to binary writers enables a simpler pattern for interop serialization.

Changes to ONNX

• Added \psi components for running the Detic and SEEM models. The components connect over a zero-mq connection to a python server that runs these models. The server functionality is provided at under Models in the Integrations folder. For installing / configuring these python servers, see the documentation available in the corresponding readme files for Detic and SEEM

• The OnnxModel base class now takes generic input and output type parameters to support models with input and output data types other than float.

Miscellaneous Changes

• Updated code analysis in projects to use .NET analyzers (version 8.0.0).

• Upgraded test projects to target .NET 8 and updated them to use the latest MS Test NuGet packages.

• Upgraded C++ projects to target the latest Windows SDK version (currently 10.0.22621.0)

• Upgraded UWP projects to target Windows SDK version 10.0.22621.0.

• Removed obsolete MicrosoftSpeech project and associated components.

2022/12/06: Beta-release, version 0.18.72.1

Overview

In this release we have continued to evolve the support for writing mixed-reality applications using \psi and included several important updates to Platform for Situated Intelligence Studio and the \psi runtime, as well as other updates and bug fixes across various \psi components.

Changes to Mixed Reality

We have continued to evolve support for writing mixed-reality applications with \psi. This release includes new components for more direct and fine-grained access to hand tracking information via OpenXR APIs and gaze tracking information via WinRT. The update also includes a refactoring of various classes and namespaces under Microsoft.Psi.MixedReality for more clarity. A more detailed list of changes is below:

• Added a new WinRT-based source component for capturing eye and head gaze: GazeSensor in the Microsoft.Psi.MixedReality.WinRT namespace.
• Added a new OpenXR-based HandsSensor component in the Microsoft.Psi.MixedReality.OpenXR namespace.
• The HoloLensCaptureApp now uses the WinRT-based GazeSensor and the OpenXR-based HandsSensor components.
• BREAKING CHANGE: Refactoring of WinRT/OpenXR/StereoKit/MediaCapture sensors and data types:
  • Reorganized Microsoft.Psi.MixedReality sensor components and datatypes into different namespaces reflecting the underlying technology used:
    • Microsoft.Psi.MixedReality.StereoKit
    • Microsoft.Psi.MixedReality.OpenXR
    • Microsoft.Psi.MixedReality.WinRT
    • Microsoft.Psi.MixedReality.ResearchMode
    • Microsoft.Psi.MixedReality.MediaCapture
  • The following classes/types from the Microsoft.Psi.MixedReality namespace have been moved to a different namespace and/or renamed:
    • Hand, EyesSensor, Handle, SpatialSound, Microphone have been moved to the Microsoft.Psi.MixedReality.StereoKit namespace.
    • All StereoKit renderer components have been moved to the Microsoft.Psi.MixedReality.StereoKit namespace and renamed from <Name>StereoKitRenderer to <Name>Renderer, e.g., Box3DStereoKitRenderer -> Box3DRenderer.
    • DepthCamera, Accelerometer, Gyroscope, Magnetometer, VisibleLightCamera have been moved to Microsoft.Psi.MixedReality.Research namespace.
    • MediaCaptureMicrophone has been renamed to Microphone and moved under the Microsoft.Psi.MixedReality.MediaCapture namespace.
    • PhotoVideoCamera has been moved to the Microsoft.Psi.MixedReality.MediaCapture namespace.
• BREAKING CHANGE: MixedReality.InitializeAsync() is now MixedReality.Initialize() (no longer async). In general, make sure that all UWP apps deployed to the HoloLens use a void Main() method, and not async Task Main().
• Updates to HandVisualizationObject to no longer require adapters.
• Created a batch processing task for projecting 3D hand poses into the 2D image space of available camera views, for visualization in PsiStudio: ProjectHandsToCamerasTask.
• Further expanded the set of format serializers in HololensCaptureInterop

Changes to PsiStudio and Visualization

Some of the primary changes to PsiStudio involve additional support for expanding streams (e.g., key expansion for Dictionary streams and script-based expansion for computing derived streams), the ability to save batch task configurations, and a new settings dialog. A number of other improvements and fixes have also been made. A more detailed list of changes is below:

• Added key expansion for Dictionary streams.
• Added script-derived streams.
• Batch processing task configuration parameters may now be saved and loaded.
• Added setting that enables skipping the security warnings upon loading 3rd party code.
• Added support for editing the settings via a dialog from within PsiStudio.
• Added setting that enables skipping the security warnings upon loading 3rd party code.
• Added support for remembering a list of most recently used dataset filenames.
• Added visualizer for dictionary of point clouds.
• Removed AnimatedModel3DVisualizationObject and replaced with a new PosedModelFromFileVisualizationObject. This new visualizer works over streams of CoordinateSystem poses, and will render a 3D model at the given pose according to a property for configuring which model file to load from disk (.obj, .stl, .3ds, .lwo, .objz, .off, and .ply files are supported).
• Added keyboard shortcuts for Zoom to Selection (Ctrl+S) and Zoom to Session (Ctrl+A)
• Added support for removing a specified partition from all sessions.
• Added support for copying the cursor time as ticks to clipboard.
• Added more copy-to-clipboard commands for partitions, sessions and streams in the dataset tree view.
• Changed the settings file to save AdditionalAssemblies as a collection instead of a string.
• Added support for type mappings in PsiStudio settings, which allows for loading streams of an unknown type under a known type. This feature is particularly useful when datatype names have changed, as PsiStudio needs to know this information when loading the store.
• Fixed a bug in expanding members for types that contain members that are nullable.
• Fixed an issue with refresh in pipeline diagnostics visualization.
• Additional fixes and cleanup in the computation of various source stream and subsumed stream properties on the stream tree nodes.
• Refactored the StreamTreeNode infrastructure to support more general derived streams. Stream bindings (which are persisted in layouts) can now reflect a binding to a derived stream. The StreamBinding stores both a derived stream adapter and a visualizer stream adapter and these two adapters are chained together (via the ChainedStreamAdapter class) to effect the binding to the stream.
• Refactored the construction of context menus for visualizers in PsiStudio by moving it from the views to the visualization objects. This puts it more in line with an MVVM approach and unifies the previous separate approaches we had for 3D and 2D visualizers.
• Moved (and refactored) some of the helper operators for accessing collections of time interval annotation from the annotation visualizer into Microsoft.Psi.Data\Annotations and Helpers.

Changes to Imaging

• The ImagePool class now allocates different subpools based on binning the allocation size, leading to improved performance.

Changes to Calibration

• BREAKING CHANGE: CalibrationExtensions.IntersectLineWithDepthMesh() is now CalibrationExtensions.ComputeRayIntersection(). The functionality is effectively the same, (intersecting a ray, rather than a line, with a depth mesh), but is more robust for different kinds of depth images.
• CameraIntrinsics.GetPixelPosition() now returns null if the 3d point passed in is behind the camera.

Changes to Spatial.Euclidean

• Added Volume property on Bounds3D and Box3D.
• Added support for exporting point clouds in PLY format from the DepthImageCameraViewAsPointCloudVisualizationObject.

Changes to Runtime

• Addressed concurrency issue in PsiStoreWriter that occured when multiple Write() commands would be generated while the pipeline is running, from Parallel() operators in the pipeline.
• Addressed concurrency issue occurring when estimating pipeline progress when new receivers are dynamically added to a component while the pipeline is running.
• Enabled the use of AdjacentValueInterpolator{T} inside Fuse/Join operators.
• The ThrottleWhenFull boolean property (and corresponding construction parameter) of DeliveryPolicy has been changed to an int? property named ThrottleQueueSize. This allows more flexibility to specify the queue size threshold at which throttling occurs.
• The behavior of delivery policies with throttling has changed. Previously, these policies would cause incoming messages to be dropped at the receiver if the upstream source attempts to post messages while throttled. These messages are no longer dropped but instead will be queued at the receiver.
• BREAKING CHANGE: The autoClone flag has been removed from the Pipeline.CreateReceiver methods. Receivers will always receive a cloned message which guarantees that any changes made to the message contents in the receiver are isolated from the sender's copy of the message. Cloned messages will be automatically recycled when the receiver method returns. If the message contents need to be retained beyond the lifetime of the receiver method, an explicit copy should be made using the DeepClone extension method.
• Added support for caching in TypeResolutionHelper class for performance improvements.
• Added support for type name synonyms with KnownSerializers.RegisterDynamicTypeSchemaNameSynonym(...). These are used when parsing type schema information to PsiStore.OpenDynamicStream(...).
• Added new BackCompatClassSerializer and BackCompatStructSerializer classes to simplify writing a custom serializer that needs to maintain backward compatibility with previous versions of a data type serialized with the auto-generated default serializer. For an example, see the AngularVelocity3D class in Microsoft.Psi.Spatial.Euclidean.

Changes to Media

• Fixed optional FFmpeg support on Linux and removed obsolete Microsoft.Psi.Media.Native.x64 project.
• Modified the Windows Mpeg4Writer component to buffer and write audio and video in a coordinated way.

Changes to ONNX

• Fixed an issue re: disposing the underlying resources in various model runner components.
• BREAKING CHANGE: MaskRCNNModelRunner now internally resizes input images to match the configured size and resizes output MaskRCNNDetection results to match the original input image size.
• MaskRCNNModelConfiguration now includes a ConfidenceThreshold used in filtering results.
• Created templated version of the OnnxModelRunner class, enabling users to provide input and output adapters via the constructor.

Annotations

• Added support for setting annotation boundaries to selection boundaries via context-menu.
• Improved performance for multi-track annotation visualizer.
• Multi-track annotation visualizer support now showing different subsets of tracks via the Show Tracks property. Users can select whether to show all tracks, only tracks with events in view, or manually specify (via the Show Tracks Selection property) which tracks are to be shown.

2022/04/20: Beta-release, version 0.17.52.1

Overview:

This release includes important updates to mixed reality support in \psi, including a set of tools for streaming data from a HoloLens 2 to a separate PC for data collection and export. The release also includes several updates to visualization and PsiStudio, the addition of a wrapper for running MaskRCNN models, updates to Azure Kinect Components, as well as some runtime updates and various other bug fixes.

Changes to Mixed Reality

• Introduced ResearchCameraConfiguration base type; now VisibleLightCameraConfiguration and DepthCameraConfiguration derive from this base type; in the process renamed Mode configuration property to DepthSensorType and VisibleLightSensorType respectively.
• Added GetCurrentTimeFromOpenXr() extension method on pipeline for retrieving the current time from OpenXR.
• Updated originating time computation at various mixed reality sensor components to more accurately reflect capture time.
• Added capability (and corresponding configuration option) for outputting camera view streams (using the new *ImageCameraView types) from the photovideo, depth, and visible light cameras.
• Changed PhotoVideoCamera to emit encoded NV12 images or image camera views (for increased performance).
• DepthCamera and VisibleLightCamera now allow for calling a Calibrate() method prior to starting the pipeline. The calibration process is compute intensive and sometimes it is desirable to compute the calibration information prior to starting the pipeline to avoid slowdowns on pipeline startup.
• Refactored and simplified the set of operators available for converting poses to/from StereoKit.
• Added PsiInput static class that provides head, hand, and eye inputs in \psi world coordinates.
• Established a pattern for creating new StereoKit renderers (deriving from StereoKitRenderer), and added a few more.
• Changed EyesSensor output representation from a CoordinateSystem to a Ray3D.
• Added an Out emitter to the HandsSensor that carries both hands in a tuple (Left, Right).

Added HoloLens Data Capture Applications

• Added a set of tools that enable capturing sensor data from the HoloLens 2 device into \psi stores, and for exporting these stores to other formats.
  • HoloLensCaptureApp: is an application runs on the HoloLens, capturing and remoting data streams.
  • HoloLensCaptureServer: is an application that runs on a Windows machine, receiving remoted data streams from the HoloLensCaptureApp and writing them to a \psi store.
  • HoloLensCaptureExporter: is a tool that can convert the data captured in the \psi stores to other formats.

Changes to Azure Kinect Components

• Updated Nuget references Microsoft.Azure.Kinect.Sensor to 1.4.1 and Microsoft.Azure.Kinect.BodyTracking to 1.1.1, which bring several improvements in the underlying SDKs, including no longer requiring CUDA dependencies and supporting more GPU setups.
• Added an additional configuration option to the AzureKinectBodyTracker component for optionally using the "lite" version of the underlying pose estimation model. According to the Azure Kinect documentation, "this model trades ~2x performance increase for ~5% accuracy decrease."

Changes to Imaging and Calibration

• Added Description member on IImageToStreamEncoder and IDepthImageToStreamEncoder; this in turns allows more informative displays in diagnostics visualization when using encoders.
• Added IImage and IDepthImage interfaces.
• Renamed EncodedImage.SetBuffer(...), which copys from a byte[], to EncodedImage.CopyFrom(...).
• Added EncodedImage.CopyFrom(...) overload allowing data to be copied from a memory pointer.
• Separated JPEG/PNG decoders into ImageFromBitmapStreamDecoder in each of Microsoft.Psi.Imaging.Windows and Microsoft.Psi.Imaging.Linux
• Separated GZip decoder into ImageFromGZipStreamDecoder in cross-platform Microsoft.Psi.Imaging
• Added NV12 decoder ImageFromNV12StreamDecoder in cross-platform Microsoft.Psi.Imaging
• ImageFromStreamDecoder now supports NV12 encoding.
• Renamed type DepthPixelSemantics to DepthValueSemantics; the depth pixel semantics information has been migrated in a breaking change from the CameraIntrinsics type to the DepthImage type, which now stores two new members: DepthValueSemantics and DepthValueToMetersScaleFactor.
• Fixed a bug in CalibrationExtensions.CalibrateCameraIntrinsicsAndExtrinsics() where the bottom-right element of the resulting intrinsics matrix was not properly being set to a value of 1.

Changes to Visualization and PsiStudio

The release includes several usability updates for PsiStudio, as well as a few adapters, visualizers and infrastructural updates:

• Added support for moving the selection to the right or the left (via Alt+Left/Alt+Right, or toolbar buttons).
• Added support for switching the currently visualized session by double-clicking on it.
• Added "Go To Time" button on toolbar, also accessible via Ctrl+G shortcut and from the context menu on any panel.
• Double-click on a regular stream in the datasets menu expands the stream members.
• Added the ability to set the default cursor epsilon on instant visualization panels.
• Added property on labeled rectangle visualizers to show/hide the label.
• Sessions now support adding multiple partitions from a specified folder.
• Dataset now supports adding session (with multiple partitions) from a specified folder.
• Dataset now supports adding multiple sessions (with multiple partitions) from a specified folder.
• Expanded support to copy to clipboard to include session name and/or cursor time.
• Added support for switching which session is being visualized from the context menu of a partition or stream in a non-visualized partition.
• Added adapters to visualize Mesh3D objects as point clouds (showing only the vertices).
• Added support for visualizing lists of Ray3D objects.
• Added support for naming stream adapters via the StreamAdapter attribute and constructing better stream context menu item names when multiple visualizers with different adapters with the same type signature are available.
• Addressed an issue that was creating a 2x slowdown in rendering visualizations that use a canvas view.
• Fixed a bug when copying cursor time to clipboard.
• A few small enhancements to text visualizers.

Changes to Spatial.Euclidean

• Converted Box3D type from struct to class and updated corresponding visualizers.
• Added Scale() method on Box3D and Bounds3D.
• Added a public constructor for PointCloud3D from matrix.
• Added epsilon tolerance parameters to AngularVelocity3D, CoordinateSystemVelocity3D constructors and Bounds3D.ContainsPoint() method.
• Added Mesh3D.ToPointCloud3D() method to convert mesh vertices to a point cloud.
• Added ImageCameraView types containing Shared<Image> along with camera intrinsics (ICameraIntrinsics) and pose (CoordinateSystem).
  • Added corresponding EncodedImageCameraView, DepthImageCameraView, and EncodedDepthImageCameraView and PsiStudio visualizers.
  • Added adapters from (<image>, ICameraIntrinsics, CoordinateSystem) to new *ImageCameraView types.

Changes to Runtime

• Added support for naming components and stream operators, as well as default names for most components and stream operators. This enables a more readable display of component names when visualizing pipeline diagnostics.
• Stores containing streams of Dictionary<TKey, TValue> are now cross-platform compatible.
• Fixed an issue in the store writer OpenStream() method regarding setting the opened time (which ensures correct behavior in batch processing / replay).
• Fixed an issue where pipeline shutdown was sometimes being delayed due to a pending message with an originating time in the future.

Changes to Annotations

• Renamed AnnotationSchema.Load() to AnnotationSchema.LoadFrom() and added TryLoadFrom() method.
• Refined/updated overloads for ToTimeIntervalAnnotations() stream operator (used for constructing time interval annotations from various source streams).

Changes to CognitiveServices.Vision

• Added stream operators for exposing various image analysis results.

Changes to Interop

• TcpSource now supports custom, user-specified deallocators to manage the lifetimes of the objects it creates.

Changes to ONNX

• Added support for using shape dictionaries when running onnx models.
• Added Mask R-CNN model runner (object detection, classification, bounding boxes and masks) and a PsiStudio visualizer.

Diagnostics

• Updated pipeline visualization to show the type of interpolator used by Join and Fuse as a tooltip on hover over the respective component.
• Added statistic for percentages of messages dropped to pipeline diagnostics visualization.

2022/02/08: Beta-release, version 0.16.92.1

Overview:

This release includes additional \psi libraries and components for writing \psi mixed-reality applications for HoloLens 2, new types and operators for representing and dealing with various spatial geometric objects, numerous updates and fixes to Platform for Situated Intelligence Studio (from updates to user interface to new and updated visualizers and support for multi-track temporal annotations), a few updates to some of the stream operators in the \psi runtime, as well as a number of other component fixes and updates. The changes are described in more detail below.

Mixed Reality

Platform for Situated Intelligence now supports the development of mixed reality applications. This release includes several components for accessing and processing the rich multimodal sensor streams (such as head tracking, hand tracking, gaze tracking, RGB and depth cameras, IMU, etc.) as well as example \psi components for rendering output on the device. For now, only the HoloLens 2 device has been tested and is supported, but in theory, it should be possible (with perhaps a little extra effort) to target other mixed reality platforms such as Oculus Quest, Windows Mixed Reality, Oculus Desktop, SteamVR, etc. Integration between \psi and the HoloLens 2 is enabled by two primary underlying technologies: Research Mode and StereoKit.

Spatial Euclidean Representations

This release includes a new project and namespace: Microsoft.Psi.Spatial.Euclidean, which extends the set of types available in MathNet.Spatial.Euclidean. New classes and operators allow for representing, manipulating, and visualizing streams of 3D rectangles, boxes, point clouds, meshes, velocities, and more.

Changes to Visualization and Platform for Situated Intelligence Studio

The updates to visualization and PsiStudio include support for multi-track annotations, improvements to the user interface, new visualizers and adapters, updates to existing visualizers, some updates and code refactoring in the visualization infrastructure, as well as bug fixes and performance improvements.

• Support for multi-track annotations:

  • Added support for manually constructing overlapping time-segment annotations in a single annotation stream. Context menus have changed on annotation stream visualizations to support creation of annotations on multiple tracks. This is a BREAKING CHANGE with respect to the definition of annotation schemas and the data types involved in persisting annotations.

• Updates and improvements in the user interface:

  • Added support for Dataset auto-save in PsiStudio -- see this pull request.
  • Added support in PsiStudio to automatically open the last loaded dataset on startup. This is configured with a settings option. Users can now toggle both this option and the option to autosave datasets from the PsiStudio File menu.
  • Added shortcut buttons on panels for showing/hiding them.
  • 2D visualizers can now be zoomed into using the mouse wheel. Users can pan the visualizer by dragging the mouse with the right mouse button down.
  • Users can now set a Threshold Value on Timeline Panels such that any values either above or below this threshold is highlighted to make it easier to find areas in the data that may require further scrutiny.
  • Added support for displaying throughput statistics (per hour, min, sec.) in dataset / session / partition / streams tree.
  • Added new properties on 3D visualization panels. These include whether to zoom/rotate around the mouse point (which is now the default mode), the zoom and rotation sensitivity, as well as the options to display various info for the viewport, such as FPS, number of triangles, coordinate system, view cube, etc.
  • Added support for deleting layouts from the layout toolbar.
  • When attempting to save a layout under a name that already exists, an option to overwrite or cancel is now presented (rather than just an error message).
  • Updated the mechanism by which PsiStudio detects live stores.
  • Added ability to set the cursor epsilon on instant visualizers directly from the visualizer context menu.
  • Added visualizer context menu option to copy the cursor time to clipboard.

• New visualizers and adapters:

  • Added a new 3D visualizer for displaying text in a 3D location.
  • Added an instant visualizer for IFormattable objects that allows to specify the format string as a property and renders the object as text.
  • Added MathNetPoint2D and NullablePoint2D to scatter plot stream adapter for visualization.
  • Added support for XY panels and Canvas panels. The former can include visualization objects that display (x,y) data, such as the scatter plot visualizer, whereas the second can include visualization objects that display any data on a flat canvas without axes, such as the text visualizer.
  • Added better support for visualizing booleans and series of booleans, i.e., Dictionary<Key, bool>, via plot visualizer.

• Updates to existing visualizers:

  • Added support for font size and border thickness in billboard visualizer.
  • Added support for XY charts in Microsoft.Psi.LiveCharts.Visualization.
  • Added support for visualizing 3D graphs based on a specified node visual 3D and edge visual 3D via ModelVisual3DGraphVisualizationObject.
  • Added support for displaying lossy delivery policies as dotted in the pipeline diagnostics visualizer (enabled by default).
  • Changed delivery policy highlight in pipeline visualizer to use transparency rather than color.
  • Updated edge labels in pipeline visualizer to use the heatmap color when using heatmap statistics. Updated default heatmap statistics to "none".
  • Added scroll bar capabilities for the text and IFormattable visualizers.
  • Added support for visualizing images in RGB_24bpp format -- see this pull request.
  • Added support for displaying (as a readonly property) the value of the pixel under mouse in the depth image visualizer.
  • Added support for controlling the range mode via context menu on the depth image visualizer.

• Updates to visualization infrastructure:

  • Refactored 2D visualization panels into two types: XYVisualizationPanel (which supports visualizers that present XY data) and CanvasVisualizationPanel (which supports 2D visualizers that do not present XY data).
  • Systematized and refactored base classes for collections of 3D visualization objects.
  • Systematized and expanded support for batch processing tasks to allow for task parameterization and parameter validation.
  • Added support for creating visualizers for interface types.
  • Enabled visualizers derived from StreamIntervalVisualizationObject to specify which data interval should be read in (by overriding a virtual GetTimeInterval() method.)

• Bug fixes and performance improvements:

  • Fixed a bug that crashed CartesianChartVisualizationObject.
  • Improved performance on point cloud visualizer by using the camera space mapping matrix.
  • Small fix in diagnostics viz to display Fuse components with a "+", like Join.
  • Changed and fixed some problems in approach for reordering panels in the visualization container. To reorder panels, left-click the handle that appears at the left edge of the panel and drag up or down. Left-click and drag elsewhere in timeline panels now only moves the timeline.

Changes to Runtime

• Added Last() and Last(count) stream operators, as well as support on Processor component for an optional action to execute when the input stream closes.
• Refactored the Interpolator<TIn,TOut> class hierarchy to allow for implicit operator conversions from RelativeTimeInterval and TimeSpan to Interpolator<T>. This allows for these conversions to be used in Fuse() operator calls, not just Join().
• Systematized the vector join operations (in the process removed the sparse vector join operator).
• Added RemoteClockExporter/RemoteClockImporter components to allow for synchronizing a local pipeline clock with a pipeline running on a remote machine (by exchanging clock information and applying a local offset).
• Added IsClearRequired to ISerializer<T> interface.
• Updates to PipelineDiagnostics data structure; these updates constitute a BREAKING CHANGE for serialization of pipeline diagnostics.

Changes to Imaging

• BREAKING CHANGE: Renamed ToPixelFormat() stream operator to Convert().
• BREAKING CHANGE: Removed ToGray() stream operator - use the Convert() operator instead.
• Added FromFile() and Save() methods to Image class.
• Added Convolve() operator (for now only for grayscale images)
• Added support for clipping the crop region to image boundaries during Crop() operations.
• Added SetBuffer() methods to the EncodedImage and EncodedDepthImage classes.
• Added additional image drawing methods and operators, including DrawText(), FillRectangle(), and FillCircle().
• Additional image drawing operators -- see this pull request.
• Depth images now support encoding/decoding via the TIFF compression format -- see this pull request.
• Added ImageToGZipStreamEncoder component and EncodeGZip() stream operator to support GZip compression for Shared<Image>

Changes to Audio

• Added a WaveStreamSampleSource component that can produce on-demand an audio sample specified by a System.IO.Stream.

Changes to Calibration

• Implemented IEquatable interface on CameraIntrinsics and added GetCameraSpaceMapping method that returns a mapping matrix that can be used to quickly transform depth value pixels into 3D space.
• Renamed some of the members of ICameraCalibration: ToPixelSpace() -> GetPixelPosition(), ToCameraSpace() -> GetCameraSpacePosition(), GetCameraSpaceMapping() -> GetPixelToCameraSpaceMapping(), DistortPoint() -> TryDistortPoint(), UndistortPoint() -> TryUndistortPoint()
• CameraIntrinsics.GetPixelPosition() now returns a Point2D? which is null if the pixel is outside the field of view of the camera (unless optional argument nullIfOutsideFieldOfView=false).
• DepthDeviceCalibrationInfo.ToColorSpace() renamed to GetPixelPosition() and now returns a nullable Point2D? which is null if the pixel is outside the field of view of the camera (unless optional argument nullIfOutsideFieldOfView=false).
• Added CameraIntrinsics.TryGetPixelPosition() and DepthDeviceCalibrationInfo.TryGetPixelPosition().
• Added a DepthPixelSemantics field which indicates whether the depth values represent distance from camera to point, or distance from camera to the camera-axis-perpendicular plane containing the point.
• Added methods in CalibrationExtensions.cs for calibrating a camera from sets of 2D image points and corresponding 3D points (intrinsics and/or extrinsics). Previous ComputeCameraIntrinscs() methods are now called CreateCameraIntrinsics().

Changes to Data Processing and Datasets

• Added support for defining configurable batch processing tasks via classes that derive from BatchProcessingTask<TBatchProcessingTaskConfiguration>.
• Renamed SessionImporter.HasStream() to SessionImporter.Contains() for consistency with other APIs.
• Removed UseRelativePaths property from Dataset class -- see this pull request.

Changes to Interop

• Added RemoteClockImporter and RemoteClockExporter components to synchronize pipeline clocks between \psi pipelines running on different machines.
• Added TcpWriter and TcpSource components to Microsoft.Psi.Interop allowing for primitive transport over TCP using existing format providers (e.g. JSON, MessagePack) or using custom implementations of IFormatSerialize/IFormatDeserialize
• Added Rendezvous system by which \psi apps may publish and discover available NetMQWriter/TcpWriter/RemoteExporter/RemoteClockExporter endpoints and available streams.

Changes to ONNX

• In ImageNetModelRunnerOutputParser class, removed the GetPredictions() method and added instead a GetLabeledPredictions() method that returns all labeled predictions, and a GetTopNLabeledPredictions() method that returns the top-n labeled predictions.

Simple Filtering Operators

A new project and namespace: Microsoft.Psi.Filters, contains some simple initial filters that operate over streams of doubles. It currently contains the following filters: Low-pass (MathNet), High-pass (MathNet), Band-pass (MathNet), Band-stop (MathNet), Denoise (MathNet), OneEuro.

Deprecation of Microsoft.Psi.MicrosoftSpeech

• The Microsoft.Psi.MicrosoftSpeech.Windows library has been deprecated as of this release. Existing users should migrate to the SystemSpeechRecognizer component available in the Microsoft.Psi.Speech.Windows project.

2021/04/23: Beta-release, version 0.15.49.1

Overview:

This release includes various updates to the runtime, tools, and components. Numerous improvements have been made in Platform for Situated Intelligence Studio, including new visualizers and improvements to existing visualizers, new features and improvements to existing features that accelerate working with the tool, improvements in the underlying data access and visualization infrastructure, and other bug fixes. In addition to the changes to PsiStudio, several updates and bug fixes have also been made to the runtime, to diagnostic metrics capture and visualization, and to imaging and calibration components and APIs.

Changes to Visualization and Platform for Situated Intelligence Studio

This release includes numerous updates to Platform for Situated Intelligence Studio, falling into 4 four broad categories: (1) new and updated visualizers, (2) new features for improved usability, (3) changes to infrastructure, and (4) other bug fixes.

(1) New visualizers and updates to existing visualizers:

• Added a visualizer for numeric series, i.e., streams containing Dictionary<TKey, TNumeric>, where TNumeric is any numeric type (e.g., int, long, double, bool, decimal, and corresponding nullables.) The dictionary keys are automatically converted to strings and act as a series name. A plot is generated for each series.
• Added a visualizer for a list of List<Point3D> as a point cloud.
• Added a visualizer for depth images which shows them as pseudo-colorized images.
• Added new visualizers and adapters for spatial camera views.
• Added a visualizer that displays strings as text in a 2D canvas.
• Added a visualizer that displays a string with position in a 3D billboard.
• Added support for displaying image resolution and mouse position in the properties of image and depth image visualizers.
• Fixed issues with tooltip rendering on nodes in diagnostics visualization.
• Added a visualizer for MathNet Ray3D objects.

(2) New features and improvements to existing features:

• X-Axis and Y-Axis scale properties have been moved from each individual timeline and 2D visualization object to the corresponding visualization panel. As a result, all visualization Objects in timeline or XY panels now automatically use the same horizontal and vertical scale that has been set via properties of the panel.
• Added support for displaying additional dataset, session, partition, and stream information in the Dataset panel, to the right of the tree nodes.
• Added hover-buttons for show/hide and remove visualization objects in the layouts tree, and for closing visualization panels.
• Added "Expand All Nodes" and "Collapse All Nodes" options on stream tree node context menu.
• Added a toolbar button in the Layout tab for clearing the layout.
• Updated playback speed textbox on the toolbar to support speed factors larger than 11. The corresponding increase/decrease buttons double or half the speed factor.
• Redesigned the context menu system in the Visualizations tab. Visualization panels and visualizers now all contribute to the context menu in a structured way.
• Added support for visualizing diagnostics information as streams, by right-clicking on an edge in the diagnostics visualizer and creating specific derived stream nodes.
• Added support for allowing mouse to move without the cursor following the mouse if Shift key is held down.
• Updated default behavior on opening a dataset to expand sessions only if there is a single session.
• Added the ability to crop a selection from a partition into a new PsiStore.

(3) Improvements and updates in the data layer and visualization infrastructure:

• Refactored large parts of the class hierarchy for VisualizationObject and VisualizationObjectView, to clarify distinctions across the dimensions of data access type (stream value or stream interval), data type, and view type.
• Refactored the data access layer to better manage object lifetimes and handle disposable (and shared) objects without generating leaks.
• Refactored stream adapters to rely on virtual methods. Apart from a GetAdaptedValue virtual method that was introduced to handle the adaptation, a Dispose(TDestination destination) method has been introduced that allows the adapters to properly dispose of items they allocate in the process of adaptation.
• Refactored StreamTreeNode for extensibility by creating a class hierarchy of different kinds of stream tree nodes, and moved functionality from PluginMap into StreamTreeNode and its related classes.
• Updated TimelineCanvasVisualizationObjectView to be able to work with any stream visualization object, not only timeline visualization objects. This allows creation of an instant value visualization objects that use a timeline view to display information.
• Removed Pool infrastructure from PsiStudio since it had become obsolete and was not being used.

(4) Other bug fixes and improvements:

• Fixed UI problem where it was necessary to click on the text of the stream tree node (rather than entire row) to activate context menu.
• Systematized displayed property names for datasets, sessions, partitions, streams.
• Fixed an issue where instant data was not immediately populated when opening a new layout.
• Fixed an issue that did not allow partition names to be changed via property editor.
• Fixed an issue whereby the partitions did not update correctly in the Datasets view after a batch processing task was run.
• Fixed an issue in DisplayImage whereby images were not immediately shown.
• Fixed an issue whereby synchronization between a visualization object containing a derived stream member and the stream tree was failing.
• Eliminated this old InitNew() pattern we had in the visualization objects as it was no longer necessary.
• Updated infrastructure for computing the order in which visualizers are added to context menus.
• Fixed an issue in PsiStudio on refreshing the stores after running a batch processing task that overwrites and existing store.
• Fixed an issue in visualization which would lead to problems if the partition name was different from the underlying store name.
• Fixed an issue in DisplayImage where initialization would not happen on the UI thread.
• Fixed an issue where users were unable to clear the selection markers if only one of the start or end markers had been set.
• Fixed an issue where PsiStudio could sometimes crash when loading a Dataset if one of the Partitions in the Dataset was invalid or missing.
• Fixed dark text rendering of expandable property names.

Changes to Runtime

• Updated infrastructure for importers to allow better lifetime management of objects. The Importer.OpenStream<>, OpenStreamIndex<>, and SessionImporter.OpenStream<> methods now take allocator and deallocator optional parameters, which in turn enables using shared pools when reading streams of a Shared<T> type. The importer component now deallocates of shared objects after post via the provided deallocator (by default it disposes of Disposable objects).
• PsiStore.Open(...) now accepts an optional usePerStreamReaders flag causing separate importers for each opened stream (default=true). This allows messages to be emitted at a pipeline time approximating the original creation time, regardless of physical interleaved ordering. Setting this parameter to false reverts to the older behavior prior to this release.
• Fixes for computations of pipeline progress estimates.
• Added an overload to UnmanagedBuffer.CopyFrom that takes an offset and length.
• Added PsiStore.SummarizeDistinctKeysInSupplementalMetadata(...) to add metadata enumerating known keys.
• Forced PsiStore.EditStream operations to use Unlimited policy to make sure no messages are dropped.
• Changed Exporter.Write() and .WriteEnvelopes() method to take in IProducer, not just Emitter.
• Added Size and StreamCount members on IStreamReader interface, as well as Dataset/Session/Partition classes. (Breaking change).
• Added deliveryPolicy and enableDiagnostics optional parameters on the CreateDerivedPartition APIs.
• The stream metadata classes (starting with the IStreamMetadata interface) have been updated:
  • MessageCount changed from int to long.
  • AverageLatencyMs renamed to AverageMessageLatencyMs and changed from int to double.
  • AverageMessageSize changed from int to double.
  • PartitionName and PartitionPath renamed to StoreName and StorePath.
  • FirstMessageTime renamed to FirstMessageCreationTime.
  • LastMessageTime renamed to LastMessageCreationTime.
• PsiStreamMetadata changes (in addition to IStreamMetadata):
  • Removed AverageFrequency.
  • AverageLatencyMs renamed to AverageMessageLatencyMs and is now computed from MessageCount and LatencyCumulativeSum, is now a double and is now in milliseconds rather than microseconds.
  • AverageMessageSize is now computed from MessageCount and MessageSizeCumulativeSum and is now a double.
  • MessageSizeCumulativeSum added (long).
  • LatencyCumulativeSum added (long).
• Message.Time renamed Message.CreationTime.
• Added IStreamMetadata.StreamTimeInterval (opened to closed time).
• Added StreamTimeInterval property to Importer, SessionImporter and MetadataCache.
• Renamed IStreamMetadata.AverageLatencyMs to AverageMessageLatencyMs.
• Fixed an issue in which streams read from a \psi stores could produce messages with a CreationTime after the current pipeline time.
• Fixed a store seek bug on index boundaries.

Changes to Diagnostics

• Expanded and systematized computation of diagnostic statistics to include counts of emitted messages, as well as emitting and receiving latency. Updated the computation of some diagnostic metrics. Statistics now include both averages and last values. Renamed some of the other members.
• Components can customize the node name in diagnostic visualization by overriding the ToString() method on the component class.
• Fixed a bug on displaying stream information in the pipeline graph when two components are connected by multiple streams.
• Changed the default for heatmap visualization to be the receiving latency.

Changes to Imaging

• Added an overload to ImageBase.CopyFrom that takes an offset and length.
• Added image operators for color averaging.
• Added support for specifying invalid values in PseudoColorize image operation and stream operator. The pseudocolorized images are now in BGRA_32bpp format.
• Added new RGB_24bpp PixelFormat.
• Changed SetPixel on DepthImages to use ushort rather than int.
• Added GetPixel and ToPixelFormat methods.
• Added GetPixelRange method for DepthImage.
• Fixed incorrect byte ordering in SetPixel for certain pixel formats.
• Fixed "Buffer too small" error when creating certain images with non-aligned strides.
• Fixed bugs in the Flip, Invert and Threshold operators.
• Fixed several bugs related to pixel format conversions.
• Small fix in image decoder and encoder components to allow pass-through of null messages.

Changes to Batch processing

• Batch processing task attributes now allow for specifying the output store name and path, the replay descriptor, the pipeline delivery policy, and whether to enable pipeline diagnostics.
• Dataset APIs extended to allow for specifying pipeline delivery policy and enabling diagnostics when computing derived partitions.
• Improved progress estimates for computing derived partitions over datasets by considering session durations.
• Expanded progress window when running batch processing tasks to include data size, time elapsed and estimated time remaining.

Changes to Calibration

• BREAKING CHANGE: Calibration: Merged DepthExtensions and RotationExtensions into a new static class, CalibrationExtensions.

Changes to PsiStoreTool

• Added OutputPath parameter to Concat, Crop, Encode operations. These operations are now outputting stores in the specified paths. If a store already exists at the specified output path, the tool asks for confirmation to overwrite.
• Systematized the options and their explanations for the various commands.
• Added analyze verb to PsiStoreTool to measure how well messages streams are interleaved within a store.

Changes to ONNX

• Fixed a bug in the bounding box computation of the TinyYoloV2 ONNX model runner.

2020/12/07: Beta-release, version 0.14.35.3

Overview:

This release includes a new ONNX model runner for ImageNet models, new components for screen and window capture, updates to annotation editing in Psistudio, as well as a number of bug fixes and updates.

Changes to Media and Imaging

• Added VisualCapture component to capture a System.Windows.Media.Visual to a stream of Shared.
• Added WindowCapture component to capture a window specified by a window handle (defaulting primary screen) to a stream of Shared.
• Decode MJPEG frames in Linux MediaCapture component.
• Bug fix to avoid serializing/persisting entire underlying MemoryStream buffer in EncodedImage.
• Fixed bug where FromImage() would crash since it doesn't support 8bpp.

Changes to Visualization and PsiStudio

• Multiple updates to annotation editing. Fixed bug where users could not drop selection markers on top of an existing annotation. If a selection marker is to be dropped "close" to an existing annotation edge, PsiStudio now drops the marker exactly on the edge so that a collection of contiguous annotations can be easily created. Users can now directly edit an annotation in place; the text of an unrestricted annotation can be edited directly in the annotation itself, and clicking on a finite annotation opens a context menu where the user can immediately select the required value from a list. The properties of a newly created annotation are now automatically displayed. PsiStudio now displays an informative error message when a user attempts but is unable to place an annotation, for example, if the new annotation would overlap an existing annotation.
• Cleanly handle mismatched types when deserializing streams of custom types. Previously, when the current structure of some type T was different to the structure of T in an existing stream, PsiStudio would crash when it attempted to deserialize that stream. PsiStudio now displays a message indicating how the structure of T has changed and will stop attempting to read from the stream.
• Fixed bug in context menu where users had the option to display stream latencies in instant visualization panels. Latency plots should only be displayed in timeline panels.
• When snap-to-stream is enabled, PsiStudio now no longer attempts to snap to messages that are outside the current viewport.
• Bug fix where adding, removing and re-adding an image visualizer would crash PsiStudio.
• Bug fix: The current value of an instant visualization object is now cleared when it becomes unbound. Previously the last value (e.g. image) would continue to be displayed after the stream became unbound.
• Added visualization adapters for 3D Points, Lines, and Rectangles.
• Bug fix where saving annotations in PsiStudio would occasionally crash.
• Fixed bug where clicking on a timeline panel occasionally caused the entire timeline to shift by a large amount.
• Replay speed can now be reduced to a slower speed than 1.0. Clicking the down button when the speed is 1.0 or less will halve the current playback speed.
• Workaround for cache misses when visualizing very large stores. If the required data has not yet been read from disk PsiStudio now explicitly requests it rather than waiting for it to be loaded naturally.
• The states of the timing buttons in the toolbar are now preserved when switching layouts.
• Fixed bug where loading layout at startup sometimes crashed.
• Bug fix for correctly handling Nullable in the Expand Members context menu item for streams.
• Fixed several layout bugs in the diagnostics visualizer.

Changes to Kinect and AzureKinect

• Added support for Wired Hardware Sync for Azure Kinect Sensors.
• Changed default TemporalSmoothing parameter in AzureKinectBodyTrackerConfiguration to 0.

Changes to Runtime

• Bug fix for possible truncation and corruption of values in StreamEnumerable<T>.
• Bug fix for potential delay in RelativeTimeWindow stream operator.
• Removed obsolete Store class.
• The StreamMetadataBase class has been moved from the Microsoft.Psi.Audio namespace to the Microsoft.Psi namespace.

Changes to ONNX

• Added ONNX model runner for a variety of ImageNet models.

Other

• Added QUICKSTART help topic for LINUX.
• Removed sample projects - These have moved to a separate Psi Samples repository.
• Moved api docs to DocFX and added DocFX build project.
• Added encode verb to PsiStoreTool to support JPEG encoding of Shared<Image>.

2020/08/31: Beta-release, version 0.13.38.2

Overview:

The release brings several important updates to Platform for Situated Intelligence Studio (PsiStudio), including support for temporal data annotations, additional visualizers for cartesian charts and histograms, general updates for various UI elements, as well as bug fixes and other improvements.

At the runtime and infrastructure level, this release adds support for writing custom store readers (via the IStreamReader interface) that can provide streaming data to \psi pipelines as well as in PsiStudio from data sources stored in third-party formats. As an example, we have written a WaveFileStreamReader that allows reading data and visualizing data from WAVE files.

Finally, the release also introduces new \psi components that enable running Onnx machine learning models.

The full, more detailed list of changes is described below:

Changes to Visualization and PsiStudio

• Added support for temporal annotations. Users can now create streams of multi-track annotations based on custom annotation schemas.
• Added support for visualizing a property or field of a stream. Right-clicking on a stream and selecting “Expand members” will display these sub-fields and they can be visualized as well.
• Added initial support for dynamically loading batch processing tasks from 3rd party assemblies. The tasks are identified by the [BatchProcessingTask] attribute, and surfaced in the context menus for Dataset and Sessions. Currently only tasks that have a signature taking a (Pipeline, SessionImporter, Exporter) are surfaced.
• Added support for opening custom, third-party stores based on the available stream readers (e.g. choose .wav in file dialog).
• Temporal selection region is now highlighted.
• Added new visualizers for camera intrinsics (visualizes a camera frustum), depth image point cloud, depth image mesh, and image with intrinsics.
• Added new visualizers for cartesian charts and histograms using LiveCharts.
• CoordinateSystem visualizer now supports size and diameter as separate properties.
• KinectBody and AzureKinectBody visualizers now support individual bone thickness and joint radius.
• Panels now have a BackgroundColor property.
• 2D and 3D panels now have a RelativeWidth property that allows for variable width instant panels in a container.
• The layout selection combobox and associated buttons have been moved from the main toolbar to the Visualizations tab. Multiple minor bug fixes around switching layouts.
• Several engineering, refactoring, and code clean-up updates, including refactoring visualization for depth images, restructuring context-menu generation, rationalizing names for the VisualizationObject attribute, increased readability of type names the property browser.
• Several additional bug fixes in DepthExtensions.ProjectTo3D, IntersectLineWithDepthMesh, Point3D visualizer, message visualization object.
• BREAKING CHANGE: The Microsoft.Psi.Visualization.Common namespace has been renamed Microsoft.Psi.Visualization and the project Microsoft.Psi.Visualization.Common.Windows has been renamed Microsoft.Psi.Visualization.Windows.

Support for Running ONNX Models

• Added a new set of packages, Microsoft.Psi.Onnx.Gpu and Microsoft.Psi.Onnx.Cpu containing a component called OnnxModelRunner that enables running a specified Onnx model.
• Added a new set of packages, Microsoft.Psi.Onnx.ModelRunners.Gpu and Microsoft.Psi.Onnx.ModelRunners.Cpu, containing wrapper components that enable more easily running specific Onnx models. As a first example, TinyYoloV2OnnxModelRunner enables running object detection over images with a TinyYoloV2 model. We plan to extend the set of model runners in the future.

Custom Store Readers

• Added IStreamReader and IStreamMetadata interfaces to support reading streams from custom storage formats.
• PsiStoreReader is an implementation of IStreamReader for \psi stores.
• WaveFileStreamReader is an implementation of IStreamReader over WAVE files.
• Added extension methods to add sessions and partitions specific to \psi stores and WAVE file stores.

BREAKING CHANGES:

• Importer, Store, Dataset, Session and Partition APIs have changed to accommodate arbitrary stream readers.
• APIs returning PsiStreamMetadata now return IStreamMetadata.
• Static APIs specific to \psi stores have moved from Store to PsiStore (Store APIs are obsolete).
• Importer constructor no longer takes name and path; instead it requires an IStreamReader.
• PsiStore.Create(...) now returns PsiExporter (base Exporter is now abstract).

Other Changes to Runtime

• Fields with the [NonSerialized] attribute are no longer skipped when cloning. To skip cloning such fields, register the containing type with the SkipNonSerializedFields cloning flag.
• Improved serialization compatibility for some .NET types across frameworks.
• Consolidated overloads for Pipeline.Create() factory method into a single method.

BREAKING CHANGES:

• Renamed PsiStreamMetadata and Importer properties:
  • Lifetime -> StreamTimeInterval
  • ActiveLifetime -> MessageCreationTimeInterval
  • OriginatingLifetime -> MessageOriginatingTimeInterval
  • FirstMessageTime -> FirstMessageCreationTime
  • LastMessageTime -> LastMessageCreationTime
• Renamed IndexEntry.Time -> IndexEntry.CreationTime.
• Renamed Envelope.Time -> Envelope.CreationTime.
• Cloning objects containing unmanaged pointers and IntPtr fields (e.g. delegates) is no longer allowed by default. This behavior may be overridden by registering the containing type using the KnownSerializers.Register method with the cloning flags ClonePointerFields and CloneIntPtrFields respectively.

Changes to Kinect and AzureKinect

• Added Configuration member to AzureKinectSensor.

Changes to Microsoft.Psi.Imaging

• Checks on Unmanagedbuffer sizes to only copy as much data as is in the actual image (height*stride).
• Fixes to resize/scale to preserve sizes.
• BREAKING CHANGE: Changed Rotate and DetermineRotateWidthHeight to support loose/tight rotation.

Changes to Microsoft.Psi.Audio.Windows

• BREAKING CHANGE: The OutputFormat parameter in AudioCaptureConfiguration has been renamed Format.
• BREAKING CHANGE: The InputFormat parameter in AudioPlayerConfiguration has been renamed Format.

Other

• BREAKING CHANGE: Renamed [Task] attribute used to define tasks to [BatchProcessingTask].
• BREAKING CHANGE: Removed SimpleVoiceActivityDetector component.
• Dataset processing APIs now support progress reporting.
• Fixed bug in ImageAnalyzer component where region was being ignored.
• Set auto CRLF handling for the whole repository.
• Updated symbol packages to the .snupkg format.

2020/06/16: Beta-release, version 0.12.53.2

Overview:

The major changes in this release include the addition of support for the Azure Kinect sensor (including body tracking), an overhaul of Microsoft.Psi.Imaging, some additional Kinect for Windows component cleanup, visualization updates, as well as an upgrade of projects to Visual Studio 2019 and .NET Core 3.1. Miscellaneous updates were also made to other areas (see more details below).

A Note About Coordinate Systems: This release begins to rationalize the handling of coordinate system bases throughout the code base. We adopt the MathNet.Spatial convention everywhere, in which the X-axis represents "forward", the Y-axis represents "left", and the Z-axis represents "up". Matrix representations assume column vectors and are stored in column-major order. These changes particularly affect code and data from Microsoft.Psi.Calibration as well as the Kinect and Azure Kinect sensor and body joint representations.

Azure Kinect Component (New)

• Added AzureKinectSensor and AzureKinectBodyTracker components that enable using the Azure Kinect sensor with \psi (see more: Overview, Sample, Azure Kinect Documentation).
• The Kinect and AzureKinect body visualizers have moved into their own projects and need to be loaded as 3rd party visualizers for PsiStudio to become available.

Updates to Microsoft.Psi.Imaging

A number of updates were made to the imaging infrastructure in Microsoft.Psi.Imaging, including the addition of support for depth images.

BREAKING CHANGES:

• EncodedImage now requires width, height and pixelFormat to be specified on construction.
• EncodedImagePool is now keyed, and GetOrCreate() now requires the encoded image width, height and pixelFormat.
• Renamed Image.FromManagedImage() to Image.FromBitmap().
• Renamed Image.ToManagedImage() to Image.ToBitmap().
• Renamed ImagePool.GetOrCreate(Bitmap) to ImagePool.GetOrCreateFromBitmap(Bitmap).
• Image.EncodeFrom() now takes in an IImageToStreamEncoder instead of an Action<>.
• Moved ImageEncoder and ImageDecoder \psi components from the platform-specific .Windows and .Linux projects into the cross-platform Microsoft.Psi.Imaging project. These components are now provided an IImageToStreamEncoder or IImageFromStreamDecoder object to do the actual encoding and decoding. A variety of implementations (e.g. for JPEG, PNG) are available in the corresponding platform-specific .Windows and .Linux projects.
• Removed the static method ImageEncoder.EncodeFrom(encoder, ...) and replaced with encoder.EncodeFrom(...).
• Renamed CompressionMethod.PNG to CompressionMethod.Png.
• Renamed the Convert() stream operator for shared images to ToPixelFormat(), for clarity and to match the underlying component.
• The 3D related operators from DepthExtensions in Calibration now operate only on depth images.
• KinectSensor.DepthImage stream now returns a stream of DepthImage rather than a stream of Image.
• The Window() operator now correctly handles TimeInterval inclusivity rather than excluding messages on the boundary. Additionally, the operator no longer accepts a waitForCompleteWindow parameter and never delays the initial window.

Other changes:

• Added new DepthImage and EncodedDepthImage types to represent depth images, and a set of associated operators.
• Overhauled the various image processing functions and corresponding stream operators to rationalize the API.
• Added support for preserving the pixel format when decoding an EncodedImage.
• Fixed a number of bugs, including issues related to pixel formats and decoding.

Changes to Visualization and PsiStudio

• Kinect body visualization removed from Visualization.Common and put in its own project so that Visualization.Common and PsiStudio no longer have a dependency on the Kinect SDK.
• KinectDepth visualizer is now a more general DepthImage visualizer supporting both mesh and point cloud views.
• Updated handling of Shared objects in instant visualizers.
• PsiStudio now starts up with no default dataset and the main window titlebar now displays the name of the current Dataset and the name of the stream currently being snapped to (if any).
• Updated visualization object selection algorithm to select the visualizer the user most likely wants to use.
• A StreamVisualizationObject can now only be snapped to when it is bound to a source, and if it becomes unbound then we remove snap-to-stream.
• A TimelineVisualizationObject can now only be zoomed to when it is bound to a source.
• Multiple bug fixes to StreamBinding serialization, we now write out the assembly qualified name of StreamAdapterType, and we now correctly serialize child objects of 3D visualization object collections.
• Code cleanup of XyzVisualizationPanel and its view including adding bidirectional properties for camera position, direction and field-of-view.
• Added support for specifying which items in an UpdatableModelVisual3DVisualizationObjectDictionary are visible via a predicate.
• Added PsiStudio cursor nudging feature.
• The pipeline visualizer now shows a dotted edge connection between subpipelines when they contain a connection (perhaps even between deep descendants) spanning them.
• Fixed bug where PsiStudio settings were sometimes not being correctly written when shutting down PsiStudio.
• Fixed bug involving the index view in StreamReaders for InstantVisualizationObjects when the user changed the value of CursorEpsilon.

Changes to Runtime

• Changes to improve backward compatibility with some older stores.
• Added BridgeTo operator to Producer<T> creating a connector as needed to bridge pipelines.
• Raise PipelineRun event for all subpipelines before starting.
• Change pipeline state to Running as soon as subpipelines have started.
• Fixed a pipeline shutdown race condition.
• Fixed DeliveryPolicy<T>.WithGuarantees to do a union between guaranteed message sets.

Changes to Stores

• BREAKING CHANGE: Removed Exporter.WriteEmitters.
• Added CropInPlace store operation.
• Added support for supplemental metadata on streams.
• Fixed a bug in Store.Copy and Store.Crop which were using the ActiveTimeInterval instead of the OriginatingTimeInterval.

Changes to Calibration

• Added "Pose" fields for DepthDeviceCalibrationInfo objects, which are the inverse of Extrinsics.
• Added ability to specify distort/undistort as either normal or reversed. Normal would be the closed form equation which returns distorted points and the reverse version returns undistorted points.

Changes to PsiStoreTool

• Added support for showing stream size information in PsiStoreTool.
• Added verb for removing a stream from a store to PsiStoreTool.

Changes to Operators

• Updated Sample operator with a new default, that allows it to sample the nearest message to the clock when no tolerance or relative-time-interval is specified.
• The Zip component and corresponding operators were changed to return T[].

Changes to Other Components

• BREAKING CHANGES to KinectSensor component:
  • Removed custom Matrix code in KinectInternalCalibration and re-implemented using MathNet.
  • Removed IKinectSensor, KinectExtensions, DepthExtensions and all GZip encoding functionality.
  • Removed DepthToColorConverter and re-implemented as a generic imaging operator.
  • Moved Orientation functions and LevenbergMarquardt optimization into Microsoft.Psi.Calibration.
  • Changed KinectBody definition to store Joint poses as MathNet CoordinateSystems, with immediate conversion into MathNet basis (X Forward, Y Left, Z Up).
• Added a constructor for AudioCapture component allowing an output format to be specified.
• Added ALSA error codes in exceptions thrown by the Linux AudioCapture and AudioSource components.
• Fixed Linux audio and video components to work on 32-bit OS.
• Converted RealSenseSensor component to use new DepthImage.
• Added a constructor for MediaSource component that accepts a System.IO.Stream as input.
• Fixed an issue with Mpeg4Writer where image data was being written in RGB instead of BGR.

Miscellaneous Engineering Updates

• This release moves .NET Core projects to version 3.1. As a result, Visual Studio 2019 is now required to build the code. Building with Visual Studio 2017 will no longer be supported.
• Added code analysis to projects.
• The following NuGet packages were updated to the versions shown:
  • Extended.Wpf.Toolkit (3.6.0)
  • HelixToolkit.Wpf (2.11.0)
  • MathNet.Numerics.Signed (4.9.1)
  • MathNet.Spatial.Signed (0.6.0)
  • MessagePack (2.1.90)
  • Microsoft.NET.Test.Sdk (16.6.1)
  • MSTest.TestAdapter (2.1.1)
  • MSTest.TestFramework (2.1.1)
  • System.ServiceModel.Primitives (4.7.0)
• Various bug fixes, typographical corrections, and code cleanup.

2020/04/14: Beta-release, version 0.11.82.2

OVERVIEW:

This update brings several new developments, and numerous improvements and bug fixes in the \psi Runtime, Tools and Components.

• In Runtime the major changes include the addition of new types of delivery policies, emitter validators, and streamlining of the APIs for running the pipelines (see more details below)
• In Tools a number of significant updates have been brought to PsiStudio, including performance improvements, support for 3rd party visualizers, and matrix layouting, and other extended functionality. Similarly, the performance of the Diagnostics subsystem has been improved and visualization functionality for diagnostics data has been extended. A number of updates have been made also to PsiStoreTool.
• In Stream Operators and Components a number of extensions and bug fixes have been added.

Details of Changes to Runtime

A number changes have been made to clean up and streamline the APIs for pipeline execution. The changes below are breaking changes:

• Removed the Pipeline.Run(TimeSpan) method as its meaning was inconsistent with running a pipeline with a finite originating time interval. To run a pipeline for a given wall-clock duration, call RunAsync(), followed by WaitAll(TimeSpan), then dispose the pipeline to stop it.
• ReplayDescriptors no longer define the UseOriginatingTime property as it was redundant. We always assume originating times when reading messages within a given time interval from a store. Consequently, Pipeline.ProposeReplayTime() now takes only a single TimeInterval parameter representing the originating time interval.
• Pipeline.WaitAny has been removed. To be notified of source component completion, use the Pipeline.ComponentCompleted event instead.
• Pipeline replay at speeds other than real-time or maximum speed are no longer supported. Consequently, the replaySpeedFactor parameter and ReplaySpeedFactor property have been removed the Pipeline.Run methods and the ReplayDescriptor class respectively.
• The deliveryPolicy parameter for the Pipeline and Subpipeline constructors has been renamed to defaultDeliveryPolicy.

The available set of delivery policies has been extended. See more information in Delivery Policies tutorial:

• Added support for throttling policies via DeliveryPolicy.Throttled.
• Added a new lossless policy that attempts synchronous delivery or throttles otherwise via DeliveryPolicy.SynchronousOrThrottle.
• Added support for data-driven delivery policies via typed delivery policies with guarantees.

A number of additional changes have been made to the Runtime:

• Pipeline.RunAsync now supports progress reporting by means of an IProgress<double> object.
• The pipeline start time is now captured in a new Pipeline.StartTime property.
• Subpipelines provide access to the parent pipeline via the protected ParentPipeline property.
• When persisting a stream to a store, the stream closing time is now saved in the store catalog.
• Emitters now support optional user-defined message validation.
• Fixed a bug where the presence of an infinite source component within a subpipeline was in some cases causing the pipeline to terminate.
• Added message validation for emitters, and support for pipeline level defaults.

Details of Changes to PsiStudio

A number of significant updates have been brought to PsiStudio, including performance improvements, support for 3rd party visualizers, and matrix layouting, and other extended functionality:

Performance updates:

• Improved performance when reading data from disk for instant visualization objects. Previously these reads were occurring on the UI thread whereas now they happen in a worker thread. Now they are optimized so that the same piece of data is not read multiple times for different visualization objects.

New functionality and bug fixes:

• In order to make better use of available screen real estate, instant visualization objects (2D and 3D visualization objects) can now be arranged in a horizontal matrix in the visualization container. These matrices can contain from one to five instant visualization objects.
• PsiStudio can now visualize the Messages and Latencies for all streams, regardless of type.
• The states of the timing info buttons are now persisted across PsiStudio sessions.
• When the use clicks in a visualization panel, PsiStudio displays the properties of the first visualization object in that panel, or if the panel has no visualization objects it displays the properties of the panel itself. Similarly, whenever the user adds a new visualization object to the visualization container its properties are immediately displayed.
• All DateTime displays in PsiStudio now show the value down to 1/10,000 of a second.
• Fixed bug where certain line segments were not drawn if the value of any of the data points was NaN or positive infinity or negative infinity.
• Users can now play multiple audio streams simultaneously.
• Added snap to stream context menu item to the timeline panel.
• 3D visualization panel now supports camera manipulation and the execution of camera animation storyboards.

Support for 3rd party visualizers:

• Added support for 3rd party visualizers. Users who have written their own visualizers can now use them in PsiStudio in a much more seamless manner. Add a new node called <AdditionalAssemblies> to the PisStudio settings file at MyDocuments/PsiStudio/PsiStudioSettings.xml. Any visualization object marked with the [VisualizationObject] attribute, and any stream adapter marked with the [StreamAdapter] attribute will be automatically added to PsiStudio’s list of visualizers.

Other updates:

• For historical reasons related to PsiStudio previously supporting a COM model, Visualization Objects and Visualization Panels each had a separate Configuration class. This configuration has now been merged back into the visualization objects or visualization panels, simplifying the code.
• Updated the 3D visualization objects to have a more hierarchial structure.
• Layout serialization now includes a version number to help prevent loading a layout that is not compatible with the current version of PsiStudio.

Details of Changes to Diagnostics

The diagnostics system enables collecting and visualizing application diagnostics in PsiStudio. This release includes many performance improvements to the diagnostics system. The overhead of collecting diagnostics information has been improved. Message sizes are half what they were and rendering in PsiStudio has been improved by orders of magnitude. Many bug fixes have been made as well, mosting having to do with visualization in PsiStudio. There have been breaking changes to the diagnostics message type affecting visualization.

Visualization:

• To simplify the graph display, we now allow checking a box Show Exporter Links to show/hide connectors and edges from them to Exporters.
• Connectors render with a little icon (☍) and Joins now render in their own color with a simple plus (+).
• Type names have been simplified from their fully qualified form to a more natural “code form” such as (List<string>, Dictionary<char, (int[], double)>).
• Connector tooltip now shows target/source.
• Emitter names can now be optionally shown on edge labels.
• Delivery policies can now be optionally shown on edge labels and hilighting of edges by delivery policy is now supported.
• Property panel now allows toggling emitter/receiver names as well as delivery policies.
• A dotted line edge is now shown indicating a connector into a descendant pipeline. That is, an edge to the direct child containing the descendant.

Naming Improvements:

• Subpipeline and Exporter node label is now the pipeline name (with type name now as a tooltip)
• Components can now provide names via ToString() used to render nodes (rather than the default type name)

Miscellaneous:

• Averaging is performed by unit time rather than past n-messages and added dropped & processed averages.

Details of Changes to PsiStoreTool

• Added concat verb to PsiStoreTool (as well as Store.Concat(…) static method) allowing concatenation of stores.
• Added crop verb to PsiStoreTool allowing cropping of stores (exposing Store.Crop(…) functionality).
• Added Microsoft.Psi.TaskAttribute to mark “task” methods as light-weight jobs that may be executed by PsiStoreTool (and future PsiStudio).
• Added tasks and exec verbs to PsiStoreTool allowing listing and executing tasks.

Details of Changes to Stream Operators

A number of BREAKING CHANGES were made to stream operators:

• Windowing operators now produce streams of T[] rather than IEnumerable<T>, because the latter are not serializable.
• The mathematical and statistical operators no longer apply to streams of IEnumerable<T>, but to streams of T[].
• Removed the Sequence<T> operator overloads that take an IEnumerator argument. Use the overloads that takes an IEnumerable instead.
• Removed the optional interval and alignmentDateTime parameters from the Once and Return operators.

A number of other additional enhancements were made to other stream operators:

• Added support for default delivery policies and naming of the Parallel composite components and operators.
• Generator changes:
  • Streams produced by Generators are now aligned by default to the pipeline start time, unless explicit message times are supplied.
  • Fixed a bug in the time alignment computation which may in rare cases result in a missing first message when an alignmentDateTime is supplied.
  • Fixed a bug where some Generators were not obeying a ReplayDescriptor with a finite end time.
• Added a DynamicWindow component and corresponding (Window(…) overload) stream operators that enable data-driven windowing.
• Small extension (with optional parameter) to RelativeTimeWindow component and corresponding Window() stream operator, enabling them to post results immediately before seeing a full window. Default behavior is still to wait for the complete window.
• Added Merge operator, which trivially combines one or more streams of type T into a single stream of type Message<T>, each with generated at the pipeline time when the message arrives at Merge, but containing the original envelope.
• Added Zip operator, which operates like Merge but ensures delivery in originating time order (ordered within single tick by stream ID)
• Added First(n) operator, which filters a stream to the first n messages.
• Added a WriteEnvelopes method for streams that enables writing only the envelopes from a stream (without the payload) to the store.
• Importer.OpenStream<T> now checks that the type of the underlying stream messages matches the supplied type argument T and throws an exception if they do not match.
• Performance improvements to the Std operator.
• Minor changes in the way Generator, Importer and Exporter apply delivery policies.

Details of Changes to Components

A number of changes were made in a variety of components:

• AudioCapture component
  • Component now uses WASAPI event-driven capture by default. The previous behavior of pulling data from the engine at the target latency is still available by setting the UseEventDrivenCapture configuration parameter to false.
  • The default audio capture buffer size may now be overridden by setting the AudioEngineBuffer configuration parameter. Larger values may help reduce the likelihood of audio glitches.
  • Component now throws an IOException if it is unable to create the audio capture device.
• AzureSpeechRecognizer component:
  • Final recognition results are now aligned in originating time to the input voice activity detection signal.
• SystemSpeechSynthesizer component:
  • Added ReceiveSsml and CancelAll receivers to support issuing SSML-based speech synthesis requests and stopping of speech synthesis.
• SystemVoiceActivityDetection component:
  • Added InitialSilenceTimeoutMs, BabbleTimeoutMs, EndSilenceTimeoutMs, and EndSilenceTimeoutAmbiguousMs parameters in the component configuration. More details about these parameters may be found in the SpeechRecognitionEngine documentation.
• Face Identification:
  • The FaceRecognizer component, which identifies faces in a stream of Shared<Image>, has been updated to the latest Azure Face APIs.
  • A PersonGroup class supports managing person groups in Azure (creating, adding faces, training, deletion).
  • Also included are PsiStoreTool tasks supporting creating, training and testing from a directory of labeled static image files.
• Microsoft.Psi.Calibration changes (these are Breaking Changes):
  • Added support for full rational distortion model (with k1-k6 parameters)
  • Calibration representations now uses a canonical coordinate system throughout, in alignment with MathNet (right-handed, X forward, Y Left, Z up, column-vectors).
  • Moved KinectCameraCalibration class and corresponding interface into Microsoft.Psi.Calibration and renamed to DepthDeviceCalibrationInfo
  • SystemCalibration was renamed to MultiCameraCalibration
  • Moved ProjectTo3D class from Microsoft.Psi.Kinect to Microsoft.Psi.Calibration.
• Added Microsoft.Psi.DeviceManagement project which enables device enumeration. Updated Microsoft.Psi.Kinect project to allow for enumerating all Kinect devices.
• The Microsoft.Psi.CognitiveServices.Vision.Windows project is now .NET Standard and has been renamed Microsoft.Psi.CognitiveServices.Vision.

Miscellaneous

• Updated projects in solution to new .csproj style.
• Updated and consolidated NuGet references to the latest versions.
• Fixed a bug that was causing two unit tests to fail occasionally.

2019/08/27: Beta-release, version 0.10.16.1

OVERVIEW:

The major changes in this release are updates and added functionality in the stream operators for fusion and synchronization, operators for interpolation and sampling, and operators for generating streams. The release also includes a number of other updates and bug fixes - see more details below.

Details of Changes to Stream Operators

A number of changes have been made to the set of interpolators, and the synchronization operators (for the latest documentation on synchronization and stream fusion and merging see here):

• The set of interpolators has been restructured, generalized and expanded. Interpolators can now produce a result that has a different type than the input message type, and allow for matching with the Nearest, First or Last message. A taxonomy of greedy versus reproducible interpolators has been defined.
• Added support for an adjacent-values-interpolator, and constructed a linear interpolator based on it.
• The existing Join operator that allows for reproducible stream fusion and correct synchronization now operates with reproducible interpolators.
• A Fuse operator has been added to enable stream fusion with any generic interpolator (reproducible or greedy).
• Interpolators now accept a user-specified default value to use.

A number of other operators for sampling, generating streams, and statistical computations have also been updated:

• Redesign of Sample and Interpolate stream operators. Both operators are driven by a clock stream. The Sample stream operator selects the nearest message to the sampling point (within a tolerance or relative time window). The Interpolate operator uses a specified interpolator to generate the result. The documentation for sampling and interpolation is available here.
• The Generators.* methods have been updated to allow the developer to specify whether the generated streams should stay open or close after the enumeration terminates. The current documentation on stream generators is available here.
• Refactoring and clean-up of mathematical and statistical operators.

Details of Updates to Pipeline Diagnostics

• Added extension methods to provide access to graph-wide statistics such as total queued messages, total dropped, etc. across the whole graph.
• Enabing diagnostics at pipeline creation time (e.g. Pipeline.Create(true, ...)) used to take an optional TimeSpan interval. Now it takes an optional DiagnosticsConfiguration; containing this and other settings.
• New DiagnosticsConfiguration.TrackMessageSize flag determines whether message byte counts are tracked (false by default, as it is an expensive operation).

Details of Changes to Platform for Situated Intelligence Studio:

• Minor icon updates.
• Ensure that dialog boxes appear above the PsiStudio main window.

Other Updates

• Moved EncodedImage and EncodedImagePool out of Microsoft.Psi.Imaging.Windows and .Linux into the central .NET Standard shared Microsoft.Psi.Imaging. This is to enable referencing a single EncodedImage type across platforms.
• Interop: Updated CSV format docs for new behavior.
• Added sensor_msgs/Image message type to the ROS bridge.
• Added more informative exception message when posting messages on a stream with out-of-order sequence IDs.

Bug Fixes:

• Fixed bug in pipeline diagnostics capture regarding counts of messages dropped.
• Fixed bug in queue-size-constrained delivery policies (including DeliveryPolicy.Latest) which sometimes led to a queue size larger by one than the max, and allowed additional messages to be processed.

Breaking Changes:

• As a result of the re-organization of interpolators, the Match static class and the interpolators it contained have been renamed. Specifically: Match.Best -> Reproducible.Nearest, Match.Exact -> Reproducible.Exact; similar for the OrDefault versions.
• Interpolator<T> was replaced with Interpolator<TIn, TOut> to support interpolation to a different output type.
• Generators.Sequence, Generators.Repeat and Generators.Range require specifying an explicit time interval (previously the time interval parameter was optional and defaulted to 1 tick).
• The existing overloads of the Sample stream operator that used an interpolator were renamed to Interpolate.
• The OpenCV Sample now requires OpenCV 4.1.1 to compile. You must also set the environment variable OpenCVDir_V4 to point to your local installation of OpenCV 4.1.1.
• Enabing diagnostics (e.g. Pipeline.Create(true, ...)) now takes an optional DiagnosticsConfiguration rather than a TimeSpan interval.

2019/07/18: Beta-release, version 0.9.6.1

OVERVIEW:

The main highlights in this release are:

• A major refresh of the Platform for Situated Intelligence Studio user interface, including a new color scheme, updated icons, and new functionality.

Details of Changes to Documentation:

• The installation documentation has been updated to detail how to build the \psi codebase on Windows (using either Visual Studio 2017 or 2019), or on Linux.

Details of Changes to Platform for Situated Intelligence Studio:

• Updated the look and feel of the user interface, including new colors and icons.
• The Datasets treeview and the Visualizations treeview now feature Expand All and Collapse All buttons to fully expand and collapse the tree.
• The Visualizations treeview now contains a new button to synchronize between it and the Datasets treeview. When a stream in the Visualizations treeview is selected, clicking this button will select the source stream in the Datasets treeveiw and bring it into view. This feature is useful for those times when you forget exactly which source stream a visualization is depicting.
• When in Live cursor mode, the timing information for the selection start and end markers is now not displayed as this information is irrelevent when in live mode.
• Added new context menu item Zoom to Stream Extents to streams in the Visualizations tree view.
• Added new context menu item Zoom to Panel Extents to panels in the Visualizations tree view.

Bug Fixes:

• Fixed a bug in the Join operator that made it not correctly take into account open relative time intervals. See this issue.
• Fixed bug in the Diagnostics visualizer where connectors between subpipelines were being incorrectly rendered.
• Fixed bug in the Diagnostics visualizer that would cause it to crash if it encountered messages with unserializable fields. We now return 0 for the size of such messages.
• Fixed bug in Rodrigues method in the Orientation class of Microsoft.Psi.Kinect.Windows. This method was returning an NaN rotation matrix when passed a zero rotation vector, it now returns an identity matrix instead.

2019/07/03: Beta-release, version 0.8.32.1

IMPORTANT NOTE:

In this release we have retargeted all .NET Framework projects to .NET Framework 4.7.2, and all native projects to Windows 10 SDK version 10.0.18362.0. In order to build the source code, please ensure that the following additional components are added to your Visual Studio installation:

• .NET Framework 4.7.2 targeting pack
• Windows 10 SDK (10.0.18362.0)

If you are using Visual Studio 2019, you may install these components using the Visual Studio Installer, or via the download links provided above. For Visual Studio 2017, you may use the Visual Studio Installer to install the .NET Framework 4.7.2 targeting pack, but you will need to download and install the Windows 10 SDK (10.0.18362.0) using the provided link.

OVERVIEW:

Some of the main highlights in this release include:

• Support for pipeline structure visualization, including message flow and diagnostics information (e.g. latencies, throughputs, stream statistics, etc.).
• Retargeted projects to .NET Framework 4.7.2 and Windows 10 SDK 10.0.18362.0.
• Updated NuGet package references in many projects to more recent versions.
• Runtime API changes and bug fixes.
• Moved documentation to wiki pages.
• Created gitter channel.

Breaking Changes:

• The ISourceComponent.Stop method now takes a finalOriginatingTime argument representing the pipeline shutdown time and a notifyCompleted delegate which the component must now call to notify the pipeline that it has completed generating source messages up to the finalOriginatingTime. For more details, see here.
• Pipeline exception handling is now exposed through new PipelineException event rather than overloading the PipelineCompleted event to also serve as an exception handler.
• Input and output connectors for subpipelines should now be constructed using the new CreateInputConnectorFrom and CreateOutputConnectorTo static methods of the Subpipeline class.
• The Match.Any interpolator has been removed.
• Enabing diagnostics at pipeline creation time (e.g. Pipeline.Create(true, ...)) used to take an optional TimeSpan interval. Now it takes an optional DiagnosticsConfiguration; containing this and other settings.

Details of Changes to the Runtime:

Several significant changes were made to the pipeline startup and shutdown logic:

• There is now a single scheduler for the main pipeline and all subpipelines. To facilitate management of work items in subpipelines, a new SchedulerContext has been introduced to which work items may be assigned.
• Message delivery in a pipeline is now delayed at startup until all source components in the pipeline have been activated.
• To enable correct and reproducibile pipeline shutdown, changes were made to the ISourceComponent that will need to be implemented by stream sources. For more details, see here.
• Renamed the PipelineElement Start and Stop methods to Activate and Deactivate respectively.
• Added a check in Receiver.OnSubscribe to ensure that emitters and receivers are not connected across pipelines (connectors should be used instead).
• Changes to the pipeline exception handling mechanism via a new PipelineException event.
• New DiagnosticsConfiguration.TrackMessageSize flag determines whether message byte counts are tracked (false by default, as it is an expensive operation).

Details of Changes to Operators and Components:

• Updated ParallelSparse and Exporter to use the composite component pattern, i.e. they are now implemented as a single subpipeline component.
• Added MessageConnector and IConnector interface to support connectors that wrap the entire message as they pass it along (used by Exporter).
• Added new static methods CreateInputConnectorFrom and CreateOutputConnectorTo on Subpipeline for creating connectors between pipelines.
• Changed the StreamEnumerable component (used by the ToObservable operator) to listen to the Unsubscribed event of its source rather than the PipelineCompleted event.

Details of Changes to Platform for Situated Intelligence Studio:

• New pipeline diagnostics visualization feature enables visualization of the pipeline structure and message flow statistics in PsiStudio.
• Added a new toolbar button that lets the user toggle whether the navigator's cursor follows the mouse cursor when in manual cursor mode. This setting can be toggled via the button and also via the shortcut "Alt+F" (enabled by default).
• When playing back in repeat/loop mode, we now ensure the cursor remains visible after looping from the end of the stream back to the beginning.
• Added support for visualizing streams of strings.
• Generic message visualizer now shows string representation of message in live legend.
• Added support for formatting in legends.
• Moved rectangle and coordinate system visualizers out of PsiStudio and into Visualization.Common.
• Created interface IView3D as well as a base version for handling collections of 3D visuals.
• Created a single base 3D visualization object, removing a lot of duplicate code.

Other Bug Fixes:

• Fixed an issue which caused the pipeline to appear to stop responding if an exception is thrown in a subpipeline's attached PipelineRun event handler.
• Fixed a bug in PsiStudio where the cursor remains a hand after dragging within a timeline visualization panel.
• Fixes to various visualizers (cleaned-up pattern for InitNew and handling config and property changes).
• Fixed an issue where live stores sometimes had no last message time.
• Fixed an AccessViolation related to IAudioEndpointVolumeCallback in the Windows audio components.
• Fixed a bug in one of the tuple-flattening Join operator overloads which was causing it to use the incorrect interpolator.
• Fixed a thread safety issue in the Join operator.
• Fixed a thread cleanup issue on disposal in InfiniteFileWriter.

---

2019/04/05: Beta-release, version 0.7.57.2

OVERVIEW:

There are many additions and updates in this release, but the major changes can be summarized as:

• Improvements and fixes to the pipeline shutdown procedure, as well as fixes to the Parallel and Join operators to support reproducible dynamic sub-pipeline construction and teardown via the Parallel operator.
• Streamlining the use of Shared<T> for more efficient messaging of large objects such as images, and added in-depth documentation on this topic.
• PsiStudio now supports connecting to live stores and fast layout switching.

Breaking Changes:

• The pipeline startup and shutdown procedure has been updated to support correct and reproducible shutdown where possible in dynamic pipelines. Specifically:
  • Removed the RegisterPipelineStart/Stop/Final handlers. If the component implements ISourceComponent, then use the Start or Stop method, or use one of the following events instead: PipelineRun, PipelineCompleted, Receiver.Unsubscribed.
  • The authoring of source components has been streamlined, and in the process the IFiniteSourceComponent and ISourceComponent interfaces have been unified into a single, new ISourceComponent interface. Implementers should call the notifyCompletionTime delegate supplied in the interface's Start method to indicate its completion time (or DateTime.MaxValue if it is infinite).
• Removed GroupBy operator.
• MessagePack no longer uses LZ4 compression during serialization, and now emits DateTime objects as Ticks rather than string representations.

Updates to Pipeline Shutdown Logic:

There have been multiple changes made to the pipeline finalization code to support an orderly shutdown process.

• Subpipeline now has independent scheduler so that it may fully shut down independently of the main pipeline.
• Renamed several Pipeline events:
  • PipelineCompletedEvent event has been renamed PipelineCompleted.
  • ComponentCompletedEvent event has been renamed ComponentCompleted.
  • PipelineCompletionEventArgs class has been renamed PipelineCompletedEventArgs.
  • A new event PipelineRun has been added, it is raised when Pipeline.Run (or Pipeline.RunAsync) has been called but before the components have started work.
• Receiver component now provides an Unsubscribed event.
• Scheduler now drops messages that were posted after the pipeline has shut down.
• Scheduler.Schedule method now returns an indication of whether the call was ignored due to the pipeline having already shut down.
• Bug fix in ParallelSparse so it works correctly with various branch termination policies.
• Pipeline pause for quiescence now includes all child subpipeline schedulers.
• Fixed bug in the ToEnumerable operator where empty streams were not being handled correctly.

Updates to Parallel Operator:

• Default branch termination policy now returns the OriginatingTime of the last message from that branch.
• Parallel operators now use a Connector to bridge to Subpipelines.
• Added a fix for Parallel with orDefault, which was not working properly in some cases due to errors in the matching function.

Updates to Join and Match Operators:

• Fixed bug in Join operator relating to the dynamic closing of secondary streams.
• Fixed bug in Match component's NearestMatch function where if there were no available messages we would always return InsufficientData. We now first check if the stream is closed, and in this case we return DoesNotExist instead.

New Features in Shared Pool APIs:

• The Shared<T>.Create method no longer supports specifying a SharedPool to which shared objects may be recycled. Recyclable shared objects must now be created from a shared pool.
• The SharedPool<T>.GetOrCreate method no longer supports specifying a construction delegate, it must now be provided when the SharedPool<T> is created.

New Features in Audio and Speech:

• Added grammar-based intent detectors using Microsoft.Speech and System.Speech
• We now ensure that messages have monotonically increasing OriginatingTimes in the MicrosoftSpeechRecognizer component to comply with the new rule on the Emitter component introduced in the last release.
• Added support for AzureSpeechServices and marked as deprecated the obsolete component BingSpeechRecognizer.
• The Speech event duration is now defined as the OriginatingTime of the last message where the VAD was still false (before it switched to true) to the last message where the VAD output was still true. Previously we were defining the event duration as the OriginatingTime of the first message where the VAD switched to true to the OriginatingTime of the first message where it went false again.
• Added Reframe operator to Audio.

New Psi Components:

• Added new Psi component PersonalityChat to the Microsoft.Psi.CognitiveServices.Languages.Windows project that wraps Project Personality Chat.

New Features in Platform for Situated Intelligence Studio:

• Removed all of the COM features that allowed a Psi application to launch and control PsiStudio. See next point.
• PsiStudio can now connect directly to the store of a running Psi application in the same way it connects to a previously created store. Users can seamlessly switch between Live mode and Playback mode when visualizing a live store.
• Simplified the hierarchy of VisualizationObject classes.
• Added option to show tracking id in the kinect body visualizer.
• When multiple audio streams are being visualized, users can now select which audio source should be played through the user's PC speakers by right-clicking the relevent stream in the Visualizations view.
• Added new Repeat button to the toolbar so that playback will loop infinitely between the Selection Start and Selection End markers.
• The position and layout of PsiStudio is now preserved when the application is shut down and is restored the next time PsiStudio is launched.
• Added Layout Chooser to the toolbar. This allows the user to quickly switch between different Visualization layouts. Users can create new layouts either from scratch or based on an existing layout and then save this new layout. Layouts are stored in /MyDocuments/PsiStudio/Layouts/.
• Loading and then switching between multiple sessions is now correctly supported. When a Visualization layout is selected, switching to a different session will automatically cause all Visualizers to rebind to the stores associated with this newly selected session.
• Added new icons to indicate when a Visualization has no stream to bind to in the current session.

Other Bug Fixes:

• Fixed bug where the final extent of the store on disk was not being correctly truncated to the actual stream size after pipeline shutdown.
• Fixed bug in PsiStudio where zooming out a long way would cause the app to apparently freeze.
• Fixed bug where StoreWriters and StoreReaders were generating different names for the same global mutex that indicated a store was currently live and being written to.
• Fixed bug where replay time intervals in parent pipelines did not take into account the replay time intervals in subpipelines.
• Fixed memory leaks in Microsoft.Psi.Imaging.Operators class.
• Improved performance of the serialization classes.
• Reduced startup delays in subpipelines.

---

2018/11/30: Beta-release, version 0.6.48.2

Breaking Changes:

• It is now a requirement that messages posted on an Emitter have strictly increasing originating times. Attempting to post multiple messages with the same originating time on the same stream will cause an exception to be thrown.
• The Buffer, History and Window operators have been unified as a single set of Window operators which take either an index-based or a relative time-based interval. The index-based variants emit the initial buffer only after the total count of messages within the specified index interval have been accumulated, whereas the time-based variants emit the initial buffer as soon as messages within the specified relative time interval are available.
• The following operators have been removed:
  • SelectMany.
  • Mirror.
  • Repeat - use Pair instead.
  • Buffer - use Window instead.
  • History - use Window instead.
  • Previous
• Delivery Policies have been simplified and renamed:
  • The Throttled policy has been removed. It will be re-introduced in a later release once issues around throttling have been resolved.
  • The Default, Immediate and ImmediateOrThrottle16 policies have been removed.
  • The QueueSize property has been renamed InitialQueueSize.
  • The MaximumLag property has been renamed MaximumLatency.
  • The IsSynchronous property has been renamed AttemptSynchronous.
• As a result of the above changes, many stream operators have been amended to take an optional DeliveryPolicy parameter.
• The constructors for the Connector component and associated CreateConnector extension methods no longer take an owner parameter.
• The serialization format for SystemCalibration has changed:
  • The ImageWidth and ImageHeight properties have been removed.
  • The NumberOfFrames property has been moved to the CameraCalibration class.
• The ICameraIntrinsics interface has changed:
  • The signature of the DistortPoint method has changed.

New Features in Platform for Situated Intelligence Studio:

• Updated the layout of the main Psi Studio screen. The Datasets tree view and the Visualizations tree view have been moved to the left hand side of the application. Furthermore they are no longer on separate tabs, they appear one above the other so that users no longer have to switch back and forth between tabs while laying out the visualizations. Datasets and Visualizations previously each had their own Properties pages, but now there is a single Properties page on the right hand side of the application that is able to display the properties of either type of object. Both the Datasets and Visualizations tree views and the Properties page can be resized or hidden completely to give more screen real estate to the main Visualization view.
• Added multi track event visualizer TimeIntervalHistoryVisualizationObject which is useful for visualizing multiple tracks of events having some finite timespan such as multiple speech-to-text streams. This visualizer will be loaded when visualizing streams containing messages of type Dictionary<string, List<(TimeInterval, string)>>. The Dictionary keys represent unique track IDs, each element in the List represents an event to display in the track and contains a tuple of the TimeInterval representing the start and end times of the event and a string representing the text that will be displayed inside the time interval. Note that since this is a history visualizer, each message should contain ALL events that have occurred up until the time of the message. This implies that the last message in the stream contains all of the data required to display the visualization.
• Users can now visualize streams by dragging them from the Datasets tree view directly into the main Visualization panel.
• Added 'Snap to Stream' functionality on certain visualizers to snap the cursor to the messages of the snapped stream.
• Added 'Visualize Messages in New Panel' and 'Visualize Latency in New Panel' commands to the stream context menu.
• Psi Studio now automatically attempts to repair corrupted stores when opening them.

New Features in Platform for Situated Intelligence Runtime/Core:

• Initial version of data interop with the introduction of Microsoft.Psi.Interop, with support for MessagePack, JSON and CSV data formats and ZeroMQ transport. See the Interop topic for more details.
• New dynamic store reader allows reading any stream from any store to dynamic primitives or to ExpandoObject of dynamic without requiring a reference to the .NET type of the stream messages.
• New PsiStoreTool command-line tool which allows exploration of available streams in a store, conversion to other formats using interop, and saving to disk or sending over a message queue for consumption by other platforms and/or languages.
• Exposed Scheduler as a parameter to Pipeline and Clock as a parameter to Scheduler.
• Multiple handlers may now be registered on start, stop and final pipeline events.
• Improved #TRACKLEAKS debug information in RecyclingPool.

New Features in Imaging:

• Added SetPixel method to Image.
• Added DrawText extension method for Image.
• Added support for CameraIntrinsics and CoordinateSystem to SystemCalibration.
• The IKinectCalibration interface and KinectCalibration class have been extended to support conversion from depth coordinates to color space coordinates using the new ToColorSpace operator.

Bug Fixes:

• Fixed several issues where visualization objects were not being displayed in the correct color in Psi Studio.
• Fixed a bug which would sometimes cause Psi Studio to crash when visualizing image streams.
• Fixed a bug which caused the mouse to move the cursor position during playback in Psi Studio.
• Fixed an issue causing streams to disappear at the end of playback in Psi Studio.
• Fixed a crash in Psi Studio when opening a layout created from a store which has since been deleted.
• Fixed a bug which sometimes caused timeline plots to be truncated when loading a layout in Psi Studio.
• Fixed an exception when closing Psi Studio after a live visualization session.
• Fixed a performance issue reading from Psi stores which occurred at the transition between consecutive data files.
• Fixed a bug where the ImageCompressor was not properly disposing of an encoded image after decoding it.
• Fixed a bug where ImagePool would sometimes return a recycled image with incorrect dimensions.
• Fixed a bug where the pipeline replay interval would sometimes extend beyond the lifetime of a stream being read from a store.
• Fixed a bug which caused KinectSample to crash when it detected no faces.
• Fixed the Scale image extension method to throw an exception when attempting to call it on an Image with an unsupported format.
• Fixed a few intermittently failing unit tests.
• Fixed a bug which sometimes caused a loss of precision when computing the current pipeline time.

---

2018/09/13: Beta-release, version 0.5.48.2

IMPORTANT NOTE:

For this release we have added the compiler switch /Qspectre to our C++ projects which helps mitigate against Spectre security vulnerabilities. In order to successfully compile the Platform for Situated Intelligence solution you must upgrade your Visual Studio instance to version 15.7 or later and add the two following components to your Visual Studio installation:

• VC++ 2017 version version_number Libs for Spectre (x86 and x64)
• Visual C++ ATL (x86/x64) with Spectre Mitigations

For more information on the Spectre vulnerabilities and information about how to add the above components to Visual Studio, please see the following pages:

• Visual C++ Team Blog - Spectre mitigations in MSVC
• Microsoft Docs - /QSpectre

New Features in Platform for Situated Intelligence Studio:

• Added new PlotStyles to Timeline Visualizer. Plots can be rendered with the following styles:
  • Direct (default): A straight line is drawn from each message to the next message to create a standard line plot.
  • Step: Messages are joined by a horizontal line followed by a vertical line for visualizing quantized data.
  • None: No lines are drawn between messages. If you select this plot style then you should also update the MarkerStyle from its default value of None or nothing will be drawn in the plot.
• Visualization Panels in Psi Studio can now be resized by dragging their bottom edge vertically.
• Visualization Panels can now also be re-ordered with the mouse via drag & drop.
• Users can now drag the visible portion of a Timeline Plot to the left or right using the mouse.
• Added modal window while loading a dataset to inform the user of the progress of the data load operation.
• Added new timing information toolbar buttons. These buttons can be used to display absolute message times, message times relative to the start of the session, and message times relative to the start of the current selection.
• Performance improvements when plotting messages.

New Features in Runtime:

• Join operator now matches against a final secondary message upon stream closing once it can be proven that no better match will exist.
• Added support for Parallel operators that take an Action rather than a Func.
• Adding components once a pipeline is running is no longer supported and now throws an exception. The recommended approach is to add a Subpipeline.
• Consolidated Windows SDK versions. Previously different parts of the toolset required different WinSDK versions, consolidated so that the only Windows SDK version that Psi now requires is 10.0.17134.0
• Improved how the pipeline shuts down to ensure that all existing messages are drained from it before stopping.

Bug Fixes:

• Fixed bug where switching Psi Studio from "Realtime" mode to "Playback" mode would result in the user being unable to move the timeline cursor or reposition the timeline.
• Fixed bug where we were trying to calculate the relative path of a partition to a dataset when the dataset was stored on the local disk but the partitions within the dataset were stored on a network share.
• Fixed bug where loading very large datasets would sometimes crash PsiStudio.

BREAKING CHANGES in this release:

• Removed several versions of the Parallel operator
• Renamed several components for consistency:
  • AudioSource -> AudioCapture
  • AudioSourceConfiguration -> AudioCaptureConfiguration
  • AudioConfiguration (on linux) was eliminated and replaced by two corresponding classes AudioCaptureConfiguration and AudioPlayerConfiguration
  • TransformImageComponent -> ImageTransformer
  • AcousticFeatures -> AcousticFeaturesExtractor
  • AcousticFeaturesConfiguration -> AcousticFeaturesExtractorConfiguration
• Renamed Parallel component to ParallelFixedLength to match naming convention with the other components.
• Partial speech recognition results for the SystemSpeechRecognizer, MicrosoftSpeechRecognizer and BingSpeechRecognizer components are now posted on a new stream named PartialRecognitionResults. The default Out stream now contains only final recognition results.
• Psi Studio will no longer load third party visualizers, for the time being it will only display its built-in visualizers.

---

2018/07/02: Beta-release, version 0.4.216.2

Interim release with support for new devices, runtime enhancements and several API changes, as well as minor bug fixes:

• Added support for RealSense depth camera
• Added the FFMPEGMediaSource component for Linux
• Added a [Subpipeline] class, enabling nested pipelines
• Parallel now uses subpipelines
• Sequence, Repeat and Range generators now allow time-aligned messages
• Additional minor bug fixes.

Several API changes have been made:

• Generators.Timer(...) is now Timers.Timer(...)
• IStartable has been replaced by ISourceControl/IFiniteSourceControl and the way that components get notified about the pipeline starting and stopping has changed

---

2018/04/04: Beta-release, version 0.3.16.5

Interim release with a few changes to the samples and some minor bug fixes:

• ArmControlROSSample is now RosArmControlSample.
• PsiRosTurtleSample is now RosTurtleSample.
• Added LinuxSpeechSample.
• KinectFaceDetector component now outputs an empty list if no face is detected.
• NuGet packages are now marked beta.
• Additional minor bug fixes.

---

2018/03/17: Beta-release, version 0.2.123.1

Initial, beta version of the Platform for Situated Intelligence. Includes the Platform for Situated Intelligence runtime, visualization tools, and an initial set of components (mostly geared towards audio and visual processing). Relevant documents:

• Documentation - top-level documentation page for Platform for Situated Intelligence.
• Platform for Situated Intelligence Overview - high-level overview of the platform.
• NuGet Packages List - list of NuGet packages available in this release.
• Building the Code - information about how to build the code.
• Brief Introduction - brief introduction on how to write a simple application.
• Samples - list of samples available.

The Roadmap document provides insights about future planned developments.