First release

This is the first release of Spyder.

Features

• Fast DER computation from Python code. Example usage:

import spyder

# reference (ground truth)
ref = [("A", 0.0, 2.0), # (speaker, start, end)
       ("B", 1.5, 3.5),
       ("A", 4.0, 5.1)]

# hypothesis (diarization result from your algorithm)
hyp = [("1", 0.0, 0.8),
       ("2", 0.6, 2.3),
       ("3", 2.1, 3.9),
       ("1", 3.8, 5.2)]

metrics = spyder.DER(ref, hyp)
print(metrics)
# DERMetrics(miss=0.098,falarm=0.216,conf=0.255,der=0.569) 

print (f"{metrics.miss:.3f}, {metrics.falarm:.3f}, {metrics.conf:3f}, {metrics.der:.3f}")
# 0.098, 0.216, 0.254, 0.569

• CLI interface to compute DER between RTTM files. Example:

> spyder ref_rttm hyp_rttm
Average error rates:
----------------------------------------------------
Missed speaker time = 11.48
False alarm speaker time = 2.27
Speaker error time = 9.81
Diarization error rate (DER) = 23.56

• Support for computing per-file DER from CLI using the --per-file flag.

Speed benchmark

We have done some basic speed comparisons in this blog post using the AMI development data as an example. Spyder is:

• 3-5x faster than md-eval.pl (when invoked from Python code);
• 10x faster than pyannote.metrics.

Processing time vs. number of turns in the recording: