Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Autodetection - setup system for validating the experiment results #1450

Closed
4 tasks
jbonisteel opened this issue Nov 17, 2023 · 1 comment
Closed
4 tasks
Assignees
Labels
epic A large user story that needs to be broken down funder/drl2022-2024

Comments

@jbonisteel
Copy link

jbonisteel commented Nov 17, 2023

This is about setting up a system that allows us to estimate how accurate our loni metric is and what it's key evaluation metrics are.

As part of this activity the following tasks are needed:

  • Research what can be used as a ground truth
  • Build a ground truth to compare the analysis against
  • Implement the ground truth to analysis comparison system
  • Run the comparison and evaluate the performance of the model

This work is highly iterative and feeds directly back into the improvement of the analysis and is a precursor to completing the PoC of the analysis. It is however also very instrumental to getting this into production so it will continue going forward (especially the last 2 points about running it and improving the comparison system)

@jbonisteel jbonisteel added epic A large user story that needs to be broken down funder/drl2022-2024 labels Nov 17, 2023
@hellais hellais changed the title Autodetection - validation Autodetection - setup system for validating the experiment results Nov 24, 2023
@hellais hellais added this to Roadmap Jan 7, 2025
@hellais
Copy link
Member

hellais commented Jan 27, 2025

Most of the work for a first iteration of this has been completed in ooni/data#84 & ooni/data#21. We can consider this done.

@hellais hellais closed this as completed Jan 27, 2025
@github-project-automation github-project-automation bot moved this to Done in Roadmap Jan 27, 2025
@hellais hellais moved this from Done to Epic in Roadmap Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
epic A large user story that needs to be broken down funder/drl2022-2024
Projects
Status: Epic
Development

No branches or pull requests

2 participants