Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLI support for unconditional sampling and assessing data likelihood of any dataset #178

Open
Schaechtle opened this issue Aug 21, 2024 · 2 comments
Assignees

Comments

@Schaechtle
Copy link

What does this issue request?

Add CLI support for two modes of evaluating posterior models, to make it easy to do either of two things:

1. Unconditional sampling of synthetic data

The goal here is to generate posterior samples to create datasets that can be compared to the training data/observations and held-out data.

2. Compute held-out likelihood scores

The goal here is to be able to compute held-out likelihood scores for training and held-out data.

Why do we want this?

This allows us to compare special cases of GenDB against relevant baselines, like previous CrossCat implementations. Note that some relevant baselines may generate synthetic data but not allow assessing held-out likelihood. The effort should be particularly relevant for evaluations that would be needed for a future publication -- but it's also not urgent right now.

@ThomasColthurst ThomasColthurst self-assigned this Aug 22, 2024
@ThomasColthurst
Copy link

For the held-out likelihoods, do you need them per held-out item or only per test set?

(Or to ask another way: if there are 1000 items in the held out test set, do you want a single log likelihood for all of them, or do you want 1000 individual log likelihoods?)

@Schaechtle
Copy link
Author

For the held-out likelihoods, do you need them per held-out item or only per test set?

(Or to ask another way: if there are 1000 items in the held out test set, do you want a single log likelihood for all of them, or do you want 1000 individual log likelihoods?)

I would return a table or list of the latter because we can aggregate easily -- but we also may want to dig into which rows generate which log-likelihood values in case we encounter bugs (e.g. NaNs for certain held-out rows).

This was referenced Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants