Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing db persistency in the local dev scenario #674

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

aittalam
Copy link
Member

@aittalam aittalam commented Jan 17, 2025

What's changing

This PR binds the sqlite database in the backend container to a local file in ~/.lumigator/db.
This allows the DB to persist from one session to another (i.e. even after make local-down+make local-up.

Closes #515

How to test it

Steps to test the changes:

  1. run make local-up
  2. add a dataset
  3. run make local-down
  4. run make local-up
  5. get datasets list

Additional notes for reviewers

This is still a draft pr because:

  • some integration tests run using this DB, which is not ideal. We should either prevent this or add in the docs that this might happen and how to avoid it
  • while retrieving datasets works properly, jobs do not work as well because currently the list jobs method returns an intersection between jobs in the DB and jobs running on ray. Unless we persist ray data (which btw is possible), or start relying on the DB only to store/get job info, we won't be able to see jobs from a previous lumigator execution. 👉 NOTE that this is partially solved by Add persistence to Ray #744, we need to decide how to deal with logs in that case: see this comment)
  • this is currently working in dev setup only. Do we want to have this as a stable feature in the make start-lumigator scenario? This still has to be discussed

@aittalam aittalam linked an issue Jan 17, 2025 that may be closed by this pull request
1 task
@aittalam aittalam changed the title Fixed local.db mapping, now pointing to a local file Fixing db persistency in the local dev scenario Jan 17, 2025
@aittalam aittalam mentioned this pull request Jan 29, 2025
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG]: sqlite database is not persisted (in dev environment at least)
1 participant