[FEATURE]: Migration to database

[surapuramakhil]

Feature summary

Currently we are using JSON file which limits max size to system memory. implement light weight DB.

SQLite vs postgrest (via docker)
After research - sqlite writes are sequential & DB updates are via automated actions - postgres becomes right choice

Feature description

Apart from log use cases, other usecases will eventually migrate to DB.

migrate following use cases to database.

• Tracking of jobs - Applied, skipped
• Failed job application - in both DB and in logs (i.e. files)

Motivation

No response

Alternatives considered

No response

Additional context

No response

[github-actions]

This issue has been marked as stale due to inactivity. Please comment or update if this is still relevant.

[yashsachdev]

Implementation of nosql for this feature

[alex-shvachko]

I would like to suggest you using DuckDB for that - this is lightweight relational db that runs only with run of ur application and dont wastes memory.

[surapuramakhil]

@alex-shvachko DuckDB is OLAP, OLTP (OLTP inclined) is ideal DB of choice for the current use case.