sqlite: expose backup api

[geeksilva97]

Closes #55413

This PR exposes the SQLite Online Backup API, which allows database backup.

The API is inspired by better-sqlite3 https://github.com/WiseLibs/better-sqlite3/blob/master/docs/api.md#backupdestination-options---promise.

Multithreading caveats

As long as writes come from the same process and handle (sqlite*), the backup will continue progressing as expected. Other than that, it can cause the backup process to restart. From docs:

...If the source database is not an in-memory database, and the write is performed from within the same process as the backup operation and uses the same database handle (pDb), then the destination database (the one opened using connection pFile) is automatically updated along with the source.

Writes to an in-memory source database, or writes to a file-based source database by an external process or thread using a database connection other than pDb are significantly more expensive than writes made to a file-based source database using pDb (as the entire backup operation must be restarted...)

[geeksilva97]

This comment mentions that DatabaseSync needs to have only sync operations (makes sense).

The backup can be performed synchronously but I don't think it should be like that.

I wonder if indeed this PR would be more suitable for async API.

Any thoughts?

Thanks in advance

[codecov]

Codecov Report

Attention: Patch coverage is 80.66038% with 41 lines in your changes missing coverage. Please review.

Project coverage is 89.18%. Comparing base (c4fb331) to head (4c4d73b).
Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
src/node_sqlite.cc	80.66%	17 Missing and 24 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #56253      +/-   ##
==========================================
- Coverage   89.21%   89.18%   -0.03%     
==========================================
  Files         665      665              
  Lines      192514   192755     +241     
  Branches    37040    37084      +44     
==========================================
+ Hits       171744   171909     +165     
- Misses      13613    13648      +35     
- Partials     7157     7198      +41     

Files with missing lines	Coverage Δ
src/node_sqlite.h	70.00% <ø> (ø)
src/node_sqlite.cc	79.97% <80.66%> (+0.12%)	⬆️

... and 38 files with indirect coverage changes

[geeksilva97]

There were unrelated failures, but there were also the same related Windows failures - https://ci.nodejs.org/job/node-test-binary-windows-js-suites/32476/#showFailuresLink.

😔 Thanks. Yeah, that will be hard to check.

Yeah, I have no idea what to try. I asked friends of mine to run this test on their machines with non-administrator permissions (Windows 11) and it worked.

[cjihrig]

Maybe worth pinging @nodejs/platform-windows for help.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/65005/

[geeksilva97]

Looks like Windows passed 🎉

https://ci.nodejs.org/job/node-test-binary-windows-js-suites/32505/

(other failed but seems unrelated)

Thanks a lot, @cjihrig @H4ad

[H4ad]

Great work, thanks for the contribution o/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/65013/

[nodejs-github-bot]

Commit Queue failed

- Loading data for nodejs/node/pull/56253
✔  Done loading data for nodejs/node/pull/56253
----------------------------------- PR info ------------------------------------
Title      sqlite: expose backup api (#56253)
Author     Edy Silva <[email protected]> (@geeksilva97)
Branch     geeksilva97:sqlite-backup -> nodejs:main
Labels     c++, author ready, sqlite
Commits    19
 - sqlite, test: expose sqlite online backup api
 - sqlite: resolve backup promise with total transferred pages
 - sqlite,test: test DatabaseSync.prototype.backup interface
 - sqlite: pass object to progress function instead of two args
 - fixup: improve tests
 - sqlite: update interface
 - doc: add backup docs
 - test: use temp dir to test invalid paths
 - doc: fixup: fix typos
 - fixup: improve formatting
 - fixup: centralize backup error handling
 - fixup: fix new env properties order
 - fixup: await rejects assertion
 - sqlite,test,doc: move backup implementation to a separate function
 - sqlite,doc: add more details about concurrent edits
 - sqlite,doc: use async/await in backup code example
 - sqlite: use proper functions to open/close database
 - test: close databases after test
 - sqlite: fix backup finalization when promise resolves
Committers 1
 - Edy Silva <[email protected]>
PR-URL: https://github.com/nodejs/node/pull/56253
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/56253
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
--------------------------------------------------------------------------------
   ℹ  This PR was created on Sat, 14 Dec 2024 01:36:33 GMT
   ✔  Approvals: 2
   ✔  - Colin Ihrig (@cjihrig): https://github.com/nodejs/node/pull/56253#pullrequestreview-2596944274
   ✔  - Vinícius Lourenço Claro Cardoso (@H4ad): https://github.com/nodejs/node/pull/56253#pullrequestreview-2596917775
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2025-02-05T20:48:52Z: https://ci.nodejs.org/job/node-test-pull-request/65013/
- Querying data for job/node-test-pull-request/65013/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  No git cherry-pick in progress
   ✔  No git am in progress
   ✔  No git rebase in progress
--------------------------------------------------------------------------------
- Bringing origin/main up to date...
From https://github.com/nodejs/node
 * branch                  main       -> FETCH_HEAD
✔  origin/main is now up-to-date
- Downloading patch for 56253
From https://github.com/nodejs/node
 * branch                  refs/pull/56253/merge -> FETCH_HEAD
✔  Fetched commits as 436723282d64..4c4d73b810fe
--------------------------------------------------------------------------------
Auto-merging src/node_sqlite.cc
Auto-merging src/node_sqlite.h
[main e6561e9bdd] sqlite, test: expose sqlite online backup api
 Author: Edy Silva <[email protected]>
 Date: Sun Jan 12 21:56:43 2025 -0300
 4 files changed, 461 insertions(+)
 create mode 100644 test/parallel/test-sqlite-backup.mjs
Auto-merging src/node_sqlite.cc
[main de3853950a] sqlite: resolve backup promise with total transferred pages
 Author: Edy Silva <[email protected]>
 Date: Sun Jan 12 23:49:34 2025 -0300
 1 file changed, 9 insertions(+), 7 deletions(-)
[main 27c17ed52d] sqlite,test: test DatabaseSync.prototype.backup interface
 Author: Edy Silva <[email protected]>
 Date: Mon Jan 13 00:01:06 2025 -0300
 1 file changed, 88 insertions(+), 10 deletions(-)
Auto-merging src/node_sqlite.cc
[main 5a6196bb7e] sqlite: pass object to progress function instead of two args
 Author: Edy Silva <[email protected]>
 Date: Tue Jan 14 12:32:56 2025 -0300
 3 files changed, 43 insertions(+), 8 deletions(-)
[main 83fa046d6b] fixup: improve tests
 Author: Edy Silva <[email protected]>
 Date: Tue Jan 14 14:05:16 2025 -0300
 1 file changed, 46 insertions(+), 58 deletions(-)
Auto-merging src/node_sqlite.cc
[main 0a61c96ed9] sqlite: update interface
 Author: Edy Silva <[email protected]>
 Date: Tue Jan 14 14:20:12 2025 -0300
 3 files changed, 15 insertions(+), 17 deletions(-)
[main 3636bebe9b] doc: add backup docs
 Author: Edy Silva <[email protected]>
 Date: Tue Jan 14 15:19:57 2025 -0300
 1 file changed, 23 insertions(+)
[main d8f89eae40] test: use temp dir to test invalid paths
 Author: Edy Silva <[email protected]>
 Date: Tue Jan 14 15:24:39 2025 -0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[main 7b218e7c10] doc: fixup: fix typos
 Author: Edy Silva <[email protected]>
 Date: Wed Jan 15 20:54:56 2025 -0300
 1 file changed, 3 insertions(+), 3 deletions(-)
Auto-merging src/node_sqlite.cc
[main a7534a16f1] fixup: improve formatting
 Author: Edy Silva <[email protected]>
 Date: Wed Jan 15 21:25:59 2025 -0300
 1 file changed, 44 insertions(+), 71 deletions(-)
Auto-merging src/node_sqlite.cc
[main b9f65e3a8f] fixup: centralize backup error handling
 Author: Edy Silva <[email protected]>
 Date: Wed Jan 15 21:59:47 2025 -0300
 3 files changed, 34 insertions(+), 45 deletions(-)
[main c4967daff8] fixup: fix new env properties order
 Author: Edy Silva <[email protected]>
 Date: Mon Jan 20 14:12:42 2025 -0300
 1 file changed, 3 insertions(+), 3 deletions(-)
[main 5560c83169] fixup: await rejects assertion
 Author: Edy Silva <[email protected]>
 Date: Mon Jan 20 15:02:09 2025 -0300
 1 file changed, 5 insertions(+), 5 deletions(-)
Auto-merging src/node_sqlite.cc
Auto-merging src/node_sqlite.h
[main 33727815e2] sqlite,test,doc: move backup implementation to a separate function
 Author: Edy Silva <[email protected]>
 Date: Fri Jan 24 12:20:35 2025 -0300
 6 files changed, 177 insertions(+), 149 deletions(-)
[main d1c3534575] sqlite,doc: add more details about concurrent edits
 Author: Edy Silva <[email protected]>
 Date: Fri Jan 31 13:58:30 2025 -0300
 1 file changed, 40 insertions(+), 2 deletions(-)
[main 5cecd76b35] sqlite,doc: use async/await in backup code example
 Author: Edy Silva <[email protected]>
 Date: Mon Feb 3 13:14:34 2025 -0300
 1 file changed, 14 insertions(+), 16 deletions(-)
Auto-merging src/node_sqlite.cc
[main dd2b54d22d] sqlite: use proper functions to open/close database
 Author: Edy Silva <[email protected]>
 Date: Mon Feb 3 18:07:10 2025 -0300
 1 file changed, 5 insertions(+), 2 deletions(-)
[main c680940ca9] test: close databases after test
 Author: Edy Silva <[email protected]>
 Date: Tue Feb 4 14:39:55 2025 -0300
 1 file changed, 11 insertions(+), 1 deletion(-)
Auto-merging src/node_sqlite.cc
[main dd8154e385] sqlite: fix backup finalization when promise resolves
 Author: Edy Silva <[email protected]>
 Date: Tue Feb 4 23:57:54 2025 -0300
 1 file changed, 3 insertions(+), 1 deletion(-)
   ✔  Patches applied
There are 19 commits in the PR. Attempting autorebase.
Rebasing (2/38)
Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
sqlite, test: expose sqlite online backup api
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD e55f86814f] sqlite, test: expose sqlite online backup api

Author: Edy Silva <[email protected]>

Date: Sun Jan 12 21:56:43 2025 -0300

4 files changed, 461 insertions(+)

create mode 100644 test/parallel/test-sqlite-backup.mjs

Rebasing (3/38)

Rebasing (4/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

sqlite: resolve backup promise with total transferred pages
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 7f7acf7ef5] sqlite: resolve backup promise with total transferred pages

Author: Edy Silva <[email protected]>

Date: Sun Jan 12 23:49:34 2025 -0300

1 file changed, 9 insertions(+), 7 deletions(-)

Rebasing (5/38)

Rebasing (6/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

sqlite,test: test DatabaseSync.prototype.backup interface
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 1250de3591] sqlite,test: test DatabaseSync.prototype.backup interface

Author: Edy Silva <[email protected]>

Date: Mon Jan 13 00:01:06 2025 -0300

1 file changed, 88 insertions(+), 10 deletions(-)

Rebasing (7/38)

Rebasing (8/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

sqlite: pass object to progress function instead of two args
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 2ba060df61] sqlite: pass object to progress function instead of two args

Author: Edy Silva <[email protected]>

Date: Tue Jan 14 12:32:56 2025 -0300

3 files changed, 43 insertions(+), 8 deletions(-)

Rebasing (9/38)

Rebasing (10/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

fixup: improve tests
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 17aba32f03] fixup: improve tests

Author: Edy Silva <[email protected]>

Date: Tue Jan 14 14:05:16 2025 -0300

1 file changed, 46 insertions(+), 58 deletions(-)

Rebasing (11/38)

Rebasing (12/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

sqlite: update interface
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 3b6a42a46b] sqlite: update interface

Author: Edy Silva <[email protected]>

Date: Tue Jan 14 14:20:12 2025 -0300

3 files changed, 15 insertions(+), 17 deletions(-)

Rebasing (13/38)

Rebasing (14/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

doc: add backup docs
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD ce69f6f1b2] doc: add backup docs

Author: Edy Silva <[email protected]>

Date: Tue Jan 14 15:19:57 2025 -0300

1 file changed, 23 insertions(+)

Rebasing (15/38)

Rebasing (16/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

test: use temp dir to test invalid paths
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD f45a2977eb] test: use temp dir to test invalid paths

Author: Edy Silva <[email protected]>

Date: Tue Jan 14 15:24:39 2025 -0300

1 file changed, 1 insertion(+), 1 deletion(-)

Rebasing (17/38)

Rebasing (18/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

doc: fixup: fix typos
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 45b5af7b9c] doc: fixup: fix typos

Author: Edy Silva <[email protected]>

Date: Wed Jan 15 20:54:56 2025 -0300

1 file changed, 3 insertions(+), 3 deletions(-)

Rebasing (19/38)

Rebasing (20/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

fixup: improve formatting
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 087cfa7a2b] fixup: improve formatting

Author: Edy Silva <[email protected]>

Date: Wed Jan 15 21:25:59 2025 -0300

1 file changed, 44 insertions(+), 71 deletions(-)

Rebasing (21/38)

Rebasing (22/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

fixup: centralize backup error handling
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 9399f5787e] fixup: centralize backup error handling

Author: Edy Silva <[email protected]>

Date: Wed Jan 15 21:59:47 2025 -0300

3 files changed, 34 insertions(+), 45 deletions(-)

Rebasing (23/38)

Rebasing (24/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

fixup: fix new env properties order
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 8072c5d392] fixup: fix new env properties order

Author: Edy Silva <[email protected]>

Date: Mon Jan 20 14:12:42 2025 -0300

1 file changed, 3 insertions(+), 3 deletions(-)

Rebasing (25/38)

Rebasing (26/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

fixup: await rejects assertion
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD f50c037adf] fixup: await rejects assertion

Author: Edy Silva <[email protected]>

Date: Mon Jan 20 15:02:09 2025 -0300

1 file changed, 5 insertions(+), 5 deletions(-)

Rebasing (27/38)

Rebasing (28/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

sqlite,test,doc: move backup implementation to a separate function
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD ba356a16b0] sqlite,test,doc: move backup implementation to a separate function

Author: Edy Silva <[email protected]>

Date: Fri Jan 24 12:20:35 2025 -0300

6 files changed, 177 insertions(+), 149 deletions(-)

Rebasing (29/38)

Rebasing (30/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

sqlite,doc: add more details about concurrent edits
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 3d527f3bd9] sqlite,doc: add more details about concurrent edits

Author: Edy Silva <[email protected]>

Date: Fri Jan 31 13:58:30 2025 -0300

1 file changed, 40 insertions(+), 2 deletions(-)

Rebasing (31/38)

Rebasing (32/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

sqlite,doc: use async/await in backup code example
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD e8ef7aa6d5] sqlite,doc: use async/await in backup code example

Author: Edy Silva <[email protected]>

Date: Mon Feb 3 13:14:34 2025 -0300

1 file changed, 14 insertions(+), 16 deletions(-)

Rebasing (33/38)

Rebasing (34/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

sqlite: use proper functions to open/close database
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 632cfa60e3] sqlite: use proper functions to open/close database

Author: Edy Silva <[email protected]>

Date: Mon Feb 3 18:07:10 2025 -0300

1 file changed, 5 insertions(+), 2 deletions(-)

Rebasing (35/38)

Rebasing (36/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

test: close databases after test
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD 108574fa0b] test: close databases after test

Author: Edy Silva <[email protected]>

Date: Tue Feb 4 14:39:55 2025 -0300

1 file changed, 11 insertions(+), 1 deletion(-)

Rebasing (37/38)

Rebasing (38/38)

Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

sqlite: fix backup finalization when promise resolves
PR-URL: #56253

Reviewed-By: Colin Ihrig <[email protected]>

Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
[detached HEAD c7edd0809a] sqlite: fix backup finalization when promise resolves

Author: Edy Silva <[email protected]>

Date: Tue Feb 4 23:57:54 2025 -0300

1 file changed, 3 insertions(+), 1 deletion(-)

Successfully rebased and updated refs/heads/main.
ℹ  Add commit-queue-squash label to land the PR as one commit, or commit-queue-rebase to land as separate commits.

https://github.com/nodejs/node/actions/runs/13167245589

[nodejs-github-bot]

Landed in 16dc29d