Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test failed in CI: db-dev::test-db-dev test_db_run #7429

Open
jgallagher opened this issue Jan 29, 2025 · 0 comments
Open

test failed in CI: db-dev::test-db-dev test_db_run #7429

jgallagher opened this issue Jan 29, 2025 · 0 comments
Labels
Test Flake Tests that work. Wait, no. Actually yes. Hang on. Something is broken.

Comments

@jgallagher
Copy link
Contributor

This test failed on a CI run on PR #7428:

https://github.com/oxidecomputer/omicron/pull/7428/checks?check_run_id=36364357016

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JJSCVHZC6J3VQW664BHDCJMJ/3H7s0iynm0IsLFeoEjnGZrLNLUsKIxrfw6DW1eaibQg4GbKo/01JJSCVWF6SBW3EVQ27VPWN5SK

Excerpt from the log showing the failure:

5438	2025-01-29T17:26:22.799Z	subproc stdout: WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
5439	2025-01-29T17:26:22.807Z	subproc stdout: WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpQowpZL"
5440	2025-01-29T17:26:22.807Z	subproc stdout: 	If you would like to access the database for debugging, run the following:
5441	2025-01-29T17:26:22.807Z	subproc stdout:
5442	2025-01-29T17:26:22.807Z	subproc stdout: 	# Run the database
5443	2025-01-29T17:26:22.807Z	subproc stdout: 	cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpQowpZL/data"
5444	2025-01-29T17:26:22.807Z	subproc stdout: 	# Access the database. Note the port may change if you run multiple databases.
5445	2025-01-29T17:26:22.807Z	subproc stdout: 	cockroach sql --host=localhost:32221 --insecure
5446	2025-01-29T17:26:22.807Z	subproc stdout: Error: populating database
5447	2025-01-29T17:26:22.807Z	subproc stdout:
5448	2025-01-29T17:26:22.808Z	subproc stdout: Caused by:
5449	2025-01-29T17:26:22.808Z	subproc stdout:     0: populate
5450	2025-01-29T17:26:22.808Z	subproc stdout:     1: populating Omicron database
5451	2025-01-29T17:26:22.808Z	subproc stdout:     2: db error: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 102014976 currently allocated, 134217728 bytes in budget
5452	2025-01-29T17:26:22.808Z	subproc stdout:        HINT: Consider increasing --max-sql-memory startup parameter.
5453	2025-01-29T17:26:22.808Z	subproc stdout:        --
5454	2025-01-29T17:26:22.808Z	subproc stdout:        Try setting a higher --max-sql-memory.
5455	2025-01-29T17:26:22.808Z	subproc stdout:        --
5456	2025-01-29T17:26:22.808Z	subproc stdout:        Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
5457	2025-01-29T17:26:22.808Z	subproc stdout:        Manual inspection may be required to determine the actual state of the database.
5458	2025-01-29T17:26:22.808Z	subproc stdout:        --
5459	2025-01-29T17:26:22.808Z	subproc stdout:        See: https://github.com/cockroachdb/cockroach/issues/42061
5460	2025-01-29T17:26:22.808Z	subproc stdout:     3: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 102014976 currently allocated, 134217728 bytes in budget
5461	2025-01-29T17:26:22.808Z	subproc stdout:        HINT: Consider increasing --max-sql-memory startup parameter.
5462	2025-01-29T17:26:22.808Z	subproc stdout:        --
5463	2025-01-29T17:26:22.808Z	subproc stdout:        Try setting a higher --max-sql-memory.
5464	2025-01-29T17:26:22.808Z	subproc stdout:        --
5465	2025-01-29T17:26:22.808Z	subproc stdout:        Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
5466	2025-01-29T17:26:22.808Z	subproc stdout:        Manual inspection may be required to determine the actual state of the database.
5467	2025-01-29T17:26:22.808Z	subproc stdout:        --
5468	2025-01-29T17:26:22.808Z	subproc stdout:        See: https://github.com/cockroachdb/cockroach/issues/42061

The failure mode here looks like #7141 (but that's a different DB test so figured I'd open this for ease of searching if this recurs).

@jgallagher jgallagher added the Test Flake Tests that work. Wait, no. Actually yes. Hang on. Something is broken. label Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Test Flake Tests that work. Wait, no. Actually yes. Hang on. Something is broken.
Projects
None yet
Development

No branches or pull requests

1 participant