Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flaky/data-race test TestPiTRAndFailureRestore/TestOperator/TestFailure #59094

Open
Tracked by #41316
D3Hunter opened this issue Jan 21, 2025 · 0 comments
Open
Tracked by #41316

flaky/data-race test TestPiTRAndFailureRestore/TestOperator/TestFailure #59094

D3Hunter opened this issue Jan 21, 2025 · 0 comments
Labels
component/br This issue is related to BR of TiDB. type/bug The issue is confirmed as a bug.

Comments

@D3Hunter
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

see https://do.pingcap.net/jenkins/blue/rest/organizations/jenkins/pipelines/pingcap/pipelines/tidb/pipelines/ghpr_check2/runs/26281/nodes/103/steps/575/log/?start=0
for TestOperator/TestFailure

=== RUN   TestOperator
    operator_test.go:80: 
        	Error Trace:	tests/realtikvtest/brietest/operator_test.go:80
        	            				tests/realtikvtest/brietest/operator_test.go:195
        	Error:      	the service gc safepoint exists
        	Test:       	TestOperator
        	Messages:   	it is struct { ServiceID string "json:\"service_id\""; ExpiredAt int64 "json:\"expired_at\""; SafePoint int64 "json:\"safe_point\"" }{ServiceID:"backup-stream-TestPiTRAndBackupInSQL-1", ExpiredAt:1737450741, SafePoint:455460398471053328}
--- FAIL: TestOperator (0.00s)
=== RUN   TestFailure
    operator_test.go:80: 
        	Error Trace:	tests/realtikvtest/brietest/operator_test.go:80
        	            				tests/realtikvtest/brietest/operator_test.go:248
        	Error:      	the service gc safepoint exists
        	Test:       	TestFailure
        	Messages:   	it is struct { ServiceID string "json:\"service_id\""; ExpiredAt int64 "json:\"expired_at\""; SafePoint int64 "json:\"safe_point\"" }{ServiceID:"backup-stream-TestPiTRAndBackupInSQL-1", ExpiredAt:1737450741, SafePoint:455460398471053328}
--- FAIL: TestFailure (0.00s)

for TestPiTRAndFailureRestore

WARNING: DATA RACE
Read at 0x00c056d155b0 by goroutine 243296:
  github.com/pingcap/tidb/pkg/domain.(*schemaValidator).IsLeaseExpired()
      pkg/domain/schema_validator.go:175 +0x4b
  github.com/pingcap/tidb/pkg/domain.(*Domain).IsLeaseExpired()
      pkg/domain/domain.go:1568 +0x6f
  github.com/pingcap/tidb/br/pkg/task.waitUntilSchemaReload.func1()
      br/pkg/task/stream.go:2065 +0x74
  github.com/pingcap/tidb/br/pkg/utils.WaitUntil()
      br/pkg/utils/wait.go:26 +0x67
  github.com/pingcap/tidb/br/pkg/task.waitUntilSchemaReload()
      br/pkg/task/stream.go:2067 +0xe4
  github.com/pingcap/tidb/br/pkg/task.restoreStream()
      br/pkg/task/stream.go:1644 +0x323a
  github.com/pingcap/tidb/br/pkg/task.RunStreamRestore()
      br/pkg/task/stream.go:1352 +0x163a
  github.com/pingcap/tidb/br/pkg/task.RunRestore()
      br/pkg/task/restore.go:722 +0x606
  tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore.(*LogBackupKit).RunStreamRestore.func11()
      tests/realtikvtest/brietest/pitr_test.go:190 +0x37e
  tests/realtikvtest/brietest/brietest_test.(*LogBackupKit).runAndCheck()
      tests/realtikvtest/brietest/pitr_test.go:279 +0x9e
  tests/realtikvtest/brietest/brietest_test.(*LogBackupKit).RunStreamRestore()
      tests/realtikvtest/brietest/pitr_test.go:181 +0x5d4
  tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore()
      tests/realtikvtest/brietest/pitr_test.go:605 +0x50a
  github.com/pingcap/failpoint.parseTerm()
      external/com_github_pingcap_failpoint/terms.go:149 +0x233
  github.com/pingcap/failpoint.parse()
      external/com_github_pingcap_failpoint/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      external/com_github_pingcap_failpoint/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).EnableCall()
      external/com_github_pingcap_failpoint/failpoint.go:94 +0x124
  github.com/pingcap/failpoint.(*Failpoints).EnableCall()
      external/com_github_pingcap_failpoint/failpoints.go:150 +0x296
  github.com/pingcap/failpoint.EnableCall()
      external/com_github_pingcap_failpoint/failpoints.go:270 +0x31b
  tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore()
      tests/realtikvtest/brietest/pitr_test.go:590 +0x31c
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1333 +0x12e
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func619()
      pkg/sessionctx/variable/sysvar.go:3309 +0x52
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).Validate()
      pkg/sessionctx/variable/variable.go:317 +0x12b
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1330 +0xd2
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).setSysVariable()
      pkg/executor/set.go:155 +0x69d
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).Next()
      pkg/executor/set.go:105 +0xee6
  github.com/pingcap/tidb/pkg/executor/internal/exec.Next()
      pkg/executor/internal/exec/executor.go:460 +0x3a8
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next()
      pkg/executor/adapter.go:1268 +0x84
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor()
      pkg/executor/adapter.go:1017 +0x624
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay()
      pkg/executor/adapter.go:850 +0x3e7
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec()
      pkg/executor/adapter.go:613 +0x180c
  github.com/pingcap/tidb/pkg/session.runStmt()
      pkg/session/session.go:2290 +0x62b
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2152 +0x1d09
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext()
      pkg/testkit/testkit.go:425 +0x1030
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext()
      pkg/testkit/testkit.go:159 +0xab
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec()
      pkg/testkit/testkit.go:154 +0x117
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).RefreshSession()
      pkg/testkit/testkit.go:126 +0x3b2
  github.com/pingcap/tidb/pkg/testkit.NewTestKit()
      pkg/testkit/testkit.go:86 +0x4dd
  tests/realtikvtest/brietest/brietest_test.initTestKit()
      tests/realtikvtest/brietest/backup_restore_test.go:42 +0x164
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1342 +0x304
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).setSysVariable()
      pkg/executor/set.go:155 +0x69d
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).Next()
      pkg/executor/set.go:105 +0xee6
  github.com/pingcap/tidb/pkg/executor/internal/exec.Next()
      pkg/executor/internal/exec/executor.go:460 +0x3a8
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next()
      pkg/executor/adapter.go:1268 +0x84
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor()
      pkg/executor/adapter.go:1017 +0x624
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay()
      pkg/executor/adapter.go:850 +0x3e7
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec()
      pkg/executor/adapter.go:613 +0x180c
  github.com/pingcap/tidb/pkg/session.runStmt()
      pkg/session/session.go:2290 +0x62b
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2152 +0x1d09
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext()
      pkg/testkit/testkit.go:425 +0x1030
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext()
      pkg/testkit/testkit.go:159 +0xab
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec()
      pkg/testkit/testkit.go:154 +0x117
  github.com/pingcap/tidb/tests/realtikvtest.CreateMockStoreAndDomainAndSetup()
      tests/realtikvtest/testkit.go:146 +0x464
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func217()
      pkg/sessionctx/variable/sysvar.go:1147 +0x52
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).ValidateWithRelaxedValidation()
      pkg/sessionctx/variable/variable.go:377 +0x242
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:142 +0xa24
  fmt.Sscanf()
      GOROOT/src/fmt/scan.go:114 +0x18e
  github.com/pingcap/tidb/pkg/sessionctx/variable.parseByteSize()
      pkg/sessionctx/variable/varsutil.go:406 +0x1d
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func261()
      pkg/sessionctx/variable/sysvar.go:1418 +0x44
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:143 +0xa94
  github.com/pingcap/tidb/pkg/sessionctx/variable.parseSchemaCacheSize()
      pkg/sessionctx/variable/varsutil.go:638 +0x17c
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func620()
      pkg/sessionctx/variable/sysvar.go:3317 +0x67
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:143 +0xa94
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func619()
      pkg/sessionctx/variable/sysvar.go:3309 +0x52
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).ValidateWithRelaxedValidation()
      pkg/sessionctx/variable/variable.go:377 +0x242
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:142 +0xa24
  github.com/pingcap/tidb/pkg/domain.(*Domain).LoadSysVarCacheLoop()
      pkg/domain/domain.go:1961 +0x93
  github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl()
      pkg/session/session.go:3538 +0x864
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func218()
      pkg/sessionctx/variable/sysvar.go:1154 +0x56
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:143 +0xa94
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func217()
      pkg/sessionctx/variable/sysvar.go:1147 +0x52
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).ValidateWithRelaxedValidation()
      pkg/sessionctx/variable/variable.go:377 +0x242
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:142 +0xa24
  fmt.Sscanf()
      GOROOT/src/fmt/scan.go:114 +0x18e
  github.com/pingcap/tidb/pkg/sessionctx/variable.parseByteSize()
      pkg/sessionctx/variable/varsutil.go:406 +0x1d
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func261()
      pkg/sessionctx/variable/sysvar.go:1418 +0x44
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:143 +0xa94
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCacheIfNeeded()
      pkg/domain/sysvar_cache.go:50 +0x1bb
  github.com/pingcap/tidb/pkg/domain.(*Domain).GetSessionCache()
      pkg/domain/sysvar_cache.go:61 +0x4a
  github.com/pingcap/tidb/pkg/session.(*session).loadCommonGlobalVariablesIfNeeded()
      pkg/session/session.go:3949 +0x2bc
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2014 +0x17a
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteInternal()
      pkg/session/session.go:1525 +0x3af
  github.com/pingcap/tidb/pkg/domain.(*Domain).LoadPrivilegeLoop()
      pkg/domain/domain.go:1901 +0x102
  github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl()
      pkg/session/session.go:3531 +0x807
  github.com/pingcap/tidb/pkg/session.BootstrapSession()
      pkg/session/session.go:3407 +0x293
  github.com/pingcap/tidb/tests/realtikvtest.CreateMockStoreAndDomainAndSetup()
      tests/realtikvtest/testkit.go:136 +0x269
  github.com/pingcap/tidb/tests/realtikvtest.CreateMockStoreAndSetup()
      tests/realtikvtest/testkit.go:109 +0xa4
  tests/realtikvtest/brietest/brietest_test.initTestKit()
      tests/realtikvtest/brietest/backup_restore_test.go:35 +0x90
  tests/realtikvtest/brietest/brietest_test.NewLogBackupKit()
      tests/realtikvtest/brietest/pitr_test.go:136 +0x3b
  tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore()
      tests/realtikvtest/brietest/pitr_test.go:573 +0x3e
  testing.tRunner()
      GOROOT/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1743 +0x44

Previous write at 0x00c056d155b0 by goroutine 243602:
  github.com/pingcap/tidb/pkg/domain.(*schemaValidator).Update()
      pkg/domain/schema_validator.go:147 +0x24e
  github.com/pingcap/tidb/pkg/domain.(*Domain).Reload()
      pkg/domain/domain.go:853 +0xe73
  github.com/pingcap/tidb/pkg/domain.(*Domain).loadSchemaInLoop()
      pkg/domain/domain.go:1149 +0x324
  github.com/pingcap/tidb/pkg/domain.(*Domain).Start.func2()
      pkg/domain/domain.go:1534 +0x52
  github.com/pingcap/tidb/pkg/util.(*WaitGroupEnhancedWrapper).Run.func1()
      pkg/util/wait_group_wrapper.go:103 +0xba

and

        	Error Trace:	pkg/testkit/testkit.go:241
        	            				pkg/testkit/testkit.go:176
        	            				tests/realtikvtest/brietest/pitr_test.go:608
        	Error:      	Received unexpected error:
        	            	[schema:1146]Table 'test.TestPiTRAndFailureRestore' doesn't exist
        	            	github.com/pingcap/errors.AddStack
        	            		external/com_github_pingcap_errors/errors.go:178
        	            	github.com/pingcap/errors.Trace
        	            		external/com_github_pingcap_errors/juju_adaptor.go:15
        	            	github.com/pingcap/tidb/pkg/planner/core.Preprocess
        	            		pkg/planner/core/preprocess.go:152
        	            	github.com/pingcap/tidb/pkg/executor.(*Compiler).Compile
        	            		pkg/executor/compiler.go:69
        	            	github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt
        	            		pkg/session/session.go:2100
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext
        	            		pkg/testkit/testkit.go:425
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQueryWithContext
        	            		pkg/testkit/testkit.go:240
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQuery
        	            		pkg/testkit/testkit.go:176
        	            	tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore
        	            		tests/realtikvtest/brietest/pitr_test.go:608
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1690
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1700
        	Test:       	TestPiTRAndFailureRestore
        	Messages:   	sql:SELECT COUNT(*) FROM test.TestPiTRAndFailureRestore, args:[]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)
@D3Hunter D3Hunter added component/br This issue is related to BR of TiDB. type/bug The issue is confirmed as a bug. labels Jan 21, 2025
@hawkingrei hawkingrei mentioned this issue Jan 21, 2025
Open
@Leavrth Leavrth mentioned this issue Jan 21, 2025
Open
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/br This issue is related to BR of TiDB. type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@D3Hunter