[Bug]: [ERROR] [datanode/data_node.go:199] ["Data Node disconnected from etcd, process will exit"]

[daiyungang]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.4.20
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 64/256
- GPU: 
- Others:

Current Behavior

When we have written one hundred million 768-dimensional vectors into Milvus, the Milvus process restarts under the circumstances of inactivity and low concurrency. Additionally, when the restarting process is loading all the collections, it will also lead to the restart of Milvus.

The disk information is as follows.

Expected Behavior

I hope that the Milvus collection can be loaded normally and vector search can be performed under the current data volume.

Steps To Reproduce

1.Deploy milvus in a standalone manner on an ARM architecture machine.
2.Construct one hundred million pieces of 768-dimensional data, with fields including id (int64), cameaId (VarChar(64)), sourceType (int64), generationTime (VarChar(64)), and feature. Among them, id is the primary key, and feature is the vector field, and the vector index is IVF_SQ8.
3.Perform low-concurrency searches with a concurrency number of 6, searching through 6 different collections.

Milvus Log

[2025/01/16 07:13:13.706 +00:00] [WARN] [rootcoord/root_coord.go:1727] ["failed to updateTimeTick"] [traceID=0ffe069c1aad452534feb09ca35ec1df] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 17"]
[2025/01/16 07:13:13.707 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 17"]
[2025/01/16 07:13:13.749 +00:00] [INFO] [datacoord/channel_manager_v2.go:495] ["Reassign standby channels to node"] [nodeID=-9223372036854775808] [channels="[by-dev-rootcoord-dml_0_454173931585144084v0,by-dev-rootcoord-dml_9_454173931610303201v0,by-dev-rootcoord-dml_2_454173931585549421v0,by-dev-rootcoord-dml_11_454173931610437806v0,by-dev-rootcoord-dml_8_454173931605100952v0,by-dev-rootcoord-dml_5_454173931585549577v0,by-dev-rootcoord-dml_12_455138126761615208v0,by-dev-rootcoord-dml_7_454173931585573850v0,by-dev-rootcoord-dml_6_454173931585550579v0,by-dev-rootcoord-dml_4_455138126753165543v0]"]
[2025/01/16 07:13:13.761 +00:00] [INFO] [datacoord/task_scheduler.go:204] ["task scheduler"] ["task num"=5141]
[2025/01/16 07:13:13.761 +00:00] [INFO] [datacoord/task_scheduler.go:235] ["task is processing"] [taskID=455138126820963583] [state=JobStateInit]
[2025/01/16 07:13:13.762 +00:00] [WARN] [datacoord/task_index.go:205] ["failed to get dim from field type params"] ["field type"=Int64] [error="dim not found in params"] [errorVerbose="dim not found in params\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n | \t/workspace/source/internal/storage/utils.go:233\n | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n | \t/workspace/source/internal/datacoord/task_index.go:203\n | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n | \t/workspace/source/internal/datacoord/task_scheduler.go:244\n | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run\n | \t/workspace/source/internal/datacoord/task_scheduler.go:211\n | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).schedule\n | \t/workspace/source/internal/datacoord/task_scheduler.go:183\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"]
[2025/01/16 07:13:13.762 +00:00] [INFO] [datacoord/index_engine_version_manager.go:74] ["index versions is empty"]
[2025/01/16 07:13:13.762 +00:00] [INFO] [datacoord/task_index.go:302] ["index task pre check successfully"] [taskID=455138126820963583]
[2025/01/16 07:13:13.762 +00:00] [INFO] [datacoord/task_scheduler.go:214] ["there is no idle indexing node, waiting for retry..."]
[2025/01/16 07:13:13.906 +00:00] [WARN] [rootcoord/root_coord.go:1727] ["failed to updateTimeTick"] [traceID=186213d282e10ad3b5750d27c710ca81] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 17"]
[2025/01/16 07:13:13.906 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 17"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [retry/retry.go:104] ["retry func failed, reach max retry"] [attempt=3]
[2025/01/16 07:13:13.921 +00:00] [WARN] [retry/retry.go:104] ["retry func failed, reach max retry"] [attempt=3]
[2025/01/16 07:13:13.921 +00:00] [WARN] [retry/retry.go:104] ["retry func failed, reach max retry"] [attempt=3]
[2025/01/16 07:13:13.921 +00:00] [WARN] [sessionutil/session_util.go:579] ["fail to retry keepAliveOnce"] [serverName=indexcoord] [LeaseID=7587884147010139249] [error="etcdserver: requested lease not found"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [retry/retry.go:104] ["retry func failed, reach max retry"] [attempt=3]
[2025/01/16 07:13:13.921 +00:00] [WARN] [sessionutil/session_util.go:579] ["fail to retry keepAliveOnce"] [serverName=proxy] [LeaseID=7587884147010139370] [error="etcdserver: requested lease not found"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [sessionutil/session_util.go:908] ["connection lost detected, shuting down"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [retry/retry.go:104] ["retry func failed, reach max retry"] [attempt=3]
[2025/01/16 07:13:13.921 +00:00] [WARN] [sessionutil/session_util.go:579] ["fail to retry keepAliveOnce"] [serverName=rootcoord] [LeaseID=7587884147010139203] [error="etcdserver: requested lease not found"]
[2025/01/16 07:13:13.921 +00:00] [ERROR] [proxy/proxy.go:177] ["Proxy disconnected from etcd, process will exit"] ["Server Id"=17] [stack="github.com/milvus-io/milvus/internal/proxy.(*Proxy).Register.func1\n\t/workspace/source/internal/proxy/proxy.go:177"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [sessionutil/session_util.go:908] ["connection lost detected, shuting down"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [sessionutil/session_util.go:579] ["fail to retry keepAliveOnce"] [serverName=datanode] [LeaseID=7587884147010139386] [error="etcdserver: requested lease not found"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [retry/retry.go:104] ["retry func failed, reach max retry"] [attempt=3]
[2025/01/16 07:13:13.921 +00:00] [WARN] [sessionutil/session_util.go:908] ["connection lost detected, shuting down"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [sessionutil/session_util.go:579] ["fail to retry keepAliveOnce"] [serverName=querynode] [LeaseID=7587884147010139403] [error="etcdserver: requested lease not found"]
[2025/01/16 07:13:13.921 +00:00] [ERROR] [datanode/data_node.go:199] ["Data Node disconnected from etcd, process will exit"] ["Server Id"=17] [stack="github.com/milvus-io/milvus/internal/datanode.(*DataNode).Register.func1\n\t/workspace/source/internal/datanode/data_node.go:199"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [sessionutil/session_util.go:908] ["connection lost detected, shuting down"]
[2025/01/16 07:13:13.921 +00:00] [WARN] [retry/retry.go:104] ["retry func failed, reach max retry"] [attempt=3]
[2025/01/16 07:13:13.921 +00:00] [ERROR] [querynodev2/server.go:173] ["Query Node disconnected from etcd, process will exit"] ["Server Id"=17] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Register.func1\n\t/workspace/source/internal/querynodev2/server.go:173"]

standalone-0.zip

Anything else?

No response

[yanliang567]

/assign @jaime0815
/unassign

[jaime0815]

It seems the crash caused restarts, can you provide more information about your arm machine

[Presburger]

Do you have a specific model for the ARM CPU you are using?