[Bug]: my collection load error

[DotaArtist]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.5.3
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): 
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus==2.5.3
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

milvus collection is always stuck at "74% is loaded".

Expected Behavior

No response

Steps To Reproduce

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

@DotaArtist

can you offer more details, including you deploy statement, your data volume and your configs.

also please offer your detailed logs so we can investigate on it

[yanliang567]

/assign @DotaArtist
If you install Milvus with Helm or Operator, please refer this doc to export the whole Milvus logs.
If you install Milvus with docker-compose, please use docker-compose logs > milvus.log to export the logs.

/unassign

[DotaArtist]

@yanliang567
milvus-standalone | [2025/01/20 08:44:52.102 +00:00] [INFO] [observers/collection_observer.go:382] ["partition load status updated"] [collectionID=455349803752882378] [partitionID=455349803752882379] [partitionLoadPercentage=75] milvus-standalone | [2025/01/20 08:44:52.102 +00:00] [INFO] [observers/collection_observer.go:396] ["collection load status updated"] [collectionID=455349803752882378] [collectionLoadPercentage=75] milvus-standalone | I20250120 08:44:53.500741 3246 VectorMemIndex.cpp:158] [SERVER][Load][CGO_LOAD][]load index files: 12 milvus-standalone | I20250120 08:44:53.500759 3246 VectorMemIndex.cpp:181] [SERVER][Load][CGO_LOAD][]load with slice meta: true milvus-standalone | I20250120 08:44:57.626650 3246 VectorMemIndex.cpp:242] [SERVER][Load][CGO_LOAD][]construct binary set... milvus-standalone | I20250120 08:44:57.626711 3246 VectorMemIndex.cpp:245] [SERVER][Load][CGO_LOAD][]add index data to binary set: HNSW milvus-standalone | I20250120 08:44:57.626726 3246 VectorMemIndex.cpp:258] [SERVER][Load][CGO_LOAD][]load index into Knowhere... milvus-standalone | I20250120 08:44:57.722681 3246 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][CGO_LOAD] Load index: done (95.820707 ms) milvus-standalone | I20250120 08:44:57.722748 3246 VectorMemIndex.cpp:261] [SERVER][Load][CGO_LOAD][]load vector index done milvus-standalone | I20250120 08:44:57.731271 3144 ChunkedSegmentSealedImpl.cpp:111] [SERVER][LoadVecIndex][CGO_DYN][]Before setting field_bit for field index, fieldID:106. segmentID:455349803850545769, milvus-standalone | I20250120 08:44:57.731366 3144 ChunkedSegmentSealedImpl.cpp:128] [SERVER][LoadVecIndex][CGO_DYN][]Has load vec index done, fieldID:106. segmentID:455349803850545769, milvus-standalone | I20250120 08:44:57.732416 3246 VectorMemIndex.cpp:158] [SERVER][Load][CGO_LOAD][]load index files: 12 milvus-standalone | I20250120 08:44:57.732431 3246 VectorMemIndex.cpp:181] [SERVER][Load][CGO_LOAD][]load with slice meta: true milvus-standalone | I20250120 08:45:00.784195 3246 VectorMemIndex.cpp:242] [SERVER][Load][CGO_LOAD][]construct binary set... milvus-standalone | I20250120 08:45:00.784252 3246 VectorMemIndex.cpp:245] [SERVER][Load][CGO_LOAD][]add index data to binary set: HNSW milvus-standalone | I20250120 08:45:00.784268 3246 VectorMemIndex.cpp:258] [SERVER][Load][CGO_LOAD][]load index into Knowhere... milvus-standalone | I20250120 08:45:00.883455 3246 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][CGO_LOAD] Load index: done (99.080345 ms) milvus-standalone | I20250120 08:45:00.883508 3246 VectorMemIndex.cpp:261] [SERVER][Load][CGO_LOAD][]load vector index done milvus-standalone | I20250120 08:45:00.892084 3294 ChunkedSegmentSealedImpl.cpp:111] [SERVER][LoadVecIndex][CGO_DYN][]Before setting field_bit for field index, fieldID:107. segmentID:455349803850545769, milvus-standalone | I20250120 08:45:00.892174 3294 ChunkedSegmentSealedImpl.cpp:128] [SERVER][LoadVecIndex][CGO_DYN][]Has load vec index done, fieldID:107. segmentID:455349803850545769, milvus-standalone | I20250120 08:45:00.893299 3246 VectorMemIndex.cpp:158] [SERVER][Load][CGO_LOAD][]load index files: 12 milvus-standalone | I20250120 08:45:00.893312 3246 VectorMemIndex.cpp:181] [SERVER][Load][CGO_LOAD][]load with slice meta: true milvus-standalone | I20250120 08:45:04.739423 3246 VectorMemIndex.cpp:242] [SERVER][Load][CGO_LOAD][]construct binary set... milvus-standalone | I20250120 08:45:04.739586 3246 VectorMemIndex.cpp:245] [SERVER][Load][CGO_LOAD][]add index data to binary set: HNSW milvus-standalone | I20250120 08:45:04.739632 3246 VectorMemIndex.cpp:258] [SERVER][Load][CGO_LOAD][]load index into Knowhere... milvus-standalone | I20250120 08:45:05.027755 3246 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][CGO_LOAD] Load index: done (287.820620 ms) milvus-standalone | I20250120 08:45:05.027953 3246 VectorMemIndex.cpp:261] [SERVER][Load][CGO_LOAD][]load vector index done milvus-standalone | I20250120 08:45:05.063761 3263 ChunkedSegmentSealedImpl.cpp:111] [SERVER][LoadVecIndex][CGO_DYN][]Before setting field_bit for field index, fieldID:104. segmentID:455349803850545769, milvus-standalone | I20250120 08:45:05.064170 3263 ChunkedSegmentSealedImpl.cpp:128] [SERVER][LoadVecIndex][CGO_DYN][]Has load vec index done, fieldID:104. segmentID:455349803850545769, milvus-standalone | I20250120 08:45:05.066830 3246 VectorMemIndex.cpp:158] [SERVER][Load][CGO_LOAD][]load index files: 1 milvus-standalone | I20250120 08:45:05.067000 3246 VectorMemIndex.cpp:181] [SERVER][Load][CGO_LOAD][]load with slice meta: false milvus-standalone | I20250120 08:45:06.447511 3246 VectorMemIndex.cpp:242] [SERVER][Load][CGO_LOAD][]construct binary set... milvus-standalone | I20250120 08:45:06.447589 3246 VectorMemIndex.cpp:245] [SERVER][Load][CGO_LOAD][]add index data to binary set: SPARSE_INVERTED_INDEX milvus-standalone | I20250120 08:45:06.447620 3246 VectorMemIndex.cpp:258] [SERVER][Load][CGO_LOAD][]load index into Knowhere... milvus-standalone | I20250120 08:45:06.540277 3246 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][CGO_LOAD] Load index: done (92.472262 ms) milvus-standalone | I20250120 08:45:06.540334 3246 VectorMemIndex.cpp:261] [SERVER][Load][CGO_LOAD][]load vector index done milvus-standalone | I20250120 08:45:06.541373 3269 ChunkedSegmentSealedImpl.cpp:111] [SERVER][LoadVecIndex][CGO_DYN][]Before setting field_bit for field index, fieldID:105. segmentID:455349803850545769, milvus-standalone | I20250120 08:45:06.541445 3269 ChunkedSegmentSealedImpl.cpp:128] [SERVER][LoadVecIndex][CGO_DYN][]Has load vec index done, fieldID:105. segmentID:455349803850545769, milvus-standalone | I20250120 08:45:06.541950 3246 ChunkedSegmentSealedImpl.cpp:261] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 109 with num_rows 39204 milvus-standalone | I20250120 08:45:06.542011 3246 ChunkedSegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 submits load field 109 task to thread pool milvus-standalone | I20250120 08:45:06.542011 3283 ChunkedSegmentSealedImpl.cpp:261] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 1 with num_rows 39204 milvus-standalone | I20250120 08:45:06.542053 3287 ChunkedSegmentSealedImpl.cpp:261] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 0 with num_rows 39204 milvus-standalone | I20250120 08:45:06.542078 3267 ChunkedSegmentSealedImpl.cpp:261] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 100 with num_rows 39204 milvus-standalone | I20250120 08:45:06.542152 3283 ChunkedSegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 submits load field 1 task to thread pool milvus-standalone | I20250120 08:45:06.542166 3228 ChunkedSegmentSealedImpl.cpp:261] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 102 with num_rows 39204 milvus-standalone | I20250120 08:45:06.542309 3287 ChunkedSegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 submits load field 0 task to thread pool milvus-standalone | I20250120 08:45:06.542371 3252 ChunkedSegmentSealedImpl.cpp:261] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 101 with num_rows 39204 milvus-standalone | I20250120 08:45:06.542375 3268 ChunkedSegmentSealedImpl.cpp:261] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 103 with num_rows 39204 milvus-standalone | I20250120 08:45:06.542428 3267 ChunkedSegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 submits load field 100 task to thread pool milvus-standalone | I20250120 08:45:06.542431 3252 ChunkedSegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 submits load field 101 task to thread pool milvus-standalone | I20250120 08:45:06.542484 3268 ChunkedSegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 submits load field 103 task to thread pool milvus-standalone | I20250120 08:45:06.542346 3260 ChunkedSegmentSealedImpl.cpp:261] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 108 with num_rows 39204 milvus-standalone | I20250120 08:45:06.542644 3228 ChunkedSegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 submits load field 102 task to thread pool milvus-standalone | I20250120 08:45:06.542650 3260 ChunkedSegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 submits load field 108 task to thread pool milvus-standalone | I20250120 08:45:06.866452 3246 ChunkedSegmentSealedImpl.cpp:286] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 109 mmap false done milvus-standalone | I20250120 08:45:07.334079 3287 ChunkedSegmentSealedImpl.cpp:286] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 0 mmap false done milvus-standalone | I20250120 08:45:07.492239 3260 ChunkedSegmentSealedImpl.cpp:286] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 108 mmap false done milvus-standalone | I20250120 08:45:07.974025 3283 ChunkedSegmentSealedImpl.cpp:286] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 1 mmap false done milvus-standalone | I20250120 08:45:08.868054 3267 ChunkedSegmentSealedImpl.cpp:286] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 100 mmap false done milvus-standalone | I20250120 08:45:09.398425 3252 ChunkedSegmentSealedImpl.cpp:286] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 101 mmap false done milvus-standalone | I20250120 08:45:10.356868 3228 ChunkedSegmentSealedImpl.cpp:286] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 102 mmap false done milvus-standalone | I20250120 08:45:10.377106 3268 ChunkedSegmentSealedImpl.cpp:286] [SERVER][LoadFieldData][CGO_LOAD][]segment 455349803850545769 loads field 103 mmap false done milvus-standalone | [2025/01/20 08:45:10.702 +00:00] [INFO] [observers/collection_observer.go:382] ["partition load status updated"] [collectionID=455349803752882378] [partitionID=455349803752882379] [partitionLoadPercentage=74] milvus-standalone | [2025/01/20 08:45:10.702 +00:00] [INFO] [observers/collection_observer.go:396] ["collection load status updated"] [collectionID=455349803752882378] [collectionLoadPercentage=74] milvus-standalone | I20250120 08:45:19.201923 241 Utils.cpp:934] [SERVER][LoadFieldDatasFromRemote][milvus][]failed to load data from remote: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433] milvus-standalone | [2025/01/20 08:45:19.202 +00:00] [WARN] [segments/segment.go:749] ["LoadMultiFieldData failed"] [traceID=b6e4cb477e7513b6428f41a614687b58] [collectionID=455349803752882378] [partitionID=455349803752882379] [segmentID=455349803797645684] [error="failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]"] [errorVerbose="failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/segcore.(*cSegmentImpl).LoadFieldData\n | \t/workspace/source/internal/util/segcore/segment.go:270\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*LocalSegment).LoadMultiFieldData.func1\n | \t/workspace/source/internal/querynodev2/segments/segment.go:745\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:82\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) failed to load field data\nWraps: (3) Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] milvus-standalone | [2025/01/20 08:45:19.260 +00:00] [WARN] [segments/segment_loader.go:323] ["load segment failed when load data into memory"] [traceID=b6e4cb477e7513b6428f41a614687b58] [collectionID=455349803752882378] [segmentType=Growing] [requestSegments="[455349803797645684]"] [preparedSegments="[455349803797645684]"] [partitionID=455349803752882379] [segmentID=455349803797645684] [segmentType=L1] [error="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]"] [errorVerbose="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:336\n | [...repeated from below...]\nWraps: (2) At LoadSegment\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/segcore.(*cSegmentImpl).LoadFieldData\n | \t/workspace/source/internal/util/segcore/segment.go:270\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*LocalSegment).LoadMultiFieldData.func1\n | \t/workspace/source/internal/querynodev2/segments/segment.go:745\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:82\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (4) failed to load field data\nWraps: (5) Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"] milvus-standalone | [2025/01/20 08:45:19.260 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]"] [errorVerbose="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:336\n | [...repeated from below...]\nWraps: (2) At LoadSegment\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/segcore.(*cSegmentImpl).LoadFieldData\n | \t/workspace/source/internal/util/segcore/segment.go:270\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*LocalSegment).LoadMultiFieldData.func1\n | \t/workspace/source/internal/querynodev2/segments/segment.go:745\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:82\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (4) failed to load field data\nWraps: (5) Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"] milvus-standalone | [2025/01/20 08:45:19.260 +00:00] [WARN] [segments/segment_loader.go:362] ["failed to load some segments"] [traceID=b6e4cb477e7513b6428f41a614687b58] [collectionID=455349803752882378] [segmentType=Growing] [requestSegments="[455349803797645684]"] [preparedSegments="[455349803797645684]"] [error="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]"] [errorVerbose="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:336\n | [...repeated from below...]\nWraps: (2) At LoadSegment\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/segcore.(*cSegmentImpl).LoadFieldData\n | \t/workspace/source/internal/util/segcore/segment.go:270\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*LocalSegment).LoadMultiFieldData.func1\n | \t/workspace/source/internal/querynodev2/segments/segment.go:745\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:82\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (4) failed to load field data\nWraps: (5) Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"] milvus-standalone | [2025/01/20 08:45:19.260 +00:00] [WARN] [segments/segment_loader.go:271] ["release new segment created due to load failure"] [traceID=b6e4cb477e7513b6428f41a614687b58] [collectionID=455349803752882378] [segmentType=Growing] [requestSegments="[455349803797645684]"] [preparedSegments="[455349803797645684]"] [segmentID=455349803797645684] [error="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]"] [errorVerbose="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:336\n | [...repeated from below...]\nWraps: (2) At LoadSegment\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/segcore.(*cSegmentImpl).LoadFieldData\n | \t/workspace/source/internal/util/segcore/segment.go:270\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*LocalSegment).LoadMultiFieldData.func1\n | \t/workspace/source/internal/querynodev2/segments/segment.go:745\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:82\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (4) failed to load field data\nWraps: (5) Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"] milvus-standalone | [2025/01/20 08:45:19.282 +00:00] [WARN] [delegator/delegator_data.go:357] ["failed to load growing segment"] [traceID=b6e4cb477e7513b6428f41a614687b58] [collectionID=455349803752882378] [channel=by-dev-rootcoord-dml_3_455349803752882378v3] [replicaID=455438769809784835] [error="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]"] [errorVerbose="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:336\n | [...repeated from below...]\nWraps: (2) At LoadSegment\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/segcore.(*cSegmentImpl).LoadFieldData\n | \t/workspace/source/internal/util/segcore/segment.go:270\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*LocalSegment).LoadMultiFieldData.func1\n | \t/workspace/source/internal/querynodev2/segments/segment.go:745\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:82\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (4) failed to load field data\nWraps: (5) Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"] milvus-standalone | [2025/01/20 08:45:19.282 +00:00] [WARN] [querynodev2/services.go:319] ["failed to load growing segments"] [traceID=b6e4cb477e7513b6428f41a614687b58] [collectionID=455349803752882378] [channel=by-dev-rootcoord-dml_3_455349803752882378v3] [currentNodeID=5] [error="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]"] [errorVerbose="At LoadSegment: failed to load field data: Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:336\n | [...repeated from below...]\nWraps: (2) At LoadSegment\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/segcore.(*cSegmentImpl).LoadFieldData\n | \t/workspace/source/internal/util/segcore/segment.go:270\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*LocalSegment).LoadMultiFieldData.func1\n | \t/workspace/source/internal/querynodev2/segments/segment.go:745\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:82\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (4) failed to load field data\nWraps: (5) Error in GetObjectBuffer[errcode:503, exception:, errmessage:, params:params, bucket=a-bucket, object=files/insert_log/455349803752882378/455349803752882379/455349803797645684/101/455349803753113433]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"]