Reindexing fail from opencti_stix_core_relationships-000001 to opencti_deleted_objects

[alexshively]

Description

When attempting to delete a large amount of indicators from the platform, about ~50k, most of them were deleted. However, once the deletion gets to the last ~10k with creation dates on and before February 2023, we receive the errors (full error at bottom):

Reindexing fail from opencti_stix_core_relationships-000001 to opencti_deleted_objects
Reindexing fail from opencti_stix_domain_objects-000001 to opencti_deleted_objects

I had noticed this other issue, but we are on 6.2.18 currently. FWIW, I believe we were on and before version 5.7.4, possibly 5.6.2, when those indicators were created.

#7031

Environment

1. OS (where OpenCTI server runs): Amazon Linux 2023
2. OpenCTI version: 6.2.18
3. OpenCTI client: frontend
4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Create indicators on version 5.6.2/5.7.4 or before
2. Upgrade to 6.2.18
3. Attempt to delete indicators

Expected Output

Indicators are deleted

Actual Output

Indicators are not removed and receive the following errors:

Reindexing fail from opencti_stix_core_relationships-000001 to opencti_deleted_objects
Reindexing fail from opencti_stix_domain_objects-000001 to opencti_deleted_objects

Additional information

Full error:
{ "category": "APP", "errors": [ { "attributes": { "body": { "dest": { "index": "opencti_deleted_objects" }, "script": { "source": "ctx._source.remove('fromType'); ctx._source.remove('toType'); ctx._source.remove('spec_version'); ctx._source.remove('representative'); ctx._source.remove('rel_has-reference');" }, "source": { "index": "opencti_stix_domain_objects-000001", "query": { "ids": { "values": [ "c3e126b4-4ab0-489c-a37c-b35c19c77b19" ] } } } }, "genre": "TECHNICAL", "http_status": 500 }, "message": "Reindexing fail from opencti_stix_domain_objects-000001 to opencti_deleted_objects", "name": "DATABASE_ERROR", "stack": "GraphQLError: Reindexing fail from opencti_stix_domain_objects-000001 to opencti_deleted_objects\n at error (/opt/opencti/build/src/config/errors.js:7:10)\n at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n at /opt/opencti/build/src/database/engine.js:3394:11\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Promise.all (index 0)\n at elDeleteElements (/opt/opencti/build/src/database/engine.js:3433:5)\n at internalDeleteElementById (/opt/opencti/build/src/database/middleware.js:3164:7)\n at deleteElementById (/opt/opencti/build/src/database/middleware.js:3186:32)\n at executeDelete (/opt/opencti/build/src/manager/taskManager.js:219:5)\n at executeProcessing (/opt/opencti/build/src/manager/taskManager.js:471:13)\n at taskHandler (/opt/opencti/build/src/manager/taskManager.js:570:22)\n at /opt/opencti/build/src/manager/taskManager.js:600:9\n at Tlt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)" }, { "message": "Response Error", "name": "ResponseError", "stack": "ResponseError: Response Error\n at onBody (/opt/opencti/build/node_modules/@opensearch-project/opensearch/lib/Transport.js:426:23)\n at IncomingMessage.onEnd (/opt/opencti/build/node_modules/@opensearch-project/opensearch/lib/Transport.js:341:11)\n at IncomingMessage.emit (node:events:531:35)\n at endReadableNT (node:internal/streams/readable:1696:12)\n at processTicksAndRejections (node:internal/process/task_queues:82:21)" } ], "level": "error", "message": "Reindexing fail from opencti_stix_domain_objects-000001 to opencti_deleted_objects", "source": "backend", "timestamp": "2024-12-06T22:26:40.040Z", "version": "6.2.18" }

[SouadHadjiat]

@alexshively could you share the json object of the indicator that fails to be deleted as it is stored on your database ?

We have fixed an issue recenlty related to this error #9173, it was released on 6.4.3.

[alexshively]

@SouadHadjiat Thank you! I will be upgrading

[alexshively]

Sorry I skimmed the first part of your message. Here is one of their raw JSON objects from OpenSearch

{ "_index": "opencti_stix_domain_objects-000001", "_id": "785e743c-b978-416a-aaba-2193f199604f", "_version": 12, "_score": null, "_source": { "pattern_type": "stix", "pattern": "[file:name = 'redacted']", "x_opencti_main_observable_type": "StixFile", "name": "redacted", "description": "Simple indicator of observable {redacted}", "x_opencti_score": 80, "x_opencti_detection": false, "valid_from": "2023-02-28T02:21:45.436Z", "valid_until": "2024-02-28T02:21:45.436Z", "entity_type": "Indicator", "internal_id": "785e743c-b978-416a-aaba-2193f199604f", "standard_id": "indicator--d6a11acd-bd10-50fa-afda-0ea341e5b92f", "creator_id": "7f817b19-2bc8-482f-8662-c0db011accea", "x_opencti_stix_ids": [], "spec_version": "2.1", "created_at": "2023-02-27T20:02:13.552Z", "updated_at": "2024-02-28T02:23:46.892Z", "revoked": true, "confidence": 15, "lang": "en", "created": "2023-02-27T20:02:13.552Z", "modified": "2024-02-28T02:23:46.892Z", "i_valid_from_day": "2023-02-28", "i_valid_from_month": "2023-02", "i_valid_from_year": "2023", "i_valid_until_day": "2024-02-28", "i_valid_until_month": "2024-02", "i_valid_until_year": "2024", "i_created_at_day": "2023-02-27", "i_created_at_month": "2023-02", "i_created_at_year": "2023", "id": "785e743c-b978-416a-aaba-2193f199604f", "base_type": "ENTITY", "parent_types": [ "Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Domain-Object" ], "rel_created-by.internal_id": [ "1e43f46b-449e-4d08-83e8-02bec6d5a1dc" ], "rel_object-marking.internal_id": [ "6df6a7e1-5b05-46e7-a912-38ac685d3a24" ], "rel_object-label.internal_id": [ "c6054a03-e9cc-45a6-91b8-31b9652e20a2" ], "rel_based-on.internal_id": [ "bb90da0d-ecfa-4f49-a9e9-90266d30629d" ] }, "fields": { "i_created_at_day": [ "2023-02-27T00:00:00.000Z" ], "modified": [ "2024-02-28T02:23:46.892Z" ], "created": [ "2023-02-27T20:02:13.552Z" ], "i_valid_until_day": [ "2024-02-28T00:00:00.000Z" ], "created_at": [ "2023-02-27T20:02:13.552Z" ], "valid_from": [ "2023-02-28T02:21:45.436Z" ], "i_valid_from_day": [ "2023-02-28T00:00:00.000Z" ], "updated_at": [ "2024-02-28T02:23:46.892Z" ], "i_valid_until_month": [ "2024-02-01T00:00:00.000Z" ], "i_valid_from_month": [ "2023-02-01T00:00:00.000Z" ], "i_created_at_month": [ "2023-02-01T00:00:00.000Z" ], "valid_until": [ "2024-02-28T02:21:45.436Z" ] }, "highlight": { "entity_type": [ "@opensearch-dashboards-highlighted-field@Indicator@/opensearch-dashboards-highlighted-field@" ] }, "sort": [ 1677528133552 ] }

[alexshively]

@SouadHadjiat I had to wait until after holidays to upgrade, and I did just upgrade to 6.4.5. However, the same issue is occurring. Upon attempting to delete the indicators again, I'm also seeing that the timestamp still has the same time from when we first attempted to delete these indicators, for what its worth. I did clear out Redis just to be sure, and the timestamp remained the same. To be safe, I haven't touched the deleted objects index.

Timestamp	Message
Nov 21, 2024, 9:14:11 PM	Reindexing fail from opencti_stix_core_relationships-000001 to opencti_deleted_objects
Nov 21, 2024, 9:14:11 PM	Reindexing fail from opencti_stix_domain_objects-000001 to opencti_deleted_objects
Nov 21, 2024, 9:14:11 PM	Reindexing fail from opencti_stix_domain_objects-000001 to opencti_deleted_objects
Nov 21, 2024, 9:14:11 PM	Reindexing fail from opencti_stix_core_relationships-000001 to opencti_deleted_objects
Nov 21, 2024, 9:14:11 PM	Reindexing fail from opencti_stix_domain_objects-000001 to opencti_deleted_objects

[SouadHadjiat]

@alexshively you're right, there are old fields (i_valid_*) that are present in your object that we need to ignore when reindexing (because these fields are not recognized by the new index mapping).

[alexshively]

Thanks @SouadHadjiat