Growing stix_ids list performance impact

[Markus98]

Prerequisites

• I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
• I went through old GitHub issues and couldn't find anything relevant
• I googled the issue and didn't find anything relevant

Description

We have an OpenCTI instance which has been running for a long time. The performance of this instance has degraded considerably and some queries take a long time and sometimes they time out. Also ingesting bundles created by some of the connectors has become really slow.

I observed that there are many objects with a growing stix_ids list caused by connectors automatically importing the same objects multiple times, but each time with a different randomly generated standard_id. We have been running several connectors with such behavior, and we have many objects with hundreds of ids in their stix_ids list.

My question is that could this be a major reason for the performance degradation that we have observed in our instance?

Environment

1. OS (where OpenCTI server runs): Docker
2. OpenCTI version: 6.2.18
3. OpenCTI client: frontend

Reproducible Steps

1. Use the official stix python library to generate new objects (creates a random uuid each time)
2. Generate and push objects with the same name several times, so that the platform deduplication method merges them
3. Observe how the stix_ids attribute of the object gets larger each time

Additional information

We have already started fixing our custom connectors by using the generate_id method included for many of the object types in the pycti library. This should prevent the stix_ids list from growing, since the standard_id should be the same every time.

I also found some previous discussion about this, but none of it had direct answer regarding degraded performance

• Opencti (Docker) consuming to much memory #3413 (comment)
• Limit stix_ids explosion by rewriting the standard_id in client python #7158
• Control Redis memory usage: limit stix_ids array length in redis stream entries to 1000 ids #6774