OpenCTI TAXII Feed - 413 Content Too Large

[agrawald]

Description

We are trying to connect OpenTAXII collection, which is very large, to OpenCTI, using TAXII feeds.
While processing the collection, we are getting following error

{"category":"APP","context":"Taxii ingestion execution","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Request failed with status code 413","name":"UNKNOWN_ERROR","stack":"GraphQLError: Request failed with status code 413
at error (/opt/opencti/build/src/config/errors.js:7:10)
at UnknownError (/opt/opencti/build/src/config/errors.js:81:47)
at Object._logWithError (/opt/opencti/build/src/config/conf.js:238:17)
at Object.error (/opt/opencti/build/src/config/conf.js:247:48)
at /opt/opencti/build/src/manager/ingestionManager.ts:402:18
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Promise.all (index 0)
at async Promise.all (index 1)
at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:508:5)
at /opt/opencti/build/src/manager/ingestionManager.ts:529:9
at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)
at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request failed with status code 413","name":"AxiosError","stack":"AxiosError: Request failed with status code 413
at settle (/opt/opencti/build/node_modules/axios/lib/core/settle.js:19:12)
at IncomingMessage.handleStreamEnd (/opt/opencti/build/node_modules/axios/lib/adapters/http.js:599:11)
at IncomingMessage.emit (node:events:531:35)
at endReadableNT (node:internal/streams/readable:1696:12)
at processTicksAndRejections (node:internal/process/task_queues:82:21)
at yKt.request (/opt/opencti/build/node_modules/axios/lib/core/Axios.js:45:41)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at taxiiHttpGet (/opt/opencti/build/src/manager/ingestionManager.ts:314:29)
at taxiiV21DataHandler (/opt/opencti/build/src/manager/ingestionManager.ts:375:24)
at async Promise.all (index 0)
at async Promise.all (index 1)
at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:508:5)
at /opt/opencti/build/src/manager/ingestionManager.ts:529:9
at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)
at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}],"level":"error","message":"Request failed with status code 413","name":"BLOCK_FEED_CONTEXT","source":"backend","timestamp":"2024-10-18T03:41:31.368Z","version":"6.3.6"}

Environment

1. OS (where OpenCTI server runs): AWS RHEL EC2
2. OpenCTI version: OpenCTI 6.3.6
3. OpenCTI client: NA
4. Other environment details: NA

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Configure a TAXII Feeds for OpenTAXII which has a large collection

Expected Output

Large Collection from OpenTAXII should make use of paginations to fetch thereby, importing all the objects.

Actual Output

Errors out as OpenCTI is trying to fetch everything in one REST API call.

Additional information

NA

Screenshots (optional)

NA

[nino-filigran]

@agrawald do you maybe a feed that we can use to be able to reproduce?

[agrawald]

Apologies @nino-filigran I am not allowed to share the feed as part of my corporate agreement. However, I can tell you that the feed has more than 45000 STIX records. I will still check with my team.

[agrawald]

Apologies @nino-filigran I am not allowed to share the feed as part of my corporate agreement. However, I can tell you that the feed has more than 45000 STIX records. I will still check with my team.

Apologies, will not be able to help you with the test data. However, if you do decide to implement pagination for TAXII feeds, I can download and test it out for you on a branch if you would prefer.

[nino-filigran]

Thanks @agrawald, I'm still trying to figure out a Taxii with this amount of data for now!

[agrawald]

@nino-filigran what is stopping you to implement pagination for the TAXII feeds.

[nino-filigran]

@agrawald nothing form a technical standpoint as far as I know, but simply the current workload of the team. We have quite some bugs opened already, which have a higher priority at the moment, in addition of the delivery of features.