[RST Threat Feed] Slow ingestion, many 'Bulk indexing fail' errors

[dominictory]

Description

We have been testing the RST Threat Feed connector, which produces a lot of bundles. We are unsure what timing to set for the connector to reduce bundle count each time it ingests, as currently it has 500k queued bundles that are ingesting at a rate of about 2/s, whereas all our other connectors ingest at 20/s plus. However, I'd be concerned that if this was a long time frame, there'd just be even more bundles at the next run. We also see a lot of Bulk indexing fail errors each run, example below:

{'name': 'DATABASE_ERROR', 'error_message': 'Bulk indexing fail'}

{"type": "indicator", "spec_version": "2.1", "id": "indicator--0f95d519-efb0-5270-a086-41eeef5da243", "created_by_ref": "identity--1ddcb2ae-1855-5c25-8096-a889c4e7918b", "created": "2025-01-19T00:00:00.000Z", "modified": "2025-01-19T00:00:00.000Z", "name": "makeitwithmichael.com", "description": "IOC with tags: malware. Related threats: ducktail_stealer\n\nWhois Registrar: Tucows Domains Inc\n--- Registrant: unknown\n--- Age: 180\n--- Created: 2024-07-23 04:20:59\n--- Updated: 2024-07-27 17:57:59\n--- Expires: 2025-07-23 04:20:59\n\nRelated IPs:\n--- A Records: ['66.81.203.135', '66.81.203.10', '66.81.203.200']\n--- Alias Records: []\n--- CNAME Records: []\n\nIs a potential false positive? false.", "pattern": "[domain-name:value = 'makeitwithmichael.com']", "pattern_type": "stix", "pattern_version": "2.1", "valid_from": "2025-01-19T00:00:00Z", "labels": ["malware", "ducktail_stealer"], "confidence": 70, "external_references": [{"source_name": "github.com", "url": "https://github.com/stamparm/maltrail/blob/master/trails/static/malware/ducktail-7.txt"}], "object_marking_refs": ["marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9"], "x_opencti_detection": true, "x_opencti_main_observable_type": "Domain-Name", "x_opencti_score": 62, "nb_deps": 1, "x_opencti_create_observables": null, "x_opencti_stix_ids": null, "x_opencti_granted_refs": null, "x_opencti_workflow_id": null}

When other connectors run, we see a spike in bundles processed per second however we suspect that is just due to other connectors ingesting faster.

Environment

6.4.8

Reproducible Steps

Integrate RST Threat Feed connector

Expected Output

Connector ingests at feasible rate without error

Actual Output

Connector ingests extremely slowly