Troubleshooting FDR Ingest

If the same message appears in the SQS queue more than once, make sure your consumer script reads, processes, and explicitly deletes the SQS message within the visibility timeout period (typically two hours). If, within the timeout period, the SQS message is not downloaded or doesn't process it, the message returns to the queue to be consumed again.

If the consumer script used is based on the sample that CrowdStrike provides, data_replicator_sample_consumer.py, be sure the msg.delete() call is not commented out. Also be sure in the data_replicator_config.py configuration file for the sample script that the VISIBILITY_TIMEOUT value is enough time for your consumer to process any downloaded files and delete the SQS message.

Duplicate messages might start to appear as the result of an increase in the volume of events. The extra events produce more files per SQS message, which in turn increases the processing time of the data in a SQS message.