Loop through each item in a large dataset

quanph2

Hello team,

I'm using DE 9.2 and I have around 1 million data records. I need to loop over each item to execute a processing flow. Please advise on how to perform the iteration efficiently without overloading or crashing the system

Thank you

Manisha_Decisions

Hi,

Thanks for reaching out.

To efficiently process 1 million records in Decisions without overloading the system:

• Use the Run Flow For List [Batch Processing] step to process records in batches, not all at once. This enables parallel processing and avoids recursion limits.
• Avoid using a standard ForEach loop for very large lists, as Flows have a default recursion limit (20,000) and may run into performance issues.
• Test with a typical and a large data set to evaluate performance and adjust batch sizes as needed.

Best practices and details:

• Batch processing: https://documentation.decisions.com/docs/best-practices
• Recursion and looping: https://documentation.decisions.com/docs/designing-flows https://documentation.decisions.com/docs/best-practices#loopinglists:~:text=in%20the%20Flow.-,Looping/Lists,-Practice

Best Regards,

Manisha