Using Batching Functionality to Avoid Rate Limit Errors

When working with AI functions, you may encounter a rate limit error (Error Code 429) if you run too many functions concurrently. To manage this, it’s essential to use batching functionality effectively.

What is Rate Limit

Rate limits are restrictions that OpenAI API imposes on the number of times a user can access services within a specified period of time. Rate limits are measured in: RPM (requests per minute), RPD (requests per day), TPM (tokens per minute), TPD (tokens per day).

What is Batching?

Batching helps manage the number of requests per minute by grouping multiple function calls into batches. This not only helps in avoiding the rate limit but also ensures smoother operation of your AI functions.

Benefits of Batching

• Prevents Requests Per Minute Limit (RPM) Errors: By controlling the number of function calls per minute.
• Prevents Tokens Per Minute Limit (TPM) Errors: Avoids the tokens per minute limit by adjusting the batch size.

How to Use Batching

1. Find the Batching Setting:
   • Navigate to the Settings -> Batching

   • You can change the batch size (from 5 to 500)

     

2. Determine the Right Batch Size:
   • The ideal batch size depends on the amount of data (text) in each row.
   • The more data you have in each row, the smaller the batch size should be.
   • Start with the highest batch size and reduce it if you encounter errors.
3. Implement Batching:
   • Try running your AI functions in smaller batches.
   • Adjust the batch size based on the amount of data to find the optimal balance.
4. Test and Optimize:
   • Experiment with different batch sizes to determine what works best for your specific use case.
   • Continuously monitor and adjust as needed to maintain performance and avoid errors.

By understanding and implementing batching functionality, you can efficiently manage your AI function calls and prevent rate limit errors, ensuring a smooth and effective operation.