Tutorials References Exercises Videos Menu
Create Website Get Certified Upgrade

AWS Serverless Scaling Considerations for SQS


Scaling Considerations for AWS SQS

The SQS queue buffers API Gateway queries, allowing for asynchronous processing.

As a Lambda event source, AWS SQS controls polling the queue for you.

But you still have control over other performance-related settings.


Scaling Considerations for AWS SQS Video

W3schools.com collaborates with Amazon Web Services to deliver digital training content to our students.


How it Works

Lambda automatically uses five parallel processes to clear the queue.

These five simultaneous processes indicate Lambda is calling your Lambda function five times.

To prevent having your Lambda function throttled, make sure the reserved concurrency is at least five.

If the Lambda service detects an increase in queue size, it raises batch size.

That implies it will call more Lambda functions concurrently.

Lambda will keep adding processes per minute until the queue slows or hits capacity.

Unless otherwise specified, the maximum concurrency is 1000.


Lambda Failures

The Lambda service will reduce the number of processes polling the queue on Lambda function failures.

Failures are placed in the dead-letter queue and deleted from your SQS queue.

If you choose a batch size that is too big, your Lambda function may timeout.

If it happens frequently, Lambda slows polling, causing a queue blockage.


Unprocessed messages

Unprocessed messages become visible if the visibility timeout expires before the Lambda function processes them.

Those messages will need to be processed again, increasing queue size.

This raises the possibility of duplicate message processing.

The ideal approach is to set your visibility timeout to 6 times the function timeout.

It is critical to set the "MaxReceive" count to balance for high concurrency.

It helps with keeping things moving in the queue when sending too many messages to the dead-letter queue.

The table below shows the characteristics of an SQS queue as a Lambda event source.

Parameter Value/Limit How the parameter is set
Batch message size limit 1-10 Configured with event source on the Lambda function
Number of default pollers 5 Managed by the Lambda service
Lambda's rate of increasing parallel pollers Up to 60 per minute Managed by the Lambda service
Number of concurrent batches Lambda handles Up to 10000 Managed by the Lambda service
Number of concurrent Lambda functions The lesser of 1,000, function or account limit Reserved concurrency limit
Messages per queue No limit N/A
Visibility timeout 0 seconds to 12 hours Configured on the queue
Number of retries 1-1000 Maxreceive count
Function timeout 0 seconds to 15 minutes Configured on the function

Related reads:

Consuming Messages Using Long Polling
Amazon SQS Limits
Lambda Developer Guide: Using Lambda with Amazon SQS
How to Use Amazon SQS as an Event Source for Lambda

AWS Serverless Exercises

Test Yourself With Exercises

Exercise:

What is the maximuzm concurrency number?


Start the Exercise