SQS Worker with Serverless ⚡️
Experimental Serverless Amazon SQS Worker with AWS Lambda.
Update: Please see Serverless Analytics for a more realistic setup using Amazon Kinesis Streams to process your data and events. You can read more about the Kinesis and Lambda connection at Serverless Analytics with Amazon Kinesis and AWS Lambda. This was only intended as an April Fools' project …
Setup
- SQS Queue with your messages
- SNS Topic to handle CloudWatch Alarms
- DynamoDB table to persist configuration
- CloudWatch Schedule as cron replacement
- Three (
scale,worker,process) AWS Lambda functions
Workflow
- CloudWatch Alarms on queue length post to SNS
- SNS Topic triggers
scaleLambda function - Function
scaleupdates configuration in DynamoDB - CloudWatch Schedule invokes
workereveryxminute(s) - Function
workerreads configuration from DynamoDB - Function
workerinvokesprocessfunction(s)
Auto-Scaling with CloudWatch Alerts
Workers with CloudWatch Schedule
Deploy
$ > yarn install
$ > yarn deploy
Add noise to SQS
You should have some data in your queue to test this setup. Use wrk to send messages to SQS, but make sure to enable anonymous access to sendMessage for your queue first!
$ > wrk -c35 -d60 -t35 \
-s helpers/wrk.lua \
https://sqs.REGION.amazonaws.com/ACCOUNT-ID/YourQueueName