This article describes the process of creating a Slack notification from a CloudWatch alarm generated from an undesirable state in an SQS (Simple Queue Service) queue, via SNS (Simple Notification Service) and Lambda.
You can of course modify any of these to suit your differing requirements. For example, the source could be a different SQS state, or the source could be some other AWS service rather than SQS, or you may want to send the notification to somewhere other than Slack (in which case a different Lambda function may be required), etc.
Lambda
IAM
We’ll firstly need to set up a new IAM user. Give it a suitable name and it shouldn’t need any particular permissions.
KMS
Next we’ll need to set up a key in KMS (Key Management Service). Create one of type symmetric, give it a name (alias), set your IAM user as key administrator, and set the IAM user you created above as the key user.
Lamba function
Now we can set up a Lambda function that will send the CloudWatch alarm notification to Slack via SNS (Simple Notification Service). In Lambda, choose “Create function” then browse the serverless app repository, then search for “cloudwatch-alarm-to-slack-python3” and select this app when it appears. This is a Python app which should suit our requirements. Give it a suitable name. In the comments at the top of the source code it tells you how to set up the webhook for the connection to the channel you need in your Slack workspace, and it also explains how to set up the encryption using the KMS key you created earlier. Follow the instructions there and your Lamba app should be ready to be triggered by CloudWatch alarms.
SNS
Create a topic in SNS of type “standard” with a suitable name, then in the topic create a subscription with protocol “AWS Lambda” and select the Lambda function you created earlier.
CloudWatch
We’ll now create a CloudWatch alarm to check when an SQS queue is not moving, i.e. when there are messages in the queue but they’re not getting sent. This requires the creation of an alarm to check there are messages in the queue, then another alarm to check the messages aren’t getting sent, and finally a composite alarm created from the first two alarms. The composite alarm will trigger the Lambda function via an SNS notification.
“messages in queue” alarm
So, for the first alarm to check there are messages in the queue, the namespace would be AWS/SQS, the metric name we want is “ApproximateNumberOfMessagesVisible”, and the QueueName is the name of the SQS queue you want to monitor. Statistic is “Average” and period can be 1 minute. Under Conditions, the threshold type is static, and we define the alarm condition as greater than 0. We don’t add any actions, and we give it a suitable name such as “SQS – queuename – messages in queue”.
“messages not sent” alarm
For the second alarm to check if messages aren’t getting sent, it’s AWS/SQS as namespace again, the metric name is “NumberOfMessagesDeleted”, the QueueName is the same as before. Statistic is “Average”, period can be 1 minute, and threshold is static. The alarm condition is lower than or equal to 0. We don’t add any actions, and we give it a suitable name such as “SQS – queuename – messages not sent”.
“queue not moving” composite alarm
We then determine that the queue isn’t moving via the criteria that a) we have some visible messages in the queue, but b) messages aren’t getting sent properly because they’re not being deleted from the queue. So for this we create a composite alarm using the previous two alarms we just created, and the condition for this composite alarm is that the “messages in queue” alarm and the “messages not sent” alarm are both in alarm state. We will configure an action on this alarm, so add an action for when the state is “in alarm” and select the SNS topic that you created earlier. Give it a suitable name such as “SQS – queuename – queue not moving”.
All being well, when the queue is stuck an alarm will now be generated and sent as a notification to the selected Slack channel.
Troubleshooting
The part I had most difficulty getting working was the Lambda function, and the output from this function appears in CloudWatch logs. Just go to “log groups” then you should be able to choose /aws/lambda/name-of-function, then choose the most recent log stream and you can see the latest log messages from the Lambda function.
I hope this has been useful in helping you to navigate around the various parts of AWS necessary to set up alarms and have the notifications sent to Slack. If I can be of further help then don’t hesitate to get in touch to ask about my SysAdmin and DevOps services.