Control your AWS spending with ChatOps

While we build our open source, cloud agnostic Heroku/Cloud Foundry-like Paas - Pipeline - on top of Kubernetes, we continue to launch lots of clusters on different cloud providers. Most of these clusters are launched on spot or preemptible instances, and managed by Hollowtrees. However, there are many smaller development clusters, control planes, instances and PoCs we launch that are marginally related to, or launched with, Pipeline. Naturally, these have an associated cost that we want to keep tight control over.

Hollowtrees manages clusters started by Pipeline and uses sophisticated spot cluster recommendations, however, a billing alert and Slack bot can be used independently from Pipeline to control spending on AWS

tl;dr1: 🔗︎

We’ve automated the creation of an AWS billing alarm, which you can get from our GitHub: banzaicloud/aws-billing-alarm

tl;dr2: 🔗︎

We’ve open sourced a Slack chatbot to help you tightly control your AWS spendings. Get the Cloudformation stack template and the open source Golang lambda function, here: banzaicloud/lambda-slack-bot.

Keep reading for more details.

Controlling costs on AWS - available options 🔗︎

Controlling costs on AWS (or any cloud provider) takes some effort, but it definitely pays off. Usually, people perform these two steps manually:

• check an estimate of the monthly total cost
• check for any unused resources that will continue not to be used

Performing these checks manually is extremely tedious, so let’s automate both by:

• creating an AWS billing alarm, which emails you if your estimated charges exceeds a limit
• creating a chatbot, which helps discover resources eligible for billing

AWS Billing Alarm 🔗︎

Creating an AWS billing alarm is relatively straightforward, and an extremely useful way to avoid surprise bills at the end of the month.

If you want to create a Billing alarm, we have a cloudformation template available on GitHub: banzaicloud/aws-billing-alarm

ChatOps 🔗︎

Now lets turn to ChatOps. For most AWS customers, charges are produced by EC2 instances. Have you ever shouted something like:

• Did anybody leave an instance running in the us-west region?
• Who started that m4.xlarge instance two weeks ago in eu-central-1?

One issue is that, on the AWS console, you always have to select a specific region to work with. So, if you want to check multiple regions, you have to repeat the process for each one.

SlackBot 🔗︎

Slack is one of the most popular team communication platforms, and it offers a great deal of extensibility. For example, Slash Commands allow you to easily trigger an App (the slack term for an extension), once you have installed/configured it, using a slack message that starts with “/” like:

/instances

During the deployment process, you will:

• deploy a CloudformationStack (api gateway + lambda function + ec2 readOnly policy)
• create a slack app
• add a slash command pointing to the apiGateway endpoint

Deploy the stack from cli - the quick way 🔗︎

If you have the aws-cli installed, you can deploy the stack with a single command:

export AWS_STACK_NAME=slackBot
aws cloudformation create-stack \
  --template-url https://s3-eu-west-1.amazonaws.com/lp-lambda-go/aws-slack-bot.json \
  --capabilities CAPABILITY_IAM \
  --stack-name $AWS_STACK_NAME

If you want to hack on the template, you can find it on GitHub: banzaicloud/lambda-slack-bot

Deploy the stack from console - it takes longer than installing awscli 🔗︎

If you don’t have aws-cli installed, or you like to click and type, you can deploy the stack from the console:

• Start the Create Stack on AWS console:
• click Next Select Template in the Select Template step.
• click Next on the Specify Details step.
• click Next on the Options step.
• On the Review step, scroll down to Capabilities and check both I acknowledge ... checkboxes
• Create ChangeSet in the Transforms section
• Once the transformation is done, scroll down and click on Execute

Verifying the deployment 🔗︎

Once the slackBot stack is deployed, watch the Output for the restAPI url.

export AWS_STACK_NAME=slackBot
aws cloudformation describe-stacks \
  --stack-name $AWS_STACK_NAME \
  --query 'Stacks[0].Outputs[0].OutputValue' \
  --out text

You can test the API endpoint via curl. The response to a GET will be the version

$ curl https://a1b2c3d4e5.execute-api.eu-west-1.amazonaws.com/Prod
version: "0.0.7"

Create a Slack App 🔗︎

• Go to SlackApi / YourApps
• Click on Create New App
• Fill out the form:
  • App Name : AwsInstancesBot
  • Development Slack Workspace: Choose the workspace (community) where you want to use the bot.
  • click Create App

Add a Slash Command 🔗︎

• Choose: Slash Commands in the left menu
• Click on Create New Command Button:
• Fillout the form:
  • Command: /instances
  • Request URL: https://a1b2c3d4e5.execute-api.eu-west-1.amazonaws.com/Prod
  • Short Description: Lists running ec2 instances
  • Usage Hint : /instances ascii
  • click Save

Activate the Bot on the Workspace 🔗︎

Although the Create a Slack App step has seemingly attached the bot to the workspace, you need to perform one additional step:

• Settings / Basic Information
• click on the Install your app to your workspace dropdown section
• push the green Install App button

Now you’re ready to use the new slash command. Try writing /instances ascii in the General channel. You should see the ascii table of running instances.

Configuring queried regions 🔗︎

The lambda function stores the coma separated list of regions for the EC2 instances you’d like to be listed in an environment variable. By default, it contains all regions except ap-northeast-2 (it takes about fours seconds with ap-northeast-2, and one second without it).

Roadmap 🔗︎

Here are some possible directions this project can go in:

• instance costs: Include a spent column, using the AWS price api and spot price history to estimate money burned by an instance.
• tag-bot: Instances should have an Owner represented by an EC2 tag, and instances without an owner should be regularly cleaned. A bot can be used to tag ownerless instances. Billing tools can use this tag for cost monitoring/management.
• listing/cleaning VPCs: VPCs are one of those resources with endless dependencies. The console helps if you want to delete dependencies, but, on the cli or in the api, you have to discover and pre-clean all dependencies.

tl;dr 🔗︎

Some additional useful resources used during development.

Resource Groups 🔗︎

Although you could use Resource Groups from the console to discover resources across regions, there appear to be two different types of resource groups:

• classic groups
• resource groups - saved groups (new)

The classic group seems to exist only on the console, and can do multi-region queries. While the new one is available from api/cli like: aws resource-groups list-groups. That group, which is programmable/scriptable, seems only to work on the regional level.

Consequentially, you’re left repeating steps for each region, an arduous task that’s begging for automation.

TabWriter 🔗︎

Creating pretty ascii tables normally involves some variation of fmt.Printf("%20s %20d", strVal, number). But even that requires you to calculate column width. Here, TabWriter comes to the rescue! Just wrap the output with TabWriter, then all you need to do is print the extra tab characters \t between fields, and the rest is taken care of. Again, this is a nice example of a hidden gem in the standard go library:

• See : GoDoc
• Docker cli uses this as well: source