Case Study: Winning Combo – Kubernetes, Terraform and Spotinst Elastigroup

Tenpa Kunga

Based in Stockholm, Sweden, [24]7 Predictive Search Bidding (formerly known as Campanja) provides a predictive bidding engine for online advertising allowing their customers to maximize each dollar spent on search advertising (Google AdWords and Bing Ads). [24]7 Predictive Bidding utilizes Real Time Tracking, Predictive Modelling, and High Frequency Bidding to harness the power of online advertising for their clients. [24]7 was an early adapter of Kubernetes (they have been using Kubernetes since version 1.0) and has been pushing forward with their usage of Kubernetes with Spot instances.

High Frequency Bidding for Google AdWords

In the early days of search engine advertising we learned that the process was very old fashioned. Customers would go to an agency and have a new campaign for Google AdWords. The agency would manually place bids for advertising on Google but since this was all done by hand for each keyword they would only be able to place at maximum a couple hundred bids a day for an AdWords account. Eventually, [24]7, then Campanja came up with the concept of a “High Frequency Bidder” through automation. With the High Frequency Bidder they can place millions of bids per day with different aspects such as keywords and cost per click. They can determine the appropriate cost per click based on each query to ensure that their clients are not paying too much when placing bids. If your ad is going to show up on the search you have to know when is the optimal time and day for you to show up on the query. Sometimes it’s not worth it to you to show up because it’s too expensive for the conversion you will get. Other times, it’s worth it and you should bid higher to make sure you get on the search page. The high frequency bidder can place bids much faster than a human can do, this ensures that bids are placed at the appropriate price at any given time and usually ensures that the company is paying the right amount at the optimal time to get a conversion. Since you get much better value from people clicking on links to your sites it makes sense to go with this method vs impressions (paying per viewing of the link). [24]7 Predictive Search tracks clicks to their client’s websites and can calculate exactly what each click costs, how much the client thinks it is worth, and what the bid should be. This type of automation around online advertising is the future and is like Nielsen and an ad bidding partner all in one

Terraform and Kubernetes

[24]7 Predictive Search utilizes separate environments that are defined by Terraform. Typically these environments include production, development, and staging. Environments for dev and staging are frequently torn down when not in use and provisioned again when they are needed. Another advantage of using Terraform is the ability to separate the environment into individual modules. VPCs, Networking, Firewall Rules, and Spotinst Elastigroup Clusters are all configured as separate modules and can easily be individually modified as needed. This granularity and control over each aspect of their environments is what makes Terraform so compelling. New versions of the infrastructure can be easily deployed when a new version of Kubernetes or CoreOS is available and makes updating systems painless.

“The Spotinst Terraform Provider makes it simple to provision EC2 clusters on Spot Instances. We simply define the the configuration and Spotinst takes care of the rest. We can easily save 80% or more on our EC2 costs”

Cost Savings and Spotinst

We currently run about one third of our Kubernetes nodes on on-demand instances. These are typically instances that have large data volumes attached and typically need to run for longer periods of time (like MySQL for example). The other two-thirds of our instances are all run as Spot via Spotinst Elastigroups. These Elastigroup clusters are all provisioned with Terraform as well and when we need to make a change like an AMI or UserData script for example we can easily make this change with Terraform. If we need to launch MySQL pods these will be provisioned on instances labeled On-Demand. Most of our services are spikey, this is why it makes more sense for us to scale our Kubernetes nodes running on Spot via memory rather than CPU. We also have a separate GPU Kubernetes nodes specifically for Data Science. These are less time critical and typically run as batches at night. We also built a Erlang program called PodMon that just collects all our Kubernetes reservations into CloudWatch. We can then scale the number of Kubernetes nodes with Spotinst via the custom CloudWatch metric created by PodMon.

Custom workflow to scale based on the number of Kubernetes reservations.

“With Spotinst we managed to reduce our kubernetes infrastructure costs by at least 80%. We also have peace of mind that our live production environment is protected from Spot Interruptions or lack of capacity concerns. The Spotinst API integration with kubernetes also ensures that there is a seamless transition from an expensive spot to a cheaper spot that is all handled Seamlessly. We’re experiencing an interruption rate that is basically 0% annually”

Learn more about [24]7 and how they are able to help their clients maximize their ad dollars at



-The Spotinst Team