Spotinst raises $15M Series A Led By Intel Capital and Vertex Ventures Read The Official Press Release

NEW: Elastic Map Reduce Breakthrough! Auto-Scaling for Task Nodes is here.

elastic use of Task Nodes within your existing EMR Clusters

Liran Polak
Chief Architect

 

Just recently we have released a breakthrough feature that allows AWS Elastic Beanstalk users to run their environments on Spot. and now, we are so happy to announce our unique Autoscaler for Elastic Map Reduce: EMR ClusterScaler.

 

EMR ClusterScaler enables you to make an elastic use of Task Nodes within your existing EMR Clusters. It lets you to define target, minimum and maximum capacity for Task Nodes, select multiple instance types and sizes, and of course, CloudWatch rules for Scale UP and Down EC2 instances based on your business rules or load.

 

How it works?

EMR ClusterScaler is defined per EMR Job-Flow. In fact it enables you to add 1..N task nodes, from multiple instance-types, and lets you to define CloudWatch rule for scaling Up/Down Task nodes.

1_emr-task-scaler

 

For example, you can connect EMR ClusterScaler to an existing EMR cluster which runs 1 master node, 20 Core-Nodes, and 5 Task-Nodes.

 

  1. Selecting your target EMR Job-Flow Id
  2. Defining the amount of Task Nodes, the instance types and sizes task_scaler-compute
  3. Define the CloudWatch rules for Scale Up and Scale Down 
    1. It can be one of your EMR Metrics such as RemainingMapTasksPerSlot
    2. It can be per instance metric, for example- CPU Usage, Network bandwidth etc..

task_scaler-cloudwatch

Once everything is configured, EMR ClusterScaler launches the desired amount of Task Nodes into your Job-Flow Id. and will automatically scale based on the metrics provided.

 

ClusterScaler makes an efficient and sophisticated use in the Spot Market, by launching Spot Instances as much as possible, to lower the cluster costs, and improve the ROI.

Spotinst ensures that your capacity won’t drop from your user-set minimum, by making a use in our in-house prediction algorithm in the Spot market. When Spotinst scales down resources, it scales first from “risky” Spot capacity, and always scales up from the most available Spot slots in AWS.

 

So, Autoscaling for Map Reduce application was around the corner for a long time. We are so happy that we have the chance to help cloud customers to make more efficient use with their resources.

As always, we value your feedback!

Try EMR ClusterScaler today.

Liran Polak
Chief Architect

Stay current

Sign up for our newsletter, and we'll send you the latest updates on Spotinst, tips, tutorials and more cool stuff!