Utilising Warm pool with EKS to reduce Boot up time

In today’s fast-paced tech environment, efficiently managing infrastructure during high-demand periods is crucial. This article outlines a strategic approach to optimize infrastructure for quicker scale-up times and enhanced system stability, focusing on the detailed steps of implementation.

Hello everyone, I’m Edwin Philip, Senior DevOps Engineer at Hackerrank, delving deep into our recent infrastructure project aimed at optimizing the scale-up time for our backend applications during peak traffic.

The Challenge

The main challenge is the delay in scaling up backend applications during peak traffic times. The existing setup, involving a Horizontal Pod Autoscaler (HPA), faces a pod readiness lag of up to 5 minutes. This delay is primarily due to the time taken for EC2 provisioning, application image pulling, and application startup, often leading to existing pods crashing and causing service outages.

Balancing Cost and Performance

The strategy involves a balance between cost and performance. By optimizing the infrastructure, significant cost savings are achieved while maintaining or enhancing performance levels.

Accelerating System Readiness

The proposed solution significantly cuts down the readiness time from 5 minutes to about 70 seconds. This is achieved by optimizing the EC2 provisioning, node boot-up process, and the Rails application’s image pull and startup times.

Architecture

The flowchart above outlines the process for managing node lifecycle and application scaling in a Kubernetes cluster, specifically one using Amazon EKS (Elastic Kubernetes Service) and EC2 instances managed by an Auto Scaling Group (ASG). Here is a step-by-step description of the flow:

Warm Pool in ASG: The Auto Scaling Group has a warm pool of pre-initialized EC2 instances that are kept in a state ready to be quickly made active, or “InService”. This reduces the time it takes to provision new instances when scaling out.

Connect to EKS and Taint Node to Disable Pod Scheduling: When a new node from the warm pool is activated, it connects to the EKS cluster. As part of its initialization, the node is tainted to temporarily prevent pods from being scheduled on it. This allows time for necessary setup operations to complete before the node starts accepting workloads.

Cache App Images Using Daemonset: A Daemonset on the cluster ensures that application images are pre-cached on the nodes. This step is crucial because it reduces the time pods take to start since they don’t have to wait for image downloads.

Check if Cache Image Complete and Continue Lifecycle: The system checks if the caching of the application images by the Daemonset is complete. If so, the node continues its lifecycle process.

Hibernate: If the node is not immediately needed, it can be placed back into hibernation, ready to be quickly brought back to an active state when required.

HPA Triggered: The process begins with the Horizontal Pod Autoscaler (HPA) being triggered, which indicates that the scaling criteria defined in the HPA configuration have been met, and more pods need to be scheduled to handle the increased load.

Pod Pending: Once the HPA triggers the scaling event, new pods enter a pending state, waiting for resources to be available in the cluster to be scheduled.

Cluster Autoscaler Requests New EC2 in ASG: The Cluster Autoscaler detects that additional nodes are needed to schedule the pending pods and requests the Auto Scaling Group to launch new EC2 instances.

New Node from Warm Pool Hibernate to InService: A new node transitions from the warm pool’s hibernated state to an active “InService” state, where it can now have pods scheduled onto it.

Implementation: Detailed Steps

Warm pool in EKS

Creating Lifecycle Hooks in ASG

The first step is to create lifecycle hooks in the Auto Scaling Group (ASG). These hooks are crucial for managing the instance states effectively during scaling operations.

aws autoscaling put-lifecycle-hook \
 --lifecycle-hook-name my-lifecycle-hook \
 --auto-scaling-group-name my-asg \
 --lifecycle-transition autoscaling:EC2_INSTANCE_LAUNCHING \
 --default-result CONTINUE

Attaching IAM Policies

Next, specific IAM policies are attached to provide necessary permissions for various operations.

{
   “Version”: “2012–10–17”,
   “Statement”: [
    {
      “Effect”: “Allow”,
      “Action”: [“autoscaling:CompleteLifecycleAction”],
      “Resource”: “arn:aws:autoscaling:*:account-id:autoScalingGroup:*:autoScalingGroupName/my-asg”
    }
   ]
}

Modifying AWS Launch Templates

AWS Launch Templates are modified to incorporate the changes required for the new setup.

INTERFACE=$(curl — silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/ | head -n1)
SUBNET_ID=$(curl — silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/$INTERFACE/subnet-id)
VPC_ID=$(curl — silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/$INTERFACE/vpc-id)
NODE_LABELS=”node.kubernetes.io/vpc-id=$VPC_IDnode.kubernetes.io/subnet-id=$SUBNET_ID”
if [[ $(type -P $(which aws)) ]] && [[ $(type -P $(which jq)) ]] ; then
 TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 2
 INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance
 REGION=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/reg
 LIFECYCLE=$(curl -s H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/autoscaling
 if [[ $LIFECYCLE == *"Warmed"* ]]; then
  TAINTS=" - register-with-taints=Lifecycle=Warmup:NoSchedule"
  WARM_NODE_EXCLUSION_LABEL="node.kubernetes.io/exclude-from-external-load-balancers=warmpool"
  rm /var/lib/cloud/instances/$INSTANCE_ID/sem/config_scripts_user
 fi
fi
# Bootstrap and join the cluster
/etc/eks/bootstrap.sh

Deploying Cache-Image Daemonsets

The deployment of Cache-Image Daemonsets is a strategic step aimed at reducing image pull time during the scale-up process. These daemonsets are responsible for pre-caching commonly used container images on all nodes within the Kubernetes cluster. By having these images pre-cached, the time it takes for new pods to start is significantly reduced, as the container runtime does not need to pull these images from the registry.

This process is particularly important in scenarios where rapid scaling is required. In such situations, the ability to quickly launch new pods without waiting for image downloads can substantially improve the responsiveness and efficiency of the infrastructure.

The daemonset ensures that this image pull runs on every node, thereby caching the required images across the entire cluster. This setup is particularly useful in large-scale deployments where minimizing startup time for new pods is critical for maintaining high availability and responsiveness.
During each deployment, we will update the image in the daemonset as below

kubectl set image daemonset cache-image \
 cache=example.com/my-image:v1.0.0

Implementing Warmpool-Node-Controller Daemonsets

The implementation of Warmpool-Node-Controller Daemonsets is a crucial step in optimizing the infrastructure for efficient scale-up. This daemonset plays a vital role in managing warm pool nodes within a Kubernetes cluster, ensuring these nodes are ready to be quickly integrated into the cluster when needed.

Create ClusterRoleBinding and ServiceAccount, granting the necessary permissions for the daemonset to interact with Kubernetes resources and perform its intended functions.

The daemonset will do the following

• Node and Instance Information: It fetches the node name, instance ID, and AWS region by querying the EC2 instance metadata. This information is crucial for correctly identifying the node within the cluster and AWS environment.
• Lifecycle State Handling: The script checks the lifecycle state of the node (either “InService” or “Warmed”). Depending on the state, it performs different actions.
• InService State: If the node is in “InService” state, the script updates the Kubernetes configuration, removes the ‘Warmup’ taint from the node, and marks the lifecycle hook as completed in AWS Auto Scaling.
• Warmed State: For nodes in the “Warmed” state, the script waits until a cache image daemonset pod is running on the node. Once this condition is met, it completes the lifecycle action in AWS Auto Scaling.

Deploying Warmpool-Node-Cleaner Deployment

The deployment of the Warmpool-Node-Cleaner is an integral part of optimizing Kubernetes infrastructure, particularly in managing the lifecycle of warm pool nodes. This deployment aims to efficiently clean up nodes that were previously used for warming up and are no longer needed, ensuring a tidy and resource-efficient environment.

Create a ClusterRole named system:node:controller, a ClusterRoleBinding, and a ServiceAccount named warmup-node-cleanup. These RBAC configurations provide the necessary permissions for the deployment to interact with Kubernetes resources, particularly for operations like getting node information and deleting nodes.

The script running within the container performs several important functions:

• It first retrieves a list of all nodes with a specific taint (node.cloudprovider.kubernetes.io/shutdown=NoSchedule), which identifies them as warm pool nodes that have been stopped.
• For each of these nodes, the script checks if a specific pod (identified by the label app.kubernetes.io/name=cache-image) is in a running state.
• If the cache-image pod is running on the node, indicating that the node is no longer needed for warming purposes, the script proceeds to delete the node from the Kubernetes cluster.

Adjusting Pod Priority Classes

Adjusting the priority classes for pods ensures that critical services receive resources preferentially during scaling.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: high-priority
value: 1000000
globalDefault: false
description: “This priority class should be used for XYZ service pods only.”

Utilizing Startup Probes with Reduced Initial Delay

In Kubernetes, Startup Probes are a critical feature used to determine when a container application has started successfully. They are particularly useful for applications that have a slow start time, as they prevent Kubernetes from killing these slow-starting containers before they’re up and running.

Reducing the Initial Delay in a Startup Probe is a strategic decision aimed at enhancing the responsiveness and efficiency of the system. When the Initial Delay is set to a lower value, Kubernetes starts checking the readiness of the pod sooner. This approach can be beneficial for applications that are expected to start quickly, allowing the system to rapidly scale up by promptly routing traffic to these newly started pods.

Conclusion

This approach to infrastructure optimization is aimed at enhancing readiness times and managing infrastructure costs effectively. By focusing on technical adjustments and strategic implementation, this strategy enables a more responsive and resilient tech infrastructure capable of handling peak demands efficiently.