Ensuring High Availability and Resilience in Amazon EKS-2

Have you ever experienced downtime or sluggish performance in your applications and wondered why? 

One common culprit is node disruption in your Kubernetes cluster, especially when running on Amazon EKS. Node disruptions can significantly impact user experience and application performance, making it crucial to understand and address these issues.

Node disruptions occur when the nodes in your Kubernetes cluster, which host your application pods, experience failures or maintenance events. These disruptions can lead to pod failures, causing temporary unavailability of services and affecting user experience.

Now before we dive deep into how we can achieve high availability and resilience in our kubernetes, running on EKS, and to narrow the impact of node disruption or pod failure on our users, we first need to understand node disruption and its effect on users and application. So, let’s get started.

Understanding Node Disruptions in EKS

It is relatively common for nodes in AWS Elastic Kubernetes Service (EKS) to go down and new nodes to be allocated. This behavior can be attributed to several factors:

• Auto Scaling: EKS clusters often use node groups that are managed by an Auto Scaling Group (ASG). If a node goes down due to failure or maintenance, the ASG automatically replaces it with a new one to maintain the desired capacity.
• Spot Instances: If your EKS cluster uses spot instances, these can be terminated by AWS when the spot price exceeds your bid or if there is insufficient capacity. In such cases, new instances are launched when the capacity becomes available again.
• Node Health Checks: AWS continuously monitors the health of nodes. If a node is deemed unhealthy, it can be automatically terminated and replaced by a new node to ensure the cluster remains healthy and available.
• Cluster Upgrades: During EKS cluster upgrades or updates to the node groups, nodes may be drained and replaced to apply the updates without downtime.

Effects of Node Disruptions on Users and Applications

When a node is terminated and a new one is initiated in an Amazon EKS cluster, there can be several effects on users and applications, depending on how the cluster and workloads are configured:

1. Pod Disruptions

If the node that is terminated was running pods, those pods will be evicted and need to be rescheduled on other nodes. If there are not enough resources available on the remaining nodes, this could lead to temporary downtime for the affected services until new nodes are up and running and the pods are rescheduled.

2. Service Availability

For stateful applications or those not designed to handle disruptions gracefully, users might experience service interruptions. Stateless applications or those using Kubernetes features like ReplicaSets, Deployments, or StatefulSets can mitigate this by maintaining desired replicas and ensuring that services remain available even when nodes are replaced.

3. Load Balancer Reconfiguration

If the terminated node was part of a load-balanced service, the load balancer might temporarily have fewer backend instances to distribute traffic to, potentially affecting performance or availability until the new node and its pods are up and running.

4. Data Loss for Non-Persistent Storage

Applications using non-persistent storage might lose data if the node is terminated unexpectedly. Persistent Volumes (PVs) should be used to ensure data persistence across node replacements.

Strategies to Minimize Impact

To ensure high performance and achieving resilience in our application over kubernetes and to reduce the impact of node replacement and pods scheduling on users we can setup different approaches such as:

1. Logging and Monitoring

Set up logging and monitoring using tools like Amazon CloudWatch, Prometheus, or Grafana. We already have our own monitoring stack so we will be focusing on using our own stack. Monitoring stack will be discussed in another article focusing on Grafana, Prometheus, Loki, Faro, On-Call and Pyroscope.

2. Pod Disruption Budgets

Use Kubernetes features such as pod disruption budgets(PDBs) to ensure application availability during node replacements. These help control the number of pods that can be disrupted during node terminations, ensuring a minimum number of replicas are always available.

3. Node Taint and Pod Affinity

Configure node taints and tolerations to control which pods can be scheduled on the nodes, which improves resilience.

4. Horizontal Pod Autoscaling

Configure horizontal pod autoscaling (HPA) so that you can have enough pods available when we have an increase in traffic and to handle load even at the time of disruption.

5. Implement Graceful Shutdown

Configure applications to handle termination signals gracefully, allowing them to clean up resources and state before being evicted.

Pod Disruption Budgets:

Pod Disruption Budgets (PDBs) in Kubernetes are a way to specify the minimum number or percentage of replicas that must be running at any given time, allowing for safe and controlled voluntary disruptions. 

These disruptions can be caused by actions like draining a node for maintenance, updating deployments or scaling down deployments. For involuntary disruptions occur unexpectedly and include node failures, kernel panics etc. 

PDBs do not protect against these disruptions directly but can help ensure there are always some pods running. 

Benefits:

• High Availability: Ensures that a certain number of pods remain available during maintenance or scaling operations.
• Controlled Disruptions: Helps in maintaining service reliability by controlling the number of pods that can be taken down simultaneously. Implementation is as follow:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ template "relname" . }}-web-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: {{ template "relname" . }}-web

Explanation:

• minAvailable: Specifies the minimum number of pods that must be available at any time. In this example, at least 1 pod must always be running.
• selector: Identifies the set of pods to which this PDB applies. It matches the pods with the label app: {{ template "relname" . }}-web.

Node Taint and Pod Affinity:

By default, Kubernetes does not guarantee that replicas of a pod will be scheduled on different nodes. The Kubernetes scheduler places pods based on available resources and does not inherently spread replicas across nodes without explicit instructions.

Without specific configuration, the scheduler may place multiple replicas on the same node if it meets the resource requirements. This can lead to reduced fault tolerance because if that node fails, all replicas on that node will be lost simultaneously.

To ensure replicas are scheduled on different nodes, you need to use Pod Anti-Affinity rules in your deployment configuration. This explicitly instructs Kubernetes to avoid placing certain pods on the same node, enhancing redundancy and availability. You need to add anti-affinity rules to your deployment as shown below:

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - my-app
        topologyKey: "kubernetes.io/hostname"

Explanation:

• Affinity: The affinity section is used to define rules for scheduling the pods.
• podAntiAffinity: This specifies that pods should avoid being scheduled on the same node.
• requiredDuringSchedulingIgnoredDuringExecution: This rule ensures that pods are strictly not scheduled on the same node.
• labelSelector: This specifies the label of the pods that should not be co-located.
• topologyKey: kubernetes.io/hostname ensures the rule is applied at the node level.

Horizontal Pod Autoscaling:

Horizontal Pod Autoscaling (HPA) in Kubernetes automatically adjusts the number of pod replicas in a deployment, replication controller, replica set, or stateful set based on observed CPU utilization (or other select metrics). 

This helps ensure that your application can handle varying loads efficiently by scaling out (adding more pods) during high traffic and scaling in (reducing the number of pods) when the load decreases.

Key Components of HPA:

1. Metrics Server: Collects and provides resource utilization data to the HPA controller. It is typically deployed as a separate component in your cluster.
2. HPA Controller: Periodically checks the current metrics against the desired thresholds and adjusts the number of replicas accordingly.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: {{ template "relname" . }}-web-hpa
 labels:
   app: {{ template "relname" . }}
   component: hpa
spec:
 scaleTargetRef:
   apiVersion: apps/v1
   kind: Deployment
   name: {{ template "relname" . }}-web
 minReplicas: {{ .Values.autoscaling.minReplicas }}
 maxReplicas: {{ .Values.autoscaling.maxReplicas }}
 metrics:
 - type: Resource
   resource:
     name: cpu
     target:
         type: Utilization
         averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
 - type: Resource
   resource:
     name: memory
     target:
         type: Utilization
         averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}

After setting up HPA, monitor its behavior and adjust the parameters as needed to better fit your application’s load patterns.

Explanation:

• scaleTargetRef: Specifies the target resource to be scaled, in this case, a deployment named {{ template "relname" . }}-web.
• minReplicas: Minimum number of pod replicas.
• maxReplicas: Maximum number of pod replicas.
• metrics: Defines the metric type and target value. Here, it’s based on CPU utilization, aiming to maintain an average utilization of 50%

Implement Graceful Shutdown

Implementing graceful shutdown in Kubernetes ensures that your applications have time to clean up and save their state before being terminated. This process involves handling the termination signals sent by Kubernetes when a pod is being shut down. Here are the steps to implement graceful shutdown in Kubernetes:

Handle Termination Signals in Your Application

Your application needs to handle termination signals (like SIGTERM) and implement logic to gracefully shut down. This might involve closing open connections, saving state, or other cleanup tasks. You do this in each of your service, an example of doing this in python is as follow:

import signal
import time
def handle_sigterm(signal_number, frame):
    print("Received SIGTERM. Shutting down gracefully...")
    # Perform cleanup tasks here
    time.sleep(10)  # Simulate cleanup time
    print("Cleanup complete. Exiting.")
signal.signal(signal.SIGTERM, handle_sigterm)
while True:
    time.sleep(1)  # Simulate application work

Configure terminationGracePeriodSeconds in Pod Spec

This field specifies the amount of time Kubernetes will wait for the pod to terminate after sending a SIGTERM signal, before forcefully killing it.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: my-app-container
    image: my-app-image:latest
  terminationGracePeriodSeconds: 30

Use PreStop Hook

A preStop hook can be defined in the pod spec to run a command or send a request to the application, triggering the graceful shutdown logic.

Ensuring Service Availability

If your application is part of a service, ensure that Kubernetes removes the pod from the service endpoints before shutting it down to avoid sending traffic to the terminating pod.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: my-app-image:latest
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 10"]
      terminationGracePeriodSeconds: 30

Conclusion

It’s crucial to ensure high availability and resilience in your Kubernetes environment on Amazon EKS. You must understand node disruptions and their impact on users. This helps in maintaining a seamless experience for users and a reliable service.

You can use strategies like Pod Disruption Budgets, Node Taints and Pod Affinity, Horizontal Pod Autoscaling, and Graceful Shutdown procedures. These will greatly reduce the effects of node replacements and pod scheduling issues.

Lastly, you need to continuously monitor and fine-tune your settings to keep your applications strong. By proactively managing node disruptions and pod scheduling, you can achieve a resilient and highly available Kubernetes environment.