Carbon-Aware Karpenter: Scheduling K8s Nodes Based on Real-Time Grid Intensity

As data centers evolve into one of the world’s most significant energy consumers, the responsibility for carbon mitigation is shifting from infrastructure providers to software engineers. While cloud providers like AWS, Azure, and Google Cloud have made strides in powering their facilities with renewable energy, the “Grid Intensity”—the amount of CO2 emitted per kilowatt-hour of electricity—varies significantly by geographic region and time of day.

In a standard Kubernetes environment, the Cluster Autoscaler or Karpenter optimizes for cost and resource availability. However, to achieve true GreenOps, we must treat carbon as a first-class scheduling constraint. This article explores how to integrate Karpenter with real-time carbon intensity data to build a carbon-aware infrastructure.

---

Technical Overview

The Shift from Cost-Optimization to Carbon-Awareness

Traditional autoscaling follows a reactive pattern: Pods are unschedulable → Provisioner adds nodes → Node is ready. Karpenter improves this by bypassing Node Groups and interacting directly with the EC2 Fleet API (on AWS), allowing for near-instantaneous, heterogeneous node provisioning.

Carbon-Aware Karpenter adds a feedback loop to this process. By consuming data from Carbon Intensity APIs (such as Electricity Maps or WattTime), we can dynamically adjust Karpenter’s NodePool configurations.

Architecture Components

1. Carbon Intensity Source: An external API providing real-time $gCO_2eq/kWh$ data.
2. Carbon-Aware Controller: A lightweight operator or CronJob running inside the cluster that fetches carbon data.
3. Karpenter NodePools: Custom Resources (CRDs) that define the constraints for node provisioning.
4. Temporal & Spatial Logic:
   • Temporal Shifting: Delaying non-critical workloads until grid intensity is low.
   • Spatial Shifting: Directing workloads to regions with cleaner grids (requires multi-region orchestration).
   • Hardware Efficiency: Prioritizing high-efficiency silicon (e.g., AWS Graviton) when the grid is “dirty.”

The Logic Flow

[Carbon API] $\rightarrow$ [Carbon Controller] $\rightarrow$ [Update NodePool CRD] $\rightarrow$ [Karpenter Provisioning]

---

Implementation Details

1. Defining the Carbon-Aware NodePool

Karpenter’s NodePool allows us to define requirements via labels. We can use a custom label carbon-intensity to influence where pods land.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: carbon-aware-pool
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot"] # Spot instances often align with renewable surpluses
        - key: "kubernetes.io/arch"
          operator: In
          values: ["arm64"] # Prioritize Graviton for better performance-per-watt
        - key: "my-org/carbon-status"
          operator: In
          values: ["low-intensity"]
      nodeClassRef:
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenUnderutilized

2. The Carbon Controller Logic

The controller acts as the bridge. Below is a conceptual implementation of a controller loop (written in Python/Go pseudocode) that updates the NodePool based on a threshold.

def reconcile_carbon_data():
    # 1. Fetch current grid intensity for the region (e.g., us-east-1)
    intensity = carbon_api.get_intensity("us-east-1") 
    THRESHOLD = 250 # gCO2eq/kWh

    # 2. Determine status
    status = "low-intensity" if intensity < THRESHOLD else "high-intensity"

    # 3. Update Karpenter NodePool Label
    # If intensity is high, we might want to restrict the NodePool 
    # to only highly efficient instance types or scale it to zero for batch jobs.
    k8s_client.patch_nodepool("carbon-aware-pool", {
        "spec": {
            "template": {
                "spec": {
                    "requirements": [
                        {"key": "my-org/carbon-status", "operator": "In", "values": [status]}
                    ]
                }
            }
        }
    })

3. Workload Scheduling with Tolerations

To ensure only carbon-flexible workloads are affected, we use Taints and Tolerations. Non-critical batch jobs will have a toleration for high-intensity nodes, while critical API services will ignore these constraints.

apiVersion: batch/v1
kind: Job
metadata:
  name: ml-training-epoch
spec:
  template:
    spec:
      tolerations:
      - key: "my-org/carbon-status"
        operator: "Equal"
        value: "low-intensity"
        effect: "NoSchedule" # Only run when the grid is green
      containers:
      - name: trainer
        image: nvidia/cuda:12.0-base

---

Best Practices and Considerations

Avoiding “Scheduling Flapping”

Grid intensity can be volatile. If your controller updates the NodePool every time the intensity fluctuates by 1%, you will trigger constant node churn (terminating and re-provisioning nodes), which consumes more energy than it saves.
* Recommendation: Implement Hysteresis. Use a wide buffer between the “High” and “Low” thresholds and a minimum update interval (e.g., 30 minutes).

Embodied vs. Operational Carbon

While shifting workloads saves operational carbon, frequent node cycling increases wear on hardware and ignores the embodied carbon (the CO2 emitted during the manufacturing of the server).
* Action: Use Karpenter’s consolidationPolicy: WhenUnderutilized to ensure that when you do spin up “green” nodes, they are packed efficiently.

Security Considerations

• API Secret Management: Carbon API keys should be stored in AWS Secrets Manager or HashiCorp Vault and injected into the controller via External Secrets Operator.
• RBAC: The Carbon Controller requires a ClusterRole with patch permissions on nodepools.karpenter.sh. Limit this scope strictly to the specific NodePools intended for carbon-aware scaling.

---

Real-World Use Cases

1. “Green” AI/ML Training

Large Language Models (LLMs) and deep learning tasks are highly compute-intensive. By using Karpenter to provision GPU instances (like p4d.24xlarge) only when the grid intensity is below $300gCO_2eq/kWh$, organizations can reduce the carbon footprint of a training run by up to 40% in regions with high renewable penetration like the Nordics or parts of the US West Coast.

2. CI/CD and Batch Processing

Github Actions runners or Jenkins agents are ephemeral. These workloads are ideal for temporal shifting. If a CI build isn’t urgent, the Carbon Controller can “pause” the NodePool by setting requirements that no available instance can meet, effectively queuing the pods until the grid clears.

Performance Metrics to Track

To measure success, integrate the following metrics into your Grafana dashboards:
* Average Carbon Intensity ($gCO_2eq/kWh$): Tracked against cluster load.
* Carbon Avoided ($kgCO_2$): Calculated as: $(Intensity_{Default} – Intensity_{Actual}) \times Energy_{Consumed}$.
* Compute Efficiency: CPU/RAM utilization per gram of CO2.

---

Conclusion

Carbon-aware scheduling is the next frontier of cloud-native engineering. By combining the rapid provisioning capabilities of Karpenter with real-time grid telemetry, we move away from passive sustainability reporting toward active environmental stewardship.

Key Takeaways:
* Karpenter is the Enabler: Its group-less architecture makes it uniquely suited for the dynamic constraints required by carbon-aware computing.
* Labels are the Interface: Use K8s labels to bridge the gap between external grid data and internal scheduling logic.
* Prioritize Efficiency: Carbon-aware isn’t just about when you run, but what you run on. Always prefer high-efficiency architectures like ARM64/Graviton when available.

As the Green Software Foundation continues to mature the Carbon Aware SDK, expect to see these patterns become integrated into standard DevOps toolchains, making “GreenOps” as common as FinOps is today.