II. 4. Cluster Architecture: Leases in Kubernetes

4.Leases in Kubernetes

Leases in Kubernetes

In Kubernetes, leases are a mechanism used for coordinating actions across distributed components, ensuring resources are managed and system-critical functions are executed reliably. Leases are particularly valuable for actions like node heartbeats and leader election among components in high-availability setups.

Key Uses of Leases in Kubernetes

1. Node Heartbeats:

Kubernetes uses leases to track node availability. Each Node in the cluster has a corresponding Lease object located in the kube-node-lease namespace. The kubelet on each node regularly updates this Lease by modifying its spec.renewTime field, which acts as a timestamp of the node’s last "heartbeat." The control plane reads this timestamp to confirm that a node is active. This helps Kubernetes quickly detect and respond to node failures or unavailability.

2. Leader Election

In a distributed setup, Kubernetes relies on leader election to prevent multiple instances of critical components (like kube-controller-manager or kube-scheduler) from running simultaneously, which could lead to conflicts. Through leases, Kubernetes ensures that only one instance of these components is active at any time. In high-availability (HA) configurations, only the elected "leader" will actively operate, while other instances remain on standby until they become the leader. “leader” will actively operate, while other instances remain on standby until they become the leader.

3. API Server Identity (from Kubernetes v1.26)

Starting with Kubernetes v1.26, each kube-apiserver instance can publish its identity using leases. This feature doesn’t directly affect functionality but allows tracking and discovering how many kube-apiserver instances are running. Leases for each API server instance are stored in the kube-system namespace, with names that include a SHA-256 hash of each instance’s hostname. These leases are useful for coordination and could support additional features in the future that require knowing the identities of running API server instances.

To view the API server leases, you can use the following command:

kubectl -n kube-system get lease -l apiserver.kubernetes.io/identity=kube-apiserver

kubectl -n kube-system get lease apiserver-07a5ea9b9b072c4a5f3d1c3702 -o yaml

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  creationTimestamp: "2023-07-02T13:16:48Z"
  labels:
    apiserver.kubernetes.io/identity: kube-apiserver
    kubernetes.io/hostname: master-1
  name: apiserver-07a5ea9b9b072c4a5f3d1c3702
  namespace: kube-system
  resourceVersion: "334899"
  uid: 90870ab5-1ba9-4523-b215-e4d4e662acb1
spec:
  holderIdentity: apiserver-07a5ea9b9b072c4a5f3d1c3702_0c8914f7-0f35-440e-8676-7844977d3a05
  leaseDurationSeconds: 3600
  renewTime: "2023-07-04T21:58:48.065888Z"

The SHA256 hash used in the lease name is based on the OS hostname as seen by that API server. Each kube-apiserver should be configured to use a hostname that is unique within the cluster. New instances of kube-apiserver that use the same hostname will take over existing Leases using a new holder identity, as opposed to instantiating new Lease objects. You can check the hostname used by kube-apisever by checking the value of the kubernetes.io/hostname label: kubectl -n kube-system get lease -l apiserver.kubernetes.io/identity=kube-apiserver

4. Custom Workload Coordination

Kubernetes leases aren’t restricted to core components; they can be used by custom workloads too. If you’re developing a controller or another component that requires a single active instance (a “leader”), you can create and manage leases to coordinate roles within your application. For example, you could name a Lease based on your application name (e.g., example-foo) to avoid confusion, especially in shared clusters where multiple instances or applications could create Lease objects. By following a unique naming convention or using a hash, you ensure that different workloads won’t interfere with each other’s leases.

Important Considerations

• Expiry and Garbage Collection: Expired leases are automatically cleaned up by new kube-apiserver instances if they detect that a corresponding instance no longer exists. This happens after one hour by default, ensuring unused or stale leases are removed.
• Disabling API Server Identity Leases: You can disable API server identity leases by turning off the APIServerIdentity feature gate if you don't need this functionality.

Leases provide Kubernetes with a reliable way to maintain consistency and coordination across distributed components, ensuring only one instance of critical components is active, monitoring node availability, and allowing custom workloads to use a similar coordination mechanism.