Chapter 3: Workload Controllers

Controller Overview

What is a Controller

Controllers are a core concept in Kubernetes, continuously monitoring cluster state through a control loop pattern to ensure the actual state matches the desired state.

flowchart LR
    subgraph ControlLoop[Control Loop]
        OBSERVE[Observe<br/>Current State] --> COMPARE[Compare<br/>Desired vs Actual]
        COMPARE --> DIFF{Consistent?}
        DIFF -->|No| ACT[Take Action<br/>Move toward desired state]
        DIFF -->|Yes| OBSERVE
        ACT --> OBSERVE
    end

    SPEC[Desired State<br/>Spec] --> COMPARE
    STATUS[Actual State<br/>Status] --> OBSERVE

Controller Types Overview

flowchart TB
    subgraph Controllers[Kubernetes Controllers]
        subgraph Stateless[Stateless Applications]
            DEPLOY[Deployment<br/>Stateless app deployment]
            RS[ReplicaSet<br/>Replica management]
        end

        subgraph Stateful[Stateful Applications]
            STS[StatefulSet<br/>Stateful app deployment]
        end

        subgraph NodeLevel[Node Level]
            DS[DaemonSet<br/>Node daemon]
        end

        subgraph Batch[Batch Processing]
            JOB[Job<br/>One-time task]
            CRON[CronJob<br/>Scheduled task]
        end
    end

    DEPLOY --> RS
    RS --> POD[Pod]
    STS --> POD
    DS --> POD
    JOB --> POD
    CRON --> JOB

Deployment Deep Dive

Deployment Architecture

flowchart TB
    subgraph Deployment
        DEPLOY_SPEC[Deployment Spec<br/>replicas: 3<br/>strategy: RollingUpdate]
    end

    subgraph ReplicaSet_v2[ReplicaSet v2 Current]
        RS2_SPEC[ReplicaSet Spec<br/>replicas: 3]
        POD1[Pod 1]
        POD2[Pod 2]
        POD3[Pod 3]
    end

    subgraph ReplicaSet_v1[ReplicaSet v1 Old Version]
        RS1_SPEC[ReplicaSet Spec<br/>replicas: 0]
    end

    DEPLOY_SPEC --> RS2_SPEC
    DEPLOY_SPEC -.Keep history.-> RS1_SPEC
    RS2_SPEC --> POD1
    RS2_SPEC --> POD2
    RS2_SPEC --> POD3

    style ReplicaSet_v2 fill:#c8e6c9
    style ReplicaSet_v1 fill:#ffcdd2

Creating a Deployment

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.24
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi

# Create Deployment
kubectl apply -f nginx-deployment.yaml

# View Deployment status
kubectl get deployment nginx-deployment

# View detailed information
kubectl describe deployment nginx-deployment

# View associated ReplicaSet
kubectl get rs -l app=nginx

# View all Pods
kubectl get pods -l app=nginx

# Quick create Deployment (without YAML)
kubectl create deployment nginx --image=nginx:1.24 --replicas=3

Rolling Updates

sequenceDiagram
    participant D as Deployment
    participant RS1 as ReplicaSet v1
    participant RS2 as ReplicaSet v2

    Note over D: Update image version

    D->>RS2: Create new ReplicaSet
    RS2->>RS2: Start 1 new Pod

    RS1->>RS1: Terminate 1 old Pod
    RS2->>RS2: Start 1 new Pod

    RS1->>RS1: Terminate 1 old Pod
    RS2->>RS2: Start 1 new Pod

    RS1->>RS1: Terminate last old Pod

    Note over RS2: 3 new Pods running
    Note over RS1: 0 Pods (keep history)

Update Strategy Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 10
  strategy:
    type: RollingUpdate  # or Recreate
    rollingUpdate:
      maxSurge: 25%        # Maximum percentage above desired replicas
      maxUnavailable: 25%  # Maximum percentage of unavailable replicas

  # Other configuration...
  minReadySeconds: 5       # Wait time after Pod ready
  progressDeadlineSeconds: 600  # Update timeout
  revisionHistoryLimit: 10     # Number of historical versions to keep

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # Maximum 1 extra Pod
    maxUnavailable: 0  # No unavailable Pods allowed

strategy:
  type: Recreate

Performing Update Operations

# Method 1: Directly modify image
kubectl set image deployment/nginx-deployment nginx=nginx:1.25

# Method 2: Edit Deployment
kubectl edit deployment nginx-deployment

# Method 3: Apply new YAML
kubectl apply -f nginx-deployment-v2.yaml

# View update status
kubectl rollout status deployment/nginx-deployment

# View update history
kubectl rollout history deployment/nginx-deployment

# View specific version details
kubectl rollout history deployment/nginx-deployment --revision=2

# Pause update
kubectl rollout pause deployment/nginx-deployment

# Resume update
kubectl rollout resume deployment/nginx-deployment

# Rollback to previous version
kubectl rollout undo deployment/nginx-deployment

# Rollback to specific version
kubectl rollout undo deployment/nginx-deployment --to-revision=1

Scaling

# Manual scaling
kubectl scale deployment nginx-deployment --replicas=5

# Conditional scaling
kubectl scale deployment nginx-deployment --replicas=3 --current-replicas=5

# Use HPA for autoscaling
kubectl autoscale deployment nginx-deployment --min=2 --max=10 --cpu-percent=80

ReplicaSet

ReplicaSet Working Principle

flowchart TB
    subgraph ReplicaSet[ReplicaSet Controller]
        WATCH[Monitor Pod count]
        COMPARE{Actual vs Desired?}
        CREATE[Create Pod]
        DELETE[Delete Pod]
    end

    WATCH --> COMPARE
    COMPARE -->|Less than desired| CREATE
    COMPARE -->|More than desired| DELETE
    COMPARE -->|Equal to desired| WATCH
    CREATE --> WATCH
    DELETE --> WATCH

    subgraph Pods
        P1[Pod 1]
        P2[Pod 2]
        P3[Pod 3]
    end

    CREATE -.-> Pods
    DELETE -.-> Pods

ReplicaSet Configuration

# Usually not used directly, managed by Deployment
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx-replicaset
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
    # Also supports matchExpressions
    matchExpressions:
    - key: tier
      operator: In
      values:
      - frontend
  template:
    metadata:
      labels:
        app: nginx
        tier: frontend
    spec:
      containers:
      - name: nginx
        image: nginx:1.24

StatefulSet for Stateful Applications

StatefulSet Features

flowchart TB
    subgraph StatefulSet[StatefulSet Features]
        subgraph Identity[Stable Identity]
            NAME[Fixed Pod names<br/>web-0, web-1, web-2]
            DNS[Fixed DNS names<br/>web-0.nginx.default.svc]
            STORAGE[Independent persistent storage<br/>PVC bound to Pod]
        end

        subgraph Order[Ordered Operations]
            DEPLOY_ORDER[Sequential deployment<br/>0 → 1 → 2]
            SCALE_ORDER[Sequential scaling]
            UPDATE_ORDER[Sequential update<br/>2 → 1 → 0]
        end
    end

StatefulSet Configuration

# mysql-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql  # Must associate with Headless Service
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        ports:
        - containerPort: 3306
          name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql

  # PersistentVolumeClaim template
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: standard
      resources:
        requests:
          storage: 10Gi

  # Update strategy
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0  # Partition update

  # Pod management policy
  podManagementPolicy: OrderedReady  # or Parallel

Headless Service

StatefulSet must be used with a Headless Service:

# mysql-headless-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  clusterIP: None  # Headless Service
  selector:
    app: mysql
  ports:
  - port: 3306
    targetPort: 3306

StatefulSet Network Identity

flowchart TB
    subgraph Cluster[Kubernetes Cluster]
        subgraph STS[StatefulSet: mysql]
            P0[mysql-0]
            P1[mysql-1]
            P2[mysql-2]
        end

        subgraph DNS[DNS Resolution]
            D0[mysql-0.mysql.default.svc.cluster.local]
            D1[mysql-1.mysql.default.svc.cluster.local]
            D2[mysql-2.mysql.default.svc.cluster.local]
        end

        P0 --> D0
        P1 --> D1
        P2 --> D2
    end

    CLIENT[Client] --> D0
    CLIENT --> D1
    CLIENT --> D2

Hands-on: Deploy MySQL Primary-Replica Cluster

# mysql-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
data:
  primary.cnf: |
    [mysqld]
    log-bin=mysql-bin
    server-id=1
  replica.cnf: |
    [mysqld]
    server-id=2
    read-only=1

---
# mysql-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
type: Opaque
data:
  password: cGFzc3dvcmQxMjM=  # base64 encoded password123

---
# mysql-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      initContainers:
      - name: init-mysql
        image: mysql:8.0
        command:
        - bash
        - "-c"
        - |
          set -ex
          [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          if [[ $ordinal -eq 0 ]]; then
            cp /mnt/config-map/primary.cnf /mnt/conf.d/
          else
            cp /mnt/config-map/replica.cnf /mnt/conf.d/
          fi
        volumeMounts:
        - name: conf
          mountPath: /mnt/conf.d
        - name: config-map
          mountPath: /mnt/config-map

      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        ports:
        - containerPort: 3306
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
        livenessProbe:
          exec:
            command: ["mysqladmin", "ping"]
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
          initialDelaySeconds: 5
          periodSeconds: 2

      volumes:
      - name: conf
        emptyDir: {}
      - name: config-map
        configMap:
          name: mysql-config

  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

# Deploy
kubectl apply -f mysql-configmap.yaml
kubectl apply -f mysql-secret.yaml
kubectl apply -f mysql-headless-service.yaml
kubectl apply -f mysql-statefulset.yaml

# Watch deployment order
kubectl get pods -l app=mysql -w

# Verify network identity
kubectl run -it --rm debug --image=mysql:8.0 --restart=Never -- \
  mysql -h mysql-0.mysql -u root -ppassword123 -e "SELECT @@hostname"

# Scale up
kubectl scale statefulset mysql --replicas=5

# View PVCs
kubectl get pvc -l app=mysql

DaemonSet Node Daemon

DaemonSet Features

flowchart TB
    subgraph Cluster[Kubernetes Cluster]
        subgraph DS[DaemonSet: node-exporter]
            direction LR
        end

        subgraph N1[Node 1]
            P1[node-exporter Pod]
        end

        subgraph N2[Node 2]
            P2[node-exporter Pod]
        end

        subgraph N3[Node 3]
            P3[node-exporter Pod]
        end

        subgraph N4[Node 4 Newly Added]
            P4[node-exporter Pod<br/>Auto-created]
        end
    end

    DS --> P1
    DS --> P2
    DS --> P3
    DS -.Auto-schedule.-> P4

DaemonSet Configuration

# node-exporter-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  labels:
    app: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true  # Use host network
      hostPID: true      # Share host PID namespace

      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.6.0
        args:
        - '--path.procfs=/host/proc'
        - '--path.sysfs=/host/sys'
        - '--path.rootfs=/host/root'
        ports:
        - containerPort: 9100
          hostPort: 9100
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: root
          mountPath: /host/root
          readOnly: true

      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /

      tolerations:
      - operator: Exists  # Tolerate all taints, ensure runs on all nodes

  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

Common DaemonSet Use Cases

# fluentd-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fluentd
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: containers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: containers
        hostPath:
          path: /var/lib/docker/containers

# calico-node-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: calico-node
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  template:
    metadata:
      labels:
        k8s-app: calico-node
    spec:
      hostNetwork: true
      containers:
      - name: calico-node
        image: calico/node:v3.26.0
        env:
        - name: DATASTORE_TYPE
          value: "kubernetes"
        securityContext:
          privileged: true

# csi-node-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: csi-node
spec:
  selector:
    matchLabels:
      app: csi-node
  template:
    metadata:
      labels:
        app: csi-node
    spec:
      containers:
      - name: csi-driver
        image: my-csi-driver:v1
        securityContext:
          privileged: true
        volumeMounts:
        - name: plugin-dir
          mountPath: /csi
        - name: pods-mount-dir
          mountPath: /var/lib/kubelet
          mountPropagation: "Bidirectional"
      volumes:
      - name: plugin-dir
        hostPath:
          path: /var/lib/kubelet/plugins/my-csi
      - name: pods-mount-dir
        hostPath:
          path: /var/lib/kubelet

Job and CronJob

Job for One-time Tasks

flowchart LR
    subgraph Job[Job Controller]
        SPEC[completions: 3<br/>parallelism: 2]
    end

    subgraph Execution[Execution]
        P1[Pod 1 ✓]
        P2[Pod 2 ✓]
        P3[Pod 3 Running]
    end

    Job --> P1
    Job --> P2
    Job --> P3

    style P1 fill:#c8e6c9
    style P2 fill:#c8e6c9
    style P3 fill:#fff9c4

Job Configuration

# batch-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: batch-process
spec:
  completions: 5       # Number of Pods that need to successfully complete
  parallelism: 2       # Number of Pods running in parallel
  backoffLimit: 4      # Number of failure retries
  activeDeadlineSeconds: 600  # Timeout

  template:
    spec:
      restartPolicy: Never  # Job must be Never or OnFailure
      containers:
      - name: worker
        image: busybox
        command: ['sh', '-c', 'echo "Processing batch $JOB_INDEX"; sleep 10; echo "Done"']
        env:
        - name: JOB_INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']

Job Types

# Single Pod runs to completion
spec:
  completions: 1
  parallelism: 1

# Specified number of Pods need to complete
spec:
  completions: 5
  parallelism: 2

# Parallel processing, any Pod success completes
spec:
  completions: 1
  parallelism: 3

# Each Pod has unique index
spec:
  completions: 5
  parallelism: 5
  completionMode: Indexed

CronJob for Scheduled Tasks

# backup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: database-backup
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  timeZone: "Asia/Shanghai"  # Timezone setting

  # Concurrency policy
  concurrencyPolicy: Forbid  # Allow | Forbid | Replace

  # History retention
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1

  # Starting deadline
  startingDeadlineSeconds: 200

  # Suspend schedule
  suspend: false

  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: backup
            image: mysql:8.0
            command:
            - /bin/sh
            - -c
            - |
              mysqldump -h mysql-primary -u root -p$MYSQL_PASSWORD --all-databases | \
              gzip > /backup/backup-$(date +%Y%m%d-%H%M%S).sql.gz
            env:
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: password
            volumeMounts:
            - name: backup-volume
              mountPath: /backup
          volumes:
          - name: backup-volume
            persistentVolumeClaim:
              claimName: backup-pvc

Cron Expressions

┌───────────── Minute (0 - 59)
│ ┌───────────── Hour (0 - 23)
│ │ ┌───────────── Day (1 - 31)
│ │ │ ┌───────────── Month (1 - 12)
│ │ │ │ ┌───────────── Day of Week (0 - 6, 0=Sunday)
│ │ │ │ │
* * * * *

Hands-on Practice

Blue-Green Deployment

# nginx-blue.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
      version: blue
  template:
    metadata:
      labels:
        app: nginx
        version: blue
    spec:
      containers:
      - name: nginx
        image: nginx:1.24
        ports:
        - containerPort: 80

---
# nginx-green.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
      version: green
  template:
    metadata:
      labels:
        app: nginx
        version: green
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80

---
# nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
    version: blue  # Switch to green for blue-green deployment
  ports:
  - port: 80
    targetPort: 80

# Deploy blue version
kubectl apply -f nginx-blue.yaml
kubectl apply -f nginx-service.yaml

# Deploy green version (doesn't receive traffic)
kubectl apply -f nginx-green.yaml

# Verify green version
kubectl port-forward deployment/nginx-green 8080:80

# Switch traffic to green version
kubectl patch service nginx-service -p '{"spec":{"selector":{"version":"green"}}}'

# After confirming no issues, delete blue version
kubectl delete deployment nginx-blue

Canary Release

# canary-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: nginx
      track: stable
  template:
    metadata:
      labels:
        app: nginx
        track: stable
    spec:
      containers:
      - name: nginx
        image: nginx:1.24

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-canary
spec:
  replicas: 1  # 10% traffic
  selector:
    matchLabels:
      app: nginx
      track: canary
  template:
    metadata:
      labels:
        app: nginx
        track: canary
    spec:
      containers:
      - name: nginx
        image: nginx:1.25

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx  # Matches both stable and canary
  ports:
  - port: 80

# Deploy
kubectl apply -f canary-deployment.yaml

# View traffic distribution
kubectl get pods -l app=nginx -L track

# Gradually increase canary percentage
kubectl scale deployment nginx-canary --replicas=3
kubectl scale deployment nginx-stable --replicas=7

# Complete switch
kubectl scale deployment nginx-canary --replicas=10
kubectl scale deployment nginx-stable --replicas=0

# Rename
kubectl patch deployment nginx-canary -p '{"metadata":{"name":"nginx-stable"}}'

Summary

Through this chapter, you should have mastered:

• Deployment: Declarative deployment and rolling updates for stateless applications
• ReplicaSet: Understanding the working principles of replica control
• StatefulSet: Stable identity and ordered deployment for stateful applications
• DaemonSet: Node-level daemon process management
• Job/CronJob: Configuration for one-time and scheduled tasks