Chapter 5: Configuration Management and Storage

Knowledge Points

Why Configuration Management is Needed

In containerized applications, separating configuration from code is a best practice. Kubernetes provides ConfigMap and Secret to manage configuration data.

flowchart LR
    subgraph Traditional["Traditional Approach"]
        Code1[Code] --> Config1[Hard-coded Config]
        Config1 --> Image1[Image]
        Note1[Configuration changes<br/>require image rebuild]
    end

    subgraph Kubernetes["Kubernetes Approach"]
        Code2[Code] --> Image2[Image]
        Config2[ConfigMap/Secret]
        Image2 --> Pod[Pod]
        Config2 --> Pod
        Note2[Config separated from image<br/>Dynamic config updates]
    end

Storage System Overview

mindmap
  root((Kubernetes Storage))
    Configuration Data
      ConfigMap
        Application Config
        Environment Variables
        Config Files
      Secret
        Passwords
        Certificates
        API Keys
    Persistent Storage
      PersistentVolume
        Static Provisioning
        Dynamic Provisioning
      PersistentVolumeClaim
        User Request
        Bind PV
      StorageClass
        Storage Type
        Dynamic Creation
    Ephemeral Storage
      emptyDir
      hostPath

ConfigMap

ConfigMap is used to store non-sensitive configuration data.

Creating ConfigMap

# Method 1: Create from literal values
kubectl create configmap app-config \
  --from-literal=APP_ENV=production \
  --from-literal=APP_DEBUG=false \
  --from-literal=LOG_LEVEL=info

# Method 2: Create from file
echo "server.port=8080
server.host=0.0.0.0
database.pool.size=10" > app.properties

kubectl create configmap app-config-file --from-file=app.properties

# Method 3: Create from directory (containing multiple files)
mkdir config
echo "database_url=postgres://localhost/mydb" > config/database.conf
echo "redis_url=redis://localhost:6379" > config/cache.conf
kubectl create configmap app-config-dir --from-file=config/

# View created ConfigMap
kubectl get configmap
kubectl describe configmap app-config
kubectl get configmap app-config -o yaml

Creating ConfigMap with YAML

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  # Simple key-value pairs
  APP_ENV: "production"
  APP_DEBUG: "false"
  LOG_LEVEL: "info"

  # Multi-line configuration file
  app.properties: |
    server.port=8080
    server.host=0.0.0.0
    database.pool.size=10

  # JSON configuration
  config.json: |
    {
      "database": {
        "host": "localhost",
        "port": 5432
      },
      "cache": {
        "enabled": true,
        "ttl": 3600
      }
    }

  # Nginx configuration example
  nginx.conf: |
    server {
        listen 80;
        server_name localhost;

        location / {
            root /usr/share/nginx/html;
            index index.html;
        }

        location /api {
            proxy_pass http://backend:8080;
        }
    }

Using ConfigMap

As Environment Variables

apiVersion: v1
kind: Pod
metadata:
  name: config-env-pod
spec:
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "env && sleep 3600"]
    env:
    # Reference a single key
    - name: APP_ENVIRONMENT
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: APP_ENV
    - name: DEBUG_MODE
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: APP_DEBUG
          optional: true  # Don't error if key doesn't exist
    # Reference all keys
    envFrom:
    - configMapRef:
        name: app-config
        optional: false

As Volume Mount

apiVersion: v1
kind: Pod
metadata:
  name: config-volume-pod
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - name: config-volume
      mountPath: /etc/nginx/conf.d
      readOnly: true
    - name: app-config-volume
      mountPath: /app/config
  volumes:
  # Mount entire ConfigMap
  - name: config-volume
    configMap:
      name: nginx-config
  # Mount specific keys
  - name: app-config-volume
    configMap:
      name: app-config
      items:
      - key: config.json
        path: settings.json  # Rename file
      - key: app.properties
        path: app.properties
        mode: 0644  # Set file permissions

Subpath Mount (without overwriting directory)

apiVersion: v1
kind: Pod
metadata:
  name: subpath-pod
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    # Use subPath to mount only a single file without overwriting entire directory
    - name: config-volume
      mountPath: /etc/nginx/nginx.conf
      subPath: nginx.conf
  volumes:
  - name: config-volume
    configMap:
      name: nginx-config

ConfigMap Hot Reload

apiVersion: apps/v1
kind: Deployment
metadata:
  name: config-reload-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
      annotations:
        # Trigger redeployment via annotation
        configHash: "{{ .Values.configHash }}"
    spec:
      containers:
      - name: app
        image: nginx
        volumeMounts:
        - name: config
          mountPath: /etc/nginx/conf.d
      volumes:
      - name: config
        configMap:
          name: nginx-config

# Update ConfigMap
kubectl edit configmap app-config

# Or use kubectl apply
kubectl apply -f updated-configmap.yaml

# Note: Volume-mounted ConfigMaps update automatically (may have delay)
# Environment variables don't auto-update, need to restart Pod

# Force restart Deployment to apply new config
kubectl rollout restart deployment config-reload-demo

Secret

Secret is used to store sensitive data such as passwords, tokens, certificates, etc.

Secret Types

flowchart TB
    subgraph SecretTypes["Secret Types"]
        Opaque["Opaque<br/>Generic type"]
        TLS["kubernetes.io/tls<br/>TLS certificates"]
        DockerConfig["kubernetes.io/dockerconfigjson<br/>Docker registry auth"]
        BasicAuth["kubernetes.io/basic-auth<br/>Basic authentication"]
        SSHAuth["kubernetes.io/ssh-auth<br/>SSH key"]
        Token["kubernetes.io/service-account-token<br/>ServiceAccount token"]
    end

Creating Secret

# Method 1: Create from literal values (Opaque type)
kubectl create secret generic db-secret \
  --from-literal=username=admin \
  --from-literal=password='S3cr3t!Pass'

# Method 2: Create from files
echo -n 'admin' > ./username.txt
echo -n 'S3cr3t!Pass' > ./password.txt
kubectl create secret generic db-secret-file \
  --from-file=username=./username.txt \
  --from-file=password=./password.txt

# Method 3: Create TLS Secret
kubectl create secret tls tls-secret \
  --cert=path/to/tls.crt \
  --key=path/to/tls.key

# Method 4: Create Docker registry auth Secret
kubectl create secret docker-registry my-registry \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=password \
  --docker-email=user@example.com

# View Secret
kubectl get secrets
kubectl describe secret db-secret
# Note: data content is base64 encoded

# Decode Secret value
kubectl get secret db-secret -o jsonpath='{.data.password}' | base64 -d

Creating Secret with YAML

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
# Values in data must be base64 encoded
data:
  username: YWRtaW4=          # echo -n 'admin' | base64
  password: UzNjcjN0IVBhc3M=  # echo -n 'S3cr3t!Pass' | base64

---
# Using stringData allows plain text (automatically converted to base64)
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials-plain
type: Opaque
stringData:
  username: admin
  password: "S3cr3t!Pass"

---
# TLS Secret
apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: kubernetes.io/tls
data:
  tls.crt: <base64-encoded-cert>
  tls.key: <base64-encoded-key>

Using Secret

apiVersion: v1
kind: Pod
metadata:
  name: secret-demo-pod
spec:
  containers:
  - name: app
    image: nginx
    env:
    # Method 1: Reference a single key
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password
    # Method 2: Reference all keys
    envFrom:
    - secretRef:
        name: db-credentials
        optional: false
    # Method 3: Volume mount
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secret-volume
    secret:
      secretName: db-credentials
      defaultMode: 0400  # Set file permissions

Using Docker Registry Secret

apiVersion: v1
kind: Pod
metadata:
  name: private-image-pod
spec:
  containers:
  - name: app
    image: registry.example.com/my-app:v1.0
  imagePullSecrets:
  - name: my-registry

---
# Or configure in ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
imagePullSecrets:
- name: my-registry

Persistent Storage

Storage Architecture

flowchart TB
    subgraph User["User Perspective"]
        Pod[Pod] --> PVC[PersistentVolumeClaim]
    end

    subgraph Admin["Administrator Perspective"]
        PVC --> PV[PersistentVolume]
        SC[StorageClass] -.-> PV
    end

    subgraph Backend["Storage Backend"]
        PV --> NFS[NFS]
        PV --> Cloud[Cloud Storage<br/>AWS EBS/GCP PD/Azure Disk]
        PV --> Local[Local Storage]
        PV --> Ceph[Ceph/GlusterFS]
    end

Volume Types

# emptyDir - Ephemeral storage, data lost when Pod is deleted
apiVersion: v1
kind: Pod
metadata:
  name: emptydir-pod
spec:
  containers:
  - name: writer
    image: busybox
    command: ["sh", "-c", "echo 'Hello' > /data/message && sleep 3600"]
    volumeMounts:
    - name: shared-data
      mountPath: /data
  - name: reader
    image: busybox
    command: ["sh", "-c", "cat /data/message && sleep 3600"]
    volumeMounts:
    - name: shared-data
      mountPath: /data
  volumes:
  - name: shared-data
    emptyDir: {}
    # Or use memory storage
    # emptyDir:
    #   medium: Memory
    #   sizeLimit: 100Mi

---
# hostPath - Mount host directory
apiVersion: v1
kind: Pod
metadata:
  name: hostpath-pod
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - name: host-data
      mountPath: /data
  volumes:
  - name: host-data
    hostPath:
      path: /var/data
      type: DirectoryOrCreate  # Directory, File, Socket, etc.

PersistentVolume (PV)

PV is a piece of storage in the cluster, pre-created by administrators or dynamically provisioned via StorageClass.

# Static PV creation
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs
  labels:
    type: nfs
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteMany      # RWX: Multi-node read-write
  persistentVolumeReclaimPolicy: Retain  # Retain/Delete/Recycle
  storageClassName: nfs-storage
  nfs:
    server: 192.168.1.100
    path: /exports/data

---
# Local storage PV
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-local
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce      # RWO: Single-node read-write
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node-1

---
# AWS EBS PV
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-ebs
spec:
  capacity:
    storage: 50Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: gp2
  awsElasticBlockStore:
    volumeID: vol-0123456789abcdef0
    fsType: ext4

PersistentVolumeClaim (PVC)

PVC is a user’s request for storage, Kubernetes automatically binds an appropriate PV.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: nfs-storage
  # Optional: select specific PV
  selector:
    matchLabels:
      type: nfs

---
# Using PVC in Pod
apiVersion: v1
kind: Pod
metadata:
  name: pvc-pod
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - name: data
      mountPath: /var/www/html
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-pvc

Access Mode Descriptions

Reclaim Policies

flowchart LR
    PVC[PVC Deleted] --> Policy{Reclaim Policy}
    Policy -->|Retain| Retain[Keep data<br/>Manual cleanup]
    Policy -->|Delete| Delete[Auto delete PV and storage]
    Policy -->|Recycle| Recycle[Clear data<br/>PV reusable<br/>Deprecated]

StorageClass

StorageClass defines storage types and supports dynamic PV creation.

# NFS StorageClass (requires NFS Provisioner)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: nfs-subdir-external-provisioner
parameters:
  archiveOnDelete: "false"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

---
# AWS EBS StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ebs
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  fsType: ext4
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

---
# Local storage StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

# View StorageClass
kubectl get storageclass

# Set default StorageClass
kubectl patch storageclass nfs-client \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# PVC using default StorageClass (without specifying storageClassName)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  # Not specifying storageClassName will use default StorageClass
  # Or explicitly specify
  storageClassName: fast-ebs

Expanding PVC Capacity

# Ensure StorageClass supports expansion
kubectl get storageclass fast-ebs -o yaml | grep allowVolumeExpansion

# Edit PVC to increase capacity
kubectl patch pvc my-pvc -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

# View expansion status
kubectl get pvc my-pvc
kubectl describe pvc my-pvc

StatefulSet Storage

StatefulSet provides stable persistent storage for each Pod.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: root-password
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      storageClassName: fast-ebs
      resources:
        requests:
          storage: 50Gi

# StatefulSet creates separate PVC for each Pod
kubectl get pvc
# NAME           STATUS   VOLUME                                     CAPACITY
# data-mysql-0   Bound    pvc-xxx-0                                  50Gi
# data-mysql-1   Bound    pvc-xxx-1                                  50Gi
# data-mysql-2   Bound    pvc-xxx-2                                  50Gi

# When deleting StatefulSet, PVCs are not automatically deleted
kubectl delete statefulset mysql
kubectl get pvc  # PVCs still exist

# Need to manually delete PVCs
kubectl delete pvc -l app=mysql

Practical Exercise

Deploying Web Application with Configuration

# Create ConfigMap for Nginx configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    events {
        worker_connections 1024;
    }
    http {
        server {
            listen 80;
            server_name localhost;

            location / {
                root /usr/share/nginx/html;
                index index.html;
            }

            location /api {
                proxy_pass http://backend:8080;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
            }
        }
    }

  index.html: |
    <!DOCTYPE html>
    <html>
    <head><title>Welcome</title></head>
    <body>
        <h1>Hello from Kubernetes!</h1>
        <p>Environment: ${APP_ENV}</p>
    </body>
    </html>

---
# Create Secret for sensitive information
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  api-key: "super-secret-api-key-12345"
  db-password: "database-password"

---
# Create PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-logs-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard

---
# Deploy Nginx
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.20
        ports:
        - containerPort: 80
        env:
        - name: APP_ENV
          value: "production"
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: api-key
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
        - name: html-content
          mountPath: /usr/share/nginx/html/index.html
          subPath: index.html
        - name: logs
          mountPath: /var/log/nginx
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "200m"
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config
      - name: html-content
        configMap:
          name: nginx-config
      - name: logs
        persistentVolumeClaim:
          claimName: nginx-logs-pvc

Deploying Database Cluster

# MySQL configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
data:
  my.cnf: |
    [mysqld]
    bind-address = 0.0.0.0
    max_connections = 200
    innodb_buffer_pool_size = 256M
    slow_query_log = 1
    slow_query_log_file = /var/log/mysql/slow.log
    long_query_time = 2

---
# MySQL password
apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
type: Opaque
stringData:
  root-password: "RootP@ssw0rd"
  user-password: "UserP@ssw0rd"
  replication-password: "ReplicaP@ssw0rd"

---
# Headless Service
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  ports:
  - port: 3306
    name: mysql
  clusterIP: None
  selector:
    app: mysql

---
# MySQL StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        ports:
        - containerPort: 3306
          name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: root-password
        - name: MYSQL_DATABASE
          value: myapp
        - name: MYSQL_USER
          value: appuser
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: user-password
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
        - name: config
          mountPath: /etc/mysql/conf.d
        - name: logs
          mountPath: /var/log/mysql
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1"
        livenessProbe:
          exec:
            command:
            - mysqladmin
            - ping
            - -h
            - localhost
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command:
            - mysql
            - -h
            - localhost
            - -u
            - root
            - -p${MYSQL_ROOT_PASSWORD}
            - -e
            - "SELECT 1"
          initialDelaySeconds: 5
          periodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: mysql-config
      - name: logs
        emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi

# Deploy MySQL
kubectl apply -f mysql.yaml

# Verify deployment
kubectl get statefulset mysql
kubectl get pods -l app=mysql
kubectl get pvc

# Connect to MySQL
kubectl exec -it mysql-0 -- mysql -uroot -p

# View configuration
kubectl exec mysql-0 -- cat /etc/mysql/conf.d/my.cnf

Configuration Hot Reload Example

# Using Reloader to auto-restart Pods
# Install: helm install reloader stakater/reloader

apiVersion: apps/v1
kind: Deployment
metadata:
  name: config-reload-app
  annotations:
    # Auto-restart when ConfigMap changes
    configmap.reloader.stakater.com/reload: "app-config"
    # Auto-restart when Secret changes
    secret.reloader.stakater.com/reload: "app-secrets"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: reload-demo
  template:
    metadata:
      labels:
        app: reload-demo
    spec:
      containers:
      - name: app
        image: nginx
        envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets

Common Issue Troubleshooting

PVC Status Pending

# Check PVC status
kubectl describe pvc my-pvc

# Common causes:
# 1. No matching PV
kubectl get pv
# Check if capacity, access mode, StorageClass match

# 2. StorageClass doesn't exist or misconfigured
kubectl get storageclass
kubectl describe storageclass <name>

# 3. Storage backend issues
# Check Provisioner logs
kubectl logs -n kube-system -l app=nfs-provisioner

Secret/ConfigMap Updates Not Taking Effect

# Check Pod usage method
# Environment variable method: Need to restart Pod
kubectl rollout restart deployment my-app

# Volume method: Auto-updates but with delay (up to 1 minute)
# Using subPath mount will NOT auto-update!

# Force trigger update
kubectl patch deployment my-app \
  -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"$(date +%s)\"}}}}}"

Storage Performance Issues

# Check storage type and IOPS
kubectl get storageclass -o yaml

# Monitor PV usage
kubectl describe pv <pv-name>

# Consider using higher performance storage types
# AWS: gp3, io1, io2
# GCP: pd-ssd
# Azure: Premium SSD

Summary

Through this chapter, you should have mastered:

• ConfigMap: Store non-sensitive configuration, supports environment variables and file mounting
• Secret: Securely store sensitive data, multiple types support different scenarios
• PV/PVC: Understand persistent storage request and binding mechanisms
• StorageClass: Implement dynamic provisioning of storage
• Best practices: Secure usage of configuration management and storage

In the next chapter, we’ll learn about Helm package management and how to use Helm to simplify the deployment and management of complex applications.