Chapter 9: CI/CD and GitOps

Haiyue

2025年9月23日

23min

Chapter 9: CI/CD and GitOps

Learning Objectives

Master Kubernetes integration with CI/CD pipelines
Learn to implement GitOps using ArgoCD
Understand continuous deployment strategies for containerized applications
Become proficient in automated releases and rollbacks

Key Concepts

CI/CD Overview

🔄 正在渲染 Mermaid 图表...

flowchart LR
    subgraph CI["Continuous Integration"]
        Code[Code Commit] --> Build[Build]
        Build --> Test[Test]
        Test --> Scan[Security Scan]
        Scan --> Push[Push Image]
    end

    subgraph CD["Continuous Deployment"]
        Push --> Deploy[Deploy]
        Deploy --> Verify[Verify]
        Verify --> Monitor[Monitor]
    end

    Monitor -->|Feedback| Code

GitOps Principles

🔄 正在渲染 Mermaid 图表...

mindmap
  root((GitOps))
    Declarative
      Infrastructure as Code
      Desired state description
      Version control
    Versioned
      Git as single source of truth
      Traceable changes
      Audit logs
    Automated
      Auto sync
      Auto heal
      Continuous reconciliation
    Observable
      State visibility
      Drift detection
      Alert notification

Traditional CI/CD vs GitOps

🔄 正在渲染 Mermaid 图表...

flowchart TB
    subgraph Traditional["Traditional CI/CD"]
        T1[CI Pipeline] -->|kubectl apply| T2[Kubernetes]
        T3[Requires cluster credentials]
        T4[Push mode]
    end

    subgraph GitOps["GitOps Mode"]
        G1[CI Pipeline] -->|Update| G2[Git Repository]
        G3[ArgoCD/Flux] -->|Pull| G2
        G3 -->|Sync| G4[Kubernetes]
        G5[No external credentials needed]
        G6[Pull mode]
    end

GitHub Actions CI

Basic CI Pipeline

# .github/workflows/ci.yaml
name: CI Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Set up Go
      uses: actions/setup-go@v4
      with:
        go-version: '1.21'

    - name: Run tests
      run: |
        go test -v -race -coverprofile=coverage.out ./...

    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        files: ./coverage.out

  lint:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Run golangci-lint
      uses: golangci/golangci-lint-action@v3
      with:
        version: latest

  build:
    needs: [test, lint]
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
    - uses: actions/checkout@v4

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Log in to Container Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=ref,event=branch
          type=ref,event=pr
          type=sha,prefix=
          type=raw,value=latest,enable={{is_default_branch}}

    - name: Build and push
      uses: docker/build-push-action@v5
      with:
        context: .
        push: true
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha
        cache-to: type=gha,mode=max

  security-scan:
    needs: build
    runs-on: ubuntu-latest
    steps:
    - name: Run Trivy vulnerability scanner
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: '${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}'
        format: 'sarif'
        output: 'trivy-results.sarif'

    - name: Upload Trivy scan results
      uses: github/codeql-action/upload-sarif@v2
      with:
        sarif_file: 'trivy-results.sarif'

Multi-stage Dockerfile

# Dockerfile
# Build stage
FROM golang:1.21-alpine AS builder

WORKDIR /app

# Dependency caching
COPY go.mod go.sum ./
RUN go mod download

# Build
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o /app/server ./cmd/server

# Runtime stage
FROM gcr.io/distroless/static-debian11

COPY --from=builder /app/server /server

USER nonroot:nonroot

EXPOSE 8080

ENTRYPOINT ["/server"]

ArgoCD GitOps

ArgoCD Architecture

🔄 正在渲染 Mermaid 图表...

flowchart TB
    subgraph GitRepo["Git Repository"]
        Manifests[Kubernetes Manifests<br/>Helm Charts<br/>Kustomize]
    end

    subgraph ArgoCD["ArgoCD"]
        API[API Server]
        Repo[Repo Server]
        Controller[Application Controller]
        Redis[Redis]

        Repo -->|Pull| GitRepo
        Controller -->|Monitor| Repo
        Controller -->|Sync| K8s
    end

    subgraph K8s["Kubernetes Cluster"]
        Apps[Application Resources]
    end

    User[User] -->|UI/CLI| API
    API --> Controller

Installing ArgoCD

# Create namespace
kubectl create namespace argocd

# Install ArgoCD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for Pods to be ready
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

# Get initial password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# Port forward to access UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Install CLI
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
chmod +x argocd && sudo mv argocd /usr/local/bin/

# Login
argocd login localhost:8080

Creating Application

# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default

  source:
    repoURL: https://github.com/myorg/myapp-manifests.git
    targetRevision: HEAD
    path: overlays/production

  destination:
    server: https://kubernetes.default.svc
    namespace: production

  syncPolicy:
    automated:
      prune: true           # Delete resources not in Git
      selfHeal: true        # Auto heal drift
      allowEmpty: false     # Don't allow empty applications
    syncOptions:
    - CreateNamespace=true  # Auto create namespace
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  # Health check
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas  # Ignore HPA-managed replicas

  # Notification
  info:
  - name: url
    value: https://myapp.example.com

Kustomize Structure

myapp-manifests/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── development/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   │       └── replicas.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   │       ├── replicas.yaml
│   │       └── resources.yaml
│   └── production/
│       ├── kustomization.yaml
│       └── patches/
│           ├── replicas.yaml
│           ├── resources.yaml
│           └── ingress.yaml
└── components/
    ├── monitoring/
    │   └── kustomization.yaml
    └── security/
        └── kustomization.yaml

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- deployment.yaml
- service.yaml
- configmap.yaml

commonLabels:
  app.kubernetes.io/name: myapp

---
# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi

---
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: production

resources:
- ../../base

components:
- ../../components/monitoring

images:
- name: myapp
  newName: ghcr.io/myorg/myapp
  newTag: v1.2.3

patches:
- path: patches/replicas.yaml
- path: patches/resources.yaml
- path: patches/ingress.yaml

configMapGenerator:
- name: myapp-config
  behavior: merge
  literals:
  - LOG_LEVEL=info
  - ENVIRONMENT=production

---
# overlays/production/patches/replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 5

ArgoCD ApplicationSet

# Auto-generate multi-environment Applications
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp-environments
  namespace: argocd
spec:
  generators:
  # List generator
  - list:
      elements:
      - env: development
        namespace: dev
        replicas: "1"
      - env: staging
        namespace: staging
        replicas: "2"
      - env: production
        namespace: production
        replicas: "5"

  template:
    metadata:
      name: 'myapp-{{env}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/myapp-manifests.git
        targetRevision: HEAD
        path: 'overlays/{{env}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true

Multi-cluster Deployment

# Add cluster
# argocd cluster add <context-name>

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp-multi-cluster
  namespace: argocd
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          env: production

  template:
    metadata:
      name: 'myapp-{{name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/myapp-manifests.git
        targetRevision: HEAD
        path: overlays/production
      destination:
        server: '{{server}}'
        namespace: production
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Deployment Strategies

Rolling Update

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%        # Maximum 25% extra Pods
      maxUnavailable: 25%  # Maximum 25% Pods unavailable
  template:
    spec:
      containers:
      - name: myapp
        image: myapp:v2
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Blue-Green Deployment

# Blue environment (current version)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
  labels:
    app: myapp
    version: blue
spec:
  replicas: 5
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: myapp
        image: myapp:v1

---
# Green environment (new version)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-green
  labels:
    app: myapp
    version: green
spec:
  replicas: 5
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: myapp
        image: myapp:v2

---
# Service switching
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
    version: green  # Switch to green environment
  ports:
  - port: 80
    targetPort: 8080

Canary Deployment (Argo Rollouts)

# Install Argo Rollouts
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# Install kubectl plugin
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64 && sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 10
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2
        ports:
        - containerPort: 8080
  strategy:
    canary:
      # Canary steps
      steps:
      - setWeight: 10        # 10% traffic to new version
      - pause: {duration: 5m} # Pause 5 minutes for observation
      - setWeight: 30
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {duration: 5m}
      - setWeight: 80
      - pause: {duration: 5m}

      # Traffic management
      canaryService: myapp-canary
      stableService: myapp-stable

      # Analysis template
      analysis:
        templates:
        - templateName: success-rate
        startingStep: 1
        args:
        - name: service-name
          value: myapp-canary

      # Auto rollback
      maxSurge: "25%"
      maxUnavailable: 0

---
# Analysis template
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 1m
    count: 5
    successCondition: result[0] >= 0.95
    failureCondition: result[0] < 0.90
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[5m]))
          /
          sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))

# Monitor Rollout status
kubectl argo rollouts get rollout myapp -w

# Manually promote
kubectl argo rollouts promote myapp

# Abort rollback
kubectl argo rollouts abort myapp

# Rollback to previous version
kubectl argo rollouts undo myapp

Complete CI/CD Example

Complete Pipeline

# .github/workflows/cicd.yaml
name: CI/CD Pipeline

on:
  push:
    branches: [main]
    tags: ['v*']

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}
  MANIFEST_REPO: myorg/myapp-manifests

jobs:
  ci:
    runs-on: ubuntu-latest
    outputs:
      image-tag: ${{ steps.meta.outputs.version }}

    steps:
    - uses: actions/checkout@v4

    - name: Run tests
      run: make test

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Log in to Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=semver,pattern={{version}}
          type=sha,prefix=

    - name: Build and push
      uses: docker/build-push-action@v5
      with:
        context: .
        push: true
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}

    - name: Scan image
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: '${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }}'
        exit-code: '1'
        severity: 'CRITICAL,HIGH'

  cd-staging:
    needs: ci
    runs-on: ubuntu-latest
    environment: staging

    steps:
    - name: Checkout manifests repo
      uses: actions/checkout@v4
      with:
        repository: ${{ env.MANIFEST_REPO }}
        token: ${{ secrets.MANIFEST_REPO_TOKEN }}
        path: manifests

    - name: Update image tag
      run: |
        cd manifests/overlays/staging
        kustomize edit set image myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ needs.ci.outputs.image-tag }}

    - name: Commit and push
      run: |
        cd manifests
        git config user.name "GitHub Actions"
        git config user.email "actions@github.com"
        git add .
        git commit -m "chore: update staging to ${{ needs.ci.outputs.image-tag }}"
        git push

    - name: Wait for ArgoCD sync
      run: |
        # Wait for ArgoCD to complete sync
        sleep 60

    - name: Verify deployment
      run: |
        # Run integration tests
        curl -f https://staging.myapp.example.com/health

  cd-production:
    needs: [ci, cd-staging]
    runs-on: ubuntu-latest
    environment: production

    steps:
    - name: Checkout manifests repo
      uses: actions/checkout@v4
      with:
        repository: ${{ env.MANIFEST_REPO }}
        token: ${{ secrets.MANIFEST_REPO_TOKEN }}
        path: manifests

    - name: Update image tag
      run: |
        cd manifests/overlays/production
        kustomize edit set image myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ needs.ci.outputs.image-tag }}

    - name: Create Pull Request
      uses: peter-evans/create-pull-request@v5
      with:
        path: manifests
        token: ${{ secrets.MANIFEST_REPO_TOKEN }}
        commit-message: 'chore: update production to ${{ needs.ci.outputs.image-tag }}'
        title: 'Deploy ${{ needs.ci.outputs.image-tag }} to production'
        body: |
          Automated PR to deploy version ${{ needs.ci.outputs.image-tag }} to production.

          ## Changes
          - Image: `${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ needs.ci.outputs.image-tag }}`

          ## Checklist
          - [ ] Staging verification passed
          - [ ] Reviewed changes
        branch: deploy/production-${{ needs.ci.outputs.image-tag }}

Notifications

# ArgoCD notification configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  template.app-deployed: |
    message: |
      Application {{.app.metadata.name}} is now {{.app.status.sync.status}}.
      Revision: {{.app.status.sync.revision}}
  template.app-health-degraded: |
    message: |
      Application {{.app.metadata.name}} health has degraded.
      Health Status: {{.app.status.health.status}}
  trigger.on-deployed: |
    - when: app.status.sync.status == 'Synced' and app.status.health.status == 'Healthy'
      send: [app-deployed]
  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [app-health-degraded]

---
# Subscribe to notifications
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  annotations:
    notifications.argoproj.io/subscribe.on-deployed.slack: deployments
    notifications.argoproj.io/subscribe.on-health-degraded.slack: alerts

Rollback Strategies

ArgoCD Rollback

# View history
argocd app history myapp

# Rollback to specific revision
argocd app rollback myapp <revision>

# Rollback to previous revision
argocd app rollback myapp

# Rollback via Git
git revert HEAD
git push origin main
# ArgoCD will auto sync

Helm Rollback

# View Release history
helm history myapp -n production

# Rollback to specific version
helm rollback myapp 3 -n production

# Rollback to previous version
helm rollback myapp -n production

Argo Rollouts Rollback

# Abort current release and rollback
kubectl argo rollouts abort myapp

# Rollback to previous version
kubectl argo rollouts undo myapp

# Rollback to specific version
kubectl argo rollouts undo myapp --to-revision=3

Practical Exercise

Complete GitOps Workflow

Project structure:
├── application/              # Application code repository
│   ├── src/
│   ├── Dockerfile
│   └── .github/workflows/
│       └── ci.yaml
│
├── manifests/                # Manifests repository
│   ├── base/
│   ├── overlays/
│   │   ├── development/
│   │   ├── staging/
│   │   └── production/
│   └── argocd/
│       ├── applications/
│       └── projects/
│
└── infrastructure/           # Infrastructure repository
    ├── terraform/
    └── clusters/

# manifests/argocd/projects/default.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: myapp
  namespace: argocd
spec:
  description: My Application Project

  # Allowed source repositories
  sourceRepos:
  - https://github.com/myorg/myapp-manifests.git

  # Allowed destination clusters and namespaces
  destinations:
  - namespace: '*'
    server: https://kubernetes.default.svc

  # Allowed cluster resources
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace

  # Namespace resource blacklist
  namespaceResourceBlacklist:
  - group: ''
    kind: ResourceQuota
  - group: ''
    kind: LimitRange

  # Roles
  roles:
  - name: developer
    policies:
    - p, proj:myapp:developer, applications, get, myapp/*, allow
    - p, proj:myapp:developer, applications, sync, myapp/*, allow
    groups:
    - dev-team

CI/CD Best Practices

Single responsibility: CI handles build/test, CD handles deployment
Immutable images: Each build generates unique tag
Declarative configuration: Use Git to manage all configurations
Progressive delivery: Use canary or blue-green deployment
Auto rollback: Automatic rollback on failure detection
Environment isolation: Development/test/production use separate configurations
Security scanning: Integrate security scanning in CI

Summary

Through this chapter, you should have mastered:

CI pipeline: Build, test, scan, push images
GitOps principles: Declarative, versioned, automated
ArgoCD: Application, ApplicationSet, sync policies
Deployment strategies: Rolling update, blue-green, canary
Rollback mechanisms: ArgoCD, Helm, Argo Rollouts rollback

In the next chapter, we will learn about production environment best practices, mastering enterprise-grade Kubernetes cluster planning and operations.

Argo Tutorial Series

Chapter 2: Argo CD Getting Started

Chapter 3: Argo CD Advanced Configuration

Chapter 4: Argo Workflows Fundamentals

Chapter 5: Argo Workflows Advanced Features

Argo Tutorial Series

Chapter 2: Argo CD Getting Started

Chapter 3: Argo CD Advanced Configuration

Chapter 4: Argo Workflows Fundamentals

Chapter 5: Argo Workflows Advanced Features

Chapter 9: CI/CD and GitOps

Chapter 9: CI/CD and GitOps

Key Concepts

CI/CD Overview

GitOps Principles

Traditional CI/CD vs GitOps

GitHub Actions CI

Basic CI Pipeline

Multi-stage Dockerfile

ArgoCD GitOps

ArgoCD Architecture

Installing ArgoCD

Creating Application

Kustomize Structure

ArgoCD ApplicationSet

Multi-cluster Deployment

Deployment Strategies

Rolling Update

Blue-Green Deployment

Canary Deployment (Argo Rollouts)

Complete CI/CD Example

Complete Pipeline

Notifications

Rollback Strategies

ArgoCD Rollback

Helm Rollback

Argo Rollouts Rollback

Practical Exercise

Complete GitOps Workflow

Summary