Chapter 6: Docker Compose Orchestration

Haiyue9/1/25About 6 min

Chapter 6: Docker Compose Orchestration

Learning Objectives

Understand the concept and importance of service orchestration
Master writing docker-compose.yml files
Learn to deploy and manage multi-container applications
Become proficient in using Compose to set up development environments

Docker Compose is Docker's official orchestration tool for defining and running multi-container Docker applications. It uses a YAML file to configure the application's services, and then with a single command, you can create and start all the services from the configuration.

Core Concepts of Compose:

Service: A component of the application, such as a database, web server, etc.
Project: A complete application composed of a set of related services.
Container: A running instance of a service.

Compose File Structure

version: '3.8'  # Compose file format version

services:       # Define services
  web:
    image: nginx:alpine
    ports:
      - "80:80"
  
  database:
    image: postgres:13
    environment:
      - POSTGRES_DB=mydb

volumes:        # Define volumes
  data-volume:

networks:       # Define networks
  app-network:
    driver: bridge

Docker Compose Installation

Linux System Installation

# Download Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/download/v2.21.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

# Add execution permissions
sudo chmod +x /usr/local/bin/docker-compose

# Create a symbolic link (optional)
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

# Verify installation
docker-compose --version

Installation with pip

# Install with pip (Python environment)
pip install docker-compose

# Verify installation
docker-compose --version

Writing a Compose File

Basic Syntax

# docker-compose.yml
version: '3.8'

services:
  # Web service
  web:
    image: nginx:alpine
    container_name: my-nginx
    ports:
      - "8080:80"
    volumes:
      - ./html:/usr/share/nginx/html:ro
      - ./conf/nginx.conf:/etc/nginx/nginx.conf:ro
    restart: unless-stopped
    depends_on:
      - api
    networks:
      - frontend

  # API service
  api:
    build:
      context: ./api
      dockerfile: Dockerfile
    container_name: my-api
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - DATABASE_URL=postgresql://user:pass@db:5432/mydb
    volumes:
      - ./api:/usr/src/app
      - /usr/src/app/node_modules
    depends_on:
      - db
      - redis
    networks:
      - frontend
      - backend

  # Database service
  db:
    image: postgres:13-alpine
    container_name: my-postgres
    environment:
      POSTGRES_DB: mydb
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    ports:
      - "5432:5432"
    restart: unless-stopped
    networks:
      - backend

  # Redis cache
  redis:
    image: redis:alpine
    container_name: my-redis
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data
    ports:
      - "6379:6379"
    restart: unless-stopped
    networks:
      - backend

# Volume definitions
volumes:
  postgres_data:
    driver: local
  redis_data:
    driver: local

# Network definitions
networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
    internal: true

Environment Variable Configuration

# .env file
POSTGRES_DB=myapp
POSTGRES_USER=admin
POSTGRES_PASSWORD=secret123
API_PORT=3000
WEB_PORT=8080

# Using environment variables in docker-compose.yml
version: '3.8'

services:
  web:
    image: nginx:alpine
    ports:
      - "${WEB_PORT}:80"
  
  api:
    build: ./api
    ports:
      - "${API_PORT}:3000"
    environment:
      - DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB}
  
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}

Build Configuration

services:
  web:
    build:
      context: ./web
      dockerfile: Dockerfile.prod
      args:
        - NODE_ENV=production
        - API_URL=http://api:3000
      target: production
      cache_from:
        - node:16-alpine
        - my-app:latest
    image: my-web-app:latest

Common Compose Commands

Basic Operation Commands

# Start services (in the background)
docker-compose up -d

# Start specific services
docker-compose up web db

# View running status
docker-compose ps

# View service logs
docker-compose logs
docker-compose logs -f web  # Follow logs of a specific service
docker-compose logs --tail=100 api  # Show the last 100 lines

# Stop services
docker-compose stop

# Stop and remove containers
docker-compose down

# Stop and remove containers, networks, and volumes
docker-compose down -v

# Restart services
docker-compose restart
docker-compose restart web

Management and Debugging Commands

# Build or rebuild services
docker-compose build
docker-compose build --no-cache web

# Pull service images
docker-compose pull

# Execute a command in a service
docker-compose exec web bash
docker-compose exec db psql -U postgres -d mydb

# Run a one-off command
docker-compose run --rm web curl http://api:3000/health

# View service configuration
docker-compose config

# Validate the compose file
docker-compose config --quiet

# Scale service instances
docker-compose up --scale web=3 --scale api=2

# View port mappings
docker-compose port web 80

Project Management Commands

# Specify a project name
docker-compose -p myproject up -d

# Specify a compose file
docker-compose -f docker-compose.prod.yml up -d

# Use multiple compose files
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# View projects
docker-compose ls

# Pause/resume services
docker-compose pause
docker-compose unpause

# Send a signal to a service
docker-compose kill -s SIGUSR1 web

Practical Cases

LAMP Stack Application

# docker-compose.yml
version: '3.8'

services:
  web:
    image: httpd:2.4-alpine
    container_name: apache-server
    ports:
      - "80:80"
    volumes:
      - ./www:/usr/local/apache2/htdocs
      - ./apache/httpd.conf:/usr/local/apache2/conf/httpd.conf:ro
    depends_on:
      - php
    networks:
      - lamp-network

  php:
    build:
      context: ./php
      dockerfile: Dockerfile
    container_name: php-fpm
    volumes:
      - ./www:/var/www/html
    depends_on:
      - mysql
    networks:
      - lamp-network

  mysql:
    image: mysql:8.0
    container_name: mysql-server
    environment:
      MYSQL_ROOT_PASSWORD: rootpass
      MYSQL_DATABASE: lampdb
      MYSQL_USER: lampuser
      MYSQL_PASSWORD: lamppass
    volumes:
      - mysql_data:/var/lib/mysql
      - ./mysql/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    ports:
      - "3306:3306"
    networks:
      - lamp-network

  phpmyadmin:
    image: phpmyadmin:latest
    container_name: phpmyadmin
    environment:
      PMA_HOST: mysql
      PMA_USER: root
      PMA_PASSWORD: rootpass
    ports:
      - "8080:80"
    depends_on:
      - mysql
    networks:
      - lamp-network

volumes:
  mysql_data:

networks:
  lamp-network:
    driver: bridge

Microservices Architecture

# docker-compose.microservices.yml
version: '3.8'

services:
  # API Gateway
  gateway:
    image: nginx:alpine
    container_name: api-gateway
    ports:
      - "80:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - user-service
      - order-service
      - product-service
    networks:
      - frontend

  # User service
  user-service:
    build: ./services/user
    container_name: user-service
    environment:
      - DATABASE_URL=postgresql://postgres:password@user-db:5432/users
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - user-db
      - redis
    networks:
      - frontend
      - user-backend

  # Order service
  order-service:
    build: ./services/order
    container_name: order-service
    environment:
      - DATABASE_URL=postgresql://postgres:password@order-db:5432/orders
      - USER_SERVICE_URL=http://user-service:3000
    depends_on:
      - order-db
    networks:
      - frontend
      - order-backend

  # Product service
  product-service:
    build: ./services/product
    container_name: product-service
    environment:
      - DATABASE_URL=postgresql://postgres:password@product-db:5432/products
    depends_on:
      - product-db
    networks:
      - frontend
      - product-backend

  # Database services
  user-db:
    image: postgres:13-alpine
    container_name: user-database
    environment:
      POSTGRES_DB: users
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - user_db_data:/var/lib/postgresql/data
    networks:
      - user-backend

  order-db:
    image: postgres:13-alpine
    container_name: order-database
    environment:
      POSTGRES_DB: orders
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - order_db_data:/var/lib/postgresql/data
    networks:
      - order-backend

  product-db:
    image: postgres:13-alpine
    container_name: product-database
    environment:
      POSTGRES_DB: products
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - product_db_data:/var/lib/postgresql/data
    networks:
      - product-backend

  # Shared services
  redis:
    image: redis:alpine
    container_name: redis-cache
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data
    networks:
      - user-backend

  # Monitoring services
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
    networks:
      - monitoring

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus
    networks:
      - monitoring

volumes:
  user_db_data:
  order_db_data:
  product_db_data:
  redis_data:
  grafana_data:

networks:
  frontend:
    driver: bridge
  user-backend:
    driver: bridge
    internal: true
  order-backend:
    driver: bridge
    internal: true
  product-backend:
    driver: bridge
    internal: true
  monitoring:
    driver: bridge

Development Environment Configuration

# docker-compose.dev.yml
version: '3.8'

services:
  # Frontend development environment
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile.dev
    container_name: frontend-dev
    ports:
      - "3000:3000"
    volumes:
      - ./frontend:/app
      - /app/node_modules
    environment:
      - CHOKIDAR_USEPOLLING=true
      - REACT_APP_API_URL=http://localhost:8000
    command: npm start

  # Backend development environment
  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile.dev
    container_name: backend-dev
    ports:
      - "8000:8000"
    volumes:
      - ./backend:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
      - DATABASE_URL=postgresql://developer:devpass@postgres:5432/devdb
      - REDIS_URL=redis://redis:6379
    depends_on:
      - postgres
      - redis
    command: npm run dev

  # Development database
  postgres:
    image: postgres:13-alpine
    container_name: postgres-dev
    ports:
      - "5432:5432"
    environment:
      POSTGRES_DB: devdb
      POSTGRES_USER: developer
      POSTGRES_PASSWORD: devpass
    volumes:
      - postgres_dev_data:/var/lib/postgresql/data
      - ./database/init-dev.sql:/docker-entrypoint-initdb.d/init.sql:ro

  # Development cache
  redis:
    image: redis:alpine
    container_name: redis-dev
    ports:
      - "6379:6379"
    volumes:
      - redis_dev_data:/data

  # Database management tool
  adminer:
    image: adminer
    container_name: adminer-dev
    ports:
      - "8080:8080"
    depends_on:
      - postgres

volumes:
  postgres_dev_data:
  redis_dev_data:

Advanced Features

Health Check Configuration

services:
  web:
    image: nginx:alpine
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

  api:
    build: ./api
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 1m30s
      timeout: 10s
      retries: 3

Resource Limits

services:
  web:
    image: nginx:alpine
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s

Configuration and Secret Management

services:
  app:
    image: my-app:latest
    configs:
      - source: app_config
        target: /etc/app/config.yml
        mode: 0644
    secrets:
      - db_password
      - api_key

configs:
  app_config:
    external: true

secrets:
  db_password:
    external: true
  api_key:
    external: true

Multi-environment Configuration

# Base configuration
# docker-compose.yml
version: '3.8'
services:
  web:
    image: nginx:alpine
    ports:
      - "80:80"

# Development environment override
# docker-compose.override.yml
version: '3.8'
services:
  web:
    volumes:
      - ./src:/usr/share/nginx/html
    environment:
      - DEBUG=true

# Production environment configuration
# docker-compose.prod.yml
version: '3.8'
services:
  web:
    restart: unless-stopped
    environment:
      - DEBUG=false
    deploy:
      replicas: 3

Using different environments:

# Development environment (automatically uses override file)
docker-compose up -d

# Production environment
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Test environment
docker-compose -f docker-compose.yml -f docker-compose.test.yml up -d

Performance Optimization and Best Practices

Build Optimization

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      cache_from:
        - my-app:latest
        - my-app:cache
      target: production
    image: my-app:latest

Startup Order Control

# Use healthcheck to control startup order
services:
  db:
    image: postgres:13
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  app:
    image: my-app:latest
    depends_on:
      db:
        condition: service_healthy

Log Management

services:
  web:
    image: nginx:alpine
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
        labels: "production_status"
        env: "NODE_ENV"

Docker Compose Best Practices

Version Control: Keep compose files under Git version control.
Environment Isolation: Use different compose files to manage different environments.
Health Checks: Configure health checks for critical services.
Resource Limits: Set appropriate resource limits for production environments.
Log Management: Configure appropriate log drivers and rotation policies.
Security Configuration: Use secrets to manage sensitive information.

Important Notes

Compose is primarily for single-host orchestration; for cluster environments, consider Kubernetes.
Production environments need to consider high availability and load balancing.
Volume backup and recovery strategies are important.
Network security configuration is crucial in production environments.

Summary

By completing this chapter, you should have mastered:

Compose Basics: Understood the role and core concepts of Docker Compose.
File Writing: Proficient in writing docker-compose.yml configuration files.
Service Orchestration: Able to orchestrate complex multi-container applications.
Environment Management: Mastered multi-environment configuration and management techniques.
Best Practices: Understood optimization strategies for production use.

In the next chapter, we will learn about container monitoring and logging to establish a complete operational monitoring system.

Chapter 6: Docker Compose Orchestration

Chapter 6: Docker Compose Orchestration

Knowledge Points

What is Docker Compose

Compose File Structure

Docker Compose Installation

Linux System Installation

Installation with pip

Writing a Compose File

Basic Syntax

Environment Variable Configuration

Build Configuration

Common Compose Commands

Basic Operation Commands

Management and Debugging Commands

Project Management Commands

Practical Cases

LAMP Stack Application

Microservices Architecture

Development Environment Configuration

Advanced Features

Health Check Configuration

Resource Limits

Configuration and Secret Management

Multi-environment Configuration

Performance Optimization and Best Practices

Build Optimization

Startup Order Control

Log Management

Summary