Chapter 11: Monitoring, Debugging, and Troubleshooting

Learning Objectives

1. Master comprehensive monitoring strategies for Lambda functions
2. Learn log analysis with CloudWatch Logs
3. Understand AWS X-Ray distributed tracing
4. Master Lambda Insights performance monitoring
5. Learn local and remote debugging techniques
6. Understand common troubleshooting methods

11.1 CloudWatch Monitoring

11.1.1 Basic Metrics Monitoring

CloudWatch provides automatic metrics:

• Invocations: Number of times function is invoked
• Duration: Execution time for each invocation
• Errors: Number of invocations that result in errors
• Throttles: Number of throttled invocations
• Concurrent Executions: Number of concurrent executions
• Iterator Age: For stream-based invocations

Custom metrics implementation:

• Create MetricsCollector class for custom metrics
• Record processing time, business metrics, errors
• Use PerformanceMonitor for operation tracking
• Implement HealthChecker for dependency monitoring
• Batch metric publishing to reduce API calls

11.1.2 CloudWatch Dashboard Configuration

Dashboard components:

• Overview Widgets: Total invocations, errors, duration, concurrency
• Lambda Function Metrics: Per-function invocations, errors, duration charts
• Error Rate Monitoring: Error percentage calculation and trending
• Performance Metrics: Cold start analysis, memory utilization
• Business Metrics: Custom application-specific metrics

CDK dashboard creation:

• Create comprehensive monitoring dashboard
• Configure performance-specific dashboard
• Set up business metrics dashboard
• Use GraphWidget for time-series data
• Use SingleValueWidget for current values

11.2 CloudWatch Logs Analysis

11.2.1 Structured Logging

Structured logging benefits:

• Machine-parseable JSON format
• Searchable and filterable fields
• Correlation IDs for request tracking
• Consistent log formatting
• Integration with log analytics tools

Implementation:

• StructuredLogger: JSON-formatted log output
• RequestTracker: Track request lifecycle
• Correlation IDs: Link related log entries
• Context Information: Function name, version, memory
• Business Events: Domain-specific log events

11.2.2 CloudWatch Logs Insights Queries

Common query patterns:

-- Query all error logs
fields @timestamp, level, message, error_type, correlation_id
| filter level = "ERROR"
| sort @timestamp desc

-- Analyze request latency distribution
fields @timestamp, duration_ms
| filter ispresent(duration_ms)
| stats avg(duration_ms), max(duration_ms) by bin(5m)

-- Find slow database operations
fields @timestamp, database_table, duration_ms
| filter ispresent(duration_ms) and duration_ms > 1000
| sort duration_ms desc

-- Error type statistics
fields @timestamp, error_type
| filter ispresent(error_type)
| stats count() by error_type

11.3 AWS X-Ray Distributed Tracing

11.3.1 X-Ray Integration

X-Ray capabilities:

• Service Map: Visualize service dependencies
• Trace Analysis: End-to-end request flow
• Performance Insights: Identify bottlenecks
• Error Analysis: Root cause identification
• Annotations: Searchable metadata
• Metadata: Additional context information

Implementation:

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

# Auto-patch AWS SDK
patch_all()

@xray_recorder.capture('operation_name')
def my_function():
    # Add annotations (indexed)
    segment = xray_recorder.current_segment()
    segment.put_annotation('user_id', user_id)

    # Add metadata (not indexed)
    segment.put_metadata('details', data)

    # Create subsegments
    with xray_recorder.in_subsegment('database_query'):
        # Database operation
        pass

11.3.2 CDK X-Ray Configuration

Enable X-Ray tracing:

function = _lambda.Function(
    self, "Function",
    runtime=_lambda.Runtime.PYTHON_3_9,
    handler="index.handler",
    code=_lambda.Code.from_asset("lambda"),
    tracing=_lambda.Tracing.ACTIVE  # Enable X-Ray
)

11.4 Lambda Insights Monitoring

11.4.1 Lambda Insights Overview

Lambda Insights provides:

• System Metrics: CPU, memory, disk, network
• Performance Metrics: Cold starts, initialization time
• Runtime Metrics: Garbage collection, thread count
• Correlation: Metrics correlated with traces
• Dashboards: Pre-built visualization dashboards

11.4.2 Enabling Lambda Insights

Add Lambda Insights layer:

insights_layer_arn = f"arn:aws:lambda:{region}:580247275435:layer:LambdaInsightsExtension:14"

function = _lambda.Function(
    self, "Function",
    runtime=_lambda.Runtime.PYTHON_3_9,
    handler="index.handler",
    code=_lambda.Code.from_asset("lambda"),
    layers=[
        _lambda.LayerVersion.from_layer_version_arn(
            self, "InsightsLayer",
            layer_version_arn=insights_layer_arn
        )
    ]
)

11.5 Debugging Techniques

11.5.1 Local Debugging

Local debugging tools:

• SAM CLI: Test functions locally
• Lambda Docker images: Run in containers
• Mock AWS services: LocalStack for testing
• IDE integration: VSCode, PyCharm debugging
• Environment variables: Match production config

11.5.2 Remote Debugging

Remote debugging approaches:

• CloudWatch Logs: Real-time log monitoring
• X-Ray Traces: Distributed request tracing
• Lambda Test Events: Invoke with sample data
• API Gateway Test: Test integrated endpoints
• CloudWatch Insights: Query and analyze logs

11.5.3 Common Issues and Solutions

Cold Start Issues:

• Use provisioned concurrency for critical paths
• Optimize initialization code
• Reduce deployment package size
• Use Lambda layers for dependencies

Timeout Issues:

• Increase function timeout limit
• Optimize long-running operations
• Use asynchronous processing
• Check external service latency

Memory Issues:

• Monitor memory usage metrics
• Increase memory allocation
• Optimize data processing
• Clean up large objects

Permission Errors:

• Review IAM policies
• Check resource-based policies
• Verify VPC configuration
• Test with AWS Policy Simulator

11.6 Chapter Summary

Effective monitoring and debugging are essential for maintaining reliable serverless applications. Use these tools and techniques to ensure your Lambda functions perform optimally in production.

Extended Reading

• CloudWatch Logs Insights Query Syntax
• AWS X-Ray Developer Guide
• Lambda Insights Documentation
• Troubleshooting Lambda Functions