mastering-aws-cli — Skillopedia

AWS CLI v2 Quick Reference A unified tool to manage AWS services from the terminal. This guide focuses on CLI v2 features, practical examples, and advanced patterns for experienced developers. Quick Start Power User Tips See Advanced Patterns for JMESPath mastery and automation tricks. Global Options | Flag | Description | |:-----|:------------| | | Use named profile from | | | Override default region (e.g., ) | | | Output: (default), , , , | | | Filter output using JMESPath expressions | | | Disable auto-pagination (first page only) | | | Check permissions without executing (EC2, etc.) | | |…

\\t'\n```\n\n### Configuration Management\n```bash\n#!/bin/bash\n# Environment-specific config\n\nENV=${1:-dev}\nCONFIG_FILE=\"config-${ENV}.json\"\n\nif [[ ! -f \"$CONFIG_FILE\" ]]; then\n echo \"Config file not found: $CONFIG_FILE\"\n exit 1\nfi\n\n# Read config\nBUCKET=$(jq -r '.bucket' \"$CONFIG_FILE\")\nREGION=$(jq -r '.region' \"$CONFIG_FILE\")\nPROFILE=$(jq -r '.profile // \"default\"' \"$CONFIG_FILE\")\n\n# Use in commands\naws s3 sync ./dist \"s3://${BUCKET}/\" \\\n --region \"$REGION\" \\\n --profile \"$PROFILE\"\n```\n\n## Useful Queries Collection\n\n```bash\n# Find untagged resources\naws ec2 describe-instances \\\n --query 'Reservations[*].Instances[?!not_null(Tags)].InstanceId'\n\n# Get total EBS storage\naws ec2 describe-volumes \\\n --query 'sum(Volumes[*].Size)' \\\n --output text\n\n# Find public S3 buckets\naws s3api list-buckets --query 'Buckets[*].Name' --output text | \\\n xargs -n1 -P4 -I{} sh -c \\\n 'aws s3api get-bucket-acl --bucket {} 2>/dev/null | grep -q AllUsers && echo {}'\n\n# Lambda functions by runtime\naws lambda list-functions \\\n --query 'Functions | group_by(@, &Runtime) | [*].{Runtime: [0].Runtime, Count: length(@)}'\n\n# Cost estimation (running instances)\naws ec2 describe-instances \\\n --filters Name=instance-state-name,Values=running \\\n --query 'Reservations[*].Instances[*].InstanceType' \\\n --output text | sort | uniq -c | sort -rn\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Use --query** | Filter on client side to reduce data transfer |\n| **Use --output text** | For scripting (avoids JSON parsing) |\n| **Use waiters** | Instead of sleep loops for async operations |\n| **Use --dry-run** | Test permissions before executing |\n| **Use aliases** | Create shortcuts for common operations |\n| **Enable debug** | Use --debug for troubleshooting |\n| **Handle pagination** | Account for paginated responses |\n| **Retry on throttling** | Implement exponential backoff |\n| **Use input files** | For complex parameters (--cli-input-json) |\n| **Parallel processing** | Use xargs/parallel for batch operations |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":13177,"content_sha256":"da65670e5b772ca8f067e22eac9a98b1b34b355a3e88b632d6f5ed211d134997"},{"filename":"references/aurora.md","content":"# Aurora & RDS\n\n## Aurora Serverless v2\n\n### Create Cluster\n```bash\n# Aurora PostgreSQL Serverless v2\naws rds create-db-cluster \\\n --db-cluster-identifier my-cluster \\\n --engine aurora-postgresql \\\n --engine-version 15.4 \\\n --master-username admin \\\n --master-user-password SecurePassword123! \\\n --serverless-v2-scaling-configuration MinCapacity=0.5,MaxCapacity=16 \\\n --db-subnet-group-name my-subnet-group \\\n --vpc-security-group-ids sg-12345678 \\\n --storage-encrypted \\\n --enable-cloudwatch-logs-exports postgresql\n\n# Aurora MySQL Serverless v2\naws rds create-db-cluster \\\n --db-cluster-identifier mysql-cluster \\\n --engine aurora-mysql \\\n --engine-version 3.04.0 \\\n --master-username admin \\\n --master-user-password SecurePassword123! \\\n --serverless-v2-scaling-configuration MinCapacity=0.5,MaxCapacity=32 \\\n --db-subnet-group-name my-subnet-group \\\n --vpc-security-group-ids sg-12345678\n```\n\n### Create Instance\n```bash\n# Primary writer instance\naws rds create-db-instance \\\n --db-instance-identifier my-primary \\\n --db-cluster-identifier my-cluster \\\n --engine aurora-postgresql \\\n --db-instance-class db.serverless\n\n# Add reader instance\naws rds create-db-instance \\\n --db-instance-identifier my-reader-1 \\\n --db-cluster-identifier my-cluster \\\n --engine aurora-postgresql \\\n --db-instance-class db.serverless\n\n# Wait for instance\naws rds wait db-instance-available --db-instance-identifier my-primary\n```\n\n## Aurora Provisioned\n\n### Create Cluster\n```bash\n# Aurora PostgreSQL with provisioned capacity\naws rds create-db-cluster \\\n --db-cluster-identifier prod-cluster \\\n --engine aurora-postgresql \\\n --engine-version 15.4 \\\n --master-username admin \\\n --master-user-password SecurePassword123! \\\n --db-subnet-group-name my-subnet-group \\\n --vpc-security-group-ids sg-12345678 \\\n --storage-encrypted \\\n --kms-key-id alias/aws/rds \\\n --backup-retention-period 7 \\\n --preferred-backup-window \"03:00-04:00\" \\\n --preferred-maintenance-window \"sun:05:00-sun:06:00\"\n\n# Create writer instance\naws rds create-db-instance \\\n --db-instance-identifier prod-primary \\\n --db-cluster-identifier prod-cluster \\\n --engine aurora-postgresql \\\n --db-instance-class db.r6g.large \\\n --publicly-accessible false\n\n# Create reader instance\naws rds create-db-instance \\\n --db-instance-identifier prod-reader-1 \\\n --db-cluster-identifier prod-cluster \\\n --engine aurora-postgresql \\\n --db-instance-class db.r6g.large \\\n --publicly-accessible false\n```\n\n## Standard RDS\n\n### Create Instance\n```bash\n# RDS PostgreSQL\naws rds create-db-instance \\\n --db-instance-identifier prod-postgres \\\n --db-instance-class db.t3.medium \\\n --engine postgres \\\n --engine-version 15.4 \\\n --allocated-storage 100 \\\n --max-allocated-storage 500 \\\n --master-username admin \\\n --master-user-password SecurePassword123! \\\n --db-subnet-group-name my-subnet-group \\\n --vpc-security-group-ids sg-12345678 \\\n --storage-encrypted \\\n --multi-az \\\n --publicly-accessible false \\\n --backup-retention-period 7\n\n# RDS MySQL\naws rds create-db-instance \\\n --db-instance-identifier prod-mysql \\\n --db-instance-class db.r6g.large \\\n --engine mysql \\\n --engine-version 8.0.35 \\\n --allocated-storage 100 \\\n --max-allocated-storage 1000 \\\n --master-username admin \\\n --master-user-password SecurePassword123! \\\n --storage-type gp3 \\\n --iops 3000 \\\n --storage-throughput 125\n```\n\n## Cluster and Instance Management\n\n### Describe and List\n```bash\n# List clusters\naws rds describe-db-clusters\n\n# Describe specific cluster\naws rds describe-db-clusters --db-cluster-identifier my-cluster\n\n# Get cluster endpoint\naws rds describe-db-clusters \\\n --db-cluster-identifier my-cluster \\\n --query 'DBClusters[0].Endpoint' \\\n --output text\n\n# Get reader endpoint\naws rds describe-db-clusters \\\n --db-cluster-identifier my-cluster \\\n --query 'DBClusters[0].ReaderEndpoint' \\\n --output text\n\n# List instances\naws rds describe-db-instances\n\n# Describe specific instance\naws rds describe-db-instances --db-instance-identifier my-primary\n```\n\n### Modify Cluster\n```bash\n# Modify scaling configuration\naws rds modify-db-cluster \\\n --db-cluster-identifier my-cluster \\\n --serverless-v2-scaling-configuration MinCapacity=1,MaxCapacity=32 \\\n --apply-immediately\n\n# Enable deletion protection\naws rds modify-db-cluster \\\n --db-cluster-identifier my-cluster \\\n --deletion-protection\n\n# Change maintenance window\naws rds modify-db-cluster \\\n --db-cluster-identifier my-cluster \\\n --preferred-maintenance-window \"sun:05:00-sun:06:00\"\n```\n\n### Modify Instance\n```bash\n# Change instance class\naws rds modify-db-instance \\\n --db-instance-identifier my-primary \\\n --db-instance-class db.r6g.xlarge \\\n --apply-immediately\n\n# Scale storage (RDS only)\naws rds modify-db-instance \\\n --db-instance-identifier prod-postgres \\\n --allocated-storage 200 \\\n --apply-immediately\n```\n\n### Delete\n```bash\n# Delete instance (skip final snapshot)\naws rds delete-db-instance \\\n --db-instance-identifier my-reader-1 \\\n --skip-final-snapshot\n\n# Delete cluster (with final snapshot)\naws rds delete-db-cluster \\\n --db-cluster-identifier my-cluster \\\n --final-db-snapshot-identifier my-cluster-final-snapshot\n\n# Delete cluster (skip final snapshot)\naws rds delete-db-cluster \\\n --db-cluster-identifier my-cluster \\\n --skip-final-snapshot\n```\n\n## Aurora Global Database\n\n### Create Global Database\n```bash\n# Create global database from existing cluster\naws rds create-global-cluster \\\n --global-cluster-identifier my-global-db \\\n --source-db-cluster-identifier arn:aws:rds:us-east-1:123456789012:cluster:my-cluster\n\n# Add secondary region\naws rds create-db-cluster \\\n --db-cluster-identifier my-cluster-eu \\\n --engine aurora-postgresql \\\n --engine-version 15.4 \\\n --global-cluster-identifier my-global-db \\\n --db-subnet-group-name eu-subnet-group \\\n --vpc-security-group-ids sg-eu-12345 \\\n --region eu-west-1\n\n# Add instance in secondary region\naws rds create-db-instance \\\n --db-instance-identifier my-cluster-eu-1 \\\n --db-cluster-identifier my-cluster-eu \\\n --engine aurora-postgresql \\\n --db-instance-class db.r6g.large \\\n --region eu-west-1\n```\n\n### Manage Global Database\n```bash\n# Describe global database\naws rds describe-global-clusters --global-cluster-identifier my-global-db\n\n# Failover to secondary (planned)\naws rds failover-global-cluster \\\n --global-cluster-identifier my-global-db \\\n --target-db-cluster-identifier arn:aws:rds:eu-west-1:123456789012:cluster:my-cluster-eu\n\n# Remove cluster from global database\naws rds remove-from-global-cluster \\\n --global-cluster-identifier my-global-db \\\n --db-cluster-identifier arn:aws:rds:eu-west-1:123456789012:cluster:my-cluster-eu\n\n# Delete global database\naws rds delete-global-cluster --global-cluster-identifier my-global-db\n```\n\n## RDS Proxy\n\n### Create Proxy\n```bash\n# Create RDS Proxy\naws rds create-db-proxy \\\n --db-proxy-name my-proxy \\\n --engine-family POSTGRESQL \\\n --auth '[{\n \"AuthScheme\": \"SECRETS\",\n \"SecretArn\": \"arn:aws:secretsmanager:us-east-1:123456789012:secret:db-creds\",\n \"IAMAuth\": \"DISABLED\"\n }]' \\\n --role-arn arn:aws:iam::123456789012:role/RDSProxyRole \\\n --vpc-subnet-ids subnet-1 subnet-2 \\\n --vpc-security-group-ids sg-12345678 \\\n --require-tls\n\n# Register target (Aurora cluster)\naws rds register-db-proxy-targets \\\n --db-proxy-name my-proxy \\\n --db-cluster-identifiers my-cluster\n\n# Register target (RDS instance)\naws rds register-db-proxy-targets \\\n --db-proxy-name my-proxy \\\n --db-instance-identifiers prod-postgres\n```\n\n### Manage Proxy\n```bash\n# Describe proxy\naws rds describe-db-proxies --db-proxy-name my-proxy\n\n# Get proxy endpoint\naws rds describe-db-proxies \\\n --db-proxy-name my-proxy \\\n --query 'DBProxies[0].Endpoint' \\\n --output text\n\n# Describe target groups\naws rds describe-db-proxy-target-groups --db-proxy-name my-proxy\n\n# Delete proxy\naws rds delete-db-proxy --db-proxy-name my-proxy\n```\n\n## Snapshots and Cloning\n\n### Manual Snapshots\n```bash\n# Create cluster snapshot\naws rds create-db-cluster-snapshot \\\n --db-cluster-snapshot-identifier my-cluster-snap-01 \\\n --db-cluster-identifier my-cluster\n\n# Create instance snapshot (RDS only)\naws rds create-db-snapshot \\\n --db-snapshot-identifier prod-postgres-snap-01 \\\n --db-instance-identifier prod-postgres\n\n# List snapshots\naws rds describe-db-cluster-snapshots --db-cluster-identifier my-cluster\n\n# Copy snapshot cross-region\naws rds copy-db-cluster-snapshot \\\n --source-db-cluster-snapshot-identifier arn:aws:rds:us-east-1:123456789012:cluster-snapshot:my-cluster-snap-01 \\\n --target-db-cluster-snapshot-identifier my-cluster-snap-01-copy \\\n --region eu-west-1\n\n# Share snapshot with another account\naws rds modify-db-cluster-snapshot-attribute \\\n --db-cluster-snapshot-identifier my-cluster-snap-01 \\\n --attribute-name restore \\\n --values-to-add 987654321098\n```\n\n### Restore from Snapshot\n```bash\n# Restore Aurora cluster\naws rds restore-db-cluster-from-snapshot \\\n --db-cluster-identifier restored-cluster \\\n --snapshot-identifier my-cluster-snap-01 \\\n --engine aurora-postgresql \\\n --db-subnet-group-name my-subnet-group \\\n --vpc-security-group-ids sg-12345678\n\n# Then create instance\naws rds create-db-instance \\\n --db-instance-identifier restored-primary \\\n --db-cluster-identifier restored-cluster \\\n --engine aurora-postgresql \\\n --db-instance-class db.serverless\n\n# Restore RDS instance\naws rds restore-db-instance-from-db-snapshot \\\n --db-instance-identifier restored-postgres \\\n --db-snapshot-identifier prod-postgres-snap-01 \\\n --db-instance-class db.t3.medium\n```\n\n### Aurora Cloning\n```bash\n# Fast clone (copy-on-write)\naws rds restore-db-cluster-to-point-in-time \\\n --source-db-cluster-identifier my-cluster \\\n --db-cluster-identifier test-clone \\\n --restore-type copy-on-write \\\n --use-latest-restorable-time\n\n# Clone to specific point in time\naws rds restore-db-cluster-to-point-in-time \\\n --source-db-cluster-identifier my-cluster \\\n --db-cluster-identifier test-clone \\\n --restore-type copy-on-write \\\n --restore-to-time \"2024-01-15T12:00:00Z\"\n```\n\n## Blue/Green Deployments\n\n### Create Blue/Green Deployment\n```bash\n# Create deployment for version upgrade\naws rds create-blue-green-deployment \\\n --blue-green-deployment-name pg-upgrade-16 \\\n --source arn:aws:rds:us-east-1:123456789012:cluster:my-cluster \\\n --target-engine-version 16.1\n\n# Create deployment with instance class change\naws rds create-blue-green-deployment \\\n --blue-green-deployment-name scale-up \\\n --source arn:aws:rds:us-east-1:123456789012:db:prod-postgres \\\n --target-db-instance-class db.r6g.xlarge\n```\n\n### Manage Blue/Green Deployment\n```bash\n# Describe deployment\naws rds describe-blue-green-deployments \\\n --blue-green-deployment-identifier bgd-12345678\n\n# Switchover (after verification)\naws rds switchover-blue-green-deployment \\\n --blue-green-deployment-identifier bgd-12345678 \\\n --switchover-timeout 300\n\n# Delete deployment (cleanup old environment)\naws rds delete-blue-green-deployment \\\n --blue-green-deployment-identifier bgd-12345678 \\\n --delete-target\n```\n\n## Parameter Groups\n\n### Create Parameter Group\n```bash\n# Create cluster parameter group\naws rds create-db-cluster-parameter-group \\\n --db-cluster-parameter-group-name my-aurora-pg-params \\\n --db-parameter-group-family aurora-postgresql15 \\\n --description \"Custom Aurora PostgreSQL parameters\"\n\n# Create instance parameter group\naws rds create-db-parameter-group \\\n --db-parameter-group-name my-postgres-params \\\n --db-parameter-group-family postgres15 \\\n --description \"Custom PostgreSQL parameters\"\n```\n\n### Modify Parameters\n```bash\n# Modify cluster parameters\naws rds modify-db-cluster-parameter-group \\\n --db-cluster-parameter-group-name my-aurora-pg-params \\\n --parameters \"ParameterName=log_statement,ParameterValue=all,ApplyMethod=immediate\" \\\n \"ParameterName=log_min_duration_statement,ParameterValue=1000,ApplyMethod=immediate\"\n\n# Apply to cluster\naws rds modify-db-cluster \\\n --db-cluster-identifier my-cluster \\\n --db-cluster-parameter-group-name my-aurora-pg-params \\\n --apply-immediately\n```\n\n## IAM Database Authentication\n\n### Enable IAM Auth\n```bash\n# Enable on cluster\naws rds modify-db-cluster \\\n --db-cluster-identifier my-cluster \\\n --enable-iam-database-authentication \\\n --apply-immediately\n\n# Enable on RDS instance\naws rds modify-db-instance \\\n --db-instance-identifier prod-postgres \\\n --enable-iam-database-authentication \\\n --apply-immediately\n```\n\n### Generate Auth Token\n```bash\n# Generate authentication token\naws rds generate-db-auth-token \\\n --hostname my-cluster.cluster-abc123.us-east-1.rds.amazonaws.com \\\n --port 5432 \\\n --username iam_user \\\n --region us-east-1\n```\n\n## Bastion Access to Aurora\n\n### Port Forwarding via SSM\n```bash\n# Get cluster endpoint\nENDPOINT=$(aws rds describe-db-clusters \\\n --db-cluster-identifier my-cluster \\\n --query 'DBClusters[0].Endpoint' \\\n --output text)\n\n# Start port forwarding through bastion\naws ssm start-session \\\n --target i-bastion-instance-id \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters \"{\\\"host\\\":[\\\"$ENDPOINT\\\"],\\\"portNumber\\\":[\\\"5432\\\"],\\\"localPortNumber\\\":[\\\"5432\\\"]}\"\n\n# In another terminal, connect via psql\npsql -h localhost -p 5432 -U admin -d mydb\n```\n\n### SSH Tunnel via SSM\n```bash\n# Configure SSH (~/.ssh/config)\n# Host i-* mi-*\n# ProxyCommand sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'\"\n\n# Create SSH tunnel\nssh -L 5432:my-cluster.cluster-abc123.us-east-1.rds.amazonaws.com:5432 \\\n ec2-user@i-bastion-instance-id -N &\n\n# Connect via tunnel\npsql -h localhost -p 5432 -U admin -d mydb\n```\n\n## Monitoring\n\n### Enable Enhanced Monitoring\n```bash\naws rds modify-db-instance \\\n --db-instance-identifier my-primary \\\n --monitoring-interval 60 \\\n --monitoring-role-arn arn:aws:iam::123456789012:role/rds-monitoring-role\n```\n\n### Enable Performance Insights\n```bash\naws rds modify-db-instance \\\n --db-instance-identifier my-primary \\\n --enable-performance-insights \\\n --performance-insights-retention-period 7\n```\n\n### CloudWatch Logs\n```bash\n# Enable log exports\naws rds modify-db-cluster \\\n --db-cluster-identifier my-cluster \\\n --cloudwatch-logs-export-configuration \\\n EnableLogTypes=postgresql,upgrade\n```\n\n## Useful Queries\n\n```bash\n# Get cluster status\naws rds describe-db-clusters \\\n --db-cluster-identifier my-cluster \\\n --query 'DBClusters[0].Status'\n\n# List all endpoints\naws rds describe-db-clusters \\\n --db-cluster-identifier my-cluster \\\n --query 'DBClusters[0].{Writer:Endpoint,Reader:ReaderEndpoint}'\n\n# Get current capacity (Serverless v2)\naws rds describe-db-clusters \\\n --db-cluster-identifier my-cluster \\\n --query 'DBClusters[0].ServerlessV2ScalingConfiguration'\n\n# Find instances in cluster\naws rds describe-db-instances \\\n --filters Name=db-cluster-id,Values=my-cluster \\\n --query 'DBInstances[*].{ID:DBInstanceIdentifier,Class:DBInstanceClass,Status:DBInstanceStatus}'\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Serverless v2** | Use MinCapacity=0.5 for dev, higher for production |\n| **Secrets Manager** | Store credentials in Secrets Manager, not scripts |\n| **Private access** | Keep PubliclyAccessible=false, use bastion/proxy |\n| **Encryption** | Enable storage encryption with KMS |\n| **Multi-AZ** | Use for production (automatic with Aurora) |\n| **RDS Proxy** | Use for Lambda or connection pooling |\n| **Global Database** | Use for cross-region disaster recovery |\n| **Blue/Green** | Use for major version upgrades |\n| **Cloning** | Use fast clones for testing |\n| **IAM Auth** | Use for temporary, rotatable credentials |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":16186,"content_sha256":"b793db7e7471c11159d79813d5479a27d723d88cb92d01e51a344df9b497a8aa"},{"filename":"references/bastion-tunneling.md","content":"# Bastion & Tunneling\n\n## SSM Session Manager\n\nThe modern, secure replacement for SSH bastion hosts. No open inbound ports required.\n\n### Prerequisites\n```bash\n# Install Session Manager plugin\n# macOS\nbrew install --cask session-manager-plugin\n\n# Verify installation\nsession-manager-plugin\n\n# Required VPC endpoints (for private subnets without NAT)\n# - ssm.region.amazonaws.com\n# - ssmmessages.region.amazonaws.com\n# - ec2messages.region.amazonaws.com\n```\n\n### Interactive Shell\n```bash\n# Start session to EC2 instance\naws ssm start-session --target i-0123456789abcdef0\n\n# Start session with specific region\naws ssm start-session \\\n --target i-0123456789abcdef0 \\\n --region us-west-2\n\n# Start session to on-premises managed instance\naws ssm start-session --target mi-0123456789abcdef0\n```\n\n### Port Forwarding (Local)\nForward a local port to a port on the remote instance.\n\n```bash\n# Forward local port to remote port\naws ssm start-session \\\n --target i-0123456789abcdef0 \\\n --document-name AWS-StartPortForwardingSession \\\n --parameters '{\"portNumber\":[\"8080\"],\"localPortNumber\":[\"8080\"]}'\n\n# RDP access (Windows)\naws ssm start-session \\\n --target i-windows-instance \\\n --document-name AWS-StartPortForwardingSession \\\n --parameters '{\"portNumber\":[\"3389\"],\"localPortNumber\":[\"33389\"]}'\n# Then: mstsc /v:localhost:33389\n\n# VNC access\naws ssm start-session \\\n --target i-linux-instance \\\n --document-name AWS-StartPortForwardingSession \\\n --parameters '{\"portNumber\":[\"5901\"],\"localPortNumber\":[\"5901\"]}'\n```\n\n### Remote Host Port Forwarding\nAccess resources *through* a bastion instance (jump host pattern).\n\n```bash\n# Access RDS through bastion\naws ssm start-session \\\n --target i-bastion-instance \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters '{\n \"host\":[\"mydb.cluster-xyz.us-east-1.rds.amazonaws.com\"],\n \"portNumber\":[\"5432\"],\n \"localPortNumber\":[\"5432\"]\n }'\n# Now connect: psql -h localhost -p 5432 -U admin mydb\n\n# Access Aurora MySQL\naws ssm start-session \\\n --target i-bastion-instance \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters '{\n \"host\":[\"aurora-cluster.cluster-xyz.us-east-1.rds.amazonaws.com\"],\n \"portNumber\":[\"3306\"],\n \"localPortNumber\":[\"3306\"]\n }'\n# Now connect: mysql -h 127.0.0.1 -P 3306 -u admin -p\n\n# Access ElastiCache Redis\naws ssm start-session \\\n --target i-bastion-instance \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters '{\n \"host\":[\"redis-cluster.xyz.cache.amazonaws.com\"],\n \"portNumber\":[\"6379\"],\n \"localPortNumber\":[\"6379\"]\n }'\n# Now connect: redis-cli -h localhost -p 6379\n\n# Access OpenSearch\naws ssm start-session \\\n --target i-bastion-instance \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters '{\n \"host\":[\"search-domain.us-east-1.es.amazonaws.com\"],\n \"portNumber\":[\"443\"],\n \"localPortNumber\":[\"9200\"]\n }'\n# Now: curl https://localhost:9200\n\n# Access internal ALB\naws ssm start-session \\\n --target i-bastion-instance \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters '{\n \"host\":[\"internal-alb-123.us-east-1.elb.amazonaws.com\"],\n \"portNumber\":[\"443\"],\n \"localPortNumber\":[\"8443\"]\n }'\n```\n\n## EKS Private Cluster Access\n\nAccess private EKS clusters through SSM without exposing the API server publicly.\n\n### Update Kubeconfig for Private Cluster\n```bash\n# Get cluster info\naws eks describe-cluster --name my-cluster \\\n --query 'cluster.{endpoint:endpoint,ca:certificateAuthority.data}'\n\n# Update kubeconfig (will fail if cluster is private)\naws eks update-kubeconfig --name my-cluster --region us-east-1\n```\n\n### Kubectl via SSM Port Forward\n```bash\n# 1. Get private endpoint\nENDPOINT=$(aws eks describe-cluster --name my-cluster \\\n --query 'cluster.endpoint' --output text)\n# Example: https://ABC123.gr7.us-east-1.eks.amazonaws.com\n\n# 2. Extract hostname\nEKS_HOST=$(echo $ENDPOINT | sed 's|https://||')\n\n# 3. Start port forwarding to EKS API (port 443)\naws ssm start-session \\\n --target i-bastion-in-eks-vpc \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters \"{\n \\\"host\\\":[\\\"$EKS_HOST\\\"],\n \\\"portNumber\\\":[\\\"443\\\"],\n \\\"localPortNumber\\\":[\\\"6443\\\"]\n }\"\n\n# 4. In another terminal, modify kubeconfig\n# Add to ~/.kube/config:\n# clusters:\n# - cluster:\n# server: https://127.0.0.1:6443\n# certificate-authority-data: \u003cbase64-ca-from-cluster>\n# name: my-cluster\n\n# 5. Run kubectl\nkubectl get nodes\nkubectl get pods -A\n```\n\n### Alternative: SSM Document for EKS\n```bash\n# Create custom SSM document for kubectl\naws ssm create-document \\\n --name \"EKS-Kubectl\" \\\n --document-type \"Session\" \\\n --content '{\n \"schemaVersion\": \"1.0\",\n \"description\": \"Run kubectl commands via SSM\",\n \"sessionType\": \"InteractiveCommands\",\n \"inputs\": {\n \"runAsEnabled\": true,\n \"runAsDefaultUser\": \"ec2-user\"\n }\n }'\n\n# Run kubectl commands directly on bastion\naws ssm start-session \\\n --target i-bastion-with-kubectl \\\n --document-name EKS-Kubectl\n```\n\n### Using SSM as SSH Proxy for EKS\n```bash\n# ~/.ssh/config addition for SSM proxy\nHost eks-bastion\n HostName i-bastion-instance-id\n User ec2-user\n ProxyCommand sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'\"\n\n# SSH to bastion, then kubectl\nssh eks-bastion\nkubectl get nodes\n```\n\n## Aurora/RDS Database Access\n\n### Direct Port Forwarding\n```bash\n# PostgreSQL (Aurora/RDS)\naws ssm start-session \\\n --target i-bastion \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters '{\n \"host\":[\"prod-db.cluster-xyz.us-east-1.rds.amazonaws.com\"],\n \"portNumber\":[\"5432\"],\n \"localPortNumber\":[\"5432\"]\n }'\n\n# Connect with psql\npsql \"host=localhost port=5432 dbname=mydb user=admin sslmode=require\"\n\n# Connect with DBeaver/pgAdmin\n# Host: localhost, Port: 5432, SSL: Required\n\n# MySQL/Aurora MySQL\naws ssm start-session \\\n --target i-bastion \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters '{\n \"host\":[\"prod-db.cluster-xyz.us-east-1.rds.amazonaws.com\"],\n \"portNumber\":[\"3306\"],\n \"localPortNumber\":[\"3306\"]\n }'\n\n# Connect with mysql client\nmysql -h 127.0.0.1 -P 3306 -u admin -p --ssl-mode=REQUIRED\n```\n\n### JDBC Connection Through Tunnel\n```bash\n# Start tunnel in background\naws ssm start-session \\\n --target i-bastion \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters '{\n \"host\":[\"aurora.cluster-xyz.us-east-1.rds.amazonaws.com\"],\n \"portNumber\":[\"5432\"],\n \"localPortNumber\":[\"5432\"]\n }' &\n\n# JDBC URL (use in application/IDE)\n# PostgreSQL: jdbc:postgresql://localhost:5432/mydb?ssl=true\n# MySQL: jdbc:mysql://localhost:3306/mydb?useSSL=true\n```\n\n### Shell Script for DB Tunnel\n```bash\n#!/bin/bash\n# db-tunnel.sh - Start database tunnel\n\nDB_HOST=\"${1:-prod-db.cluster-xyz.us-east-1.rds.amazonaws.com}\"\nDB_PORT=\"${2:-5432}\"\nLOCAL_PORT=\"${3:-5432}\"\nBASTION_ID=\"${BASTION_INSTANCE_ID:-i-0123456789abcdef0}\"\n\necho \"Starting tunnel to $DB_HOST:$DB_PORT on localhost:$LOCAL_PORT\"\naws ssm start-session \\\n --target \"$BASTION_ID\" \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters \"{\n \\\"host\\\":[\\\"$DB_HOST\\\"],\n \\\"portNumber\\\":[\\\"$DB_PORT\\\"],\n \\\"localPortNumber\\\":[\\\"$LOCAL_PORT\\\"]\n }\"\n```\n\n## SSH Access Methods\n\n### EC2 Instance Connect\nPush a temporary SSH key (valid 60 seconds).\n\n```bash\n# Push public key\naws ec2-instance-connect send-ssh-public-key \\\n --instance-id i-0123456789abcdef0 \\\n --instance-os-user ec2-user \\\n --ssh-public-key file://~/.ssh/id_rsa.pub\n\n# SSH within 60 seconds\nssh ec2-user@\u003cpublic-ip>\n\n# One-liner with AWS CLI\naws ec2-instance-connect send-ssh-public-key \\\n --instance-id i-123 \\\n --instance-os-user ec2-user \\\n --ssh-public-key file://~/.ssh/id_rsa.pub && \\\n ssh ec2-user@$(aws ec2 describe-instances --instance-ids i-123 \\\n --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)\n```\n\n### SSM as SSH Proxy\nConfigure SSH to tunnel through SSM (no public IP needed).\n\n```bash\n# ~/.ssh/config\nHost i-* mi-*\n ProxyCommand sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'\"\n User ec2-user\n IdentityFile ~/.ssh/my-key.pem\n\nHost bastion-prod\n HostName i-0123456789abcdef0\n User ec2-user\n ProxyCommand sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'\"\n\n# Usage\nssh i-0123456789abcdef0\nssh bastion-prod\n\n# SCP through SSM\nscp -o ProxyCommand=\"aws ssm start-session --target i-123 --document-name AWS-StartSSHSession --parameters portNumber=22\" \\\n myfile.txt ec2-user@i-123:/home/ec2-user/\n```\n\n### Multi-Hop SSH Through Bastion\n```bash\n# ~/.ssh/config for jump host pattern\nHost bastion\n HostName i-bastion-id\n User ec2-user\n ProxyCommand sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'\"\n\nHost private-server\n HostName 10.0.1.100\n User ec2-user\n ProxyJump bastion\n IdentityFile ~/.ssh/private-key.pem\n\n# Usage\nssh private-server\n```\n\n## Run Command (Remote Execution)\n\n### Execute Commands\n```bash\n# Run command on single instance\naws ssm send-command \\\n --instance-ids i-0123456789abcdef0 \\\n --document-name \"AWS-RunShellScript\" \\\n --parameters 'commands=[\"df -h\",\"free -m\"]'\n\n# Run on multiple instances\naws ssm send-command \\\n --instance-ids i-123 i-456 i-789 \\\n --document-name \"AWS-RunShellScript\" \\\n --parameters 'commands=[\"yum update -y\"]'\n\n# Run on instances by tag\naws ssm send-command \\\n --targets Key=tag:Environment,Values=Production \\\n --document-name \"AWS-RunShellScript\" \\\n --parameters 'commands=[\"systemctl restart nginx\"]'\n\n# Run with timeout\naws ssm send-command \\\n --instance-ids i-123 \\\n --document-name \"AWS-RunShellScript\" \\\n --parameters 'commands=[\"./long-running-script.sh\"]' \\\n --timeout-seconds 3600\n```\n\n### Get Command Output\n```bash\n# Get command invocation results\nCOMMAND_ID=$(aws ssm send-command \\\n --instance-ids i-123 \\\n --document-name \"AWS-RunShellScript\" \\\n --parameters 'commands=[\"hostname\"]' \\\n --query 'Command.CommandId' \\\n --output text)\n\n# Wait and get output\naws ssm get-command-invocation \\\n --command-id $COMMAND_ID \\\n --instance-id i-123 \\\n --query '{Status:Status,Output:StandardOutputContent}'\n\n# List all command invocations\naws ssm list-command-invocations \\\n --command-id $COMMAND_ID \\\n --details\n```\n\n### Windows Commands\n```bash\n# PowerShell command\naws ssm send-command \\\n --instance-ids i-windows-123 \\\n --document-name \"AWS-RunPowerShellScript\" \\\n --parameters 'commands=[\"Get-Process | Sort-Object CPU -Descending | Select-Object -First 10\"]'\n\n# Install Windows feature\naws ssm send-command \\\n --instance-ids i-windows-123 \\\n --document-name \"AWS-RunPowerShellScript\" \\\n --parameters 'commands=[\"Install-WindowsFeature -Name Web-Server -IncludeManagementTools\"]'\n```\n\n## Session Manager Preferences\n\n### Configure Logging\n```bash\n# Create preferences document\naws ssm update-document \\\n --name \"SSM-SessionManagerRunShell\" \\\n --document-version \"\\$LATEST\" \\\n --content '{\n \"schemaVersion\": \"1.0\",\n \"description\": \"Session Manager Preferences\",\n \"sessionType\": \"Standard_Stream\",\n \"inputs\": {\n \"s3BucketName\": \"my-session-logs-bucket\",\n \"s3KeyPrefix\": \"session-logs/\",\n \"s3EncryptionEnabled\": true,\n \"cloudWatchLogGroupName\": \"/aws/ssm/session-logs\",\n \"cloudWatchEncryptionEnabled\": true,\n \"kmsKeyId\": \"alias/session-manager-key\",\n \"runAsEnabled\": true,\n \"runAsDefaultUser\": \"ssm-user\",\n \"idleSessionTimeout\": \"20\",\n \"shellProfile\": {\n \"linux\": \"cd ~ && bash\",\n \"windows\": \"\"\n }\n }\n }'\n```\n\n### Custom Session Documents\n```bash\n# Create interactive command document\naws ssm create-document \\\n --name \"Custom-InteractiveSession\" \\\n --document-type \"Session\" \\\n --content '{\n \"schemaVersion\": \"1.0\",\n \"description\": \"Custom interactive session\",\n \"sessionType\": \"InteractiveCommands\",\n \"inputs\": {\n \"runAsEnabled\": true,\n \"runAsDefaultUser\": \"admin\"\n }\n }'\n```\n\n## Automation Patterns\n\n### Dynamic Bastion Discovery\nFind bastion instances by tag instead of hardcoding instance IDs.\n\n```bash\n# Find bastion by tag\nBASTION_ID=$(aws ec2 describe-instances \\\n --filters \"Name=tag:Role,Values=bastion\" \\\n \"Name=instance-state-name,Values=running\" \\\n --query 'Reservations[0].Instances[0].InstanceId' \\\n --output text)\n\n# Verify bastion found\nif [ \"$BASTION_ID\" = \"None\" ] || [ -z \"$BASTION_ID\" ]; then\n echo \"Error: No running bastion instance found\"\n exit 1\nfi\n\n# Use in SSM session\naws ssm start-session --target \"$BASTION_ID\"\n\n# Find by multiple tags (environment + role)\nBASTION_ID=$(aws ec2 describe-instances \\\n --filters \"Name=tag:Environment,Values=production\" \\\n \"Name=tag:Role,Values=bastion\" \\\n \"Name=instance-state-name,Values=running\" \\\n --query 'Reservations[0].Instances[0].InstanceId' \\\n --output text)\n```\n\n### Clean Shell Environment\nEnsure SSO credentials aren't overridden by static environment variables.\n\n```bash\n# Clear any static credentials before SSO login\nunset AWS_ACCESS_KEY_ID\nunset AWS_SECRET_ACCESS_KEY\nunset AWS_SESSION_TOKEN\n\n# Now SSO login works correctly\nexport AWS_PROFILE=my-sso-profile\naws sso login\n\n# Verify using SSO credentials (not static)\naws configure list\n# Should show: profile my-sso-profile, not environment variables\n\n# Full clean environment script\nclean_aws_env() {\n unset AWS_ACCESS_KEY_ID\n unset AWS_SECRET_ACCESS_KEY\n unset AWS_SESSION_TOKEN\n unset AWS_SECURITY_TOKEN\n echo \"Cleared static AWS credentials from environment\"\n}\n```\n\n### Credential Validation Patterns\nCheck if credentials are valid before proceeding.\n\n```bash\n# Validate credentials (returns 0 if valid, non-zero if expired/invalid)\nvalidate_aws_creds() {\n if aws sts get-caller-identity &>/dev/null; then\n echo \"✓ AWS credentials valid\"\n aws sts get-caller-identity --query 'Arn' --output text\n return 0\n else\n echo \"✗ AWS credentials invalid or expired\"\n return 1\n fi\n}\n\n# Auto-login if credentials expired (SSO profiles)\nensure_aws_login() {\n if ! aws sts get-caller-identity &>/dev/null; then\n echo \"Credentials expired, logging in...\"\n aws sso login --profile \"${AWS_PROFILE:-default}\"\n fi\n}\n\n# Use in scripts\nensure_aws_login\naws s3 ls\n\n# Check profile before operations\ncheck_profile() {\n local current=$(aws configure list --query 'profile' --output text 2>/dev/null)\n echo \"Current profile: ${AWS_PROFILE:-default}\"\n echo \"Account: $(aws sts get-caller-identity --query 'Account' --output text)\"\n echo \"ARN: $(aws sts get-caller-identity --query 'Arn' --output text)\"\n}\n```\n\n### Port Conflict Resolution\nHandle \"Address already in use\" errors.\n\n```bash\n# Check what's using a port\nlsof -i :5432\n\n# Find and show process on port\ncheck_port() {\n local port=$1\n local pid=$(lsof -ti :$port 2>/dev/null)\n if [ -n \"$pid\" ]; then\n echo \"Port $port in use by PID $pid:\"\n ps -p $pid -o pid,user,command\n return 1\n else\n echo \"Port $port is available\"\n return 0\n fi\n}\n\n# Kill process on port (use with caution)\nfree_port() {\n local port=$1\n local pid=$(lsof -ti :$port 2>/dev/null)\n if [ -n \"$pid\" ]; then\n echo \"Killing process $pid on port $port\"\n kill -9 $pid\n sleep 1\n fi\n}\n\n# Smart port selection (find available port)\nfind_available_port() {\n local start_port=${1:-5432}\n local port=$start_port\n while lsof -ti :$port &>/dev/null; do\n ((port++))\n done\n echo $port\n}\n\n# Example: auto-select port for DB tunnel\nLOCAL_PORT=$(find_available_port 5432)\necho \"Using port $LOCAL_PORT\"\n```\n\n### Multiple Cluster Access\nWork with multiple EKS clusters simultaneously.\n\n```bash\n# Each terminal: different cluster\n# Terminal 1 (Dev)\nexport AWS_PROFILE=dev-admin\nexport KUBECONFIG=~/.kube/config-dev\n./connect-cluster.sh dev-cluster\n\n# Terminal 2 (Staging)\nexport AWS_PROFILE=staging-admin\nexport KUBECONFIG=~/.kube/config-staging\n./connect-cluster.sh staging-cluster\n\n# Terminal 3 (Prod - read only)\nexport AWS_PROFILE=prod-readonly\nexport KUBECONFIG=~/.kube/config-prod\n./connect-cluster.sh prod-cluster\n\n# Quick cluster context switch script\nuse_cluster() {\n local env=$1\n case $env in\n dev)\n export AWS_PROFILE=dev-admin\n export KUBECONFIG=~/.kube/config-dev\n ;;\n staging)\n export AWS_PROFILE=staging-admin\n export KUBECONFIG=~/.kube/config-staging\n ;;\n prod)\n export AWS_PROFILE=prod-readonly\n export KUBECONFIG=~/.kube/config-prod\n ;;\n *)\n echo \"Unknown environment: $env\"\n return 1\n ;;\n esac\n echo \"Switched to $env environment\"\n kubectl config current-context\n}\n```\n\n### Session Cleanup on Exit\nEnsure SSM sessions are cleaned up when shell exits.\n\n```bash\n#!/bin/bash\n# ssm-connect.sh - Connect with automatic cleanup\n\nSESSION_PID=\"\"\nSSM_LOG=\"/tmp/ssm-session-$.log\"\n\ncleanup() {\n echo \"Cleaning up SSM session...\"\n [ -n \"$SESSION_PID\" ] && kill $SESSION_PID 2>/dev/null\n rm -f \"$SSM_LOG\"\n echo \"Session terminated\"\n}\n\n# Register cleanup on exit\ntrap cleanup EXIT INT TERM\n\n# Start SSM port forwarding in background\naws ssm start-session \\\n --target \"$BASTION_ID\" \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters \"{\n \\\"host\\\":[\\\"$REMOTE_HOST\\\"],\n \\\"portNumber\\\":[\\\"$REMOTE_PORT\\\"],\n \\\"localPortNumber\\\":[\\\"$LOCAL_PORT\\\"]\n }\" > \"$SSM_LOG\" 2>&1 &\n\nSESSION_PID=$!\n\n# Wait for tunnel to establish\nsleep 3\n\n# Check if session is running\nif ! kill -0 $SESSION_PID 2>/dev/null; then\n echo \"Failed to start SSM session\"\n cat \"$SSM_LOG\"\n exit 1\nfi\n\necho \"Tunnel established on localhost:$LOCAL_PORT\"\necho \"Press Ctrl+C or type 'exit' to disconnect\"\n\n# Start interactive shell\n$SHELL\n\n# Cleanup happens automatically via trap\n```\n\n### EKS Connection Script\nComplete script for connecting to private EKS clusters.\n\n```bash\n#!/bin/bash\n# eks-connect.sh \u003ccluster-name> [local-port]\n\nCLUSTER_NAME=${1:?Usage: eks-connect.sh \u003ccluster-name> [local-port]}\nLOCAL_PORT=${2:-6443}\n\n# Ensure clean environment\nunset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN\n\n# Validate credentials\nif ! aws sts get-caller-identity &>/dev/null; then\n echo \"Please login first: aws sso login\"\n exit 1\nfi\n\n# Get cluster endpoint\nENDPOINT=$(aws eks describe-cluster --name \"$CLUSTER_NAME\" \\\n --query 'cluster.endpoint' --output text)\nEKS_HOST=$(echo \"$ENDPOINT\" | sed 's|https://||')\n\n# Find bastion\nBASTION_ID=$(aws ec2 describe-instances \\\n --filters \"Name=tag:Role,Values=bastion\" \\\n \"Name=instance-state-name,Values=running\" \\\n --query 'Reservations[0].Instances[0].InstanceId' \\\n --output text)\n\nif [ \"$BASTION_ID\" = \"None\" ]; then\n echo \"Error: No bastion found\"\n exit 1\nfi\n\n# Check port availability\nif lsof -ti :$LOCAL_PORT &>/dev/null; then\n echo \"Port $LOCAL_PORT in use, finding alternative...\"\n LOCAL_PORT=$(find_available_port $LOCAL_PORT)\nfi\n\necho \"Connecting to $CLUSTER_NAME via $BASTION_ID on port $LOCAL_PORT\"\n\n# Update kubeconfig\naws eks update-kubeconfig --name \"$CLUSTER_NAME\" 2>/dev/null\n# Override server to use local port\nkubectl config set-cluster \"$CLUSTER_NAME\" --server=\"https://127.0.0.1:$LOCAL_PORT\"\n\n# Start session (cleanup via trap)\ntrap 'kill $SSM_PID 2>/dev/null; exit' EXIT INT TERM\n\naws ssm start-session \\\n --target \"$BASTION_ID\" \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters \"{\n \\\"host\\\":[\\\"$EKS_HOST\\\"],\n \\\"portNumber\\\":[\\\"443\\\"],\n \\\"localPortNumber\\\":[\\\"$LOCAL_PORT\\\"]\n }\" &\nSSM_PID=$!\n\nsleep 3\n\necho \"Connected! Run kubectl commands in this shell.\"\necho \"Type 'exit' to disconnect.\"\n\n# Interactive shell with custom KUBECONFIG\nKUBECONFIG=~/.kube/config $SHELL\n```\n\n---\n\n## Useful Queries\n\n```bash\n# List SSM-managed instances\naws ssm describe-instance-information \\\n --query 'InstanceInformationList[*].{ID:InstanceId,IP:IPAddress,Platform:PlatformType,Status:PingStatus}'\n\n# Find instances by tag\naws ssm describe-instance-information \\\n --filters Key=tag:Environment,Values=Production \\\n --query 'InstanceInformationList[*].InstanceId'\n\n# List active sessions\naws ssm describe-sessions \\\n --state Active \\\n --query 'Sessions[*].{SessionId:SessionId,Target:Target,Owner:Owner}'\n\n# Get session history\naws ssm describe-sessions \\\n --state History \\\n --filters key=Owner,value=$(aws sts get-caller-identity --query 'Arn' --output text) \\\n --query 'Sessions[*].{SessionId:SessionId,Target:Target,StartDate:StartDate}'\n\n# Terminate session\naws ssm terminate-session --session-id session-id-here\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **SSM over SSH** | Use Session Manager for audit, logging, IAM control |\n| **No public IPs** | Keep instances in private subnets, use SSM |\n| **VPC endpoints** | Deploy SSM endpoints for private subnet access |\n| **Logging** | Enable S3/CloudWatch logging for compliance |\n| **IAM policies** | Restrict ssm:StartSession by instance tags |\n| **Idle timeout** | Configure automatic session termination |\n| **Port forwarding** | Use for database access instead of VPNs |\n| **Run Command** | Prefer over SSH for automated tasks |\n| **Session preferences** | Standardize shell profiles across team |\n| **Multi-account** | Use cross-account IAM roles with SSM |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":22252,"content_sha256":"f666a556ed54a5a4766f74369224216cb4f82b48ebaef8a4710eb6f65d7d9c01"},{"filename":"references/dynamodb.md","content":"# DynamoDB\n\n## Table Management\n\n### Create Table (On-Demand)\n```bash\n# Simple table with on-demand billing\naws dynamodb create-table \\\n --table-name Users \\\n --attribute-definitions \\\n AttributeName=UserId,AttributeType=S \\\n --key-schema \\\n AttributeName=UserId,KeyType=HASH \\\n --billing-mode PAY_PER_REQUEST\n\n# With Global Secondary Index\naws dynamodb create-table \\\n --table-name Users \\\n --attribute-definitions \\\n AttributeName=UserId,AttributeType=S \\\n AttributeName=Email,AttributeType=S \\\n --key-schema \\\n AttributeName=UserId,KeyType=HASH \\\n --billing-mode PAY_PER_REQUEST \\\n --global-secondary-indexes '[\n {\n \"IndexName\": \"EmailIndex\",\n \"KeySchema\": [{\"AttributeName\": \"Email\", \"KeyType\": \"HASH\"}],\n \"Projection\": {\"ProjectionType\": \"ALL\"}\n }\n ]'\n\n# Composite key (partition + sort)\naws dynamodb create-table \\\n --table-name Orders \\\n --attribute-definitions \\\n AttributeName=CustomerId,AttributeType=S \\\n AttributeName=OrderDate,AttributeType=S \\\n --key-schema \\\n AttributeName=CustomerId,KeyType=HASH \\\n AttributeName=OrderDate,KeyType=RANGE \\\n --billing-mode PAY_PER_REQUEST\n```\n\n### Create Table (Provisioned)\n```bash\naws dynamodb create-table \\\n --table-name Logs \\\n --attribute-definitions AttributeName=LogId,AttributeType=S \\\n --key-schema AttributeName=LogId,KeyType=HASH \\\n --billing-mode PROVISIONED \\\n --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5\n```\n\n### Manage Tables\n```bash\n# List tables\naws dynamodb list-tables\n\n# Describe table\naws dynamodb describe-table --table-name Users\n\n# Delete table\naws dynamodb delete-table --table-name Users\n\n# Wait for table to be active\naws dynamodb wait table-exists --table-name Users\n```\n\n### Update Table\n```bash\n# Switch to provisioned capacity\naws dynamodb update-table \\\n --table-name Users \\\n --billing-mode PROVISIONED \\\n --provisioned-throughput ReadCapacityUnits=10,WriteCapacityUnits=10\n\n# Switch to on-demand\naws dynamodb update-table \\\n --table-name Users \\\n --billing-mode PAY_PER_REQUEST\n\n# Add GSI\naws dynamodb update-table \\\n --table-name Users \\\n --attribute-definitions AttributeName=Status,AttributeType=S \\\n --global-secondary-index-updates '[\n {\n \"Create\": {\n \"IndexName\": \"StatusIndex\",\n \"KeySchema\": [{\"AttributeName\": \"Status\", \"KeyType\": \"HASH\"}],\n \"Projection\": {\"ProjectionType\": \"ALL\"}\n }\n }\n ]'\n\n# Delete GSI\naws dynamodb update-table \\\n --table-name Users \\\n --global-secondary-index-updates '[\n {\"Delete\": {\"IndexName\": \"StatusIndex\"}}\n ]'\n```\n\n## Item Operations\n\n### Put Item\n```bash\n# Basic put\naws dynamodb put-item \\\n --table-name Users \\\n --item '{\n \"UserId\": {\"S\": \"u-123\"},\n \"Name\": {\"S\": \"Alice\"},\n \"Age\": {\"N\": \"30\"},\n \"Email\": {\"S\": \"[email protected]\"},\n \"Tags\": {\"SS\": [\"admin\", \"active\"]}\n }'\n\n# Put with condition (only if not exists)\naws dynamodb put-item \\\n --table-name Users \\\n --item '{\"UserId\": {\"S\": \"u-123\"}, \"Name\": {\"S\": \"Alice\"}}' \\\n --condition-expression \"attribute_not_exists(UserId)\"\n\n# Return old values\naws dynamodb put-item \\\n --table-name Users \\\n --item '{\"UserId\": {\"S\": \"u-123\"}, \"Name\": {\"S\": \"Bob\"}}' \\\n --return-values ALL_OLD\n```\n\n### Get Item\n```bash\n# Basic get\naws dynamodb get-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}'\n\n# Get specific attributes only\naws dynamodb get-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}' \\\n --projection-expression \"Name, Email\"\n\n# Consistent read\naws dynamodb get-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}' \\\n --consistent-read\n```\n\n### Update Item\n```bash\n# Update with expression\naws dynamodb update-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}' \\\n --update-expression \"SET Age = :age, #status = :status\" \\\n --expression-attribute-names '{\"#status\": \"Status\"}' \\\n --expression-attribute-values '{\":age\": {\"N\": \"31\"}, \":status\": {\"S\": \"active\"}}'\n\n# Increment counter\naws dynamodb update-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}' \\\n --update-expression \"SET LoginCount = if_not_exists(LoginCount, :zero) + :inc\" \\\n --expression-attribute-values '{\":zero\": {\"N\": \"0\"}, \":inc\": {\"N\": \"1\"}}'\n\n# Add to set\naws dynamodb update-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}' \\\n --update-expression \"ADD Tags :newTags\" \\\n --expression-attribute-values '{\":newTags\": {\"SS\": [\"premium\"]}}'\n\n# Remove attribute\naws dynamodb update-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}' \\\n --update-expression \"REMOVE TemporaryField\"\n\n# Conditional update\naws dynamodb update-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}' \\\n --update-expression \"SET Balance = Balance - :amount\" \\\n --condition-expression \"Balance >= :amount\" \\\n --expression-attribute-values '{\":amount\": {\"N\": \"100\"}}'\n```\n\n### Delete Item\n```bash\n# Basic delete\naws dynamodb delete-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}'\n\n# Conditional delete\naws dynamodb delete-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}' \\\n --condition-expression \"#status = :inactive\" \\\n --expression-attribute-names '{\"#status\": \"Status\"}' \\\n --expression-attribute-values '{\":inactive\": {\"S\": \"inactive\"}}'\n\n# Return deleted item\naws dynamodb delete-item \\\n --table-name Users \\\n --key '{\"UserId\": {\"S\": \"u-123\"}}' \\\n --return-values ALL_OLD\n```\n\n## Query and Scan\n\n### Query (Efficient)\n```bash\n# Query by partition key\naws dynamodb query \\\n --table-name Orders \\\n --key-condition-expression \"CustomerId = :cid\" \\\n --expression-attribute-values '{\":cid\": {\"S\": \"c-123\"}}'\n\n# Query with sort key condition\naws dynamodb query \\\n --table-name Orders \\\n --key-condition-expression \"CustomerId = :cid AND OrderDate BETWEEN :start AND :end\" \\\n --expression-attribute-values '{\n \":cid\": {\"S\": \"c-123\"},\n \":start\": {\"S\": \"2024-01-01\"},\n \":end\": {\"S\": \"2024-12-31\"}\n }'\n\n# Query with filter\naws dynamodb query \\\n --table-name Orders \\\n --key-condition-expression \"CustomerId = :cid\" \\\n --filter-expression \"Amount > :min\" \\\n --expression-attribute-values '{\":cid\": {\"S\": \"c-123\"}, \":min\": {\"N\": \"100\"}}'\n\n# Query GSI\naws dynamodb query \\\n --table-name Users \\\n --index-name EmailIndex \\\n --key-condition-expression \"Email = :email\" \\\n --expression-attribute-values '{\":email\": {\"S\": \"[email protected]\"}}'\n\n# Reverse order (descending)\naws dynamodb query \\\n --table-name Orders \\\n --key-condition-expression \"CustomerId = :cid\" \\\n --expression-attribute-values '{\":cid\": {\"S\": \"c-123\"}}' \\\n --scan-index-forward false \\\n --limit 10\n```\n\n### Scan (Use Sparingly)\n```bash\n# Basic scan\naws dynamodb scan --table-name Users\n\n# Scan with filter\naws dynamodb scan \\\n --table-name Users \\\n --filter-expression \"Age > :age\" \\\n --expression-attribute-values '{\":age\": {\"N\": \"25\"}}'\n\n# Parallel scan (for large tables)\naws dynamodb scan \\\n --table-name Users \\\n --segment 0 \\\n --total-segments 4\n```\n\n### PartiQL\n```bash\n# Select\naws dynamodb execute-statement \\\n --statement \"SELECT * FROM Users WHERE UserId = 'u-123'\"\n\n# Select with filter\naws dynamodb execute-statement \\\n --statement \"SELECT Name, Email FROM Users WHERE Age > 25\"\n\n# Insert\naws dynamodb execute-statement \\\n --statement \"INSERT INTO Users VALUE {'UserId': 'u-456', 'Name': 'Bob'}\"\n\n# Update\naws dynamodb execute-statement \\\n --statement \"UPDATE Users SET Age = 32 WHERE UserId = 'u-123'\"\n\n# Delete\naws dynamodb execute-statement \\\n --statement \"DELETE FROM Users WHERE UserId = 'u-123'\"\n```\n\n## Batch Operations\n\n### Batch Write\n```bash\n# Write multiple items (max 25)\naws dynamodb batch-write-item \\\n --request-items '{\n \"Users\": [\n {\"PutRequest\": {\"Item\": {\"UserId\": {\"S\": \"u-1\"}, \"Name\": {\"S\": \"Alice\"}}}},\n {\"PutRequest\": {\"Item\": {\"UserId\": {\"S\": \"u-2\"}, \"Name\": {\"S\": \"Bob\"}}}},\n {\"DeleteRequest\": {\"Key\": {\"UserId\": {\"S\": \"u-old\"}}}}\n ]\n }'\n```\n\n### Batch Get\n```bash\n# Get multiple items (max 100)\naws dynamodb batch-get-item \\\n --request-items '{\n \"Users\": {\n \"Keys\": [\n {\"UserId\": {\"S\": \"u-1\"}},\n {\"UserId\": {\"S\": \"u-2\"}},\n {\"UserId\": {\"S\": \"u-3\"}}\n ],\n \"ProjectionExpression\": \"UserId, Name, Email\"\n }\n }'\n```\n\n## Transactions\n\n### TransactWrite (All-or-nothing writes)\n```bash\naws dynamodb transact-write-items \\\n --transact-items '[\n {\n \"Put\": {\n \"TableName\": \"Orders\",\n \"Item\": {\"OrderId\": {\"S\": \"o-123\"}, \"CustomerId\": {\"S\": \"c-123\"}, \"Amount\": {\"N\": \"100\"}}\n }\n },\n {\n \"Update\": {\n \"TableName\": \"Customers\",\n \"Key\": {\"CustomerId\": {\"S\": \"c-123\"}},\n \"UpdateExpression\": \"SET Balance = Balance - :amount\",\n \"ConditionExpression\": \"Balance >= :amount\",\n \"ExpressionAttributeValues\": {\":amount\": {\"N\": \"100\"}}\n }\n }\n ]'\n```\n\n### TransactGet (Consistent reads)\n```bash\naws dynamodb transact-get-items \\\n --transact-items '[\n {\"Get\": {\"TableName\": \"Users\", \"Key\": {\"UserId\": {\"S\": \"u-123\"}}}},\n {\"Get\": {\"TableName\": \"Orders\", \"Key\": {\"OrderId\": {\"S\": \"o-123\"}}}}\n ]'\n```\n\n## DynamoDB Streams\n\n### Enable Streams\n```bash\n# Enable streams on table\naws dynamodb update-table \\\n --table-name Users \\\n --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES\n\n# Stream view types:\n# KEYS_ONLY - Only key attributes\n# NEW_IMAGE - Item after modification\n# OLD_IMAGE - Item before modification\n# NEW_AND_OLD_IMAGES - Both before and after\n```\n\n### Read Streams\n```bash\n# Get stream ARN\naws dynamodb describe-table \\\n --table-name Users \\\n --query 'Table.LatestStreamArn' \\\n --output text\n\n# List stream shards\naws dynamodbstreams describe-stream \\\n --stream-arn arn:aws:dynamodb:us-east-1:123456789012:table/Users/stream/2024-01-01T00:00:00.000\n\n# Get shard iterator\naws dynamodbstreams get-shard-iterator \\\n --stream-arn arn:aws:dynamodb:us-east-1:123456789012:table/Users/stream/2024-01-01T00:00:00.000 \\\n --shard-id shardId-00000001 \\\n --shard-iterator-type TRIM_HORIZON\n\n# Get records\naws dynamodbstreams get-records \\\n --shard-iterator \u003cshard-iterator>\n```\n\n## Global Tables\n\n### Create Global Table\n```bash\n# Create table in first region\naws dynamodb create-table \\\n --table-name GlobalUsers \\\n --attribute-definitions AttributeName=UserId,AttributeType=S \\\n --key-schema AttributeName=UserId,KeyType=HASH \\\n --billing-mode PAY_PER_REQUEST \\\n --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES \\\n --region us-east-1\n\n# Wait for table\naws dynamodb wait table-exists --table-name GlobalUsers --region us-east-1\n\n# Add replica in another region\naws dynamodb update-table \\\n --table-name GlobalUsers \\\n --replica-updates '[{\"Create\": {\"RegionName\": \"eu-west-1\"}}]' \\\n --region us-east-1\n\n# Add more replicas\naws dynamodb update-table \\\n --table-name GlobalUsers \\\n --replica-updates '[{\"Create\": {\"RegionName\": \"ap-southeast-1\"}}]' \\\n --region us-east-1\n```\n\n### Manage Global Tables\n```bash\n# Describe global table\naws dynamodb describe-table \\\n --table-name GlobalUsers \\\n --query 'Table.Replicas'\n\n# Remove replica\naws dynamodb update-table \\\n --table-name GlobalUsers \\\n --replica-updates '[{\"Delete\": {\"RegionName\": \"ap-southeast-1\"}}]' \\\n --region us-east-1\n```\n\n## Time to Live (TTL)\n\n### Enable TTL\n```bash\n# Enable TTL on attribute\naws dynamodb update-time-to-live \\\n --table-name Sessions \\\n --time-to-live-specification Enabled=true,AttributeName=ExpiresAt\n\n# Check TTL status\naws dynamodb describe-time-to-live --table-name Sessions\n```\n\n### Use TTL\n```bash\n# Put item with TTL (Unix timestamp)\nEXPIRES=$(date -d '+24 hours' +%s)\naws dynamodb put-item \\\n --table-name Sessions \\\n --item \"{\n \\\"SessionId\\\": {\\\"S\\\": \\\"sess-123\\\"},\n \\\"UserId\\\": {\\\"S\\\": \\\"u-123\\\"},\n \\\"ExpiresAt\\\": {\\\"N\\\": \\\"$EXPIRES\\\"}\n }\"\n```\n\n## Backups and Recovery\n\n### On-Demand Backup\n```bash\n# Create backup\naws dynamodb create-backup \\\n --table-name Users \\\n --backup-name Users-2024-01-15\n\n# List backups\naws dynamodb list-backups --table-name Users\n\n# Describe backup\naws dynamodb describe-backup \\\n --backup-arn arn:aws:dynamodb:us-east-1:123456789012:table/Users/backup/01234567890123-abc123\n\n# Restore from backup\naws dynamodb restore-table-from-backup \\\n --target-table-name Users-Restored \\\n --backup-arn arn:aws:dynamodb:us-east-1:123456789012:table/Users/backup/01234567890123-abc123\n\n# Delete backup\naws dynamodb delete-backup \\\n --backup-arn arn:aws:dynamodb:us-east-1:123456789012:table/Users/backup/01234567890123-abc123\n```\n\n### Point-in-Time Recovery (PITR)\n```bash\n# Enable PITR\naws dynamodb update-continuous-backups \\\n --table-name Users \\\n --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true\n\n# Check PITR status\naws dynamodb describe-continuous-backups --table-name Users\n\n# Restore to point in time\naws dynamodb restore-table-to-point-in-time \\\n --source-table-name Users \\\n --target-table-name Users-Restored \\\n --restore-date-time 2024-01-15T12:00:00Z\n```\n\n## Export to S3\n\n```bash\n# Export table to S3 (for analytics)\naws dynamodb export-table-to-point-in-time \\\n --table-arn arn:aws:dynamodb:us-east-1:123456789012:table/Users \\\n --s3-bucket my-exports-bucket \\\n --s3-prefix dynamodb-exports/ \\\n --export-format DYNAMODB_JSON\n\n# List exports\naws dynamodb list-exports --table-arn arn:aws:dynamodb:us-east-1:123456789012:table/Users\n\n# Describe export\naws dynamodb describe-export \\\n --export-arn arn:aws:dynamodb:us-east-1:123456789012:table/Users/export/01234567890123-abc123\n```\n\n## Import from S3\n\n```bash\n# Import from S3\naws dynamodb import-table \\\n --s3-bucket-source S3Bucket=my-import-bucket,S3KeyPrefix=imports/ \\\n --input-format DYNAMODB_JSON \\\n --table-creation-parameters '{\n \"TableName\": \"ImportedUsers\",\n \"AttributeDefinitions\": [{\"AttributeName\": \"UserId\", \"AttributeType\": \"S\"}],\n \"KeySchema\": [{\"AttributeName\": \"UserId\", \"KeyType\": \"HASH\"}],\n \"BillingMode\": \"PAY_PER_REQUEST\"\n }'\n\n# List imports\naws dynamodb list-imports\n\n# Describe import\naws dynamodb describe-import --import-arn \u003cimport-arn>\n```\n\n## Auto Scaling (Provisioned Mode)\n\n```bash\n# Register scalable target (read capacity)\naws application-autoscaling register-scalable-target \\\n --service-namespace dynamodb \\\n --resource-id \"table/Users\" \\\n --scalable-dimension \"dynamodb:table:ReadCapacityUnits\" \\\n --min-capacity 5 \\\n --max-capacity 1000\n\n# Create scaling policy\naws application-autoscaling put-scaling-policy \\\n --service-namespace dynamodb \\\n --resource-id \"table/Users\" \\\n --scalable-dimension \"dynamodb:table:ReadCapacityUnits\" \\\n --policy-name \"UsersReadScaling\" \\\n --policy-type TargetTrackingScaling \\\n --target-tracking-scaling-policy-configuration '{\n \"TargetValue\": 70.0,\n \"PredefinedMetricSpecification\": {\n \"PredefinedMetricType\": \"DynamoDBReadCapacityUtilization\"\n },\n \"ScaleOutCooldown\": 60,\n \"ScaleInCooldown\": 60\n }'\n```\n\n## Useful Queries\n\n```bash\n# Get table item count\naws dynamodb describe-table \\\n --table-name Users \\\n --query 'Table.ItemCount'\n\n# Get table size in bytes\naws dynamodb describe-table \\\n --table-name Users \\\n --query 'Table.TableSizeBytes'\n\n# List all GSIs\naws dynamodb describe-table \\\n --table-name Users \\\n --query 'Table.GlobalSecondaryIndexes[*].IndexName'\n\n# Check table status\naws dynamodb describe-table \\\n --table-name Users \\\n --query 'Table.TableStatus'\n\n# Get stream ARN\naws dynamodb describe-table \\\n --table-name Users \\\n --query 'Table.LatestStreamArn' \\\n --output text\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **On-demand billing** | Start with PAY_PER_REQUEST for new workloads |\n| **Partition key design** | Use high-cardinality keys to avoid hot partitions |\n| **GSI projection** | Only project needed attributes to reduce costs |\n| **Conditional writes** | Use conditions to prevent race conditions |\n| **TTL** | Enable for session data, caches, temporary records |\n| **Streams** | Use for change data capture, event-driven architectures |\n| **Global Tables** | Enable for multi-region active-active |\n| **PITR** | Enable for critical tables (35-day recovery window) |\n| **Batch operations** | Use batch-write-item for bulk inserts (max 25) |\n| **Transactions** | Use for ACID operations across multiple items |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":17102,"content_sha256":"8353e6ecabff7f97e7bbf84dfb2f4ec8a2ea3580f1bc0dc4abba0c1691678930"},{"filename":"references/ecr.md","content":"# Elastic Container Registry (ECR)\n\n## Repository Management\n\n### Create Repository\n```bash\n# Basic repository with scanning and immutability\naws ecr create-repository \\\n --repository-name my-app/backend \\\n --image-scanning-configuration scanOnPush=true \\\n --image-tag-mutability IMMUTABLE\n\n# With KMS encryption\naws ecr create-repository \\\n --repository-name my-app/backend \\\n --image-scanning-configuration scanOnPush=true \\\n --image-tag-mutability IMMUTABLE \\\n --encryption-configuration encryptionType=KMS,kmsKey=alias/ecr-key\n\n# With tags\naws ecr create-repository \\\n --repository-name my-app/backend \\\n --tags Key=Environment,Value=Production Key=Team,Value=Backend\n```\n\n### List and Describe Repositories\n```bash\n# List all repositories\naws ecr describe-repositories\n\n# Describe specific repository\naws ecr describe-repositories --repository-names my-app/backend\n\n# Get repository URI\naws ecr describe-repositories \\\n --repository-names my-app/backend \\\n --query 'repositories[0].repositoryUri' \\\n --output text\n```\n\n### Delete Repository\n```bash\n# Delete empty repository\naws ecr delete-repository --repository-name my-app/backend\n\n# Force delete (removes all images)\naws ecr delete-repository --repository-name my-app/backend --force\n```\n\n## Authentication\n\n### Docker Login\n```bash\n# Get login password and authenticate Docker\naws ecr get-login-password --region us-east-1 | \\\n docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com\n\n# For a specific profile\naws ecr get-login-password --region us-east-1 --profile production | \\\n docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com\n```\n\n### Get Authorization Token Details\n```bash\n# Get base64-encoded token (valid 12 hours)\naws ecr get-authorization-token \\\n --query 'authorizationData[0].authorizationToken' \\\n --output text | base64 -d\n\n# Get expiration time\naws ecr get-authorization-token \\\n --query 'authorizationData[0].expiresAt'\n```\n\n## Push/Pull Workflows\n\n### Build and Push Image\n```bash\n# Set variables\nACCOUNT_ID=123456789012\nREGION=us-east-1\nREPO_NAME=my-app/backend\nTAG=v1.2.0\n\n# Authenticate\naws ecr get-login-password --region $REGION | \\\n docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com\n\n# Build image\ndocker build -t $REPO_NAME:$TAG .\n\n# Tag for ECR\ndocker tag $REPO_NAME:$TAG $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/$REPO_NAME:$TAG\n\n# Push\ndocker push $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/$REPO_NAME:$TAG\n\n# Also tag as latest\ndocker tag $REPO_NAME:$TAG $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/$REPO_NAME:latest\ndocker push $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/$REPO_NAME:latest\n```\n\n### Pull Image\n```bash\n# Authenticate first\naws ecr get-login-password --region us-east-1 | \\\n docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com\n\n# Pull image\ndocker pull 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app/backend:v1.2.0\n```\n\n## Image Management\n\n### List Images\n```bash\n# List all images in repository\naws ecr list-images --repository-name my-app/backend\n\n# List with details\naws ecr describe-images --repository-name my-app/backend\n\n# List images sorted by push date\naws ecr describe-images \\\n --repository-name my-app/backend \\\n --query 'sort_by(imageDetails, &imagePushedAt)[*].{Tag:imageTags[0],Pushed:imagePushedAt,Size:imageSizeInBytes}'\n\n# Find untagged images\naws ecr describe-images \\\n --repository-name my-app/backend \\\n --filter tagStatus=UNTAGGED\n```\n\n### Delete Images\n```bash\n# Delete by tag\naws ecr batch-delete-image \\\n --repository-name my-app/backend \\\n --image-ids imageTag=v1.0.0\n\n# Delete by digest\naws ecr batch-delete-image \\\n --repository-name my-app/backend \\\n --image-ids imageDigest=sha256:abc123...\n\n# Delete multiple images\naws ecr batch-delete-image \\\n --repository-name my-app/backend \\\n --image-ids imageTag=v1.0.0 imageTag=v1.0.1 imageTag=v1.0.2\n\n# Delete all untagged images\naws ecr describe-images \\\n --repository-name my-app/backend \\\n --filter tagStatus=UNTAGGED \\\n --query 'imageDetails[*].imageDigest' \\\n --output text | \\\n xargs -I {} aws ecr batch-delete-image \\\n --repository-name my-app/backend \\\n --image-ids imageDigest={}\n```\n\n### Get Image Details\n```bash\n# Get specific image by tag\naws ecr describe-images \\\n --repository-name my-app/backend \\\n --image-ids imageTag=v1.2.0\n\n# Get image manifest\naws ecr batch-get-image \\\n --repository-name my-app/backend \\\n --image-ids imageTag=v1.2.0 \\\n --query 'images[0].imageManifest' \\\n --output text\n```\n\n## Image Scanning\n\n### Basic Scanning (On-Push)\n```bash\n# Enable scan on push for repository\naws ecr put-image-scanning-configuration \\\n --repository-name my-app/backend \\\n --image-scanning-configuration scanOnPush=true\n```\n\n### On-Demand Scanning\n```bash\n# Start manual scan\naws ecr start-image-scan \\\n --repository-name my-app/backend \\\n --image-id imageTag=v1.2.0\n\n# Wait for scan to complete\naws ecr wait image-scan-complete \\\n --repository-name my-app/backend \\\n --image-id imageTag=v1.2.0\n```\n\n### Get Scan Findings\n```bash\n# Get scan results\naws ecr describe-image-scan-findings \\\n --repository-name my-app/backend \\\n --image-id imageTag=v1.2.0\n\n# Get only critical and high vulnerabilities\naws ecr describe-image-scan-findings \\\n --repository-name my-app/backend \\\n --image-id imageTag=v1.2.0 \\\n --query 'imageScanFindings.findings[?severity==`CRITICAL` || severity==`HIGH`]'\n\n# Get vulnerability counts by severity\naws ecr describe-image-scan-findings \\\n --repository-name my-app/backend \\\n --image-id imageTag=v1.2.0 \\\n --query 'imageScanFindings.findingSeverityCounts'\n```\n\n### Enhanced Scanning (Amazon Inspector)\n```bash\n# Enable enhanced scanning at registry level\naws ecr put-registry-scanning-configuration \\\n --scan-type ENHANCED \\\n --rules '[{\"repositoryFilters\":[{\"filter\":\"*\",\"filterType\":\"WILDCARD\"}],\"scanFrequency\":\"SCAN_ON_PUSH\"}]'\n\n# Enable continuous scanning for specific repositories\naws ecr put-registry-scanning-configuration \\\n --scan-type ENHANCED \\\n --rules '[{\"repositoryFilters\":[{\"filter\":\"prod/*\",\"filterType\":\"WILDCARD\"}],\"scanFrequency\":\"CONTINUOUS_SCAN\"}]'\n\n# Get registry scanning configuration\naws ecr get-registry-scanning-configuration\n```\n\n## Lifecycle Policies\n\n### Create Lifecycle Policy\n```bash\naws ecr put-lifecycle-policy \\\n --repository-name my-app/backend \\\n --lifecycle-policy-text file://lifecycle-policy.json\n```\n\n**lifecycle-policy.json (Keep tagged, expire old untagged):**\n```json\n{\n \"rules\": [\n {\n \"rulePriority\": 1,\n \"description\": \"Expire untagged images older than 1 day\",\n \"selection\": {\n \"tagStatus\": \"untagged\",\n \"countType\": \"sinceImagePushed\",\n \"countUnit\": \"days\",\n \"countNumber\": 1\n },\n \"action\": {\n \"type\": \"expire\"\n }\n },\n {\n \"rulePriority\": 2,\n \"description\": \"Keep only 10 dev images\",\n \"selection\": {\n \"tagStatus\": \"tagged\",\n \"tagPrefixList\": [\"dev-\", \"feature-\"],\n \"countType\": \"imageCountMoreThan\",\n \"countNumber\": 10\n },\n \"action\": {\n \"type\": \"expire\"\n }\n },\n {\n \"rulePriority\": 3,\n \"description\": \"Keep last 50 production images\",\n \"selection\": {\n \"tagStatus\": \"tagged\",\n \"tagPrefixList\": [\"v\", \"release-\"],\n \"countType\": \"imageCountMoreThan\",\n \"countNumber\": 50\n },\n \"action\": {\n \"type\": \"expire\"\n }\n }\n ]\n}\n```\n\n### Manage Lifecycle Policies\n```bash\n# Get lifecycle policy\naws ecr get-lifecycle-policy --repository-name my-app/backend\n\n# Preview lifecycle policy (dry run)\naws ecr get-lifecycle-policy-preview \\\n --repository-name my-app/backend \\\n --lifecycle-policy-text file://lifecycle-policy.json\n\n# Delete lifecycle policy\naws ecr delete-lifecycle-policy --repository-name my-app/backend\n```\n\n## Repository Policies (Cross-Account Access)\n\n### Set Repository Policy\n```bash\naws ecr set-repository-policy \\\n --repository-name my-app/backend \\\n --policy-text file://repo-policy.json\n```\n\n**repo-policy.json (Cross-account pull access):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"AllowCrossAccountPull\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"AWS\": [\n \"arn:aws:iam::111111111111:root\",\n \"arn:aws:iam::222222222222:root\"\n ]\n },\n \"Action\": [\n \"ecr:GetDownloadUrlForLayer\",\n \"ecr:BatchGetImage\",\n \"ecr:BatchCheckLayerAvailability\"\n ]\n }\n ]\n}\n```\n\n**repo-policy.json (Allow specific role):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"AllowECSTaskRole\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"AWS\": \"arn:aws:iam::111111111111:role/ECSTaskRole\"\n },\n \"Action\": [\n \"ecr:GetDownloadUrlForLayer\",\n \"ecr:BatchGetImage\",\n \"ecr:BatchCheckLayerAvailability\"\n ]\n }\n ]\n}\n```\n\n### Manage Repository Policies\n```bash\n# Get current policy\naws ecr get-repository-policy --repository-name my-app/backend\n\n# Delete policy\naws ecr delete-repository-policy --repository-name my-app/backend\n```\n\n## Cross-Region Replication\n\n### Configure Registry Replication\n```bash\naws ecr put-replication-configuration \\\n --replication-configuration file://replication-config.json\n```\n\n**replication-config.json (Single region):**\n```json\n{\n \"rules\": [\n {\n \"destinations\": [\n {\n \"region\": \"eu-west-1\",\n \"registryId\": \"123456789012\"\n }\n ],\n \"repositoryFilters\": [\n {\n \"filter\": \"prod/\",\n \"filterType\": \"PREFIX_MATCH\"\n }\n ]\n }\n ]\n}\n```\n\n**replication-config.json (Multi-region):**\n```json\n{\n \"rules\": [\n {\n \"destinations\": [\n {\"region\": \"eu-west-1\", \"registryId\": \"123456789012\"},\n {\"region\": \"ap-southeast-1\", \"registryId\": \"123456789012\"},\n {\"region\": \"us-west-2\", \"registryId\": \"123456789012\"}\n ],\n \"repositoryFilters\": [\n {\n \"filter\": \"prod/\",\n \"filterType\": \"PREFIX_MATCH\"\n }\n ]\n }\n ]\n}\n```\n\n### Cross-Account Replication\n```json\n{\n \"rules\": [\n {\n \"destinations\": [\n {\n \"region\": \"us-east-1\",\n \"registryId\": \"999999999999\"\n }\n ]\n }\n ]\n}\n```\n\n### Get Replication Configuration\n```bash\naws ecr describe-registry\n```\n\n## Pull Through Cache\n\nCache images from external registries (Docker Hub, GitHub, Quay, etc.).\n\n### Create Pull Through Cache Rule\n```bash\n# Docker Hub (public)\naws ecr create-pull-through-cache-rule \\\n --ecr-repository-prefix docker-hub \\\n --upstream-registry-url registry-1.docker.io\n\n# GitHub Container Registry\naws ecr create-pull-through-cache-rule \\\n --ecr-repository-prefix ghcr \\\n --upstream-registry-url ghcr.io\n\n# Quay.io\naws ecr create-pull-through-cache-rule \\\n --ecr-repository-prefix quay \\\n --upstream-registry-url quay.io\n\n# With credentials (for private registries)\naws ecr create-pull-through-cache-rule \\\n --ecr-repository-prefix docker-hub \\\n --upstream-registry-url registry-1.docker.io \\\n --credential-arn arn:aws:secretsmanager:us-east-1:123456789012:secret:dockerhub-creds\n```\n\n### Use Cached Images\n```bash\n# Instead of: docker pull nginx:latest\ndocker pull 123456789012.dkr.ecr.us-east-1.amazonaws.com/docker-hub/library/nginx:latest\n\n# Instead of: docker pull ghcr.io/owner/image:tag\ndocker pull 123456789012.dkr.ecr.us-east-1.amazonaws.com/ghcr/owner/image:tag\n```\n\n### Manage Pull Through Cache\n```bash\n# List rules\naws ecr describe-pull-through-cache-rules\n\n# Delete rule\naws ecr delete-pull-through-cache-rule --ecr-repository-prefix docker-hub\n```\n\n## Registry Settings\n\n### Registry Policy\n```bash\n# Set registry policy (for replication permissions)\naws ecr put-registry-policy --policy-text file://registry-policy.json\n```\n\n**registry-policy.json:**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"AllowReplicationFromAccount\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"AWS\": \"arn:aws:iam::111111111111:root\"\n },\n \"Action\": [\n \"ecr:CreateRepository\",\n \"ecr:ReplicateImage\"\n ],\n \"Resource\": \"arn:aws:ecr:us-east-1:123456789012:repository/*\"\n }\n ]\n}\n```\n\n### Get Registry Settings\n```bash\n# Describe registry\naws ecr describe-registry\n\n# Get registry policy\naws ecr get-registry-policy\n```\n\n## Useful Queries\n\n```bash\n# Get all repository URIs\naws ecr describe-repositories \\\n --query 'repositories[*].repositoryUri' \\\n --output table\n\n# Find largest images\naws ecr describe-images \\\n --repository-name my-app/backend \\\n --query 'sort_by(imageDetails, &imageSizeInBytes)[-5:].{Tag:imageTags[0],SizeMB:imageSizeInBytes}' \\\n --output table\n\n# Count images per repository\nfor repo in $(aws ecr describe-repositories --query 'repositories[*].repositoryName' --output text); do\n count=$(aws ecr list-images --repository-name $repo --query 'length(imageIds)' --output text)\n echo \"$repo: $count images\"\ndone\n\n# Find images with vulnerabilities\naws ecr describe-images \\\n --repository-name my-app/backend \\\n --query 'imageDetails[?imageScanFindingsSummary.findingSeverityCounts.CRITICAL > `0`].imageTags'\n```\n\n## Image Tagging Best Practices\n\n| Strategy | Example | Use Case |\n|:---------|:--------|:---------|\n| **Semantic versioning** | `v1.2.3` | Production releases |\n| **Git SHA** | `abc1234` | Traceability to commits |\n| **Build number** | `build-456` | CI/CD pipelines |\n| **Combined** | `v1.2.3-abc1234` | Best of both worlds |\n| **Environment** | `prod-v1.2.3` | Multi-environment |\n| **Date-based** | `2024-01-15-abc1234` | Rolling deployments |\n\n```bash\n# Multi-tag strategy in CI/CD\nTAG_VERSION=\"v1.2.3\"\nTAG_SHA=$(git rev-parse --short HEAD)\nTAG_DATE=$(date +%Y%m%d)\n\ndocker tag app:latest $ECR_URI:$TAG_VERSION\ndocker tag app:latest $ECR_URI:$TAG_SHA\ndocker tag app:latest $ECR_URI:$TAG_VERSION-$TAG_SHA\ndocker tag app:latest $ECR_URI:latest\n\ndocker push $ECR_URI --all-tags\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Immutable tags** | Enable for release tags to prevent overwrites |\n| **Scan on push** | Enable basic scanning for all repositories |\n| **Enhanced scanning** | Use for production (continuous vulnerability monitoring) |\n| **Lifecycle policies** | Always set policies to clean up old/untagged images |\n| **Cross-region replication** | Replicate production images to DR regions |\n| **Pull through cache** | Cache external images to reduce Docker Hub rate limits |\n| **Repository naming** | Use namespaces like `team/app` or `env/app` |\n| **Tag strategy** | Combine semantic versions with Git SHAs |\n| **Cross-account access** | Use repository policies, not IAM policies |\n| **KMS encryption** | Use customer-managed keys for sensitive images |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":15816,"content_sha256":"ca007d05aa00873ca45edbd4154647f9a39e63e7b312243292dfd2318e011713"},{"filename":"references/ecs.md","content":"# Elastic Container Service (ECS)\n\n## Clusters\n\n### Create Cluster\n```bash\n# Fargate-only cluster with Container Insights\naws ecs create-cluster \\\n --cluster-name production \\\n --capacity-providers FARGATE FARGATE_SPOT \\\n --default-capacity-provider-strategy \\\n capacityProvider=FARGATE,weight=1,base=1 \\\n capacityProvider=FARGATE_SPOT,weight=4 \\\n --settings name=containerInsights,value=enabled\n\n# EC2 cluster with managed scaling\naws ecs create-cluster \\\n --cluster-name ec2-cluster \\\n --settings name=containerInsights,value=enabled\n```\n\n### Manage Clusters\n```bash\n# List clusters\naws ecs list-clusters\n\n# Describe cluster\naws ecs describe-clusters --clusters production\n\n# Delete cluster (must be empty)\naws ecs delete-cluster --cluster production\n```\n\n### Capacity Providers\n```bash\n# Create capacity provider for EC2 Auto Scaling group\naws ecs create-capacity-provider \\\n --name my-ec2-capacity \\\n --auto-scaling-group-provider \\\n autoScalingGroupArn=arn:aws:autoscaling:us-east-1:123456789012:autoScalingGroup:abc123:autoScalingGroupName/my-asg,\\\nmanagedScaling='{status=ENABLED,targetCapacity=100}',\\\nmanagedTerminationProtection=ENABLED\n\n# Update cluster capacity providers\naws ecs put-cluster-capacity-providers \\\n --cluster production \\\n --capacity-providers FARGATE FARGATE_SPOT my-ec2-capacity \\\n --default-capacity-provider-strategy \\\n capacityProvider=FARGATE,weight=1\n```\n\n## Task Definitions\n\n### Register Task Definition\n```bash\naws ecs register-task-definition \\\n --cli-input-json file://task-def.json\n```\n\n**task-def.json (Fargate):**\n```json\n{\n \"family\": \"web-api\",\n \"networkMode\": \"awsvpc\",\n \"requiresCompatibilities\": [\"FARGATE\"],\n \"cpu\": \"256\",\n \"memory\": \"512\",\n \"executionRoleArn\": \"arn:aws:iam::123456789012:role/ecsTaskExecutionRole\",\n \"taskRoleArn\": \"arn:aws:iam::123456789012:role/ecsTaskRole\",\n \"containerDefinitions\": [\n {\n \"name\": \"app\",\n \"image\": \"123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest\",\n \"essential\": true,\n \"portMappings\": [\n {\"containerPort\": 8080, \"protocol\": \"tcp\"}\n ],\n \"environment\": [\n {\"name\": \"NODE_ENV\", \"value\": \"production\"}\n ],\n \"secrets\": [\n {\n \"name\": \"DB_PASSWORD\",\n \"valueFrom\": \"arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/db-password\"\n },\n {\n \"name\": \"API_KEY\",\n \"valueFrom\": \"arn:aws:ssm:us-east-1:123456789012:parameter/prod/api-key\"\n }\n ],\n \"logConfiguration\": {\n \"logDriver\": \"awslogs\",\n \"options\": {\n \"awslogs-group\": \"/ecs/web-api\",\n \"awslogs-region\": \"us-east-1\",\n \"awslogs-stream-prefix\": \"ecs\"\n }\n },\n \"healthCheck\": {\n \"command\": [\"CMD-SHELL\", \"curl -f http://localhost:8080/health || exit 1\"],\n \"interval\": 30,\n \"timeout\": 5,\n \"retries\": 3,\n \"startPeriod\": 60\n }\n }\n ]\n}\n```\n\n### Manage Task Definitions\n```bash\n# List task definition families\naws ecs list-task-definition-families\n\n# List task definitions\naws ecs list-task-definitions --family-prefix web-api\n\n# Describe task definition\naws ecs describe-task-definition --task-definition web-api:1\n\n# Deregister (soft delete)\naws ecs deregister-task-definition --task-definition web-api:1\n```\n\n## Services\n\n### Create Service\n```bash\n# Fargate service with ALB\naws ecs create-service \\\n --cluster production \\\n --service-name web-api-svc \\\n --task-definition web-api:1 \\\n --desired-count 2 \\\n --launch-type FARGATE \\\n --network-configuration \\\n \"awsvpcConfiguration={subnets=[subnet-1,subnet-2],securityGroups=[sg-1],assignPublicIp=DISABLED}\" \\\n --load-balancers \\\n \"targetGroupArn=arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-tg/abc123,containerName=app,containerPort=8080\" \\\n --health-check-grace-period-seconds 60 \\\n --enable-execute-command\n\n# With capacity provider strategy\naws ecs create-service \\\n --cluster production \\\n --service-name web-api-svc \\\n --task-definition web-api:1 \\\n --desired-count 4 \\\n --capacity-provider-strategy \\\n capacityProvider=FARGATE,weight=1,base=1 \\\n capacityProvider=FARGATE_SPOT,weight=4 \\\n --network-configuration \\\n \"awsvpcConfiguration={subnets=[subnet-1,subnet-2],securityGroups=[sg-1]}\"\n```\n\n### Update Service\n```bash\n# Deploy new task definition\naws ecs update-service \\\n --cluster production \\\n --service web-api-svc \\\n --task-definition web-api:2\n\n# Force new deployment (pull latest image)\naws ecs update-service \\\n --cluster production \\\n --service web-api-svc \\\n --force-new-deployment\n\n# Scale\naws ecs update-service \\\n --cluster production \\\n --service web-api-svc \\\n --desired-count 5\n\n# Update deployment configuration\naws ecs update-service \\\n --cluster production \\\n --service web-api-svc \\\n --deployment-configuration \\\n \"minimumHealthyPercent=50,maximumPercent=200,deploymentCircuitBreaker={enable=true,rollback=true}\"\n```\n\n### List and Describe Services\n```bash\n# List services\naws ecs list-services --cluster production\n\n# Describe service\naws ecs describe-services \\\n --cluster production \\\n --services web-api-svc\n\n# Get service events\naws ecs describe-services \\\n --cluster production \\\n --services web-api-svc \\\n --query 'services[0].events[:10]'\n```\n\n### Delete Service\n```bash\n# Scale to 0 first\naws ecs update-service \\\n --cluster production \\\n --service web-api-svc \\\n --desired-count 0\n\n# Then delete\naws ecs delete-service \\\n --cluster production \\\n --service web-api-svc\n\n# Or force delete (stops tasks immediately)\naws ecs delete-service \\\n --cluster production \\\n --service web-api-svc \\\n --force\n```\n\n## Tasks\n\n### Run Standalone Task\n```bash\naws ecs run-task \\\n --cluster production \\\n --task-definition migration-task:1 \\\n --launch-type FARGATE \\\n --network-configuration \\\n \"awsvpcConfiguration={subnets=[subnet-1],securityGroups=[sg-1],assignPublicIp=ENABLED}\" \\\n --overrides \\\n '{\"containerOverrides\":[{\"name\":\"app\",\"command\":[\"python\",\"migrate.py\"]}]}'\n```\n\n### List and Stop Tasks\n```bash\n# List running tasks\naws ecs list-tasks --cluster production\n\n# List tasks for service\naws ecs list-tasks --cluster production --service-name web-api-svc\n\n# Describe tasks\naws ecs describe-tasks \\\n --cluster production \\\n --tasks arn:aws:ecs:us-east-1:123456789012:task/production/abc123\n\n# Stop task\naws ecs stop-task \\\n --cluster production \\\n --task arn:aws:ecs:us-east-1:123456789012:task/production/abc123 \\\n --reason \"Debugging\"\n```\n\n## ECS Exec (Container Debugging)\n\n### Enable ECS Exec\nService must be created with `--enable-execute-command`:\n```bash\naws ecs update-service \\\n --cluster production \\\n --service web-api-svc \\\n --enable-execute-command\n```\n\n### Execute Commands\n```bash\n# Interactive shell\naws ecs execute-command \\\n --cluster production \\\n --task arn:aws:ecs:us-east-1:123456789012:task/production/abc123 \\\n --container app \\\n --interactive \\\n --command \"/bin/sh\"\n\n# Run single command\naws ecs execute-command \\\n --cluster production \\\n --task arn:aws:ecs:us-east-1:123456789012:task/production/abc123 \\\n --container app \\\n --interactive \\\n --command \"cat /app/config.json\"\n```\n\n### Check ECS Exec Readiness\n```bash\n# Install amazon-ecs-exec-checker\naws ecs describe-tasks \\\n --cluster production \\\n --tasks abc123 \\\n --query 'tasks[0].containers[0].managedAgents'\n```\n\n## Service Auto Scaling\n\n### Register Scalable Target\n```bash\naws application-autoscaling register-scalable-target \\\n --service-namespace ecs \\\n --resource-id service/production/web-api-svc \\\n --scalable-dimension ecs:service:DesiredCount \\\n --min-capacity 2 \\\n --max-capacity 10\n```\n\n### Target Tracking Policy\n```bash\naws application-autoscaling put-scaling-policy \\\n --service-namespace ecs \\\n --resource-id service/production/web-api-svc \\\n --scalable-dimension ecs:service:DesiredCount \\\n --policy-name cpu-scaling \\\n --policy-type TargetTrackingScaling \\\n --target-tracking-scaling-policy-configuration \\\n \"TargetValue=70.0,PredefinedMetricSpecification={PredefinedMetricType=ECSServiceAverageCPUUtilization},ScaleOutCooldown=60,ScaleInCooldown=60\"\n```\n\n### Step Scaling Policy\n```bash\naws application-autoscaling put-scaling-policy \\\n --service-namespace ecs \\\n --resource-id service/production/web-api-svc \\\n --scalable-dimension ecs:service:DesiredCount \\\n --policy-name memory-step-scaling \\\n --policy-type StepScaling \\\n --step-scaling-policy-configuration \\\n \"AdjustmentType=ChangeInCapacity,StepAdjustments=[{MetricIntervalLowerBound=0,ScalingAdjustment=2}],Cooldown=60\"\n```\n\n## Blue-Green Deployments (with CodeDeploy)\n\n### Create Deployment Group\n```bash\naws deploy create-deployment-group \\\n --application-name my-ecs-app \\\n --deployment-group-name my-dg \\\n --service-role-arn arn:aws:iam::123456789012:role/CodeDeployRole \\\n --deployment-config-name CodeDeployDefault.ECSAllAtOnce \\\n --ecs-services clusterName=production,serviceName=web-api-svc \\\n --load-balancer-info \\\n \"targetGroupPairInfoList=[{targetGroups=[{name=blue-tg},{name=green-tg}],prodTrafficRoute={listenerArns=[arn:aws:elasticloadbalancing:...]}}]\" \\\n --blue-green-deployment-configuration \\\n \"terminateBlueInstancesOnDeploymentSuccess={action=TERMINATE,terminationWaitTimeInMinutes=5},deploymentReadyOption={actionOnTimeout=CONTINUE_DEPLOYMENT}\"\n```\n\n## Useful Queries\n\n```bash\n# Get task private IPs\naws ecs describe-tasks \\\n --cluster production \\\n --tasks $(aws ecs list-tasks --cluster production --service-name web-api-svc --query 'taskArns[*]' --output text) \\\n --query 'tasks[*].attachments[*].details[?name==`privateIPv4Address`].value' \\\n --output text\n\n# Find stopped tasks with reason\naws ecs list-tasks --cluster production --desired-status STOPPED\naws ecs describe-tasks \\\n --cluster production \\\n --tasks TASK_ARN \\\n --query 'tasks[*].{id:taskArn,reason:stoppedReason,code:stopCode}'\n\n# Get container instance ARNs (EC2 launch type)\naws ecs list-container-instances --cluster ec2-cluster\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Fargate** | Use for reduced ops overhead unless GPU/specific instance needed |\n| **Fargate Spot** | Use for fault-tolerant workloads (up to 70% savings) |\n| **Secrets** | Use Secrets Manager or SSM Parameter Store, never env vars |\n| **Logging** | Always configure awslogs driver to CloudWatch |\n| **Health checks** | Define container health checks for ALB integration |\n| **ECS Exec** | Enable for debugging, disable in highly secure environments |\n| **Task roles** | Separate execution role (ECR/logs) from task role (app perms) |\n| **Circuit breaker** | Enable deployment circuit breaker for automatic rollback |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":11292,"content_sha256":"3aa562cdd0945d6578f5ba749627f9988757b4c7d1363b5130b09eff27126860"},{"filename":"references/eks.md","content":"# Elastic Kubernetes Service (EKS)\n\n## Cluster Management\n\n### Create Cluster\n```bash\n# Create cluster with Kubernetes 1.32\naws eks create-cluster \\\n --name production-cluster \\\n --version 1.32 \\\n --role-arn arn:aws:iam::123456789012:role/EKSClusterRole \\\n --resources-vpc-config \\\n subnetIds=subnet-1,subnet-2,subnet-3,\\\nsecurityGroupIds=sg-1,\\\nendpointPublicAccess=false,\\\nendpointPrivateAccess=true \\\n --kubernetes-network-config serviceIpv4Cidr=10.100.0.0/16 \\\n --logging '{\"clusterLogging\":[{\"types\":[\"api\",\"audit\",\"authenticator\",\"controllerManager\",\"scheduler\"],\"enabled\":true}]}'\n\n# Wait for cluster to be active\naws eks wait cluster-active --name production-cluster\n```\n\n### Describe and List Clusters\n```bash\n# List clusters\naws eks list-clusters\n\n# Describe cluster\naws eks describe-cluster --name production-cluster\n\n# Get cluster endpoint\naws eks describe-cluster \\\n --name production-cluster \\\n --query 'cluster.endpoint' \\\n --output text\n\n# Get cluster OIDC issuer\naws eks describe-cluster \\\n --name production-cluster \\\n --query 'cluster.identity.oidc.issuer' \\\n --output text\n```\n\n### Update Cluster\n```bash\n# Update Kubernetes version\naws eks update-cluster-version \\\n --name production-cluster \\\n --kubernetes-version 1.33\n\n# Update cluster config (logging)\naws eks update-cluster-config \\\n --name production-cluster \\\n --logging '{\"clusterLogging\":[{\"types\":[\"api\",\"audit\"],\"enabled\":true}]}'\n\n# Update endpoint access\naws eks update-cluster-config \\\n --name production-cluster \\\n --resources-vpc-config \\\n endpointPublicAccess=true,\\\nendpointPrivateAccess=true,\\\npublicAccessCidrs=[\"203.0.113.0/24\"]\n```\n\n## Managed Node Groups\n\n### Create Node Group\n```bash\n# On-Demand node group\naws eks create-nodegroup \\\n --cluster-name production-cluster \\\n --nodegroup-name workers-on-demand \\\n --node-role arn:aws:iam::123456789012:role/EKSNodeRole \\\n --subnets subnet-1 subnet-2 subnet-3 \\\n --instance-types m6g.large m6g.xlarge \\\n --capacity-type ON_DEMAND \\\n --scaling-config minSize=2,maxSize=10,desiredSize=3 \\\n --disk-size 50 \\\n --labels environment=production,tier=workers\n\n# Spot node group (cost optimization)\naws eks create-nodegroup \\\n --cluster-name production-cluster \\\n --nodegroup-name workers-spot \\\n --node-role arn:aws:iam::123456789012:role/EKSNodeRole \\\n --subnets subnet-1 subnet-2 subnet-3 \\\n --instance-types m6g.large m6g.xlarge c6g.large c6g.xlarge \\\n --capacity-type SPOT \\\n --scaling-config minSize=0,maxSize=20,desiredSize=5\n\n# With launch template\naws eks create-nodegroup \\\n --cluster-name production-cluster \\\n --nodegroup-name custom-workers \\\n --node-role arn:aws:iam::123456789012:role/EKSNodeRole \\\n --subnets subnet-1 subnet-2 \\\n --launch-template name=eks-node-template,version=1 \\\n --scaling-config minSize=1,maxSize=5,desiredSize=2\n```\n\n### Manage Node Groups\n```bash\n# List node groups\naws eks list-nodegroups --cluster-name production-cluster\n\n# Describe node group\naws eks describe-nodegroup \\\n --cluster-name production-cluster \\\n --nodegroup-name workers-on-demand\n\n# Update scaling\naws eks update-nodegroup-config \\\n --cluster-name production-cluster \\\n --nodegroup-name workers-on-demand \\\n --scaling-config minSize=3,maxSize=15,desiredSize=5\n\n# Update node group version (rolling update)\naws eks update-nodegroup-version \\\n --cluster-name production-cluster \\\n --nodegroup-name workers-on-demand \\\n --kubernetes-version 1.32\n\n# Delete node group\naws eks delete-nodegroup \\\n --cluster-name production-cluster \\\n --nodegroup-name workers-spot\n```\n\n## Fargate Profiles\n\n### Create Fargate Profile\n```bash\naws eks create-fargate-profile \\\n --cluster-name production-cluster \\\n --fargate-profile-name app-profile \\\n --pod-execution-role-arn arn:aws:iam::123456789012:role/EKSFargatePodRole \\\n --subnets subnet-private-1 subnet-private-2 \\\n --selectors \\\n namespace=production,labels={compute=fargate} \\\n namespace=kube-system,labels={k8s-app=kube-dns}\n```\n\n### Manage Fargate Profiles\n```bash\n# List profiles\naws eks list-fargate-profiles --cluster-name production-cluster\n\n# Describe profile\naws eks describe-fargate-profile \\\n --cluster-name production-cluster \\\n --fargate-profile-name app-profile\n\n# Delete profile\naws eks delete-fargate-profile \\\n --cluster-name production-cluster \\\n --fargate-profile-name app-profile\n```\n\n## Add-ons\n\n### Manage Add-ons\n```bash\n# List available add-ons\naws eks describe-addon-versions --kubernetes-version 1.32\n\n# List installed add-ons\naws eks list-addons --cluster-name production-cluster\n\n# Create add-on\naws eks create-addon \\\n --cluster-name production-cluster \\\n --addon-name vpc-cni \\\n --addon-version v1.16.0-eksbuild.1 \\\n --service-account-role-arn arn:aws:iam::123456789012:role/VPCCNIRole\n\naws eks create-addon \\\n --cluster-name production-cluster \\\n --addon-name coredns\n\naws eks create-addon \\\n --cluster-name production-cluster \\\n --addon-name kube-proxy\n\naws eks create-addon \\\n --cluster-name production-cluster \\\n --addon-name aws-ebs-csi-driver \\\n --service-account-role-arn arn:aws:iam::123456789012:role/EBSCSIRole\n\n# Update add-on\naws eks update-addon \\\n --cluster-name production-cluster \\\n --addon-name vpc-cni \\\n --addon-version v1.17.0-eksbuild.1\n\n# Delete add-on\naws eks delete-addon \\\n --cluster-name production-cluster \\\n --addon-name aws-ebs-csi-driver\n```\n\n## Kubeconfig Setup\n\n### Basic Configuration\n```bash\n# Update kubeconfig (uses current AWS credentials)\naws eks update-kubeconfig \\\n --name production-cluster \\\n --region us-east-1\n\n# With custom alias\naws eks update-kubeconfig \\\n --name production-cluster \\\n --alias prod-cluster\n\n# With role assumption (cross-account or different role)\naws eks update-kubeconfig \\\n --name production-cluster \\\n --role-arn arn:aws:iam::123456789012:role/EKSAdminRole\n\n# Dry run (show kubeconfig without writing)\naws eks update-kubeconfig \\\n --name production-cluster \\\n --dry-run\n```\n\n### Verify Access\n```bash\n# Test connection\nkubectl get nodes\nkubectl get pods -A\n\n# Get cluster info\nkubectl cluster-info\n```\n\n## IAM Roles for Service Accounts (IRSA)\n\nIRSA allows Kubernetes pods to assume IAM roles without using node-level credentials.\n\n### Step 1: Create OIDC Provider\n```bash\n# Get OIDC issuer URL\nOIDC_URL=$(aws eks describe-cluster \\\n --name production-cluster \\\n --query 'cluster.identity.oidc.issuer' \\\n --output text)\n\n# Extract OIDC ID\nOIDC_ID=$(echo $OIDC_URL | sed 's|https://||')\n\n# Create OIDC provider (if not exists)\naws iam create-open-id-connect-provider \\\n --url $OIDC_URL \\\n --client-id-list sts.amazonaws.com \\\n --thumbprint-list $(openssl s_client -servername oidc.eks.us-east-1.amazonaws.com \\\n -connect oidc.eks.us-east-1.amazonaws.com:443 2>/dev/null \\\n | openssl x509 -fingerprint -sha1 -noout \\\n | sed 's/://g' | cut -d= -f2 | tr '[:upper:]' '[:lower:]')\n```\n\n### Step 2: Create IAM Role with Trust Policy\n```bash\n# Create trust policy file\ncat > trust-policy.json \u003c\u003c EOF\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Federated\": \"arn:aws:iam::123456789012:oidc-provider/${OIDC_ID}\"\n },\n \"Action\": \"sts:AssumeRoleWithWebIdentity\",\n \"Condition\": {\n \"StringEquals\": {\n \"${OIDC_ID}:sub\": \"system:serviceaccount:default:my-service-account\",\n \"${OIDC_ID}:aud\": \"sts.amazonaws.com\"\n }\n }\n }]\n}\nEOF\n\n# Create role\naws iam create-role \\\n --role-name MyPodRole \\\n --assume-role-policy-document file://trust-policy.json\n\n# Attach policy\naws iam attach-role-policy \\\n --role-name MyPodRole \\\n --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\n```\n\n### Step 3: Annotate Service Account\n```bash\n# Create service account with annotation (kubectl)\nkubectl create serviceaccount my-service-account\nkubectl annotate serviceaccount my-service-account \\\n eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/MyPodRole\n```\n\n## Accessing Private EKS Clusters\n\n### Method 1: SSM Port Forwarding\n```bash\n# Get EKS API endpoint (without https://)\nEKS_ENDPOINT=$(aws eks describe-cluster \\\n --name production-cluster \\\n --query 'cluster.endpoint' \\\n --output text | sed 's|https://||')\n\n# Start port forwarding through bastion\naws ssm start-session \\\n --target i-bastion-instance-id \\\n --document-name AWS-StartPortForwardingSessionToRemoteHost \\\n --parameters \"{\\\"host\\\":[\\\"$EKS_ENDPOINT\\\"],\\\"portNumber\\\":[\\\"443\\\"],\\\"localPortNumber\\\":[\\\"9443\\\"]}\"\n\n# In another terminal, update kubeconfig\naws eks update-kubeconfig --name production-cluster\n\n# Modify kubeconfig to use localhost:9443\nkubectl config set-cluster arn:aws:eks:us-east-1:123456789012:cluster/production-cluster \\\n --server=https://localhost:9443\n\n# Disable certificate verification (for port forwarding)\nkubectl config set-cluster arn:aws:eks:us-east-1:123456789012:cluster/production-cluster \\\n --insecure-skip-tls-verify=true\n\n# Test access\nkubectl get nodes\n```\n\n### Method 2: SOCKS Proxy via SSH over SSM\n\n**Configure SSH (~/.ssh/config):**\n```\nHost i-* mi-*\n ProxyCommand sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'\"\n User ec2-user\n StrictHostKeyChecking no\n```\n\n**Start SOCKS proxy:**\n```bash\n# Start SSH SOCKS proxy through SSM bastion\nssh -N -D 127.0.0.1:1080 i-bastion-instance-id &\n\n# Configure kubectl to use proxy\nexport HTTPS_PROXY=socks5h://127.0.0.1:1080\n\n# Update kubeconfig normally\naws eks update-kubeconfig --name production-cluster\n\n# Use kubectl (traffic goes through SOCKS proxy)\nkubectl get nodes\n```\n\n### Prerequisites for Private Cluster Access\n```bash\n# Required VPC endpoints\naws ec2 create-vpc-endpoint \\\n --vpc-id vpc-xxxxx \\\n --service-name com.amazonaws.us-east-1.ssm \\\n --vpc-endpoint-type Interface \\\n --subnet-ids subnet-xxxxx \\\n --security-group-ids sg-xxxxx\n\naws ec2 create-vpc-endpoint \\\n --vpc-id vpc-xxxxx \\\n --service-name com.amazonaws.us-east-1.ssmmessages \\\n --vpc-endpoint-type Interface \\\n --subnet-ids subnet-xxxxx \\\n --security-group-ids sg-xxxxx\n\naws ec2 create-vpc-endpoint \\\n --vpc-id vpc-xxxxx \\\n --service-name com.amazonaws.us-east-1.ec2messages \\\n --vpc-endpoint-type Interface \\\n --subnet-ids subnet-xxxxx \\\n --security-group-ids sg-xxxxx\n```\n\n## Access Entries (EKS Access Management)\n\n### Create Access Entry\n```bash\n# Grant cluster access to IAM principal\naws eks create-access-entry \\\n --cluster-name production-cluster \\\n --principal-arn arn:aws:iam::123456789012:role/DeveloperRole \\\n --type STANDARD\n\n# Associate access policy\naws eks associate-access-policy \\\n --cluster-name production-cluster \\\n --principal-arn arn:aws:iam::123456789012:role/DeveloperRole \\\n --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy \\\n --access-scope type=namespace,namespaces=default\n\n# Admin access\naws eks associate-access-policy \\\n --cluster-name production-cluster \\\n --principal-arn arn:aws:iam::123456789012:role/AdminRole \\\n --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \\\n --access-scope type=cluster\n```\n\n### Manage Access Entries\n```bash\n# List access entries\naws eks list-access-entries --cluster-name production-cluster\n\n# Describe access entry\naws eks describe-access-entry \\\n --cluster-name production-cluster \\\n --principal-arn arn:aws:iam::123456789012:role/DeveloperRole\n\n# Delete access entry\naws eks delete-access-entry \\\n --cluster-name production-cluster \\\n --principal-arn arn:aws:iam::123456789012:role/DeveloperRole\n```\n\n## Useful Queries\n\n```bash\n# Get node group status\naws eks describe-nodegroup \\\n --cluster-name production-cluster \\\n --nodegroup-name workers-on-demand \\\n --query 'nodegroup.status'\n\n# List all node groups with status\naws eks list-nodegroups --cluster-name production-cluster \\\n --query 'nodegroups[]' --output text | \\\n xargs -I {} aws eks describe-nodegroup \\\n --cluster-name production-cluster \\\n --nodegroup-name {} \\\n --query '{name:nodegroup.nodegroupName,status:nodegroup.status}'\n\n# Get cluster CA certificate\naws eks describe-cluster \\\n --name production-cluster \\\n --query 'cluster.certificateAuthority.data' \\\n --output text | base64 -d\n\n# Check cluster health\naws eks describe-cluster \\\n --name production-cluster \\\n --query 'cluster.{status:status,version:version,endpoint:endpoint}'\n```\n\n## Waiters\n\n```bash\n# Wait for cluster creation\naws eks wait cluster-active --name production-cluster\n\n# Wait for node group creation\naws eks wait nodegroup-active \\\n --cluster-name production-cluster \\\n --nodegroup-name workers-on-demand\n\n# Wait for node group deletion\naws eks wait nodegroup-deleted \\\n --cluster-name production-cluster \\\n --nodegroup-name old-workers\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Private endpoints** | Use private cluster endpoints for production |\n| **IRSA** | Use IAM Roles for Service Accounts instead of node role |\n| **Spot instances** | Use Spot for stateless, fault-tolerant workloads |\n| **Managed node groups** | Use managed node groups for easier operations |\n| **Add-on management** | Use EKS add-ons for VPC CNI, CoreDNS, kube-proxy |\n| **Version alignment** | Keep control plane and node group versions aligned |\n| **Access entries** | Use EKS access entries for cluster access management |\n| **Logging** | Enable control plane logging for audit and troubleshooting |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":13783,"content_sha256":"bc4cb6132d49e52dbffb80dfc102eebc70e9363171d3373662434abe68bb2146"},{"filename":"references/github-cicd.md","content":"# GitHub Actions CI/CD\n\n## OIDC Integration (No Long-lived Keys)\n\n### 1. Create OIDC Provider\nOne-time setup per AWS account.\n\n```bash\n# Create OIDC provider\naws iam create-open-id-connect-provider \\\n --url \"https://token.actions.githubusercontent.com\" \\\n --client-id-list \"sts.amazonaws.com\" \\\n --thumbprint-list \"6938fd4d98bab03faadb97b34396831e3780aea1\" \\\n --tags Key=Purpose,Value=GitHubActions\n\n# List OIDC providers\naws iam list-open-id-connect-providers\n\n# Get provider details\naws iam get-open-id-connect-provider \\\n --open-id-connect-provider-arn arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com\n```\n\n### 2. Create IAM Role\n\n**Basic Trust Policy (`github-trust.json`):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Federated\": \"arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com\"\n },\n \"Action\": \"sts:AssumeRoleWithWebIdentity\",\n \"Condition\": {\n \"StringEquals\": {\n \"token.actions.githubusercontent.com:aud\": \"sts.amazonaws.com\"\n },\n \"StringLike\": {\n \"token.actions.githubusercontent.com:sub\": \"repo:my-org/my-repo:*\"\n }\n }\n }\n ]\n}\n```\n\n**Restricted Trust Policy (main branch only):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Federated\": \"arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com\"\n },\n \"Action\": \"sts:AssumeRoleWithWebIdentity\",\n \"Condition\": {\n \"StringEquals\": {\n \"token.actions.githubusercontent.com:aud\": \"sts.amazonaws.com\",\n \"token.actions.githubusercontent.com:sub\": \"repo:my-org/my-repo:ref:refs/heads/main\"\n }\n }\n }\n ]\n}\n```\n\n**Environment-based Trust Policy:**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Federated\": \"arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com\"\n },\n \"Action\": \"sts:AssumeRoleWithWebIdentity\",\n \"Condition\": {\n \"StringEquals\": {\n \"token.actions.githubusercontent.com:aud\": \"sts.amazonaws.com\"\n },\n \"StringLike\": {\n \"token.actions.githubusercontent.com:sub\": \"repo:my-org/my-repo:environment:production\"\n }\n }\n }\n ]\n}\n```\n\n**Multi-repo Trust Policy:**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Federated\": \"arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com\"\n },\n \"Action\": \"sts:AssumeRoleWithWebIdentity\",\n \"Condition\": {\n \"StringEquals\": {\n \"token.actions.githubusercontent.com:aud\": \"sts.amazonaws.com\"\n },\n \"StringLike\": {\n \"token.actions.githubusercontent.com:sub\": [\n \"repo:my-org/frontend:*\",\n \"repo:my-org/backend:*\",\n \"repo:my-org/infrastructure:ref:refs/heads/main\"\n ]\n }\n }\n }\n ]\n}\n```\n\n```bash\n# Create role\naws iam create-role \\\n --role-name GitHubDeployRole \\\n --assume-role-policy-document file://github-trust.json \\\n --description \"Role for GitHub Actions deployments\" \\\n --max-session-duration 3600 \\\n --tags Key=Purpose,Value=CICD\n\n# Attach managed policies\naws iam attach-role-policy \\\n --role-name GitHubDeployRole \\\n --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess\n\naws iam attach-role-policy \\\n --role-name GitHubDeployRole \\\n --policy-arn arn:aws:iam::aws:policy/AmazonECR-FullAccess\n\n# Or attach custom policy\naws iam put-role-policy \\\n --role-name GitHubDeployRole \\\n --policy-name DeploymentPolicy \\\n --policy-document file://deploy-policy.json\n```\n\n## Workflow Configurations\n\n### Basic S3 Deployment\n```yaml\nname: Deploy to S3\non:\n push:\n branches: [main]\n\npermissions:\n id-token: write # Required for OIDC\n contents: read\n\njobs:\n deploy:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n\n - name: Configure AWS Credentials\n uses: aws-actions/configure-aws-credentials@v4\n with:\n role-to-assume: arn:aws:iam::123456789012:role/GitHubDeployRole\n aws-region: us-east-1\n\n - name: Deploy to S3\n run: aws s3 sync ./dist s3://my-bucket/ --delete\n```\n\n### ECR Push and ECS Deploy\n```yaml\nname: Build and Deploy\non:\n push:\n branches: [main]\n\npermissions:\n id-token: write\n contents: read\n\nenv:\n AWS_REGION: us-east-1\n ECR_REPOSITORY: my-app\n ECS_SERVICE: my-service\n ECS_CLUSTER: my-cluster\n CONTAINER_NAME: app\n\njobs:\n deploy:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n\n - name: Configure AWS credentials\n uses: aws-actions/configure-aws-credentials@v4\n with:\n role-to-assume: arn:aws:iam::123456789012:role/GitHubDeployRole\n aws-region: ${{ env.AWS_REGION }}\n\n - name: Login to Amazon ECR\n id: login-ecr\n uses: aws-actions/amazon-ecr-login@v2\n\n - name: Build and push image\n id: build-image\n env:\n ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}\n IMAGE_TAG: ${{ github.sha }}\n run: |\n docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .\n docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG\n echo \"image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG\" >> $GITHUB_OUTPUT\n\n - name: Deploy to ECS\n uses: aws-actions/amazon-ecs-deploy-task-definition@v1\n with:\n task-definition: task-definition.json\n service: ${{ env.ECS_SERVICE }}\n cluster: ${{ env.ECS_CLUSTER }}\n wait-for-service-stability: true\n```\n\n### Lambda Deployment\n```yaml\nname: Deploy Lambda\non:\n push:\n branches: [main]\n paths:\n - 'src/**'\n - 'requirements.txt'\n\npermissions:\n id-token: write\n contents: read\n\njobs:\n deploy:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n\n - name: Set up Python\n uses: actions/setup-python@v5\n with:\n python-version: '3.11'\n\n - name: Configure AWS credentials\n uses: aws-actions/configure-aws-credentials@v4\n with:\n role-to-assume: arn:aws:iam::123456789012:role/GitHubDeployRole\n aws-region: us-east-1\n\n - name: Package and deploy\n run: |\n pip install -r requirements.txt -t package/\n cp -r src/* package/\n cd package && zip -r ../function.zip .\n aws lambda update-function-code \\\n --function-name my-function \\\n --zip-file fileb://function.zip\n```\n\n### Multi-Environment Deployment\n```yaml\nname: Deploy\non:\n push:\n branches: [main, develop]\n workflow_dispatch:\n inputs:\n environment:\n description: 'Environment to deploy'\n required: true\n type: choice\n options:\n - staging\n - production\n\npermissions:\n id-token: write\n contents: read\n\njobs:\n deploy:\n runs-on: ubuntu-latest\n environment: ${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production' || 'staging') }}\n steps:\n - uses: actions/checkout@v4\n\n - name: Configure AWS credentials (Staging)\n if: github.ref != 'refs/heads/main'\n uses: aws-actions/configure-aws-credentials@v4\n with:\n role-to-assume: arn:aws:iam::111111111111:role/GitHubDeployRole\n aws-region: us-east-1\n\n - name: Configure AWS credentials (Production)\n if: github.ref == 'refs/heads/main'\n uses: aws-actions/configure-aws-credentials@v4\n with:\n role-to-assume: arn:aws:iam::222222222222:role/GitHubDeployRole\n aws-region: us-east-1\n\n - name: Deploy\n run: |\n echo \"Deploying to ${{ github.event.inputs.environment || 'auto-detected' }}\"\n # deployment commands\n```\n\n## AWS CodeBuild Integration\n\n### Create CodeBuild Project\n```bash\n# Create CodeBuild project\naws codebuild create-project \\\n --name my-build-project \\\n --source '{\n \"type\": \"GITHUB\",\n \"location\": \"https://github.com/my-org/my-repo.git\",\n \"buildspec\": \"buildspec.yml\",\n \"auth\": {\n \"type\": \"OAUTH\"\n }\n }' \\\n --artifacts '{\n \"type\": \"S3\",\n \"location\": \"my-artifacts-bucket\",\n \"name\": \"build-output\"\n }' \\\n --environment '{\n \"type\": \"LINUX_CONTAINER\",\n \"image\": \"aws/codebuild/amazonlinux2-x86_64-standard:5.0\",\n \"computeType\": \"BUILD_GENERAL1_SMALL\",\n \"privilegedMode\": true,\n \"environmentVariables\": [\n {\"name\": \"AWS_DEFAULT_REGION\", \"value\": \"us-east-1\"},\n {\"name\": \"ECR_REPO\", \"value\": \"123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app\"}\n ]\n }' \\\n --service-role arn:aws:iam::123456789012:role/CodeBuildServiceRole\n\n# Create webhook for GitHub\naws codebuild create-webhook \\\n --project-name my-build-project \\\n --filter-groups '[[\n {\"type\": \"EVENT\", \"pattern\": \"PUSH\"},\n {\"type\": \"HEAD_REF\", \"pattern\": \"^refs/heads/main$\"}\n ]]'\n```\n\n### CodeBuild buildspec.yml\n```yaml\nversion: 0.2\n\nenv:\n variables:\n AWS_DEFAULT_REGION: us-east-1\n secrets-manager:\n DOCKER_HUB_TOKEN: docker-hub-credentials:token\n\nphases:\n pre_build:\n commands:\n - echo Logging in to Amazon ECR...\n - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $ECR_REPO\n - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)\n - IMAGE_TAG=${COMMIT_HASH:=latest}\n\n build:\n commands:\n - echo Build started on `date`\n - docker build -t $ECR_REPO:$IMAGE_TAG .\n - docker tag $ECR_REPO:$IMAGE_TAG $ECR_REPO:latest\n\n post_build:\n commands:\n - echo Build completed on `date`\n - docker push $ECR_REPO:$IMAGE_TAG\n - docker push $ECR_REPO:latest\n - printf '[{\"name\":\"app\",\"imageUri\":\"%s\"}]' $ECR_REPO:$IMAGE_TAG > imagedefinitions.json\n\nartifacts:\n files:\n - imagedefinitions.json\n\ncache:\n paths:\n - '/root/.docker/**/*'\n```\n\n### Manage CodeBuild\n```bash\n# Start build\naws codebuild start-build \\\n --project-name my-build-project \\\n --source-version main\n\n# Start build with overrides\naws codebuild start-build \\\n --project-name my-build-project \\\n --environment-variables-override '[\n {\"name\": \"DEPLOY_ENV\", \"value\": \"staging\"}\n ]'\n\n# Get build status\naws codebuild batch-get-builds \\\n --ids my-build-project:build-id \\\n --query 'builds[0].{Status:buildStatus,Phase:currentPhase}'\n\n# List builds\naws codebuild list-builds-for-project \\\n --project-name my-build-project \\\n --sort-order DESCENDING\n\n# Stop build\naws codebuild stop-build --id my-build-project:build-id\n\n# List projects\naws codebuild list-projects\n\n# Delete project\naws codebuild delete-project --name my-build-project\n```\n\n## CodePipeline\n\n### Create Pipeline\n```bash\n# Create pipeline\naws codepipeline create-pipeline \\\n --pipeline file://pipeline.json\n\n# Get pipeline state\naws codepipeline get-pipeline-state \\\n --name my-pipeline\n\n# Start pipeline execution\naws codepipeline start-pipeline-execution \\\n --name my-pipeline\n\n# List pipelines\naws codepipeline list-pipelines\n\n# Get pipeline\naws codepipeline get-pipeline \\\n --name my-pipeline\n\n# Delete pipeline\naws codepipeline delete-pipeline \\\n --name my-pipeline\n```\n\n**pipeline.json:**\n```json\n{\n \"pipeline\": {\n \"name\": \"my-pipeline\",\n \"roleArn\": \"arn:aws:iam::123456789012:role/CodePipelineServiceRole\",\n \"stages\": [\n {\n \"name\": \"Source\",\n \"actions\": [\n {\n \"name\": \"GitHub\",\n \"actionTypeId\": {\n \"category\": \"Source\",\n \"owner\": \"AWS\",\n \"provider\": \"CodeStarSourceConnection\",\n \"version\": \"1\"\n },\n \"configuration\": {\n \"ConnectionArn\": \"arn:aws:codestar-connections:us-east-1:123456789012:connection/abc-123\",\n \"FullRepositoryId\": \"my-org/my-repo\",\n \"BranchName\": \"main\"\n },\n \"outputArtifacts\": [{\"name\": \"SourceOutput\"}]\n }\n ]\n },\n {\n \"name\": \"Build\",\n \"actions\": [\n {\n \"name\": \"CodeBuild\",\n \"actionTypeId\": {\n \"category\": \"Build\",\n \"owner\": \"AWS\",\n \"provider\": \"CodeBuild\",\n \"version\": \"1\"\n },\n \"configuration\": {\n \"ProjectName\": \"my-build-project\"\n },\n \"inputArtifacts\": [{\"name\": \"SourceOutput\"}],\n \"outputArtifacts\": [{\"name\": \"BuildOutput\"}]\n }\n ]\n },\n {\n \"name\": \"Deploy\",\n \"actions\": [\n {\n \"name\": \"ECS\",\n \"actionTypeId\": {\n \"category\": \"Deploy\",\n \"owner\": \"AWS\",\n \"provider\": \"ECS\",\n \"version\": \"1\"\n },\n \"configuration\": {\n \"ClusterName\": \"my-cluster\",\n \"ServiceName\": \"my-service\",\n \"FileName\": \"imagedefinitions.json\"\n },\n \"inputArtifacts\": [{\"name\": \"BuildOutput\"}]\n }\n ]\n }\n ]\n }\n}\n```\n\n## Useful Queries\n\n```bash\n# Get OIDC provider ARN\naws iam list-open-id-connect-providers \\\n --query 'OpenIDConnectProviderList[?contains(Arn, `github`)].Arn'\n\n# List roles with GitHub trust\naws iam list-roles \\\n --query 'Roles[?contains(AssumeRolePolicyDocument|to_string(@), `github`)].RoleName'\n\n# Get recent CodeBuild builds\naws codebuild list-builds \\\n --sort-order DESCENDING \\\n --max-items 10\n\n# Get failed builds\naws codebuild batch-get-builds \\\n --ids $(aws codebuild list-builds-for-project \\\n --project-name my-project \\\n --query 'ids[:5]' \\\n --output text) \\\n --query 'builds[?buildStatus==`FAILED`].{ID:id,Phase:phases[-1].phaseType}'\n\n# Get pipeline execution history\naws codepipeline list-pipeline-executions \\\n --pipeline-name my-pipeline \\\n --query 'pipelineExecutionSummaries[*].{ID:pipelineExecutionId,Status:status,Time:startTime}'\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **OIDC** | Use OIDC instead of long-lived access keys |\n| **Branch restrictions** | Limit trust policy to specific branches |\n| **Environment protection** | Use GitHub environments for approvals |\n| **Least privilege** | Grant minimal permissions to deploy role |\n| **Audit logging** | Enable CloudTrail for AssumeRoleWithWebIdentity |\n| **Session duration** | Set short max-session-duration on roles |\n| **Multi-account** | Use separate roles/accounts per environment |\n| **Secrets** | Use Secrets Manager, not GitHub secrets for AWS |\n| **CodeBuild cache** | Enable caching to speed up builds |\n| **Webhook filters** | Filter CodeBuild webhooks to relevant events |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":16116,"content_sha256":"10622c825ad212dace0f109e1c2a72e2025830a19c07134b4bdd24ad33b8ec14"},{"filename":"references/glue.md","content":"# AWS Glue\n\n## Data Catalog\n\n### Databases\n```bash\n# Create database\naws glue create-database \\\n --database-input '{\n \"Name\": \"analytics\",\n \"Description\": \"Data Lake Catalog\",\n \"LocationUri\": \"s3://my-datalake/\"\n }'\n\n# List databases\naws glue get-databases\n\n# Get database\naws glue get-database --name analytics\n\n# Delete database\naws glue delete-database --name analytics\n```\n\n### Tables\n```bash\n# Create table\naws glue create-table \\\n --database-name analytics \\\n --table-input file://table-definition.json\n\n# Get table\naws glue get-table --database-name analytics --name raw_logs\n\n# List tables\naws glue get-tables --database-name analytics\n\n# Update table\naws glue update-table \\\n --database-name analytics \\\n --table-input file://updated-table.json\n\n# Delete table\naws glue delete-table --database-name analytics --name raw_logs\n```\n\n**table-definition.json:**\n```json\n{\n \"Name\": \"raw_logs\",\n \"StorageDescriptor\": {\n \"Columns\": [\n {\"Name\": \"timestamp\", \"Type\": \"timestamp\"},\n {\"Name\": \"user_id\", \"Type\": \"string\"},\n {\"Name\": \"event\", \"Type\": \"string\"},\n {\"Name\": \"properties\", \"Type\": \"map\u003cstring,string>\"}\n ],\n \"Location\": \"s3://my-datalake/raw/logs/\",\n \"InputFormat\": \"org.apache.hadoop.mapred.TextInputFormat\",\n \"OutputFormat\": \"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\",\n \"SerdeInfo\": {\n \"SerializationLibrary\": \"org.openx.data.jsonserde.JsonSerDe\"\n }\n },\n \"PartitionKeys\": [\n {\"Name\": \"year\", \"Type\": \"string\"},\n {\"Name\": \"month\", \"Type\": \"string\"},\n {\"Name\": \"day\", \"Type\": \"string\"}\n ],\n \"TableType\": \"EXTERNAL_TABLE\"\n}\n```\n\n### Partitions\n```bash\n# Get partitions\naws glue get-partitions \\\n --database-name analytics \\\n --table-name raw_logs\n\n# Create partition\naws glue create-partition \\\n --database-name analytics \\\n --table-name raw_logs \\\n --partition-input '{\n \"Values\": [\"2024\", \"01\", \"15\"],\n \"StorageDescriptor\": {\n \"Location\": \"s3://my-datalake/raw/logs/year=2024/month=01/day=15/\"\n }\n }'\n\n# Batch create partitions\naws glue batch-create-partition \\\n --database-name analytics \\\n --table-name raw_logs \\\n --partition-input-list file://partitions.json\n```\n\n## Crawlers\n\n### Create Crawler\n```bash\n# S3 crawler\naws glue create-crawler \\\n --name s3-data-crawler \\\n --role arn:aws:iam::123456789012:role/GlueServiceRole \\\n --database-name analytics \\\n --targets '{\n \"S3Targets\": [\n {\"Path\": \"s3://my-datalake/raw/\"},\n {\"Path\": \"s3://my-datalake/processed/\"}\n ]\n }' \\\n --schedule \"cron(0 12 * * ? *)\" \\\n --schema-change-policy '{\n \"UpdateBehavior\": \"UPDATE_IN_DATABASE\",\n \"DeleteBehavior\": \"LOG\"\n }' \\\n --configuration '{\n \"Version\": 1.0,\n \"CrawlerOutput\": {\n \"Partitions\": {\"AddOrUpdateBehavior\": \"InheritFromTable\"}\n },\n \"Grouping\": {\"TableGroupingPolicy\": \"CombineCompatibleSchemas\"}\n }'\n\n# JDBC crawler (database)\naws glue create-crawler \\\n --name rds-crawler \\\n --role arn:aws:iam::123456789012:role/GlueServiceRole \\\n --database-name analytics \\\n --targets '{\n \"JdbcTargets\": [{\n \"ConnectionName\": \"rds-connection\",\n \"Path\": \"mydb/%\"\n }]\n }'\n```\n\n### Manage Crawlers\n```bash\n# Start crawler\naws glue start-crawler --name s3-data-crawler\n\n# Stop crawler\naws glue stop-crawler --name s3-data-crawler\n\n# Get crawler status\naws glue get-crawler --name s3-data-crawler\n\n# List crawlers\naws glue list-crawlers\n\n# Update crawler\naws glue update-crawler \\\n --name s3-data-crawler \\\n --schedule \"cron(0 */6 * * ? *)\"\n\n# Delete crawler\naws glue delete-crawler --name s3-data-crawler\n```\n\n## ETL Jobs\n\n### Create Job\n```bash\n# Spark ETL job\naws glue create-job \\\n --name parquet-transformer \\\n --role arn:aws:iam::123456789012:role/GlueServiceRole \\\n --command '{\n \"Name\": \"glueetl\",\n \"ScriptLocation\": \"s3://my-scripts/transform.py\",\n \"PythonVersion\": \"3\"\n }' \\\n --glue-version \"4.0\" \\\n --worker-type G.1X \\\n --number-of-workers 10 \\\n --default-arguments '{\n \"--job-bookmark-option\": \"job-bookmark-enable\",\n \"--enable-metrics\": \"true\",\n \"--enable-continuous-cloudwatch-log\": \"true\",\n \"--TempDir\": \"s3://my-temp-bucket/glue/\"\n }'\n\n# Python Shell job (simpler, cheaper)\naws glue create-job \\\n --name simple-etl \\\n --role arn:aws:iam::123456789012:role/GlueServiceRole \\\n --command '{\n \"Name\": \"pythonshell\",\n \"ScriptLocation\": \"s3://my-scripts/simple.py\",\n \"PythonVersion\": \"3.9\"\n }' \\\n --max-capacity 0.0625\n```\n\n### Run Jobs\n```bash\n# Start job run\naws glue start-job-run \\\n --job-name parquet-transformer \\\n --arguments '{\n \"--input_path\": \"s3://source-bucket/data/\",\n \"--output_path\": \"s3://dest-bucket/processed/\"\n }'\n\n# Start with override settings\naws glue start-job-run \\\n --job-name parquet-transformer \\\n --worker-type G.2X \\\n --number-of-workers 20\n\n# Get job run status\naws glue get-job-run \\\n --job-name parquet-transformer \\\n --run-id jr_abc123\n\n# List job runs\naws glue get-job-runs --job-name parquet-transformer\n\n# Stop job run\naws glue batch-stop-job-run \\\n --job-name parquet-transformer \\\n --job-run-ids jr_abc123\n```\n\n### Manage Jobs\n```bash\n# List jobs\naws glue list-jobs\n\n# Get job details\naws glue get-job --job-name parquet-transformer\n\n# Update job\naws glue update-job \\\n --job-name parquet-transformer \\\n --job-update '{\n \"NumberOfWorkers\": 15,\n \"WorkerType\": \"G.2X\"\n }'\n\n# Delete job\naws glue delete-job --job-name parquet-transformer\n\n# Reset job bookmark\naws glue reset-job-bookmark --job-name parquet-transformer\n```\n\n## Triggers\n\n### Create Triggers\n```bash\n# Scheduled trigger\naws glue create-trigger \\\n --name daily-etl-trigger \\\n --type SCHEDULED \\\n --schedule \"cron(0 8 * * ? *)\" \\\n --actions '[{\"JobName\": \"parquet-transformer\"}]' \\\n --start-on-creation\n\n# On-demand trigger\naws glue create-trigger \\\n --name manual-trigger \\\n --type ON_DEMAND \\\n --actions '[{\"JobName\": \"parquet-transformer\"}]'\n\n# Conditional trigger (run when another job succeeds)\naws glue create-trigger \\\n --name conditional-trigger \\\n --type CONDITIONAL \\\n --predicate '{\n \"Logical\": \"AND\",\n \"Conditions\": [{\n \"LogicalOperator\": \"EQUALS\",\n \"JobName\": \"extract-job\",\n \"State\": \"SUCCEEDED\"\n }]\n }' \\\n --actions '[{\"JobName\": \"transform-job\"}]' \\\n --start-on-creation\n\n# Event trigger (EventBridge)\naws glue create-trigger \\\n --name event-trigger \\\n --type EVENT \\\n --actions '[{\"JobName\": \"parquet-transformer\"}]'\n```\n\n### Manage Triggers\n```bash\n# Start trigger\naws glue start-trigger --name daily-etl-trigger\n\n# Stop trigger\naws glue stop-trigger --name daily-etl-trigger\n\n# Activate on-demand trigger\naws glue start-trigger --name manual-trigger\n\n# Get trigger\naws glue get-trigger --name daily-etl-trigger\n\n# List triggers\naws glue list-triggers\n\n# Delete trigger\naws glue delete-trigger --name daily-etl-trigger\n```\n\n## Workflows\n\n### Create Workflow\n```bash\n# Create workflow\naws glue create-workflow \\\n --name data-pipeline \\\n --description \"Daily data processing pipeline\"\n\n# Add triggers to workflow\naws glue create-trigger \\\n --name workflow-start \\\n --workflow-name data-pipeline \\\n --type SCHEDULED \\\n --schedule \"cron(0 8 * * ? *)\" \\\n --actions '[{\"JobName\": \"extract-job\"}]'\n\naws glue create-trigger \\\n --name extract-complete \\\n --workflow-name data-pipeline \\\n --type CONDITIONAL \\\n --predicate '{\"Conditions\": [{\"JobName\": \"extract-job\", \"State\": \"SUCCEEDED\"}]}' \\\n --actions '[{\"JobName\": \"transform-job\"}]'\n\naws glue create-trigger \\\n --name transform-complete \\\n --workflow-name data-pipeline \\\n --type CONDITIONAL \\\n --predicate '{\"Conditions\": [{\"JobName\": \"transform-job\", \"State\": \"SUCCEEDED\"}]}' \\\n --actions '[{\"JobName\": \"load-job\"}]'\n```\n\n### Manage Workflows\n```bash\n# Start workflow run\naws glue start-workflow-run --name data-pipeline\n\n# Get workflow\naws glue get-workflow --name data-pipeline\n\n# Get workflow run\naws glue get-workflow-run \\\n --name data-pipeline \\\n --run-id wr_abc123\n\n# List workflow runs\naws glue get-workflow-runs --name data-pipeline\n\n# Stop workflow run\naws glue stop-workflow-run \\\n --name data-pipeline \\\n --run-id wr_abc123\n\n# Delete workflow\naws glue delete-workflow --name data-pipeline\n```\n\n## Connections\n\n### Create Connection\n```bash\n# JDBC connection (RDS/Aurora)\naws glue create-connection \\\n --connection-input '{\n \"Name\": \"rds-connection\",\n \"ConnectionType\": \"JDBC\",\n \"ConnectionProperties\": {\n \"JDBC_CONNECTION_URL\": \"jdbc:postgresql://mydb.abc123.us-east-1.rds.amazonaws.com:5432/mydb\",\n \"USERNAME\": \"admin\",\n \"PASSWORD\": \"password\"\n },\n \"PhysicalConnectionRequirements\": {\n \"SubnetId\": \"subnet-12345\",\n \"SecurityGroupIdList\": [\"sg-12345\"],\n \"AvailabilityZone\": \"us-east-1a\"\n }\n }'\n\n# Network connection (for VPC resources)\naws glue create-connection \\\n --connection-input '{\n \"Name\": \"vpc-connection\",\n \"ConnectionType\": \"NETWORK\",\n \"PhysicalConnectionRequirements\": {\n \"SubnetId\": \"subnet-12345\",\n \"SecurityGroupIdList\": [\"sg-12345\"]\n }\n }'\n```\n\n### Manage Connections\n```bash\n# List connections\naws glue get-connections\n\n# Get connection\naws glue get-connection --name rds-connection\n\n# Test connection (via job)\naws glue start-job-run \\\n --job-name connection-test \\\n --arguments '{\"--connection_name\": \"rds-connection\"}'\n\n# Delete connection\naws glue delete-connection --name rds-connection\n```\n\n## Schema Registry\n\n### Create Registry\n```bash\naws glue create-registry \\\n --registry-name streaming-schemas \\\n --description \"Schemas for streaming data\"\n```\n\n### Create Schema\n```bash\n# Avro schema\naws glue create-schema \\\n --registry-id RegistryName=streaming-schemas \\\n --schema-name user-events \\\n --data-format AVRO \\\n --compatibility BACKWARD \\\n --schema-definition file://schema.avsc\n\n# JSON schema\naws glue create-schema \\\n --registry-id RegistryName=streaming-schemas \\\n --schema-name api-logs \\\n --data-format JSON \\\n --compatibility FULL \\\n --schema-definition file://schema.json\n```\n\n### Manage Schemas\n```bash\n# List schemas\naws glue list-schemas --registry-id RegistryName=streaming-schemas\n\n# Get schema\naws glue get-schema --schema-id SchemaName=user-events,RegistryName=streaming-schemas\n\n# Register new version\naws glue register-schema-version \\\n --schema-id SchemaName=user-events,RegistryName=streaming-schemas \\\n --schema-definition file://schema-v2.avsc\n\n# List schema versions\naws glue list-schema-versions \\\n --schema-id SchemaName=user-events,RegistryName=streaming-schemas\n```\n\n## Useful Queries\n\n```bash\n# Get job run status\naws glue get-job-runs \\\n --job-name my-job \\\n --query 'JobRuns[0].{Status:JobRunState,Started:StartedOn,Duration:ExecutionTime}'\n\n# Find failed job runs\naws glue get-job-runs \\\n --job-name my-job \\\n --query 'JobRuns[?JobRunState==`FAILED`].{RunId:Id,Error:ErrorMessage}'\n\n# List all tables in database\naws glue get-tables \\\n --database-name analytics \\\n --query 'TableList[*].Name'\n\n# Get crawler metrics\naws glue get-crawler-metrics --crawler-name-list s3-data-crawler\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Worker types** | Use G.1X for most jobs; G.2X for memory-intensive |\n| **Job bookmarks** | Enable for incremental processing |\n| **Crawlers** | Use specific prefixes, schedule during low-traffic |\n| **Partitions** | Partition by date for time-series data |\n| **Workflows** | Use for complex multi-job pipelines |\n| **Connections** | Use VPC connections for private resources |\n| **Triggers** | Prefer conditional triggers over polling |\n| **Schema Registry** | Use for Kafka/Kinesis data validation |\n| **Monitoring** | Enable CloudWatch logs and metrics |\n| **Cost** | Use Python Shell for simple scripts |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":12324,"content_sha256":"cee9ec01b26f28a855639fd7018d098cf0c7d38b70d2bb33551a62ff05198546"},{"filename":"references/iam-security.md","content":"# IAM & Security\n\n## Contents\n\n- [Users & Credentials](#users--credentials)\n- [Roles](#roles)\n - [Cross-Account Role Setup Checklist](#cross-account-role-setup-checklist)\n - [GitHub Actions OIDC Setup Checklist](#github-actions-oidc-setup-checklist)\n- [Policies](#policies)\n- [STS (Security Token Service)](#sts-security-token-service)\n- [Instance Profiles](#instance-profiles)\n- [Permission Boundaries](#permission-boundaries)\n- [IAM Access Analyzer](#iam-access-analyzer)\n- [Service-Linked Roles](#service-linked-roles)\n- [OIDC Providers](#oidc-providers)\n- [Useful Queries](#useful-queries)\n- [Best Practices](#best-practices)\n\n---\n\n## Users & Credentials\n\n### User Management\n```bash\n# Create user with path-based organization\naws iam create-user --user-name developer --path /engineering/\n\n# Create user with tags\naws iam create-user \\\n --user-name developer \\\n --tags Key=Team,Value=Backend Key=Environment,Value=Production\n\n# List users\naws iam list-users\naws iam list-users --path-prefix /engineering/\n\n# Get user details\naws iam get-user --user-name developer\n\n# Delete user (must remove from groups and delete keys first)\naws iam delete-user --user-name developer\n```\n\n### Access Keys\n```bash\n# Create access key (SAVE OUTPUT - shown only once!)\naws iam create-access-key --user-name developer\n\n# List access keys\naws iam list-access-keys --user-name developer\n\n# Rotate key (create new, then deactivate old)\naws iam update-access-key \\\n --user-name developer \\\n --access-key-id AKIAIOSFODNN7EXAMPLE \\\n --status Inactive\n\n# Delete old key\naws iam delete-access-key \\\n --user-name developer \\\n --access-key-id AKIAIOSFODNN7EXAMPLE\n\n# Get last used info\naws iam get-access-key-last-used --access-key-id AKIAIOSFODNN7EXAMPLE\n```\n\n### MFA Management\n```bash\n# Create virtual MFA device\naws iam create-virtual-mfa-device \\\n --virtual-mfa-device-name developer-mfa \\\n --outfile QRCode.png \\\n --bootstrap-method QRCodePNG\n\n# Enable MFA for user (requires two consecutive codes)\naws iam enable-mfa-device \\\n --user-name developer \\\n --serial-number arn:aws:iam::123456789012:mfa/developer-mfa \\\n --authentication-code1 123456 \\\n --authentication-code2 789012\n\n# List MFA devices\naws iam list-mfa-devices --user-name developer\n\n# Deactivate MFA\naws iam deactivate-mfa-device \\\n --user-name developer \\\n --serial-number arn:aws:iam::123456789012:mfa/developer-mfa\n```\n\n### Groups\n```bash\n# Create group\naws iam create-group --group-name Developers\n\n# Add user to group\naws iam add-user-to-group --user-name developer --group-name Developers\n\n# List groups for user\naws iam list-groups-for-user --user-name developer\n\n# Remove user from group\naws iam remove-user-from-group --user-name developer --group-name Developers\n```\n\n## Roles\n\n### Create Role with Trust Policy\n\n**Basic role creation:**\n```bash\naws iam create-role \\\n --role-name MyRole \\\n --assume-role-policy-document file://trust-policy.json \\\n --description \"Role for service X\" \\\n --max-session-duration 3600\n```\n\n### Trust Policy Examples\n\n**EC2 Service Role:**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"Service\": \"ec2.amazonaws.com\"},\n \"Action\": \"sts:AssumeRole\"\n }]\n}\n```\n\n**Lambda Service Role:**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"Service\": \"lambda.amazonaws.com\"},\n \"Action\": \"sts:AssumeRole\"\n }]\n}\n```\n\n**ECS Tasks Role:**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"Service\": \"ecs-tasks.amazonaws.com\"},\n \"Action\": \"sts:AssumeRole\"\n }]\n}\n```\n\n**Cross-Account Role (with external ID):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": \"arn:aws:iam::OTHER_ACCOUNT_ID:root\"},\n \"Action\": \"sts:AssumeRole\",\n \"Condition\": {\n \"StringEquals\": {\"sts:ExternalId\": \"UniqueSecretId123\"}\n }\n }]\n}\n```\n\n**GitHub OIDC (for GitHub Actions):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Federated\": \"arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com\"\n },\n \"Action\": \"sts:AssumeRoleWithWebIdentity\",\n \"Condition\": {\n \"StringEquals\": {\n \"token.actions.githubusercontent.com:aud\": \"sts.amazonaws.com\"\n },\n \"StringLike\": {\n \"token.actions.githubusercontent.com:sub\": \"repo:myorg/myrepo:*\"\n }\n }\n }]\n}\n```\n\n**EKS OIDC (IRSA - IAM Roles for Service Accounts):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Federated\": \"arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E\"\n },\n \"Action\": \"sts:AssumeRoleWithWebIdentity\",\n \"Condition\": {\n \"StringEquals\": {\n \"oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub\": \"system:serviceaccount:default:my-service-account\",\n \"oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud\": \"sts.amazonaws.com\"\n }\n }\n }]\n}\n```\n\n### Cross-Account Role Setup Checklist\n\n```\n[ ] 1. Create trust-policy.json with target account ARN and optional external ID\n[ ] 2. Create role: aws iam create-role --role-name CrossAccountRole --assume-role-policy-document file://trust-policy.json\n[ ] 3. Attach policy: aws iam attach-role-policy --role-name CrossAccountRole --policy-arn \u003cpolicy-arn>\n[ ] 4. Verify role: aws iam get-role --role-name CrossAccountRole\n[ ] 5. Test from target account: aws sts assume-role --role-arn arn:aws:iam::ACCOUNT:role/CrossAccountRole --role-session-name test\n[ ] 6. Verify assumed identity: aws sts get-caller-identity\n```\n\n### GitHub Actions OIDC Setup Checklist\n\n```\n[ ] 1. Create OIDC provider (one-time per account):\n aws iam create-open-id-connect-provider \\\n --url \"https://token.actions.githubusercontent.com\" \\\n --client-id-list \"sts.amazonaws.com\" \\\n --thumbprint-list \"6938fd4d98bab03faadb97b34396831e3780aea1\"\n[ ] 2. Create trust-policy.json with GitHub OIDC principal (see template above)\n[ ] 3. Create role: aws iam create-role --role-name GitHubActionsRole --assume-role-policy-document file://trust-policy.json\n[ ] 4. Attach permissions: aws iam attach-role-policy --role-name GitHubActionsRole --policy-arn \u003cpolicy-arn>\n[ ] 5. Verify provider: aws iam list-open-id-connect-providers\n[ ] 6. Test in GitHub Actions workflow with aws-actions/configure-aws-credentials\n```\n\n### Role Management\n```bash\n# List roles\naws iam list-roles\naws iam list-roles --path-prefix /service-roles/\n\n# Get role details\naws iam get-role --role-name MyRole\n\n# Update trust policy\naws iam update-assume-role-policy \\\n --role-name MyRole \\\n --policy-document file://new-trust-policy.json\n\n# Delete role (must detach policies first)\naws iam delete-role --role-name MyRole\n```\n\n## Policies\n\n### Create Managed Policy\n```bash\naws iam create-policy \\\n --policy-name S3ReadAccess \\\n --policy-document file://policy.json \\\n --description \"Read-only access to S3\"\n```\n\n**Policy Example (S3 Read-Only):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Action\": [\n \"s3:GetObject\",\n \"s3:ListBucket\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::my-bucket\",\n \"arn:aws:s3:::my-bucket/*\"\n ]\n }]\n}\n```\n\n**Policy Example (DynamoDB CRUD):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Action\": [\n \"dynamodb:GetItem\",\n \"dynamodb:PutItem\",\n \"dynamodb:UpdateItem\",\n \"dynamodb:DeleteItem\",\n \"dynamodb:Query\",\n \"dynamodb:Scan\"\n ],\n \"Resource\": \"arn:aws:s3:us-east-1:123456789012:table/MyTable\"\n }]\n}\n```\n\n### Attach Policies\n```bash\n# Attach managed policy to role\naws iam attach-role-policy \\\n --role-name MyRole \\\n --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\n\n# Attach to user\naws iam attach-user-policy \\\n --user-name developer \\\n --policy-arn arn:aws:iam::123456789012:policy/S3ReadAccess\n\n# Attach to group\naws iam attach-group-policy \\\n --group-name Developers \\\n --policy-arn arn:aws:iam::aws:policy/PowerUserAccess\n\n# List attached policies\naws iam list-attached-role-policies --role-name MyRole\naws iam list-attached-user-policies --user-name developer\n```\n\n### Inline Policies\n```bash\n# Put inline policy on role\naws iam put-role-policy \\\n --role-name MyRole \\\n --policy-name InlineS3Policy \\\n --policy-document file://inline-policy.json\n\n# Get inline policy\naws iam get-role-policy --role-name MyRole --policy-name InlineS3Policy\n\n# Delete inline policy\naws iam delete-role-policy --role-name MyRole --policy-name InlineS3Policy\n```\n\n### Policy Versioning\n```bash\n# Create new version\naws iam create-policy-version \\\n --policy-arn arn:aws:iam::123456789012:policy/S3ReadAccess \\\n --policy-document file://updated-policy.json \\\n --set-as-default\n\n# List versions\naws iam list-policy-versions \\\n --policy-arn arn:aws:iam::123456789012:policy/S3ReadAccess\n\n# Set default version\naws iam set-default-policy-version \\\n --policy-arn arn:aws:iam::123456789012:policy/S3ReadAccess \\\n --version-id v2\n```\n\n## STS (Security Token Service)\n\n### Assume Role\n```bash\n# Basic assume role\naws sts assume-role \\\n --role-arn arn:aws:iam::123456789012:role/CrossAccountRole \\\n --role-session-name MySession \\\n --duration-seconds 3600\n\n# With MFA\naws sts assume-role \\\n --role-arn arn:aws:iam::123456789012:role/MFAProtectedRole \\\n --role-session-name MFASession \\\n --serial-number arn:aws:iam::123456789012:mfa/myuser \\\n --token-code 123456\n\n# With external ID (for cross-account)\naws sts assume-role \\\n --role-arn arn:aws:iam::123456789012:role/VendorRole \\\n --role-session-name VendorSession \\\n --external-id UniqueSecretId123\n```\n\n### Use Assumed Credentials\n```bash\n# Capture and export credentials\nCREDS=$(aws sts assume-role \\\n --role-arn arn:aws:iam::123456789012:role/AdminRole \\\n --role-session-name AdminSession \\\n --output json)\n\nexport AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r '.Credentials.AccessKeyId')\nexport AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r '.Credentials.SecretAccessKey')\nexport AWS_SESSION_TOKEN=$(echo $CREDS | jq -r '.Credentials.SessionToken')\n\n# Verify new identity\naws sts get-caller-identity\n```\n\n### Identity Verification\n```bash\n# Get current identity\naws sts get-caller-identity\n# Returns: Account, UserId, Arn\n\n# Get session token (for MFA)\naws sts get-session-token \\\n --serial-number arn:aws:iam::123456789012:mfa/myuser \\\n --token-code 123456 \\\n --duration-seconds 3600\n```\n\n### Web Identity (OIDC)\n```bash\n# Assume role with web identity token\naws sts assume-role-with-web-identity \\\n --role-arn arn:aws:iam::123456789012:role/FederatedRole \\\n --role-session-name WebSession \\\n --web-identity-token file://token.jwt\n```\n\n## Instance Profiles\n\n```bash\n# Create instance profile\naws iam create-instance-profile --instance-profile-name WebServerProfile\n\n# Add role to instance profile\naws iam add-role-to-instance-profile \\\n --instance-profile-name WebServerProfile \\\n --role-name EC2-S3-Access\n\n# Associate with EC2 instance\naws ec2 associate-iam-instance-profile \\\n --instance-id i-1234567890abcdef0 \\\n --iam-instance-profile Name=WebServerProfile\n\n# Replace instance profile\naws ec2 replace-iam-instance-profile-association \\\n --association-id iip-assoc-0e7736511a163c209 \\\n --iam-instance-profile Name=NewProfile\n\n# List instance profiles\naws iam list-instance-profiles\n\n# Get instance profile for role\naws iam list-instance-profiles-for-role --role-name EC2-S3-Access\n```\n\n## Permission Boundaries\n\n```bash\n# Set permission boundary on role\naws iam put-role-permissions-boundary \\\n --role-name DeveloperRole \\\n --permissions-boundary arn:aws:iam::123456789012:policy/DeveloperBoundary\n\n# Set on user\naws iam put-user-permissions-boundary \\\n --user-name developer \\\n --permissions-boundary arn:aws:iam::123456789012:policy/DeveloperBoundary\n\n# Remove boundary\naws iam delete-role-permissions-boundary --role-name DeveloperRole\n```\n\n**Permission Boundary Policy Example:**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"s3:*\",\n \"dynamodb:*\",\n \"lambda:*\",\n \"logs:*\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Deny\",\n \"Action\": [\n \"iam:*\",\n \"organizations:*\",\n \"account:*\"\n ],\n \"Resource\": \"*\"\n }\n ]\n}\n```\n\n## IAM Access Analyzer\n\n```bash\n# Create analyzer (account-level)\naws accessanalyzer create-analyzer \\\n --analyzer-name ExternalAccessAnalyzer \\\n --type ACCOUNT\n\n# Create analyzer (organization-level)\naws accessanalyzer create-analyzer \\\n --analyzer-name OrgAnalyzer \\\n --type ORGANIZATION\n\n# List analyzers\naws accessanalyzer list-analyzers\n\n# List findings\naws accessanalyzer list-findings \\\n --analyzer-arn arn:aws:access-analyzer:us-east-1:123456789012:analyzer/ExternalAccessAnalyzer\n\n# Filter findings\naws accessanalyzer list-findings \\\n --analyzer-arn arn:aws:access-analyzer:us-east-1:123456789012:analyzer/ExternalAccessAnalyzer \\\n --filter '{\"resourceType\": {\"eq\": [\"AWS::S3::Bucket\"]}, \"status\": {\"eq\": [\"ACTIVE\"]}}'\n\n# Get finding details\naws accessanalyzer get-finding \\\n --analyzer-arn arn:aws:access-analyzer:us-east-1:123456789012:analyzer/ExternalAccessAnalyzer \\\n --id finding-id\n\n# Archive finding (mark as reviewed)\naws accessanalyzer update-findings \\\n --analyzer-arn arn:aws:access-analyzer:us-east-1:123456789012:analyzer/ExternalAccessAnalyzer \\\n --ids finding-id \\\n --status ARCHIVED\n```\n\n## Service-Linked Roles\n\n```bash\n# Create service-linked role\naws iam create-service-linked-role \\\n --aws-service-name elasticloadbalancing.amazonaws.com\n\n# List service-linked roles\naws iam list-roles --path-prefix /aws-service-role/\n\n# Get service-linked role\naws iam get-role --role-name AWSServiceRoleForElasticLoadBalancing\n\n# Delete service-linked role (if not in use)\naws iam delete-service-linked-role \\\n --role-name AWSServiceRoleForElasticLoadBalancing\n```\n\n## OIDC Providers\n\n```bash\n# Create OIDC provider (GitHub Actions)\naws iam create-open-id-connect-provider \\\n --url \"https://token.actions.githubusercontent.com\" \\\n --client-id-list \"sts.amazonaws.com\" \\\n --thumbprint-list \"6938fd4d98bab03faadb97b34396831e3780aea1\"\n\n# Get OIDC provider details\naws iam get-open-id-connect-provider \\\n --open-id-connect-provider-arn arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com\n\n# List OIDC providers\naws iam list-open-id-connect-providers\n\n# Delete OIDC provider\naws iam delete-open-id-connect-provider \\\n --open-id-connect-provider-arn arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com\n```\n\n## Useful Queries\n\n```bash\n# Find unused credentials (access keys not used in 90 days)\naws iam generate-credential-report\naws iam get-credential-report --output text --query 'Content' | base64 -d\n\n# List all policies attached to a role\naws iam list-attached-role-policies --role-name MyRole\naws iam list-role-policies --role-name MyRole # inline policies\n\n# Get effective permissions for identity\naws iam simulate-principal-policy \\\n --policy-source-arn arn:aws:iam::123456789012:user/developer \\\n --action-names s3:GetObject s3:PutObject \\\n --resource-arns arn:aws:s3:::my-bucket/*\n\n# Find roles that can be assumed by a service\naws iam list-roles \\\n --query 'Roles[?contains(AssumeRolePolicyDocument.Statement[0].Principal.Service, `lambda.amazonaws.com`)].RoleName'\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Least privilege** | Start with minimal permissions, add as needed |\n| **Use roles** | Prefer roles over long-lived access keys for compute |\n| **MFA** | Require MFA for sensitive operations |\n| **Permission boundaries** | Limit maximum permissions for delegated admins |\n| **External IDs** | Use external IDs for cross-account roles with third parties |\n| **Rotate keys** | Rotate access keys every 90 days maximum |\n| **Access Analyzer** | Enable to detect unintended external access |\n| **Service-linked roles** | Use AWS-managed roles for services when available |\n| **Audit** | Use CloudTrail to monitor IAM actions |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":16887,"content_sha256":"3c4188d1d1b90c0dc8b0da569a2c33b22dfb0d6b69aad6a3907e953661a99402"},{"filename":"references/kinesis.md","content":"# Amazon Kinesis\n\n## Data Streams\n\n### Create Stream\n```bash\n# Provisioned mode (fixed shards)\naws kinesis create-stream \\\n --stream-name app-clickstream \\\n --shard-count 4\n\n# On-demand mode (auto-scaling)\naws kinesis create-stream \\\n --stream-name app-events \\\n --stream-mode-details StreamMode=ON_DEMAND\n\n# Wait for stream to be active\naws kinesis wait stream-exists --stream-name app-clickstream\n```\n\n### Manage Streams\n```bash\n# List streams\naws kinesis list-streams\n\n# Describe stream\naws kinesis describe-stream --stream-name app-clickstream\n\n# Describe stream summary (faster)\naws kinesis describe-stream-summary --stream-name app-clickstream\n\n# Update shard count (provisioned mode)\naws kinesis update-shard-count \\\n --stream-name app-clickstream \\\n --target-shard-count 8 \\\n --scaling-type UNIFORM_SCALING\n\n# Update stream mode\naws kinesis update-stream-mode \\\n --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream \\\n --stream-mode-details StreamMode=ON_DEMAND\n\n# Delete stream\naws kinesis delete-stream --stream-name app-clickstream\n\n# Enable encryption\naws kinesis start-stream-encryption \\\n --stream-name app-clickstream \\\n --encryption-type KMS \\\n --key-id alias/aws/kinesis\n```\n\n## Producing Data\n\n### Put Single Record\n```bash\n# Put record with data\naws kinesis put-record \\\n --stream-name app-clickstream \\\n --partition-key \"user-123\" \\\n --data \"$(echo -n '{\"event\":\"click\",\"page\":\"/home\"}' | base64)\"\n\n# Put record from file\naws kinesis put-record \\\n --stream-name app-clickstream \\\n --partition-key \"user-456\" \\\n --data fileb://event.json\n\n# With explicit hash key (direct shard targeting)\naws kinesis put-record \\\n --stream-name app-clickstream \\\n --partition-key \"user-123\" \\\n --explicit-hash-key \"170141183460469231731687303715884105727\" \\\n --data \"...\"\n```\n\n### Put Multiple Records (Batch)\n```bash\n# Put records batch\naws kinesis put-records \\\n --stream-name app-clickstream \\\n --records '[\n {\"Data\": \"eyJldmVudCI6ImNsaWNrIn0=\", \"PartitionKey\": \"user-1\"},\n {\"Data\": \"eyJldmVudCI6InZpZXcifQ==\", \"PartitionKey\": \"user-2\"},\n {\"Data\": \"eyJldmVudCI6InB1cmNoYXNlIn0=\", \"PartitionKey\": \"user-3\"}\n ]'\n\n# From file\naws kinesis put-records \\\n --stream-name app-clickstream \\\n --records file://records.json\n```\n\n**records.json:**\n```json\n[\n {\"Data\": \"base64data1\", \"PartitionKey\": \"key1\"},\n {\"Data\": \"base64data2\", \"PartitionKey\": \"key2\"}\n]\n```\n\n## Consuming Data\n\n### Get Shard Iterator\n```bash\n# Latest records (new data only)\nITERATOR=$(aws kinesis get-shard-iterator \\\n --stream-name app-clickstream \\\n --shard-id shardId-000000000000 \\\n --shard-iterator-type LATEST \\\n --query 'ShardIterator' \\\n --output text)\n\n# From beginning (all data)\nITERATOR=$(aws kinesis get-shard-iterator \\\n --stream-name app-clickstream \\\n --shard-id shardId-000000000000 \\\n --shard-iterator-type TRIM_HORIZON \\\n --query 'ShardIterator' \\\n --output text)\n\n# From specific sequence number\nITERATOR=$(aws kinesis get-shard-iterator \\\n --stream-name app-clickstream \\\n --shard-id shardId-000000000000 \\\n --shard-iterator-type AT_SEQUENCE_NUMBER \\\n --starting-sequence-number \"12345678901234567890\" \\\n --query 'ShardIterator' \\\n --output text)\n\n# From timestamp\nITERATOR=$(aws kinesis get-shard-iterator \\\n --stream-name app-clickstream \\\n --shard-id shardId-000000000000 \\\n --shard-iterator-type AT_TIMESTAMP \\\n --timestamp \"2024-01-15T12:00:00Z\" \\\n --query 'ShardIterator' \\\n --output text)\n```\n\n### Get Records\n```bash\n# Get records\naws kinesis get-records --shard-iterator $ITERATOR\n\n# Get records with limit\naws kinesis get-records \\\n --shard-iterator $ITERATOR \\\n --limit 100\n\n# Decode records (using jq)\naws kinesis get-records --shard-iterator $ITERATOR | \\\n jq -r '.Records[].Data' | \\\n while read data; do echo $data | base64 -d; echo; done\n```\n\n### List Shards\n```bash\n# List all shards\naws kinesis list-shards --stream-name app-clickstream\n\n# List with filter\naws kinesis list-shards \\\n --stream-name app-clickstream \\\n --shard-filter Type=AT_LATEST\n```\n\n## Enhanced Fan-Out\n\n### Register Consumer\n```bash\n# Register enhanced fan-out consumer\naws kinesis register-stream-consumer \\\n --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream \\\n --consumer-name my-app-consumer\n\n# Wait for consumer to be active\naws kinesis describe-stream-consumer \\\n --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream \\\n --consumer-name my-app-consumer\n```\n\n### Manage Consumers\n```bash\n# List consumers\naws kinesis list-stream-consumers \\\n --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream\n\n# Describe consumer\naws kinesis describe-stream-consumer \\\n --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream \\\n --consumer-name my-app-consumer\n\n# Deregister consumer\naws kinesis deregister-stream-consumer \\\n --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream \\\n --consumer-name my-app-consumer\n```\n\n### Subscribe to Shard (Enhanced Fan-Out)\n```bash\n# Subscribe (returns subscription ARN for SubscribeToShard API)\naws kinesis subscribe-to-shard \\\n --consumer-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream/consumer/my-app-consumer \\\n --shard-id shardId-000000000000 \\\n --starting-position Type=LATEST\n```\n\n## Kinesis Data Firehose\n\n### Create Delivery Stream\n```bash\n# Direct to S3\naws firehose create-delivery-stream \\\n --delivery-stream-name raw-events-to-s3 \\\n --delivery-stream-type DirectPut \\\n --s3-destination-configuration '{\n \"RoleARN\": \"arn:aws:iam::123456789012:role/FirehoseRole\",\n \"BucketARN\": \"arn:aws:s3:::my-data-bucket\",\n \"Prefix\": \"raw/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/\",\n \"ErrorOutputPrefix\": \"errors/\",\n \"BufferingHints\": {\n \"SizeInMBs\": 128,\n \"IntervalInSeconds\": 300\n },\n \"CompressionFormat\": \"GZIP\"\n }'\n\n# From Kinesis Data Stream to S3\naws firehose create-delivery-stream \\\n --delivery-stream-name stream-to-s3 \\\n --delivery-stream-type KinesisStreamAsSource \\\n --kinesis-stream-source-configuration '{\n \"KinesisStreamARN\": \"arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream\",\n \"RoleARN\": \"arn:aws:iam::123456789012:role/FirehoseRole\"\n }' \\\n --s3-destination-configuration '{\n \"RoleARN\": \"arn:aws:iam::123456789012:role/FirehoseRole\",\n \"BucketARN\": \"arn:aws:s3:::my-data-bucket\",\n \"Prefix\": \"processed/\"\n }'\n\n# To OpenSearch\naws firehose create-delivery-stream \\\n --delivery-stream-name logs-to-opensearch \\\n --delivery-stream-type DirectPut \\\n --elasticsearch-destination-configuration '{\n \"RoleARN\": \"arn:aws:iam::123456789012:role/FirehoseRole\",\n \"DomainARN\": \"arn:aws:es:us-east-1:123456789012:domain/my-domain\",\n \"IndexName\": \"logs\",\n \"TypeName\": \"_doc\",\n \"IndexRotationPeriod\": \"OneDay\",\n \"BufferingHints\": {\n \"IntervalInSeconds\": 60,\n \"SizeInMBs\": 5\n },\n \"S3BackupMode\": \"FailedDocumentsOnly\",\n \"S3Configuration\": {\n \"RoleARN\": \"arn:aws:iam::123456789012:role/FirehoseRole\",\n \"BucketARN\": \"arn:aws:s3:::my-backup-bucket\"\n }\n }'\n```\n\n### Manage Firehose\n```bash\n# List delivery streams\naws firehose list-delivery-streams\n\n# Describe delivery stream\naws firehose describe-delivery-stream --delivery-stream-name raw-events-to-s3\n\n# Put record to Firehose\naws firehose put-record \\\n --delivery-stream-name raw-events-to-s3 \\\n --record '{\"Data\": \"eyJldmVudCI6InRlc3QifQ==\"}'\n\n# Put batch\naws firehose put-record-batch \\\n --delivery-stream-name raw-events-to-s3 \\\n --records '[{\"Data\": \"...\"}, {\"Data\": \"...\"}]'\n\n# Update destination\naws firehose update-destination \\\n --delivery-stream-name raw-events-to-s3 \\\n --current-delivery-stream-version-id 1 \\\n --destination-id destinationId-000000000001 \\\n --s3-destination-update '{\n \"BufferingHints\": {\"SizeInMBs\": 256}\n }'\n\n# Delete delivery stream\naws firehose delete-delivery-stream --delivery-stream-name raw-events-to-s3\n```\n\n## Lambda Integration\n\n### Create Event Source Mapping\n```bash\n# Kinesis to Lambda\naws lambda create-event-source-mapping \\\n --function-name process-stream \\\n --event-source-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream \\\n --starting-position LATEST \\\n --batch-size 100 \\\n --maximum-batching-window-in-seconds 5 \\\n --parallelization-factor 2 \\\n --maximum-retry-attempts 3 \\\n --bisect-batch-on-function-error \\\n --destination-config '{\n \"OnFailure\": {\n \"Destination\": \"arn:aws:sqs:us-east-1:123456789012:dlq\"\n }\n }'\n\n# With tumbling window\naws lambda create-event-source-mapping \\\n --function-name aggregate-stream \\\n --event-source-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream \\\n --starting-position LATEST \\\n --batch-size 1000 \\\n --tumbling-window-in-seconds 60\n\n# With event filtering\naws lambda create-event-source-mapping \\\n --function-name process-orders \\\n --event-source-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-events \\\n --starting-position LATEST \\\n --filter-criteria '{\n \"Filters\": [{\n \"Pattern\": \"{\\\"data\\\": {\\\"eventType\\\": [\\\"ORDER_PLACED\\\"]}}\"\n }]\n }'\n```\n\n### Manage Event Source Mapping\n```bash\n# List mappings\naws lambda list-event-source-mappings \\\n --function-name process-stream\n\n# Update mapping\naws lambda update-event-source-mapping \\\n --uuid abc-123-uuid \\\n --batch-size 200 \\\n --parallelization-factor 4\n\n# Disable mapping\naws lambda update-event-source-mapping \\\n --uuid abc-123-uuid \\\n --no-enabled\n```\n\n## Kinesis Data Analytics\n\n### Create Application\n```bash\n# Create SQL application\naws kinesisanalytics create-application \\\n --application-name real-time-analytics \\\n --application-description \"Real-time stream processing\" \\\n --runtime-environment SQL-1_0 \\\n --service-execution-role arn:aws:iam::123456789012:role/KinesisAnalyticsRole\n\n# Create Flink application\naws kinesisanalyticsv2 create-application \\\n --application-name flink-processor \\\n --runtime-environment FLINK-1_15 \\\n --service-execution-role arn:aws:iam::123456789012:role/KinesisAnalyticsRole \\\n --application-configuration '{\n \"FlinkApplicationConfiguration\": {\n \"ParallelismConfiguration\": {\n \"ConfigurationType\": \"CUSTOM\",\n \"Parallelism\": 4,\n \"ParallelismPerKPU\": 1,\n \"AutoScalingEnabled\": true\n }\n },\n \"ApplicationCodeConfiguration\": {\n \"CodeContent\": {\n \"S3ContentLocation\": {\n \"BucketARN\": \"arn:aws:s3:::my-apps-bucket\",\n \"FileKey\": \"flink-app.jar\"\n }\n },\n \"CodeContentType\": \"ZIPFILE\"\n }\n }'\n```\n\n## Useful Queries\n\n```bash\n# Get stream status\naws kinesis describe-stream-summary \\\n --stream-name app-clickstream \\\n --query 'StreamDescriptionSummary.StreamStatus'\n\n# Get shard count\naws kinesis describe-stream-summary \\\n --stream-name app-clickstream \\\n --query 'StreamDescriptionSummary.OpenShardCount'\n\n# Get consumer count\naws kinesis list-stream-consumers \\\n --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/app-clickstream \\\n --query 'Consumers | length(@)'\n\n# Get Firehose status\naws firehose describe-delivery-stream \\\n --delivery-stream-name raw-events-to-s3 \\\n --query 'DeliveryStreamDescription.DeliveryStreamStatus'\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **On-demand mode** | Use for variable or unpredictable traffic |\n| **Partition keys** | Randomize to avoid hot shards |\n| **Enhanced fan-out** | Use for multiple low-latency consumers |\n| **Batching** | Use PutRecords for higher throughput |\n| **Firehose** | Use for simple S3/OpenSearch delivery |\n| **Lambda parallelization** | Increase for high-volume streams |\n| **Filtering** | Use event filters to reduce Lambda invocations |\n| **DLQ** | Configure for failed record handling |\n| **Encryption** | Enable KMS encryption for sensitive data |\n| **Retention** | Extend beyond 24h for replay capability |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":12477,"content_sha256":"5cfba48a1a7bc65582eb92e447bda5684e6855c568d4f99a16fc94bbdc32ea74"},{"filename":"references/lambda.md","content":"# Lambda Functions\n\n## Contents\n\n- [Function Creation](#function-creation)\n- [Code & Configuration Updates](#code--configuration-updates)\n- [Layers](#layers)\n- [Versions & Aliases](#versions--aliases)\n - [Safe Deployment Checklist](#safe-deployment-checklist-blue-green-with-aliases)\n- [Invocation](#invocation)\n- [Event Source Mappings](#event-source-mappings)\n- [Function URLs](#function-urls)\n- [Concurrency](#concurrency)\n- [Permissions (Resource Policies)](#permissions-resource-policies)\n- [Useful Queries](#useful-queries)\n- [Best Practices](#best-practices)\n\n---\n\n## Function Creation\n\n### From ZIP (Python/Node.js)\n```bash\n# Python with ARM64 architecture (Graviton2 - cheaper & often faster)\naws lambda create-function \\\n --function-name my-processor \\\n --runtime python3.12 \\\n --architectures arm64 \\\n --handler lambda_function.handler \\\n --role arn:aws:iam::123456789012:role/LambdaExecutionRole \\\n --zip-file fileb://function.zip \\\n --memory-size 512 \\\n --timeout 30 \\\n --environment Variables='{LOG_LEVEL=INFO,DB_HOST=prod-db.example.com}'\n\n# Node.js with x86_64\naws lambda create-function \\\n --function-name my-api \\\n --runtime nodejs22.x \\\n --architectures x86_64 \\\n --handler index.handler \\\n --role arn:aws:iam::123456789012:role/LambdaExecutionRole \\\n --zip-file fileb://function.zip\n```\n\n### From S3\n```bash\naws lambda create-function \\\n --function-name my-function \\\n --runtime python3.12 \\\n --code S3Bucket=my-deployment-bucket,S3Key=deployments/function.zip,S3ObjectVersion=abc123 \\\n --handler app.lambda_handler \\\n --role arn:aws:iam::123456789012:role/LambdaExecutionRole\n```\n\n### From Container Image (ECR)\n```bash\naws lambda create-function \\\n --function-name my-container-func \\\n --package-type Image \\\n --code ImageUri=123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest \\\n --role arn:aws:iam::123456789012:role/LambdaExecutionRole \\\n --architectures arm64 \\\n --memory-size 1024 \\\n --timeout 60\n```\n\n## Code & Configuration Updates\n\n### Update Code\n```bash\n# From local ZIP\naws lambda update-function-code \\\n --function-name my-function \\\n --zip-file fileb://new-code.zip\n\n# From S3\naws lambda update-function-code \\\n --function-name my-function \\\n --s3-bucket my-bucket \\\n --s3-key deployments/new-code.zip\n\n# From ECR image\naws lambda update-function-code \\\n --function-name my-container-func \\\n --image-uri 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:v2\n\n# Publish version immediately\naws lambda update-function-code \\\n --function-name my-function \\\n --zip-file fileb://new-code.zip \\\n --publish\n```\n\n### Update Configuration\n```bash\n# Memory and timeout\naws lambda update-function-configuration \\\n --function-name my-function \\\n --memory-size 1024 \\\n --timeout 60\n\n# Environment variables\naws lambda update-function-configuration \\\n --function-name my-function \\\n --environment Variables='{KEY1=value1,KEY2=value2}'\n\n# Add layers\naws lambda update-function-configuration \\\n --function-name my-function \\\n --layers arn:aws:lambda:us-east-1:123456789012:layer:my-layer:3 \\\n arn:aws:lambda:us-east-1:123456789012:layer:other-layer:1\n\n# VPC configuration\naws lambda update-function-configuration \\\n --function-name my-function \\\n --vpc-config SubnetIds=subnet-1,subnet-2,SecurityGroupIds=sg-1\n\n# Remove VPC\naws lambda update-function-configuration \\\n --function-name my-function \\\n --vpc-config SubnetIds=[],SecurityGroupIds=[]\n```\n\n### Wait for Update Completion\n```bash\n# Wait for function update to complete\naws lambda wait function-updated --function-name my-function\n\n# Wait for function to be active\naws lambda wait function-active --function-name my-function\n```\n\n## Layers\n\n### Create and Publish Layer\n```bash\n# Publish layer version\naws lambda publish-layer-version \\\n --layer-name common-libs \\\n --description \"Shared Python libraries\" \\\n --zip-file fileb://layer.zip \\\n --compatible-runtimes python3.11 python3.12 \\\n --compatible-architectures arm64 x86_64\n\n# Grant layer usage to another account\naws lambda add-layer-version-permission \\\n --layer-name common-libs \\\n --version-number 1 \\\n --statement-id xaccount \\\n --action lambda:GetLayerVersion \\\n --principal 987654321098\n\n# Grant to all accounts (public layer)\naws lambda add-layer-version-permission \\\n --layer-name common-libs \\\n --version-number 1 \\\n --statement-id public \\\n --action lambda:GetLayerVersion \\\n --principal \"*\"\n```\n\n### Manage Layers\n```bash\n# List layers\naws lambda list-layers\n\n# List layer versions\naws lambda list-layer-versions --layer-name common-libs\n\n# Get layer version details\naws lambda get-layer-version \\\n --layer-name common-libs \\\n --version-number 1\n\n# Delete layer version\naws lambda delete-layer-version \\\n --layer-name common-libs \\\n --version-number 1\n```\n\n## Versions & Aliases\n\n### Publish Version\n```bash\n# Publish immutable version from $LATEST\naws lambda publish-version \\\n --function-name my-function \\\n --description \"Release v1.2.0\"\n\n# List versions\naws lambda list-versions-by-function --function-name my-function\n```\n\n### Create and Manage Aliases\n```bash\n# Create alias pointing to version\naws lambda create-alias \\\n --function-name my-function \\\n --name PROD \\\n --function-version 5 \\\n --description \"Production traffic\"\n\n# Update alias to new version\naws lambda update-alias \\\n --function-name my-function \\\n --name PROD \\\n --function-version 6\n\n# Weighted routing (canary deployment - 10% to new version)\naws lambda update-alias \\\n --function-name my-function \\\n --name PROD \\\n --function-version 5 \\\n --routing-config AdditionalVersionWeights='{\"6\": 0.1}'\n\n# Complete traffic shift (100% to new version)\naws lambda update-alias \\\n --function-name my-function \\\n --name PROD \\\n --function-version 6 \\\n --routing-config AdditionalVersionWeights='{}'\n\n# List aliases\naws lambda list-aliases --function-name my-function\n\n# Delete alias\naws lambda delete-alias --function-name my-function --name STAGING\n```\n\n### Safe Deployment Checklist (Blue-Green with Aliases)\n\n```\n[ ] 1. Update code: aws lambda update-function-code --function-name my-function --zip-file fileb://code.zip\n[ ] 2. Wait for update: aws lambda wait function-updated --function-name my-function\n[ ] 3. Test $LATEST: aws lambda invoke --function-name my-function --qualifier '$LATEST' test-response.json\n[ ] 4. Publish version: aws lambda publish-version --function-name my-function --description \"v1.2.0\"\n[ ] 5. Canary (10%): aws lambda update-alias --function-name my-function --name PROD --routing-config AdditionalVersionWeights='{\"NEW_VERSION\": 0.1}'\n[ ] 6. Monitor CloudWatch for errors (wait 5-10 min)\n[ ] 7. Full shift: aws lambda update-alias --function-name my-function --name PROD --function-version NEW_VERSION --routing-config AdditionalVersionWeights='{}'\n[ ] 8. Verify: aws lambda get-alias --function-name my-function --name PROD\n```\n\n## Invocation\n\n### Synchronous Invocation\n```bash\n# Basic invocation\naws lambda invoke \\\n --function-name my-function \\\n response.json\n\n# With payload\naws lambda invoke \\\n --function-name my-function \\\n --cli-binary-format raw-in-base64-out \\\n --payload '{\"key\": \"value\", \"items\": [1, 2, 3]}' \\\n response.json\n\n# Invoke specific version or alias\naws lambda invoke \\\n --function-name my-function \\\n --qualifier PROD \\\n response.json\n\n# With logs in response\naws lambda invoke \\\n --function-name my-function \\\n --log-type Tail \\\n --query 'LogResult' \\\n --output text \\\n response.json | base64 -d\n```\n\n### Asynchronous Invocation\n```bash\n# Fire-and-forget\naws lambda invoke \\\n --function-name my-function \\\n --invocation-type Event \\\n --payload '{\"event\": \"data\"}' \\\n response.json\n\n# Dry run (validation only)\naws lambda invoke \\\n --function-name my-function \\\n --invocation-type DryRun \\\n response.json\n```\n\n## Event Source Mappings\n\n### SQS Trigger\n```bash\naws lambda create-event-source-mapping \\\n --function-name my-function \\\n --event-source-arn arn:aws:sqs:us-east-1:123456789012:my-queue \\\n --batch-size 10 \\\n --maximum-batching-window-in-seconds 5\n```\n\n### DynamoDB Streams\n```bash\naws lambda create-event-source-mapping \\\n --function-name my-function \\\n --event-source-arn arn:aws:dynamodb:us-east-1:123456789012:table/MyTable/stream/2024-01-01T00:00:00.000 \\\n --batch-size 100 \\\n --starting-position LATEST \\\n --maximum-retry-attempts 3 \\\n --bisect-batch-on-function-error\n```\n\n### Kinesis Stream\n```bash\naws lambda create-event-source-mapping \\\n --function-name my-function \\\n --event-source-arn arn:aws:kinesis:us-east-1:123456789012:stream/my-stream \\\n --batch-size 100 \\\n --starting-position LATEST \\\n --parallelization-factor 2 \\\n --destination-config '{\"OnFailure\":{\"Destination\":\"arn:aws:sqs:us-east-1:123456789012:dlq\"}}'\n```\n\n### MSK (Kafka)\n```bash\naws lambda create-event-source-mapping \\\n --function-name my-function \\\n --event-source-arn arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abc123 \\\n --batch-size 100 \\\n --starting-position LATEST \\\n --topics my-topic\n```\n\n### Event Filtering\n```bash\n# Filter events before invoking Lambda\naws lambda create-event-source-mapping \\\n --function-name my-function \\\n --event-source-arn arn:aws:sqs:us-east-1:123456789012:my-queue \\\n --filter-criteria '{\"Filters\": [{\"Pattern\": \"{\\\"body\\\": {\\\"type\\\": [\\\"order\\\"]}}\"}]}'\n\n# Update existing mapping with filter\naws lambda update-event-source-mapping \\\n --uuid abc123-uuid \\\n --filter-criteria '{\"Filters\": [{\"Pattern\": \"{\\\"eventName\\\": [\\\"INSERT\\\", \\\"MODIFY\\\"]}\"}]}'\n```\n\n### Manage Event Source Mappings\n```bash\n# List all mappings\naws lambda list-event-source-mappings --function-name my-function\n\n# Get mapping details\naws lambda get-event-source-mapping --uuid abc123-uuid\n\n# Enable/disable mapping\naws lambda update-event-source-mapping \\\n --uuid abc123-uuid \\\n --enabled\n\naws lambda update-event-source-mapping \\\n --uuid abc123-uuid \\\n --no-enabled\n\n# Delete mapping\naws lambda delete-event-source-mapping --uuid abc123-uuid\n```\n\n## Function URLs\n\n### Create Function URL\n```bash\n# Public endpoint (no auth)\naws lambda create-function-url-config \\\n --function-name my-function \\\n --auth-type NONE\n\n# IAM auth required\naws lambda create-function-url-config \\\n --function-name my-function \\\n --auth-type AWS_IAM\n\n# With CORS\naws lambda create-function-url-config \\\n --function-name my-function \\\n --auth-type NONE \\\n --cors '{\n \"AllowOrigins\": [\"https://example.com\"],\n \"AllowMethods\": [\"GET\", \"POST\"],\n \"AllowHeaders\": [\"Content-Type\"],\n \"MaxAge\": 86400\n }'\n```\n\n### Grant Public Access (for NONE auth type)\n```bash\naws lambda add-permission \\\n --function-name my-function \\\n --statement-id FunctionURLAllowPublicAccess \\\n --action lambda:InvokeFunctionUrl \\\n --principal \"*\" \\\n --function-url-auth-type NONE\n```\n\n### Manage Function URLs\n```bash\n# Get URL configuration\naws lambda get-function-url-config --function-name my-function\n\n# Update CORS\naws lambda update-function-url-config \\\n --function-name my-function \\\n --cors '{\"AllowOrigins\": [\"*\"]}'\n\n# Delete function URL\naws lambda delete-function-url-config --function-name my-function\n```\n\n## Concurrency\n\n### Reserved Concurrency\n```bash\n# Set reserved concurrency (guarantees capacity, also limits max)\naws lambda put-function-concurrency \\\n --function-name my-function \\\n --reserved-concurrent-executions 100\n\n# Get concurrency configuration\naws lambda get-function-concurrency --function-name my-function\n\n# Remove reserved concurrency (use unreserved pool)\naws lambda delete-function-concurrency --function-name my-function\n```\n\n### Provisioned Concurrency\n```bash\n# Provision warm instances for alias/version (eliminates cold starts)\naws lambda put-provisioned-concurrency-config \\\n --function-name my-function \\\n --qualifier PROD \\\n --provisioned-concurrent-executions 50\n\n# Get provisioned concurrency status\naws lambda get-provisioned-concurrency-config \\\n --function-name my-function \\\n --qualifier PROD\n\n# List all provisioned concurrency configs\naws lambda list-provisioned-concurrency-configs --function-name my-function\n\n# Delete provisioned concurrency\naws lambda delete-provisioned-concurrency-config \\\n --function-name my-function \\\n --qualifier PROD\n```\n\n## Permissions (Resource Policies)\n\n```bash\n# Add permission for service to invoke\naws lambda add-permission \\\n --function-name my-function \\\n --statement-id AllowS3Invoke \\\n --action lambda:InvokeFunction \\\n --principal s3.amazonaws.com \\\n --source-arn arn:aws:s3:::my-bucket \\\n --source-account 123456789012\n\n# Add permission for API Gateway\naws lambda add-permission \\\n --function-name my-function \\\n --statement-id AllowAPIGatewayInvoke \\\n --action lambda:InvokeFunction \\\n --principal apigateway.amazonaws.com \\\n --source-arn \"arn:aws:execute-api:us-east-1:123456789012:api-id/*\"\n\n# Add permission for CloudWatch Events\naws lambda add-permission \\\n --function-name my-function \\\n --statement-id AllowCloudWatchEventsInvoke \\\n --action lambda:InvokeFunction \\\n --principal events.amazonaws.com \\\n --source-arn arn:aws:events:us-east-1:123456789012:rule/my-rule\n\n# Get resource policy\naws lambda get-policy --function-name my-function\n\n# Remove permission\naws lambda remove-permission \\\n --function-name my-function \\\n --statement-id AllowS3Invoke\n```\n\n## Useful Queries\n\n```bash\n# List all functions with memory > 512MB\naws lambda list-functions \\\n --query 'Functions[?MemorySize > `512`].[FunctionName, MemorySize, Runtime]' \\\n --output table\n\n# Find functions by runtime\naws lambda list-functions \\\n --query 'Functions[?Runtime==`python3.12`].FunctionName'\n\n# Get function ARN\naws lambda get-function \\\n --function-name my-function \\\n --query 'Configuration.FunctionArn' \\\n --output text\n\n# Find functions with specific tag\naws lambda list-functions \\\n --query 'Functions[?Tags.Environment==`production`].FunctionName'\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **ARM64** | Use Graviton2 (`arm64`) for better price/performance |\n| **Layers** | Share common code via layers to reduce package size |\n| **Aliases** | Use aliases (PROD, STAGING) for safe deployments |\n| **Reserved concurrency** | Set limits to prevent runaway costs |\n| **Provisioned concurrency** | Use for latency-sensitive workloads |\n| **Environment variables** | Externalize configuration, never hardcode |\n| **IAM** | Use minimal permissions, function-specific roles |\n| **VPC** | Only use VPC if accessing private resources |\n| **Event filtering** | Filter at source to reduce invocations |\n| **Dead-letter queues** | Configure DLQ for async invocations |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":15000,"content_sha256":"66ed72dd2d52ddcce04a5a0ce180fb3aae0d5019b4382637ab94e5dbce5db1fc"},{"filename":"references/msk.md","content":"# Amazon MSK (Managed Kafka)\n\n## Cluster Types\n\n### Serverless Cluster\n```bash\n# Create serverless cluster (auto-scaling)\naws kafka create-cluster-v2 \\\n --cluster-name app-events \\\n --serverless '{\n \"VpcConfigs\": [{\n \"SubnetIds\": [\"subnet-1\", \"subnet-2\", \"subnet-3\"],\n \"SecurityGroupIds\": [\"sg-12345\"]\n }],\n \"ClientAuthentication\": {\n \"Sasl\": {\"Iam\": {\"Enabled\": true}}\n }\n }'\n```\n\n### Provisioned Cluster\n```bash\n# Create provisioned cluster\naws kafka create-cluster \\\n --cluster-name production-cluster \\\n --kafka-version \"3.5.1\" \\\n --number-of-broker-nodes 3 \\\n --broker-node-group-info '{\n \"InstanceType\": \"kafka.m5.large\",\n \"ClientSubnets\": [\"subnet-1\", \"subnet-2\", \"subnet-3\"],\n \"SecurityGroups\": [\"sg-12345\"],\n \"StorageInfo\": {\n \"EbsStorageInfo\": {\n \"VolumeSize\": 100,\n \"ProvisionedThroughput\": {\n \"Enabled\": true,\n \"VolumeThroughput\": 250\n }\n }\n }\n }' \\\n --encryption-info '{\n \"EncryptionInTransit\": {\n \"ClientBroker\": \"TLS\",\n \"InCluster\": true\n },\n \"EncryptionAtRest\": {\n \"DataVolumeKMSKeyId\": \"alias/aws/kafka\"\n }\n }' \\\n --client-authentication '{\n \"Sasl\": {\"Iam\": {\"Enabled\": true}},\n \"Tls\": {\"CertificateAuthorityArnList\": []}\n }' \\\n --enhanced-monitoring PER_TOPIC_PER_BROKER \\\n --logging-info '{\n \"BrokerLogs\": {\n \"CloudWatchLogs\": {\n \"Enabled\": true,\n \"LogGroup\": \"/aws/msk/production-cluster\"\n }\n }\n }'\n```\n\n## Cluster Management\n\n### List and Describe\n```bash\n# List clusters\naws kafka list-clusters-v2\n\n# Describe cluster\naws kafka describe-cluster-v2 --cluster-arn $CLUSTER_ARN\n\n# Get bootstrap brokers\naws kafka get-bootstrap-brokers --cluster-arn $CLUSTER_ARN\n\n# Get cluster operation (during updates)\naws kafka describe-cluster-operation --cluster-operation-arn $OP_ARN\n```\n\n### Update Cluster\n```bash\n# Update broker count (scale out)\naws kafka update-broker-count \\\n --cluster-arn $CLUSTER_ARN \\\n --current-version $CLUSTER_VERSION \\\n --target-number-of-broker-nodes 6\n\n# Update broker storage\naws kafka update-broker-storage \\\n --cluster-arn $CLUSTER_ARN \\\n --current-version $CLUSTER_VERSION \\\n --target-broker-ebs-volume-info '[{\"KafkaBrokerNodeId\": \"1\", \"VolumeSizeGB\": 200}]'\n\n# Update broker type\naws kafka update-broker-type \\\n --cluster-arn $CLUSTER_ARN \\\n --current-version $CLUSTER_VERSION \\\n --target-instance-type kafka.m5.xlarge\n\n# Update Kafka version\naws kafka update-cluster-kafka-version \\\n --cluster-arn $CLUSTER_ARN \\\n --current-version $CLUSTER_VERSION \\\n --target-kafka-version \"3.6.0\"\n\n# Update monitoring\naws kafka update-monitoring \\\n --cluster-arn $CLUSTER_ARN \\\n --current-version $CLUSTER_VERSION \\\n --enhanced-monitoring PER_TOPIC_PER_PARTITION \\\n --open-monitoring '{\n \"Prometheus\": {\n \"JmxExporter\": {\"EnabledInBroker\": true},\n \"NodeExporter\": {\"EnabledInBroker\": true}\n }\n }'\n```\n\n### Delete Cluster\n```bash\naws kafka delete-cluster --cluster-arn $CLUSTER_ARN\n```\n\n## Configuration\n\n### Create Configuration\n```bash\n# Create custom configuration\naws kafka create-configuration \\\n --name production-config \\\n --kafka-versions \"3.5.1\" \"3.6.0\" \\\n --server-properties \"$(cat \u003c\u003cEOF\nauto.create.topics.enable=false\ndefault.replication.factor=3\nmin.insync.replicas=2\nnum.partitions=6\nlog.retention.hours=168\nlog.segment.bytes=1073741824\ncompression.type=lz4\nEOF\n)\"\n```\n\n### Apply Configuration\n```bash\n# Update cluster to use configuration\naws kafka update-cluster-configuration \\\n --cluster-arn $CLUSTER_ARN \\\n --current-version $CLUSTER_VERSION \\\n --configuration-info Arn=$CONFIG_ARN,Revision=1\n```\n\n### Manage Configurations\n```bash\n# List configurations\naws kafka list-configurations\n\n# Describe configuration\naws kafka describe-configuration --arn $CONFIG_ARN\n\n# List configuration revisions\naws kafka list-configuration-revisions --arn $CONFIG_ARN\n\n# Update configuration (creates new revision)\naws kafka update-configuration \\\n --arn $CONFIG_ARN \\\n --server-properties \"auto.create.topics.enable=true\"\n```\n\n## MSK Connect\n\n### Create Connector\n```bash\n# Create MSK Connect connector\naws kafkaconnect create-connector \\\n --connector-name s3-sink-connector \\\n --kafka-cluster '{\n \"ApacheKafkaCluster\": {\n \"BootstrapServers\": \"b-1.cluster.abc123.kafka.us-east-1.amazonaws.com:9098\",\n \"Vpc\": {\n \"SecurityGroups\": [\"sg-12345\"],\n \"Subnets\": [\"subnet-1\", \"subnet-2\"]\n }\n }\n }' \\\n --kafka-cluster-client-authentication '{\n \"AuthenticationType\": \"IAM\"\n }' \\\n --kafka-cluster-encryption-in-transit '{\n \"EncryptionType\": \"TLS\"\n }' \\\n --capacity '{\n \"AutoScaling\": {\n \"MaxWorkerCount\": 4,\n \"MinWorkerCount\": 1,\n \"ScaleInPolicy\": {\"CpuUtilizationPercentage\": 20},\n \"ScaleOutPolicy\": {\"CpuUtilizationPercentage\": 80},\n \"McuCount\": 1\n }\n }' \\\n --connector-configuration '{\n \"connector.class\": \"io.confluent.connect.s3.S3SinkConnector\",\n \"tasks.max\": \"2\",\n \"topics\": \"my-topic\",\n \"s3.bucket.name\": \"my-bucket\",\n \"s3.region\": \"us-east-1\",\n \"flush.size\": \"1000\",\n \"storage.class\": \"io.confluent.connect.s3.storage.S3Storage\",\n \"format.class\": \"io.confluent.connect.s3.format.json.JsonFormat\"\n }' \\\n --plugins '[{\n \"CustomPlugin\": {\n \"CustomPluginArn\": \"arn:aws:kafkaconnect:us-east-1:123456789012:custom-plugin/s3-plugin/abc123\",\n \"Revision\": 1\n }\n }]' \\\n --service-execution-role-arn arn:aws:iam::123456789012:role/MSKConnectRole\n```\n\n### Manage Connectors\n```bash\n# List connectors\naws kafkaconnect list-connectors\n\n# Describe connector\naws kafkaconnect describe-connector --connector-arn $CONNECTOR_ARN\n\n# Update connector\naws kafkaconnect update-connector \\\n --connector-arn $CONNECTOR_ARN \\\n --current-version $VERSION \\\n --capacity '{\n \"AutoScaling\": {\n \"MaxWorkerCount\": 8,\n \"MinWorkerCount\": 2,\n \"McuCount\": 2\n }\n }'\n\n# Delete connector\naws kafkaconnect delete-connector --connector-arn $CONNECTOR_ARN\n```\n\n### Custom Plugins\n```bash\n# Create custom plugin\naws kafkaconnect create-custom-plugin \\\n --name s3-sink-plugin \\\n --content-type ZIP \\\n --location '{\n \"S3Location\": {\n \"BucketArn\": \"arn:aws:s3:::my-plugins-bucket\",\n \"FileKey\": \"plugins/s3-sink-connector.zip\"\n }\n }'\n\n# List plugins\naws kafkaconnect list-custom-plugins\n```\n\n## Replicator\n\n### Create Replicator\n```bash\n# Create cross-region replicator\naws kafka create-replicator \\\n --replicator-name cross-region-replicator \\\n --kafka-clusters '[\n {\n \"AmazonMskCluster\": {\"MskClusterArn\": \"arn:aws:kafka:us-east-1:123456789012:cluster/source\"},\n \"VpcConfig\": {\"SecurityGroupIds\": [\"sg-source\"], \"SubnetIds\": [\"subnet-source\"]}\n },\n {\n \"AmazonMskCluster\": {\"MskClusterArn\": \"arn:aws:kafka:eu-west-1:123456789012:cluster/target\"},\n \"VpcConfig\": {\"SecurityGroupIds\": [\"sg-target\"], \"SubnetIds\": [\"subnet-target\"]}\n }\n ]' \\\n --replication-info-list '[{\n \"SourceKafkaClusterArn\": \"arn:aws:kafka:us-east-1:123456789012:cluster/source\",\n \"TargetKafkaClusterArn\": \"arn:aws:kafka:eu-west-1:123456789012:cluster/target\",\n \"TopicReplication\": {\n \"TopicsToReplicate\": [\"orders.*\", \"events.*\"],\n \"CopyTopicConfigurations\": true,\n \"DetectAndCopyNewTopics\": true\n },\n \"ConsumerGroupReplication\": {\n \"ConsumerGroupsToReplicate\": [\".*\"]\n },\n \"TargetCompressionType\": \"LZ4\"\n }]' \\\n --service-execution-role-arn arn:aws:iam::123456789012:role/MSKReplicatorRole\n```\n\n## Monitoring\n\n### CloudWatch Metrics\n```bash\n# Get broker CPU metrics\naws cloudwatch get-metric-statistics \\\n --namespace \"AWS/Kafka\" \\\n --metric-name \"CpuUser\" \\\n --dimensions Name=ClusterName,Value=production-cluster Name=BrokerId,Value=1 \\\n --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ) \\\n --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \\\n --period 300 \\\n --statistics Average\n\n# Get consumer lag\naws cloudwatch get-metric-statistics \\\n --namespace \"AWS/Kafka\" \\\n --metric-name \"EstimatedMaxTimeLag\" \\\n --dimensions Name=ClusterName,Value=production-cluster Name=Topic,Value=my-topic Name=ConsumerGroup,Value=my-group \\\n --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ) \\\n --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \\\n --period 60 \\\n --statistics Maximum\n```\n\n### Enable Prometheus Monitoring\n```bash\naws kafka update-monitoring \\\n --cluster-arn $CLUSTER_ARN \\\n --current-version $CLUSTER_VERSION \\\n --open-monitoring '{\n \"Prometheus\": {\n \"JmxExporter\": {\"EnabledInBroker\": true},\n \"NodeExporter\": {\"EnabledInBroker\": true}\n }\n }'\n```\n\n## Useful Queries\n\n```bash\n# Get bootstrap brokers for IAM auth\naws kafka get-bootstrap-brokers \\\n --cluster-arn $CLUSTER_ARN \\\n --query 'BootstrapBrokerStringSaslIam' \\\n --output text\n\n# Get cluster status\naws kafka describe-cluster-v2 \\\n --cluster-arn $CLUSTER_ARN \\\n --query 'ClusterInfo.State'\n\n# List cluster operations\naws kafka list-cluster-operations --cluster-arn $CLUSTER_ARN\n\n# Get Zookeeper connection string\naws kafka describe-cluster \\\n --cluster-arn $CLUSTER_ARN \\\n --query 'ClusterInfo.ZookeeperConnectString'\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Serverless** | Use for variable or unpredictable workloads |\n| **IAM auth** | Use IAM for AWS service integration |\n| **Encryption** | Enable TLS in-transit and KMS at-rest |\n| **Monitoring** | Enable PER_TOPIC_PER_BROKER for debugging |\n| **Replication** | Set min.insync.replicas=2 for durability |\n| **Partitions** | Plan partition count based on throughput needs |\n| **MSK Connect** | Use for managed Kafka Connect deployments |\n| **Tiered storage** | Enable for cost-effective long retention |\n| **Multi-AZ** | Use 3 AZs for high availability |\n| **Prometheus** | Enable for detailed JMX metrics |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":10516,"content_sha256":"dd94c854a65541dc184b55f598d096eaf2e5fd4e83e62b3f6ff29c94710ebd33"},{"filename":"references/private-parameters.md","content":"# Secrets & Parameters Management\n\n## When to Use Which\n\n| Need | Use | Why |\n|------|-----|-----|\n| App config, feature flags | Parameter Store | Free, simple hierarchies |\n| API keys, passwords | Secrets Manager | Auto-rotation, audit |\n| Database credentials | Secrets Manager | RDS integration |\n| Simple key-value | Parameter Store | Lower cost |\n| Cross-region secrets | Secrets Manager | Built-in replication |\n\n## Parameter Store\n\n### Create/Update Parameters\n\n```bash\n# String parameter\naws ssm put-parameter --name \"/app/prod/db/host\" --value \"db.example.com\" --type String\n\n# SecureString with default KMS\naws ssm put-parameter --name \"/app/prod/db/password\" --value \"secret123\" --type SecureString\n\n# SecureString with custom KMS key\naws ssm put-parameter --name \"/app/prod/api/key\" --value \"sk-xxx\" --type SecureString --key-id alias/my-key\n\n# Update existing (requires --overwrite)\naws ssm put-parameter --name \"/app/prod/db/host\" --value \"new-db.example.com\" --type String --overwrite\n\n# StringList\naws ssm put-parameter --name \"/app/prod/allowed-ips\" --value \"10.0.0.1,10.0.0.2,10.0.0.3\" --type StringList\n```\n\n### Retrieve Parameters\n\n```bash\n# Single parameter\naws ssm get-parameter --name \"/app/prod/db/host\"\n\n# Decrypt SecureString\naws ssm get-parameter --name \"/app/prod/db/password\" --with-decryption\n\n# Value only\naws ssm get-parameter --name \"/app/prod/db/host\" --query \"Parameter.Value\" --output text\n\n# Multiple parameters\naws ssm get-parameters --names \"/app/prod/db/host\" \"/app/prod/db/port\" --with-decryption\n\n# By path (hierarchy)\naws ssm get-parameters-by-path --path \"/app/prod\" --recursive --with-decryption\n\n# By path with filter\naws ssm get-parameters-by-path --path \"/app/prod\" --parameter-filters \"Key=Type,Values=SecureString\"\n```\n\n### Parameter Hierarchies\n\n```\n/app\n /prod\n /db\n /host\n /password\n /api\n /key\n /dev\n /db\n /host\n```\n\nBenefits:\n- Organize by environment, service, component\n- Retrieve entire subtrees with `get-parameters-by-path`\n- Apply IAM policies to paths\n\n### Delete Parameters\n\n```bash\n# Single delete\naws ssm delete-parameter --name \"/app/prod/old-param\"\n\n# Bulk delete\naws ssm delete-parameters --names \"/app/prod/old1\" \"/app/prod/old2\"\n```\n\n### Parameter History\n\n```bash\n# View version history\naws ssm get-parameter-history --name \"/app/prod/db/host\"\n\n# Get specific version\naws ssm get-parameter --name \"/app/prod/db/host:2\"\n\n# Get labels\naws ssm get-parameter-history --name \"/app/prod/db/host\" --with-decryption\n```\n\n## Secrets Manager\n\n### Create Secrets\n\n```bash\n# Simple string secret\naws secretsmanager create-secret --name \"prod/api/stripe-key\" --secret-string \"sk_live_xxx\"\n\n# JSON secret (common for credentials)\naws secretsmanager create-secret --name \"prod/db/credentials\" --secret-string '{\"username\":\"admin\",\"password\":\"secret123\",\"host\":\"db.example.com\"}'\n\n# With description and tags\naws secretsmanager create-secret \\\n --name \"prod/api/key\" \\\n --secret-string \"xxx\" \\\n --description \"Production API key\" \\\n --tags Key=Environment,Value=prod Key=Team,Value=backend\n\n# Binary secret\naws secretsmanager create-secret --name \"prod/certs/tls\" --secret-binary fileb://cert.der\n```\n\n### Retrieve Secrets\n\n```bash\n# Get current version\naws secretsmanager get-secret-value --secret-id \"prod/api/key\"\n\n# Value only\naws secretsmanager get-secret-value --secret-id \"prod/api/key\" --query SecretString --output text\n\n# Parse JSON secret\naws secretsmanager get-secret-value --secret-id \"prod/db/credentials\" --query SecretString --output text | jq -r .password\n\n# Get specific version\naws secretsmanager get-secret-value --secret-id \"prod/api/key\" --version-id \"abc123\"\n\n# Get by stage\naws secretsmanager get-secret-value --secret-id \"prod/api/key\" --version-stage AWSPREVIOUS\n```\n\n### Update Secrets\n\n```bash\n# Update value\naws secretsmanager put-secret-value --secret-id \"prod/api/key\" --secret-string \"new-value\"\n\n# Update metadata only\naws secretsmanager update-secret --secret-id \"prod/api/key\" --description \"Updated description\"\n\n# Rotate with custom KMS key\naws secretsmanager update-secret --secret-id \"prod/api/key\" --kms-key-id alias/new-key\n```\n\n### Rotation\n\n```bash\n# Enable rotation with Lambda\naws secretsmanager rotate-secret \\\n --secret-id \"prod/db/credentials\" \\\n --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:rotate-db-creds \\\n --rotation-rules AutomaticallyAfterDays=30\n\n# Trigger immediate rotation\naws secretsmanager rotate-secret --secret-id \"prod/db/credentials\"\n\n# Check rotation status\naws secretsmanager describe-secret --secret-id \"prod/db/credentials\" --query \"RotationEnabled\"\n```\n\n### Delete and Restore\n\n```bash\n# Schedule deletion (7-30 days)\naws secretsmanager delete-secret --secret-id \"prod/old-key\" --recovery-window-in-days 7\n\n# Force immediate deletion (cannot recover!)\naws secretsmanager delete-secret --secret-id \"prod/old-key\" --force-delete-without-recovery\n\n# Cancel deletion/restore\naws secretsmanager restore-secret --secret-id \"prod/old-key\"\n```\n\n### Cross-Region Replication\n\n```bash\n# Add replica regions\naws secretsmanager replicate-secret-to-regions \\\n --secret-id \"prod/api/key\" \\\n --add-replica-regions Region=eu-west-1 Region=ap-southeast-1\n\n# Remove replica\naws secretsmanager remove-regions-from-replication \\\n --secret-id \"prod/api/key\" \\\n --remove-replica-regions eu-west-1\n```\n\n## Cross-Service Patterns\n\n### Lambda Environment from Secrets Manager\n\n```bash\n# Store Lambda config as secret\naws secretsmanager create-secret --name \"prod/lambda/my-func/config\" \\\n --secret-string '{\"API_KEY\":\"xxx\",\"DB_HOST\":\"db.example.com\"}'\n\n# Lambda reads at runtime:\n# boto3.client(\"secretsmanager\").get_secret_value(SecretId=\"prod/lambda/my-func/config\")\n```\n\n### ECS Task from Parameter Store\n\n```bash\n# Store task secrets\naws ssm put-parameter --name \"/ecs/prod/my-service/db-password\" --value \"xxx\" --type SecureString\n\n# Reference in task definition:\n# \"secrets\": [{\"name\": \"DB_PASSWORD\", \"valueFrom\": \"/ecs/prod/my-service/db-password\"}]\n```\n\n## Useful Queries\n\n```bash\n# List all parameters by path\naws ssm describe-parameters --parameter-filters \"Key=Path,Values=/app/prod\"\n\n# Find parameters by tag\naws ssm describe-parameters --parameter-filters \"Key=tag:Environment,Values=production\"\n\n# List all secrets\naws secretsmanager list-secrets\n\n# Find secrets by name pattern\naws secretsmanager list-secrets --filters Key=name,Values=prod\n\n# Find secrets by tag\naws secretsmanager list-secrets --filters Key=tag-key,Values=Environment Key=tag-value,Values=prod\n```\n\n## Best Practices\n\n1. **Use hierarchies**: `/app/env/service/key` for organization\n2. **Tag everything**: Environment, Team, Application for filtering\n3. **Prefer SecureString**: Always encrypt sensitive values\n4. **Rotate regularly**: Enable auto-rotation for Secrets Manager\n5. **Limit IAM scope**: Grant access to specific paths, not `*`\n6. **Audit access**: Enable CloudTrail for both services\n7. **Use resource policies**: Control cross-account access\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7004,"content_sha256":"b4cb28e2206b43bcaa0ba0b9d67d44149a1ece096e4c8e11805fdee3ed5c155c"},{"filename":"references/s3.md","content":"# S3 Storage\n\n## Contents\n\n- [Bucket Management](#bucket-management)\n- [Block Public Access](#block-public-access)\n- [Bucket Policies](#bucket-policies)\n- [Object Operations](#object-operations)\n- [Presigned URLs](#presigned-urls)\n- [Versioning](#versioning)\n- [Encryption](#encryption)\n- [Lifecycle Policies](#lifecycle-policies)\n- [Transfer Acceleration](#transfer-acceleration)\n- [Static Website Hosting](#static-website-hosting)\n- [CORS Configuration](#cors-configuration)\n- [Cross-Region Replication](#cross-region-replication)\n- [S3 Select](#s3-select)\n- [Object Lock (WORM)](#object-lock-worm)\n- [Event Notifications](#event-notifications)\n- [Useful Queries](#useful-queries)\n- [Performance Optimization](#performance-optimization)\n- [Best Practices](#best-practices)\n\n---\n\n## Bucket Management\n\n### Create Bucket\n```bash\n# Create bucket (us-east-1 doesn't need LocationConstraint)\naws s3 mb s3://my-bucket\n\n# Create in specific region\naws s3 mb s3://my-bucket --region eu-west-1\n\n# Using s3api with explicit region constraint\naws s3api create-bucket \\\n --bucket my-bucket \\\n --region eu-west-1 \\\n --create-bucket-configuration LocationConstraint=eu-west-1\n```\n\n### List and Describe Buckets\n```bash\n# List all buckets\naws s3 ls\n\n# List bucket contents\naws s3 ls s3://my-bucket/\naws s3 ls s3://my-bucket/prefix/ --recursive\n\n# Get bucket location\naws s3api get-bucket-location --bucket my-bucket\n\n# Get bucket info\naws s3api head-bucket --bucket my-bucket\n```\n\n### Delete Bucket\n```bash\n# Delete empty bucket\naws s3 rb s3://my-bucket\n\n# Force delete (remove all objects first)\naws s3 rb s3://my-bucket --force\n```\n\n## Block Public Access\n\n### Configure Block Public Access\n```bash\n# Block all public access (recommended for most buckets)\naws s3api put-public-access-block \\\n --bucket my-bucket \\\n --public-access-block-configuration \\\n BlockPublicAcls=true,\\\nIgnorePublicAcls=true,\\\nBlockPublicPolicy=true,\\\nRestrictPublicBuckets=true\n\n# Get current settings\naws s3api get-public-access-block --bucket my-bucket\n```\n\n### Account-Level Block\n```bash\n# Block public access for entire account\naws s3control put-public-access-block \\\n --account-id 123456789012 \\\n --public-access-block-configuration \\\n BlockPublicAcls=true,\\\nIgnorePublicAcls=true,\\\nBlockPublicPolicy=true,\\\nRestrictPublicBuckets=true\n```\n\n## Bucket Policies\n\n### Set Bucket Policy\n```bash\naws s3api put-bucket-policy \\\n --bucket my-bucket \\\n --policy file://bucket-policy.json\n```\n\n**bucket-policy.json (Deny non-HTTPS):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"DenyNonHTTPS\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": \"s3:*\",\n \"Resource\": [\n \"arn:aws:s3:::my-bucket\",\n \"arn:aws:s3:::my-bucket/*\"\n ],\n \"Condition\": {\n \"Bool\": {\"aws:SecureTransport\": \"false\"}\n }\n }\n ]\n}\n```\n\n**bucket-policy.json (Cross-account read access):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"CrossAccountRead\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"AWS\": \"arn:aws:iam::OTHER_ACCOUNT_ID:root\"\n },\n \"Action\": [\n \"s3:GetObject\",\n \"s3:ListBucket\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::my-bucket\",\n \"arn:aws:s3:::my-bucket/*\"\n ]\n }\n ]\n}\n```\n\n**bucket-policy.json (VPC endpoint only):**\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"VPCEndpointOnly\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": \"s3:*\",\n \"Resource\": [\n \"arn:aws:s3:::my-bucket\",\n \"arn:aws:s3:::my-bucket/*\"\n ],\n \"Condition\": {\n \"StringNotEquals\": {\n \"aws:SourceVpce\": \"vpce-1234567890abcdef0\"\n }\n }\n }\n ]\n}\n```\n\n### Manage Bucket Policy\n```bash\n# Get current policy\naws s3api get-bucket-policy --bucket my-bucket\n\n# Delete policy\naws s3api delete-bucket-policy --bucket my-bucket\n```\n\n## Object Operations\n\n### Copy and Upload\n```bash\n# Upload single file\naws s3 cp document.pdf s3://my-bucket/docs/\n\n# Upload with storage class\naws s3 cp archive.zip s3://my-bucket/ --storage-class STANDARD_IA\n\n# Upload with metadata\naws s3 cp file.txt s3://my-bucket/ \\\n --metadata '{\"project\":\"alpha\",\"owner\":\"team-a\"}'\n\n# Upload with content type\naws s3 cp index.html s3://my-bucket/ --content-type \"text/html\"\n\n# Upload directory recursively\naws s3 cp ./local-dir s3://my-bucket/remote-dir/ --recursive\n```\n\n### Download\n```bash\n# Download single file\naws s3 cp s3://my-bucket/docs/document.pdf ./\n\n# Download directory\naws s3 cp s3://my-bucket/logs/ ./logs/ --recursive\n\n# Download with specific version\naws s3api get-object \\\n --bucket my-bucket \\\n --key myfile.txt \\\n --version-id abc123 \\\n output.txt\n```\n\n### Move and Rename\n```bash\n# Move (copy + delete)\naws s3 mv s3://my-bucket/old.txt s3://my-bucket/new.txt\n\n# Move between buckets\naws s3 mv s3://source-bucket/file.txt s3://dest-bucket/file.txt\n```\n\n### Sync\n```bash\n# Sync local to S3 (upload changes only)\naws s3 sync ./build s3://my-bucket/site/\n\n# Sync with delete (mirror)\naws s3 sync ./build s3://my-bucket/site/ --delete\n\n# Sync with exclusions\naws s3 sync ./project s3://my-bucket/ \\\n --exclude \"*.git/*\" \\\n --exclude \"node_modules/*\" \\\n --include \"*.js\"\n\n# Sync between buckets\naws s3 sync s3://source-bucket s3://dest-bucket\n\n# Dry run (show what would happen)\naws s3 sync ./build s3://my-bucket/ --dryrun\n```\n\n### Delete Objects\n```bash\n# Delete single object\naws s3 rm s3://my-bucket/file.txt\n\n# Delete recursively\naws s3 rm s3://my-bucket/logs/ --recursive\n\n# Delete with prefix\naws s3 rm s3://my-bucket/ --recursive --exclude \"*\" --include \"temp/*\"\n```\n\n## Presigned URLs\n\n### Generate Download URLs\n```bash\n# Default 1 hour expiry\naws s3 presign s3://my-bucket/report.pdf\n\n# Custom expiry (seconds)\naws s3 presign s3://my-bucket/report.pdf --expires-in 86400 # 24 hours\n```\n\n### Generate Upload URLs\n```bash\n# Using s3api for upload presigned URL\naws s3api put-object \\\n --bucket my-bucket \\\n --key uploads/file.txt \\\n --generate-presigned-url \\\n --expires-in 3600\n```\n\n## Versioning\n\n### Enable Versioning\n```bash\n# Enable\naws s3api put-bucket-versioning \\\n --bucket my-bucket \\\n --versioning-configuration Status=Enabled\n\n# Suspend (keeps existing versions)\naws s3api put-bucket-versioning \\\n --bucket my-bucket \\\n --versioning-configuration Status=Suspended\n\n# Get status\naws s3api get-bucket-versioning --bucket my-bucket\n```\n\n### Work with Versions\n```bash\n# List all versions\naws s3api list-object-versions --bucket my-bucket\n\n# List versions for specific key\naws s3api list-object-versions \\\n --bucket my-bucket \\\n --prefix myfile.txt\n\n# Get specific version\naws s3api get-object \\\n --bucket my-bucket \\\n --key myfile.txt \\\n --version-id \"abc123xyz\" \\\n output.txt\n\n# Delete specific version\naws s3api delete-object \\\n --bucket my-bucket \\\n --key myfile.txt \\\n --version-id \"abc123xyz\"\n\n# Restore deleted object (delete the delete marker)\naws s3api delete-object \\\n --bucket my-bucket \\\n --key myfile.txt \\\n --version-id \"delete-marker-version-id\"\n```\n\n## Encryption\n\n### Server-Side Encryption (Default)\n```bash\n# SSE-S3 (AES-256, AWS managed keys)\naws s3api put-bucket-encryption \\\n --bucket my-bucket \\\n --server-side-encryption-configuration '{\n \"Rules\": [{\n \"ApplyServerSideEncryptionByDefault\": {\n \"SSEAlgorithm\": \"AES256\"\n },\n \"BucketKeyEnabled\": true\n }]\n }'\n\n# SSE-KMS (Customer managed key)\naws s3api put-bucket-encryption \\\n --bucket my-bucket \\\n --server-side-encryption-configuration '{\n \"Rules\": [{\n \"ApplyServerSideEncryptionByDefault\": {\n \"SSEAlgorithm\": \"aws:kms\",\n \"KMSMasterKeyID\": \"arn:aws:kms:us-east-1:123456789012:key/abc123\"\n },\n \"BucketKeyEnabled\": true\n }]\n }'\n```\n\n### Upload with Encryption\n```bash\n# SSE-S3\naws s3 cp file.txt s3://my-bucket/ --sse AES256\n\n# SSE-KMS\naws s3 cp file.txt s3://my-bucket/ \\\n --sse aws:kms \\\n --sse-kms-key-id alias/my-key\n```\n\n## Lifecycle Policies\n\n### Create Lifecycle Policy\n```bash\naws s3api put-bucket-lifecycle-configuration \\\n --bucket my-bucket \\\n --lifecycle-configuration file://lifecycle.json\n```\n\n**lifecycle.json (Comprehensive):**\n```json\n{\n \"Rules\": [\n {\n \"ID\": \"TransitionToIA\",\n \"Status\": \"Enabled\",\n \"Filter\": {\"Prefix\": \"data/\"},\n \"Transitions\": [\n {\"Days\": 30, \"StorageClass\": \"STANDARD_IA\"},\n {\"Days\": 90, \"StorageClass\": \"GLACIER\"},\n {\"Days\": 365, \"StorageClass\": \"DEEP_ARCHIVE\"}\n ]\n },\n {\n \"ID\": \"ExpireTempFiles\",\n \"Status\": \"Enabled\",\n \"Filter\": {\"Prefix\": \"temp/\"},\n \"Expiration\": {\"Days\": 7}\n },\n {\n \"ID\": \"CleanupIncompleteMultipart\",\n \"Status\": \"Enabled\",\n \"Filter\": {\"Prefix\": \"\"},\n \"AbortIncompleteMultipartUpload\": {\"DaysAfterInitiation\": 7}\n },\n {\n \"ID\": \"DeleteOldVersions\",\n \"Status\": \"Enabled\",\n \"Filter\": {\"Prefix\": \"\"},\n \"NoncurrentVersionExpiration\": {\"NoncurrentDays\": 30}\n }\n ]\n}\n```\n\n### Manage Lifecycle\n```bash\n# Get lifecycle configuration\naws s3api get-bucket-lifecycle-configuration --bucket my-bucket\n\n# Delete lifecycle\naws s3api delete-bucket-lifecycle --bucket my-bucket\n```\n\n## Transfer Acceleration\n\n### Enable Transfer Acceleration\n```bash\n# Enable\naws s3api put-bucket-accelerate-configuration \\\n --bucket my-bucket \\\n --accelerate-configuration Status=Enabled\n\n# Get status\naws s3api get-bucket-accelerate-configuration --bucket my-bucket\n```\n\n### Use Accelerated Endpoint\n```bash\n# Upload using accelerated endpoint\naws s3 cp large-file.zip s3://my-bucket/ \\\n --endpoint-url https://my-bucket.s3-accelerate.amazonaws.com\n\n# Or set as default\naws configure set s3.use_accelerate_endpoint true\n```\n\n## Static Website Hosting\n\n### Enable Website Hosting\n```bash\naws s3api put-bucket-website \\\n --bucket my-bucket \\\n --website-configuration '{\n \"IndexDocument\": {\"Suffix\": \"index.html\"},\n \"ErrorDocument\": {\"Key\": \"error.html\"}\n }'\n```\n\n### Public Website Policy\n```bash\n# First, disable block public access\naws s3api put-public-access-block \\\n --bucket my-bucket \\\n --public-access-block-configuration \\\n BlockPublicAcls=false,\\\nIgnorePublicAcls=false,\\\nBlockPublicPolicy=false,\\\nRestrictPublicBuckets=false\n\n# Then apply public read policy\naws s3api put-bucket-policy \\\n --bucket my-bucket \\\n --policy '{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Sid\": \"PublicRead\",\n \"Effect\": \"Allow\",\n \"Principal\": \"*\",\n \"Action\": \"s3:GetObject\",\n \"Resource\": \"arn:aws:s3:::my-bucket/*\"\n }]\n }'\n```\n\n### Get Website URL\n```bash\n# Website endpoint format\n# http://my-bucket.s3-website-us-east-1.amazonaws.com\naws s3api get-bucket-website --bucket my-bucket\n```\n\n## CORS Configuration\n\n### Set CORS Rules\n```bash\naws s3api put-bucket-cors \\\n --bucket my-bucket \\\n --cors-configuration file://cors.json\n```\n\n**cors.json:**\n```json\n{\n \"CORSRules\": [\n {\n \"AllowedHeaders\": [\"*\"],\n \"AllowedMethods\": [\"GET\", \"PUT\", \"POST\"],\n \"AllowedOrigins\": [\"https://example.com\", \"https://app.example.com\"],\n \"ExposeHeaders\": [\"ETag\"],\n \"MaxAgeSeconds\": 3600\n }\n ]\n}\n```\n\n### Manage CORS\n```bash\n# Get CORS\naws s3api get-bucket-cors --bucket my-bucket\n\n# Delete CORS\naws s3api delete-bucket-cors --bucket my-bucket\n```\n\n## Cross-Region Replication\n\n### Enable Replication\n```bash\n# First enable versioning on both buckets\naws s3api put-bucket-versioning \\\n --bucket source-bucket \\\n --versioning-configuration Status=Enabled\n\naws s3api put-bucket-versioning \\\n --bucket dest-bucket \\\n --versioning-configuration Status=Enabled\n\n# Set replication configuration\naws s3api put-bucket-replication \\\n --bucket source-bucket \\\n --replication-configuration file://replication.json\n```\n\n**replication.json:**\n```json\n{\n \"Role\": \"arn:aws:iam::123456789012:role/S3ReplicationRole\",\n \"Rules\": [\n {\n \"ID\": \"ReplicateAll\",\n \"Status\": \"Enabled\",\n \"Priority\": 1,\n \"Filter\": {},\n \"Destination\": {\n \"Bucket\": \"arn:aws:s3:::dest-bucket\",\n \"StorageClass\": \"STANDARD\"\n },\n \"DeleteMarkerReplication\": {\"Status\": \"Enabled\"}\n }\n ]\n}\n```\n\n## S3 Select\n\nQuery data directly in S3 without downloading.\n\n```bash\n# Query CSV file\naws s3api select-object-content \\\n --bucket my-bucket \\\n --key data.csv \\\n --expression \"SELECT * FROM s3object s WHERE s.status = 'active'\" \\\n --expression-type SQL \\\n --input-serialization '{\"CSV\": {\"FileHeaderInfo\": \"USE\"}}' \\\n --output-serialization '{\"JSON\": {}}' \\\n output.json\n\n# Query JSON file\naws s3api select-object-content \\\n --bucket my-bucket \\\n --key data.json \\\n --expression \"SELECT s.name, s.email FROM s3object s WHERE s.age > 30\" \\\n --expression-type SQL \\\n --input-serialization '{\"JSON\": {\"Type\": \"LINES\"}}' \\\n --output-serialization '{\"JSON\": {}}' \\\n output.json\n```\n\n## Object Lock (WORM)\n\n### Enable Object Lock\n```bash\n# Must be enabled at bucket creation\naws s3api create-bucket \\\n --bucket my-bucket \\\n --object-lock-enabled-for-bucket\n\n# Set default retention\naws s3api put-object-lock-configuration \\\n --bucket my-bucket \\\n --object-lock-configuration '{\n \"ObjectLockEnabled\": \"Enabled\",\n \"Rule\": {\n \"DefaultRetention\": {\n \"Mode\": \"GOVERNANCE\",\n \"Days\": 365\n }\n }\n }'\n```\n\n### Apply Retention to Object\n```bash\n# Governance mode (can be overridden with permission)\naws s3api put-object-retention \\\n --bucket my-bucket \\\n --key important-file.txt \\\n --retention '{\"Mode\": \"GOVERNANCE\", \"RetainUntilDate\": \"2025-12-31T00:00:00Z\"}'\n\n# Compliance mode (cannot be overridden)\naws s3api put-object-retention \\\n --bucket my-bucket \\\n --key legal-doc.pdf \\\n --retention '{\"Mode\": \"COMPLIANCE\", \"RetainUntilDate\": \"2030-12-31T00:00:00Z\"}'\n\n# Legal hold\naws s3api put-object-legal-hold \\\n --bucket my-bucket \\\n --key evidence.pdf \\\n --legal-hold '{\"Status\": \"ON\"}'\n```\n\n## Event Notifications\n\n### Configure Notifications\n```bash\naws s3api put-bucket-notification-configuration \\\n --bucket my-bucket \\\n --notification-configuration file://notifications.json\n```\n\n**notifications.json (Lambda trigger):**\n```json\n{\n \"LambdaFunctionConfigurations\": [\n {\n \"Id\": \"ProcessNewUploads\",\n \"LambdaFunctionArn\": \"arn:aws:lambda:us-east-1:123456789012:function:ProcessS3\",\n \"Events\": [\"s3:ObjectCreated:*\"],\n \"Filter\": {\n \"Key\": {\n \"FilterRules\": [\n {\"Name\": \"prefix\", \"Value\": \"uploads/\"},\n {\"Name\": \"suffix\", \"Value\": \".json\"}\n ]\n }\n }\n }\n ]\n}\n```\n\n**notifications.json (SQS):**\n```json\n{\n \"QueueConfigurations\": [\n {\n \"Id\": \"NewFileQueue\",\n \"QueueArn\": \"arn:aws:sqs:us-east-1:123456789012:s3-events\",\n \"Events\": [\"s3:ObjectCreated:*\"]\n }\n ]\n}\n```\n\n## Useful Queries\n\n```bash\n# Get bucket size\naws s3 ls s3://my-bucket --recursive --summarize | tail -2\n\n# Find largest objects\naws s3api list-objects-v2 \\\n --bucket my-bucket \\\n --query 'sort_by(Contents, &Size)[-10:].{Key:Key,SizeMB:Size}' \\\n --output table\n\n# Count objects by prefix\naws s3api list-objects-v2 \\\n --bucket my-bucket \\\n --prefix \"logs/\" \\\n --query 'length(Contents)'\n\n# Get storage class distribution\naws s3api list-objects-v2 \\\n --bucket my-bucket \\\n --query 'Contents[].StorageClass' | sort | uniq -c\n\n# Find objects modified in last 24 hours\naws s3api list-objects-v2 \\\n --bucket my-bucket \\\n --query \"Contents[?LastModified>='$(date -u -d '1 day ago' '+%Y-%m-%dT%H:%M:%S')'].Key\"\n```\n\n## Performance Optimization\n\n| Technique | When to Use |\n|:----------|:------------|\n| **Transfer Acceleration** | Large files over long distances |\n| **Multipart Upload** | Files > 100MB (automatic in `aws s3 cp`) |\n| **Byte-range fetches** | Parallel downloads of large files |\n| **S3 Select** | Query subset of data without downloading |\n| **Prefix distribution** | Millions of objects, add random prefix |\n\n```bash\n# Parallel uploads with concurrent requests\naws configure set default.s3.max_concurrent_requests 20\naws configure set default.s3.multipart_chunksize 64MB\n\n# Upload with higher concurrency\naws s3 sync ./large-dataset s3://my-bucket/ --only-show-errors\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Block public access** | Enable account-wide public access block |\n| **Encryption** | Use SSE-S3 or SSE-KMS for all buckets |\n| **Versioning** | Enable for critical data buckets |\n| **Lifecycle policies** | Transition to cheaper storage classes |\n| **HTTPS only** | Use bucket policy to deny non-SSL requests |\n| **VPC endpoints** | Access S3 without internet for private subnets |\n| **Object Lock** | Use for compliance/regulatory requirements |\n| **Bucket naming** | Use DNS-compliant names, avoid periods |\n| **Logging** | Enable access logging for auditing |\n| **Replication** | Use CRR for disaster recovery |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":17984,"content_sha256":"b10654c95279764292f6ec2bcaf1c0e1dbc6aeb4c8654ee03c9a250761c62e0d"},{"filename":"references/setup.md","content":"# Installation & Configuration\n\n## Contents\n\n- [Installation](#installation)\n- [Configuration](#configuration)\n- [CLI v2 Features](#cli-v2-features)\n- [Troubleshooting](#troubleshooting)\n- [Best Practices](#best-practices)\n\n---\n\n## Installation\n\nAWS CLI v2 is a self-contained bundle with its own Python runtime, eliminating dependency conflicts.\n\n### Windows\n\n**MSI Installer (Recommended):**\n```powershell\n# Download and install (admin required)\nmsiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi\n\n# Silent install\nmsiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi /qn\n\n# Update existing installation\nmsiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi /qn REINSTALL=ALL REINSTALLMODE=omus\n```\n\n**Winget:**\n```powershell\nwinget install -e --id Amazon.AWSCLI\nwinget upgrade Amazon.AWSCLI\n```\n\n**Chocolatey:**\n```powershell\nchoco install awscli\nchoco upgrade awscli\n```\n\n### macOS\n\n**PKG Installer (All Users):**\n```bash\n# Download and install\ncurl \"https://awscli.amazonaws.com/AWSCLIV2.pkg\" -o \"AWSCLIV2.pkg\"\nsudo installer -pkg AWSCLIV2.pkg -target /\n\n# Verify\nwhich aws\naws --version\n```\n\n**Homebrew:**\n```bash\nbrew install awscli\nbrew upgrade awscli\n```\n\n### Linux\n\n**Bundled Installer (Recommended):**\n```bash\n# x86_64\ncurl \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"\nunzip awscliv2.zip\nsudo ./aws/install\n\n# ARM64 (Graviton)\ncurl \"https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip\" -o \"awscliv2.zip\"\nunzip awscliv2.zip\nsudo ./aws/install\n\n# Update existing installation\nsudo ./aws/install --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli --update\n\n# Custom install location (no sudo)\n./aws/install --install-dir ~/aws-cli --bin-dir ~/bin\n```\n\n**Snap (Auto-updating):**\n```bash\nsudo snap install aws-cli --classic\n```\n\n**Docker:**\n```bash\ndocker run --rm -it amazon/aws-cli --version\ndocker run --rm -it -v ~/.aws:/root/.aws amazon/aws-cli s3 ls\n```\n\n### Verify Installation\n```bash\naws --version\n# aws-cli/2.x.x Python/3.x.x Linux/x86_64 source/x86_64.amzn.2\n```\n\n## Configuration\n\n### Interactive Setup\n\n```bash\n# Basic configuration (access keys)\naws configure\n# AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE\n# AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n# Default region name [None]: us-west-2\n# Default output format [None]: json\n\n# Named profile\naws configure --profile production\n\n# IAM Identity Center (SSO) - Recommended for organizations\naws configure sso\n# SSO session name: my-sso\n# SSO start URL: https://my-sso-portal.awsapps.com/start\n# SSO region: us-east-1\n# SSO registration scopes: sso:account:access\n\n# View current configuration\naws configure list\naws configure list --profile production\n```\n\n### Configuration Files\n\n**~/.aws/credentials** (access keys only):\n```ini\n[default]\naws_access_key_id = AKIAIOSFODNN7EXAMPLE\naws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n\n[production]\naws_access_key_id = AKIAI44QH8DHBEXAMPLE\naws_secret_access_key = je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY\n```\n\n**~/.aws/config** (settings and role configurations):\n```ini\n[default]\nregion = us-west-2\noutput = json\ncli_pager =\n\n[profile production]\nregion = us-east-1\noutput = text\n\n[profile cross-account]\nrole_arn = arn:aws:iam::123456789012:role/AdminRole\nsource_profile = default\nrole_session_name = MySession\nduration_seconds = 3600\n\n[profile mfa-protected]\nrole_arn = arn:aws:iam::123456789012:role/MFARole\nsource_profile = default\nmfa_serial = arn:aws:iam::111111111111:mfa/myuser\n\n[profile sso-dev]\nsso_session = my-sso\nsso_account_id = 123456789012\nsso_role_name = PowerUserAccess\nregion = us-east-1\n\n[sso-session my-sso]\nsso_start_url = https://my-sso-portal.awsapps.com/start\nsso_region = us-east-1\nsso_registration_scopes = sso:account:access\n```\n\n### Environment Variables\n\n| Variable | Description | Example |\n|:---------|:------------|:--------|\n| `AWS_ACCESS_KEY_ID` | Access key | `AKIAIOSFODNN7EXAMPLE` |\n| `AWS_SECRET_ACCESS_KEY` | Secret key | `wJalrXUtnFEMI/...` |\n| `AWS_SESSION_TOKEN` | Session token (temp creds) | From STS |\n| `AWS_PROFILE` | Named profile to use | `production` |\n| `AWS_REGION` | AWS region | `us-west-2` |\n| `AWS_DEFAULT_REGION` | Fallback region | `us-east-1` |\n| `AWS_DEFAULT_OUTPUT` | Output format | `json`, `text`, `table` |\n| `AWS_PAGER` | Pager program | `\"\"` (disable) |\n| `AWS_CONFIG_FILE` | Config file path | `~/.aws/config` |\n| `AWS_SHARED_CREDENTIALS_FILE` | Credentials path | `~/.aws/credentials` |\n| `AWS_CA_BUNDLE` | CA certificate bundle | `/path/to/cert.pem` |\n| `AWS_RETRY_MODE` | Retry behavior | `standard`, `adaptive` |\n| `AWS_MAX_ATTEMPTS` | Max retry attempts | `5` |\n\n### Credential Precedence\n\n1. Command-line options (`--profile`)\n2. Environment variables (`AWS_ACCESS_KEY_ID`, etc.)\n3. Web identity token file (EKS IRSA)\n4. SSO credentials (`sso_*` config)\n5. Credentials file (`~/.aws/credentials`)\n6. Config file credential_process\n7. Container credentials (ECS task role)\n8. Instance metadata (EC2 instance profile)\n\n## CLI v2 Features\n\n### Auto-Prompt Mode\nInteractive parameter completion for discovering commands:\n```bash\n# Enable for single command\naws dynamodb --cli-auto-prompt\n\n# Enable globally\naws configure set cli_auto_prompt on\n\n# Disable paging during auto-prompt\naws configure set cli_pager \"\"\n```\n\n### Output Formats\n```bash\naws ec2 describe-instances --output json # Default, machine-parseable\naws ec2 describe-instances --output text # Tab-separated (grep/awk)\naws ec2 describe-instances --output table # Human-readable ASCII\naws ec2 describe-instances --output yaml # YAML (v2 only)\naws ec2 describe-instances --output yaml-stream # Streaming YAML\n```\n\n### SSO Integration\n```bash\n# Configure SSO\naws configure sso\n\n# Login (opens browser)\naws sso login --profile sso-dev\n\n# Logout\naws sso logout\n\n# List available accounts/roles\naws sso list-accounts\naws sso list-account-roles --account-id 123456789012\n```\n\n### Wizards\nInteractive configuration for complex setups:\n```bash\naws configure wizard # General configuration wizard\n```\n\n## Troubleshooting\n\n### Common Issues\n\n| Issue | Symptom | Solution |\n|:------|:--------|:---------|\n| **Command not found** | `aws: command not found` | Add to PATH: `/usr/local/bin` (Linux/Mac), `C:\\Program Files\\Amazon\\AWSCLIV2\\` (Windows) |\n| **Invalid credentials** | `InvalidClientTokenId` | Run `aws configure` or check `~/.aws/credentials` |\n| **Expired SSO token** | `Token has expired` | Run `aws sso login --profile \u003cname>` |\n| **Clock skew** | `SignatureDoesNotMatch` | Sync system clock: `sudo ntpdate pool.ntp.org` |\n| **SSL certificate error** | `SSL: CERTIFICATE_VERIFY_FAILED` | Set `AWS_CA_BUNDLE` or update CA certs |\n| **Permission denied** | `AccessDenied` | Check IAM policy with `aws sts get-caller-identity` |\n| **Region not set** | `You must specify a region` | Set `AWS_REGION` or use `--region` |\n| **Profile not found** | `Profile not found` | Check `~/.aws/config` syntax, use `[profile name]` |\n| **MFA required** | `AccessDenied` with MFA | Use `--serial-number` and `--token-code` with assume-role |\n| **Throttling** | `Throttling` or `Rate exceeded` | Add retry logic, use exponential backoff |\n\n### Debugging\n```bash\n# Enable debug output\naws s3 ls --debug\n\n# Debug specific request\naws s3 ls --debug 2>&1 | grep -i \"x-amz\"\n\n# Check configured values\naws configure list\n\n# Verify identity\naws sts get-caller-identity\n\n# Test specific profile\naws sts get-caller-identity --profile production\n```\n\n### Credential Troubleshooting\n```bash\n# Check which credentials are being used\naws configure list\n# Shows: profile, access_key, secret_key, region (and source)\n\n# Verify identity\naws sts get-caller-identity\n# Returns: Account, UserId, Arn\n\n# Clear cached SSO credentials\nrm -rf ~/.aws/sso/cache/*\n\n# Check environment overrides\nenv | grep AWS_\n```\n\n### Network Issues\n```bash\n# Test connectivity\naws s3 ls --debug 2>&1 | grep -i \"establishing\"\n\n# Use proxy\nexport HTTP_PROXY=http://proxy.example.com:8080\nexport HTTPS_PROXY=http://proxy.example.com:8080\n\n# Disable SSL verification (not recommended for production)\naws s3 ls --no-verify-ssl\n\n# Custom CA bundle\nexport AWS_CA_BUNDLE=/path/to/ca-bundle.crt\n```\n\n### Version Conflicts (v1 vs v2)\n```bash\n# Check installed version\naws --version\n\n# Check path\nwhich aws\ntype aws\n\n# Remove old v1 (pip-installed)\npip uninstall awscli\n\n# Ensure v2 is in PATH first\nexport PATH=\"/usr/local/bin:$PATH\"\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **Named profiles** | Use profiles for different accounts/roles |\n| **Disable pager** | Set `AWS_PAGER=\"\"` in scripts |\n| **Explicit regions** | Always specify region in automation |\n| **Credential rotation** | Rotate access keys every 90 days if using them |\n\nSee [SKILL.md Best Practices](../SKILL.md#best-practices) for security recommendations including SSO, MFA, and roles.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":8895,"content_sha256":"a834c06cfcca3b61cd2bc81e60c88b764a796b223a9b87942fd3a3a621ad4345"},{"filename":"references/vpc-networking.md","content":"# VPC & Networking\n\n## VPC Management\n\n### Create VPC\n```bash\n# Create VPC with DNS support\naws ec2 create-vpc \\\n --cidr-block 10.0.0.0/16 \\\n --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=production-vpc}]'\n\n# Enable DNS hostnames\naws ec2 modify-vpc-attribute \\\n --vpc-id vpc-123 \\\n --enable-dns-hostnames\n\n# Enable DNS support\naws ec2 modify-vpc-attribute \\\n --vpc-id vpc-123 \\\n --enable-dns-support\n```\n\n### Subnets\n```bash\n# Create public subnet\naws ec2 create-subnet \\\n --vpc-id vpc-123 \\\n --cidr-block 10.0.1.0/24 \\\n --availability-zone us-east-1a \\\n --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=public-1a}]'\n\n# Create private subnet\naws ec2 create-subnet \\\n --vpc-id vpc-123 \\\n --cidr-block 10.0.10.0/24 \\\n --availability-zone us-east-1a \\\n --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=private-1a}]'\n\n# Enable auto-assign public IP (public subnets)\naws ec2 modify-subnet-attribute \\\n --subnet-id subnet-public \\\n --map-public-ip-on-launch\n\n# List subnets\naws ec2 describe-subnets \\\n --filters Name=vpc-id,Values=vpc-123 \\\n --query 'Subnets[*].{ID:SubnetId,CIDR:CidrBlock,AZ:AvailabilityZone,Name:Tags[?Key==`Name`].Value|[0]}'\n```\n\n### Internet Gateway\n```bash\n# Create IGW\naws ec2 create-internet-gateway \\\n --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=production-igw}]'\n\n# Attach to VPC\naws ec2 attach-internet-gateway \\\n --vpc-id vpc-123 \\\n --internet-gateway-id igw-123\n\n# Detach IGW\naws ec2 detach-internet-gateway \\\n --vpc-id vpc-123 \\\n --internet-gateway-id igw-123\n```\n\n### NAT Gateway\n```bash\n# Allocate Elastic IP\nEIP_ALLOC=$(aws ec2 allocate-address \\\n --domain vpc \\\n --query 'AllocationId' \\\n --output text)\n\n# Create NAT Gateway (public subnet)\naws ec2 create-nat-gateway \\\n --subnet-id subnet-public-1a \\\n --allocation-id $EIP_ALLOC \\\n --tag-specifications 'ResourceType=natgateway,Tags=[{Key=Name,Value=nat-1a}]'\n\n# Wait for NAT Gateway\naws ec2 wait nat-gateway-available \\\n --nat-gateway-ids nat-123\n\n# Create private NAT Gateway (VPC-to-VPC traffic)\naws ec2 create-nat-gateway \\\n --subnet-id subnet-private-1a \\\n --connectivity-type private \\\n --tag-specifications 'ResourceType=natgateway,Tags=[{Key=Name,Value=private-nat}]'\n\n# Delete NAT Gateway\naws ec2 delete-nat-gateway --nat-gateway-id nat-123\n```\n\n### Route Tables\n```bash\n# Create route table\naws ec2 create-route-table \\\n --vpc-id vpc-123 \\\n --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=public-rt}]'\n\n# Add route to IGW (public)\naws ec2 create-route \\\n --route-table-id rtb-public \\\n --destination-cidr-block 0.0.0.0/0 \\\n --gateway-id igw-123\n\n# Add route to NAT Gateway (private)\naws ec2 create-route \\\n --route-table-id rtb-private \\\n --destination-cidr-block 0.0.0.0/0 \\\n --nat-gateway-id nat-123\n\n# Associate route table with subnet\naws ec2 associate-route-table \\\n --route-table-id rtb-public \\\n --subnet-id subnet-public-1a\n\n# Replace route\naws ec2 replace-route \\\n --route-table-id rtb-private \\\n --destination-cidr-block 0.0.0.0/0 \\\n --nat-gateway-id nat-456\n\n# Delete route\naws ec2 delete-route \\\n --route-table-id rtb-123 \\\n --destination-cidr-block 10.1.0.0/16\n```\n\n## Security Groups\n\n### Create and Configure\n```bash\n# Create security group\naws ec2 create-security-group \\\n --group-name web-servers \\\n --description \"Allow HTTP/HTTPS traffic\" \\\n --vpc-id vpc-123\n\n# Allow inbound HTTPS\naws ec2 authorize-security-group-ingress \\\n --group-id sg-web \\\n --protocol tcp \\\n --port 443 \\\n --cidr 0.0.0.0/0\n\n# Allow inbound from CIDR range\naws ec2 authorize-security-group-ingress \\\n --group-id sg-web \\\n --protocol tcp \\\n --port 22 \\\n --cidr 10.0.0.0/8\n\n# Allow from another security group\naws ec2 authorize-security-group-ingress \\\n --group-id sg-db \\\n --protocol tcp \\\n --port 5432 \\\n --source-group sg-web\n\n# Allow all traffic from security group\naws ec2 authorize-security-group-ingress \\\n --group-id sg-internal \\\n --protocol -1 \\\n --source-group sg-internal\n\n# Allow port range\naws ec2 authorize-security-group-ingress \\\n --group-id sg-app \\\n --protocol tcp \\\n --port 8080-8090 \\\n --cidr 10.0.0.0/16\n```\n\n### Using IP Permissions JSON\n```bash\n# Complex ingress rules\naws ec2 authorize-security-group-ingress \\\n --group-id sg-web \\\n --ip-permissions '[\n {\n \"IpProtocol\": \"tcp\",\n \"FromPort\": 443,\n \"ToPort\": 443,\n \"IpRanges\": [{\"CidrIp\": \"0.0.0.0/0\", \"Description\": \"HTTPS from anywhere\"}]\n },\n {\n \"IpProtocol\": \"tcp\",\n \"FromPort\": 80,\n \"ToPort\": 80,\n \"IpRanges\": [{\"CidrIp\": \"0.0.0.0/0\", \"Description\": \"HTTP redirect\"}]\n }\n ]'\n\n# Egress rules\naws ec2 authorize-security-group-egress \\\n --group-id sg-app \\\n --ip-permissions '[\n {\n \"IpProtocol\": \"tcp\",\n \"FromPort\": 443,\n \"ToPort\": 443,\n \"IpRanges\": [{\"CidrIp\": \"0.0.0.0/0\"}]\n },\n {\n \"IpProtocol\": \"tcp\",\n \"FromPort\": 5432,\n \"ToPort\": 5432,\n \"UserIdGroupPairs\": [{\"GroupId\": \"sg-db\"}]\n }\n ]'\n```\n\n### Revoke Rules\n```bash\n# Revoke ingress\naws ec2 revoke-security-group-ingress \\\n --group-id sg-web \\\n --protocol tcp \\\n --port 22 \\\n --cidr 0.0.0.0/0\n\n# Revoke from security group\naws ec2 revoke-security-group-ingress \\\n --group-id sg-db \\\n --protocol tcp \\\n --port 5432 \\\n --source-group sg-old-app\n```\n\n## Network ACLs\n\n### Create and Configure\n```bash\n# Create NACL\naws ec2 create-network-acl \\\n --vpc-id vpc-123 \\\n --tag-specifications 'ResourceType=network-acl,Tags=[{Key=Name,Value=public-nacl}]'\n\n# Add inbound rule (allow HTTPS)\naws ec2 create-network-acl-entry \\\n --network-acl-id acl-123 \\\n --rule-number 100 \\\n --protocol 6 \\\n --port-range From=443,To=443 \\\n --cidr-block 0.0.0.0/0 \\\n --ingress \\\n --rule-action allow\n\n# Add outbound rule (ephemeral ports)\naws ec2 create-network-acl-entry \\\n --network-acl-id acl-123 \\\n --rule-number 100 \\\n --protocol 6 \\\n --port-range From=1024,To=65535 \\\n --cidr-block 0.0.0.0/0 \\\n --egress \\\n --rule-action allow\n\n# Associate with subnet\naws ec2 replace-network-acl-association \\\n --association-id aclassoc-123 \\\n --network-acl-id acl-new\n```\n\n## VPC Endpoints\n\n### Gateway Endpoints (S3, DynamoDB)\n```bash\n# Create S3 gateway endpoint\naws ec2 create-vpc-endpoint \\\n --vpc-id vpc-123 \\\n --service-name com.amazonaws.us-east-1.s3 \\\n --route-table-ids rtb-private-1a rtb-private-1b \\\n --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=s3-endpoint}]'\n\n# Create DynamoDB gateway endpoint\naws ec2 create-vpc-endpoint \\\n --vpc-id vpc-123 \\\n --service-name com.amazonaws.us-east-1.dynamodb \\\n --route-table-ids rtb-private-1a rtb-private-1b\n\n# Add endpoint policy\naws ec2 modify-vpc-endpoint \\\n --vpc-endpoint-id vpce-123 \\\n --policy-document '{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": \"*\",\n \"Action\": \"s3:*\",\n \"Resource\": [\n \"arn:aws:s3:::my-bucket\",\n \"arn:aws:s3:::my-bucket/*\"\n ]\n }]\n }'\n```\n\n### Interface Endpoints (PrivateLink)\n```bash\n# Create interface endpoint (ECR)\naws ec2 create-vpc-endpoint \\\n --vpc-id vpc-123 \\\n --vpc-endpoint-type Interface \\\n --service-name com.amazonaws.us-east-1.ecr.api \\\n --subnet-ids subnet-private-1a subnet-private-1b \\\n --security-group-ids sg-endpoints \\\n --private-dns-enabled\n\n# Create SSM endpoints (all 3 required)\nfor svc in ssm ssmmessages ec2messages; do\n aws ec2 create-vpc-endpoint \\\n --vpc-id vpc-123 \\\n --vpc-endpoint-type Interface \\\n --service-name com.amazonaws.us-east-1.$svc \\\n --subnet-ids subnet-private-1a \\\n --security-group-ids sg-endpoints \\\n --private-dns-enabled\ndone\n\n# Create Secrets Manager endpoint\naws ec2 create-vpc-endpoint \\\n --vpc-id vpc-123 \\\n --vpc-endpoint-type Interface \\\n --service-name com.amazonaws.us-east-1.secretsmanager \\\n --subnet-ids subnet-private-1a subnet-private-1b \\\n --security-group-ids sg-endpoints \\\n --private-dns-enabled\n\n# List endpoint services\naws ec2 describe-vpc-endpoint-services \\\n --query 'ServiceNames[?contains(@, `secretsmanager`)]'\n```\n\n### Manage Endpoints\n```bash\n# List endpoints\naws ec2 describe-vpc-endpoints \\\n --filters Name=vpc-id,Values=vpc-123 \\\n --query 'VpcEndpoints[*].{ID:VpcEndpointId,Service:ServiceName,Type:VpcEndpointType}'\n\n# Modify endpoint (add subnets)\naws ec2 modify-vpc-endpoint \\\n --vpc-endpoint-id vpce-123 \\\n --add-subnet-ids subnet-private-1c\n\n# Delete endpoint\naws ec2 delete-vpc-endpoints \\\n --vpc-endpoint-ids vpce-123\n```\n\n## VPC Peering\n\n### Create Peering\n```bash\n# Request peering (same account)\naws ec2 create-vpc-peering-connection \\\n --vpc-id vpc-requester \\\n --peer-vpc-id vpc-accepter \\\n --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Name,Value=vpc1-to-vpc2}]'\n\n# Request peering (cross-account)\naws ec2 create-vpc-peering-connection \\\n --vpc-id vpc-123 \\\n --peer-vpc-id vpc-456 \\\n --peer-owner-id 987654321098 \\\n --peer-region us-west-2\n\n# Accept peering\naws ec2 accept-vpc-peering-connection \\\n --vpc-peering-connection-id pcx-123\n\n# Add routes for peering\naws ec2 create-route \\\n --route-table-id rtb-vpc1 \\\n --destination-cidr-block 10.1.0.0/16 \\\n --vpc-peering-connection-id pcx-123\n\naws ec2 create-route \\\n --route-table-id rtb-vpc2 \\\n --destination-cidr-block 10.0.0.0/16 \\\n --vpc-peering-connection-id pcx-123\n\n# Describe peering\naws ec2 describe-vpc-peering-connections \\\n --vpc-peering-connection-ids pcx-123\n\n# Delete peering\naws ec2 delete-vpc-peering-connection \\\n --vpc-peering-connection-id pcx-123\n```\n\n## Transit Gateway\n\n### Create Transit Gateway\n```bash\n# Create Transit Gateway\naws ec2 create-transit-gateway \\\n --description \"Central hub for VPC connectivity\" \\\n --options '{\n \"AmazonSideAsn\": 64512,\n \"AutoAcceptSharedAttachments\": \"enable\",\n \"DefaultRouteTableAssociation\": \"enable\",\n \"DefaultRouteTablePropagation\": \"enable\",\n \"VpnEcmpSupport\": \"enable\",\n \"DnsSupport\": \"enable\"\n }' \\\n --tag-specifications 'ResourceType=transit-gateway,Tags=[{Key=Name,Value=central-tgw}]'\n\n# Wait for TGW to be available\naws ec2 describe-transit-gateways \\\n --transit-gateway-ids tgw-123 \\\n --query 'TransitGateways[0].State'\n```\n\n### Attach VPCs\n```bash\n# Create VPC attachment\naws ec2 create-transit-gateway-vpc-attachment \\\n --transit-gateway-id tgw-123 \\\n --vpc-id vpc-prod \\\n --subnet-ids subnet-private-1a subnet-private-1b \\\n --tag-specifications 'ResourceType=transit-gateway-attachment,Tags=[{Key=Name,Value=prod-vpc}]'\n\n# Create attachment with appliance mode (for firewalls)\naws ec2 create-transit-gateway-vpc-attachment \\\n --transit-gateway-id tgw-123 \\\n --vpc-id vpc-security \\\n --subnet-ids subnet-fw-1a subnet-fw-1b \\\n --options ApplianceModeSupport=enable\n\n# List attachments\naws ec2 describe-transit-gateway-vpc-attachments \\\n --filters Name=transit-gateway-id,Values=tgw-123\n```\n\n### Route Tables\n```bash\n# Create route table\naws ec2 create-transit-gateway-route-table \\\n --transit-gateway-id tgw-123 \\\n --tag-specifications 'ResourceType=transit-gateway-route-table,Tags=[{Key=Name,Value=prod-rt}]'\n\n# Associate attachment with route table\naws ec2 associate-transit-gateway-route-table \\\n --transit-gateway-route-table-id tgw-rtb-123 \\\n --transit-gateway-attachment-id tgw-attach-vpc1\n\n# Enable propagation\naws ec2 enable-transit-gateway-route-table-propagation \\\n --transit-gateway-route-table-id tgw-rtb-123 \\\n --transit-gateway-attachment-id tgw-attach-vpc1\n\n# Create static route\naws ec2 create-transit-gateway-route \\\n --transit-gateway-route-table-id tgw-rtb-123 \\\n --destination-cidr-block 10.0.0.0/8 \\\n --transit-gateway-attachment-id tgw-attach-vpc1\n\n# Create blackhole route\naws ec2 create-transit-gateway-route \\\n --transit-gateway-route-table-id tgw-rtb-123 \\\n --destination-cidr-block 192.168.0.0/16 \\\n --blackhole\n\n# Get routes\naws ec2 search-transit-gateway-routes \\\n --transit-gateway-route-table-id tgw-rtb-123 \\\n --filters Name=type,Values=static,propagated\n```\n\n### Cross-Account Sharing\n```bash\n# Share via RAM (Resource Access Manager)\naws ram create-resource-share \\\n --name \"TGW-Share\" \\\n --resource-arns arn:aws:ec2:us-east-1:123456789012:transit-gateway/tgw-123 \\\n --principals 987654321098\n\n# Accept in target account\naws ram accept-resource-share-invitation \\\n --resource-share-invitation-arn arn:aws:ram:us-east-1:123456789012:resource-share-invitation/abc123\n```\n\n## VPC Flow Logs\n\n### Enable Flow Logs\n```bash\n# Flow logs to CloudWatch\naws ec2 create-flow-logs \\\n --resource-type VPC \\\n --resource-ids vpc-123 \\\n --traffic-type ALL \\\n --log-destination-type cloud-watch-logs \\\n --log-group-name /aws/vpc/flow-logs \\\n --deliver-logs-permission-arn arn:aws:iam::123456789012:role/FlowLogsRole \\\n --tag-specifications 'ResourceType=vpc-flow-log,Tags=[{Key=Name,Value=vpc-flow-logs}]'\n\n# Flow logs to S3\naws ec2 create-flow-logs \\\n --resource-type VPC \\\n --resource-ids vpc-123 \\\n --traffic-type ALL \\\n --log-destination-type s3 \\\n --log-destination arn:aws:s3:::my-flow-logs-bucket/vpc-logs/ \\\n --log-format '${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}'\n\n# Subnet-level flow logs\naws ec2 create-flow-logs \\\n --resource-type Subnet \\\n --resource-ids subnet-123 \\\n --traffic-type REJECT \\\n --log-destination-type s3 \\\n --log-destination arn:aws:s3:::my-flow-logs-bucket/rejected/\n\n# ENI-level flow logs\naws ec2 create-flow-logs \\\n --resource-type NetworkInterface \\\n --resource-ids eni-123 \\\n --traffic-type ALL \\\n --log-destination-type cloud-watch-logs \\\n --log-group-name /aws/vpc/eni-logs\n```\n\n### Manage Flow Logs\n```bash\n# List flow logs\naws ec2 describe-flow-logs \\\n --filter Name=resource-id,Values=vpc-123\n\n# Delete flow logs\naws ec2 delete-flow-logs \\\n --flow-log-ids fl-123\n```\n\n## Useful Queries\n\n```bash\n# List all VPCs\naws ec2 describe-vpcs \\\n --query 'Vpcs[*].{ID:VpcId,CIDR:CidrBlock,Name:Tags[?Key==`Name`].Value|[0]}'\n\n# Find subnets by type\naws ec2 describe-subnets \\\n --filters Name=tag:Name,Values=*private* \\\n --query 'Subnets[*].{ID:SubnetId,AZ:AvailabilityZone,CIDR:CidrBlock}'\n\n# List security group rules\naws ec2 describe-security-groups \\\n --group-ids sg-123 \\\n --query 'SecurityGroups[0].IpPermissions'\n\n# Find unused security groups\naws ec2 describe-network-interfaces \\\n --query 'NetworkInterfaces[*].Groups[*].GroupId' \\\n --output text | tr '\\t' '\\n' | sort -u > used-sgs.txt\naws ec2 describe-security-groups \\\n --query 'SecurityGroups[*].GroupId' \\\n --output text | tr '\\t' '\\n' | sort -u > all-sgs.txt\ncomm -23 all-sgs.txt used-sgs.txt\n\n# List NAT Gateway status\naws ec2 describe-nat-gateways \\\n --query 'NatGateways[*].{ID:NatGatewayId,State:State,Subnet:SubnetId}'\n\n# Get Transit Gateway routes\naws ec2 search-transit-gateway-routes \\\n --transit-gateway-route-table-id tgw-rtb-123 \\\n --filters Name=state,Values=active \\\n --query 'Routes[*].{CIDR:DestinationCidrBlock,Attachment:TransitGatewayAttachments[0].TransitGatewayAttachmentId}'\n```\n\n## Best Practices\n\n| Practice | Description |\n|:---------|:------------|\n| **CIDR planning** | Use /16 for VPCs, leave room for growth |\n| **Multi-AZ** | Deploy subnets across 3 AZs for HA |\n| **Security groups** | Reference other SGs instead of CIDRs for internal traffic |\n| **NAT Gateway** | Deploy one per AZ for HA |\n| **VPC endpoints** | Use for S3/DynamoDB/ECR to reduce NAT costs |\n| **Transit Gateway** | Use for hub-and-spoke with 3+ VPCs |\n| **Flow logs** | Enable for security and troubleshooting |\n| **Private subnets** | Keep databases and internal services private |\n| **NACLs** | Use sparingly, prefer security groups |\n| **DNS** | Enable DNS hostnames for service discovery |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":16439,"content_sha256":"b3db8d6437d148dbb82fea4022d97702e252bdf26b32f6c18cccd54b1a2bbd9d"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"AWS CLI v2 Quick Reference","type":"text"}]},{"type":"paragraph","content":[{"text":"A unified tool to manage AWS services from the terminal. This guide focuses on CLI v2 features, practical examples, and advanced patterns for experienced developers.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Quick Start","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Verify installation and version\naws --version\n\n# Interactive configuration\naws configure # Access keys + region + output format\naws configure sso # IAM Identity Center (SSO) - recommended\n\n# Verify identity\naws sts get-caller-identity # Shows Account, UserId, ARN\n\n# Enable auto-prompt for command discovery\naws dynamodb --cli-auto-prompt","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Power User Tips","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# See all waiter commands for a service\naws ec2 wait help\n\n# Generate command skeleton (fill in the blanks)\naws lambda create-function --generate-cli-skeleton > create-fn.json\n\n# Create CLI alias for common commands\naws configure set cli_alias.whoami \"sts get-caller-identity\"\naws whoami # Now works!\n\n# Disable pager for scripting\nexport AWS_PAGER=\"\"","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text"},{"text":"Advanced Patterns","type":"text","marks":[{"type":"link","attrs":{"href":"references/advanced-patterns.md","title":null}}]},{"text":" for JMESPath mastery and automation tricks.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Global Options","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Flag","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Description","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"--profile NAME","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Use named profile from ","type":"text"},{"text":"~/.aws/credentials","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"--region REGION","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Override default region (e.g., ","type":"text"},{"text":"us-east-1","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"--output FORMAT","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Output: ","type":"text"},{"text":"json","type":"text","marks":[{"type":"code_inline"}]},{"text":" (default), ","type":"text"},{"text":"text","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"table","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"yaml-stream","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"--query EXPR","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Filter output using JMESPath expressions","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"--no-paginate","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Disable auto-pagination (first page only)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"--dry-run","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Check permissions without executing (EC2, etc.)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"--debug","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Verbose HTTP/API debug logging","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"--cli-auto-prompt","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Interactive parameter completion","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"--no-cli-pager","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Disable output paging","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Decision Trees","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Compute & Containers","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Need compute?\n├── Serverless functions ────────────► Lambda (references/lambda.md)\n├── Docker containers\n│ ├── Managed orchestration ───────► ECS (references/ecs.md)\n│ ├── Kubernetes ──────────────────► EKS (references/eks.md)\n│ └── Container registry ──────────► ECR (references/ecr.md)\n└── Virtual machines ────────────────► EC2 (use aws ec2 commands)","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Data & Storage","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Need data storage?\n├── Object/blob storage ─────────────► S3 (references/s3.md)\n├── NoSQL (key-value/document) ──────► DynamoDB (references/dynamodb.md)\n├── Relational SQL ──────────────────► Aurora/RDS (references/aurora.md)\n├── Data catalog & ETL ──────────────► Glue (references/glue.md)\n└── Data warehouse ──────────────────► Redshift (aws redshift commands)","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Streaming & Messaging","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Need streaming/messaging?\n├── Kafka-compatible ────────────────► MSK (references/msk.md)\n├── Real-time streams ───────────────► Kinesis (references/kinesis.md)\n├── Message queues ──────────────────► SQS (aws sqs commands)\n└── Pub/Sub notifications ───────────► SNS (aws sns commands)","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Security & Access","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Need security/access management?\n├── Users, roles, policies ──────────► IAM (references/iam-security.md)\n├── Secrets & credentials ───────────► Secrets Manager/SSM (references/private-parameters.md)\n├── Private network access ──────────► VPC (references/vpc-networking.md)\n└── Secure tunneling ────────────────► SSM/Bastion (references/bastion-tunneling.md)","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reference File Navigation","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Reference","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Description","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Key Triggers","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Setup","type":"text","marks":[{"type":"link","attrs":{"href":"references/setup.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Installation, configuration, profiles, SSO","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"install","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"configure","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"sso","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"profile","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"IAM & Security","type":"text","marks":[{"type":"link","attrs":{"href":"references/iam-security.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Roles, policies, STS, MFA, cross-account","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"iam","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"role","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"policy","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"sts","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"assume-role","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Lambda","type":"text","marks":[{"type":"link","attrs":{"href":"references/lambda.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Functions, layers, aliases, URLs, events","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"lambda","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"serverless","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"function","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"ECS","type":"text","marks":[{"type":"link","attrs":{"href":"references/ecs.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Clusters, tasks, services, Fargate","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"ecs","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"fargate","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"task","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"container","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"EKS","type":"text","marks":[{"type":"link","attrs":{"href":"references/eks.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Clusters, node groups, kubeconfig, IRSA","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"eks","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"kubernetes","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"kubectl","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"k8s","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"ECR","type":"text","marks":[{"type":"link","attrs":{"href":"references/ecr.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Repositories, auth, scanning, lifecycle","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"ecr","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"registry","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"image","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"S3","type":"text","marks":[{"type":"link","attrs":{"href":"references/s3.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Buckets, objects, sync, presign, lifecycle","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"s3","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"bucket","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"upload","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"sync","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"DynamoDB","type":"text","marks":[{"type":"link","attrs":{"href":"references/dynamodb.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Tables, items, queries, streams, backups","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"dynamodb","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"ddb","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"nosql","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Aurora/RDS","type":"text","marks":[{"type":"link","attrs":{"href":"references/aurora.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Clusters, serverless v2, cloning, blue-green","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"rds","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"aurora","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"mysql","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"postgresql","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Glue","type":"text","marks":[{"type":"link","attrs":{"href":"references/glue.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Catalog, crawlers, ETL jobs, workflows","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"glue","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"etl","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"catalog","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"crawler","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"MSK","type":"text","marks":[{"type":"link","attrs":{"href":"references/msk.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Kafka clusters, serverless, configuration","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"msk","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"kafka","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"streaming","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Kinesis","type":"text","marks":[{"type":"link","attrs":{"href":"references/kinesis.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Data streams, Firehose, consumers","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"kinesis","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"stream","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"firehose","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Secrets & Params","type":"text","marks":[{"type":"link","attrs":{"href":"references/private-parameters.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Parameter Store, Secrets Manager, rotation","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"ssm","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"secrets","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"parameter","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"rotation","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"VPC & Networking","type":"text","marks":[{"type":"link","attrs":{"href":"references/vpc-networking.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"VPCs, subnets, security groups, endpoints","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"vpc","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"subnet","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"security-group","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"endpoint","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Bastion & Tunneling","type":"text","marks":[{"type":"link","attrs":{"href":"references/bastion-tunneling.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"SSM Session Manager, port forwarding","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"bastion","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"tunnel","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"ssm","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"ssh","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"GitHub CI/CD","type":"text","marks":[{"type":"link","attrs":{"href":"references/github-cicd.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"OIDC, GitHub Actions, CodeBuild","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"github","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"actions","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"oidc","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"cicd","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Advanced Patterns","type":"text","marks":[{"type":"link","attrs":{"href":"references/advanced-patterns.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"JMESPath, waiters, skeletons, aliases","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"jmespath","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"query","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"waiter","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"alias","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Environment Variables","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Variable","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Purpose","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Example","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_ACCESS_KEY_ID","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Access key for authentication","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AKIAIOSFODNN7EXAMPLE","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_SECRET_ACCESS_KEY","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Secret key for authentication","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"wJalrXUtnFEMI/...","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_SESSION_TOKEN","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Session token (temporary credentials)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"For STS assume-role","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_PROFILE","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Named profile to use","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"production","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_REGION","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS region for requests","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"us-west-2","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_DEFAULT_OUTPUT","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Default output format","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"json","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"text","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"table","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_PAGER","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Pager program (empty to disable)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"\"\"","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_CONFIG_FILE","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Custom config file path","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"~/.aws/config","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_SHARED_CREDENTIALS_FILE","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Custom credentials file path","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"~/.aws/credentials","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_CA_BUNDLE","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Custom CA certificate bundle","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"/path/to/cert.pem","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AWS_RETRY_MODE","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Retry mode","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"standard","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"adaptive","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Credential Precedence","type":"text"}]},{"type":"paragraph","content":[{"text":"The CLI resolves credentials in this order (first match wins):","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Command-line options","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"--profile","type":"text","marks":[{"type":"code_inline"}]},{"text":", explicit credentials)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Environment variables","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"AWS_ACCESS_KEY_ID","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"AWS_SECRET_ACCESS_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Web identity token","type":"text","marks":[{"type":"strong"}]},{"text":" (EKS IRSA, OIDC)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"SSO credentials","type":"text","marks":[{"type":"strong"}]},{"text":" (IAM Identity Center)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Credentials file","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"~/.aws/credentials","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Config file","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"~/.aws/config","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"credential_process","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Container credentials","type":"text","marks":[{"type":"strong"}]},{"text":" (ECS task role)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Instance metadata","type":"text","marks":[{"type":"strong"}]},{"text":" (EC2 instance profile, IMDSv2)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Common Patterns","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Profile Switching","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Use specific profile for one command\naws s3 ls --profile production\n\n# Set default profile for session\nexport AWS_PROFILE=production\n\n# List configured profiles\naws configure list-profiles","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Output Filtering with JMESPath","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Get specific fields\naws ec2 describe-instances \\\n --query 'Reservations[*].Instances[*].[InstanceId,State.Name]' \\\n --output table\n\n# Filter running instances\naws ec2 describe-instances \\\n --query 'Reservations[*].Instances[?State.Name==`running`].InstanceId' \\\n --output text","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Wait for Resource State","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Wait for instance to be running\naws ec2 wait instance-running --instance-ids i-1234567890abcdef0\n\n# Wait for Lambda function update\naws lambda wait function-updated --function-name my-function","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Best Practices","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Category","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Recommendation","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Security","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"aws configure sso","type":"text","marks":[{"type":"code_inline"}]},{"text":" over long-lived access keys","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Security","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Use IAM roles for compute (EC2/Lambda/ECS) instead of embedded keys","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Security","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Enable MFA for sensitive operations","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Scripting","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--output json","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"--output text","type":"text","marks":[{"type":"code_inline"}]},{"text":" for parsing","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Scripting","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--query","type":"text","marks":[{"type":"code_inline"}]},{"text":" to filter data and reduce output","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Safety","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--dry-run","type":"text","marks":[{"type":"code_inline"}]},{"text":" before destructive operations","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Performance","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--page-size","type":"text","marks":[{"type":"code_inline"}]},{"text":" to control memory on large lists","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Regions","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Explicitly set region in scripts to avoid surprises","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Cost","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Use lifecycle policies (S3/ECR) for automatic cleanup","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Debugging","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--debug","type":"text","marks":[{"type":"code_inline"}]},{"text":" to see raw HTTP requests/responses","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Common Errors Quick Reference","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Error","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Cause","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Fix","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"ExpiredToken","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Session credentials expired","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"aws sso login","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"aws sts get-session-token","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"AccessDenied","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Missing IAM permissions","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Check IAM policy; use ","type":"text"},{"text":"--debug","type":"text","marks":[{"type":"code_inline"}]},{"text":" to see required action","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"InvalidClientTokenId","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Invalid access key","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Verify ","type":"text"},{"text":"AWS_ACCESS_KEY_ID","type":"text","marks":[{"type":"code_inline"}]},{"text":" or run ","type":"text"},{"text":"aws configure","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"UnauthorizedAccess","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Wrong region or account","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Check ","type":"text"},{"text":"--region","type":"text","marks":[{"type":"code_inline"}]},{"text":" flag and ","type":"text"},{"text":"aws sts get-caller-identity","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"ThrottlingException","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"API rate limit exceeded","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Add retry logic with exponential backoff","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"NoCredentialProviders","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"No credentials found","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":"left"},"content":[{"type":"paragraph","content":[{"text":"Check credential chain; run ","type":"text"},{"text":"aws configure list","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"For detailed troubleshooting, see ","type":"text"},{"text":"Setup","type":"text","marks":[{"type":"link","attrs":{"href":"references/setup.md#troubleshooting","title":null}}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"When Not to Use","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"AWS SDK code","type":"text","marks":[{"type":"strong"}]},{"text":" — For boto3, AWS SDK for JavaScript, etc., use programming documentation","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"CloudFormation/Terraform","type":"text","marks":[{"type":"strong"}]},{"text":" — This skill covers CLI commands, not IaC templates","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Console UI steps","type":"text","marks":[{"type":"strong"}]},{"text":" — CLI-focused; use AWS documentation for console walkthroughs","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Pricing/billing","type":"text","marks":[{"type":"strong"}]},{"text":" — Use AWS pricing calculator or Cost Explorer documentation","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Quick Command Reference","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Identity & Access\naws sts get-caller-identity\n# → {\"Account\": \"123456789012\", \"UserId\": \"AIDAEXAMPLE\", \"Arn\": \"arn:aws:iam::123456789012:user/dev\"}\n\naws sts assume-role --role-arn arn:aws:iam::123456789012:role/Admin --role-session-name mysession\n# → {\"Credentials\": {\"AccessKeyId\": \"ASIA...\", \"SecretAccessKey\": \"...\", \"SessionToken\": \"...\"}}\n\n# S3\naws s3 ls\n# → 2024-01-15 bucket-name-1\n# → 2024-02-20 bucket-name-2\n\naws s3 sync ./local s3://bucket/prefix --delete\n\n# Lambda\naws lambda invoke --function-name fn response.json\n# → {\"StatusCode\": 200, \"ExecutedVersion\": \"$LATEST\"}\n\naws lambda update-function-code --function-name fn --zip-file fileb://code.zip\n# → {\"FunctionName\": \"fn\", \"LastModified\": \"2024-12-28T...\", \"State\": \"Active\"}\n\n# ECS\naws ecs list-clusters\n# → {\"clusterArns\": [\"arn:aws:ecs:us-east-1:123456789012:cluster/prod\"]}\n\naws ecs update-service --cluster prod --service api --force-new-deployment\n\n# EKS\naws eks update-kubeconfig --name my-cluster\n# → Added new context arn:aws:eks:us-east-1:123456789012:cluster/my-cluster\n\naws eks list-clusters\n# → {\"clusters\": [\"my-cluster\", \"dev-cluster\"]}\n\n# Secrets\naws secretsmanager get-secret-value --secret-id prod/api/key --query SecretString --output text\n# → sk_live_xxxxxxxxxxxxx\n\naws ssm get-parameter --name /app/prod/db/host --with-decryption --query Parameter.Value --output text\n# → db.example.com\n\n# Debugging\naws ssm start-session --target i-0123456789abcdef0\n# → Starting session with SessionId: user-0a1b2c3d4e5f67890","type":"text"}]}]},"metadata":{"date":"2026-11-02","name":"mastering-aws-cli","author":"@skillopedia","source":{"stars":9,"repo_name":"mastering-aws-cli","origin_url":"https://github.com/SpillwaveSolutions/mastering-aws-cli/blob/HEAD/SKILL.md","repo_owner":"SpillwaveSolutions","body_sha256":"850a9fe8b35e2724383c03dc549f1abdeafcf5718529f121b1dddbc0f56665e2","cluster_key":"93479d0cf34f1556189ed470c35cbf27be095f2cb39a5d211014c9b33486787a","clean_bundle":{"format":"clean-skill-bundle-v1","source":"SpillwaveSolutions/mastering-aws-cli/SKILL.md","text_bytes":243250,"source_commit":"87f407bf571fae95fd920a351e0690199b548ff1","skills_sh_rank":6850,"attachment_count":17,"text_attachments":17,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"devops-infrastructure","category_label":"DevOps"},"exact_dupes_collapsed_into_this":0},"license":"MIT","version":"v1","category":"devops-infrastructure","metadata":{"author":"Spillwave","version":"2.1.0"},"triggers":["aws cli","aws command line","aws commands","ec2","s3","lambda","iam","eks","ecs","ecr","dynamodb","rds","aurora","glue","msk","kinesis","ssm","secrets manager","parameter store","vpc","cloudwatch","sts","assume role","aws configure","aws sso","github actions aws","oidc aws","bastion","ssm tunnel","kubectl eks"],"import_tag":"skills-sh-top20k-2026-11","description":"AWS CLI v2 quick-reference for experienced developers. Covers compute (Lambda, ECS, EKS), storage (S3, DynamoDB, Aurora), networking (VPC, SSM tunneling), security (IAM, Secrets Manager), and GitHub Actions CI/CD. Use when asked to \"write aws commands\", \"debug aws access\", \"set up cross-account roles\", \"configure aws cli\", \"assume role\", \"S3 bucket operations\", or \"deploy to ECS\".\n","allowed-tools":["Read","Bash","WebFetch"]}},"renderedAt":1793671362599}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.