Module 02: Web Servers & Content Delivery Networks (CDN)

User in Tokyo → Origin Server in Virginia (USA)
├─ Distance: 11,000 km (6,800 miles)
├─ Light speed limit: ~7,300 km/ms
├─ Minimum latency: 150ms (physics limit)
├─ Real latency: 300-500ms (routing overhead)
└─ User experience: Website feels slow and laggy

1 million users → 1 origin server
├─ Each server handles max 10,000 requests/second
├─ Traffic spike: 50,000 requests/second
├─ Result: Server crashes, website goes down
└─ Lost revenue: $10,000/minute for e-commerce

260M subscribers streaming from 1 data center in California:
├─ Each 4K stream: 25 Mbps bandwidth
├─ Total bandwidth needed: 6.5 Petabits/second
├─ Cost of that bandwidth: $650 million/month
└─ Result: Impossible and economically unfeasible

260M subscribers streaming from 17,000+ servers in 1,000+ cities:
├─ Tokyo users → Tokyo server (2ms latency)
├─ London users → London server (3ms latency)
├─ São Paulo users → São Paulo server (5ms latency)
├─ Total bandwidth cost: $50 million/month (13× cheaper)
└─ Result: Perfect 4K streaming globally

User Request → Internet (150-500ms) → Origin Server → Internet (150-500ms) → User
├─ Total latency: 300-1000ms
├─ Origin bandwidth: 10 Gbps ($5,000/month)
├─ Requests handled: 10,000/second max
└─ Single point of failure

1. User in Paris requests photo.jpg
2. Paris CDN server: "Don't have it, fetching from origin..."
3. Paris CDN → Virginia origin → Gets photo.jpg
4. Paris CDN: Saves photo.jpg locally (caches it)
5. Returns photo.jpg to user (300ms total)

1. Users in Paris request photo.jpg
2. Paris CDN: "Already have it!"
3. Returns photo.jpg immediately (5ms total)
4. No origin server contact needed
5. Origin saved 9,999 requests

User types: https://www.example.com
    ↓
Browser sends HTTP request to web server
    ↓
Web server processes request:
    1. Parse URL and headers
    2. Check if file exists or needs application processing
    3. Generate or retrieve response
    4. Send HTTP response back to browser
    ↓
Browser receives HTML, CSS, JavaScript, images
    ↓
Browser renders webpage

Free Tier (26M+ websites):
    - Unlimited DDoS protection
    - Free SSL/TLS certificates
    - Global CDN
    - DNS hosting
    Cost to Cloudflare: $1-2/site/month
    Strategy: Scale economies, attract paid upgrades

Pro Tier ($20/month):
    - Advanced performance features
    - Image optimization
    - Mobile optimization
    - 15%+ faster page loads

Business Tier ($200/month):
    - Custom SSL certificates
    - Advanced WAF rules
    - PCI compliance
    - 24/7 email support

Enterprise (Custom pricing, $2K-50K/month):
    - Dedicated support
    - Custom contracts
    - 100% uptime SLA
    - Advanced features (Workers, R2 storage)
    - Customers: Fortune 500, major websites

Apache Prefork MPM (Multi-Processing Module):
    - One process per connection
    - 10,000 concurrent users = 10,000 Apache processes
    - Each process: 5-10 MB RAM
    - Total RAM: 50-100 GB just for connections!
    - Context switching overhead (CPU thrashing)

Problem at Netflix scale:
    - 260M+ subscribers browsing simultaneously
    - 100,000+ concurrent connections per server
    - Apache servers running out of RAM
    - CPU spending 50%+ time context switching
    - Page loads: 1-3 seconds (unacceptable)

NGINX Event-Driven Model:
    - One worker process per CPU core
    - Event loop handles 10,000+ connections per process
    - 100,000 concurrent users = 16 worker processes (16-core server)
    - Each worker: 10-20 MB RAM
    - Total RAM: 200-300 MB for connections (vs 50-100 GB Apache!)
    - No context switching (event loop is efficient)

Results at Netflix:
    - 260M+ subscribers supported with 1/10th servers
    - Server count: 1,000 web servers → 100 (90% reduction)
    - RAM usage: 95% reduction per server
    - Page loads: 200-500ms (5-15x faster)
    - Infrastructure cost: $10M/year savings

# /etc/nginx/nginx.conf
user www-data;
worker_processes auto;  # One per CPU core (typically 16-96 cores)
worker_rlimit_nofile 100000;  # Max file descriptors

events {
    worker_connections 10000;  # Each worker handles 10K connections
    use epoll;  # Linux-specific, most efficient
    multi_accept on;  # Accept multiple connections at once
}

http {
    # Performance optimizations
    sendfile on;  # Zero-copy file transfers (kernel-level)
    tcp_nopush on;  # Send headers in one packet (Nagle's algorithm)
    tcp_nodelay on;  # Don't wait for packets to fill (low latency)
    
    keepalive_timeout 65;  # Keep connections alive (reduce handshakes)
    keepalive_requests 100;  # 100 requests per connection
    
    # Compression
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;  # Balance CPU vs compression ratio
    gzip_types text/plain text/css text/xml text/javascript 
               application/json application/javascript application/xml+rss;
    
    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    
    # Rate limiting (DDoS protection)
    limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
    limit_req zone=one burst=20;  # Allow 20 burst requests, then rate limit
    
    # Upstream application servers
    upstream netflix_api {
        least_conn;  # Send to server with fewest connections
        server 10.0.1.10:8080 weight=1;
        server 10.0.1.11:8080 weight=1;
        server 10.0.1.12:8080 weight=1;
        keepalive 32;  # Connection pool to backends
    }
    
    # Virtual host
    server {
        listen 80;
        listen [::]:80;
        server_name netflix.com www.netflix.com;
        
        # Redirect HTTP to HTTPS
        return 301 https://$server_name$request_uri;
    }
    
    server {
        listen 443 ssl http2;  # HTTP/2 enabled
        listen [::]:443 ssl http2;
        server_name netflix.com www.netflix.com;
        
        # SSL/TLS configuration
        ssl_certificate /etc/nginx/ssl/netflix.crt;
        ssl_certificate_key /etc/nginx/ssl/netflix.key;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
        ssl_prefer_server_ciphers off;
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;
        
        # Static assets (served directly by NGINX)
        location /static/ {
            root /var/www/netflix;
            expires 1y;  # Browser cache for 1 year
            add_header Cache-Control "public, immutable";
        }
        
        # API requests (proxy to application servers)
        location /api/ {
            proxy_pass http://netflix_api;
            proxy_http_version 1.1;
            proxy_set_header Connection "";  # Keep alive to upstream
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            
            # Timeouts
            proxy_connect_timeout 5s;
            proxy_send_timeout 60s;
            proxy_read_timeout 60s;
            
            # Caching (for GET requests)
            proxy_cache_key $scheme$request_method$host$request_uri;
            proxy_cache netflix_cache;
            proxy_cache_valid 200 10m;  # Cache successful responses for 10 minutes
            proxy_cache_use_stale error timeout http_500 http_502 http_503;
        }
        
        # Health check endpoint
        location /health {
            access_log off;  # Don't log health checks
            return 200 "healthy\n";
            add_header Content-Type text/plain;
        }
    }
}

Apache 2.4 (Prefork MPM):
    Requests per second: 12,450
    Latency (avg): 803ms
    Failed requests: 234 (2.3% error rate)
    RAM usage: 8.2 GB
    CPU usage: 85%

Apache 2.4 (Worker MPM - threaded):
    Requests per second: 28,300
    Latency (avg): 353ms
    Failed requests: 89 (0.9% error rate)
    RAM usage: 3.1 GB
    CPU usage: 72%

NGINX:
    Requests per second: 87,650
    Latency (avg): 114ms
    Failed requests: 0 (0% error rate)
    RAM usage: 420 MB
    CPU usage: 45%

Performance Winner: NGINX
    - 3.1x more requests/second vs Apache Worker
    - 7.0x more requests/second vs Apache Prefork
    - 3.1x faster response time
    - 7.4x less RAM usage (vs Worker), 19.5x less (vs Prefork)
    - 40% less CPU usage

Apache Approach:
    New connection arrives
        → Fork new process (or spawn thread)
        → Process handles ONE connection
        → When done, process killed/recycled
    Cost: 5-10 MB RAM per connection, context switching overhead

NGINX Approach:
    New connection arrives
        → Added to event loop
        → Worker process handles 10,000+ connections
        → Non-blocking I/O (async)
    Cost: <1 KB RAM per connection, minimal context switching

Apache (Blocking):
    Worker thread reads file from disk
        → Thread BLOCKS until disk I/O completes
        → Other requests must wait
        → Slow for high concurrency

NGINX (Non-Blocking):
    Worker initiates file read
        → Immediately moves to next request (doesn't wait)
        → Kernel notifies when disk I/O done
        → Worker processes result
    Result: 10,000+ requests in-flight simultaneously

Apache:
    read(file) → copy to userspace → copy to socket
    (2 copies, slower)

NGINX:
    sendfile() system call
        → Kernel copies file directly to network socket
        → Zero-copy (no userspace involvement)
    Result: 3-5x faster for static files

Apache:
    - Per-directory config files (.htaccess)
    - Users can override settings without sudo
    - Great for shared hosting
    - WordPress, Drupal rely on .htaccess

NGINX:
    - No .htaccess support
    - All config in main files (requires sudo)
    - Better security (users can't misconfigure)
    - Requires reload for config changes

Apache:
    - 100+ modules available
    - LoadModule directive (dynamic loading)
    - No recompilation needed

NGINX:
    - Fewer modules (but growing)
    - Some modules require recompilation
    - Commercial version (NGINX Plus) has more dynamic modules

Market Reality:
    - More developers know Apache (25+ years old)
    - More WordPress hosting uses Apache
    - More tutorials/StackOverflow answers for Apache
    - Easier for beginners

NGINX (Reverse Proxy + Static Files)
    ↓
Apache (Dynamic Content + .htaccess)
    ↓
Application (PHP, Python, Node.js)

Benefits:
    - NGINX handles SSL, static files, caching (fast)
    - Apache handles dynamic content (compatible)
    - Best of both worlds

Without CDN:
    User in Tokyo requests image from server in New York
        Distance: 10,900 km (6,775 miles)
        Speed of light in fiber: 200,000 km/s
        Theoretical minimum latency: 54ms ONE WAY
        Actual latency: 150-250ms (routing overhead)
        3-way TCP handshake: 450-750ms
        TLS handshake: 900-1,500ms
        Total before content: 1.35-2.25 seconds!

With CDN:
    User in Tokyo requests image from Tokyo edge server
        Distance: 20 km (12 miles)
        Latency: 5-10ms
        Cached content: 0ms origin time
        Total: 5-10ms (99% faster!)

Images (i.redd.it):
    - TTL: 1 year (immutable URLs)
    - Cache everywhere: RAM + SSD
    - Compression: WebP format (30% smaller than JPEG)
    - Total size: 300+ million images cached

CSS/JavaScript (reddit.com/static/):
    - TTL: 1 year (versioned URLs: bundle.abc123.js)
    - Brotli compression (20% better than gzip)
    - HTTP/2 push: Send CSS before browser requests

Avatars/Thumbnails:
    - TTL: 1 hour (users change profiles)
    - Lazy loading (only fetch visible images)

Post listings (reddit.com/r/programming):
    - TTL: 60 seconds (frequently updated)
    - ESI (Edge Side Includes): Cache page template, fetch fresh posts
    - Personalization at edge (logged-in vs logged-out)

API responses (reddit.com/api/info.json):
    - TTL: 10-30 seconds
    - Vary by auth token (different cache per user)

AWS S3 + CloudFront (Alternative):
    S3 storage: 300M images × 500 KB avg = 150 TB
    S3 cost: 150,000 GB × $0.023/GB = $3,450/month
    
    CloudFront bandwidth: 1.7B pageviews × 5 MB avg = 8.5 PB/month
    CloudFront cost: 8,500 TB × $0.085/GB first 10 TB, then cheaper
    Estimated: $500K/month
    
    Total: $503K/month = $6M/year

Fastly CDN (Actual):
    Traffic-based pricing: ~$0.12/GB (average)
    8.5 PB/month × $0.12/GB = $1.02M/month
    
    Total: $1M/month = $12M/year
    
Wait, that's MORE expensive?!

BUT: Fastly includes:
    - Real-time purging (vs 15-minute CloudFront invalidation)
    - Custom VCL logic (vs limited CloudFront)
    - Real-time logs (vs 6-hour CloudFront delay)
    - Better support (vs AWS support tickets)
    
Reddit's decision: Pay 2x for better features & developer experience
ROI: Faster deployment = more features = more users = $100M+ revenue

User posts offensive content to Reddit
    ↓
Moderator removes post
    ↓
But image cached on CDN for 1 year!
    ↓
Standard CDN: Wait 5-15 minutes for cache invalidation
    ↓
Result: Offensive content still visible (bad user experience)

Moderator clicks "Remove"
    ↓
Reddit API calls Fastly purge endpoint:
        POST /purge/https://i.redd.it/offensive123.jpg
    ↓
Fastly purges from ALL edge servers globally
    ↓
Time to purge: 150-500ms (instant!)
    ↓
Next user request: Cache miss, fetches from origin
    ↓
Origin returns 404 (image deleted)
    ↓
Result: Content removed worldwide in <1 second

# Custom logic at edge (Varnish Configuration Language)

sub vcl_recv {
    # Block bad bots
    if (req.http.User-Agent ~ "BadBot|Scraper") {
        error 403 "Forbidden";
    }
    
    # Normalize Accept-Encoding (better cache hit rate)
    if (req.http.Accept-Encoding) {
        if (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            unset req.http.Accept-Encoding;
        }
    }
    
    # Remove tracking parameters (better caching)
    if (req.url ~ "\?(utm_|fbclid|gclid)") {
        set req.url = regsub(req.url, "\?.*$", "");
    }
    
    # Logged-in users: Don't cache personalized content
    if (req.http.Cookie ~ "reddit_session") {
        return (pass);  # Bypass cache
    }
}

sub vcl_fetch {
    # Cache images for 1 year
    if (beresp.url ~ "\.(jpg|jpeg|png|gif|webp)$") {
        set beresp.ttl = 365d;
        set beresp.http.Cache-Control = "public, max-age=31536000, immutable";
    }
    
    # Cache HTML for 60 seconds
    if (beresp.http.Content-Type ~ "text/html") {
        set beresp.ttl = 60s;
        set beresp.http.Cache-Control = "public, max-age=60";
    }
}

sub vcl_deliver {
    # Add custom header showing cache status
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
    } else {
        set resp.http.X-Cache = "MISS";
    }
}

Akamai Edge Servers:
    - 4,100+ locations (vs Cloudflare 310+, Fastly 73)
    - Even in remote regions (Africa, South America, rural areas)
    - 365,000+ servers (vs Cloudflare ~200K estimated)
    - Average user <25ms from Akamai edge

Result: Better performance in underserved regions

Apple Product Launch Architecture:
    ↓
Apple Media Services (origin servers in US, Europe, Asia)
    ↓
Akamai EdgeSuite (video streaming platform)
    - 365K+ edge servers worldwide
    - Adaptive bitrate streaming (multiple quality levels)
    - Pre-positioning (content cached before event)
    ↓
End User Experience:
    - Automatic quality adjustment (based on bandwidth)
    - <2 second buffering start time
    - 99.9%+ stream success rate

iOS 17 Release Day (September 2023):
    - Available users: 1.2+ billion iPhones worldwide
    - Download size: 6.2 GB average
    - Simultaneous downloaders: 100+ million first day
    - Total data: 620+ petabytes first week

Without CDN (Hypothetical):
    - Apple origin servers: Would need 50,000+ servers
    - Bandwidth cost: $50M+ first week
    - Network congestion: Internet backbone overload
    - User experience: 10+ hour download times

With Akamai CDN:
    - Edge servers: 365K+ worldwide cache iOS update
    - Bandwidth cost: $5M first week (origin to edge once, edge to users many times)
    - Network: Distributed load, no congestion
    - User experience: 20-60 minute downloads (smooth)
    - Apple's cost: $300M/year Akamai contract (estimated)

Base Platform: $5,000-50,000/month minimum
    - Includes basic CDN, SSL, monitoring

Bandwidth Pricing (Volume discounts):
    - 0-10 TB/month: $0.15/GB
    - 10-50 TB/month: $0.12/GB
    - 50-150 TB/month: $0.08/GB
    - 150-500 TB/month: $0.05/GB
    - 500+ TB/month: Custom (Apple: estimated $0.02-0.03/GB)

Apple's Estimated Costs:
    - Bandwidth: 10+ PB/month = $200-300K/month
    - Platform fees: $50K/month
    - Professional services: $100K/month
    - Total: ~$25M/year (conservative estimate)
    - Actual contract: $300M/year (includes security, media services, consulting)

Typical AWS Architecture:
    S3 (Origin storage)
        → Lambda@Edge (custom logic at edge)
        → CloudFront (global distribution)
        → Route 53 (DNS)
        → ACM (free SSL certificates)
        → WAF (web application firewall)
    
All in one console, one bill, one support contract

CloudFront Pricing (Pay-as-you-go, no minimums):
    Data Transfer Out (North America):
        - First 10 TB: $0.085/GB
        - Next 40 TB: $0.080/GB
        - Next 100 TB: $0.060/GB
        - Over 150 TB: $0.040/GB
        - Over 5 PB: $0.020/GB
    
    Requests:
        - HTTP requests: $0.0075 per 10,000
        - HTTPS requests: $0.010 per 10,000
    
Example - 10 TB/month:
    Bandwidth: 10,000 GB × $0.085 = $850
    Requests: 100M requests × $0.010/10K = $100
    Total: $950/month
    
vs Akamai: $1,500/month (10 TB) + $5K minimum = $6,500/month
Savings: $5,550/month (85% cheaper for small/medium sites)

Traditional CDN (Akamai/Fastly):
    1. Contact sales (wait for sales call)
    2. Sign contract (legal review, 2-4 weeks)
    3. Account setup (onboarding, 1-2 weeks)
    4. Technical integration (engineering, 1-2 weeks)
    Total: 1-2 months to go live

CloudFront:
    1. Log into AWS Console
    2. Click "Create Distribution"
    3. Enter origin domain (S3 bucket or custom)
    4. Configure cache behaviors
    5. Click "Create"
    Total: 15 minutes to go live

Architecture:
    - Own CDN infrastructure (rented servers in data centers)
    - 50 locations worldwide
    - Cost: $2M/year (server rentals + bandwidth)
    - Engineering: 5 full-time engineers maintaining CDN
    - Issues:
        * Slow in Africa, South America (poor coverage)
        * Manual scaling for new courses
        * DDoS attacks required manual mitigation
        * Video buffering complaints (30% of users)

Architecture:
    Videos stored in S3 → CloudFront (450+ edge locations)
    
Configuration:
    - S3 bucket (origin): 500K videos, 2 PB total
    - CloudFront distribution: Automatic video optimization
    - Lambda@Edge: Geo-restriction logic (licensing)
    - CloudFront cache: 90%+ hit rate

Results:
    - Cost: $800K/year (60% reduction vs self-hosted)
    - Engineering: 0 engineers (fully managed)
    - Coverage: 450 locations (vs 50) = 9x improvement
    - Performance: 
        * Video start time: 5 seconds → 1.2 seconds (76% faster)
        * Buffering complaints: 30% → 3% (90% reduction)
        * Completion rates: 45% → 58% (students finish more courses)
    - Revenue impact: +$50M/year (more completions = more subscriptions)
    
ROI: Spent $800K/year, saved $1.2M/year in costs + gained $50M revenue

Video Qualities Generated:
    - 240p (0.3 Mbps): Low bandwidth users
    - 360p (0.7 Mbps): Mobile users
    - 480p (1.5 Mbps): Standard quality
    - 720p (3 Mbps): HD quality
    - 1080p (6 Mbps): Full HD (premium users)

CloudFront automatically serves:
    - Fast connection (10 Mbps+): 1080p
    - Medium connection (3-10 Mbps): 720p
    - Slow connection (1-3 Mbps): 480p
    - Very slow (<1 Mbps): 240p

Result: Smooth playback for all users (no buffering)

// Lambda@Edge function (runs at CloudFront edge)
exports.handler = async (event) => {
    const request = event.Records[0].cf.request;
    const headers = request.headers;
    
    // Get user's country from CloudFront header
    const country = headers['cloudfront-viewer-country'][0].value;
    
    // Check course licensing for this country
    const courseId = request.uri.split('/')[2];
    const allowedCountries = await getAllowedCountries(courseId);
    
    if (!allowedCountries.includes(country)) {
        return {
            status: '403',
            body: 'This course is not available in your country due to licensing restrictions.'
        };
    }
    
    return request; // Allow request to proceed
};

CloudFront provides metrics:
    - Requests per second (current: 50K req/sec average)
    - Bandwidth usage (current: 500 TB/month)
    - Cache hit rate (current: 92%)
    - Error rates (4xx, 5xx responses)
    - Top requested objects
    - Geographic distribution

Coursera uses this to:
    - Identify slow videos (re-encode)
    - Find popular courses (recommend to others)
    - Detect DDoS attacks (spike in requests)
    - Optimize costs (invalidate rarely-accessed videos)

Akamai: 4,100+ locations
CloudFront: 450+ locations

Impact: Rural areas may be 50-100ms farther from edge
Example: 
    - Rural India: Akamai 30ms, CloudFront 80ms
    - Sub-Saharan Africa: Akamai 40ms, CloudFront 120ms
    
For Coursera: Acceptable (education can tolerate 50ms extra)
For Apple: Unacceptable (need <25ms p95 latency everywhere)

Akamai: Full custom logic via EdgeWorkers (JavaScript at edge)
CloudFront: Limited to Lambda@Edge (some restrictions)

Lambda@Edge Restrictions:
    - Max execution time: 5 seconds (vs unlimited Akamai)
    - Max package size: 50 MB (vs unlimited Akamai)
    - Limited triggers (vs any request stage Akamai)
    
For most use cases: Lambda@Edge sufficient
For advanced edge computing: Akamai more flexible

Akamai: 
    - Dedicated account team
    - 24/7 phone support
    - <15 minute response time (enterprise SLA)
    - Regular business reviews

CloudFront:
    - Standard AWS Support: Email only, 24-hour response
    - Business Support ($100+/month): 1-hour response
    - Enterprise Support ($15K+/month): 15-minute response + TAM
    
For large enterprises: Akamai support better
For startups/SMBs: CloudFront support adequate

CDN Provider Rankings:
1. Akamai: 18ms average TTFB
2. Cloudflare: 22ms average TTFB
3. Fastly: 26ms average TTFB
4. AWS CloudFront: 31ms average TTFB
5. Azure CDN: 38ms average TTFB
6. Google Cloud CDN: 41ms average TTFB

Winner: Akamai (fastest, most consistent)
Best Value: Cloudflare (fast + free tier)
Best for AWS users: CloudFront (integration)

North America:
    - Akamai: 12ms
    - Cloudflare: 14ms
    - CloudFront: 16ms
    - All CDNs excellent (mature infrastructure)

Europe:
    - Akamai: 15ms
    - Cloudflare: 18ms
    - CloudFront: 22ms
    - All CDNs good

Asia (Urban):
    - Akamai: 20ms
    - Cloudflare: 25ms
    - CloudFront: 35ms
    - Cloudflare gaining ground

Asia (Rural):
    - Akamai: 35ms (still acceptable)
    - Cloudflare: 65ms (noticeable lag)
    - CloudFront: 80ms (sluggish)
    - Akamai's 4,100 locations shine here

Latin America:
    - Akamai: 40ms
    - Cloudflare: 75ms
    - CloudFront: 95ms
    - Underserved region, Akamai best

Africa:
    - Akamai: 55ms
    - Cloudflare: 120ms
    - CloudFront: 150ms
    - Significant performance gap

                    Price               Performance
Akamai:             $$$$$ (highest)     5/5 (best)
Fastly:             $$$$ (high)         4/5 (excellent)
Cloudflare Ent:     $$$ (moderate)      4/5 (excellent)
CloudFront:         $$ (affordable)     3/5 (good)
Azure CDN:          $$ (affordable)     3/5 (good)
Cloud CDN:          $$ (affordable)     3/5 (good)
Cloudflare Free:    $ (free!)           3/5 (good)

Sweet Spot: Cloudflare (great performance, reasonable cost)
Best Premium: Akamai (worth the cost for mission-critical)
Best Budget: CloudFront (if already on AWS)

Q1: What's your budget?
    - Unlimited → Consider Akamai (best performance)
    - Limited → Go to Q2

Q2: Are you already using a cloud provider?
    - AWS → CloudFront (easy integration, good performance)
    - Azure → Azure CDN (integration benefits)
    - GCP → Cloud CDN (integration benefits)
    - Multiple/None → Go to Q3

Q3: What's your traffic volume?
    - <1 TB/month → Cloudflare Free (unbeatable value)
    - 1-10 TB/month → Cloudflare Pro ($20/month)
    - 10-100 TB/month → CloudFront or Cloudflare Business
    - 100+ TB/month → Negotiate with Akamai/Cloudflare/Fastly

Q4: Do you need advanced features?
    - Instant purge → Fastly (best-in-class)
    - Custom edge logic → Cloudflare Workers or Akamai EdgeWorkers
    - Video streaming → Akamai (industry leader)
    - DDoS protection → Cloudflare (free unlimited) or Akamai
    - Compliance (HIPAA, etc.) → Akamai or CloudFront

Q5: Geographic coverage needed?
    - Global + rural areas → Akamai (4,100+ locations)
    - Urban areas only → Cloudflare/CloudFront (sufficient)
    - Specific regions → Check provider maps

Q6: Support requirements?
    - 24/7 phone support → Akamai Enterprise
    - Email support OK → CloudFront/Cloudflare
    - Community support OK → Cloudflare Free

To get SSL certificate (before Let's Encrypt):
1. Purchase certificate from CA (Certificate Authority)
   Cost: $50-300/year per domain
   
2. Prove domain ownership
   Method: Email verification or file upload
   Time: 1-7 days wait
   
3. Generate CSR (Certificate Signing Request)
   Complexity: OpenSSL commands, cryptography knowledge
   Error-prone: Wrong parameters = invalid cert
   
4. Install certificate on server
   Complexity: Different process for Apache, NGINX, IIS
   Risk: Misconfiguration = site down
   
5. Remember to renew (certificates expire)
   Problem: 1 in 4 sites experienced outage from expired cert
   Manual process: Check expiry dates, repeat steps 1-4

Result: Only large companies could afford proper HTTPS
Small businesses, hobbyists: HTTP (insecure)

Traditional CA (Slow, Manual):
    1. Buy certificate ($50-300, manual payment)
    2. Email verification (wait hours/days)
    3. Manual CSR generation (error-prone)
    4. Manual installation (complex)
    5. Manual renewal (every year, often forgotten)
    
Let's Encrypt (Fast, Automated):
    1. Install Certbot (automated client)
    2. Run: certbot --nginx -d example.com
    3. Certbot proves domain ownership (automatic)
    4. Let's Encrypt issues certificate (seconds)
    5. Certbot installs certificate (automatic)
    6. Auto-renewal every 60 days (set it and forget it)
    
Time: 5 minutes (vs 1-7 days)
Cost: $0 (vs $50-300/year)
Renewal: Automatic (vs manual, often failed)

# Install Certbot (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install certbot python3-certbot-nginx

# Get certificate and auto-configure NGINX
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com

# Output:
# Saving debug log to /var/log/letsencrypt/letsencrypt.log
# Requesting certificate for yourdomain.com and www.yourdomain.com
# 
# Successfully received certificate.
# Certificate is saved at: /etc/letsencrypt/live/yourdomain.com/fullchain.pem
# Key is saved at: /etc/letsencrypt/live/yourdomain.com/privkey.pem
# This certificate expires on 2024-06-15.
# 
# Deploying certificate to nginx config
# Reloading nginx configuration
# 
# HTTPS is now enabled!
# Your site is now available at https://yourdomain.com

# Test auto-renewal (runs twice daily via cron)
sudo certbot renew --dry-run

# Output:
# Congratulations, all renewals succeeded!

ACME Challenge Process:
1. Certbot contacts Let's Encrypt server
   Request: "I want certificate for yourdomain.com"
   
2. Let's Encrypt responds with challenge
   Challenge: "Prove you control this domain"
   Method: "Create file .well-known/acme-challenge/TOKEN"
   
3. Certbot creates challenge file
   File: /var/www/html/.well-known/acme-challenge/random-token
   Content: Verification string
   
4. Let's Encrypt validates
   HTTP request to: http://yourdomain.com/.well-known/acme-challenge/random-token
   Checks: Content matches expected value
   
5. Validation succeeds
   Let's Encrypt: "You control this domain"
   Issues: Certificate (valid 90 days)

Total time: 10-30 seconds (automated)

Certbot modifies NGINX configuration:

Before:
server {
    listen 80;
    server_name yourdomain.com;
    
    location / {
        proxy_pass http://localhost:3000;
    }
}

After (Certbot auto-configuration):
server {
    listen 80;
    server_name yourdomain.com;
    
    # Redirect HTTP to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name yourdomain.com;
    
    # Let's Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    
    # Modern SSL configuration (Certbot adds these)
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;
    
    # Security headers (best practices)
    add_header Strict-Transport-Security "max-age=31536000" always;
    
    location / {
        proxy_pass http://localhost:3000;
    }
}

Result: HTTPS enabled, HTTP redirects to HTTPS, modern security

# Certbot adds cron job (auto-renewal twice daily)
cat /etc/cron.d/certbot

# 0 */12 * * * root certbot renew --quiet

# This means:
# - Runs every 12 hours
# - Checks if certificates expire in <30 days
# - If yes, automatically renews
# - If renewal succeeds, reloads nginx
# - If renewal fails, sends email alert

# You never have to manually renew again!

HTTPS Usage Over Time:
2014: 30% HTTPS (before Let's Encrypt)
2016: 40% HTTPS (Let's Encrypt launches)
2018: 60% HTTPS (Chrome "Not Secure" warnings)
2020: 80% HTTPS (HTTPS becomes default)
2022: 90% HTTPS (HTTP nearly extinct)
2024: 95%+ HTTPS (HTTPS is the standard)

Let's Encrypt Share of SSL Certificates:
2016: 0% (just launched)
2018: 25% (rapid growth)
2020: 45% (majority of new certs)
2022: 60% (dominant player)
2024: 68% (2 of 3 certificates worldwide)

Competitor Impact:
Comodo: 25% → 10% (dropped 60%)
DigiCert: 20% → 8% (dropped 60%)
GoDaddy: 15% → 5% (dropped 67%)
Symantec: 10% → defunct (acquired by DigiCert)

Why: Free vs $50-300/year is a no-brainer for most sites

Before Let's Encrypt (2014):
    - Average SSL cost: $150/year per domain
    - Websites needing HTTPS: 1 billion
    - Industry revenue: $150B/year (estimated)
    - Small businesses: Often skipped HTTPS (too expensive)

After Let's Encrypt (2024):
    - Let's Encrypt: Free (sponsored by tech giants)
    - Certificates issued: 430M active
    - Money saved by website owners: $64B/year
    - Small businesses: Can afford proper security

Example - 100 Domain Portfolio:
    Before: 100 domains × $150/year = $15,000/year
    After: 100 domains × $0/year = $0/year
    Savings: $15,000/year (100% cost reduction)

Traditional HTTPS Setup (Complex):
1. Buy SSL certificate ($50-300/year)
2. Generate CSR, prove domain ownership
3. Install certificate on your web server
4. Configure NGINX/Apache for HTTPS
5. Test configuration
6. Set up renewal process
Total: 2-8 hours, technical expertise required

Cloudflare Universal SSL (Simple):
1. Add your website to Cloudflare (free account)
2. Point your domain's nameservers to Cloudflare
3. Cloudflare automatically provisions SSL certificate
4. Enable "Flexible SSL" or "Full SSL" (one click)
5. Done - your site is now HTTPS
Total: 15 minutes, no technical expertise required

1. Flexible SSL (Free):
    Browser → [HTTPS] → Cloudflare → [HTTP] → Your Server
    
    Pros: Instant HTTPS (no certificate on your server)
    Cons: Traffic between Cloudflare and your server is unencrypted
    Use case: Legacy servers that can't do HTTPS
    
2. Full SSL (Free):
    Browser → [HTTPS] → Cloudflare → [HTTPS] → Your Server
    
    Pros: End-to-end encryption
    Cons: Requires certificate on your server (self-signed OK)
    Use case: Modern servers, better security
    
3. Full SSL (Strict) (Free):
    Browser → [HTTPS] → Cloudflare → [HTTPS validated] → Your Server
    
    Pros: Maximum security (validates your server certificate)
    Cons: Requires valid certificate (not self-signed)
    Use case: Enterprise security requirements
    Recommended: Use with Let's Encrypt on origin server

How Cloudflare Provides Free SSL to 28M Websites:

1. Multi-Domain Certificates:
   - Traditional: 1 certificate per domain ($50-300 each)
   - Cloudflare: 1 certificate covers 50-100 domains
   - Result: 500x cost reduction per certificate
   
2. Automated Issuance:
   - Cloudflare has partnerships with DigiCert, Let's Encrypt
   - Automated API requests certificates for new domains
   - No human involvement (fully automated pipeline)
   
3. Certificate Caching:
   - Certificate stored on all 310+ Cloudflare data centers
   - Shared across multiple customers (with SNI)
   - Amortized cost: <$0.01 per website
   
4. Economies of Scale:
   - 28M websites = massive negotiating power
   - Cloudflare pays bulk rates to CAs
   - Cost passed as "free" to customers (subsidized by enterprise plans)

Result: $150/year certificate → free for all users

User Types: https://example.com

Step 1: TCP Connection (Not Encrypted Yet)
    Time: ~30ms (round trip to server)
    Browser → Server: SYN (synchronize)
    Server → Browser: SYN-ACK (acknowledge)
    Browser → Server: ACK (established)
    
Step 2: TLS ClientHello (Browser → Server)
    Time: +30ms (round trip)
    Browser sends:
        - Supported TLS versions: [1.2, 1.3]
        - Supported cipher suites: [256-bit AES, etc.]
        - Random number (for key generation)
    
Step 3: TLS ServerHello (Server → Browser)
    Time: Same round trip as ClientHello
    Server responds:
        - Chosen TLS version: 1.3
        - Chosen cipher suite: TLS_AES_256_GCM_SHA384
        - Server certificate (proves identity)
        - Random number (for key generation)
    
Step 4: Certificate Verification (Browser)
    Time: ~10ms (local processing)
    Browser checks:
        - Is certificate signed by trusted CA? 
        - Does domain match certificate? 
        - Is certificate expired?  (valid until 2025-06-15)
        - Is certificate revoked?  (checks OCSP)
    
Step 5: Key Exchange (Browser → Server)
    Time: +30ms (round trip)
    Browser:
        - Generates pre-master secret
        - Encrypts with server's public key
        - Sends encrypted pre-master secret
    Server:
        - Decrypts with private key
        - Both sides now have shared secret
    
Step 6: Session Keys Generated (Both Sides)
    Time: ~5ms (local processing)
    Browser + Server:
        - Use pre-master secret + random numbers
        - Generate session keys (symmetric encryption)
        - These keys encrypt all subsequent traffic
    
Step 7: Finished Messages (Both Sides)
    Time: +30ms (round trip)
    Browser → Server: "Finished" (encrypted with session key)
    Server → Browser: "Finished" (encrypted with session key)
    
    Handshake complete! Can now send HTTP request.

Total TLS 1.2 Handshake Time: ~120-150ms (4 round trips)
    - TCP: 1 round trip (30ms)
    - TLS: 2-3 round trips (60-90ms)
    - Processing: 15ms
    
Then HTTP Request:
    Time: +30ms (round trip)
    Browser → Server: GET / HTTP/1.1 (encrypted)
    Server → Browser: HTML response (encrypted)

Total Time to First Byte (TTFB): ~180ms (HTTPS)
vs HTTP: ~60ms (no TLS handshake)

HTTPS Penalty: 120ms extra latency (due to TLS handshake)

TLS 1.2 Handshake: 2-3 round trips (120-150ms)
TLS 1.3 Handshake: 1 round trip (30-40ms)

Improvement: 70-75% faster handshake!

How TLS 1.3 Achieves This:
1. Combine ClientHello + Key Exchange (1 round trip instead of 2)
2. Remove unnecessary cipher suites (simplify negotiation)
3. 0-RTT mode: Resume previous sessions instantly (0ms handshake)

TLS 1.3 with 0-RTT (Resumption):
    Browser remembers previous session
    Sends encrypted HTTP request immediately
    No handshake needed!
    
Result: HTTPS can be as fast as HTTP (0ms TLS overhead)

Adoption (2024):
    - 80%+ of browsers support TLS 1.3
    - 60%+ of servers support TLS 1.3
    - Cloudflare, CloudFront: 100% TLS 1.3 support

Date: May 2023
Duration: 4 hours
Impact: 900M users unable to access LinkedIn

Root Cause:
    - SSL certificate expired at 08:00 UTC
    - Automated renewal failed (process issue)
    - No monitoring alert for cert expiry
    - Manual intervention required

Timeline:
    08:00 UTC - Certificate expires
    08:03 UTC - Users report "Your connection is not private" errors
    08:15 UTC - Engineering team notified
    08:45 UTC - Root cause identified (expired cert)
    09:00 UTC - New certificate generated
    09:30 UTC - Certificate deployed to CDN
    10:00 UTC - CDN cache cleared globally
    12:00 UTC - 100% recovery

Business Impact:
    - Revenue loss: $8M (4 hours × $2M/hour estimated)
    - User trust: 100K+ users complained on Twitter
    - Stock impact: -2.3% that day ($4B market cap loss)
    - Reputation: "How does LinkedIn forget to renew?"

Lessons Learned:
    1. Monitor certificate expiry (alert 30 days before)
    2. Test renewal process monthly (dry run)
    3. Have backup certificates ready
    4. Implement automated renewal (Let's Encrypt style)
    5. Use multiple redundant systems

Prevention:
    After incident, LinkedIn implemented:
        - 3 independent cert expiry monitoring systems
        - Automated renewal 45 days before expiry
        - Backup renewal 15 days before expiry
        - Manual fallback 7 days before expiry
        - Executive dashboard showing all cert expiry dates
    
    Result: No cert expiry outages since (18+ months)

Ericsson Network Outage (2018):
    Cause: Expired software certificate
    Impact: 32M phones couldn't make calls (UK, Japan)
    Duration: 11 hours
    Cost: $100M+ (customer compensation + fixes)

Microsoft Teams Outage (2020):
    Cause: Expired authentication certificate  
    Impact: 75M users couldn't sign in
    Duration: 3 hours
    Cost: $10M+ estimated productivity loss

Equifax Breach (2017):
    Cause: Expired certificate = monitoring failed
    Impact: 143M users' data stolen
    Cost: $1.4B (settlements + fines)
    Note: Certificate expiry disabled security monitoring

Pattern: Certificate expiry is a top cause of outages
Solution: Automated renewal (Let's Encrypt prevents 99% of these)

Solution: Let's Encrypt + Certbot
    - Free certificates
    - Automated renewal
    - 5-minute setup
    
Commands:
    # Get certificate
    sudo certbot --nginx -d yourdomain.com
    
    # Auto-renewal runs twice daily (automatic)
    # Test it:
    sudo certbot renew --dry-run

Cost: $0/year
Time: 5 minutes initial, 0 minutes maintenance
Reliability: 99.9%+ (auto-renewal prevents expiry)

Solution: Cloudflare + Let's Encrypt on origin
    - Cloudflare: Free SSL between users and CDN
    - Let's Encrypt: Free SSL between CDN and origin
    - Automatic renewal on both sides
    
Setup:
    1. Add domains to Cloudflare (bulk import CSV)
    2. Enable "Full SSL (Strict)" mode
    3. Install Certbot on origin server
    4. Get wildcard certificate: *.yourdomain.com
    
Cost: $0/year (Cloudflare Free plan)
Time: 30 minutes initial, 0 minutes maintenance
Reliability: 99.99%+ (dual redundancy)

Solution: ACM (AWS Certificate Manager) or similar
    - AWS ACM: Free SSL for AWS resources
    - Automatic renewal (no manual intervention)
    - Centralized management (dashboard)
    
Setup:
    1. Request certificate in ACM console
    2. Validate domain (DNS record)
    3. Attach to CloudFront/ALB/API Gateway
    4. ACM auto-renews 60 days before expiry
    
Features:
    - Wildcard certificates (*.domain.com)
    - Multi-domain certificates (SAN)
    - Automatic deployment to resources
    - Centralized expiry monitoring
    
Cost: $0/year (free for AWS resources)
Time: 10 minutes per domain, 0 minutes maintenance
Reliability: 99.999%+ (AWS SLA)

Note: ACM certificates only work with AWS resources
For non-AWS, use Let's Encrypt or paid CA

1991: HTTP/0.9 - Single-line protocol (GET /page.html)
1996: HTTP/1.0 - Headers added, POST/HEAD methods
1999: HTTP/1.1 - Persistent connections, chunked transfer
2015: HTTP/2 - Binary protocol, multiplexing, header compression
2022: HTTP/3 - QUIC (UDP-based), even faster

Gap: 16 years between HTTP/1.1 and HTTP/2
Reason: HTTP/1.1 "good enough" for text-based web
Catalyst: Mobile devices, rich media, need for speed

HTTP/1.1 Problems:
1. Head-of-Line Blocking
   - Browser opens 6 concurrent connections per domain
   - Each connection handles 1 request at a time
   - If request 1 is slow, requests 2-6 wait (blocked)
   
2. No Request Prioritization
   - Critical CSS loads same priority as low-priority image
   - Browser can't tell server "I need CSS first!"
   - Result: Suboptimal loading order
   
3. Redundant Headers
   - Every request sends full headers (1-2 KB)
   - Same headers repeated: User-Agent, Cookies, etc.
   - Wasted bandwidth: 40% of request size is headers
   
4. Workarounds Required (Hacks):
   - Domain sharding: assets1.shopify.com, assets2.shopify.com
   - CSS sprites: Combine images to reduce requests
   - Inlining: Embed CSS/JS in HTML (bloats pages)
   - Concatenation: Combine 20 JS files into 1 (cache bust on any change)

Page Assets:
    - 1 HTML document (50 KB)
    - 3 CSS files (120 KB total)
    - 8 JavaScript files (400 KB total)
    - 30 images (2 MB total)
    - 5 fonts (150 KB total)
    Total: 42 requests, 2.72 MB

Load Timeline (HTTP/1.1):
    0ms: Request HTML
    80ms: Receive HTML, parse, discover assets
    80ms: Request 6 assets (max concurrent connections)
    180ms: First 6 assets received
    180ms: Request next 6 assets (second batch)
    280ms: Second batch received
    280ms: Request next 6 assets (third batch)
    ... continues for 7 batches (42 requests ÷ 6)
    
Total Load Time: ~1,800ms (1.8 seconds)

Bottleneck: Only 6 requests at a time = waterfall effect

HTTP/1.1: 
    6 connections × 1 request each = 6 concurrent requests
    
HTTP/2:
    1 connection × unlimited requests = all 42 requests concurrent!

How it works:
    - Single TCP connection to server
    - Multiple "streams" within connection
    - Each stream = 1 request/response
    - Streams interleaved (no head-of-line blocking)
    
Example:
    Stream 1: HTML document (50 KB) - high priority
    Stream 2: CSS file (40 KB) - high priority
    Stream 3: JS file (100 KB) - medium priority
    Stream 4-33: Images (2 MB) - low priority
    
    All requests sent immediately (no waiting)
    Server sends high-priority first
    Browser receives data as available (interleaved)

HTTP/1.1 Headers (Repeated Every Request):
    GET /product/t-shirt HTTP/1.1
    Host: store.shopify.com
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)...
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: en-US,en;q=0.5
    Accept-Encoding: gzip, deflate, br
    Cookie: session=abc123; cart=xyz789; preferences=...
    Referer: https://store.shopify.com/collections/shirts
    
    Total: ~1,500 bytes per request
    42 requests × 1,500 bytes = 63 KB wasted on headers!

HTTP/2 Headers (HPACK Compression):
    First request: Full headers (1,500 bytes)
    Subsequent requests: Only differences sent
    
    Request 2:
        :path: /style.css
        (all other headers same as request 1 = not sent)
        Total: 20 bytes (vs 1,500 bytes HTTP/1.1)
    
    42 requests: 1,500 + (41 × 20) = 2,320 bytes
    Savings: 63 KB → 2.3 KB (96% reduction!)

HTTP/1.1 (Reactive):
    Browser: "Give me index.html"
    Server: "Here's index.html"
    Browser: (parses HTML) "Oh, I need style.css"
    Server: "Here's style.css"
    Browser: (parses CSS) "Oh, I need logo.png"
    Server: "Here's logo.png"
    
    Problem: Round-trip for each discovery (80ms × 3 = 240ms wasted)

HTTP/2 (Proactive):
    Browser: "Give me index.html"
    Server: "Here's index.html, and I know you'll need style.css and logo.png, so here they are too"
    Browser: (receives everything immediately)
    
    Savings: 2 round trips eliminated (160ms saved)
    
Shopify's Server Push Config:
    Link: </style.css>; rel=preload; as=style
    Link: </logo.png>; rel=preload; as=image
    Link: </script.js>; rel=preload; as=script
    
    Server automatically pushes these when HTML is requested

HTTP/1.1:
    All requests equal priority
    Large image might load before critical CSS
    Result: Slow visual rendering

HTTP/2 Stream Priority (Browser → Server):
    Priority 1 (Highest): HTML, CSS
    Priority 2 (High): JavaScript (critical)
    Priority 3 (Medium): Fonts, JavaScript (non-critical)
    Priority 4 (Low): Images (above fold)
    Priority 5 (Lowest): Images (below fold)
    
    Server respects priorities = optimal load order
    Visual rendering starts 200-300ms faster

Test Conditions:
    - Sample: 1,000 diverse Shopify stores
    - Location: 20 global test points
    - Network: 3G, 4G, WiFi, Fiber
    - Metric: Time to Interactive (TTI)

Before HTTP/2 (HTTP/1.1):
    Average TTI: 3.2 seconds
    P50 (median): 2.8 seconds
    P95 (slow): 6.5 seconds
    Bounce rate: 18% (users leave before page loads)

After HTTP/2:
    Average TTI: 1.6 seconds (50% faster)
    P50 (median): 1.4 seconds (50% improvement)
    P95 (slow): 3.2 seconds (51% improvement)
    Bounce rate: 12% (33% reduction)

Business Impact:
    Conversion rate: +15% (faster loads = more sales)
    Revenue: +$3.5B GMV annually (15% of $235B)
    Customer satisfaction: +12% (NPS survey)
    
Shopify's Investment:
    Engineering time: 6 months, 8 engineers
    Infrastructure upgrade: $2M (HTTP/2 support in CDN, load balancers)
    
ROI: Spent $2M, gained $3.5B in merchant sales (1,750x return)

Browser Support: 98%+ (all modern browsers)
Server Support:
    - NGINX: Since 1.9.5 (2015)
    - Apache: Since 2.4.17 (2015)
    - Cloudflare: 100% of sites
    - CloudFront: Enabled by default
    - Akamai: 100% of sites
    
Adoption Rate:
    2016: 5% of websites
    2018: 25% of websites
    2020: 50% of websites
    2024: 65% of top 10M websites
    
Top 1M sites: 85%+ HTTP/2
Long tail: Still 35% HTTP/1.1 (legacy)

TCP (Transmission Control Protocol):
    Pros:
        - Reliable (guarantees delivery, order)
        - Congestion control
        - Universal support (every device)
    
    Cons:
        - Head-of-line blocking (packet loss blocks all streams)
        - Slow start (gradual speed increase)
        - 3-way handshake (latency)
        - TCP + TLS = 2 handshakes (extra latency)

Example - Packet Loss Impact on HTTP/2 over TCP:
    Sending 10 streams simultaneously
    Stream 5 packet lost
    TCP must retransmit stream 5 packet
    Streams 6-10 blocked until stream 5 recovered
    
    Result: 1 packet loss = all streams delayed (HOL blocking)
    Problem: Mobile networks have 1-5% packet loss (common)

HTTP/2 over TCP:
    Stream 1: ████████░░ (packet lost, blocking all)
    Stream 2: ░░░░░░░░░░ (waiting for stream 1)
    Stream 3: ░░░░░░░░░░ (waiting for stream 1)
    
    Result: 1 lost packet delays entire page

HTTP/3 over QUIC:
    Stream 1: ████████░░ (packet lost, retransmitting)
    Stream 2: ██████████ (continues unaffected)
    Stream 3: ██████████ (continues unaffected)
    
    Result: Only affected stream delayed (others unblocked)

HTTP/2 over TCP + TLS:
    0ms: TCP SYN →
    30ms: TCP SYN-ACK ←
    60ms: TLS ClientHello →
    90ms: TLS ServerHello ←
    120ms: HTTP Request →
    150ms: HTTP Response ←
    
    Total: 150ms to first byte (5 round trips)

HTTP/3 over QUIC (First Connection):
    0ms: QUIC + TLS handshake (combined) →
    30ms: Response ←
    60ms: HTTP Request →
    90ms: HTTP Response ←
    
    Total: 90ms to first byte (3 round trips)
    Savings: 60ms (40% faster)

HTTP/3 over QUIC (Resumption 0-RTT):
    0ms: HTTP Request + session ticket →
    30ms: HTTP Response ←
    
    Total: 30ms to first byte (1 round trip)
    Savings: 120ms vs HTTP/2 (80% faster!)

Scenario: User on train, switches WiFi → 4G

HTTP/2 over TCP:
    TCP connection tied to IP address
    IP changes (WiFi → 4G) = connection lost
    Must re-establish: TCP handshake + TLS handshake
    Time: 150-200ms interruption
    User experience: Video buffering, image loading pauses
    
HTTP/3 over QUIC:
    QUIC connection tied to "Connection ID" (not IP)
    IP changes = same Connection ID
    Connection continues seamlessly
    Time: 0ms interruption
    User experience: No buffering, smooth transition
    
Real-world benefit: Video streaming on mobile (critical)

HTTP/2: Can be used without TLS (rare, but possible)
HTTP/3: TLS 1.3 mandatory (encryption always on)

Result: 100% of HTTP/3 traffic is encrypted
No plaintext HTTP/3 possible (security by design)

Test Methodology:
    - 1 million page loads across 28M Cloudflare sites
    - 100 global test locations
    - Network conditions: 3G, 4G, 5G, WiFi, Fiber
    - Metric: Time to First Byte (TTFB)

Results (Median TTFB):
    HTTP/1.1: 350ms
    HTTP/2: 180ms (49% faster than HTTP/1.1)
    HTTP/3: 125ms (64% faster than HTTP/1.1, 31% faster than HTTP/2)
    
Mobile Performance (4G with 2% packet loss):
    HTTP/2: 450ms (degraded due to packet loss)
    HTTP/3: 180ms (resilient to packet loss)
    
    Improvement: 60% faster on lossy networks

Connection Migration (WiFi → 4G):
    HTTP/2: 220ms interruption (re-handshake)
    HTTP/3: 0ms interruption (seamless)
    
    Benefit: Zero buffering on network change

Enable HTTP/3 on Cloudflare (Literally 1 Click):
1. Log into Cloudflare dashboard
2. Navigate to Network tab
3. Toggle "HTTP/3 (with QUIC)" ON
4. Done - your site now supports HTTP/3

Behind the scenes:
    - Cloudflare edge servers advertise HTTP/3 support
    - Browsers supporting HTTP/3 use it automatically
    - Browsers without HTTP/3 fall back to HTTP/2
    - No changes needed on your origin server
    
Cost: $0 (included in Free plan)
Time: 5 seconds (literally a toggle)

Why Only 30% Adoption (vs 65% HTTP/2)?

1. UDP Blocking:
    - Some corporate firewalls block UDP
    - Reason: Old security policy (pre-QUIC era)
    - Impact: 5-10% of users can't use HTTP/3
    - Fallback: Browser detects, uses HTTP/2 instead
    
2. Middlebox Interference:
    - Some ISPs inspect/modify traffic (deep packet inspection)
    - QUIC encryption prevents inspection
    - ISPs might block/throttle QUIC
    - Adoption: Slowly improving as ISPs adapt
    
3. Server Support:
    - NGINX: Experimental support (requires compilation)
    - Apache: No stable HTTP/3 yet (in development)
    - Node.js: Experimental (behind flag)
    - CDNs: Full support (Cloudflare, Fastly, Cloudfront)
    
    Reality: Most sites rely on CDN for HTTP/3
    Origin servers still use HTTP/2 or HTTP/1.1
    
4. Debugging Complexity:
    - TCP: 40+ years of tools (Wireshark, tcpdump)
    - QUIC: Newer, fewer tools, encrypted
    - Engineers less familiar with UDP debugging
    - Learning curve: 6-12 months for teams

High Impact (Use HTTP/3):
    Mobile-first applications (connection migration)
    Video streaming (resilience to packet loss)
    Real-time apps (gaming, chat, video calls)
    Global users (high latency, lossy networks)
    Behind CDN (Cloudflare, etc. - free & easy)

Moderate Impact:
    ~ E-commerce (faster loads, but HTTP/2 sufficient)
    ~ News sites (incremental benefit)
    ~ Corporate sites (may have UDP firewall issues)

Low Impact:
     Intranet applications (low latency, reliable networks)
     APIs (HTTP/2 already excellent for APIs)
     Static sites (HTTP/2 sufficient)

# /etc/nginx/nginx.conf

server {
    listen 443 ssl http2;  # Enable HTTP/2 (requires SSL)
    server_name example.com;
    
    # SSL Certificates
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    
    # HTTP/2 Push (Preload Critical Assets)
    location = /index.html {
        http2_push /style.css;
        http2_push /logo.png;
        http2_push /script.js;
    }
    
    # Increase HTTP/2 concurrent streams (default 128)
    http2_max_concurrent_streams 256;
    
    # Increase HTTP/2 max header size
    large_client_header_buffers 4 32k;
    
    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name example.com;
    return 301 https://$server_name$request_uri;
}

# Check if NGINX has HTTP/2 support
nginx -V 2>&1 | grep -o with-http_v2_module
# Output: with-http_v2_module

# If not, install NGINX with HTTP/2 (Ubuntu)
sudo apt update
sudo apt install nginx-extras

# Edit config (add 'http2' to listen directive)
sudo nano /etc/nginx/sites-available/default

# Test configuration
sudo nginx -t

# Reload NGINX
sudo systemctl reload nginx

# Verify HTTP/2 is working
curl -I --http2 https://example.com
# Output: HTTP/2 200 (success!)

# Use h2load (HTTP/2 benchmark tool)
h2load -n 10000 -c 100 https://example.com

# Output:
# finished in 3.52s, 2841.90 req/s
# requests: 10000 total, 10000 started, 10000 done, 10000 succeeded
# status codes: 10000 2xx
# traffic: 150.2MB (157517000) total
#
# HTTP/2: 2,841 requests/sec
# vs HTTP/1.1: ~400 requests/sec (7x faster!)

Without Reverse Proxy (Direct Connection):
    User → Web Server 1 (overloaded, slow)
    User → Web Server 2 (idle, wasted capacity)
    User → Web Server 3 (down, users get errors)
    
    Problems:
    - Uneven load distribution
    - Single point of failure
    - No health checking
    - SSL termination on every server

With Reverse Proxy (Smart Distribution):
    User → Reverse Proxy → Web Server 1 (20% load)
                        → Web Server 2 (20% load)
                        → Web Server 3 (20% load)
                        → Web Server 4 (20% load)
                        → Web Server 5 (20% load)
    
    Benefits:
    - Even load distribution
    - Automatic failover (server 3 down? skip it)
    - Health checks (only send to healthy servers)
    - SSL termination once (proxy handles HTTPS)
    - One public IP (N servers behind)

Global Traffic (1.3B requests/month):
    ↓
Cloudflare CDN (70% cached, 30% passed to origin)
    - Static assets: 90%+ cache hit
    - HTML pages: 40%+ cache hit
    - API requests: Pass through (dynamic)
    ↓
HAProxy Load Balancers (2 servers, active-passive):
    - Primary HAProxy (handles 100% traffic)
    - Secondary HAProxy (hot standby, takes over if primary fails)
    - Health checks every 2 seconds
    - SSL termination (TLS 1.3)
    ↓
Web Servers (9 IIS servers running ASP.NET):
    - Each server: Intel Xeon, 64 GB RAM
    - Each handles: 15,000 requests/second peak
    - Total capacity: 135,000 requests/second
    - Actual usage: 50,000 requests/second average
    - Headroom: 2.7x capacity (for traffic spikes)
    ↓
SQL Server Cluster (2 servers):
    - Primary: Read/write
    - Secondary: Read replica
    - 2 TB database (21M questions + 100M answers)

# /etc/haproxy/haproxy.cfg (simplified)

global
    maxconn 500000          # 500K concurrent connections
    nbproc 4                # 4 processes (multi-core)
    cpu-map auto:1/1-4 0-3  # Pin to CPU cores 0-3
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:...
    ssl-default-bind-options ssl-min-ver TLSv1.2

defaults
    mode http
    timeout connect 5s      # 5 seconds to connect to backend
    timeout client 50s      # 50 seconds client inactivity
    timeout server 50s      # 50 seconds server inactivity
    option httplog
    option dontlognull
    option forwardfor       # Add X-Forwarded-For header

# Frontend (receives requests)
frontend stackoverflow_https
    bind *:443 ssl crt /etc/ssl/stackoverflow.com.pem
    bind *:80               # Redirect HTTP to HTTPS
    redirect scheme https if !{ ssl_fc }
    
    # Rate limiting (prevent abuse)
    stick-table type ip size 100k expire 30s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny if { sc_http_req_rate(0) gt 100 }
    
    # Route to backend
    default_backend stackoverflow_web_servers

# Backend (web servers)
backend stackoverflow_web_servers
    balance roundrobin      # Distribution algorithm
    option httpchk GET /health HTTP/1.1\r\nHost:\ stackoverflow.com
    
    # Web servers (9 total)
    server web01 10.0.1.11:80 check inter 2s fall 3 rise 2 maxconn 20000
    server web02 10.0.1.12:80 check inter 2s fall 3 rise 2 maxconn 20000
    server web03 10.0.1.13:80 check inter 2s fall 3 rise 2 maxconn 20000
    server web04 10.0.1.14:80 check inter 2s fall 3 rise 2 maxconn 20000
    server web05 10.0.1.15:80 check inter 2s fall 3 rise 2 maxconn 20000
    server web06 10.0.1.16:80 check inter 2s fall 3 rise 2 maxconn 20000
    server web07 10.0.1.17:80 check inter 2s fall 3 rise 2 maxconn 20000
    server web08 10.0.1.18:80 check inter 2s fall 3 rise 2 maxconn 20000
    server web09 10.0.1.19:80 check inter 2s fall 3 rise 2 maxconn 20000
    
    # Server parameters explained:
    # check - Enable health checks
    # inter 2s - Check every 2 seconds
    # fall 3 - Mark down after 3 failed checks (6 seconds)
    # rise 2 - Mark up after 2 successful checks (4 seconds)
    # maxconn 20000 - Max 20K connections per server

Requests distributed evenly in order:
    Request 1 → Server 1
    Request 2 → Server 2
    Request 3 → Server 3
    Request 4 → Server 1 (cycles back)
    Request 5 → Server 2
    ...

Pros:
    - Simple, predictable
    - Even distribution (if requests similar duration)
    - No overhead (no tracking needed)

Cons:
    - Doesn't consider server load
    - Long request on server 1 doesn't affect distribution
    - All servers must be equal capacity

Best for: Stateless applications with similar request times

Requests go to server with fewest active connections:
    Server 1: 5,000 connections
    Server 2: 3,000 connections ← Next request goes here
    Server 3: 7,000 connections
    
Pros:
    - Better for varying request durations
    - Adapts to actual load
    - Self-balancing

Cons:
    - Slightly more overhead (track connections)
    - Doesn't consider connection weight (1 heavy = 1 light)

Best for: Long-lived connections (websockets, streaming)

Hash user's IP address, always send to same server:
    User 1.2.3.4 → hash → Server 2 (always Server 2)
    User 5.6.7.8 → hash → Server 5 (always Server 5)

Pros:
    - Session persistence (user always gets same server)
    - No need for shared session storage
    - Caching benefits (same user = cached data)

Cons:
    - Uneven distribution (large NAT = many users = 1 server)
    - Server failure = sessions lost
    - Not scalable (adding/removing servers = rehash)

Best for: Applications requiring sticky sessions (legacy apps)

Send to server with fastest recent response times:
    Server 1: Average 50ms response
    Server 2: Average 120ms response (maybe overloaded)
    Server 3: Average 45ms response ← Next request goes here

Pros:
    - Optimal performance
    - Adapts to actual server performance
    - Handles varying server capacity

Cons:
    - Most overhead (track response times)
    - Complex algorithm
    - Can oscillate under certain conditions

Best for: Heterogeneous servers (different capacities)

Hash key (user ID, session ID) to point on ring:
    Ring: [Server 1 -- Server 2 -- Server 3 -- Server 1]
    User ID 12345 → hash → Position X → Nearest server = Server 2
    
Adding/removing server:
    Only 1/N keys rehash (vs all keys in simple hash)
    
Pros:
    - Minimal disruption when scaling
    - Cache-friendly
    - Predictable

Cons:
    - More complex
    - Requires careful implementation
    - Hotspot issues possible

Best for: Distributed caches (Redis, Memcached clusters)

Every 2 seconds, HAProxy checks each web server:
    HTTP GET /health → Server 1
    
Server 1 responds:
    HTTP/1.1 200 OK
    {
        "status": "healthy",
        "sql_connection": "ok",
        "redis_connection": "ok",
        "cpu_usage": "45%",
        "memory_usage": "62%",
        "active_requests": 12450
    }

Health Check Logic:
    HTTP 200 response = Healthy (keep sending traffic)
    HTTP 5xx response = Unhealthy (stop sending traffic)
    Timeout (>2 seconds) = Unhealthy
    3 consecutive failures = Mark server DOWN
    2 consecutive successes = Mark server UP

Example Scenario:
    10:00:00 - Server 3 healthy (200 OK)
    10:00:02 - Server 3 healthy (200 OK)
    10:00:04 - Server 3 timeout (deployment in progress)
    10:00:06 - Server 3 timeout (still deploying)
    10:00:08 - Server 3 timeout (3rd failure → MARK DOWN)
    10:00:10 - No traffic sent to Server 3
    10:00:12 - Server 3 healthy (deployment complete)
    10:00:14 - Server 3 healthy (2nd success → MARK UP)
    10:00:16 - Traffic resumes to Server 3
    
Result: Zero user impact (traffic automatically rerouted)

Rolling Deployment with HAProxy:

Step 1: Mark Server 1 as DRAIN (new connections stopped)
    HAProxy: "Finish existing requests, no new requests to Server 1"
    Wait: 10 seconds (let existing requests complete)
    Servers 2-9 handle new traffic

Step 2: Deploy to Server 1
    Stop IIS, update code, start IIS
    Duration: ~30 seconds
    Users: No impact (traffic on Servers 2-9)

Step 3: Health check Server 1
    HAProxy checks /health endpoint
    If healthy: Resume traffic
    If unhealthy: Alert engineers, rollback

Step 4: Repeat for Servers 2-9
    One at a time, 9 × 30 seconds = 4.5 minutes total
    
Result: Deploy 9 servers with zero downtime
Users never notice (always 8 servers available)

Infrastructure (Remarkably Small):
    - 2 HAProxy servers (load balancers)
    - 9 IIS web servers (ASP.NET)
    - 2 SQL Server nodes (database cluster)
    - 2 Redis nodes (caching)
    - 3 Elasticsearch nodes (search)
    Total: 18 servers handle 1.3B requests/month

Costs (Estimated 2024):
    - Servers: $50K/month (own hardware in NYC data center)
    - Bandwidth: $30K/month (900 TB/month)
    - Cloudflare: $5K/month (Enterprise plan)
    - Staff: $150K/month (5 SREs)
    Total: ~$235K/month = $2.82M/year

Cost per Request: $0.0000022 (0.00022 cents per page view)
Cost per User: $0.0016 (0.16 cents per monthly user)

Comparison to "Cloud Native" Approach:
    AWS EC2 equivalent:
        - 20 m5.2xlarge instances: $15K/month
        - RDS SQL Server Enterprise: $25K/month
        - ElastiCache Redis: $5K/month
        - Elasticsearch Service: $8K/month
        - CloudFront: $20K/month
        - Load Balancer: $2K/month
        Total: ~$75K/month = $900K/year
    
    Stack Overflow approach: $2.82M/year (own hardware + NYC data center)
    Cloud approach: $900K/year (AWS equivalent)
    
    Wait, cloud is cheaper?
        Yes! But Stack Overflow owns hardware (no vendor lock-in)
        Flexibility to optimize exactly how they want
        Learning platform for community
        "We run our own infrastructure because we can" philosophy
    
    Reality: Most companies should use cloud
    Exception: Infrastructure experts like Stack Overflow can optimize bare metal

Cache Layers:
    1. CDN (Cloudflare): 70% of requests never reach origin
    2. Redis (in-memory): 25% of remaining requests hit Redis
    3. SQL Server: Only 5% of requests hit database
    
Example:
    1,000 requests to "What is recursion?" question
    - 700 served from CDN (already cached globally)
    - 250 served from Redis (origin cache hit)
    - 50 served from SQL Server (cache miss, query database)
    
Result: Database handles 5% of traffic (20x reduction)

Stack Overflow Traffic Pattern:
    - Reads (questions, answers, voting): 99.9%
    - Writes (new questions, answers): 0.1%
    
Optimization:
    - 9 web servers (all handle reads)
    - 1 SQL Server (writes)
    - 1 SQL Server (read replica)
    - Heavy caching (reads from cache)
    
Compare to social media:
    - Facebook/Twitter: 50% reads, 50% writes (real-time feeds)
    - Stack Overflow: 99.9% reads (Q&A doesn't change often)
    
Result: 10x fewer servers needed than typical social network

Stack Overflow Philosophy: "Scale up before scaling out"

Each web server:
    - CPU: 32 cores (Intel Xeon)
    - RAM: 64 GB
    - SSD: 1 TB NVMe
    - Cost: $8,000 (one-time hardware)
    
vs Cloud "Scale Out" Philosophy: "Many small servers"
    - AWS m5.large: 2 cores, 8 GB RAM
    - Need 16 m5.large to equal 1 Stack Overflow server
    - Cost: 16 × $70/month = $1,120/month = $13,440/year
    - 3 years: $40,320 vs $8,000 hardware (5x more expensive)
    
Trade-off:
    - Scale up: Fewer servers, simpler management, cheaper (if you can manage)
    - Scale out: More flexible, easier to automate, better for cloud
    
Stack Overflow: Owns infrastructure = scale up makes sense
Most companies: Use cloud = scale out makes sense

Lyft Monolith Architecture:
    Single Python application (Django)
    ↓
    All features in one codebase:
        - Rider app logic
        - Driver app logic
        - Pricing calculations
        - Routing/ETA
        - Payment processing
        - Fraud detection
        - Analytics
    ↓
    PostgreSQL database (everything in one DB)

Scaling Problems:
    1. Deploy entire app to change 1 feature (30+ minute deploys)
    2. Can't scale individual features (scale everything or nothing)
    3. One bug crashes entire app (all users down)
    4. Engineers stepping on each other (merge conflicts daily)
    5. Slow development (100+ engineers, 1 codebase)

Lyft Microservices Architecture (2024):
    10,000+ independent services
    ↓
    Example services:
        - rider-service (100 instances)
        - driver-service (80 instances)
        - pricing-service (50 instances)
        - routing-service (200 instances)
        - payment-service (30 instances)
        - fraud-service (40 instances)
        - eta-service (150 instances)
        - surge-pricing (60 instances)
        ... 9,900+ more services
    ↓
    Each service:
        - Independent codebase
        - Independent deploy (1-2 minute deploys)
        - Independent scaling
        - Different languages (Python, Go, Java, Node.js)

Benefits:
    Fast deploys (1 service, not entire app)
    Independent scaling (scale pricing separately from routing)
    Team autonomy (teams own services)
    Technology choice (use best tool for job)
    Fault isolation (one service down ≠ all down)

New Problems:
    Service discovery (10K services, where is pricing-service?)
    Load balancing (which of 50 pricing-service instances?)
    Retries (pricing-service timeout? Retry? How many times?)
    Circuit breaking (pricing-service down? Stop sending requests?)
    Observability (which service is slow? Where's the bottleneck?)
    Security (service-to-service authentication? Encryption?)

Traditional Reverse Proxy (HAProxy, NGINX):
    External traffic → Proxy → Backend servers
    
Envoy (Service Mesh):
    Every service has sidecar Envoy proxy
    Service A → Envoy A → Envoy B → Service B
    
Key Difference:
    - Traditional: 1 proxy for external traffic
    - Envoy: N proxies (1 per service) for ALL traffic

Request Flow (Rider requests a ride):
    
1. Rider App → API Gateway (Envoy)
    - Authentication (valid user?)
    - Rate limiting (not abusing API?)
    - Routing (which service handles this request?)

2. API Gateway → Rider Service (via Envoy sidecar)
    GET /rider/profile/12345
    
    Rider Service Envoy does:
        - Service discovery (find rider-service instances)
        - Load balancing (pick healthy instance)
        - Retry logic (failed? Try another instance)
        - Timeout (don't wait forever)
        - Metrics (how long did this take?)

3. Rider Service → Pricing Service (via Envoy sidecars)
    POST /pricing/calculate
    {
        "pickup": {"lat": 37.7749, "lon": -122.4194},
        "dropoff": {"lat": 37.8044, "lon": -122.2712},
        "time": "2024-02-15T18:30:00Z"
    }
    
    Pricing Service Envoy does:
        - Service discovery (50 pricing instances available)
        - Load balancing (least-request algorithm)
        - Circuit breaking (if pricing down, fail fast)
        - TLS encryption (secure service-to-service)
        - Tracing (tag request for observability)

4. Pricing Service → Surge Service (via Envoy sidecars)
    GET /surge/current?region=san-francisco
    
    Surge Pricing Service may be slow or down:
        - Envoy timeout: 200ms
        - If timeout: Use cached value (graceful degradation)
        - If circuit open: Fail fast (don't wait)

5. Pricing Service → Rider Service → API Gateway → Rider App
    Response: $23.45 (with surge: 1.8x)
    
    Envoy tracks:
        - Total time: 350ms
        - Breakdown: Rider 50ms, Pricing 180ms, Surge 120ms
        - Bottleneck: Surge service (needs optimization)

# envoy-pricing-service.yaml

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: pricing_service
                  timeout: 5s
                  retry_policy:
                    retry_on: 5xx,gateway-error,connect-failure,refused-stream
                    num_retries: 3
                    per_try_timeout: 1s
          http_filters:
          - name: envoy.filters.http.router

  clusters:
  - name: pricing_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST  # Send to instance with fewest active requests
    health_checks:
    - timeout: 1s
      interval: 5s
      unhealthy_threshold: 2
      healthy_threshold: 2
      http_health_check:
        path: /health
    load_assignment:
      cluster_name: pricing_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: pricing-1.lyft.internal
                port_value: 8080
        - endpoint:
            address:
              socket_address:
                address: pricing-2.lyft.internal
                port_value: 8080
        # ... 48 more pricing service instances
    
    # Circuit Breaker (prevent cascading failures)
    circuit_breakers:
      thresholds:
      - priority: DEFAULT
        max_connections: 1000
        max_pending_requests: 1000
        max_requests: 1000
        max_retries: 3
    
    # Outlier Detection (automatically remove unhealthy instances)
    outlier_detection:
      consecutive_5xx: 5              # 5 errors in a row = ejected
      interval: 10s                   # Check every 10 seconds
      base_ejection_time: 30s         # Remove for 30 seconds minimum
      max_ejection_percent: 50        # Don't eject more than 50% of instances

Scenario: Pricing service instance has transient error

Without Envoy:
    Rider Service → Pricing Service (error)
    Return error to user: "Unable to calculate price. Try again."
    User experience: Frustrating, may abandon ride

With Envoy Retry:
    Attempt 1: Pricing-1 (500 error) - failed
    Wait: 25ms (exponential backoff)
    Attempt 2: Pricing-2 (500 error) - failed
    Wait: 50ms (exponential backoff)
    Attempt 3: Pricing-3 (200 OK) - success!
    Return: $23.45 to user
    User experience: Seamless, didn't notice error

Envoy automatically retried 3 times (took 75ms extra)
User got result without seeing error
Success rate: Went from 95% to 99.8% (retries saved 4.8% of requests)

Scenario: Surge pricing service is completely down

Without Circuit Breaker:
    1. Pricing service calls Surge service
    2. Wait 5 seconds (timeout)
    3. Retry 3 times (15 seconds total wasted)
    4. All pricing requests take 15+ seconds
    5. Pricing service queues back up
    6. Rider service queues back up
    7. API gateway queues back up
    8. All of Lyft is slow/down (cascading failure)

With Envoy Circuit Breaker:
    1. Envoy detects: 5 consecutive errors to Surge service
    2. Open circuit (stop sending requests to Surge)
    3. Fail fast: Return cached surge value (or 1.0x, no surge)
    4. Pricing responds in 200ms (not 15 seconds)
    5. Rider gets price (may be slightly outdated surge)
    6. User can still request ride
    7. Envoy periodically retries Surge (every 30s)
    8. When Surge recovers, circuit closes automatically

Result: Surge service down, but Lyft still works
User experience: Slight inaccuracy (1.5x surge shown as 1.0x)
Alternative: Entire Lyft down (unacceptable)

Old Approach (Hardcoded IPs):
    pricing_service = [
        "10.0.1.101:8080",
        "10.0.1.102:8080",
        "10.0.1.103:8080"
    ]
    
    Problem: Add server? Update config, redeploy all services
    Reality: 10K services × 100 instances = 1M config changes/day

Envoy + Consul (Service Registry):
    1. Pricing service starts
    2. Registers with Consul: "I'm pricing-4 at 10.0.1.104:8080"
    3. Envoy queries Consul: "Where are pricing service instances?"
    4. Consul responds: "50 instances at [IPs]"
    5. Envoy updates internal routing table
    6. If instance crashes, Consul detects (heartbeat missed)
    7. Envoy automatically removes from rotation
    8. New instance? Envoy detects in <5 seconds

Result: Zero manual configuration for service scaling
Deploy 10 new pricing instances? Envoy finds them automatically
Remove 5 old instances? Envoy stops routing in 5 seconds

Lyft Request Trace (via Envoy):

Trace ID: abc-123-def-456 (same ID across all services)

Span 1: API Gateway → Rider Service
    Start: 0ms
    End: 50ms
    Duration: 50ms
    Status: 200 OK

Span 2: Rider Service → Pricing Service
    Start: 10ms
    End: 250ms
    Duration: 240ms
    Status: 200 OK
    
Span 3: Pricing Service → Surge Service
    Start: 50ms
    End: 180ms
    Duration: 130ms
    Status: 200 OK

Span 4: Pricing Service → ETA Service
    Start: 50ms
    End: 230ms
    Duration: 180ms ← BOTTLENECK!
    Status: 200 OK
    
Span 5: Pricing Service → Rider Service → API Gateway
    Total: 250ms

Analysis:
    - ETA Service is slowest (180ms)
    - Surge Service is acceptable (130ms)
    - Optimize ETA Service = faster pricing

Before Envoy:
    "Pricing is slow, why?"
    Engineers guess, debug for hours

After Envoy:
    "ETA service is the bottleneck (180ms p95)"
    Engineers optimize ETA service immediately

Before Envoy (2015):
    - Deploy time: 30+ minutes (entire monolith)
    - Scaling: All or nothing (can't scale one feature)
    - Reliability: 95% (cascading failures common)
    - Debugging: Hours to find slow service
    - Team velocity: Slow (merge conflicts, coordination)

After Envoy (2018-2024):
    - Deploy time: 1-2 minutes (single service)
    - Scaling: Per-service (scale pricing independently)
    - Reliability: 99.9%+ (circuit breakers prevent cascades)
    - Debugging: Minutes (distributed tracing shows bottlenecks)
    - Team velocity: Fast (teams deploy independently)

Specific Improvements:
    - P95 latency: 1,200ms → 350ms (71% faster)
    - Success rate: 95% → 99.8% (retries recovered 4.8% of requests)
    - Cascading failures: 10/month → 0/month (circuit breakers work)
    - Mean time to recovery: 30 minutes → 5 minutes (auto-detection)
    - Engineering efficiency: 100 engineers → 2,000 engineers (20x scale)

Infrastructure Costs:
    - Envoy overhead: +10% CPU, +5% latency (acceptable trade-off)
    - Benefit: 4.8% more successful requests (saved $50M+/year in lost rides)
    - Net: Spent $5M/year on Envoy infrastructure, saved $50M+/year
    - ROI: 10x return on investment

HAProxy/NGINX:
    Extremely fast (C++ optimized for performance)
    Mature (20+ years of production hardening)
    Simple configuration (for basic use cases)
    Designed for edge load balancing (not service mesh)
    Limited observability (basic logs, no distributed tracing)
    Manual service discovery (static configuration)
    No built-in retries, circuit breakers for complex scenarios

Envoy:
    Designed for microservices (service mesh first)
    Advanced observability (distributed tracing, rich metrics)
    Dynamic service discovery (integrates with Consul, Kubernetes)
    Built-in retries, circuit breakers, outlier detection
    HTTP/2 and gRPC first-class support
    More complex configuration (YAML can be verbose)
    Higher resource usage (+10% CPU vs HAProxy)
    Newer (less production time than HAProxy/NGINX)

When to use each:
    - HAProxy: Edge load balancing, need maximum performance
    - NGINX: Web server + reverse proxy, simple use cases
    - Envoy: Microservices architecture, need observability

Web Application Attacks (2024):
    - DDoS attacks: 15 million/year (41,000/day globally)
    - Average attack size: 30 Gbps (largest: 71M requests/second)
    - Bot traffic: 47% of all web traffic (28% malicious bots)
    - Web application attacks: 10 billion/year (OWASP Top 10)
    - Cost per breach: $4.45M average (IBM 2024 report)
    
Security is not optional: It's survival

Comparison to Normal Traffic:
    - Twitter: ~20,000 tweets/second average
    - Google: ~100,000 searches/second
    - Netflix: ~800,000 streams/second
    - Facebook: ~2M likes/second
    - This DDoS: 71,000,000 requests/second

Scale:
    - Equivalent to 71 million people hitting refresh simultaneously
    - 4.2 billion requests per minute
    - 252 billion requests per hour (if sustained)
    - Larger than most websites' entire monthly traffic

Why Didn't Target Go Down?
    - Cloudflare absorbed entire attack (distributed across 310+ datacenters)
    - Customer's origin servers: Received 0 attack traffic
    - Automatic mitigation: No human intervention required
    - Attack detected and blocked: <3 seconds

Cloudflare learns normal patterns for each customer:

Normal Traffic (Financial Services Site):
    - Requests: 5,000 req/sec average
    - Peak hours: 8,000 req/sec (market open)
    - Geographic: 80% USA, 15% Europe, 5% Asia
    - User-Agent: 60% Chrome, 25% Safari, 10% Firefox, 5% mobile
    - HTTP methods: 95% GET, 4% POST, 1% other
    - Response codes: 85% 200 OK, 10% 304 Not Modified, 5% other

Attack Traffic (Anomaly Detected):
    - Requests: 71,000,000 req/sec (14,200x normal!)
    - Geographic: Distributed (121 countries, unusual)
    - User-Agent: Suspicious patterns (IoT devices, old browsers)
    - HTTP methods: 100% GET (attacking specific endpoint)
    - Response codes: Many 503 (origin struggling)
    - Request pattern: Identical requests (not natural variation)

Cloudflare's System:
    Time 0:00 - Normal traffic (5,000 req/sec)
    Time 0:05 - Traffic spike detected (50,000 req/sec, +900%)
    Time 0:06 - Anomaly confirmed (1M req/sec, characteristics match DDoS)
    Time 0:07 - Mitigation activated (challenge suspicious requests)
    Time 0:08 - Origin protected (attack traffic blocked at edge)
    Time 0:30 - Attack ends (71M peak req/sec absorbed)
    Time 0:31 - Normal traffic resumed (5,000 req/sec)
    
Total outage: 0 seconds (customers never noticed)

Layer 1: Network (Bandwidth Absorption)
    - Cloudflare network: 100+ Tbps capacity
    - Attack bandwidth: 1.2 Tbps (71M × 17 KB average)
    - Utilization: 1.2% of total capacity (easily absorbed)
    - Anycast routing: Attack distributed across 310+ locations
    
    Why it works: 
        Attack targets 1 IP address
        Cloudflare's Anycast: Same IP announced from 310 locations
        Result: Attack traffic split 310 ways (230K req/sec per location)
        Each location easily handles 230K req/sec (normal capacity: 10M+)

Layer 2: Rate Limiting (Per-IP Throttling)
    - Legitimate user: 1-10 requests/second (normal browsing)
    - Attack IP: 1,000+ requests/second (clearly malicious)
    - Action: Block IPs exceeding 100 req/sec threshold
    - Result: 99% of attack traffic blocked at this layer

Layer 3: Challenge/Response (Prove You're Human)
    - Suspicious traffic: JavaScript challenge
    - Bot: Can't execute JavaScript (blocked)
    - Human: Challenge completes automatically (allowed)
    - Overhead: 200ms delay (acceptable for security)
    
    Example JavaScript Challenge:
        Browser receives: <script>
            var a = 123, b = 456;
            document.cookie = "proof=" + (a + b);
            window.location.reload();
        </script>
        
        Real browser: Executes, sets cookie, passes
        Bot: Can't execute JavaScript, blocked

Layer 4: CAPTCHA (Last Resort)
    - Persistent attackers: Shown CAPTCHA
    - Solve puzzle: Access granted
    - Fail puzzle: Blocked
    - Used for: <1% of traffic (most automated attacks blocked earlier)

Layer 5: Behavioral Analysis (Machine Learning)
    - ML models trained on 28 million websites
    - Patterns learned: Attack vs legitimate traffic
    - Real-time scoring: Each request gets risk score 0-100
    - Threshold: Score >80 = blocked, 50-80 = challenged, <50 = allowed
    
    Attack Request Analysis:
        - No referrer header: +20 points (suspicious)
        - User-Agent: IoT device: +30 points (unusual for finance site)
        - Request rate: 1000/sec: +40 points (bot-like)
        - Total score: 90 → BLOCKED

Without Origin Shield:
    Attack: 71M req/sec → Cloudflare edge → Origin servers
    Even if 99.9% blocked: 71,000 req/sec hit origin
    Origin capacity: 10,000 req/sec
    Result: Origin overwhelmed, site down

With Origin Shield (Cloudflare's Implementation):
    Attack: 71M req/sec → Cloudflare edge (310 locations)
                        → Origin shield (1 regional cache)
                        → Origin servers
    
    Edge layer: Blocks 99.99% (70,993,000 req/sec blocked)
    Origin shield: Receives 7,000 req/sec (handles easily)
    Cache hit: 90% (6,300 req/sec served from shield cache)
    Origin: Receives 700 req/sec (well below capacity)
    
    Result: Origin never stressed, site stays up

DIY DDoS Protection (Without CDN):
    Infrastructure:
        - 100 Gbps DDoS scrubbing: $50,000/month
        - Load balancers: $10,000/month
        - Failover infrastructure: $20,000/month
    Staff:
        - 24/7 security team: $500,000/year (4 engineers)
        - Incident response: $200,000/year
    Total: ~$1.16M/year minimum

Cloudflare DDoS Protection:
    - Free tier: Unmetered DDoS protection (no size limit)
    - Pro tier: $20/month (small business)
    - Business tier: $200/month (medium business)
    - Enterprise tier: Custom ($2,000-10,000/month typical)
    
    Customer in 71M attack: Likely paid <$10K/month
    DIY equivalent: $1.16M/year ($96K/month)
    Savings: ~$86K/month = $1M+/year

How Cloudflare Offers Free DDoS Protection:
    1. Economics of scale (28M customers share infrastructure)
    2. Anycast architecture (attacks distributed automatically)
    3. Excess capacity (100 Tbps network, average use <10 Tbps)
    4. Bot traffic subsidizes humans (95% of DDoS traffic filtered with minimal cost)

Goal: Saturate bandwidth

UDP Flood:
    - Attacker sends millions of UDP packets
    - Target server: Must process each packet
    - Bandwidth: 100 Gbps attack saturates 10 Gbps link
    - Result: Legitimate traffic can't get through
    
    Defense: 
        - CDN with 100+ Tbps capacity (absorb flood)
        - Rate limiting (drop excessive UDP)
        - Anycast (distribute across 310 locations)

DNS Amplification:
    - Attacker spoofs target's IP address
    - Sends DNS query to open resolvers
    - Response 100x larger than query
    - Result: Target overwhelmed with DNS responses
    
    Example:
        Query: 60 bytes ("What's IP of example.com?")
        Response: 4,000 bytes (full DNS record + DNSSEC)
        Amplification: 67x
        
        1 Gbps of queries → 67 Gbps of responses to target
    
    Defense:
        - Block spoofed IPs (BCP 38 filtering)
        - Rate limit DNS responses
        - Use CDN (absorb amplified traffic)

Goal: Consume server connections

SYN Flood:
    - TCP handshake: SYN → SYN-ACK → ACK
    - Attacker sends millions of SYN packets
    - Server allocates memory for each half-open connection
    - Never sends final ACK (connections stay open)
    - Result: Server runs out of memory, rejects legitimate connections
    
    Defense:
        - SYN cookies (stateless TCP handshake)
        - Connection timeout (drop half-open after 10 seconds)
        - Rate limiting (limit SYNs per IP)

Slowloris:
    - Open connections to server
    - Send partial HTTP requests (very slowly)
    - Keep connections alive indefinitely
    - Server waits for complete request
    - Result: All connection slots filled, no room for legitimate users
    
    Defense:
        - Connection timeout (close slow connections)
        - Reverse proxy (buffer requests before backend)
        - NGINX: Default timeout 60s (closes slow connections)

Goal: Exhaust application resources

HTTP Flood:
    - Send valid HTTP requests (hard to distinguish from legitimate)
    - Target expensive endpoints (search, database queries)
    - Example: GET /search?q=* (searches everything)
    - Result: Database overwhelmed, site slow/down
    
    The 71M req/sec attack: This type (HTTPS flood)
    
    Defense:
        - Rate limiting (per IP, per endpoint)
        - CAPTCHA challenges (prove human)
        - Behavioral analysis (bot detection)
        - Caching (serve from cache, not database)

Slowread:
    - Opposite of Slowloris
    - Request large file
    - Receive response very slowly (1 byte/second)
    - Keep connection open for hours
    - Result: Exhaust connection pool
    
    Defense:
        - Minimum connection speed (close if too slow)
        - Timeout idle connections
        - Connection limits per IP

Without Rate Limiting:
    Scenario: Poorly written script
        for i in range(1000000):
            response = requests.get("https://api.github.com/users/torvalds")
        
    Impact:
        - 1 user makes 1M requests in 10 minutes
        - API servers overwhelmed
        - Database saturated
        - ALL users affected (slow/down)
        - Cost: $10K+ in wasted infrastructure

With Rate Limiting:
    Same script runs:
        - Requests 1-5,000: Accepted
        - Request 5,001: 
            HTTP 403 Forbidden
            { "message": "API rate limit exceeded" }
        - Script stops (error handling)
        - Other users unaffected

User Type Limits (per hour):

Unauthenticated (IP-based):
    - Limit: 60 requests/hour
    - Use case: Anonymous browsing, documentation
    - Reset: Every hour at :00
    - Reason: Prevent abuse while allowing public access

Authenticated (Token-based):
    - Limit: 5,000 requests/hour
    - Use case: Normal development workflow
    - Reset: Rolling window (not fixed hour)
    - Reason: Generous limit for legitimate use

GitHub Apps:
    - Limit: 15,000 requests/hour
    - Use case: CI/CD, automation tools
    - Reset: Rolling window
    - Reason: Higher limits for business tools

Enterprise:
    - Limit: Negotiable (50,000+ requests/hour)
    - Use case: Large organizations
    - Custom: Based on usage patterns
    - Cost: $21+/user/month

Every API response includes rate limit info:

HTTP/1.1 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4999
X-RateLimit-Reset: 1677721200
X-RateLimit-Used: 1
X-RateLimit-Resource: core

Headers explained:
    - Limit: 5000 = Your total limit per hour
    - Remaining: 4999 = Requests left this window
    - Reset: 1677721200 = Unix timestamp when limit resets
    - Used: 1 = Requests used so far
    - Resource: core = Which limit bucket (core, search, graphql)

Developer experience:
    Check remaining before making requests
    If remaining < 10: Wait until reset
    Prevents hitting limit unexpectedly

GitHub has separate limits for different resources:

Core API (General):
    - Limit: 5,000/hour
    - Endpoints: repos, issues, pulls, users
    - Most requests use this bucket

Search API (Expensive):
    - Limit: 30/minute (720/hour)
    - Endpoints: /search/repositories, /search/code
    - Why lower: Database-intensive queries
    - Example: Search all code for "TODO" = expensive

GraphQL API (Flexible):
    - Limit: Points-based, not request-based
    - Simple query: 1 point
    - Complex query: 100+ points
    - Total: 5,000 points/hour
    - Why: Fairer (simple queries don't count as much)

Example GraphQL Points:
    query {
        repository(owner: "github", name: "hub") {
            name  # 1 point
        }
    }
    Cost: 1 point
    
    query {
        repository(owner: "github", name: "hub") {
            issues(first: 100) {  # 100 points
                edges {
                    node {
                        comments(first: 50) {  # 50 points per issue
                            edges {
                                node { body }
                            }
                        }
                    }
                }
            }
        }
    }
    Cost: 100 + (100 × 50) = 5,100 points → DENIED (over limit)

Problem:
    - Docker Hub: Container image registry
    - Free tier: Unlimited pulls (no rate limit)
    - Abuse: Automated systems pulling terabytes daily
    - Cost: Docker paying $millions in bandwidth
    - Decision: Implement rate limiting (November 2020)

New Limits (2020):
    - Anonymous: 100 pulls per 6 hours per IP
    - Free account: 200 pulls per 6 hours
    - Pro account: Unlimited pulls
    - Cost: Pro = $5/month

Developer Reaction:
    - Outrage: "Breaking CI/CD pipelines!"
    - Reality: Most developers <20 pulls/day (under limit)
    - Problem: Corporate CI/CD behind 1 NAT IP = hundreds of developers sharing limit
    
Corporate Impact Example:
    - Company: 500 developers
    - All behind 1 IP: Share 100 pulls per 6 hours
    - CI pipeline: 10 pulls per build
    - Result: 10 builds every 6 hours (not enough)
    - Solution: Pay for Pro account (unlimited)

Lessons Learned:
    1. Rate limiting saves costs ($millions for Docker)
    2. Implement gradually (warn users first)
    3. Exempt legitimate use (whitelist known good actors)
    4. Consider shared IPs (corporate NAT problem)

Implementation:
    - Window: 1 hour (0:00-1:00, 1:00-2:00, etc.)
    - Limit: 5,000 requests per window
    - Counter resets at window boundary
    
Example:
    12:59:59 - Request 5,000 (last request of hour)
    13:00:00 - Counter resets to 0
    13:00:01 - Request 5,001 allowed (new window)
    
Burst Problem:
    12:30:00 - Made 5,000 requests (hit limit)
    12:59:59 - Wait 1 second...
    13:00:00 - Can make 5,000 more requests
    Total: 10,000 requests in 30 seconds! (burst)

Pros:
    Simple to implement
    Low memory (just counter + timestamp)
    
Cons:
    Burst problem (2x limit at window boundary)
    Unfair (user A at 0:01 vs user B at 0:59)

Implementation:
    - Log timestamp of each request
    - Check last hour of timestamps
    - Count requests in sliding 60-minute window
    
Example:
    13:45:00 - Check requests from 12:45:00 to 13:45:00
    13:45:01 - Check requests from 12:45:01 to 13:45:01 (slides)
    
No burst problem:
    12:59:59 - Made 5,000 requests (hit limit)
    13:00:00 - Still have 5,000 requests from 12:00-13:00
    13:00:01 - Request from 12:00:01 expires (now have 4,999)
    Gradual: Limit releases 1 request at a time

Pros:
    No burst problem (true sliding window)
    Fair (exact 60-minute window always)
    
Cons:
    Memory intensive (store all timestamps)
    5,000 limit = store 5,000 timestamps = ~100 KB per user
    100M users = 10 TB memory needed!

Implementation:
    - Bucket holds tokens (max 5,000)
    - Tokens refill at rate (e.g., 83.33/minute = 5,000/hour)
    - Each request consumes 1 token
    - No tokens = request denied
    
Example:
    Bucket: 5000 tokens (full)
    Request 1: 4999 tokens left
    Wait 1 minute: 4999 + 83.33 = 5,082.33 tokens
    Cap at max: 5,000 tokens (can't go over)
    
Burst allowed (but limited):
    Start with 5,000 tokens (full bucket)
    Make 5,000 requests instantly (empty bucket)
    Wait 1 hour: Bucket refills to 5,000
    Can't exceed 5,000 (bucket size is hard limit)
    
    Burst: 5,000 in 1 second  (but then must wait)
    vs Fixed window: 10,000 in 1 second  (at boundary)

Pros:
    Allows short bursts (natural traffic pattern)
    Low memory (just 2 numbers: tokens + timestamp)
    Smooth over time (refill rate prevents abuse)
    
Cons:
    Slight burst possible (by design, often desired)
    
GitHub uses this algorithm (token bucket)

# /etc/nginx/nginx.conf

http {
    # Define rate limit zone (token bucket algorithm)
    # Zone name: api_limit
    # Key: $binary_remote_addr (client IP)
    # Size: 10m (10 megabytes = ~160K IP addresses)
    # Rate: 5000r/h (5000 requests per hour = 83.33/minute)
    limit_req_zone $binary_remote_addr zone=api_limit:10m rate=83r/m;
    
    server {
        listen 443 ssl http2;
        server_name api.example.com;
        
        location /api/ {
            # Apply rate limit
            # Zone: api_limit (defined above)
            # Burst: 100 (allow burst of 100 requests above rate)
            # nodelay: Don't delay burst requests (process immediately)
            limit_req zone=api_limit burst=100 nodelay;
            
            # Custom error response for rate limit
            limit_req_status 429;  # HTTP 429 Too Many Requests
            
            # Add rate limit headers to response
            add_header X-RateLimit-Limit "5000" always;
            add_header X-RateLimit-Remaining $limit_req_remaining always;
            
            proxy_pass http://backend;
        }
    }
}

# Test GitHub API rate limit
curl -I https://api.github.com/users/octocat

# Response headers:
# HTTP/2 200
# X-RateLimit-Limit: 60
# X-RateLimit-Remaining: 59
# X-RateLimit-Reset: 1677721200

# With authentication (higher limit):
curl -H "Authorization: token YOUR_TOKEN" \
     -I https://api.github.com/users/octocat

# Response:
# X-RateLimit-Limit: 5000
# X-RateLimit-Remaining: 4999

# Exceed limit test:
for i in {1..100}; do
    curl https://api.github.com/users/octocat
done

# After 60 requests (unauthenticated limit):
# HTTP/2 403
# {
#   "message": "API rate limit exceeded for <IP>.",
#   "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"
# }

1. Broken Access Control (34% of applications tested)
2. Cryptographic Failures (formerly Sensitive Data Exposure)
3. Injection (SQL, NoSQL, Command injection)
4. Insecure Design
5. Security Misconfiguration
6. Vulnerable and Outdated Components
7. Identification and Authentication Failures
8. Software and Data Integrity Failures
9. Security Logging and Monitoring Failures
10. Server-Side Request Forgery (SSRF)

Cost of OWASP Top 10 breaches: $4.45M average per incident

Vulnerable Code (PHP):
    $username = $_POST['username'];
    $password = $_POST['password'];
    
    $query = "SELECT * FROM users WHERE username='$username' AND password='$password'";
    $result = mysqli_query($conn, $query);
    
    if (mysqli_num_rows($result) > 0) {
        echo "Login successful!";
    }

Attack:
    Attacker enters:
        Username: admin' OR '1'='1
        Password: anything
    
    Query becomes:
        SELECT * FROM users WHERE username='admin' OR '1'='1' AND password='anything'
        
    '1'='1' is always true
    Result: Bypasses authentication, logs in as admin!
    
Advanced Attack (Extract Data):
    Username: admin' UNION SELECT credit_card, cvv, exp_date FROM payment_cards--
    
    Query becomes:
        SELECT * FROM users WHERE username='admin' 
        UNION SELECT credit_card, cvv, exp_date FROM payment_cards--' AND password='anything'
        
    Result: Attacker gets all credit card data!

Fix 1 - Prepared Statements (Correct):
    $stmt = $conn->prepare("SELECT * FROM users WHERE username=? AND password=?");
    $stmt->bind_param("ss", $username, $password);
    $stmt->execute();
    
    User input treated as data (not SQL code)
    Injection impossible

Fix 2 - WAF (Defense in Depth):
    WAF detects: ' OR '1'='1
    Pattern: Classic SQL injection signature
    Action: Block request before it reaches application
    Response: HTTP 403 Forbidden

ModSecurity Core Rule Set (CRS) - Industry Standard:

Rule 1: SQL Injection Detection
    Pattern: (\bOR\b|\bAND\b).*(=|<|>)
    Examples blocked:
        - ' OR 1=1
        - admin' AND 1=1
        - ' OR 'a'='a
    Paranoia level: 1 (basic)

Rule 2: XSS (Cross-Site Scripting) Detection
    Pattern: &lt;script|javascript&#x3A;|onerror&#x3D;|onclick&#x3D;
    Examples blocked:
        - &lt;script>alert('XSS')</script>
        - &lt;img src=x onerror&#x3D;alert(1)>
        - javascript&#x3A;alert(document.cookie)
    Paranoia level: 1 (basic)

Rule 3: Path Traversal Detection
    Pattern: \.\./|\.\.\\|%2e%2e
    Examples blocked:
        - ../../../../etc/passwd
        - ..\..\windows\system32
        - %2e%2e%2f (URL encoded)
    Paranoia level: 1 (basic)

Rule 4: Command Injection Detection
    Pattern: ;|\||&&|\$\(|\`
    Examples blocked:
        - ; rm -rf /
        - | cat /etc/passwd
        - && shutdown -h now
    Paranoia level: 1 (basic)

Rule 5: File Upload Restrictions
    Pattern: \.php$|\.exe$|\.sh$
    Examples blocked:
        - malware.php (web shell)
        - virus.exe (executable)
        - backdoor.sh (shell script)
    Action: Block dangerous file extensions

Rule 6: Rate Limiting (Application Layer)
    Pattern: Same IP, same endpoint, >100 req/min
    Examples blocked:
        - Credential stuffing (try 1000 passwords)
        - Comment spam (post 100 comments)
        - Scraping (crawl entire site)
    Action: Block IP for 10 minutes

Real Attack Blocked (SQL Injection Attempt):

Request:
    POST /login HTTP/1.1
    Host: example.com
    Content-Type: application/x-www-form-urlencoded
    
    username=admin' OR '1'='1'--&password=test

Cloudflare WAF Analysis:
    1. Parse request body
    2. Check username parameter: "admin' OR '1'='1'--"
    3. Match SQL injection pattern: ' OR '1'='1
    4. Severity: High (OWASP Top 10 #3)
    5. Action: Block (configured by site owner)

Response to Attacker:
    HTTP/1.1 403 Forbidden
    Server: cloudflare
    
    <html>
    <body>
    <h1>Access Denied</h1>
    <p>This request has been blocked by our Web Application Firewall.</p>
    <p>If you believe this is an error, please contact support.</p>
    <p>Ray ID: 12345abc (for debugging)</p>
    </body>
    </html>

Legitimate Application:
    Never receives malicious request
    Protected from SQL injection
    No code changes needed

Alert to Security Team:
    Time: 2024-02-15 10:30:45 UTC
    Event: SQL injection attempt blocked
    Source IP: 203.0.113.45 (Russia)
    Target: /login
    Payload: admin' OR '1'='1'--
    Action: Blocked
    Ray ID: 12345abc

Problem: Legitimate requests blocked by WAF

Example 1 - SQL in Content:
    Blog post about databases:
        Title: "10 Best Practices for SQL Queries"
        Content: "Always use SELECT * FROM users WHERE id=? instead of string concatenation"
        
    WAF sees: "SELECT * FROM users WHERE"
    Match: SQL injection pattern
    Result: Legitimate blog post blocked!
    
    Solution:
        - WAF whitelist for /admin/blog/create endpoint
        - Lower paranoia level for authenticated admins
        - Inspect context (POST to /blog vs /login)

Example 2 - Code in Comments:
    Developer forum discussion:
        Comment: "My <script> tag isn't working, help!"
        
    WAF sees: "<script>"
    Match: XSS pattern
    Result: Legitimate question blocked!
    
    Solution:
        - HTML entity encoding (<script> → <script>)
        - WAF exception for authenticated users
        - User education (paste code in code blocks)

Example 3 - International Names:
    User registration:
        Name: "O'Brien" (Irish surname)
        
    WAF sees: O'Brien (contains single quote)
    Match: SQL injection pattern (' character suspicious)
    Result: Legitimate user can't register!
    
    Solution:
        - Context-aware rules (name field vs SQL query field)
        - Allow single quotes in form fields (but not URL parameters)
        - Validate input format (letters, hyphen, apostrophe only)

Balancing Security vs Usability:
    - Too strict: Block legitimate users (false positives)
    - Too loose: Allow attacks through (false negatives)
    - Sweet spot: Block 99.9% attacks, allow 99.9% legitimate
    - Tuning required: Monitor false positives, adjust rules

# /etc/nginx/nginx.conf

load_module modules/ngx_http_modsecurity_module.so;

http {
    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsecurity/modsecurity.conf;
    
    server {
        listen 443 ssl;
        server_name example.com;
        
        location / {
            # Enable ModSecurity for this location
            modsecurity on;
            
            # Custom rule: Block SQL injection
            modsecurity_rules '
                SecRule ARGS "@rx (\bOR\b|\bAND\b).*(=|<|>)" \
                    "id:1001,\
                    phase:2,\
                    deny,\
                    status:403,\
                    msg:\'SQL Injection Detected\',\
                    log,\
                    auditlog"
            ';
            
            proxy_pass http://backend;
        }
        
        location /admin {
            # Higher security for admin area
            modsecurity on;
            
            # Load OWASP Core Rule Set
            modsecurity_rules_file /etc/nginx/modsecurity/crs/crs-setup.conf;
            modsecurity_rules_file /etc/nginx/modsecurity/crs/rules/*.conf;
            
            proxy_pass http://admin_backend;
        }
    }
}

Benchmarks (ModSecurity with OWASP CRS):

Without WAF:
    - Requests/second: 50,000
    - Average latency: 10ms
    - P95 latency: 20ms
    - CPU usage: 40%

With WAF (Basic Rules):
    - Requests/second: 45,000 (-10%)
    - Average latency: 12ms (+20%)
    - P95 latency: 25ms (+25%)
    - CPU usage: 55% (+15%)

With WAF (Full OWASP CRS):
    - Requests/second: 35,000 (-30%)
    - Average latency: 15ms (+50%)
    - P95 latency: 35ms (+75%)
    - CPU usage: 70% (+30%)

Trade-off Analysis:
    Cost: 30% performance reduction
    Benefit: 99.9%+ attack protection
    ROI: 1 prevented breach ($4.45M) >> performance cost ($10K/month servers)
    
Optimization:
    - Run WAF on edge (Cloudflare, not origin)
    - Cache rule evaluations (same patterns repeat)
    - Selective rules (only for sensitive endpoints)
    - Hardware acceleration (specialized WAF appliances)

Essential Security Headers:

1. Strict-Transport-Security (HSTS)
    Header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
    
    What it does:
        - Forces HTTPS (browsers refuse HTTP)
        - Prevents MITM attacks (SSL stripping)
        - Lasts 1 year (31536000 seconds)
    
    Without HSTS:
        User types: http://bank.com
        Browser: Connects to HTTP first
        Attacker: Intercepts, blocks HTTPS redirect
        Result: User on HTTP (credentials stolen)
    
    With HSTS:
        User types: http://bank.com
        Browser: "I remember, this site requires HTTPS"
        Browser: Automatically upgrades to HTTPS
        Attacker: Can't intercept (HTTPS encrypted)
        Result: User protected

2. Content-Security-Policy (CSP)
    Header: Content-Security-Policy: default-src 'self'; script-src 'self' https://cdn.example.com
    
    What it does:
        - Controls where resources can load from
        - Prevents XSS attacks (blocks inline scripts)
        - Whitelist trusted domains
    
    XSS Attack Blocked by CSP:
        Attacker injects: &lt;script>alert(document.cookie)</script>
        Browser: "CSP policy forbids inline scripts"
        Browser: Blocks script execution
        Result: XSS attack fails
    
    Allowed sources:
        - 'self' = same domain only
        - https://cdn.example.com = trusted CDN
        - 'unsafe-inline' = allow inline (dangerous, avoid!)

3. X-Frame-Options
    Header: X-Frame-Options: DENY
    
    What it does:
        - Prevents clickjacking attacks
        - Stops site from being embedded in <iframe>
    
    Clickjacking Attack:
        Attacker site:
            <iframe src="https://bank.com/transfer" style="opacity:0"></iframe>
            <button>Click to win $1000!</button>
        
        User clicks "win button"
        Actually clicking hidden bank transfer button
        Result: Money transferred to attacker
    
    With X-Frame-Options: DENY:
        Browser: "This site can't be framed"
        Browser: Refuses to load in iframe
        Result: Clickjacking impossible

4. X-Content-Type-Options
    Header: X-Content-Type-Options: nosniff
    
    What it does:
        - Prevents MIME type sniffing
        - Forces browser to respect Content-Type header
    
    MIME Sniffing Attack:
        Attacker uploads: malicious.jpg (actually HTML with &lt;script>)
        Server: Content-Type: image/jpeg
        Browser (without nosniff): "This looks like HTML, I'll execute it"
        Result: XSS attack via image upload
    
    With nosniff:
        Browser: "Content-Type says image/jpeg, I'll treat it as image"
        Browser: Won't execute as HTML
        Result: Attack fails

5. Referrer-Policy
    Header: Referrer-Policy: strict-origin-when-cross-origin
    
    What it does:
        - Controls what Referer header is sent
        - Prevents leaking sensitive URLs
    
    Without Referrer-Policy:
        User: https://example.com/account/user123/transactions?month=2024-02
        Clicks link to: https://external-site.com
        Referer sent: https://example.com/account/user123/transactions?month=2024-02
        External site: Sees full URL (including user ID!)
    
    With strict-origin-when-cross-origin:
        Same domain: Send full URL
        Cross-domain: Send only origin (https://example.com)
        Result: Privacy protected

6. Permissions-Policy
    Header: Permissions-Policy: geolocation=(), microphone=(), camera=()
    
    What it does:
        - Controls browser features available
        - Prevents unauthorized access to sensors
    
    Example:
        Malicious ad: <iframe src="ad.html"></iframe>
        Ad script: navigator.geolocation.getCurrentPosition()
        Without policy: Gets user location
        With policy: Permission denied
        Result: Privacy protected

# /etc/nginx/nginx.conf

server {
    listen 443 ssl http2;
    server_name example.com;
    
    # SSL/TLS Configuration
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    
    # Security Headers (copy-paste ready)
    
    # HSTS (force HTTPS for 1 year)
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    
    # Prevent clickjacking
    add_header X-Frame-Options "DENY" always;
    
    # Prevent MIME sniffing
    add_header X-Content-Type-Options "nosniff" always;
    
    # XSS Protection (legacy, but still useful)
    add_header X-XSS-Protection "1; mode=block" always;
    
    # Referrer Policy (privacy)
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    
    # Content Security Policy (adjust for your needs)
    add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' https://cdn.example.com; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self'; frame-ancestors 'none';" always;
    
    # Permissions Policy (restrict features)
    add_header Permissions-Policy "geolocation=(), microphone=(), camera=()" always;
    
    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name example.com;
    return 301 https://$server_name$request_uri;
}

# Check headers with curl
curl -I https://example.com

# Output:
# HTTP/2 200
# strict-transport-security: max-age=31536000; includeSubDomains; preload
# x-frame-options: DENY
# x-content-type-options: nosniff
# content-security-policy: default-src 'self'; ...
# permissions-policy: geolocation=(), microphone=(), camera=()

# Security scan (online tools):
# - https://securityheaders.com
# - https://observatory.mozilla.org

# Example scan result:
# securityheaders.com grade: A+
# - HSTS:  Enabled
# - CSP:  Enabled
# - X-Frame-Options:  Enabled
# - X-Content-Type-Options:  Enabled
# - Referrer-Policy:  Enabled

Study: Scott Helme (2024)
Sample: Top 1 million websites

Security Headers Adoption:
    - HSTS: 25% of sites (improving from 15% in 2020)
    - CSP: 8% of sites (most with weak policies)
    - X-Frame-Options: 40% of sites
    - X-Content-Type-Options: 35% of sites
    - Referrer-Policy: 12% of sites

Grade Distribution:
    - A+ grade: 1.2% of sites (12,000)
    - A grade: 3.5% of sites (35,000)
    - B grade: 10% of sites (100,000)
    - C or below: 50% of sites (500,000)
    - No headers: 35% of sites (350,000)

Cost to implement: $0 (just configuration)
Time to implement: 15 minutes (add headers to nginx.conf)
Protection gained: Major (XSS, clickjacking, MITM, MIME sniffing)

Sites with A+ grade:
    - GitHub: A+
    - Google: A+
    - Facebook: A+
    - Netflix: A
    - Twitter: A
    - Most sites: F (no headers)

Your site: Should be A+ (no excuse for F)

Google Research (2023):
    - 1 second delay: -7% conversions
    - 2 second delay: -20% conversions
    - 3 second delay: -40% conversions
    - >3 seconds: 53% mobile users abandon

Amazon Research (2019):
    - Every 100ms delay: -1% revenue
    - 1 second slower: -$1.6 billion/year lost

Pinterest (2017):
    - 40% faster load time: +15% signups
    - Performance improvement: +15% SEO traffic

Performance = Revenue

Typical Shopify Product Page (Before):

Desktop (1920×1080 screen):
    - Hero image: 5 MB (4000×3000 pixels, 8MP camera photo)
    - Thumbnail 1: 3 MB (3000×2000 pixels)
    - Thumbnail 2: 4 MB (3500×2500 pixels)
    ... 9 more thumbnails (3-5 MB each)
    Total: 45 MB page weight
    Load time: 15 seconds on 25 Mbps connection
    
Mobile (390×844 iPhone screen):
    - Same 5 MB hero image (displayed at 390px width!)
    - Wasted: 4000px downloaded, only 390px displayed
    - Bandwidth waste: 90%+ (downloading 10x more pixels than needed)
    - Load time: 30+ seconds on 4G (slower connection)

Impact:
    - 53% mobile users abandon (>3 second load)
    - Conversion rate: 2.3% (industry average: 3.5%)
    - Lost revenue: $200M+/year (calculated from traffic × abandoned carts)

Before (Manual):
    Merchant uploads: product.jpg (5 MB, 4000×3000)
    Shopify stores: product.jpg (5 MB)
    User on mobile: Downloads 5 MB (displays at 390px)
    Wasted: 4.7 MB bandwidth

After (Automatic):
    Merchant uploads: product.jpg (5 MB, 4000×3000)
    Shopify generates:
        - product_large.jpg (2000×1500, 800 KB) - Desktop hero
        - product_medium.jpg (1200×900, 350 KB) - Tablet
        - product_small.jpg (800×600, 180 KB) - Mobile
        - product_thumb.jpg (400×300, 60 KB) - Thumbnails
    
    User on mobile: Downloads 180 KB (not 5 MB)
    Savings: 96% bandwidth (5 MB → 180 KB)
    
HTML Implementation (Responsive Images):
    <img 
      srcset="product_small.jpg 400w,
              product_medium.jpg 800w,
              product_large.jpg 1200w"
      sizes="(max-width: 600px) 400px,
             (max-width: 1200px) 800px,
             1200px"
      src="product_medium.jpg"
      alt="Product photo"
      loading="lazy"
    />
    
    Browser automatically selects correct size:
        - 390px iPhone: Loads product_small.jpg (180 KB)
        - 768px iPad: Loads product_medium.jpg (350 KB)
        - 1920px desktop: Loads product_large.jpg (800 KB)

JPEG vs WebP vs AVIF Comparison:

Original JPEG:
    - File: product.jpg
    - Size: 800 KB
    - Quality: Good
    - Support: 100% browsers

WebP (Google, 2010):
    - File: product.webp
    - Size: 320 KB (-60% vs JPEG)
    - Quality: Same visual quality
    - Support: 97% browsers (all modern)
    
AVIF (AOMedia, 2019):
    - File: product.avif
    - Size: 240 KB (-70% vs JPEG)
    - Quality: Better than JPEG at same size
    - Support: 85% browsers (growing)

Shopify Implementation:
    <picture>
      <source srcset="product.avif" type="image/avif">
      <source srcset="product.webp" type="image/webp">
      &lt;img src="product.jpg" alt="Fallback for old browsers">
    </picture>
    
    Browser support cascade:
        - Modern Chrome/Firefox: Loads AVIF (240 KB, best compression)
        - Safari/older Chrome: Loads WebP (320 KB, good compression)
        - Ancient IE11: Loads JPEG (800 KB, universal support)

Real Savings:
    Before: 800 KB JPEG × 12 images = 9.6 MB page
    After: 240 KB AVIF × 12 images = 2.9 MB page
    Reduction: 70% (9.6 MB → 2.9 MB)

Traditional Loading:
    Page loads → Browser downloads all 12 images immediately
    Images below fold: Downloaded but not visible (wasted)
    Time: 15 seconds to download all images
    User sees: Only first 2 images above fold

Lazy Loading:
    Page loads → Browser downloads 2 visible images only
    User scrolls down → Browser downloads next images (just-in-time)
    Images never scrolled to: Never downloaded (bandwidth saved)
    Time: 3 seconds to download visible images (5x faster)

HTML Implementation (Native):
    <img src="product.jpg" loading="lazy" alt="Product">
    
    Browser behavior:
        - Above fold: Loads immediately
        - Below fold: Loads when user scrolls near (200px threshold)
        - Never visible: Never loads
    
    Browser support: 93% (all modern browsers)

JavaScript Fallback (Old Browsers):
    <img data-src="product.jpg" class="lazy" alt="Product">
    
    <script>
    document.addEventListener("DOMContentLoaded", function() {
        const lazyImages = document.querySelectorAll('.lazy');
        
        const imageObserver = new IntersectionObserver((entries) => {
            entries.forEach(entry => {
                if (entry.isIntersecting) {
                    const img = entry.target;
                    img.src = img.dataset.src;
                    img.classList.remove('lazy');
                    imageObserver.unobserve(img);
                }
            });
        });
        
        lazyImages.forEach(img => imageObserver.observe(img));
    });
    </script>

Shopify CDN (Powered by Fastly + Cloudflare):

Automatic Transformations:
    URL: https://cdn.shopify.com/s/files/1/0001/2345/products/shirt.jpg?width=600
    
    CDN automatically:
        1. Resizes to 600px width (on-the-fly)
        2. Converts to WebP if browser supports
        3. Compresses (optimal quality vs size)
        4. Caches result (subsequent requests instant)
        5. Serves from nearest edge (low latency)
    
    Response:
        Content-Type: image/webp
        Content-Length: 45000 (45 KB, was 800 KB JPEG)
        Cache-Control: public, max-age=31536000 (cache 1 year)
        CF-Cache-Status: HIT (served from cache)

Image URL Parameters:
    ?width=600 - Resize to 600px width
    ?height=400 - Resize to 400px height
    ?crop=center - Crop to center
    ?format=webp - Force WebP format
    ?quality=80 - Set quality (0-100)
    
Example usage:
    Mobile: ?width=400&format=avif&quality=85
    Tablet: ?width=800&format=webp&quality=90
    Desktop: ?width=1200&format=webp&quality=95

Performance Improvements:

Image Delivery:
    Before: 45 MB average product page
    After: 2.9 MB average product page
    Reduction: 93.6% (45 MB → 2.9 MB)

Load Times:
    Desktop (25 Mbps):
        Before: 15 seconds
        After: 3 seconds (80% faster)
    
    Mobile (10 Mbps):
        Before: 35 seconds
        After: 7 seconds (80% faster)
    
    Mobile (4G, 5 Mbps):
        Before: 70 seconds
        After: 14 seconds (80% faster)

Business Impact:
    Bounce rate: 35% → 18% (49% reduction)
    Conversion rate: 2.3% → 3.1% (+0.8 percentage points)
    Average order value: $87 (unchanged)
    
    Revenue calculation:
        100M monthly visitors
        18% fewer bounces = 17M more engaged visitors
        3.1% conversion (vs 2.3%) = +800K extra orders
        $87 average order = $69.6M extra monthly revenue
        Annual impact: $835M additional GMV
        
    Shopify take rate: 2.5% of GMV
    Shopify benefit: ~$20M+/year additional revenue

Infrastructure Savings:
    Bandwidth: 45 MB → 2.9 MB (93.6% reduction)
    CDN costs: $30M/year → $8M/year ($22M savings)
    Storage: Unchanged (source images kept)
    Processing: +$2M/year (image optimization servers)
    
    Net savings: $20M/year infrastructure costs

Total Impact:
    Revenue: +$20M/year (Shopify's take from increased GMV)
    Cost savings: +$20M/year (reduced bandwidth)
    Total benefit: $40M+/year
    
    Investment: $5M (one-time engineering + infrastructure)
    ROI: 8x first year, ongoing benefit

Merchant Benefit:
    4.4M stores × $835M GMV impact = $190 per store additional annual revenue
    Zero effort required (automatic optimization)
    Faster sites = happier customers = better reviews

1. Compress Images:
    Tools: ImageOptim, TinyPNG, Squoosh
    JPEG: 80-85% quality (sweet spot, visually identical to 100%)
    PNG: Use pngquant (lossless compression)
    
    Example:
        Original: 2.5 MB (100% quality JPEG)
        80% quality: 400 KB (visually same, 84% smaller)
        WebP: 250 KB (37% more savings)

2. Responsive Images:
    Use srcset + sizes attributes
    Serve different sizes for different viewports
    Save 60-90% bandwidth on mobile
    
    Don't: <img src="huge-4000px.jpg" width="300">
    Do: <img srcset="small.jpg 300w, medium.jpg 600w, large.jpg 1200w" sizes="(max-width: 600px) 300px, 600px">

3. Lazy Load:
    Native: loading="lazy" attribute
    javascript: IntersectionObserver for old browsers
    Save 50-70% initial page weight
    
    Lazy load everything below fold (not above!)

4. Use CDN:
    Automatic image optimization (Cloudflare, Fastly)
    On-the-fly resizing + format conversion
    Global edge caching (low latency)
    
    Example: Shopify CDN, Imgix, Cloudinary

5. Next-Gen Formats:
    AVIF: Best compression (-70% vs JPEG)
    WebP: Good compression (-60% vs JPEG), better support
    JPEG: Fallback for old browsers
    
    Use <picture> element for progressive enhancement

6. Preload Critical Images:
    <link rel="preload" as="image" href="hero.webp">
    Loads hero image immediately (before HTML parsing)
    Improves Largest Contentful Paint (LCP)
    
    Only preload 1-2 critical images (not all!)

7. Avoid Common Mistakes:
    Uploading 5 MB camera photos directly
    Using PNG for photos (use JPEG/WebP/AVIF)
    Using JPEG for logos/icons (use SVG or PNG)
    Not specifying width/height (causes layout shift)
    Loading all images eagerly (wastes bandwidth)

1992: gzip (DEFLATE algorithm)
2004: 7-Zip (LZMA algorithm)
2013: Brotli (Google, optimized for web)
2024: Brotli adoption: 60%+ of top 10K websites

Test File: app.js (1 MB uncompressed JavaScript)

No Compression:
    - Size: 1,000 KB
    - Transfer time (10 Mbps): 0.8 seconds
    - Browser overhead: 0 ms (no decompression)

Gzip (Level 6, default):
    - Size: 250 KB (75% compression)
    - Transfer time: 0.2 seconds (4x faster)
    - Browser overhead: 10 ms (decompression)
    - CPU: Low (fast decompression)
    - Total: 0.21 seconds (3.8x faster than uncompressed)

Brotli (Level 6):
    - Size: 200 KB (80% compression, 20% better than gzip)
    - Transfer time: 0.16 seconds (5x faster)
    - Browser overhead: 12 ms (decompression)
    - CPU: Low (fast decompression)
    - Total: 0.172 seconds (4.65x faster, 18% faster than gzip)

Why Brotli is Better:
    1. Better dictionary (predefined for common web patterns)
    2. Larger context window (16 MB vs 32 KB gzip)
    3. Optimized for UTF-8 text (HTML, CSS, JS, JSON)
    4. More modern algorithm (2013 vs 1992)

HTML (index.html, 150 KB):
    Uncompressed: 150 KB
    Gzip: 31 KB (79% compression)
    Brotli: 26 KB (83% compression, 16% better)

CSS (styles.css, 500 KB):
    Uncompressed: 500 KB
    Gzip: 85 KB (83% compression)
    Brotli: 68 KB (86% compression, 20% better)

JavaScript (app.js, 2 MB):
    Uncompressed: 2,000 KB
    Gzip: 450 KB (77.5% compression)
    Brotli: 360 KB (82% compression, 20% better)

JSON (api-response.json, 100 KB):
    Uncompressed: 100 KB
    Gzip: 15 KB (85% compression)
    Brotli: 12 KB (88% compression, 20% better)

SVG (icons.svg, 200 KB):
    Uncompressed: 200 KB
    Gzip: 45 KB (77.5% compression)
    Brotli: 38 KB (81% compression, 15% better)

Images (JPEG, PNG):
    Already compressed (no benefit from gzip/brotli)
    Don't compress: Wastes CPU, negligible size change

# /etc/nginx/nginx.conf

# Install brotli module first:
# Ubuntu: sudo apt install nginx-module-brotli
# Or compile: --with-brotli_module

load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_brotli_static_module.so;

http {
    # Enable Brotli compression
    brotli on;
    brotli_comp_level 6;  # 0-11 (6 = balanced speed vs compression)
    brotli_types 
        text/plain
        text/css
        text/javascript
        text/xml
        text/markdown
        application/javascript
        application/json
        application/xml
        application/rss+xml
        application/atom+xml
        image/svg+xml;
    
    # Don't compress already-compressed files
    brotli_min_length 1000;  # Only compress files >1 KB
    
    # Enable gzip as fallback (for old browsers)
    gzip on;
    gzip_comp_level 6;
    gzip_types 
        text/plain
        text/css
        text/javascript
        application/javascript
        application/json
        application/xml
        image/svg+xml;
    
    server {
        listen 443 ssl http2;
        server_name example.com;
        
        location / {
            # Compression enabled (from http block)
            proxy_pass http://backend;
            
            # Headers to origin
            proxy_set_header Accept-Encoding "";  # Don't double-compress
        }
        
        # Pre-compressed static files (best performance)
        location ~* \.(js|css|svg)$ {
            # Serve pre-compressed files if available
            brotli_static on;  # Serve .br files if they exist
            gzip_static on;     # Serve .gz files as fallback
            
            # Example: app.js.br, app.js.gz, app.js
            # Browser sends: Accept-Encoding: br, gzip
            # NGINX serves: app.js.br (best compression, zero CPU)
        }
    }
}

# Build time compression (no runtime CPU cost)

# Compress all JS files at build time
find dist/ -name '*.js' -exec brotli -q 11 -o {}.br {} \;
find dist/ -name '*.js' -exec gzip -9 -k {} \;

# Result:
# dist/app.js (1000 KB) - original
# dist/app.js.br (200 KB) - Brotli level 11 (maximum)
# dist/app.js.gz (250 KB) - Gzip level 9 (maximum)

# Why pre-compress at build time:
# 1. Higher compression levels (11 vs 6) = smaller files
# 2. Zero runtime CPU (NGINX just serves static file)
# 3. Faster response (no compression delay)
# 4. Works with CDN (upload .br files to CDN)

# Trade-off:
# Build time: +30 seconds (one-time cost)
# Runtime: Instant (zero CPU)
# File size: 15% smaller (level 11 vs 6)

Brotli Level Trade-offs:

Level 0 (Fastest):
    - Compression: 60% (worse than gzip)
    - Speed: 400 MB/sec
    - Use: Never (worse than gzip)

Level 1-3 (Fast):
    - Compression: 70-75% (similar to gzip)
    - Speed: 200-300 MB/sec
    - Use: High-traffic dynamic content

Level 4-6 (Balanced - Default):
    - Compression: 78-82% (20% better than gzip)
    - Speed: 80-150 MB/sec
    - Use: General purpose (most common)

Level 7-9 (High Compression):
    - Compression: 83-85% (25% better than gzip)
    - Speed: 20-50 MB/sec
    - Use: Static assets, pre-compression

Level 10-11 (Maximum):
    - Compression: 86-88% (30% better than gzip)
    - Speed: 2-10 MB/sec (very slow!)
    - Use: Pre-compression only (never real-time)

Recommendation:
    - Real-time: Level 6 (balanced)
    - Build time: Level 11 (maximum, no runtime cost)

Brotli Support (2024):
    - Chrome: Since v50 (2016)
    - Firefox: Since v44 (2016)
    - Safari: Since v11 (2017)
    - Edge: Since v15 (2017)
    - Total: 98%+ global browser share

Content-Encoding Negotiation:
    Browser request:
        GET /app.js HTTP/1.1
        Accept-Encoding: br, gzip, deflate
    
    Server response (Brotli):
        HTTP/1.1 200 OK
        Content-Type: application/javascript
        Content-Encoding: br
        Content-Length: 200000
        
        [Brotli-compressed data]
    
    Old browser (no Brotli support):
        GET /app.js HTTP/1.1
        Accept-Encoding: gzip, deflate
    
    Server response (Gzip fallback):
        HTTP/1.1 200 OK
        Content-Encoding: gzip
        Content-Length: 250000
        
        [Gzip-compressed data]

LinkedIn (2017 Case Study):
    Before (Gzip only):
        - javascript: 1.2 MB compressed (gzip)
        - CSS: 180 KB compressed (gzip)
        - Total: 1.38 MB

    After (Brotli):
        - javascript: 960 KB compressed (20% smaller)
        - CSS: 144 KB compressed (20% smaller)
        - Total: 1.10 MB

    Results:
        - Page load: 2.7s → 2.3s (15% faster)
        - Bounce rate: -5% (fewer users leaving)
        - Mobile users: Biggest benefit (slower connections)
        - Cost: $0 (just configuration change)

Wikipedia (2021):
    - 500M+ pageviews/day
    - Average page: 100 KB HTML (gzip: 25 KB, brotli: 20 KB)
    - Bandwidth saved: 5 KB × 500M = 2.5 TB/day
    - Annual savings: 912 TB/year (2.5 TB × 365 days)
    - Cost savings: $45K/year ($0.05/GB × 912 TB)
    - Setup cost: $0 (configuration change)

Cloudflare Advantages:
1. Performance:
   - 310+ locations globally (covers Europe, Asia well)
   - Average TTFB: 22ms (well under 200ms requirement)
   - Workers run at edge (dynamic content fast)

2. Cost:
   - Business plan: $200/month base
   - Bandwidth: 100 TB/month typical
     * Images: 100K users × 7.5 MB page × 5 pages = 3.75 TB/month
     * Videos: 30K video views × 5 MB = 150 GB
     * Total: ~4 TB/month (well under $15K budget)
   - Estimated cost: $5K/month (within budget)

3. Traffic Spike:
   - Unmetered DDoS protection (Black Friday spikes)
   - Auto-scaling (no capacity planning)
   - Pay-as-you-go (only pay for usage)

4. Dynamic Content:
   - Cloudflare Workers (JavaScript at edge)
   - Run checkout logic at 310+ locations
   - No need to cache dynamic pages (Workers execute)

Architecture:
    User (Europe) 
        → Cloudflare Edge (London, 15ms away)
        → Workers (checkout logic runs in London)
        → Origin (us-east-1, only if Workers need data)
        → Response (cached if static, dynamic if Workers)

CloudFront Issues:
1. Cost:
   - 450+ locations but pricing higher
   - Lambda@Edge: $0.60 per 1M requests
   - Black Friday spike: 10x traffic = 10M requests = $6,000 just Lambda
   - Bandwidth: $0.085/GB × 4,000 GB = $340
   - Total: $6,340/month (acceptable but more expensive)

2. Cold Starts:
   - Lambda@Edge cold start: 50-200ms
   - Adds latency to dynamic requests
   - Cloudflare Workers: <1ms (always warm)

3. Complexity:
   - Separate services (CloudFront + Lambda + API Gateway)
   - More configuration vs Cloudflare (integrated)

Not wrong, just more expensive and complex for this use case

Akamai Issues:
1. Cost:
   - Enterprise-only (minimum $5K-10K/month commitment)
   - 4,100+ locations (overkill for 180K users)
   - Bandwidth: $0.15/GB (2x CloudFront)
   - 4 TB × $0.15 = $600 base + $10K minimum = $10,600/month

2. Overengineered:
   - Need: 95th percentile <200ms (easily achievable)
   - Akamai provides: <25ms globally (gold-plated)
   - Paying for performance you don't need

3. Setup Time:
   - Sales cycle: 2-4 weeks
   - Contract negotiation
   - Professional services required

Use Akamai when:
   - Mission-critical (Apple iOS updates)
   - Absolute best performance needed
   - Large enterprise budget

This is growing e-commerce, not Apple. Akamai is overkill.

Regional Server Issues:
1. Cost:
   - 3 regions (US, EU, Asia) × 5 servers each = 15 servers
   - m5.xlarge: $140/month × 15 = $2,100/month servers
   - Data transfer: $0.09/GB × 4 TB = $360/month
   - Load balancers: $16/month × 3 = $48/month
   - Total: $2,508/month (seems cheaper...)

2. Hidden Costs:
   - Engineering: 3 deployments (US, EU, Asia)
   - Database replication: Cross-region sync complexity
   - Monitoring: 3 separate infrastructures
   - On-call: Issues in Asia = wake up at 2 AM
   - Maintenance: 3x operational burden

3. Traffic Spikes:
   - Black Friday: Need 150 servers (10x)
   - Auto-scaling across regions: Complex
   - Capacity planning: Guessing peak load
   - CDN: Automatically handles spikes

4. Performance:
   - Images still slow (no edge caching)
   - Videos: 5 MB × 30K = 150 GB (slow without CDN)
   - No edge optimization (compress, WebP conversion)

Regional servers + CDN is better than regional servers alone.
Cloudflare CDN is better than managing regional servers.

// Cloudflare Worker (runs at edge)
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const url = new URL(request.url)
  
  // Static content: Cache aggressively
  if (url.pathname.match(/\.(jpg|png|css|js|woff2)$/)) {
    return fetch(request, {
      cf: {
        cacheTtl: 86400,  // Cache 24 hours
        cacheEverything: true
      }
    })
  }
  
  // Dynamic checkout: Run at edge
  if (url.pathname.startsWith('/checkout')) {
    // Fetch user cart from origin
    const cartResponse = await fetch('https://origin.example.com/api/cart', {
      headers: {
        'Cookie': request.headers.get('Cookie')
      }
    })
    const cart = await cartResponse.json()
    
    // Calculate tax at edge (based on user location)
    const country = request.cf.country
    const tax = calculateTax(cart.total, country)
    
    // Generate HTML at edge (no origin round-trip)
    return new Response(renderCheckout(cart, tax), {
      headers: { 'Content-Type': 'text/html' }
    })
  }
  
  // Default: Pass to origin
  return fetch(request)
}

Capacity Analysis:

Apache (Current):
    - 5 servers × 3,000 connections = 15,000 total
    - Architecture: Process-per-connection (Prefork MPM)
    - Memory: 300 processes × 40 MB = 12 GB per server
    - CPU: 60% (context switching overhead)

NGINX (Proposed):
    - Event-driven architecture (1 worker per CPU core)
    - Memory: ~500 MB (handles all connections in worker pool)
    - Connections: 100,000+ per server easily (vs 3,000 Apache)
    
    1 NGINX server capacity:
        - m5.2xlarge: 8 vCPUs, 32 GB RAM
        - NGINX workers: 8 (one per core)
        - Connections per worker: 10,000 (typical)
        - Total capacity: 80,000 concurrent connections
        - Current need: 15,000 connections
        - Utilization: 18.75% (plenty of headroom)
        - Memory: 500 MB (1.5% of 32 GB)
        - CPU: 20% (event-driven efficiency)

Cost Comparison:
    Before: 5 × $280/month = $1,400/month
    After: 1 × $280/month = $280/month
    Savings: $1,120/month (80% reduction)  Exceeds 50% goal

Why it works:
    - NGINX: 1 connection = 1 KB memory
    - Apache: 1 connection = 40 MB memory (40,000x more!)
    - 15K connections: 15 MB NGINX vs 600 GB Apache (theoretical)

# NGINX configuration (replaces 5 Apache servers)

worker_processes auto;  # 8 workers (one per CPU)
worker_rlimit_nofile 100000;  # File descriptor limit

events {
    worker_connections 10000;  # 10K per worker = 80K total
    use epoll;  # Linux efficient event notification
}

http {
    # Keep-alive (reuse connections)
    keepalive_timeout 65;
    keepalive_requests 100;
    
    # Upstream (application servers)
    upstream app_servers {
        least_conn;  # Balance by active connections
        server app1.internal:8080 max_conns=5000;
        server app2.internal:8080 max_conns=5000;
        server app3.internal:8080 max_conns=5000;
        keepalive 1000;  # Persistent connections to backend
    }
    
    server {
        listen 80;
        
        location / {
            proxy_pass http://app_servers;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

Overprovisioning:
    - 5 NGINX servers: 400,000 connection capacity
    - Current need: 15,000 connections
    - Utilization: 3.75% (massively underutilized)
    - Cost: $1,400/month (no savings)
    - Waste: $1,120/month vs 1 server

When to use 5 servers:
    - Traffic: 200K+ concurrent connections
    - High availability: 5-nines requirement (99.999%)
    - Geographic distribution: 5 regions
    
For 15K connections: 1 server sufficient

Still Overprovisioned:
    - 3 servers: 240,000 connection capacity
    - Utilization: 6.25% (underutilized)
    - Cost: 3 × $280 = $840/month
    - Savings: $560/month (40% reduction)
    - CEO wanted: 50% reduction ($700/month)
    - Falls short of goal

Better than A, but not optimal

Auto-scaling Complexity:
    - 2 servers: Base capacity 160,000 connections
    - Utilization: 9.375% (still low)
    - Cost: 2 × $280 = $560/month (60% savings)
    -  Exceeds CEO's 50% goal
    
But:
    - Auto-scaling adds complexity (ALB, CloudWatch, etc.)
    - 1 server handles current + 5x growth headroom
    - When to add 2nd server: >40K concurrent connections
    
1 server simpler and cheaper until you actually need scaling

Before Migration (2010):
    - 1,000 Apache servers (US data center)
    - 2 million concurrent streams
    - Cost: $2M/month (servers + bandwidth)
    - Problem: Linear scaling (2x traffic = 2x servers)

After Migration (2011):
    - 100 NGINX servers (AWS us-east-1)
    - 2 million concurrent streams (same capacity)
    - Cost: $200K/month servers + $1.8M AWS services
    - Savings: $10M/year infrastructure
    
Why 90% reduction:
    - Apache: 2,000 streams per server
    - NGINX: 20,000 streams per server (10x more efficient)
    - Event-driven architecture vs process-per-connection
    - Memory: 10 GB NGINX vs 120 GB Apache

# Benchmark Apache vs NGINX

# Apache Prefork (existing)
ab -n 100000 -c 5000 http://apache-server/
# Results:
# Requests per second: 3,250
# Time per request: 1,538ms (mean)
# Failed requests: 420 (2K+ connection limit hit)

# NGINX (proposed)
ab -n 100000 -c 5000 http://nginx-server/
# Results:
# Requests per second: 28,500 (8.8x faster)
# Time per request: 175ms (mean) (88% faster)
# Failed requests: 0 (handles 5K connections easily)

# Memory usage during test:
# Apache: 12 GB (300 processes × 40 MB)
# NGINX: 420 MB (8 workers, shared memory)
# Difference: 96.5% less memory

Defense in Depth Strategy:

Layer 1: Automated Renewal (Let's Encrypt + Certbot)
    - Certbot cron job: Runs twice daily
    - Auto-renews: 30 days before expiry
    - Success rate: 99.9% (if properly configured)
    - Cost: Free

Layer 2: Monitoring (Multiple Checks)
    - Check 1: Internal (every 6 hours)
        Script checks: openssl s_client -connect domain.com:443
        Alert: If cert expires in <30 days
    
    - Check 2: External (third-party service)
        Service: SSL Labs, UptimeRobot, or Pingdom
        Checks: Every hour from multiple locations
        Alert: If cert expires in <14 days
    
    - Check 3: Business Hours Check
        Daily email to devops@company: "Cert status for all 20 domains"
        Human verification: Glance confirms all >30 days

Layer 3: Backup Certificates
    - Pre-generate backup cert (different CA)
    - Store in secrets manager (AWS Secrets Manager, HashiCorp Vault)
    - If primary expires: Emergency script swaps to backup
    - Recovery time: 5 minutes (vs 2 hours)

Layer 4: Alerting Escalation
    - Day 45: Info alert to devops@
    - Day 30: Warning to devops@ + infra-lead@
    - Day 14: Critical to devops@ + infra-lead@ + cto@
    - Day 7: PagerDuty page (wake someone up)
    - Day 1: Auto-deploy backup certificate

Implementation:
    Cost: $0 (Let's Encrypt + open-source tools)
    Setup time: 4 hours (one-time)
    Maintenance: 0 hours/month (fully automated)
    Risk reduction: 99.99% (4 layers of protection)

#!/bin/bash
# /usr/local/bin/check-ssl-expiry.sh

DOMAINS=(
    "example.com"
    "api.example.com"
    "www.example.com"
    # ... 17 more domains
)

ALERT_DAYS=30
CRITICAL_DAYS=7

for domain in "${DOMAINS[@]}"; do
    # Get certificate expiry date
    expiry=$(echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | \
             openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
    
    # Convert to Unix timestamp
    expiry_epoch=$(date -d "$expiry" +%s)
    now_epoch=$(date +%s)
    
    # Calculate days until expiry
    days_left=$(( ($expiry_epoch - $now_epoch) / 86400 ))
    
    # Alert based on days left
    if [ $days_left -lt $CRITICAL_DAYS ]; then
        echo "CRITICAL: $domain expires in $days_left days!"
        # Send PagerDuty alert
        curl -X POST https://events.pagerduty.com/v2/enqueue \
          -H 'Content-Type: application/json' \
          -d "{\"routing_key\":\"$PAGERDUTY_KEY\",\"event_action\":\"trigger\",\"payload\":{\"summary\":\"SSL cert for $domain expires in $days_left days\",\"severity\":\"critical\"}}"
    elif [ $days_left -lt $ALERT_DAYS ]; then
        echo "WARNING: $domain expires in $days_left days"
        # Send Slack notification
        curl -X POST https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
          -H 'Content-Type: application/json' \
          -d "{\"text\":\"SSL cert for $domain expires in $days_left days\"}"
    else
        echo "OK: $domain expires in $days_left days"
    fi
done

# Cron job (runs every 6 hours)
# 0 */6 * * * /usr/local/bin/check-ssl-expiry.sh

Calendar Reminder Issues:
    - Human dependency (someone must act)
    - Vacation/sick leave: Person away, cert expires
    - Reminder fatigue: "I'll do it tomorrow" × 60 days
    - No verification: Did renewal succeed?
    - Manual process: Error-prone (wrong command, wrong domain)

LinkedIn incident: HAD reminders, still failed
Reason: Person saw reminder, renewal failed, no follow-up
Solution: Automation (not better reminders)

AWS ACM Advantages:
    - Auto-renewal: 60 days before expiry
    - Free: No cost
    - Easy: One-click setup
    
AWS ACM Limitations:
    - AWS-only: Only works with CloudFront, ALB, API Gateway
    - Cannot export: Private key stays in AWS (can't use with NGINX on EC2)
    - Vendor lock-in: Tied to AWS ecosystem
    - Multi-cloud issue: Your domains across 5 providers

Scenario: 20 domains across 5 providers
    - 8 domains on AWS → ACM works
    - 12 domains on GCP, Azure, DigitalOcean, Cloudflare → ACM doesn't work
    
Solution B doesn't solve for 12 domains not on AWS

10-Year Certificate Problems:
    1. Deprecated: Browsers limit to 398 days (13 months) since 2020
       - Chrome 85+ enforces 398-day limit
       - 10-year certs show "Certificate Invalid" error
       - Not an option anymore
    
    2. Security Risk: If private key compromised
       - 10-year cert: Attacker has 10 years unless revoked
       - 90-day cert: Attacker has 90 days max
       - Shorter validity = better security
    
    3. Revocation Nightmare:
       - Need to revoke: Must reissue + deploy to all servers
       - With 90-day certs: Already rotating frequently
       - With 10-year certs: Revocation is rare emergency
    
    4. Industry Standard: 90 days (Let's Encrypt)
       - Forces automation (good practice)
       - Limits damage from compromised keys
       - Aligns with modern DevOps

10-year certs are not just wrong, they're impossible

# docker-compose.yml (Complete SSL Management)

version: '3'
services:
  # NGINX web server
  nginx:
    image: nginx:latest
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - certbot-certs:/etc/letsencrypt
      - certbot-www:/var/www/certbot
    
  # Certbot (auto-renewal)
  certbot:
    image: certbot/certbot
    volumes:
      - certbot-certs:/etc/letsencrypt
      - certbot-www:/var/www/certbot
    command: certonly --webroot --webroot-path=/var/www/certbot \
             --email admin@example.com --agree-tos --no-eff-email \
             -d example.com -d www.example.com
    
  # Certificate monitor
  ssl-monitor:
    image: custom/ssl-monitor:latest
    environment:
      - DOMAINS=example.com,api.example.com,www.example.com
      - ALERT_DAYS=30
      - CRITICAL_DAYS=7
      - SLACK_WEBHOOK=https://hooks.slack.com/services/YOUR/WEBHOOK
      - PAGERDUTY_KEY=your-pagerduty-key
    command: /check-ssl-expiry.sh

volumes:
  certbot-certs:
  certbot-www:

# Cron job for renewal (runs twice daily)
# 0 */12 * * * docker-compose run certbot renew && docker-compose exec nginx nginx -s reload