Building Scalable Video Processing Pipelines

Video processing pipelines should be asynchronous and event-driven. When a user uploads a video, immediately return a job ID and process in the background.

Why asynchronous? - Video processing takes minutes to hours - Users shouldn't wait for completion - Resources scale independently of API servers - Failed jobs can be retried without user action

Core pipeline stages: 1. Upload: Receive file, validate, store in staging 2. Ingest: Extract metadata, create job record 3. Transcode: Convert to target formats/resolutions 4. Package: Generate streaming manifests 5. Deliver: Move to CDN origin, update status

Message queues decouple upload handling from transcoding, allowing horizontal scaling of workers independent of API servers.

Queue architecture components:

Job Queue (SQS, RabbitMQ, Redis) - Receives transcoding jobs from upload handler - Provides at-least-once delivery guarantee - Dead letter queue for failed jobs - Visibility timeout prevents duplicate processing

Worker Pool - Pulls jobs from queue - Auto-scales based on queue depth - Stateless—can be terminated anytime - Reports progress to status service

Status Service - Tracks job state (pending, processing, complete, failed) - Provides webhook/polling for completion - Stores job metadata and output URLs

This architecture handles traffic spikes gracefully—queue absorbs bursts while workers process at sustainable pace.

Complex video pipelines have multiple stages that must execute in order, with parallel processing and error handling.

AWS Step Functions example flow: 1. Validate input (parallel: check format, virus scan) 2. Extract metadata (resolution, duration, codec) 3. Transcode (parallel: multiple renditions) 4. Generate thumbnails (parallel with transcode) 5. Create streaming manifests 6. Update CDN and database 7. Send completion webhook

Temporal/Cadence for complex workflows: - Long-running workflows (hours/days) - Complex branching and conditionals - Human-in-the-loop approval steps - Versioned workflow definitions

Error handling patterns: - Retry with exponential backoff - Dead letter queues for inspection - Partial success handling (some renditions fail) - Alerting on failure rate thresholds

Auto-scaling strategies:

Queue-depth scaling (recommended) - Scale workers based on queue length, not CPU - Target: queue length / workers = desired processing time - Aggressive scale-up, gradual scale-down

Spot instances for batch processing - 70-90% cost savings over on-demand - Handle interruption gracefully (checkpoint progress) - Use spot fleet with multiple instance types - Keep on-demand capacity for time-sensitive jobs

GPU acceleration: - 5-10x faster encoding with NVENC/QuickSync - Cost-effective for high-volume processing - Limited codec support (primarily H.264/H.265)

Right-sizing considerations: - Transcoding is CPU-bound, not memory-bound - Network bandwidth matters for large files - Local SSD improves I/O for complex filters

Managed transcoding services:

AWS MediaConvert - Pay-per-minute, no infrastructure management - Scales automatically, supports all major formats - Good for: variable workloads, teams without video expertise

Mux, Cloudflare Stream, api.video - End-to-end solutions including player and analytics - Fastest time-to-market - Higher per-minute cost, but zero ops burden

Self-hosted FFmpeg clusters: - Maximum control and flexibility - Lower cost at scale (millions of minutes/month) - Requires significant DevOps investment - Good for: video-core businesses with engineering resources

Hybrid approach: - Use managed services for standard transcoding - Custom pipeline for specialized processing (AI, custom filters) - Migrate components in-house as scale justifies investment

We typically recommend starting with managed services and bringing specific components in-house only when scale and requirements justify the investment.