Microservices for Video Processing

Service Decomposition

Split by function, not by feature:

Recommended Services

• Upload Service: Handles file uploads, chunking, URL fetching
• Job Service: Creates jobs, tracks state, manages workflow
• Worker Service: GPU processing, runs on different infrastructure
• Delivery Service: Signed URLs, output lifecycle
• Webhook Service: Customer notifications, retry logic
• Billing Service: Usage tracking, credit management

Why This Split

• Upload: CPU-bound, needs high network
• Job: Mostly database, low compute
• Worker: GPU-bound, expensive infrastructure
• Delivery: Storage-bound, integrates with CDN
• Webhook: Network-bound, needs retry logic
• Billing: Sensitive, separate trust boundary

Communication Patterns

Synchronous (Request-Response)

For operations needing immediate response:

• Job creation (API → Job Service)
• Status queries (API → Job Service)
• Authentication (all services → Auth)

Asynchronous (Queue-Based)

For work that can be deferred:

• Job processing (Job Service → Queue → Worker)
• Webhook delivery (Job Service → Queue → Webhook Service)
• Cleanup tasks (Scheduler → Queue → Various)

Event-Driven

For reactions to state changes:

• Job completed → Update billing
• Upload finished → Create job
• Webhook failed → Alert

Data Flow

Upload Service → Storage → Worker Service → Storage → Delivery Service
                    ↓                           ↓
              Job Service ←──────────────────────
                    ↓
            Webhook Service → Customer

What Not to Microservice

Don't Split the Processing Pipeline

Bad idea:

• Decode Service → Frame Service → Enhance Service → Encode Service

Why it fails:

• Massive data transfer between services (gigabytes)
• Latency at every boundary
• Complex coordination
• Failure handling nightmare

The processing pipeline should be one service.

Don't Microservice Configuration

• Configuration should be environment or file-based
• A "Config Service" adds latency to every operation
• Cache invalidation becomes a nightmare

Don't Create Single-Use Services

If a service only serves one other service, it's probably not a service:

• "Frame Extractor Service" → Just a library
• "Hash Calculator Service" → Just a function

Service Communication

API Gateway

Single entry point for external clients:

• Authentication/authorization
• Rate limiting
• Request routing
• API versioning

Service Mesh (Optional)

For complex deployments:

• Service discovery
• Load balancing
• Mutual TLS
• Observability

Often overkill for video processing — start simple.

Shared Libraries

Common code as libraries, not services:

• Authentication helpers
• Logging formatters
• Error handling
• Configuration parsing

Data Management

Database per Service

Each service owns its data:

• Job Service: job records, status
• Billing Service: usage, credits
• Webhook Service: delivery attempts

Shared Storage

Video files are shared via object storage:

• Upload Service writes to S3
• Worker Service reads from S3
• Delivery Service generates signed URLs

Event Sourcing (Optional)

For audit and replay:

• Every state change is an event
• Current state derived from event stream
• Useful for billing reconciliation