1. The Problem & Technical Context
The team required an automated, reliable system to schedule, format, transcode, and publish multi-media content across multiple third-party social media providers (Facebook, YouTube, TikTok, Telegram) through a unified REST/OAuth gateway (Zernio integration).
In early prototypes, high concurrency and external API flakiness caused operational bottlenecks:
- OAuth Token Refresh Stampede: When an access token expired, multiple concurrent background workers attempted to exchange the refresh token simultaneously, causing invalidation of active session tokens and cascade authentication failures.
- Duplicate Dispatch on Network Retry: A brief network timeout during third-party publishing could cause worker retries, leading to identical marketing videos being published 2 or 3 times to user channels.
- Desynchronized Scheduled State: Scheduled posts would occasionally remain in a "PENDING" or "PROCESSING" state indefinitely if a worker pod restarted mid-flight.
2. System Architecture & Data Flow
The microservice is built with NestJS 11 on top of the high-throughput Fastify adapter, utilizing Drizzle ORM with PostgreSQL for type-safe schema migrations and atomic SQL transactions.
3. Key Engineering Decisions & Trade-Offs
Decision 1: In-Memory Single-Flight Mutex for OAuth Token Refresh
SingleFlightMutex service. The first incoming request initiates the async HTTP token refresh and stores the pending Promise. All 9 subsequent concurrent requests join and await the exact same Promise without making additional network calls.
SELECT ... FOR UPDATE) is used as a secondary fence.
Decision 2: Atomic Job Claiming via PostgreSQL SKIP LOCKED
SELECT ... FOR UPDATE SKIP LOCKED inside an atomic transaction. A worker locks and transitions rows to 'PROCESSING' atomically, allowing concurrent workers to immediately skip to unreserved records with zero lock contention.
Decision 3: Multi-Layer Video Transcoding & Fallback Pipeline
4. Resilient Publishing State Machine Implementation
// Single-Flight Mutex pattern for OAuth token refresh deduplication
@Injectable()
export class TokenManagerService {
private inFlightRefreshes = new Map<string, Promise<string>>();
async getValidToken(accountId: string): Promise<string> {
const existingRefresh = this.inFlightRefreshes.get(accountId);
if (existingRefresh) {
// Coalesce duplicate callers into the single in-flight promise
return existingRefresh;
}
const refreshPromise = this.executeTokenRefresh(accountId).finally(() => {
this.inFlightRefreshes.delete(accountId);
});
this.inFlightRefreshes.set(accountId, refreshPromise);
return refreshPromise;
}
}
5. Verification & Testing Strategy
To ensure production zero-regression, I authored over 30 Jest test suites encompassing unit tests for services and integration tests with a live test database container.
- Mutex Deduplication Tests: Verified that firing 20 concurrent
getValidToken()calls results in exactly 1 HTTP network invocation. - Idempotency Validation: Verified that duplicate webhook payloads with matching idempotency keys return
200 OKwithout duplicate DB inserts. - Reconcile Sweep Tests: Simulated crashed workers and verified that stale tasks older than 15 minutes are reset to
FAILEDor re-queued gracefully.
If scaling to 100,000+ posts/hour, I would transition the PostgreSQL SKIP LOCKED queue to a dedicated distributed message broker like BullMQ (Redis) or Apache Kafka with partitioned partition keys by account ID to ensure horizontal FIFO scheduling.
Additionally, implementing OpenTelemetry distributed tracing across the worker lifecycle would provide granular p99 latency insights into third-party API response degradation.