1. The Problem: Large Media Streaming & IoT Saturation
This dual-discipline engineering initiative solved two distinct operational problems:
- Media Automation Pipeline: Uploading 4K/1080p long-form video files (>1GB) to YouTube via standard automation scripts frequently caused server out-of-memory (OOM) crashes and incomplete uploads due to flaky connection drops.
- IoT Protocol Vulnerability: Lightweight IoT publish/subscribe architectures (Mosquitto MQTT) are susceptible to Flood-DoS and unauthorized topic writes, where rogue sensor nodes overwhelm the central message broker.
2. Resumable Chunking & IoT Defense Architecture
To eliminate server memory pressure, the n8n pipeline avoids loading entire video files into RAM. Instead, a custom JavaScript indexing node calculates discrete 100MB byte ranges (Range: bytes=start-end) streamed sequentially:
Figure 4.1: Byte-Range Resumable Workflow & IoT Broker Isolation Topology
3. Key Engineering Decisions
Decision 1: Zero-Memory Byte-Range Slicing for Video Uploads
Problem
Loading full 2GB video binary into workflow execution memory triggered pod crashes under constrained server limits.
Decision
Developed a JavaScript chunking script computing an array of offset boundaries
[{start: 0, end: 104857599}, {start: 104857600, end: ...}], streaming only 100MB per HTTP request with Content-Range headers.
Trade-off
Requires managing intermediate loop state and handling HTTP 308 (Resume Incomplete) status codes without marking execution as failed.
Decision 2: Automated Failure Escalation via Telegram Webhooks
Problem
Silent automation failures left missing uploads unnoticed until business teams reported missing daily posts.
Decision
Integrated try/catch error routing branching directly to an internal Telegram bot with execution error stack traces and row identifiers.
Trade-off
Slightly increased workflow node count, but reduced incident response time to under 5 minutes.
4. Resumable Byte Chunking Algorithm
n8n-code-node-byte-chunker.js
// Computes discrete byte ranges for YouTube Resumable Upload
const totalSize = parseInt($json.fileSize, 10);
const CHUNK_SIZE = 100 * 1024 * 1024; // 100MB chunks
const totalChunks = Math.ceil(totalSize / CHUNK_SIZE);
const chunks = [];
for (let i = 0; i < totalChunks; i++) {
const start = i * CHUNK_SIZE;
const end = Math.min(start + CHUNK_SIZE - 1, totalSize - 1);
chunks.push({
chunkIndex: i + 1,
totalChunks: totalChunks,
start: start,
end: end,
contentRange: `bytes ${start}-${end}/${totalSize}`,
isLastChunk: (i === totalChunks - 1)
});
}
return chunks.map(c => ({ json: c }));
What I Would Improve Next (Engineering Reflection)
For larger deployments, replacing JSON workflow files with declarative Terraform/GitOps pipelines would ensure consistent environment promotion, and adding automated dead-letter queue (DLQ) retry mechanisms would further improve enterprise resilience.