Design YouTube/Netflix
Video Streaming কেন আলাদা?
YouTube প্রতি মিনিটে 500 ঘণ্টার video upload হয়। Netflix-এর internet traffic-এর ১৫% শুধু Netflix-এর। এই দুটো system-এ সবচেয়ে বড় challenge হলো video storage, processing, এবং delivery — এটা regular web app থেকে সম্পূর্ণ আলাদা।
📌 Key Insight
Video streaming = massive storage + video transcoding + adaptive bitrate streaming + global CDN। একটা 1-hour video কে 10+ different formats/qualities-এ encode করতে হয়।
📤 Upload Challenge
Creator uploads → raw storage → Kafka trigger → transcoding workers → multiple quality versions → CDN এ push।
▶️ Stream Challenge
Viewer requests → API gets CDN URL → Viewer directly streams from nearest CDN edge node। API server video bits transmit করে না।
Features কী কী?
✅ Functional Requirements
- →Video upload করা
- →Video stream করা (watch)
- →Video search করা
- →Like, comment, subscribe
- →Video recommendations
- →Resume from where left off
- →Multiple quality (360p/720p/1080p/4K)
⚡ Non-Functional Requirements
- →2 billion+ monthly active users
- →Video starts in < 2 seconds
- →No buffering (smooth playback)
- →99.9% availability
- →Uploaded video available within 1 min
- →Support any device (mobile/TV/web)
YouTube-এর Numbers
🔢 Storage Calculation
500 hrs/min × 60 min = 30,000 hrs/hr upload। প্রতি ঘণ্টা video ≈ 1GB (compressed) × 10 formats = 10GB/hr of video। Daily: 30,000 × 24 × 10GB = 7.2 petabytes/day! এই scale-এ dedicated storage infrastructure দরকার।
YouTube-এর Two Flows
YouTube-এ দুটো completely আলাদা path: Upload path এবং Stream path।
YouTube Architecture — Upload & Stream Flows
Video Processing Pipeline
🎬 Transcoding — সবচেয়ে গুরুত্বপূর্ণ Step
একটা raw video upload হলে সেটাকে অনেকগুলো format-এ convert করতে হয়। এটাকে transcoding বলে।
| Output Format | Resolution | Bitrate | Use Case |
|---|---|---|---|
| 360p | 640×360 | ~400 Kbps | Slow mobile/2G |
| 480p | 854×480 | ~700 Kbps | Mobile 3G |
| 720p HD | 1280×720 | ~2.5 Mbps | Normal streaming |
| 1080p FHD | 1920×1080 | ~5 Mbps | Good connection |
| 4K UHD | 3840×2160 | ~20 Mbps | Netflix/Fast internet |
💡 Adaptive Bitrate Streaming (ABR)
Player automatically quality switch করে network speed অনুযায়ী। Netflix/YouTube HLS (HTTP Live Streaming) বা DASH protocol use করে। Video small chunks (2-10 sec) এ ভাগ হয়। Slow network → 360p, Fast → 1080p auto switch।
import subprocess
import boto3
def transcode_video(raw_video_path: str, video_id: str):
qualities = [
{"name": "360p", "width": 640, "height": 360, "bitrate": "400k"},
{"name": "720p", "width": 1280, "height": 720, "bitrate": "2500k"},
{"name": "1080p", "width": 1920, "height": 1080, "bitrate": "5000k"},
]
s3 = boto3.client('s3')
output_files = []
for q in qualities:
output_path = f"/tmp/{video_id}_{q['name']}.mp4"
# FFmpeg দিয়ে transcode করুন
subprocess.run([
"ffmpeg", "-i", raw_video_path,
"-vf", f"scale={q['width']}:{q['height']}",
"-b:v", q['bitrate'],
"-c:v", "libx264", "-c:a", "aac",
output_path
])
# S3-এ upload করুন
s3_key = f"videos/{video_id}/{q['name']}.mp4"
s3.upload_file(output_path, "my-video-bucket", s3_key)
output_files.append(s3_key)
return output_files # CDN-এ push করা হবে📊 Video Chunking — HLS Protocol
# Master playlist — player এটা দেখে quality choose করে
#EXTM3U
#EXT-X-VERSION:3
# 360p stream
#EXT-X-STREAM-INF:BANDWIDTH=400000,RESOLUTION=640x360
https://cdn.youtube.com/video123/360p/playlist.m3u8
# 720p stream
#EXT-X-STREAM-INF:BANDWIDTH=2500000,RESOLUTION=1280x720
https://cdn.youtube.com/video123/720p/playlist.m3u8
# 1080p stream
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
https://cdn.youtube.com/video123/1080p/playlist.m3u8
# Player automatically best quality select করে bandwidth অনুযায়ীAdaptive Bitrate & CDN — Global Delivery
YouTube-এর streaming architecture-এর দুটো secret weapon: ABR (buffer-free playback) এবং CDN (global fast delivery)।
ABR Streaming — How HLS/DASH Works
🐌 Slow Network (≤ 1 Mbps)
Player switches to 360p। No buffering। Small chunk size।
🚶 Normal Network (5 Mbps)
Player uses 720p। Balanced quality/buffer।
🚀 Fast Network (20+ Mbps)
Player serves 1080p / 4K। Full quality।
| Protocol | Developer | Chunk Size | Supported By | Best For |
|---|---|---|---|---|
| HLS | Apple | 2-10 sec | iOS, Safari, all | VOD + Live streaming |
| DASH | MPEG (ISO) | 2-4 sec | Android, Chrome | Adaptive VOD |
| RTMP | Adobe | Real-time | Flash (legacy) | Live ingest only |
| WebRTC | W3C/IETF | Sub-second | All browsers | Ultra-low latency |
💡 Netflix Open Connect
Netflix Open Connect Appliance (OCA) — Netflix নিজেই hardware box তৈরি করে ISP-দের কাছে বিনামূল্যে দেয়। ISP network-এর ভেতরে Netflix content cache থাকে। Bandwidth cost কমে, speed বাড়ে। Bangladesh-এর user India বা Singapore edge থেকে video পাবেন — origin US server থেকে না। Latency 200ms থেকে 10ms-এ নামে।
Data কোথায় রাখবো?
| Data | Database | Why? |
|---|---|---|
| Video metadata (title, desc) | MySQL + Redis cache | Structured, relational, cacheable |
| Raw video files | Amazon S3 / Google Cloud Storage | Object storage, terabytes-scale |
| Processed videos | S3 → CDN (CloudFront) | Edge delivery globally |
| User watch history | Cassandra | Time-series, massive scale |
| Video search index | Elasticsearch | Full-text search, tags |
| Recommendations data | Neo4j / ML model store | Graph-based recommendations |
| View counts, likes | Redis (counters) | Fast atomic increments |
⚠️ Never Store Videos in DB
Database-এ video binary কখনো store করবেন না। Database slow binary retrieval-এর জন্য। সর্বদা Object Storage (S3) + CDN ব্যবহার করুন। DB শুধু metadata রাখে।
Scale করার উপায়
CDN is Everything: 95%+ video traffic CDN থেকে serve হয়। Origin server-এ কোনো load নেই। Netflix Open Connect — নিজেদের CDN boxes ISP-র কাছে রাখে।
Parallel Transcoding: একটা video কে segments-এ ভাগ করে parallel workers-এ transcode করুন। 1-hour video → 1 worker 1 hour লাগবে, 60 workers = 1 minute!
Pre-warm CDN: Popular video detect হলে immediately CDN edge locations-এ push করুন। Viral হওয়ার আগেই ready।
Storage Cost: 10x storage multiplication (multiple qualities)। Storage cheap কিন্তু at 1 exabyte scale expensive হয়। Cold storage (Glacier) use করুন old videos-এর জন্য।
Transcoding Time: 4K video transcode করতে সময় লাগে। User upload করার সাথে সাথে সব quality available হয় না — lower quality আগে, higher quality পরে।
YouTube/Netflix-এর Technologies
Backend & Processing
Storage
Delivery & Monitoring
Recommendations & Interview Tips
🤖 Recommendation System
YouTube recommendation হলো ML-based। User watch history (Cassandra) + likes + search queries → collaborative filtering। Graph DB (Neo4j) দিয়ে similar users find করা হয়। Apache Spark large-scale batch processing করে।
✅ Interview Tips
- →Upload path এবং stream path আলাদা করুন
- →CDN-first strategy mention করুন
- →Kafka দিয়ে async transcoding বলুন
- →Never DB for video binary — S3 বলুন
- →ABR / HLS explain করতে পারেন
🎯 Interview Must-Know: Viral Video Handling
Viral video (suddenly 10M views in 1 hour) — CDN automatically handle করে। Viral video CDN edge servers globally cached। 10M concurrent viewers CDN থেকে serve হয়, origin server-এ almost zero load। এজন্যই YouTube viral videos-এও crash করে না।
SUMMARY — আজকে যা শিখলাম
| Challenge | Solution | Technology |
|---|---|---|
| Video storage at scale | Object storage | Amazon S3 |
| Multiple quality formats | Transcoding pipeline | FFmpeg + Workers |
| Global fast delivery | Edge CDN | CloudFront/Akamai |
| Smooth streaming | Adaptive bitrate | HLS / DASH |
| Async processing | Message queue | Apache Kafka |
| Watch history | Time-series DB | Cassandra |
| HLS (HTTP Live Streaming) | 2-10s chunks, m3u8 playlist, ABR | Apple / Industry standard |
| DASH (MPEG-DASH) | 2-4s segments, ISO standard, codec-agnostic | MPEG / Android / Chrome |
| RTMP | Real-time ingest for live streams | Adobe (legacy ingest) |
| Netflix Open Connect | ISP-level CDN appliances | Netflix proprietary CDN |