ORIGINAL SYSTEM DESIGN QUESTION
Bounded authorized web crawler
Crawl an explicitly authorized set of public domains with duplicate detection and resource budgets.
Original exercise · intermediate. Editorial role relevance does not imply a company has asked this question.
Requirements and constraints
- One million allowed URLs; 100 fetch workers; average page 200 KB.
- Respect domain scope and access restrictions; links must not broaden the authorization set.
- State assumptions and justify security, cost, operational ownership and data retention.
Pressure round
An allowed page links to a loop generating infinite URL variants.
Build and explain the design
Requirements & scope
Who uses it? What must it do? What is explicitly out of scope? State latency, durability and privacy goals.
Self-review guidance
Define the actors, required operations and out-of-scope behavior. Explicit requirement: Respect domain scope and access restrictions; links must not broaden the authorization set.
Capacity estimates
State assumptions, peak requests/second, data volume and retention. Show your arithmetic and units.
Self-review guidance
Use this workload: One million allowed URLs; 100 fetch workers; average page 200 KB. Convert rates and payload sizes into peak compute/storage assumptions; show units, headroom and what remains unmeasured.
APIs & data model
Describe endpoints/events, entity keys, indexes, authorization, and idempotency.
Self-review guidance
Define allowed origins, canonical URL identity, fetch provenance and budget limits.
Architecture & data flow
Describe components and request/event paths. Use plain text arrows to sketch your design; explain each responsibility.
Self-review guidance
Use a deduplicated frontier, per-origin scheduling and bounded parsers.
Failures & observability
What happens on overload, duplicate events, partial failures or lost dependencies? Discuss monitoring and recovery.
Self-review guidance
Pressure-test: An allowed page links to a loop generating infinite URL variants. Describe bounded recovery, deduplication, observability and a user-visible failure state.
Tradeoffs & alternatives
Compare two plausible approaches. State the costs, consistency model and what would make you change the design.
Self-review guidance
Compare canonicalization completeness with preserving meaningful query identity.
Design guidance describes assumptions and tradeoffs; no architecture has been simulated or benchmarked.
Practice this question
Save your work privately, explain decisions and return to your revision queue.
Open practice workspaceBrowse the question library →