Why my first chunking strategy was wrong

Sole author. Self-directed design exercise, not shipped software. No production deployment, no team, no users. There are no measured results below — only design targets and the reasoning behind them. Where I couldn't validate something, I say so.

Focus: Enterprise SaaS · Asynchronous architecture · API reliability · SLA definition Target roles: Technical Program Manager · IT Project Manager · Business Systems Analyst


Why I picked this problem

As a senior engineer on an HR and payroll platform, I watched large bulk uploads collide with payroll deadlines more than once. An HR admin uploads a CSV of tens of thousands of employee changes, the request hangs, the browser eventually gives up, and nobody knows how many records actually landed. Then support gets a P1 during the worst possible hour.

We never got the runway to redesign it properly. This is the design I would have proposed.


The problem

The original ingestion path is synchronous: one HTTP request holds a database connection open while it validates schemas, maps fields, and writes every record inside a single transaction.

That works fine at 500 records. It falls apart at scale:

The important reframe: the user does not need a fast database write. They need a fast acknowledgment, a UI that stays usable, and row-level errors precise enough to fix without starting over. Those are three different requirements, and the original design conflated all of them into "make the request finish."

EDITOR'S NOTE — DELETE THIS BLOCK BEFORE PUBLISHING. The old version claimed "informal conversations with people in my network." Pick one: (a) It happened → replace this block with what they actually told you. How many people, what surprised you, and what you changed in the design because of it. The most valuable version is something you got wrong before you talked to them. (b) It didn't happen → delete this block and change nothing. The section above already stands on your firsthand experience, which is stronger than vague research anyway.


What I got wrong first

My initial design chunked large jobs into batches of 5,000 records. The reasoning was throughput: fewer round trips, less per-batch overhead, faster overall completion. On paper it was the efficient choice.