You've written this loop. I've written this loop. It's in production somewhere right now:
User::whereNull('slug')->chunk(1000, function ($users) {
foreach ($users as $user) {
$user->update(['slug' => Str::slug($user->name)]);
}
});It looks fine. It runs without error. It finishes suspiciously quickly.
And it processes roughly half your table.
Walk through it
chunk() pages with LIMIT and OFFSET:
SELECT * FROM users WHERE slug IS NULL ORDER BY id LIMIT 1000 OFFSET 0;
SELECT * FROM users WHERE slug IS NULL ORDER BY id LIMIT 1000 OFFSET 1000;
SELECT * FROM users WHERE slug IS NULL ORDER BY id LIMIT 1000 OFFSET 2000;That's correct for a result set that holds still. A backfill's result set does not hold still — the whole point of whereNull('slug') is that a row stops matching once you've given it a slug.
So: batch one takes offsets 0–999 and slugs all thousand. Those rows now fail slug IS NULL and leave the result set, which shrinks by a thousand. Everything below shifts down.
Batch two asks for offsets 1000–1999 of the new, smaller result set. That's where rows 2000–2999 now live. Rows 1000–1999 slid down into offsets 0–999 and are never visited.
Every batch skips as many rows as the last batch removed. No exception, no warning — just a job that ends early and a count() close enough to believe.
The fix is to stop counting positions and start remembering a value:
SELECT * FROM users WHERE slug IS NULL AND id > 1000 ORDER BY id LIMIT 1000;A cursor that says "everything up to here is done" cannot drift when rows leave underneath it. It's also dramatically faster on a large table: OFFSET 4000000 makes the database walk and discard four million rows before returning anything, while id > 4000000 seeks straight into the index.
Why not just put it in a migration?
Because migrations run synchronously during deploy, usually inside a transaction. A multi-million row UPDATE there means your deploy blocks until the data change finishes, your statement timeout eventually kills it half way through, and there's no cursor — so the next attempt starts from zero, redoing everything it already did.
Schema changes belong in migrations. Data changes belong in something that can be paused, resumed, throttled and watched.
That thing didn't exist, so I built it.
laravel-backfill
composer require kstmostofa/laravel-backfill
php artisan migratephp artisan make:backfill BackfillUserSlugsclass BackfillUserSlugs extends Backfill
{
public int $batchSize = 1000;
public function collection(): Builder
{
return User::query()->whereNull('slug');
}
public function process($record): void
{
$record->update(['slug' => Str::slug($record->name)]);
}
}php artisan backfill:run user-slugsThe whole package is built around one sentence:
A backfill can be killed at any instant, restarted, and must arrive at the same end state — no duplicated side effects, no skipped rows.
Every design decision is downstream of that. It's why the cursor is written after the work rather than before, why each row gets its own savepoint, and why the run lock is a unique index rather than a cache key.
Look before you leap
The command I actually reach for first:
php artisan backfill:run user-slugs --dry-run Dry run: user-slugs — nothing was written.
Rows matching ........................................... 8,412,663
Batch size .................................................. 1,000
Index ....................... indexed — Walks index PRIMARY (type=range), no sort.
Estimated duration ........................................... ~4.2h
Sampled 5 rows, rolled back:
| Row | What would change |
| 1041 | slug: null -> ada-lovelace, updated_at: … -> … |
| 1042 | slug: null -> alan-turing, updated_at: … -> … |
Side effects intercepted (these would have escaped in a real run):
mail ................. 5 across 5 rows — roughly 8,412,663 in fullFour questions answered before you commit to anything: how much work is there, is the cursor column indexed, roughly how long, and does process() actually do what you think.
Those diffs are real. Five rows are genuinely processed inside a transaction that's then rolled back — because a dry run that only prints the query tells you nothing about whether your code works. It catches the case a query preview can't:
| 1041 | no change |
| 1042 | no change |
None of the sampled rows changed. Either the work is already done,
or process() is not doing what you expect.Everything that can't be rolled back is intercepted first — mail, notifications, queued jobs, HTTP. A "dry" run that emails four million customers is the single worst thing this package could do.
That interception has one detail I want to flag, because it nearly bit me. Laravel's MailFake::raw() is an empty method that records nothing. A dry run built on Mail::fake() silently loses every Mail::raw() call and cheerfully reports that no mail would be sent — exactly the false reassurance that makes a dry run dangerous. So mail goes through the array transport instead, which sends nothing and captures everything.
The index check is narrower than you'd guess
"Is any index used" is the wrong question. A backfill can happily use an index for its WHERE clause and still sort the entire table on every single batch, because the cursor column is unindexed. That's the difference between a ten-minute job and a three-day one.
So what gets checked is whether the ORDER BY is satisfied by an index — a filesort on MySQL, a Sort node on PostgreSQL, a temp b-tree on SQLite. And the query is explained with the cursor predicate in place, which matters more than it sounds: without id > ?, SQLite reports SCAN bf_users; with it, SEARCH bf_users USING INTEGER PRIMARY KEY. Explaining the wrong query gives you a confident answer about something that never runs.
Support staff can run it
This is the part teams keep installed after the migration that prompted it is long forgotten.
class BackfillOrderReceipts extends Backfill
{
public bool $operatorRunnable = true;
public function description(): string
{
return 'Re-issue refund receipts';
}
public function parameters(): array
{
return [
Parameter::ids('order_ids', 'Order IDs')->required()->max(50_000),
Parameter::select('tone', ['formal' => 'Formal', 'friendly' => 'Friendly']),
];
}
public function collection(): Builder
{
return Order::query()
->whereNull('receipt_sent_at')
->whereIn('id', $this->parameter('order_ids', []));
}
}That appears in a separate, separately-gated panel where someone in support pastes a list of ids and presses Run. The list is split however their spreadsheet formatted it — commas, newlines, semicolons — de-duplicated, and checked against its ceiling before anything is queued. Progress reads "Working — 1,204 done so far" rather than exposing cursors and batch counts.
They can't reach anything you didn't deliberately expose, and they can't override a production guard. That's an engineer's decision, made on the engineer's dashboard.
The numbers
Measured against a real 8,000,000-row table on MySQL 8.4, with a stock 128 MB buffer pool against a 923 MB table — so these include real disk I/O, not a warm cache.
RowsBulk SQL pathEloquent path1,000,0008s~3.6m2,000,00017s~7m8,000,00075s~29m
Roughly 110,000 rows/sec on the un-hydrated path against 4,600 rows/sec with models — a 24× gap, and the most consequential choice you'll make on a large backfill. Use models when you need model logic; use processBatch() when the change is expressible in SQL.
Throughput stayed flat from 1M to 8M, which is keyset pagination doing its job. An OFFSET loop degrades as the offset grows. This doesn't.
"Killable at any instant" is worth what the test is worth
So the test doesn't simulate a crash. It forks, and the child sends itself a real SIGKILL from inside a batch transaction — uncatchable, no destructors, no shutdown handlers, no finally. Exactly what the OOM killer or a kill -9 during a deploy does to a worker.
Against the 8M-row table, killed 25 seconds in:
status: running (nothing got the chance to say otherwise)
processed_count: 3,045,000
rows actually written: 3,045,000 ← exactly equalThe counter and the data agree precisely. The batch in flight was discarded whole when the connection dropped. After resuming:
rows unprocessed: 0
processed exactly once: 8,000,000
processed more than once: 0
run rows: 1 (resumed, not restarted)Eight million rows, one hard kill, zero duplicated and zero skipped.
And because a passing test proves nothing if it would pass regardless: flipping useTransactions to false makes it fail immediately, leaving two rows written while the cursor still reads zero. The savepoint check is more interesting still — remove the per-row savepoint and SQLite and MySQL both pass, while PostgreSQL fails with SQLSTATE[25P02]: current transaction is aborted. On PostgreSQL a failed statement poisons the whole transaction, taking the other rows, the error records and the cursor with it.
Also in the box
Queue mode that chains short jobs, so a deploy costs one batch instead of the run
Adaptive throttling on replication lag — slows down before your replicas notice, pauses rather than pushing them further behind
A circuit breaker that stops a run when failures stop looking like bad rows and start looking like a bad assumption
backfill:retry-failed, so you re-process the two hundred rows that failed rather than walking eight million to reach themProduction guards — a row-count ceiling and deploy-freeze windows
Multi-tenancy with a cursor, run and lock per tenant
A Livewire dashboard and a Pulse card
261 tests, green on SQLite, MySQL 8.4 and PostgreSQL 18, with the chaos test running for real on each.
Try it
composer require kstmostofa/laravel-backfill
php artisan migrate
php artisan make:backfill BackfillUserSlugs
php artisan backfill:run user-slugs --dry-runDocumentation: https://kstmostofa.github.io/laravel-backfill/ Source: https://github.com/kstmostofa/laravel-backfill
And if you've got a chunk() backfill in production right now, it's worth a couple of minutes to check whether it did what you think it did:
User::whereNull('slug')->count();
