Laravel LLM Integration: Queues and Real-Time Frontend UI

Direct API Calls Will Kill Your Queue Workers

Making synchronous HTTP requests to Anthropic or OpenAI inside a web request lifecycle is asking for 504 Gateway Timeouts. LLM API latencies swing wildly. A simple prompt might take 800ms, while a complex context window with 10k tokens can easily stretch to 12 seconds. If three users hit an endpoint that calls Claude 3.5 Sonnet synchronously, your FPM worker pool fills up immediately, blocking every subsequent HTTP request to your application.

The solution isn't just pushing the call into a queue. If you push 500 LLM calls to your default Redis queue, your standard background jobs like sending transactional emails or processing Stripe webhooks will stall behind third-party API rate limits. Anthropic limits tier 1 accounts to 50 requests per minute. When you hit that ceiling, OpenAI or Anthropic throws a 429 status code. If your job retries immediately without backoff strategies, you exhaust your quota and crash queue workers with unhandled exceptions.

You need a dedicated queue connection, explicit rate limiting using Laravel's Redis::throttle(), exponential backoff, and real-time frontend updates to inform users without forcing browser refreshes. Here is how we built this for a clean laravel llm integration using Laravel 12 with PHP 8.3 on the backend and Next.js 16 with React 19 on the frontend.

Structuring Prompts with Dedicated Service Classes

Stop putting raw prompt strings inside your jobs or controllers. When you inline prompts, testing becomes annoying and modifying model instructions requires scanning through application logic. Build a dedicated prompt builder or service class that handles variable substitution, token estimation, and output formatting.

In PHP 8.3, we take advantage of typed class constants and readonly properties to define clean prompt templates. Here is a clean pattern for managing structured prompts before dispatching jobs.

<?php

namespace App\Services\Llm;

readonly class SummaryPromptBuilder
{
    private const string SYSTEM_INSTRUCTION = <<<'TEXT'
You are a technical document parser. Extract key insights and actionable items.
Return ONLY valid JSON with keys: "summary" (string), "key_points" (array of strings), and "action_items" (array of strings).
Do not include markdown code block backticks in your final output.
TEXT;

    public function __construct(
        private string $documentText,
        private int $maxWords = 250
    ) {}

    public function buildPayload(): array
    {
        return [
            'model' => 'gpt-4o-mini',
            'response_format' => ['type' => 'json_object'],
            'messages' => [
                [
                    'role' => 'system',
                    'content' => self::SYSTEM_INSTRUCTION,
                ],
                [
                    'role' => 'user',
                    'content' => "Summarize the following document in under {$this->maxWords} words:\n\n{$this->documentText}",
                ],
            ],
            'temperature' => 0.2,
        ];
    }
}

This design separates your prompt strategy from your queue execution logic. When OpenAI updates JSON mode configurations or Anthropic changes tool definition formats, you modify single builder classes rather than touching queue workers or database logic.

Queuing the Job with Rate Limiting and Retry Backoff

In Laravel 12, create a dedicated queue for LLM jobs (for instance, queues: 'llm'). Configure your supervisor process to allocate dedicated workers to this queue so long-running LLM jobs don't starve critical app queues.

The job must handle three specific failures: HTTP 429 rate limits, 5xx server errors from the provider, and JSON parse failures when responses don't match expected schemas. Here is a complete queued job class implementing exponential backoff and WebSockets broadcasting via Laravel Reverb.

<?php

namespace App\Jobs;

use App\Events\DocumentSummarized;
use App\Models\Document;
use App\Services\Llm\SummaryPromptBuilder;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Redis;

class ProcessDocumentSummary implements ShouldQueue
{
    use Queueable;

    public int $tries = 5;
    public array $backoff = [10, 30, 90, 300];

    public function __construct(
        public Document $document
    ) {}

    public function handle(): void
    {
        Redis::throttle('openai-api')
            ->allow(30)
            ->every(60)
            ->then(function () {
                $this->executeRequest();
            }, function () {
                $this->release(15);
            });
    }

    private function executeRequest(): void
    {
        $prompt = new SummaryPromptBuilder($this->document->content);

        $response = Http::withToken(config('services.openai.key'))
            ->timeout(45)
            ->post('https://api.openai.com/v1/chat/completions', $prompt->buildPayload());

        if ($response->failed()) {
            if ($response->status() === 429) {
                $retryAfter = (int) $response->header('Retry-After', 30);
                $this->release($retryAfter);
                return;
            }

            $response->throw();
        }

        $data = $response->json();
        $rawContent = $data['choices'][0]['message']['content'] ?? null;

        if (!$rawContent) {
            throw new \RuntimeException('Empty payload returned from OpenAI.');
        }

        $parsed = json_decode($rawContent, true);

        if (json_last_error() !== JSON_ERROR_NONE) {
            Log::error('Failed to parse LLM JSON output', ['raw' => $rawContent]);
            $this->fail(new \RuntimeException('Invalid JSON returned from model.'));
            return;
        }

        $this->document->update([
            'summary' => $parsed['summary'],
            'metadata' => [
                'key_points' => $parsed['key_points'] ?? [],
                'action_items' => $parsed['action_items'] ?? [],
                'tokens_used' => $data['usage']['total_tokens'] ?? 0,
            ],
            'status' => 'completed',
        ]);

        event(new DocumentSummarized($this->document->id, $parsed));
    }
}

Notice how we set Http::timeout(45) on the HTTP client. OpenAI endpoints occasionally hang when GPU clusters experience high traffic. Without an explicit timeout, your PHP worker hangs until the CLI timeout kills it, leaving jobs stuck in an ambiguous state.

The Redis::throttle() call prevents your worker nodes from overwhelming your API limits across multi-server deployments. If five Horizon workers run simultaneously, they coordinate through Redis before firing outbound requests.

Keeping the UI Responsive with React 19 and Next.js 16

Polling an endpoint every 2 seconds via setInterval works for small demos, but it generates hundreds of unnecessary HTTP requests when scaled across active users. A modern stack uses WebSockets. In Laravel 12, Laravel Reverb provides first-party WebSocket serving directly within your infrastructure.

On the client, Next.js 16 App Router components paired with React 19 hooks handle real-time state updates cleanly. When the user clicks Summarize Document, we immediately push an optimistic state to the UI showing a pending status. When the job finishes processing and fires DocumentSummarized, Echo updates the React client state without forcing a manual refresh.

'use client';

import { useState, useEffect } from 'react';
import { echo } from '@/lib/echo';

interface SummaryData {
  summary: string;
  key_points: string[];
  action_items: string[];
}

interface DocumentSummaryViewerProps {
  documentId: number;
  initialStatus: string;
  initialSummary: SummaryData | null;
}

export function DocumentSummaryViewer({
  documentId,
  initialStatus,
  initialSummary,
}: DocumentSummaryViewerProps) {
  const [status, setStatus] = useState<string>(initialStatus);
  const [summary, setSummary] = useState<SummaryData | null>(initialSummary);

  useEffect(() => {
    if (status === 'completed') return;

    const channel = echo.private(`documents.${documentId}`)
      .listen('DocumentSummarized', (e: { payload: SummaryData }) => {
        setSummary(e.payload);
        setStatus('completed');
      });

    return () => {
      channel.stopListening('DocumentSummarized');
    };
  }, [documentId, status]);

  if (status === 'processing') {
    return (
      <div className="p-4 border rounded shadow-sm bg-gray-50">
        <p className="text-sm text-gray-600 animate-pulse">
          Generating summary with AI worker... You can safely leave this page.
        </p>
      </div>
    );
  }

  if (status === 'failed') {
    return (
      <div className="p-4 border rounded border-red-200 bg-red-50">
        <p className="text-sm text-red-600">
          Failed to process document summary. Please retry.
        </p>
      </div>
    );
  }

  if (!summary) return null;

  return (
    <div className="space-y-4 p-6 border rounded-lg bg-white">
      <h3 className="text-xl font-bold">Document Summary</h3>
      <p className="text-gray-800">{summary.summary}</p>
      
      {summary.key_points.length > 0 && (
        <div>
          <h3 className="font-semibold text-md mb-2">Key Points</h3>
          <ul className="list-disc pl-5 space-y-1">
            {summary.key_points.map((point, i) => (
              <li key={i}>{point}</li>
            ))}
          </ul>
        </div>
      )}

      {summary.action_items.length > 0 && (
        <div>
          <h3 className="font-semibold text-md mb-2">Action Items</h3>
          <ul className="list-disc pl-5 space-y-1">
            {summary.action_items.map((item, i) => (
              <li key={i}>{item}</li>
            ))}
          </ul>
        </div>
      )}
    </div>
  );
}

Production Gotchas That Will Bite You

When deploying this setup to production environments, several edge cases emerge that don't appear in local testing.

1. Token Limit Errors on Payload Construction

If users upload a 100-page PDF, sending raw text directly inside a user prompt will trigger HTTP 400 invalid context length errors. You must calculate estimated tokens before dispatching or building payloads. PHP packages like gkreitz/gpt-3-encoder or native character truncation prevent malformed requests. If the payload exceeds 8,000 tokens for standard models, truncate or chunk the text beforehand.

2. Memory Leaks in Long-Running Workers

Queue workers running under php artisan queue:work keep application state in memory between job executions. If your prompt builder or HTTP response handling creates large string buffers or circular object references, memory consumption creeps up over time. Always set a maximum memory limit on your workers (php artisan queue:work --memory=128) or run queue:listen in development environments to catch memory growth early.

3. Silent Timeout Mismatches

If your Nginx reverse proxy has a 60-second read timeout, your Laravel HTTP client has a 45-second timeout, and your queue worker has a 30-second --timeout, your worker process gets forcibly killed while waiting for the LLM response. The HTTP request never cleanly finishes, no failure exception is caught, and the database record stays stuck in a processing state forever. Align these numbers: worker timeout should be longer than HTTP client timeout by at least 15 seconds.

Decoupling LLM generation from synchronous request threads through queues, Redis throttling, and WebSockets ensures your app remains fast and resilient regardless of third-party API latency spikes.