Every morning at 11:00, the crawler behind swinghop.ch reads about forty pages from swing dance schools and organisers in French-speaking Switzerland. It sends the text and the image of each page to an LLM, Claude Sonnet, which pulls out the upcoming events. The events it creates then go through a moderation queue before they are published.
From 19 to 30 September, the crawler ran every day. The URL called by the cron job answered 200 before the crawl had even started, and the report then said “0 extracted, ok” for every source. In those twelve days it did not create a single event, published or rejected.
Nothing flagged it. I found out on 1 October, while counting events by origin for an unrelated reason.
Why no alert went off
The Claude account had hit its spending limit, and every call to the model failed. Until 29 September, image reading went through Claude alone. The free fallback added that day reads about one page a minute, too slow for a crawl.
The real problem was somewhere else. This is how the crawl handled a page whose extraction failed:
try {
return $this->extractor->extract($candidate);
} catch (\Throwable $e) {
// One unparseable page must not abort the whole source.
$this->logger->warning('Extraction failed for a candidate: {error}', ['error' => $e->getMessage()]);
return [];
}The intent was sound: one malformed page should not stop the other pages from being crawled. But that return [] returns exactly the same thing as a page with no events on it. A quiet week and a complete model outage produced the same report.
Each failure wrote a warning to the logs, but none of them raised an alert.
Telling an empty page apart from an unavailable model
First fix: if every provider fails, we throw an exception that keeps the error of each one. Before, only the error of the last provider tried came through.
final class LlmUnavailableException extends \RuntimeException
{
/**
* @param array<string, string> $errors provider name => its error message
*/
public function __construct(public readonly array $errors, ?\Throwable $previous = null)
{
$message = implode('; ', array_map(
static fn (string $name, string $error): string => $name.': '.$error,
array_keys($errors),
$errors,
));
parent::__construct('All LLM providers failed. '.$message, 0, $previous);
}
}Extraction now has three outcomes. A list, empty or not, means the model read the page. null means extraction failed for another reason, such as an unusable answer. The exception means no provider answered, and only those pages count as unread.
try {
return $this->extractor->extract($candidate);
} catch (LlmUnavailableException $e) {
throw $e;
} catch (\Throwable $e) {
$this->logger->warning('Extraction failed for a candidate: {error}', ['error' => $e->getMessage()]);
return null;
}The crawl loop then counts unread pages, instead of mixing them up with empty ones.
Pages that return null do not count towards the alert. The model did answer, and the page will be read again the next day. An unreadable answer now and then happens. The case that cost me twelve days is the one where nobody answers at all.
Alerting on the whole crawl
One unread page now and then is noise: a timeout, a slow answer. An email every time would soon be ignored. What matters is the share across the whole run.
public function modelLooksUnavailable(): bool
{
$unread = $this->unread();
return $unread > 0 && 2 * $unread > $this->sentToModel();
}If more than half of the pages sent to the model come back unread, I get an email. It gives the number of pages read out of the total, the error from each provider, and what to check: the Claude credit first, then the fallback quotas.
A run started by hand with --dry-run sends no email, since someone is already watching the output.
Not paying twice for the same page
While looking for the cause, I noticed something else: the crawl re-read every page with Claude every day, even when it had not changed. That is very likely what drained the credit in the first place.
The crawl now computes a fingerprint of what the model receives for a page, except today’s date:
public function fingerprint(string $sourceKey, EventCandidate $candidate, ?string $imageBase64, SwingGate $gate): string
{
return hash('sha256', json_encode([
$sourceKey,
$candidate->sourceUrl,
$candidate->text,
null === $imageBase64 ? null : hash('sha256', $imageBase64),
$gate->name,
EventExtractor::instructionsFingerprint(),
], \JSON_THROW_ON_ERROR));
}The extraction instructions are part of the fingerprint. When I change the prompt, every page is read again the next day, instead of waiting a week for the new prompt to take effect.
If the same fingerprint was read in the last 7 days, the page is skipped. The prompt also contains today’s date, so the model knows which day “this Saturday” means. That date is left out of the fingerprint, or the fingerprint would change every day and no page would ever be skipped. So an unchanged page is still read again once a week: if it says “every Wednesday”, that weekly read is what brings in the following Wednesdays.
One detail matters: the fingerprint is stored only after every event extracted from the page has been processed. A page that could not be read, or whose events failed to save, is read again on the next run. Without that, the memory would have hidden the outage a second time.
The result
On 4 October, the crawl re-read 3 pages out of 42, in 10 seconds instead of 8 minutes. About 93% of the Claude calls were avoided that day.
And if more than half of the pages sent fail with every provider, the crawl now emails me the same day.
A job that answers “ok” only tells you it did not crash. It does not tell you it did its work. For a crawler that depends on an LLM, count what it produces and what it could not read, and keep “nothing to find” clearly apart from “nobody looked”.