How I Built DealerProxy.ca: Every Car at 3,000 Canadian Dealerships, Tracked Daily

The question I wanted answered was simple: when a dealer drops a price, how far did it fall, how long had the car been sitting, and had it already failed to sell somewhere else? Dealer sites answer none of that. They show today. So on September 20 I started a crawler for the 40 Lexus dealerships in Canada, and within three days it had BMW, Audi, Mercedes-Benz, Land Rover, Subaru, Toyota, Kia, Mazda and Honda too. Two weeks later DealerProxy.ca tracks 3,254 dealerships of every brand sold here and a little under 300,000 vehicles.

The negotiation half came once the data was there. If the site knows a car has been on the lot for 40 days and has been marked down twice, it is in a good position to ring the dealership on your behalf. That is what the name means: DealerProxy stands in for you at the dealer.

The screenshots were captured from the live site on October 3, 2026, and the figures in this article (3,254 dealerships, 298,017 vehicles, 20,520 price drops, 812 verified extractors) are from that day. They are a snapshot, not a promise about what the site shows when you open it.

Two processes, one event log

The system has the same shape as the inventory tool I wrote about in August: a Next.js app that owns the event store and exposes command and query endpoints, and a background processor that does the slow work and talks to the app only through those endpoints. The processor never opens the main database. It asks the app what is due, does it, and reports back with a command.

It is CQRS with event sourcing, designed with Event Modeling. Every fact the system knows is an append-only row in one SQLite table, main.db, and everything the site shows is a projection of that table. The pipeline is four steps.

The modelling choice that made the rest fall into place is the aggregate identity. A vehicle with a VIN is the same vehicle wherever it turns up, so the VIN is the stream id. When the same car appears on a different dealership’s site, that is one more event in one stream, not a new record. A car without a VIN gets the dealer’s stock number scoped to the dealership, or failing that the listing URL, hashed into a novin- id.

The dealership, the crawl, the customer and the negotiation are the other aggregates, all in the same table. Events carry one photo per vehicle as a BLOB, which turned out to matter for the indexes; more on that below.

A real browser, and every byte it receives

Most dealer sites render their inventory with JavaScript, and several refuse plain HTTP clients outright. So the spider is a headless Chromium driven by Playwright, six dealerships side by side, each one crawled politely with a delay between pages and robots.txt honoured. It never leaves the dealership’s own hosts.

Its artifact is a single SQLite file per crawl, crawls/<dealership>/<date>-<crawlId>.db. It is an append-only event log of its own, and it holds every response the browser received, verbatim, in a BLOB column, plus the DOM of each page after its JavaScript ran. I built the same thing for DIY SEO Hub and it has paid for itself twice here: when an extractor misreads a site, the exact bytes it misread are on disk, and when a model’s reading is in doubt, so is the page it was shown.

CREATE TABLE events (
  id          INTEGER PRIMARY KEY AUTOINCREMENT,
  event_type  TEXT NOT NULL,   -- ResponseReceived, PageRendered, ...
  url         TEXT,
  event_data  TEXT NOT NULL,   -- headers, status, timing as JSON
  body_sha256 TEXT,
  body        BLOB             -- the response, byte for byte
);

Images are the exception. Photos were most of a crawl’s bytes, so only their headers are kept, and the interpreter fetches a vehicle’s first photo itself when the crawl does not hold it. The spider refuses to start a crawl with under ten gigabytes free, writes each database on a work path and moves it into place when done, and an hourly retention job deletes a crawl file once it has been interpreted or superseded. What it showed lives on as events.

Finding all of a dealer’s stock took persistence, and every rule in the link selector came from a real first crawl: scroll the listing until it stops growing, click “load more”, turn the pages of a listing paged by a JavaScript button, recognise the many spellings of ?page=, follow the links to the certified and demo lists, find the real stock list when the URL on file is a model showroom, and stay out of filter links, model catalogues and articles.

Reading a dealer page: a model first, then code that earns the job

The interpretation job opens a finished crawl, reduces each rendered page to its visible text, links, image URLs, vehicle data attributes and any schema.org vehicle JSON-LD, and has a model list the vehicles on it. The reduction usually shrinks a page by more than 90%, and a long listing is split into parts rather than cut off, because cutting at a character budget would silently drop every vehicle below the cut.

The model is asked for a strict JSON schema, through a small gateway that is toggled with one variable. openai is gpt-5-mini on the Responses API. ollama is a local Qwen 3.5 served from my own GPU boxes. hybrid is what production runs: a page goes to the cheapest tier that can take a page of its size and has a healthy host, and climbs a tier when that one is resting, too small, or failed on the page.

LLM_PROVIDER=hybrid
OLLAMA_BASE_URL=https://qwen.fusenv.com,https://qwen3.fusenv.com,...,\
  https://[email protected]/api?model=mistralai/mistral-nemo&tier=1#openai
LLM_LOCAL_MAX_TOKENS=2500       # pages bigger than this skip the GPU boxes
LLM_BACKLOG_TARGET_HOURS=1      # past this, busy tiers are skipped at once

A model call per page is the running cost, and with 300,000 vehicles that cost is real. So the second half of the design is dealership extractors: code written for a site that reads it with no model at all. Nearly every dealer site in Canada runs on one of a handful of platforms (D2C Media, SM360, eDealer, Convertus, Magnetis, and a long tail), so there is one engine per platform and a short definition per dealership naming its engine, any options, and a baseline of what its crawls looked like when verified.

An extractor is not trusted because it exists. Before its reading is used, a gate checks that at least 80% of the pages look like the platform it was written for, that no page is in a shape it does not expect, that it found vehicles, and that the share of them with a VIN, price, year and model is not far below its baseline. Any of those failing sends the crawl to the model and records LlmFallbackUsed with the reason, which puts the dealership on the “extractor needs rebuilding” list in diagnostics.

Verification is the part I am most pleased with. Crawl files are deleted soon after interpretation, so a script snapshots what the interpreter reads of every crawl, and each snapshot gets an expected.json: the vehicles the model recorded when it read that same crawl. A dealership is registered only when an engine finds at least 95% of the model’s vehicles in every crawl the model read and agrees on at least 90% of each core fact. The script writes the registry, a test file per dealership, and a status page. By September 28 there were 29 platform engines and 812 dealerships verified.

The model is the referee, and the model is not always right. It misreads VINs, takes an internal id for a stock number, reads a similar-vehicles carousel as stock, and calls a demo new. The verification recognises those cases and reports them apart, and an overrides file records, per dealership and with the reason, the fields where the model is known to read that site wrong. It is the only hand-edited part of the registry. Of the vehicles set aside on September 28, the biggest categories were condition (demo or certified read as new), stock numbers the model took from the URL, and about 2,000 vehicles it invented, listed twice, or gave a placeholder VIN.

The app decides what happened

The processor only ever says “this is what the scan saw”. It sends one record-vehicle-observation command per vehicle, whether an extractor or a model produced it, and the command handler replays the vehicle’s stream and appends only what is new.

if (!state.seen) return [VehicleFirstSeen(observed)];

if (state.dealershipId !== dealershipId) decided.push(VehicleMovedDealership);
else if (!state.listed)                  decided.push(VehicleRelisted);

if (observed.price   !== null && observed.price   !== known.price)   decided.push(VehiclePriceChanged);
if (observed.mileage !== null && observed.mileage !== known.mileage) decided.push(VehicleMileageChanged);
// ...details that changed, photos only when none of the new ones were there before

// Nothing changed: record the sighting once per crawl, so "last seen" stays honest
if (decided.length === 0 && state.lastCrawlId !== crawlId) decided.push(VehicleSighted);

Two rules in there took a few crawls to learn. A scan that did not publish a fact never erases a fact an earlier scan established, because dealer pages drop fields all the time. And a reading lists a few of a vehicle’s photos, not always the same few, so the photos count as changed only when none of the new ones were there before. Completing the interpretation appends VehicleDelisted for every vehicle the dealership listed before and the scan no longer found, with one exception: a scan that finds no vehicles at all delists nothing, because that is a broken crawl, not an empty lot.

The two jobs are the automation pattern from Event Modeling. A read model lists the work, the processor does it and reports back, and the resulting event takes the item off the list. A failed interpretation is retried up to five times with a growing delay. A failed crawl is retried after an hour, and crawls cut short by a processor restart are failed at startup and retried straight away.

The site is a projection

projections.db holds the vehicles, dealerships, crawls and negotiations tables that the pages read. A checkpointed projector catches it up at the top of every query, so the site is never behind the log by more than one request. In production the projection lives in the server’s memory: the whole log is replayed at boot, a batch at a time so the server keeps answering, and the health endpoint reports 503 until that first catch-up is done. Delete the file and it is rebuilt from the events.

Replaying hundreds of thousands of events taught me two SQLite lessons I did not have from the last time I wrote about event sourcing on SQLite. The first was that compiling the projector’s SQL on every event was most of the replay, so statements are now prepared once per connection. The second was the photos. The BLOB sits before timestamp and version in each row, so reading those columns off the table walked every photo’s overflow pages: the boot replay was reading 115,000 photos it did not need and taking 460 seconds. Two covering indexes that hold every column except the photo answer the replay and a stream load from the index alone.

With the projection in place, the questions I started with are each a page. Price drops lists every vehicle for sale below its first asking price, sortable by the drop in dollars or percent, the number of price changes, or time on market.

Sales rank estimates sales per dealership from the vehicles that left the lot, counted on the day a crawl found them gone. It is an estimate, and the first weeks overstate it: a dealer whose first crawl was a partial one shows a burst of “sales” when the next crawl finds the rest, and a site that changed its listing shape shows the opposite. The tracked column says how long each number has had to settle.

Then the site negotiates the car for you

A customer joins with $5.00 of credit, buys more in packs through Stripe, and asks the site to negotiate a vehicle: a target price, extras to have thrown in, and notes. Tokens spent on their behalf are charged at a dollar per million, and a negotiation pauses when the credit runs out and carries on when more is bought. Every one of these is a slice in the same event model, with a read model as its to-do list and the processor working it.

The phone side reuses the gateway I built for giving OpenClaw a phone. The negotiation brief tells the agent how to handle a phone menu, a hold and a voicemail, to stop ringing only once sales is reached, and never to accept anything: it brings the offer to the buyer. The processor reads the transcript afterwards for offers the agent did not record itself.

The whole flow runs end to end in CI against stand-ins for Stripe, Migadu, the mail servers, the phone gateway and OpenAI. The other end-to-end suite builds the app on an empty data directory, starts a fixture dealership and a stand-in OpenAI API, and runs the real processor: add a dealership, crawl, interpret, search, change the fixture’s inventory, crawl again, and watch the price drop and the delisting appear in the vehicle’s history. No test calls a real dealership.

Knowing when it is broken

A pipeline that spans a browser, a model pool and two processes fails quietly, so the diagnostics page reads the event store and the projections and says what is wrong: a stalled pipeline, crawl failures clustered by cause, dealerships that stopped yielding vehicles after earlier crawls found them, extractors that stopped fitting their site, how deep the interpretation backlog is and how many hours it will take to drain at the current rate. Token usage per model host for the last hour is on the same page, which is how I watch the hybrid routing actually behave.

The diagnostics read from five-minute rollups the projector keeps rather than replaying the log, because the first version replayed it on every load and stalled the event loop while it did. The processor logs a minute’s CPU profile by area five minutes after start and hourly, which is how I found that reading crawls and parsing their pages belonged on a pool of worker threads.

Shipping it

A push to main runs both unit suites and both end-to-end suites, builds two images, and sends a dispatch to my devops repository, whose workflow runs both containers on my own server. It is the same two-repository pipeline every project here uses.

What I would tell someone starting the same build: keep every byte the browser receives, because you will want it the week after you deleted it; let the model read first and write the code second, with the model’s own readings as the test suite; and put the decision about what changed in one place, on the app side, so an extractor and a model produce the same history. The rest is a to-do list and a loop.

If you are shopping for a car in Canada, DealerProxy.ca is live. Find the car, look at its history, and if the price has been falling, let the site make the call.