Jobs NightWatch

Job boards show you what exists. This shows you what changed.

System architecture

Numbered edges trace a single request, from pressing Run now (or Cloud Scheduler firing) through to results on the dashboard.

EXTERNAL Your browser public URL, no sign-in judges can click anything Greenhouse Job Board API 10 company boards · public unauthenticated · one call each GOOGLE CLOUD · jobs-nightwatch · us-central1 TRIGGER Cloud Scheduler cron · every 3 hours OIDC service account Dashboard Cloud Run · server-rendered public · “Run now” button INGEST & DETECT Collector Cloud Run · private, OIDC only Deterministic core — no model SHA-256 diff → new / modified / removed relevance gate · 2752 → 295 work-authorisation hard block a hash comparison is faster, cheaper and more accurate than asking a model if two records differ Pub/Sub topic: job-changes one message per change push + OIDC · ack 600s retry with backoff fault isolation & parallelism REASON Agent Cloud Run + ADK POST /pubsub-push idempotent by content hash 4 tools · model picks which Vertex AI Gemini 3.7 Flash the ONLY model call in the whole pipeline ~35s per change STATE Firestore Native mode · keyed by (board_token, external_id) so every read is get-by-id — no queries, no composite indexes companies postings posting_history decisions profiles 1 2 GET /jobs?content=true 3 read previous state, write posting_history BEFORE overwriting 4 only changes that pass the gate 5 6 7 tools read profile & history, then write the decision 8 9
Cloud Run · Scheduler Pub/Sub Firestore Deterministic code — no model Vertex AI · Gemini External

Reading the diagram

Everything inside the blue boundary is one Google Cloud project. The dashed box above it holds the two things outside our control: the visitor’s browser and Greenhouse’s public API.

The four tiers run top to bottom — trigger, ingest & detect, reason, state. Firestore spans the full width because every tier touches it, which is exactly why it is drawn as a shared substrate rather than another box in the chain.

Why the deterministic core is nested inside the Collector

The green block is not a separate service — it is ordinary Python running inside the Collector, drawn separately because it is where the architectural decision lives. Change detection is a SHA-256 comparison and the relevance gate is string matching. Neither involves a model, and both run in about two seconds across 2752 postings.

Why Pub/Sub sits between detection and reasoning

The obvious design is a loop: for each change, call the agent. If Gemini times out on the fourteenth posting, postings fifteen through forty never run, and you are left half-finished with no record of what completed.

One message per change fixes it. Each retries independently with backoff, and they process in parallel — three changes finish in about 48 seconds rather than three sequential 35-second runs. The ack deadline is set to 600s because a 35-second job under the 10-second default would be redelivered three times mid-flight, quadrupling model spend while looking perfectly healthy.

Where the model is, and is not

Only edge 6 reaches Gemini. Everything before it is deterministic. That boundary is the central design decision: the model is spent on judgement — does this change matter to this candidate, and what would they say about it — never on comparison.