New applications in a council every morning

Goal: every morning, collect the applications that are new in one council, store them, and never pay twice for the same one within a month.

Approach

• Ask for the council's applications received in the last 14 days (authority plus since), and page through with cursor.
• Store each one keyed on id, inserting or updating, so overlapping windows never create duplicates.
• The overlap is free: an application you already received this month is not charged again. You pay once a month for each application in the window.
• Why 14 days and not 1: since filters on start_date, the date the council received the application, and councils often publish applications days later. A longer window catches late arrivals.

Code

// Run once a morning (cron: 0 7 * * *). Node 18+.
const KEY = process.env.PLANNING_SIGNAL_API_KEY;
const COUNCIL = "Camden";

// Look back 14 days: councils often publish an application some days after
// they received it, and start_date is the date received.
const since = new Date(Date.now() - 14 * 86400_000).toISOString().slice(0, 10);

let cursor = null;
let fresh = 0;
do {
  const url = new URL("https://planningsignal.co.uk/api/v1/applications");
  url.searchParams.set("authority", COUNCIL);
  url.searchParams.set("since", since);
  url.searchParams.set("limit", "100");
  if (cursor) url.searchParams.set("cursor", cursor);
  const res = await fetch(url, { headers: { Authorization: `Bearer ${KEY}` } });
  const body = await res.json();
  if (!res.ok) throw new Error(`${res.status} ${body.code}`);
  for (const app of body.data) {
    if (await db.upsertApplication(app)) fresh++; // your storage, keyed on app.id
  }
  cursor = body.next_cursor;
} while (cursor);
console.log(`${fresh} new applications`);

What it costs

Roughly the number of applications the council receives in a month, plus the 14-day overlap again at the start of each month (the free-repeat list starts again on the 1st). A busy London borough receives a few hundred applications a month, which fits API Starter. Check X-Records-Remaining as you go.

Variations

• Around a site instead of a council: replace authority with postcode and radius_m.
• Only some kinds of work: add q, for example q=basement.
• Several councils: one loop per council. Get exact names from /api/v1/authorities (free).