[Wed 18 Feb] Scaling from one cinema in Belfast to 240 venues across London

[alistairjcbrown]

[alistairjcbrown]

Notes:

Scaling From 3 Cinemas to 240 Venues: What Broke and What Evolved

Key points to cover:

The retrieve/transform split

• Originally, retrieve grabbed the main page and transform figured out if it needed more. That made transform impure — it was making network calls, which got messy
• The clean separation: retrieve does all the fetching (even if it needs to parse the DOM to find links to follow), transform makes zero network calls. Just takes inputs, produces a schema. That's the guarantee
• This matters practically: GitHub Actions has time limits, and if you're using self-hosted runners, you don't want your transforms suddenly needing network access on a Pi

The variety of retrieval strategies

• Single page: one big page with everything you need — the dream
• Main page + listing pages: e.g. Curzon — get the main page to know what's on, then fetch each film's listing page for rich data needed for matching
• JSON/API endpoints: the best — raw structured data, no DOM parsing needed
• GraphQL: same benefit, structured data, no HTML wrangling
• Cheerio: for HTML parsing — like jQuery but no actual DOM, lets you do selectors and extraction. Used for a lot of the simpler sites
• Playwright: for the heavier sites that need a real browser
• Complex multi-page setups: e.g. the Science Museum, where listing pages and booking pages are separate things and you have to search/match to connect them
• The BFI: a massive site that loads slowly and inconsistently — takes the longest to extract from

Shared cinema platforms

• OCAPI (used by Odeon and possibly Vue) — a standard set of endpoints, so one unified codebase handles two of the biggest cinema chains
• Savoy and similar systems: when you find a new cinema and discover it's powered by a platform you already support, it's a joy — just point the existing module at the new venue
• When a cinema migrates between platforms you already know, updating is trivial

Event platforms as a data source

• Not every screening venue has its own website with listings — some only list events on platforms like Eventbrite, Dice, or Savvy
• The pipeline pulls all London film-specific events from these platforms once per retrieve run
• Each venue's transform can call out to event sources by passing in its name/address/coordinates, and the source returns matching events
• Some venues are "event source only" — no website to retrieve, no HTML to transform. Their transform just returns whatever the event source found
• Nice pattern: new venue shows up on Eventbrite? Minimal effort to add it — the event data is already being pulled daily, you just need to register the venue and let it match

⚠️ Needs: specific examples per retrieval type, maybe a diagram showing the different paths data can take from venue to final output

[alistairjcbrown]

This one might actually be two posts — the retrieve/transform architecture could be separate from the "zoo of different venue types" content. Or it could be one longer piece.