George Willis / Data scientist

Personal project · In development

Giggity

A London gig finder that started because I kept finding out about things too late.

Why I started

I kept hearing about gigs after they had happened, or finding a band I wanted to see only to discover the tickets had sold out. The information was there, but scattered across random instagram posts, venue programmes and emails.

The gap for me was between finding an individual gig and keeping track of what was coming up. The apps and sites I used were useful starting points, but I still ended up checking venue programmes separately and trying to remember what I had already seen. I wanted to browse the venues I cared about, explore somewhere new and notice useful changes without repeating the same searches.

I approached it as a data problem first: bring the listings together, preserve their source links and work out which records described the same event. A searchable list and venue map would make that information easier to use; change tracking would help distinguish a real update from another copy of a listing.

That seemed like a manageable personal project. It has since become a much broader exercise in learning how to build an application around the data, with the original annoyance providing a useful test of what was worth adding.

From collecting listings to building an app

The first step was collecting listings from Ticketmaster. That gave me a useful starting point, but not everything I wanted to see, so I added selected venue programmes. This brought with it the dangers and legality of web scraping.

The interesting part wasn’t just getting the data into one table. Different sources describe events differently. Listings can overlap, names vary and a page can mix gigs with other types of events. Some decisions are straightforward to automate; others need a closer look.

Giggity now combines collection, local review and change tracking with a searchable web interface and an optional venue map.

What it does

  • Brings together Ticketmaster listings and selected London venue programmes.
  • Lets me search and filter events, browse venues and explore the map.
  • Tracks useful changes, such as a different date or venue.
  • Keeps links back to the original listing or ticket page.
  • Leaves uncertain cases for review rather than forcing an answer.

Getting the data right

A listing disappearing from a feed doesn’t necessarily mean the gig has been cancelled. Equally, two similar titles don’t always describe the same performance.

Those details became a large part of the project. I wanted to preserve the source information, make matching decisions understandable and avoid creating a stream of updates that weren’t actually useful.

The collection process checks what it has fetched, records what was accepted or excluded and flags cases that need attention. The aim is to spend less time checking listings manually without losing sight of where the information came from.

How it actually works

From local data collection and review to global edge deployment.

Giggity data pipeline: collect listings, check and review, publish event data, browse the app 1. COLLECT LISTINGS Ticketmaster and venue programmes 2. CHECK AND REVIEW Python, Pydantic and SQLite 3. PUBLISH EVENT DATA Reviewed JSON snapshot 4. BROWSE THE APP SvelteKit, venue map and Cloudflare

Scroll horizontally to inspect full pipeline ↔

Why I chose this stack

Python was the natural starting point for collecting and working with the data. Pydantic gave me a way to define what an event record should contain and check it before passing it further through the app.

SQLite keeps the local data straightforward to inspect and query, without needing a separate database service. It holds the current events and the information used to compare changes.

SvelteKit and TypeScript took me further beyond my usual analysis work. They gave me a way to turn the data into something I could browse and use, rather than another output file.

MapLibre adds the venue map. I kept that optional because a list is often all I need, while the map is useful when I want to explore an area.

Cloudflare hosts the app and the published event data. Keeping collection separate from the website means I can review the data locally, then publish it when it is ready.

Learning what to leave out

Before Giggity, I built an earlier version called GigRadar with a separate FastAPI backend and a PostgreSQL database. SQLAlchemy handled the database models and queries, while Alembic managed schema changes. I used Podman to run the local database in a container.

That version took me beyond collecting and analysing data. I had to connect the interface to an API, manage relationships between events and sources, and work through database migrations without losing existing records.

I also prototyped duplicate detection with pgvector, using similarity between event embeddings to suggest matches and send less certain cases for review. It was an experiment rather than a validated matching system. Similar wording alone does not establish that two listings describe the same performance.

For Giggity, I chose a simpler path: local collection and review, SQLite for current records and changes, and a published JSON snapshot for the website. That removed the need for a separate discovery API and a running PostgreSQL service. The earlier work helped me understand what those tools offered, and which parts this app actually needed.

What I’ve enjoyed learning

As a data scientist, I was already used to collecting awkward data and questioning what it showed. Building everything around that analysis was less familiar.

Giggity has pushed me to learn more about application structure, frontend development, database changes, testing and deployment. Each new part has raised another question: where should this logic live, what happens if a source changes, and how do I make the next update easier?

I’ve enjoyed that process. It is satisfying to take something that began as an irritation and gradually turn it into something I can open and use.

Where it is now

The app is online, with Ticketmaster collection, selected venue programmes, local review and change tracking in place. I’m continuing to improve the coverage and make the collection process easier to maintain.

Further ideas include following artists or venues, useful notifications and experimenting with local models through Ollama to help extract information from flyers and posters. Those are later steps; the current focus is making the listings useful.