For a Text Technologies of Data Science coursework project at Edinburgh, a team of six of us built a search engine that finds a song from a lyric snippet. Optional filters covered title, artist, and album. We called it Lyrics Wizard.
My part, alongside two teammates, was the information retrieval system. That's the part worth writing up. It's where most of the interesting decisions actually were.
The dataset was large enough to make some of those decisions non-optional. Around 1.62 million songs from Spotify's API, cross-referenced with lyrics from Genius. Roughly 5 million tracks once everything was combined. At that scale, a few approaches that look appealing on paper turned out too slow to ship.
Getting the Data Was Its Own Retrieval Problem
Building the corpus took as much engineering attention as the ranking algorithm. A few decisions there shaped what the retrieval layer had to handle later.
Song metadata came from Spotify through three sequential scripts: artists first, then albums for each artist, then tracks for each album. All six of us ran those scripts in parallel on a rotating basis just to pull data fast enough.
That got us to roughly 1.62 million songs across 433 thousand albums and 142 thousand artists, stored across three MongoDB collections. Each document kept its Spotify ID for building the inverted index later.
Lyrics were the harder half. The first approach used the lyricsgenius library, which calls the Genius API twice per song: once to find it, once to fetch the lyrics. At our volume that meant constant timeout exceptions.
Switching to scraping the same data directly from Genius's pages and parsing it with BeautifulSoup and a couple of regex passes (one for the lyrics, one for metadata like title and release date) turned out roughly three times faster in practice. That's what we shipped with.
The API wasn't the wrong choice conceptually. It just stopped being the right tool once volume got large enough that per-song round trips became the bottleneck.
The corpus itself needed cleaning before retrieval could trust it. Some of what Genius had indexed as "songs" were actually literature, Shakespeare among them, mixed in because Genius hosts more than just song lyrics.
Rather than hand-filtering those out, we used a signal Spotify already gave us for free: its danceability score. It came back negative for that kind of non-musical text and positive for actual songs. That one existing field did the filtering work a custom classifier would otherwise have had to do.
On top of that, we ran a language check with the langdetect library over Genius's own language field. Some non-English lyrics were still slipping through, and the system was scoped to English only.
Three Query Shapes, Three Scoring Paths
The system splits on what the user actually gave it. It doesn't force every query through one generic ranker.
A lyrics-only query goes through the lyric search algorithm. A filter-only query, meaning artist name, album, or title, goes through TF-IDF. A query with both goes through a combined path that blends the two.
Each input shape needs a different kind of scoring. Handling all three with one pipeline would have meant compromising on all of them.
Lyric Search: Two Signals That Disagree on Purpose
The lyric search algorithm is really two searches run in parallel and then blended.
The first is a pattern search. Tokens are preprocessed, but deliberately without removing stop words. A half-remembered lyric snippet loses meaning if you strip the small connecting words out of it.
If the query is a single token, every instance of that term is returned directly. If there's more than one token, the algorithm generates permutations that preserve the original left-to-right order.
It starts from the full query and shortens it one token at a time from either end. Then it checks each pattern against the index until it has collected 15 matching songs or run out of patterns.
This rewards an exact, in-order phrase match over a scattered bag-of-words match. That's the right bias for lyrics. Word order is part of what makes a lyric recognizable.
The second is a rarity search. It runs plain TF-IDF over the same tokens and returns the top 15 songs by term rarity. Word order doesn't matter here at all. This catches songs where the query terms are distinctive, even if the user's word order is a little off. That happens constantly with half-remembered lyrics.
The two get combined with a simple weighted formula:
final_score = rarity_search_score * 0.7 + pattern_search_score * 0.3TF-IDF gets the larger weight because it's more forgiving of an imperfect memory of the exact phrasing. That's the common case.
The exact-order pattern match gets less weight because it's a stronger signal when it fires, but fires less often. Treating it as equal to TF-IDF would have let it dominate the ranking on the rare queries where it hits, at the expense of everything else.
Filters Reuse the Same TF-IDF Machinery
When the query is filter-only (title, artist, or album), TF-IDF runs directly against the positional index built for that filter type and returns the top 15 by score. There's no pattern search step here.
Filter values don't carry the same sequential structure lyrics do. An artist name isn't something a user half-remembers the word order of the way they might a line from a song.
Combining Both When the User Gives Both
When a query includes both lyrics and a filter, the system runs the lyric search to get its top 15 songs. Then it looks up the corresponding filter score for each of those candidates.
Since one song can appear under multiple albums or artists, only the maximum filter score for each song is kept. The final score blends the two:
final_score = 0.7 * lyrics_score + 0.3 * max_filter_scoreThe same 0.7/0.3 split shows up in both places. In both cases, the harder and more information-dense signal gets the majority weight. That's either TF-IDF or the full lyric search. The second signal only acts as a booster, not an equal partner.
Once that pattern held up in one place, we reused it rather than inventing a new weighting scheme for the second.
Long Queries Get Trimmed Before Anything Else Runs
Sometimes a query comes in longer than 10 tokens, like someone pasting in a whole verse. When that happens, it gets reduced to the 10 most frequent tokens in the positional index before any of the scoring above runs.
Permutation generation is why this matters. The number of order-preserving permutations grows fast with token count. Running that step on a 20-token query without trimming first would have made the whole system noticeably slower for no real gain in match quality.
What We Tried for Fuzzy Matching That Didn't Survive Contact With the Data
A meaningful chunk of the engineering time went into approaches that didn't make it into the final system. I think those are more instructive than the ones that did.
We first tried Double Metaphone, a phonetic algorithm meant to encode similarly-pronounced words to the same token. We hoped it would help match a lyric the way someone actually misheard it rather than how it's spelled.
It was more demanding about matching exact pronunciation than real misheard lyrics turned out to need. It didn't help the way we expected.
Next we tried N-grams with edit distance, a more standard way to catch near-matches. That ran into a harder problem: it was computationally expensive at the scale of roughly 5 million tracks. It wasn't practical to run in the actual system.
We ended up back at plain stemming, deliberately without stop word removal. That's a simpler and cheaper technique than either of the two we tried first. It's the one that actually survived contact with the dataset's real size.
We also tried query expansion, which means generating additional related terms to broaden a lyric query. Different variants were tested. Expanding based on similarity from the same artist or similar tokens didn't produce better results.
Sequencing matters in lyric search in a way that generic term-similarity expansion doesn't respect. We dropped it in favor of relying on the combined search algorithm to do that work instead.
Keeping Retrieval Fast Once the Index Got Big
Two infrastructure problems showed up once the dataset was actually loaded. Neither was about the ranking math. They were about making the same math run fast enough to feel instant.
The positional index file got large enough that reading it was slow before a single score was even computed. We fixed the worst of that by encoding document names with their Spotify ID rather than a longer descriptive string. A small change, purely about making lookups cheaper.
Separately, hitting MongoDB directly while scoring tokens during a live search was slow enough to matter. So we generated a JSON file server-side ahead of time and read from that instead of the database during the scoring pass.
Neither change touched what got ranked. They changed how fast the system could rank it. That matters just as much once real users are waiting on a response instead of a notebook cell finishing.
None of this was security theater either. The app used CSRF tokens on its forms and restricted direct MongoDB access to an IP allowlist covering just the team. A low bar, but a real one. Worth doing even on a coursework project handling a database this size.
How We Actually Measured the Ranking
Rather than reporting one summary number, we built 100 of our own test cases from real lyric snippets. For each one, we checked where the correct song landed in the results.
Three numbers came out of that. 95.4% of the time the correct song appeared somewhere in the top 10. 87.2% of the time it was the literal top result. 93.8% of the time it was in the top 3.
Keeping those as three separate numbers instead of collapsing them into one score matters. A system that gets the right answer on the first try most of the time behaves very differently from one that usually needs the user to scan through several results. Both might report a similar "found it eventually" number.
Where We'd Take Ranking Next
The most interesting idea we identified but didn't build was ranking songs by popularity using emotional tags rather than raw play counts. It was based on a personalized PageRank random walk over a graph of tag similarity.
PageRank captures how central a tag is within a network of related tags and songs. That behaves differently from a simple cosine similarity score or counting how often tags co-occur.
Two songs can share very few tags directly and still be strongly connected through the graph if enough intermediate songs bridge them. A plain similarity score would miss that entirely.
We left this as future work rather than shipping it. It fits the same philosophy as everything above: before reaching for a bigger model, check whether the ranking is actually missing a structurally different signal. Not just a stronger version of the one it already has.
The General Lesson
Ranking design here wasn't one algorithm. It was a small set of complementary signals (exact-order pattern matching, term-rarity TF-IDF, and filter-based scores) combined with weights chosen to reflect which signal we trusted more by default.
Plus a trimming step that exists purely so the elegant version of the algorithm doesn't fall over on an ordinary real query. None of the fuzzy-matching ideas that didn't make it in were bad ideas.
They just didn't hold up once actual data volume and actual user behavior were in the picture. That's usually where a retrieval system's real constraints show up first.
Learn More
The full project, including the IR system and ranking logic discussed here, is open source: Lyrics Search Engine on GitHub.