1. Check that the yardstick measures the right thing. I nearly dropped the noise tag over an agreement number. The LLM and the keyword tagger agree on noise only 30% of the time, and I read that as "noise is too contextual to tag."
The one-line version: the keyword tagger scored "cozy" and "hidden gem" as evidence of quiet, so it called 301 of 464 cafes quiet and never once said loud — it was measuring charm, not sound.
Its keyword list put "cozy", "small cafe", "intimate", "tucked away" and "hidden gem" in the quiet bucket, each worth +2. Those words describe ambiance and appear in most cafe reviews, so almost every cafe tipped into quiet before a real noise word was weighed. The LLM reads the sentence instead of matching the word: of its 88 quiet tags carrying a quote, 87 cite an actual acoustic statement — "quiet enough to actually hear the person sitting across from you." Across the 197 cafes where the two disagree, the LLM cites supporting evidence 89% of the time. Low agreement meant my baseline was wrong, not that the feature was unbuildable.
What I did about it: 28 cafes were still displaying "Quiet" on keyword evidence alone, because the app fell back to the keyword tag wherever the LLM had none. Those now show nothing. An empty space is honest; a confident wrong label is not.
Agreement with a baseline is not quality. I replaced it with two measures that need no ground truth — how often the tagger has to answer unknown, and how often it can cite a quote for the answer it gave — and gated the monthly pipeline on those instead. A metric that would have talked me out of a working feature is worse than no metric.
2. Check with the possible options in the beginning phase. I spent hours building keyword-based filters before realising they miss the context of a review. Exploring the options first would have saved all of it.