TF-IDF in SEO and why counting keywords stopped working

TF-IDF, short for term frequency times inverse document frequency, is a formula from information retrieval that scores how important a word is to one document compared with a whole collection. It mattered when search engines ranked pages by matching words. Modern Google reads meaning through neural language systems, so TF-IDF survives mostly as a sales pitch inside content tools that promise a score to hit. This post covers what the formula actually does, the honest history of what replaced it, why keyword-density targets mislead, and the small job TF-IDF is still genuinely good at.

How the math works

Two numbers multiplied. Term frequency asks how often a word appears in your document. Inverse document frequency asks how rare that word is across the whole collection, so common words score near zero. The word "the" appears constantly everywhere, so its weight collapses. A word like "endodontics" appears rarely across the web, so a page using it repeatedly is probably about endodontics. That insight, credited to Karen Sparck Jones in 1972, let retrieval systems rank documents against queries without understanding either. Elegant, cheap to compute, and blind to meaning: it cannot tell "cheap SEO" from "SEO is not cheap".

What search engines actually ran

Here is a detail the tool vendors skip. Production search engines did not stop at raw TF-IDF even within word-matching retrieval. BM25, the family of scoring functions that refined it with saturation and length normalization, became the standard decades ago, and open-source search software has defaulted to BM25-style scoring for years. So a tool selling you raw TF-IDF comparisons trails not just modern Google but the retrieval technology its own name invokes.

The timeline of what replaced word counting

1. RankBrain, 2015

   Google's first deep-learning ranking system, built to interpret the never-seen-before queries that make up a large share of daily searches by mapping them to concepts rather than words.

2. Neural matching, 2018

   Connects queries to pages that answer them without sharing vocabulary. A page can rank for a phrase it never contains because the system relates ideas, not strings.

3. BERT, 2019

   Reads words in both directions at once, so prepositions and word order finally count. Google said at launch it touched roughly one in ten English queries, and it has run in ranking ever since.

4. Passage ranking and MUM onward

   Google now scores individual passages, not just whole pages, and its later language systems handle queries across formats and languages. Every step moves further from anything a word counter can imitate.

The common thread is embeddings: words, sentences, and whole documents represented as points in space where distance means similarity of meaning. Two pages about the same topic sit close together even if their vocabularies barely overlap. Against that machinery, a density spreadsheet is a sundial next to a clock.

Why density tools mislead

• False precision. A score of 2.3 percent looks like engineering, but no engine publishes a target, so the number is a dartboard with decimals.
• Wrong direction of causation. Top pages share vocabulary because they cover the topic well. Copying their word counts copies the symptom, not the cause.
• Stuffing with a dashboard. Chasing a term quota in finished copy produces the exact unnatural repetition spam systems are trained to catch.
• The honest use. Corpus comparison as a discovery step: see which subtopics and entities the ranking pages discuss that yours skips, then decide as an editor whether your page should cover them.

What to do instead

The work that replaced the score chase is intent coverage. Start from keyword research that maps what buyers actually ask and groups terms by intent, so each page owns one topic instead of thin variants competing with each other. Then build each page to answer its topic completely: the direct answer up front, the subtopics a real expert would cover, the entities named plainly, titles and headings that say what the page is. That page-level craft is on-page SEO, and it is the discipline TF-IDF tools gesture at without delivering. Write for the reader who searched, and the vocabulary the tools measure shows up on its own.

Questions people ask about TF-IDF

Is TF-IDF a Google ranking factor?

Not in the way the tools imply. Term weighting of the TF-IDF family sits somewhere deep in the retrieval plumbing of every search engine, but Google has spent a decade layering meaning-based systems on top of word counting. Hitting a TF-IDF score does not move rankings, and Google engineers have publicly waved people off the metric.

What is a good keyword density?

There is no target number, and there never was one published by any search engine. Write the term where a reader needs it, which is the title, the H1, the opening answer, and wherever the copy genuinely discusses it. If repeating it once more would read badly aloud, you have passed the useful amount.

What replaced TF-IDF in search?

Inside retrieval, BM25-family scoring refined it decades ago. Inside ranking, meaning-based systems took over: RankBrain in 2015, neural matching in 2018, BERT in 2019, and passage ranking after that. These systems compare what text means, represented as embeddings, rather than counting which words it contains.

Are TF-IDF content tools worth using?

As term-discovery aids, sometimes; as score targets, no. A corpus comparison can surface subtopics the top-ranking pages cover and yours skips, which is useful input. The failure mode is treating the output as a quota and sprinkling words into finished copy, which reads badly and adds nothing Google cannot already infer.

Does keyword stuffing still get penalized?

Yes. Stuffing sits in Google spam policies, and modern language systems make unnatural repetition easier to detect, not harder. The practical risk today is less a manual penalty and more quiet mediocrity: stuffed pages read badly, earn no links, and lose to pages people actually want to cite.