TF-IDF, short for term frequency times inverse document frequency, is a formula from information retrieval that scores how important a word is to one document compared with a whole collection. It mattered when search engines ranked pages by matching words. Modern Google reads meaning through neural language systems, BERT and MUM among them, so TF-IDF survives mostly as a sales pitch inside content tools that promise a score to hit. This post covers what the formula actually does, with the math, the honest history of what replaced it inside both retrieval and ranking, why keyword-density targets mislead, and the one small job TF-IDF is still genuinely good at.
The short version
- TF-IDF weighs a term's frequency in your page against its rarity across many pages.
- Real engines refined it into BM25 decades ago, then neural systems took over ranking.
- RankBrain, neural matching, BERT, and MUM shifted Google from matching words to matching meaning.
- Use TF-IDF for subtopic discovery if you like; never as a density quota.
How the math works, with a worked example
Two numbers multiplied. Term frequency asks how often a word appears in your document. Inverse document frequency asks how rare that word is across the whole collection, usually as the logarithm of total documents divided by documents containing the word, so common words collapse toward zero. Walk it through: in a collection of 1,000 pages, the word "the" appears on all 1,000, so its IDF is log(1000/1000), which is zero, and any amount of repetition scores nothing. The word "endodontics" appears on 10 of the 1,000, so its IDF is log(1000/10), a meaningful weight, and a page using it eight times scores high for it. The formula concludes the page is probably about endodontics, and it is probably right. That insight, credited to Karen Sparck Jones in 1972, let retrieval systems rank documents against queries without understanding either one. Elegant, cheap to compute, and blind to meaning: it cannot tell "cheap SEO" from "SEO is not cheap", and it has no idea that "root canal specialist" and "endodontist" are the same person.
Search engines outgrew raw TF-IDF twice
Here is the detail the tool vendors skip. Production search engines moved past raw TF-IDF even within word-matching retrieval. BM25, the family of scoring functions that refined it with term saturation and document length normalization, became the standard decades ago, and open-source search software has defaulted to BM25-style scoring for years. Saturation alone breaks the density pitch: in BM25, the fifth occurrence of a term adds far less than the second, and additional repetition asymptotes toward nothing. So a tool selling raw TF-IDF comparisons trails not just modern Google but the retrieval technology its own name invokes. Then engines outgrew word matching itself, which is the second and bigger move.
The timeline from counting words to reading meaning
-
RankBrain arrived in 2015
Google's first deep-learning ranking system, built to interpret the never-seen-before queries that make up a large share of daily searches by mapping them to concepts rather than strings. The first public signal that word overlap was no longer the game.
-
Neural matching followed in 2018
Connects queries to pages that answer them without sharing vocabulary. A page can rank for a phrase it never contains because the system relates ideas, not words, which no term-frequency spreadsheet can model even in principle.
-
BERT landed in ranking in 2019
A language model that reads words in both directions at once, so prepositions, negation, and word order finally count. "Parking on a hill with no curb" stopped ranking pages about curbs. Google said at launch it touched roughly one in ten English queries, and it has run in ranking ever since.
-
Passage ranking and MUM pushed past the page
Passage ranking, rolled out after BERT, scores individual sections, so one buried answer can rank on its own merits. MUM, introduced in 2021, understands across languages and formats and now underpins how Google handles complex queries. Each step moves further from anything a word counter can imitate, and the AI answers now sitting on top of results are built on the same meaning-first machinery.
The common thread is embeddings: words, sentences, and whole documents represented as points in space where distance means similarity of meaning. Two pages about the same topic sit close together even if their vocabularies barely overlap, and a query lands near the pages that answer it rather than the pages that echo it. Against that machinery, a density spreadsheet is a sundial next to a clock.
What each approach can actually see
| Situation | TF-IDF sees | Neural relevance sees |
|---|---|---|
| Synonyms ("endodontist" vs "root canal specialist") | Two unrelated terms | The same concept |
| Negation ("SEO is not cheap") | A page about cheap SEO | The opposite claim |
| Word order and prepositions | Nothing; a bag of words | Grammar that changes meaning |
| A page missing the exact phrase | Irrelevant | Possibly the best answer |
| Repeating a term ten more times | A higher score | The same topic, now stuffed |
Where TF-IDF still earns its keep
None of this makes the formula useless; it makes the formula useless as a ranking target. TF-IDF still does honest work where cheap and explainable beats deep and opaque: internal site search on small collections, keyword extraction that pulls the distinctive terms out of a document, deduplication passes, and quick first-cut relevance in data pipelines that will never see a GPU. Machine-learning courses still teach it first because you can compute it by hand and reason about every number it produces. That is also the fair way to read Google's public comments on the subject: its search advocates have repeatedly called TF-IDF an old retrieval metric and steered site owners away from tools built on it, not because the math is wrong, but because tuning your writing toward it tunes it for a system Google stopped being years ago.
Why density tools mislead
- False precision. A score of 2.3 percent looks like engineering, but no engine publishes a target, so the number is a dartboard with decimals.
- Wrong direction of causation. Top pages share vocabulary because they cover the topic well. Copying their word counts copies the symptom, not the cause.
- Stuffing with a dashboard. Chasing a term quota in finished copy produces the exact unnatural repetition spam systems are trained to catch.
- The honest use. Corpus comparison as a discovery step: see which subtopics and entities the ranking pages discuss that yours skips, then decide as an editor whether your page should cover them.
A fair question: if term weighting still lives somewhere in the retrieval plumbing, why not target it anyway? Because the layer you can influence with word counts is the layer that only decides whether your page enters the candidate pool, and a page that genuinely covers its topic clears that bar without trying. The layers that decide position read meaning, structure, and authority. Effort spent tuning frequencies is effort taken from the only inputs those layers reward.
What to do instead of chasing a score
-
Map the intent before the vocabulary
Start from keyword research that groups terms by what the searcher wants, so each page owns one topic instead of thin variants competing with each other. The grouping work kills most density questions before they start.
-
Outline from what the winners cover, then add what they lack
Read the top three ranking pages and list their subtopics; this is the legitimate version of what TF-IDF tools automate. Cover the topic fully, then add the thing they cannot copy: your pricing, your mistakes, your data.
-
Name entities plainly and answer up front
Say who and what the page is about in ordinary sentences, put the direct answer in the opening, and let headings say what each section is. This entity-first construction is the working core of semantic SEO, and it is what meaning-reading systems reward.
-
Finish with page-level craft, not word counts
Titles and headings that state the topic, internal links with descriptive anchors, clean structure a machine can quote. That discipline is on-page SEO, and it is what TF-IDF tools gesture at without delivering. Write for the reader who searched, and the vocabulary the tools measure shows up on its own.
Where Egochi fits
Egochi builds pages for the meaning-reading systems that actually rank them: intent-mapped keyword sets, complete topical coverage, and entity-forward copy that AI answers can quote. That work runs through our SEO services, and no client of ours has ever been handed a density score to hit.
Questions people ask about TF-IDF
Is TF-IDF a Google ranking factor?
Not in the way the tools imply. Term weighting of the TF-IDF family sits somewhere deep in the retrieval plumbing of every search engine, but Google has spent a decade layering meaning-based systems on top of word counting. Hitting a TF-IDF score does not move rankings, and Google engineers have publicly waved people off the metric.
What is a good keyword density?
There is no target number, and no search engine ever published one. Write the term where a reader needs it: the title, the H1, the opening answer, and wherever the copy genuinely discusses it. If repeating it once more would read badly aloud, you have passed the useful amount.
What replaced TF-IDF in search?
Inside retrieval, BM25-family scoring refined it decades ago. Inside ranking, meaning-based systems took over: RankBrain in 2015, neural matching in 2018, BERT in 2019, passage ranking and MUM after that. These systems compare what text means, represented as embeddings, rather than counting which words it contains.
Are TF-IDF content tools worth using?
As term-discovery aids, sometimes; as score targets, no. A corpus comparison can surface subtopics the top-ranking pages cover and yours skips, which is useful editorial input. The failure mode is treating the output as a quota and sprinkling words into finished copy, which reads badly and adds nothing Google cannot already infer.
Is TF-IDF the same as keyword density?
No. Keyword density is a bare percentage of one document, with no reference point. TF-IDF at least compares your page against a collection, which is why it beat density in early retrieval. Both share the fatal limit: they count words, and modern ranking systems read meaning.
Does keyword stuffing still get penalized?
Yes. Stuffing sits in Google spam policies, and modern language systems make unnatural repetition easier to detect, not harder. The practical risk today is less a manual penalty and more quiet mediocrity: stuffed pages read badly, earn no links, and lose to pages people actually want to cite.
