Stemming vs Lemmatization in NLP: Differences with Python Code

If you’ve spent any time in NLP, you’ve run into stemming vs lemmatization — two techniques that both reduce words to their base form, but work very differently and produce very different results.

Both are core text pre-processing steps. Both strip words down to their roots. But one is fast and crude, the other is slow and precise. Choosing the wrong one for your task can silently hurt your model’s accuracy.

In this guide, you’ll understand exactly what each technique does, see the differences side by side with real Python code, and know precisely when to use each one.

Table of Contents

1. Why Word Normalization Matters in NLP
2. What is Stemming?
3. Types of Stemming Algorithms
4. What is Lemmatization?
5. Stemming vs Lemmatization: Side-by-Side Code
6. Where Stemming Goes Wrong
7. Where Lemmatization Gets It Right
8. The Role of POS Tags in Lemmatization
9. Stemming vs Lemmatization: Full Comparison Table
10. When to Use Stemming vs Lemmatization
11. Stemming and Lemmatization in a Full NLP Pipeline
12. FAQs

Why Word Normalization Matters in NLP

Before comparing stemming vs lemmatization, let’s understand the problem they both solve.

Consider these four words: run, runs, running, ran

To a human, these are all forms of the same concept. To a machine — without normalization — they’re four completely different tokens with no connection. This means a model trained on “running” will fail to recognize “ran” as related.

Word normalization (reducing variants to a common base form) fixes this. It:

• Reduces vocabulary size significantly
• Groups semantically related words together
• Improves model generalization on unseen data
• Reduces noise in text classification and search tasks

Both stemming and lemmatization perform this normalization — just with different trade-offs.

What is Stemming?

Stemming is a rule-based technique that reduces words to their root form — called a stem — by chopping off prefixes and suffixes using heuristic patterns.

The key word is heuristic. Stemming doesn’t look up words in a dictionary or understand grammar. It applies mechanical rules like “remove -ing”, “remove -ed”, “remove -es”. Fast, simple, and sometimes wrong.

"running"    → "run"      # correct
"studies"    → "studi"    # not a real word
"better"     → "better"   # misses "good"
"caring"     → "care"     # correct
"happiness"  → "happi"    # not a real word
"wolves"     → "wolv"     # not a real word

The output of a stemmer is called a stem — and it’s often not a real English word. That’s intentional and acceptable when you only care about grouping, not readability.

Types of Stemming Algorithms

There are several stemming algorithms, each with different aggressiveness levels.

Porter Stemmer (Most Popular)

The classic. Developed in 1980, still widely used. Applies a series of rule-based suffix stripping passes.

from nltk.stem import PorterStemmer
import nltk

stemmer = PorterStemmer()

words = ["running", "studies", "happiness", "wolves",
         "caring", "better", "flies", "generously", "electricity"]

print(f"{'Word':<15} {'Porter Stem'}")
print("-" * 30)
for word in words:
    print(f"{word:<15} {stemmer.stem(word)}")

Output:

Word            Porter Stem
------------------------------
running         run
studies         studi
happiness       happi
wolves          wolv
caring          care
better          better
flies           fli
generously      generous
electricity     electr

Snowball Stemmer (Porter2 — More Aggressive)

An improved version of Porter. Supports multiple languages.

from nltk.stem import SnowballStemmer

snowball = SnowballStemmer("english")

words = ["running", "studies", "happiness", "wolves",
         "caring", "better", "flies", "generously", "electricity"]

print(f"{'Word':<15} {'Snowball Stem'}")
print("-" * 30)
for word in words:
    print(f"{word:<15} {snowball.stem(word)}")

Output:

Word            Snowball Stem
------------------------------
running         run
studies         studi
happiness       happi
wolves          wolv
caring          care
better          better
flies           fli
generously      generous
electricity     electr

Lancaster Stemmer (Most Aggressive)

The most aggressive stemmer. Strips words down to minimal forms — can over-stem significantly.

from nltk.stem import PorterStemmer, SnowballStemmer, LancasterStemmer

porter = PorterStemmer()
snowball = SnowballStemmer("english")
lancaster = LancasterStemmer()

words = ["running", "happiness", "electricity", "generously", "computational"]

print(f"{'Word':<16} {'Porter':<14} {'Snowball':<14} {'Lancaster'}")
print("-" * 58)
for word in words:
    print(f"{word:<16} {porter.stem(word):<14} {snowball.stem(word):<14} {lancaster.stem(word)}")

Output:

Word             Porter         Snowball       Lancaster
----------------------------------------------------------
running          run            run            run
happiness        happi          happi          happy
electricity      electr         electr         elect
generously       generous       generous       gen
computational    comput         comput         comput

Lancaster over-stems "generously" to "gen" — barely recognizable. Porter and Snowball are safer choices for most use cases.

What is Lemmatization?

Lemmatization reduces words to their lemma — the canonical, dictionary form of a word. Unlike stemming, it actually understands the word’s meaning and grammatical context.

"running"    → "run"       # correct (verb)
"studies"    → "study"     # correct (real word)
"better"     → "good"      # correct (comparative adjective)
"caring"     → "care"      # correct (verb)
"happiness"  → "happiness" # (no change — already base form)
"wolves"     → "wolf"      # correct (real word)

Every output of a lemmatizer is a real, valid English word that you’d find in a dictionary. That’s the fundamental difference from stemming.

Lemmatization uses lexical resources (like WordNet) and POS tags to determine the correct base form. This is why it’s slower — it’s doing real linguistic work.

Stemming vs Lemmatization: Side-by-Side Code

Let’s see the full difference between stemming vs lemmatization on the same word list.

import nltk
from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk.corpus import wordnet

nltk.download('wordnet', quiet=True)
nltk.download('omw-1.4', quiet=True)

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

test_words = [
    "running", "studies", "better", "wolves",
    "caring", "flies", "happiness", "went",
    "children", "geese", "electricity", "ate"
]

print(f"{'Word':<14} {'Stemmed':<14} {'Lemmatized (verb)':<20} {'Lemmatized (noun)'}")
print("-" * 65)
for word in test_words:
    stemmed = stemmer.stem(word)
    lemma_v = lemmatizer.lemmatize(word, pos='v')   # as verb
    lemma_n = lemmatizer.lemmatize(word, pos='n')   # as noun
    print(f"{word:<14} {stemmed:<14} {lemma_v:<20} {lemma_n}")

Output:

Word           Stemmed        Lemmatized (verb)    Lemmatized (noun)
-----------------------------------------------------------------
running        run            run                  running
studies        studi          study                study
better         better         better               better
wolves         wolv           wolves               wolf
caring         care           care                 caring
flies          fli            fly                  fly
happiness      happi          happiness            happiness
went           went           go                   went
children       children       children             child
geese          gees           geese                goose
electricity    electr         electricity          electricity
ate            ate            eat                  ate

This output reveals everything about the difference between stemming vs lemmatization:

• "went" → Stemmed: "went" (unchanged, wrong). Lemmatized as verb: "go" # correct
• "wolves" → Stemmed: "wolv" (not a word). Lemmatized as noun: "wolf" # correct
• "geese" → Stemmed: "gees" (wrong). Lemmatized as noun: "goose" # correct
• "ate" → Stemmed: "ate" (unchanged). Lemmatized as verb: "eat" # correct
• "children" → Stemmed: "children" (unchanged). Lemmatized as noun: "child" # correct

Lemmatization handles irregular forms perfectly. Stemming completely misses them.

Where Stemming Goes Wrong

Stemming has two failure modes: over-stemming and under-stemming.

from nltk.stem import PorterStemmer

stemmer = PorterStemmer()

# Over-stemming: different words collapse to the same stem (false positives)
over_stem_examples = {
    "universal": "univers",
    "university": "univers",   # completely different meaning!
    "universe": "univers"      # all map to same stem
}

print("Over-stemming (different words → same stem):")
for word, expected_stem in over_stem_examples.items():
    actual = stemmer.stem(word)
    print(f"  {word:<15} → {actual}")

print()

# Under-stemming: related words don't map to same stem (false negatives)
print("Under-stemming (related words → different stems):")
under_examples = ["alumnus", "alumni", "alumna", "alumnae"]
for word in under_examples:
    print(f"  {word:<15} → {stemmer.stem(word)}")

Output:

Over-stemming (different words → same stem):
  universal       → univers
  university      → univers
  universe        → univers

Under-stemming (related words → different stems):
  alumnus         → alumnus
  alumni          → alumni
  alumna          → alumna
  alumnae         → alumna

"universal", "university", and "universe" have very different meanings but collapse to the same stem "univers". A search engine using this stemmer would return results for "universe" when you searched for "university" — a real accuracy problem.

Where Lemmatization Gets It Right

import spacy
nlp = spacy.load("en_core_web_sm")

# Irregular forms — stemming fails, lemmatization handles perfectly
irregular_sentences = [
    "The geese flew south while the children ran home.",
    "She went to the store and ate an apple.",
    "The mice were bigger than the men expected.",
    "He is better at swimming than his brothers."
]

print("Lemmatization handling irregular forms:\n")
for sent in irregular_sentences:
    doc = nlp(sent)
    pairs = [(token.text, token.lemma_) for token in doc
             if token.text.lower() != token.lemma_ and token.is_alpha]
    if pairs:
        print(f"Sentence: {sent}")
        for original, lemma in pairs:
            print(f"  {original:<12} → {lemma}")
        print()

Output:

Lemmatization handling irregular forms:

Sentence: The geese flew south while the children ran home.
  geese        → goose
  flew         → fly
  children     → child
  ran          → run

Sentence: She went to the store and ate an apple.
  went         → go
  ate          → eat

Sentence: The mice were bigger than the men expected.
  mice         → mouse
  were         → be
  bigger       → big
  men          → man

Sentence: He is better at swimming than his brothers.
  is           → be
  better       → good
  swimming     → swim
  brothers     → brother

Every irregular form — geese→goose, flew→fly, went→go, ate→eat, mice→mouse, better→good — handled correctly. No stemmer gets this right.

The Role of POS Tags in Lemmatization

This is the most important and most misunderstood aspect of lemmatization. The same word can have different lemmas depending on its part of speech.

from nltk.stem import WordNetLemmatizer
import nltk

nltk.download('averaged_perceptron_tagger', quiet=True)

lemmatizer = WordNetLemmatizer()

# "better" as adjective vs verb
word = "better"
print(f"'better' as adjective (pos='a'): {lemmatizer.lemmatize(word, pos='a')}")
print(f"'better' as verb     (pos='v'): {lemmatizer.lemmatize(word, pos='v')}")
print(f"'better' default (no pos)     : {lemmatizer.lemmatize(word)}")

print()

# "saw" — completely different meanings
word = "saw"
print(f"'saw' as verb (pos='v'): {lemmatizer.lemmatize(word, pos='v')}")  # past tense of "see"
print(f"'saw' as noun (pos='n'): {lemmatizer.lemmatize(word, pos='n')}")  # a cutting tool

Output:

'better' as adjective (pos='a'): good
'better' as verb     (pos='v'): better
'better' default (no pos)     : better

'saw' as verb (pos='v'): see
'saw' as noun (pos='n'): saw

Without a POS tag, lemmatizer.lemmatize("better") returns "better" — wrong. With pos='a' (adjective), it correctly returns "good".

This is why spaCy is preferred for lemmatization in production — it automatically determines POS tags before lemmatizing:

import spacy
nlp = spacy.load("en_core_web_sm")

sentences = [
    "He saw the man with a saw.",        # "saw" used as verb AND noun
    "The better solution is to lemmatize properly."
]

for sent in sentences:
    doc = nlp(sent)
    print(f"Sentence: {sent}")
    for token in doc:
        if token.is_alpha:
            print(f"  {token.text:<12} POS: {token.pos_:<6} → Lemma: {token.lemma_}")
    print()

Output:

Sentence: He saw the man with a saw.
  He           POS: PRON   → Lemma: he
  saw          POS: VERB   → Lemma: see
  the          POS: DET    → Lemma: the
  man          POS: NOUN   → Lemma: man
  with         POS: ADP    → Lemma: with
  a            POS: DET    → Lemma: a
  saw          POS: NOUN   → Lemma: saw

Sentence: The better solution is to lemmatize properly.
  The          POS: DET    → Lemma: the
  better       POS: ADJ    → Lemma: good
  solution     POS: NOUN   → Lemma: solution
  is           POS: AUX    → Lemma: be
  to           POS: PART   → Lemma: to
  lemmatize    POS: VERB   → Lemma: lemmatize
  properly     POS: ADV    → Lemma: properly

spaCy correctly identifies both uses of "saw" — as a verb (see) and as a noun (saw) — in the same sentence. NLTK without POS tags would get both wrong.

Stemming vs Lemmatization: Full Comparison Table

When to Use Stemming vs Lemmatization

Use stemming when:

• Speed is critical and you’re processing millions of documents
• You’re building a search/information retrieval system where approximate matching is acceptable
• Exact word forms don’t matter — you just need grouping
• Dataset is so large that lemmatization would be a bottleneck
• You’re doing keyword-based tasks, not meaning-based tasks

Use lemmatization when:

• Word meaning and accuracy matter
• You’re building chatbots, QA systems, or language understanding pipelines
• The text contains irregular word forms (past tense verbs, plurals)
• You need output that’s human-readable or will be displayed
• You’re doing sentiment analysis, text summarization, or semantic search
• You’re preparing data for transformer-based models

Use neither when:

• You’re using BERT, GPT, or other transformer models — they use subword tokenization and don’t need normalization. Stemming or lemmatizing before feeding into a transformer can actually hurt performance.

Stemming and Lemmatization in a Full NLP Pipeline

Here’s how both fit into a real preprocessing pipeline, with a toggle to switch between them:

import re
import nltk
import spacy
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

nltk.download('punkt', quiet=True)
nltk.download('stopwords', quiet=True)
nltk.download('wordnet', quiet=True)

nlp_spacy = spacy.load("en_core_web_sm")

def preprocess(text, method="lemmatize"):
    """
    Full NLP preprocessing pipeline.
    method: 'stem' or 'lemmatize'
    """
    # Step 1: Lowercase + clean
    text = text.lower()
    text = re.sub(r'[^a-z\s]', '', text)

    # Step 2: Tokenize
    tokens = word_tokenize(text)

    # Step 3: Remove stop words
    stop_words = set(stopwords.words('english'))
    tokens = [t for t in tokens if t not in stop_words and len(t) > 2]

    # Step 4: Stem or Lemmatize
    if method == "stem":
        stemmer = PorterStemmer()
        tokens = [stemmer.stem(t) for t in tokens]
    elif method == "lemmatize":
        lemmatizer = WordNetLemmatizer()
        # Simple POS mapping for better lemmatization
        from nltk import pos_tag
        nltk.download('averaged_perceptron_tagger', quiet=True)
        tagged = pos_tag(tokens)
        def get_wordnet_pos(tag):
            if tag.startswith('J'): return 'a'
            elif tag.startswith('V'): return 'v'
            elif tag.startswith('N'): return 'n'
            elif tag.startswith('R'): return 'r'
            return 'n'
        tokens = [lemmatizer.lemmatize(word, get_wordnet_pos(tag))
                  for word, tag in tagged]

    return tokens

# Test both
sample = "The children were running faster than expected. They ate well and slept better."

stem_result = preprocess(sample, method="stem")
lemma_result = preprocess(sample, method="lemmatize")

print(f"Original  : {sample}")
print(f"\nStemmed   : {stem_result}")
print(f"\nLemmatized: {lemma_result}")

Output:

Original  : The children were running faster than expected. They ate well and slept better.

Stemmed   : ['children', 'run', 'faster', 'expect', 'ate', 'well', 'slept', 'better']

Lemmatized: ['child', 'run', 'fast', 'expect', 'eat', 'well', 'sleep', 'good']

Lemmatization correctly resolves: children→child, ate→eat, slept→sleep, better→good. Stemming misses all of these irregular forms entirely.

Conclusion

The choice between stemming vs lemmatization comes down to one question: do you need speed or accuracy?

Stemming is fast and good enough for information retrieval, search engines, and large-scale indexing where approximate grouping is the goal. Lemmatization is slower but produces real, meaningful base forms — essential for any task that needs to understand language rather than just index it.

In practice, most modern NLP pipelines use lemmatization via spaCy for classical tasks, and skip both entirely when working with transformer-based models like BERT or GPT. Learn both, understand the trade-offs, and always benchmark on your actual task.

FAQs

1. What is the difference between stemming and lemmatization in NLP?

Stemming cuts words down to a stem using mechanical rules — the output may not be a real word (e.g., "studies" → "studi"). Lemmatization maps words to their actual dictionary form using linguistic knowledge (e.g., "studies" → "study"). Stemming is faster; lemmatization is more accurate.

2. Which is better — stemming or lemmatization?

Neither is universally better — it depends on the task. Stemming is better when speed matters and exact word forms are not critical (search engines, large-scale indexing). Lemmatization is better when meaning matters — sentiment analysis, chatbots, question answering, and semantic analysis tasks.

3. Can stemming and lemmatization handle irregular words?

Stemming cannot — it applies fixed rules and fails on irregular forms like went, geese, children, ate. Lemmatization handles these correctly because it uses dictionary lookups and understands that went is the past tense of go.

4. Should I use stemming or lemmatization with BERT or GPT?

Neither. Transformer models like BERT and GPT use subword tokenization (WordPiece or BPE) and are trained on raw, unprocessed text. Applying stemming or lemmatization before feeding text into these models can actually hurt performance by distorting the input the model was trained on.

5. What Python libraries support lemmatization?

The two main options are NLTK’s WordNetLemmatizer (requires manual POS tags for best results) and spaCy (automatically determines POS tags and lemmatizes in a single call). spaCy is recommended for production pipelines.

6. What is over-stemming and under-stemming?

Over-stemming is when different words with different meanings collapse to the same stem — e.g., "university" and "universe" both becoming "univers". Under-stemming is when related words fail to map to the same stem — e.g., "alumnus" and "alumni" producing different stems. Both are accuracy problems that lemmatization avoids.

Related reading on Nomidl: Tokenization in NLP — the step that comes before stemming and lemmatization in any NLP pipeline. Also see How Does Natural Language Processing Work? for the full preprocessing pipeline, and What is Natural Language Processing? for NLP fundamentals.

External reference: NLTK Stemming and Lemmatization documentation — official reference for all NLTK stemming algorithms.

Popular Posts

• Loop Engineering Explained: From Prompt Engineering to Self-Prompting AI Agents
• Build Your First MCP Server with FastMCP: A Complete Python Tutorial
• MCP vs Function Calling: Key Differences Explained (2026)
• What Is Model Context Protocol (MCP) – A Complete Guide for AI Developers
• MCP Primitives Explained: Tools, Resources, and Prompts With Real Examples