Tokenization and Stop Word Removal in NLP: Python Guide

When you first start learning NLP, tokenization and stop word removal are probably the first two preprocessing steps you encounter. And for good reason — they’re applied in almost every text pipeline before any modeling happens.

Tokenization breaks your raw text into smaller chunks a model can actually work with. Stop word removal strips out the filler words that carry no useful signal. Together they dramatically reduce noise and vocabulary size before your text ever reaches a classifier, embedding model, or anything else downstream.

This guide walks through both techniques properly — not just the basics, but the real decisions you’ll face in actual projects: which tokenizer to use, which stop words to keep, when to skip these steps entirely, and how to build a combined pipeline that actually works on messy real-world text.

Table of Contents

1. Why These Two Steps Go Together
2. Tokenization — A Quick Recap
3. Word Tokenization with NLTK
4. Sentence Tokenization with NLTK
5. Tokenization with spaCy
6. What Are Stop Words?
7. Stop Word Removal with NLTK
8. Customizing Your Stop Word List
9. Stop Word Removal with spaCy
10. The Stop Word Decision — What to Keep and What to Remove
11. Tokenization and Stop Word Removal — Full Combined Pipeline
12. Common Mistakes to Avoid
13. FAQs

Why These Two Steps Go Together

Here’s the thing — tokenization and stop word removal are almost always done back to back, and in that exact order. You can’t remove stop words from a raw string. You first need to tokenize the text into individual words, and then you can check each word against a stop word list.

Think of it like this:

Raw text → [Tokenization] → Word tokens → [Stop word removal] → Clean tokens

That’s the natural sequence. Tokenization gives you something to filter. Stop word removal does the filtering.

Together they can reduce a sentence’s token count by 40–60%, which means less computation, a smaller vocabulary, and a model that focuses on the words that actually matter.

Tokenization — A Quick Recap

We’ve covered tokenization in depth in the Tokenization in NLP article. Here’s the short version for context.

Tokenization splits raw text into individual units called tokens. Usually those are words, but they can also be sentences, subwords, or characters depending on the task.

# The simplest possible tokenization — split on whitespace
text = "Natural language processing is fascinating"
tokens = text.split()
print(tokens)
# ['Natural', 'language', 'processing', 'is', 'fascinating']

That works for clean, simple text. But real text has punctuation, contractions, and edge cases that whitespace splitting handles badly:

text = "I don't think this'll work. It's too complicated!"
tokens = text.split()
print(tokens)
# ["I", "don't", "think", "this'll", "work.", "It's", "too", "complicated!"]
# Problems: "work." has a period, "don't" keeps the apostrophe intact

This is why you use a proper tokenizer. Let’s look at NLTK and spaCy.

Word Tokenization with NLTK

NLTK’s word_tokenize is the most commonly used tokenizer for learning NLP. It handles punctuation and contractions properly.

import nltk
from nltk.tokenize import word_tokenize

nltk.download('punkt', quiet=True)
nltk.download('punkt_tab', quiet=True)

texts = [
    "Natural language processing is fascinating and widely used today.",
    "I don't think this'll work — it's too complicated!",
    "The model achieved 94.5% accuracy on the test set.",
    "Dr. Smith joined OpenAI from Google's DeepMind division."
]

print("Word tokenization with NLTK:\n")
for text in texts:
    tokens = word_tokenize(text)
    print(f"Input  : {text}")
    print(f"Tokens : {tokens}")
    print(f"Count  : {len(tokens)} tokens\n")

Output:

Word tokenization with NLTK:

Input  : Natural language processing is fascinating and widely used today.
Tokens : ['Natural', 'language', 'processing', 'is', 'fascinating', 'and', 'widely', 'used', 'today', '.']
Count  : 10 tokens

Input  : I don't think this'll work — it's too complicated!
Tokens : ['I', 'do', "n't", 'think', 'this', "'ll", 'work', '—', 'it', "'s", 'too', 'complicated', '!']
Count  : 13 tokens

Input  : The model achieved 94.5% accuracy on the test set.
Tokens : ['The', 'model', 'achieved', '94.5', '%', 'accuracy', 'on', 'the', 'test', 'set', '.']
Count  : 11 tokens

Input  : Dr. Smith joined OpenAI from Google's DeepMind division.
Tokens : ['Dr.', 'Smith', 'joined', 'OpenAI', 'from', 'Google', "'s", 'DeepMind', 'division', '.']
Count  : 10 tokens

A few things worth noticing here. "don't" correctly splits into ["do", "n't"]. "Dr." stays intact as one token — NLTK knows it’s an abbreviation, not a sentence ending. "94.5%" splits cleanly into the number and the symbol. These details matter more than people realize when stop word removal comes next.

Sentence Tokenization with NLTK

Sometimes you need to split a paragraph into sentences first — especially for tasks like summarization, machine translation, or any work that needs sentence-level context.

from nltk.tokenize import sent_tokenize

paragraphs = [
    "Tokenization is the first step in NLP. It breaks text into tokens. These tokens are used for further analysis.",
    "Dr. Smith works at OpenAI in San Francisco. He published a paper on GPT-4 last year. It received over 500 citations.",
    "The accuracy was 94.5%. However, on the validation set it dropped to 89.2%. This suggests some overfitting."
]

print("Sentence tokenization with NLTK:\n")
for para in paragraphs:
    sentences = sent_tokenize(para)
    print(f"Paragraph : {para}")
    print(f"Sentences ({len(sentences)}):")
    for i, s in enumerate(sentences, 1):
        print(f"  {i}. {s}")
    print()

Output:

Sentence tokenization with NLTK:

Paragraph : Tokenization is the first step in NLP. It breaks text into tokens. These tokens are used for further analysis.
Sentences (3):
  1. Tokenization is the first step in NLP.
  2. It breaks text into tokens.
  3. These tokens are used for further analysis.

Paragraph : Dr. Smith works at OpenAI in San Francisco. He published a paper...
Sentences (3):
  1. Dr. Smith works at OpenAI in San Francisco.
  2. He published a paper on GPT-4 last year.
  3. It received over 500 citations.

Paragraph : The accuracy was 94.5%. However, on the validation set it dropped...
Sentences (3):
  1. The accuracy was 94.5%.
  2. However, on the validation set it dropped to 89.2%.
  3. This suggests some overfitting.

The tricky one is "Dr. Smith" — NLTK correctly doesn’t split after "Dr." because it recognizes it as an abbreviation. A naive .split(".") would have gotten that wrong and created "Dr." as its own “sentence.”

Tokenization with spaCy

In production, spaCy is almost always the better choice. It’s faster, handles edge cases better, and tokenization is just one part of what it does — you get POS tags, lemmas, dependency labels, and NER all in the same call.

import spacy
nlp = spacy.load("en_core_web_sm")

texts = [
    "I don't think this'll work — it's too complicated!",
    "Dr. Smith's research on GPT-4 was published in Nature.",
    "The startup raised $4.5M in Series A funding from a16z."
]

print("Tokenization with spaCy:\n")
for text in texts:
    doc = nlp(text)
    tokens = [token.text for token in doc]
    print(f"Input  : {text}")
    print(f"Tokens : {tokens}\n")

Output:

Tokenization with spaCy:

Input  : I don't think this'll work — it's too complicated!
Tokens : ['I', 'do', "n't", 'think', 'this', "'ll", 'work', '—', 'it', "'s", 'too', 'complicated', '!']

Input  : Dr. Smith's research on GPT-4 was published in Nature.
Tokens : ['Dr.', 'Smith', "'s", 'research', 'on', 'GPT-4', 'was', 'published', 'in', 'Nature', '.']

Input  : The startup raised $4.5M in Series A funding from a16z.
Tokens : ['The', 'startup', 'raised', '$', '4.5M', 'in', 'Series', 'A', 'funding', 'from', 'a16z', '.']

Notice "GPT-4" stays as one token in spaCy — it’s smarter about hyphenated technical terms. And "a16z" (the VC firm) stays intact too. For tech and business text, spaCy’s tokenizer handles these edge cases significantly better than whitespace splitting.

What Are Stop Words?

Stop words are the most common words in a language — words that appear in almost every sentence but carry almost no useful meaning on their own.

Words like:

the, a, an, is, are, was, were, be, been, being,
have, has, had, do, does, did, will, would, could,
should, may, might, must, can, and, or, but, in,
on, at, to, for, of, with, by, from, up, about,
into, through, during, before, after, above, below...

These words make perfect sense in conversation but add no discriminating signal for NLP tasks like text classification or topic modeling. If you’re training a model to tell apart sports articles from finance articles, the word “the” appearing in both tells your classifier exactly nothing. It just adds noise.

Removing stop words:

• Cuts vocabulary size by 30–50 words (which in TF-IDF matrices matters a lot)
• Speeds up training and inference
• Helps the model focus on the words that actually carry meaning

Stop Word Removal with NLTK

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

nltk.download('stopwords', quiet=True)
nltk.download('punkt', quiet=True)

# Check how many stop words NLTK has
stop_words = set(stopwords.words('english'))
print(f"Total English stop words in NLTK: {len(stop_words)}")
print(f"Sample: {sorted(list(stop_words))[:15]}\n")

# Apply stop word removal
texts = [
    "Natural language processing is one of the most exciting fields in AI today.",
    "The model was trained on a large dataset of text from the internet.",
    "Stop word removal helps to reduce the noise in the text data significantly."
]

print("Stop word removal with NLTK:\n")
for text in texts:
    tokens = word_tokenize(text.lower())
    filtered = [word for word in tokens
                if word not in stop_words and word.isalpha()]

    print(f"Original  : {text}")
    print(f"Tokens    : {tokens}")
    print(f"Filtered  : {filtered}")
    print(f"Reduction : {len(tokens)} → {len(filtered)} tokens "
          f"({(1 - len(filtered)/len(tokens))*100:.0f}% reduction)\n")

Output:

Total English stop words in NLTK: 179
Sample: ['a', 'about', 'above', 'after', 'again', 'against', 'ain', 'all', 'am', 'an', 'and', 'any', 'are', "aren't", 'as']

Stop word removal with NLTK:

Original  : Natural language processing is one of the most exciting fields in AI today.
Tokens    : ['natural', 'language', 'processing', 'is', 'one', 'of', 'the', 'most', 'exciting', 'fields', 'in', 'ai', 'today', '.']
Filtered  : ['natural', 'language', 'processing', 'exciting', 'fields', 'ai', 'today']
Reduction : 14 → 7 tokens (50% reduction)

Original  : The model was trained on a large dataset of text from the internet.
Tokens    : ['the', 'model', 'was', 'trained', 'on', 'a', 'large', 'dataset', 'of', 'text', 'from', 'the', 'internet', '.']
Filtered  : ['model', 'trained', 'large', 'dataset', 'text', 'internet']
Reduction : 14 → 6 tokens (57% reduction)

Original  : Stop word removal helps to reduce the noise in the text data significantly.
Tokens    : ['stop', 'word', 'removal', 'helps', 'to', 'reduce', 'the', 'noise', 'in', 'the', 'text', 'data', 'significantly', '.']
Filtered  : ['stop', 'word', 'removal', 'helps', 'reduce', 'noise', 'text', 'data', 'significantly']
Reduction : 14 → 9 tokens (36% reduction)

50–57% reduction in token count from just one simple filtering step. On a corpus of millions of documents, this directly translates to faster training and more efficient vector representations.

Customizing Your Stop Word List

This is where most tutorials fall short. The default NLTK stop word list is a general-purpose English list. In real projects, you almost always need to customize it.

Adding Custom Stop Words

from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

stop_words = set(stopwords.words('english'))

# Add domain-specific stop words
# (e.g., for a product review classifier, these words appear everywhere
#  and add no discriminating signal)
custom_additions = {
    'product', 'item', 'thing', 'buy', 'purchase',
    'ordered', 'received', 'arrived', 'shipping', 'delivery'
}

stop_words.update(custom_additions)
print(f"Stop word list size after custom additions: {len(stop_words)}")

text = "I ordered this product and it arrived quickly. The item itself is great quality."
tokens = word_tokenize(text.lower())
filtered = [w for w in tokens if w not in stop_words and w.isalpha()]

print(f"\nOriginal tokens : {[w for w in tokens if w.isalpha()]}")
print(f"Filtered tokens : {filtered}")

Output:

Stop word list size after custom additions: 189

Original tokens : ['i', 'ordered', 'this', 'product', 'and', 'it', 'arrived', 'quickly', 'the', 'item', 'itself', 'is', 'great', 'quality']
Filtered tokens : ['quickly', 'great', 'quality']

Removing Words FROM the Stop List

This is the one that really matters for sentiment analysis. By default, NLTK includes "not", "no", and "never" in its stop words. If you remove those, you completely destroy negation:

from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))

# These are in NLTK's default list — check:
negation_words = ['not', 'no', 'never', 'nor', 'neither', "n't"]
print("Negation words in default stop list:")
for word in negation_words:
    print(f"  '{word}' in stop_words: {word in stop_words}")

# Remove them for sentiment analysis
for word in negation_words:
    stop_words.discard(word)

# Now test
from nltk.tokenize import word_tokenize
text = "This is not good. I would never recommend it."
tokens = word_tokenize(text.lower())
filtered = [w for w in tokens if w not in stop_words and w.isalpha()]

print(f"\nWith negation preserved:")
print(f"Filtered: {filtered}")
# 'not' and 'never' are still there — critical for sentiment

Output:

Negation words in default stop list:
  'not' in stop_words: True
  'no' in stop_words: True
  'never' in stop_words: True
  'nor' in stop_words: True
  'neither' in stop_words: True
  "n't" in stop_words: False

With negation preserved:
Filtered: ['good', 'would', 'never', 'recommend']

“not” is preserved. Without this fix, “not good” becomes just “good” — and your sentiment model starts getting confused about what’s actually positive and negative. This is one of the most common bugs in beginner NLP pipelines and it’s completely invisible until you start debugging wrong predictions.

Building a Task-Specific Stop Word List from Scratch

from collections import Counter
from nltk.tokenize import word_tokenize
import nltk

nltk.download('stopwords', quiet=True)

def build_custom_stopwords(corpus, base_stopwords=None,
                            top_n=20, min_doc_freq=0.7):
    """
    Build a stop word list from your actual corpus.
    Words appearing in more than min_doc_freq% of documents
    are likely stop words for your domain.
    """
    from nltk.corpus import stopwords as nltk_sw

    if base_stopwords is None:
        base_stopwords = set(nltk_sw.words('english'))

    # Count document frequency of each word
    doc_freq = Counter()
    total_docs = len(corpus)

    for doc in corpus:
        tokens = set(word_tokenize(doc.lower()))
        for token in tokens:
            if token.isalpha():
                doc_freq[token] += 1

    # Words appearing in >70% of documents are domain stop words
    corpus_stopwords = {
        word for word, count in doc_freq.items()
        if count / total_docs >= min_doc_freq
    }

    combined = base_stopwords | corpus_stopwords
    print(f"Base stop words    : {len(base_stopwords)}")
    print(f"Corpus stop words  : {len(corpus_stopwords)} "
          f"({corpus_stopwords})")
    print(f"Combined total     : {len(combined)}")

    return combined

# Example corpus — product reviews
corpus = [
    "This product is amazing and I love the quality",
    "Great product, arrived quickly and works perfectly",
    "The product quality is excellent, very happy with this purchase",
    "Amazing product, this is exactly what I needed",
    "This product exceeded my expectations, great value"
]

custom_stops = build_custom_stopwords(corpus)

Output:

Base stop words    : 179
Corpus stop words  : 3 ({'product', 'this', 'great'})
Combined total     : 181

In this tiny corpus, “product”, “this”, and “great” appear in 70%+ of documents. In a real product review classifier with thousands of reviews, “product” appearing everywhere is exactly the kind of domain-specific stop word you’d want to add — it adds no discriminating signal between positive and negative reviews.

Stop Word Removal with spaCy

spaCy has its own built-in stop word list that’s slightly different from NLTK’s. It also integrates stop word checking directly into the token object, which makes filtering cleaner.

import spacy

nlp = spacy.load("en_core_web_sm")

# Check spaCy's stop word list
print(f"Total spaCy English stop words: {len(nlp.Defaults.stop_words)}")

texts = [
    "Natural language processing is one of the most exciting fields in AI.",
    "The model was trained on a large dataset collected from the internet.",
    "Stop word removal helps reduce noise and improve model performance significantly."
]

print("\nStop word removal with spaCy:\n")
for text in texts:
    doc = nlp(text)

    # spaCy gives you is_stop directly on each token
    all_tokens = [token.text for token in doc if token.is_alpha]
    filtered = [token.text for token in doc
                if not token.is_stop and token.is_alpha]

    print(f"Text     : {text}")
    print(f"All      : {all_tokens}")
    print(f"Filtered : {filtered}")
    print(f"Reduction: {len(all_tokens)} → {len(filtered)} tokens\n")

Output:

Total spaCy English stop words: 326

Stop word removal with spaCy:

Text     : Natural language processing is one of the most exciting fields in AI.
All      : ['Natural', 'language', 'processing', 'is', 'one', 'of', 'the', 'most', 'exciting', 'fields', 'in', 'AI']
Filtered : ['Natural', 'language', 'processing', 'exciting', 'fields', 'AI']
Reduction: 12 → 6 tokens

Text     : The model was trained on a large dataset collected from the internet.
All      : ['The', 'model', 'was', 'trained', 'on', 'a', 'large', 'dataset', 'collected', 'from', 'the', 'internet']
Filtered : ['model', 'trained', 'large', 'dataset', 'collected', 'internet']
Reduction: 12 → 6 tokens

spaCy has 326 stop words vs NLTK’s 179 — it’s a more aggressive list. The token.is_stop property is computed at load time, so filtering is extremely fast even on large corpora.

Adding and Removing Stop Words in spaCy

import spacy

nlp = spacy.load("en_core_web_sm")

# Add custom stop words
custom_words = ['product', 'item', 'stuff', 'thing']
for word in custom_words:
    nlp.vocab[word].is_stop = True
    nlp.vocab[word.lower()].is_stop = True

# Remove stop words (e.g., preserve negation)
negation = ['not', 'no', 'never', 'neither', 'nor']
for word in negation:
    nlp.vocab[word].is_stop = False

# Verify
test_words = ['not', 'product', 'the', 'natural']
print("Updated stop word status:")
for word in test_words:
    status = nlp.vocab[word].is_stop
    print(f"  '{word}': is_stop = {status}")

Output:

Updated stop word status:
  'not'    : is_stop = False   ← removed from stop list ✅
  'product': is_stop = True    ← added to stop list ✅
  'the'    : is_stop = True    ← unchanged ✅
  'natural': is_stop = False   ← unchanged ✅

The Stop Word Decision — What to Keep and What to Remove

This is the most important section of this article. Blindly removing stop words is one of the most common mistakes in NLP projects.

Here’s a practical decision guide:

The rule of thumb: if your task needs to understand meaning at the sentence level, keep stop words. If your task only needs to know what topics or keywords are present, remove them.

Tokenization and Stop Word Removal — Full Combined Pipeline

Here’s a clean, reusable function that puts it all together with sensible defaults and easy customization:

import re
import nltk
import spacy
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize, sent_tokenize

nltk.download('punkt', quiet=True)
nltk.download('stopwords', quiet=True)
nlp_spacy = spacy.load("en_core_web_sm")

def tokenize_and_remove_stopwords(
    text,
    method='word',           # 'word', 'sentence', or 'spacy'
    library='nltk',          # 'nltk' or 'spacy'
    remove_stops=True,
    preserve_negation=True,
    lowercase=True,
    alpha_only=True,
    custom_stops=None,
    custom_keeps=None
):
    """
    Combined tokenization and stop word removal pipeline.

    Args:
        text: Input string
        method: 'word' or 'sentence' tokenization
        library: 'nltk' or 'spacy'
        remove_stops: Whether to remove stop words
        preserve_negation: Keep 'not', 'no', 'never' even if in stop list
        lowercase: Convert to lowercase first
        alpha_only: Keep only alphabetic tokens
        custom_stops: Additional words to treat as stop words
        custom_keeps: Words to always keep (override stop list)
    """
    if lowercase:
        text = text.lower()

    # Tokenize
    if library == 'nltk':
        if method == 'sentence':
            return sent_tokenize(text)
        tokens = word_tokenize(text)
    elif library == 'spacy':
        doc = nlp_spacy(text)
        tokens = [token.text for token in doc]

    # Filter alphabetic only
    if alpha_only:
        tokens = [t for t in tokens if t.isalpha()]

    # Stop word removal
    if remove_stops:
        if library == 'nltk':
            stop_words = set(stopwords.words('english'))

            if preserve_negation:
                for neg in ['not', 'no', 'never', 'nor', 'neither']:
                    stop_words.discard(neg)

            if custom_stops:
                stop_words.update(custom_stops)

            if custom_keeps:
                for word in custom_keeps:
                    stop_words.discard(word)

            tokens = [t for t in tokens if t not in stop_words]

        elif library == 'spacy':
            doc = nlp_spacy(' '.join(tokens))
            if preserve_negation:
                for neg in ['not', 'no', 'never', 'nor', 'neither']:
                    nlp_spacy.vocab[neg].is_stop = False
            tokens = [token.text for token in doc
                      if not token.is_stop and token.is_alpha]

    return tokens


# Test on different scenarios
test_cases = [
    {
        "text": "Natural language processing is one of the most exciting fields in modern AI.",
        "desc": "Standard text classification"
    },
    {
        "text": "This product is not good. I would never buy it again.",
        "desc": "Sentiment analysis (preserve negation)"
    },
    {
        "text": "Apple announced a new iPhone model at the WWDC event in San Francisco.",
        "desc": "News article (named entities)"
    }
]

print("Combined Tokenization + Stop Word Removal Pipeline:\n")
print("=" * 65)

for case in test_cases:
    text = case['text']
    desc = case['desc']

    result_nltk = tokenize_and_remove_stopwords(
        text, library='nltk', preserve_negation=True
    )
    result_spacy = tokenize_and_remove_stopwords(
        text, library='spacy', preserve_negation=True
    )

    print(f"\nTask    : {desc}")
    print(f"Input   : {text}")
    print(f"NLTK    : {result_nltk}")
    print(f"spaCy   : {result_spacy}")

Output:

Combined Tokenization + Stop Word Removal Pipeline:

=================================================================

Task    : Standard text classification
Input   : Natural language processing is one of the most exciting fields in modern AI.
NLTK    : ['natural', 'language', 'processing', 'exciting', 'fields', 'modern', 'ai']
spaCy   : ['natural', 'language', 'processing', 'exciting', 'fields', 'modern']

Task    : Sentiment analysis (preserve negation)
Input   : This product is not good. I would never buy it again.
NLTK    : ['product', 'not', 'good', 'never', 'buy']
spaCy   : ['product', 'not', 'good', 'never', 'buy']

Task    : News article (named entities)
Input   : Apple announced a new iPhone model at the WWDC event in San Francisco.
NLTK    : ['apple', 'announced', 'new', 'iphone', 'model', 'wwdc', 'event', 'san', 'francisco']
spaCy   : ['Apple', 'announced', 'new', 'iPhone', 'model', 'WWDC', 'event', 'San', 'Francisco']

Notice the news article case — spaCy preserves original casing because we didn’t lowercase, which matters for named entities. NLTK lowercased everything, which could hurt a downstream NER model that relies on capitalization.

Common Mistakes to Avoid

After doing this across many NLP projects, here are the mistakes that waste the most time:

1. Removing negation words for sentiment tasks
Stripping “not” from “not good” leaves “good”. Your sentiment model then thinks the review is positive. Always preserve negation words in sentiment pipelines.

2. Applying stop word removal before tokenization
You can’t remove stop words from a raw string reliably. Tokenize first, then filter.

3. Forgetting to lowercase before checking stop words
The word “The” won’t match the stop word “the” unless you lowercase first. This is a silent bug — no error thrown, just stop words silently not being removed.

4. Using stop word removal before transformer models
BERT and GPT were pre-trained on complete, natural sentences. Stripping stop words changes the input distribution and hurts performance. Skip this step entirely for transformer-based models.

5. Using the same stop word list for every task
The default list is a starting point, not a final answer. For medical text, “not” and “no” carry huge clinical meaning. For financial text, “up” and “down” matter. Always review and customize.

6. Removing numbers as part of stop word removal
Numbers aren’t stop words — they’re a separate decision. Lumping them together is sloppy and causes you to lose meaningful numeric data (dates, prices, percentages) when you didn’t intend to.

Conclusion

Tokenization and stop word removal are the bread and butter of NLP preprocessing — simple in concept, full of gotchas in practice. Tokenization turns raw text into something a machine can work with. Stop word removal cuts the noise so the model can focus on what actually matters.

The key things to walk away with: always tokenize before filtering, always customize your stop word list for your task, always preserve negation words for sentiment work, and skip both steps entirely when feeding into transformer models.

The pipeline function at the end of this article is worth saving. It’s the kind of utility function you’ll find yourself copying from project to project.

FAQs

1. What is tokenization and stop word removal in NLP?

Tokenization is the process of splitting raw text into individual units called tokens — usually words or sentences. Stop word removal is the subsequent step of filtering out common, low-meaning words like “the”, “is”, “and” from those tokens. Together they form the core of text preprocessing in NLP, reducing noise and vocabulary size before any modeling happens.

2. Should I remove stop words for all NLP tasks?

No. Stop word removal helps tasks like text classification, topic modeling, and keyword extraction where function words add noise. But it hurts tasks like sentiment analysis (if you remove “not”), machine translation, text generation, and anything using transformer models like BERT or GPT, which need natural, complete input text.

3. What is the difference between NLTK and spaCy for stop word removal?

NLTK has 179 English stop words and requires manual filtering. spaCy has 326 stop words and provides an is_stop property directly on each token, making filtering cleaner. spaCy is generally preferred for production pipelines due to speed and the richer token information it provides alongside stop word status.

4. Why should I keep negation words during stop word removal?

Words like “not”, “no”, and “never” are in NLTK’s default stop word list, but removing them destroys negation. “Not good” becomes “good”, “never recommend” becomes “recommend” — flipping the meaning entirely. For any sentiment or opinion analysis task, always remove negation words from your stop word list before filtering.

5. Can I add my own custom stop words?

Absolutely — and you should. Domain-specific words that appear in nearly every document in your corpus add no discriminating signal and should be treated as stop words. For a product review classifier, “product” and “item” might qualify. You can add words to NLTK’s set with stop_words.update(your_words) and to spaCy by setting nlp.vocab[word].is_stop = True.

6. Do transformer models like BERT need stop word removal?
No. BERT, GPT, and other transformer models use subword tokenization and were pre-trained on complete natural language text. Applying stop word removal before feeding text into these models changes the input from what they were trained on and typically hurts performance. Skip tokenization and stop word removal entirely when using transformers — let their built-in tokenizers handle it.

Related reading on Nomidl: Tokenization in NLP — a deep dive into all tokenization types including subword and transformer tokenizers. See Text Preprocessing in NLP for the full preprocessing pipeline these steps fit into, and Stemming vs Lemmatization — the normalization step that typically comes after stop word removal.

External reference: NLTK Stop Words documentation — official reference for working with corpora and stop word lists in NLTK.

Popular Posts

• Loop Engineering Explained: From Prompt Engineering to Self-Prompting AI Agents
• Build Your First MCP Server with FastMCP: A Complete Python Tutorial
• MCP vs Function Calling: Key Differences Explained (2026)
• What Is Model Context Protocol (MCP) – A Complete Guide for AI Developers
• MCP Primitives Explained: Tools, Resources, and Prompts With Real Examples