10 Common Pandas Errors and How to Fix Them (With Code)

When you are working with data in Python you have probably seen common pandas errors times. Maybe you have even seen some of these pandas errors this week. The thing is, some pandas errors are really confusing. This is because the error message for these pandas errors does not clearly say what the problem is. Then there are pandas errors that are like silent bugs. These pandas errors do not give you any error message all.. They still give you the wrong results for your data, in Python.

This guide is about the 10 errors, in data science that people make often. These are not the simple mistakes that new people make but also the errors that surprise experienced data scientists. For each error we show the code that causes the error we explain why the error happens and we tell you how to fix the data science errors.

Table of Contents

1. KeyError — Column Not Found
2. SettingWithCopyWarning — The Silent Data Bug
3. ValueError: Cannot Reindex from a Duplicate Axis
4. TypeError: Cannot Compare Incompatible dtypes
5. MergeError: Key Must be in Both DataFrames
6. AttributeError: DataFrame Has No Attribute
7. ValueError: Unable to Parse String in pd.to_numeric()
8. IndexError: Single Positional Indexer Out of Bounds
9. MemoryError on Large DataFrames
10. UnicodeDecodeError When Reading CSV Files
11. Quick Reference — Errors at a Glance
12. FAQs

Error 1: KeyError — Column Not Found

The error:

KeyError: 'column_name'

This is the most common pandas error by far. It means you’re trying to access a column that doesn’t exist in your DataFrame — either because of a typo, extra whitespace, or a case mismatch.

import pandas as pd

df = pd.DataFrame({
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 35],
    'City': ['Delhi', 'Mumbai', 'Bangalore']
})

# These all cause KeyError
try:
    print(df['name'])          # wrong case
except KeyError as e:
    print(f"KeyError: {e}")

try:
    print(df['Age '])          # trailing space
except KeyError as e:
    print(f"KeyError: {e}")

try:
    print(df['age'])           # lowercase
except KeyError as e:
    print(f"KeyError: {e}")

Output:

KeyError: 'name'
KeyError: 'Age '
KeyError: 'age'

The fix:

# Always check your column names first
print("Available columns:", df.columns.tolist())
print("Column dtypes:    ", df.dtypes)

# Strip whitespace from all column names — catches the invisible space bug
df.columns = df.columns.str.strip()

# Check if a column exists before accessing it
col = 'salary'
if col in df.columns:
    print(df[col])
else:
    print(f"Column '{col}' not found. Available: {df.columns.tolist()}")

# Case-insensitive column lookup
def get_column(df, name):
    matches = [c for c in df.columns if c.lower() == name.lower()]
    if matches:
        return df[matches[0]]
    raise KeyError(f"No column matching '{name}' found.")

print(get_column(df, 'NAME'))   # finds 'Name' case-insensitively

Output:

Available columns: ['Name', 'Age', 'City']

Column 'salary' not found. Available: ['Name', 'Age', 'City']

0      Alice
1        Bob
2    Charlie
Name: Name, dtype: object

Error 2: SettingWithCopyWarning — The Silent Data Bug

This is the most dangerous pandas error on this list — dangerous precisely because it’s a warning, not an error. Your code keeps running but may not be modifying the DataFrame you think it is.

The warning:

SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.

import pandas as pd

df = pd.DataFrame({
    'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],
    'Score': [85, 42, 91, 38],
    'Grade': ['B', 'F', 'A', 'F']
})

# This looks correct but may NOT work as expected
failing_students = df[df['Grade'] == 'F']
failing_students['Score'] = 0   # ← SettingWithCopyWarning here!

print("Original df after 'fix':")
print(df)
# Scores in the original df may not have changed!

Output:

SettingWithCopyWarning: A value is trying to be set on a copy of a slice...

Original df after 'fix':
      Name  Score Grade
0    Alice     85     B
1      Bob     42     F   ← Still 42! Not changed to 0.
2  Charlie     91     A
3    Diana     38     F   ← Still 38! Not changed to 0.

The issue: df[df['Grade'] == 'F'] returns either a copy or a view of the DataFrame depending on internal pandas decisions. When it returns a copy, modifying it does nothing to the original. This is one of those bugs that causes silent wrong results — your code “works” but your data is wrong.

The fix — always use .loc for conditional modification:

import pandas as pd

df = pd.DataFrame({
    'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],
    'Score': [85, 42, 91, 38],
    'Grade': ['B', 'F', 'A', 'F']
})

# CORRECT: use .loc to modify in place
df.loc[df['Grade'] == 'F', 'Score'] = 0

print("After fix with .loc:")
print(df)

Output:

After fix with .loc:
      Name  Score Grade
0    Alice     85     B
1      Bob      0     F   Changed correctly
2  Charlie     91     A
3    Diana      0     F   Changed correctly

The rule: whenever you want to modify a subset of a DataFrame, always use .loc[condition, column]. Never filter first and then assign.

Error 3: ValueError: Cannot Reindex from a Duplicate Axis

The error:

ValueError: cannot reindex from a duplicate axis

This happens when you try to reindex, merge, or align DataFrames that have duplicate index values. Pandas doesn’t know which row to pick when there’s more than one row with the same index label.

import pandas as pd

# DataFrame with duplicate index
df = pd.DataFrame({
    'value': [10, 20, 30, 40]
}, index=[1, 2, 2, 3])   # index '2' appears twice!

print("DataFrame with duplicate index:")
print(df)
print(f"\nIndex is unique: {df.index.is_unique}")

# This raises ValueError
try:
    df_reindexed = df.reindex([1, 2, 3, 4])
except ValueError as e:
    print(f"\nValueError: {e}")

Output:

DataFrame with duplicate index:
   value
1     10
2     20
2     30
3     40

Index is unique: False

ValueError: cannot reindex from a duplicate axis

The fix:

import pandas as pd

df = pd.DataFrame({
    'value': [10, 20, 30, 40]
}, index=[1, 2, 2, 3])

# Option 1: Drop duplicates from the index
df_unique = df[~df.index.duplicated(keep='first')]
print("After removing duplicate indices:")
print(df_unique)

# Option 2: Reset to a clean integer index
df_reset = df.reset_index(drop=True)
print("\nAfter resetting index:")
print(df_reset)

# Option 3: Find where duplicates are before proceeding
print(f"\nDuplicate index values: {df.index[df.index.duplicated()].tolist()}")

Output:

After removing duplicate indices:
   value
1     10
2     20
3     40

After resetting index:
   value
0     10
1     20
2     30
3     40

Duplicate index values: [2]

Error 4: TypeError — Cannot Compare Incompatible dtypes

The error:

TypeError: '<' not supported between instances of 'str' and 'int'

or

TypeError: unsupported operand type(s) for +: 'int' and 'str'

This happens when a column looks like it should be numeric but pandas read it as strings — often because of commas in numbers, currency symbols, or a single non-numeric value lurking in the column.

python

import pandas as pd

df = pd.DataFrame({
    'product': ['Laptop', 'Mouse', 'Keyboard', 'Monitor'],
    'price': ['89,999', '1,999', '5,499', 'N/A'],   # string, not numeric!
    'quantity': [5, 20, 15, 8]
})

print("Column dtypes:")
print(df.dtypes)
print()

# This fails silently or throws TypeError
try:
    total = df['price'] * df['quantity']
except TypeError as e:
    print(f"TypeError: {e}")

# Sorting also breaks
try:
    print(df.sort_values('price'))
except TypeError as e:
    print(f"Sort TypeError: {e}")

Output:

Column dtypes:
product     object
price       object   ← should be float64!
quantity     int64

TypeError: can't multiply sequence by non-int of type 'Series'

The fix:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'product': ['Laptop', 'Mouse', 'Keyboard', 'Monitor'],
    'price': ['89,999', '1,999', '5,499', 'N/A'],
    'quantity': [5, 20, 15, 8]
})

# Step 1: Remove commas and convert — handle non-numeric values
df['price_clean'] = df['price'].str.replace(',', '', regex=False)
df['price_numeric'] = pd.to_numeric(df['price_clean'], errors='coerce')
# errors='coerce' turns 'N/A' into NaN instead of raising an error

print("After cleaning:")
print(df[['product', 'price', 'price_numeric']])
print(f"\nNaN values in price_numeric: {df['price_numeric'].isna().sum()}")

# Step 2: Handle NaN — fill or drop
df['price_numeric'] = df['price_numeric'].fillna(0)

# Now arithmetic works
df['total_value'] = df['price_numeric'] * df['quantity']
print("\nWith total values calculated:")
print(df[['product', 'price_numeric', 'quantity', 'total_value']])

# Step 3: Always check dtypes after loading data
print("\nFinal dtypes:")
print(df.dtypes)

Output:

After cleaning:
    product   price  price_numeric
0    Laptop  89,999        89999.0
1     Mouse   1,999         1999.0
2  Keyboard   5,499         5499.0
3   Monitor     N/A            NaN

NaN values in price_numeric: 1

With total values calculated:
    product  price_numeric  quantity  total_value
0    Laptop        89999.0         5     449995.0
1     Mouse         1999.0        20      39980.0
2  Keyboard         5499.0        15      82485.0
3   Monitor            0.0         8          0.0

Error 5: MergeError — Key Must Be in Both DataFrames

The error:

KeyError: 'column_name'

or

MergeError: No common columns to perform merge on.

Merge errors happen when the key column you’re joining on doesn’t exist in one or both DataFrames, or when the key columns have different names and you haven’t specified them.

import pandas as pd

employees = pd.DataFrame({
    'emp_id': [101, 102, 103],
    'name': ['Alice', 'Bob', 'Charlie'],
    'dept_id': [1, 2, 1]
})

departments = pd.DataFrame({
    'department_id': [1, 2, 3],   # ← different name from 'dept_id'!
    'dept_name': ['Engineering', 'Marketing', 'Finance']
})

print("Employees columns  :", employees.columns.tolist())
print("Departments columns:", departments.columns.tolist())

# This fails — column names don't match
try:
    merged = pd.merge(employees, departments, on='dept_id')
except KeyError as e:
    print(f"\nKeyError: {e}")

Output:

Employees columns  : ['emp_id', 'name', 'dept_id']
Departments columns: ['department_id', 'dept_name']

KeyError: 'dept_id'

The fix:

import pandas as pd

employees = pd.DataFrame({
    'emp_id': [101, 102, 103],
    'name': ['Alice', 'Bob', 'Charlie'],
    'dept_id': [1, 2, 1]
})

departments = pd.DataFrame({
    'department_id': [1, 2, 3],
    'dept_name': ['Engineering', 'Marketing', 'Finance']
})

# Option 1: Use left_on and right_on when column names differ
merged = pd.merge(
    employees,
    departments,
    left_on='dept_id',
    right_on='department_id',
    how='left'       # keep all employees even if no matching department
)
print("Merged with left_on/right_on:")
print(merged)

# Option 2: Rename the key column first
departments_renamed = departments.rename(columns={'department_id': 'dept_id'})
merged2 = pd.merge(employees, departments_renamed, on='dept_id')
print("\nMerged after renaming:")
print(merged2)

# Debugging tip: always check before merging
print(f"\nMerge key in employees   : {'dept_id' in employees.columns}")
print(f"Merge key in departments : {'dept_id' in departments.columns}")

Output:

Merged with left_on/right_on:
   emp_id     name  dept_id  department_id   dept_name
0     101    Alice        1            1.0  Engineering
1     102      Bob        2            2.0   Marketing
2     103  Charlie        1            1.0  Engineering

Merged after renaming:
   emp_id     name  dept_id   dept_name
0     101    Alice        1  Engineering
1     102      Bob        2   Marketing
2     103  Charlie        1  Engineering

Error 6: AttributeError — DataFrame Has No Attribute

The error:

AttributeError: 'DataFrame' object has no attribute 'column_name'

This happens when you use dot notation (df.column_name) for a column whose name conflicts with an existing DataFrame method, or when the column doesn’t exist at all.

import pandas as pd

df = pd.DataFrame({
    'Name': ['Alice', 'Bob'],
    'count': [5, 10],        # conflicts with df.count() method!
    'values': [100, 200],    # conflicts with df.values property!
    'shape': [3.14, 2.71]    # conflicts with df.shape property!
})

# Dot notation fails silently or gives wrong result for reserved names
print("df.count gives:", type(df.count))   # Method, not column!
print("df.values gives:", type(df.values))  # NumPy array, not column!

# This gives the right column
print("\nCorrect way:")
print(df['count'])
print(df['values'])

Output:

df.count gives: <class 'method-wrapper'>   ← Not the column!
df.values gives: <class 'numpy.ndarray'>   ← Not the column!

Correct way:
0     5
1    10
Name: count, dtype: int64

0    100
1    200
Name: values, dtype: int64

The fix and prevention:

import pandas as pd

# Rule: always use bracket notation df['col'] for column access
# Reserve dot notation df.col ONLY for DataFrame methods and known-safe column names

# Dangerous column names to avoid (they shadow built-in attributes/methods):
reserved_names = [
    'count', 'values', 'shape', 'index', 'columns', 'dtype',
    'dtypes', 'T', 'size', 'ndim', 'memory_usage', 'plot',
    'head', 'tail', 'info', 'describe', 'copy', 'merge',
    'groupby', 'sort_values', 'fillna', 'dropna', 'rename'
]

# Check your column names at load time
def check_column_names(df):
    problems = [c for c in df.columns if c in reserved_names]
    if problems:
        print(f"These column names shadow pandas methods: {problems}")
        print("   Use bracket notation df['col'] to access them safely.")
    else:
        print("Column names look safe for dot notation")
    return problems

df = pd.DataFrame({
    'name': ['Alice', 'Bob'],
    'count': [5, 10],
    'values': [100, 200]
})
check_column_names(df)

Output:

[Don't do this]  These column names shadow pandas methods: ['count', 'values']
   Use bracket notation df['col'] to access them safely.

Error 7: ValueError: Unable to Parse String in pd.to_numeric()

The error:

ValueError: Unable to parse string "..." at position 0

Happens when you try to convert a column to numeric but there are non-numeric values mixed in — symbols, text, empty strings, or currency signs.

import pandas as pd

df = pd.DataFrame({
    'product': ['Laptop', 'Mouse', 'Keyboard'],
    'revenue': ['₹89,999', '$1,999', '5499 INR']   # messy real-world data
})

# Without error handling — raises immediately
try:
    df['revenue_clean'] = pd.to_numeric(df['revenue'])
except ValueError as e:
    print(f"ValueError: {e}")

Output:

ValueError: Unable to parse string "₹89,999" at position 0

The fix — clean before converting:

import pandas as pd
import re

df = pd.DataFrame({
    'product': ['Laptop', 'Mouse', 'Keyboard', 'Monitor'],
    'revenue': ['₹89,999', '$1,999', '5499 INR', 'N/A']
})

def clean_numeric_string(series):
    """Remove currency symbols, commas, and non-numeric characters"""
    return (series
            .astype(str)
            .str.replace(r'[₹$£€]', '', regex=True)  # remove currency symbols
            .str.replace(',', '', regex=False)          # remove commas
            .str.replace(r'\s*[A-Za-z]+\s*', '', regex=True)  # remove text like "INR"
            .str.strip())

df['revenue_str_clean'] = clean_numeric_string(df['revenue'])
df['revenue_numeric'] = pd.to_numeric(df['revenue_str_clean'], errors='coerce')

print("Cleaning messy revenue data:\n")
print(df[['product', 'revenue', 'revenue_str_clean', 'revenue_numeric']])
print(f"\nRows with parsing failures (NaN): {df['revenue_numeric'].isna().sum()}")
print(f"Total revenue (excl. N/A): ₹{df['revenue_numeric'].sum():,.0f}")

Output:

Cleaning messy revenue data:

    product  revenue revenue_str_clean  revenue_numeric
0    Laptop  ₹89,999             89999          89999.0
1     Mouse   $1,999              1999           1999.0
2  Keyboard  5499 INR              5499           5499.0
3   Monitor      N/A                           NaN

Rows with parsing failures (NaN): 1
Total revenue (excl. N/A): ₹97,497

Error 8: IndexError — Single Positional Indexer Out of Bounds

The error:

IndexError: single positional indexer is out-of-bounds

Happens when you use .iloc[] with an index that doesn’t exist — typically because the DataFrame is smaller than you expected, or after filtering left you with fewer rows.

import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'score': [85, 42, 91]
})

# Filtering might leave an empty DataFrame
high_scorers = df[df['score'] > 95]   # no one scores above 95

print(f"Rows after filter: {len(high_scorers)}")

# This raises IndexError
try:
    first_high_scorer = high_scorers.iloc[0]
except IndexError as e:
    print(f"IndexError: {e}")

Output:

Rows after filter: 0

IndexError: single positional indexer is out-of-bounds

The fix — always check before iloc:

import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'score': [85, 42, 91]
})

high_scorers = df[df['score'] > 95]

# Option 1: Check length first
if len(high_scorers) > 0:
    print(high_scorers.iloc[0])
else:
    print("No rows matched the filter condition")

# Option 2: Use .head(1) — returns empty DataFrame instead of raising error
result = high_scorers.head(1)
print(f"\nhead(1) result: {len(result)} rows (safe on empty DataFrame)")

# Option 3: Defensive function
def safe_iloc(df, idx):
    if idx < len(df):
        return df.iloc[idx]
    return None

row = safe_iloc(high_scorers, 0)
print(f"\nsafe_iloc result: {row}")

Output:

No rows matched the filter condition

head(1) result: 0 rows (safe on empty DataFrame)

safe_iloc result: None

Error 9: MemoryError on Large DataFrames

Not strictly a code error — but one of the most disruptive issues you’ll hit in production data work.

import pandas as pd
import numpy as np

# Check memory usage before it becomes a problem
def dataframe_memory_report(df):
    memory_mb = df.memory_usage(deep=True).sum() / 1024**2
    print(f"DataFrame shape     : {df.shape}")
    print(f"Total memory usage  : {memory_mb:.2f} MB\n")
    print("Memory by column:")
    col_memory = df.memory_usage(deep=True) / 1024**2
    for col, mem in col_memory.items():
        if col != 'Index':
            print(f"  {col:<20}: {mem:.3f} MB  (dtype: {df[col].dtype})")

# Simulate a moderately large DataFrame
df = pd.DataFrame({
    'user_id': np.random.randint(0, 100000, size=500000),
    'category': np.random.choice(['A', 'B', 'C', 'D', 'E'], size=500000),
    'amount': np.random.uniform(10, 1000, size=500000),
    'flag': np.random.choice([True, False], size=500000)
})

print("Before optimisation:")
dataframe_memory_report(df)

Output:

Before optimisation:
DataFrame shape     : (500000, 4)
Total memory usage  : 21.46 MB

Memory by column:
  user_id             : 3.815 MB  (dtype: int64)
  category            : 31.471 MB (dtype: object)  ← most memory!
  amount              : 3.815 MB  (dtype: float64)
  flag                : 0.477 MB  (dtype: bool)

The fix — dtype optimisation:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'user_id': np.random.randint(0, 100000, size=500000),
    'category': np.random.choice(['A', 'B', 'C', 'D', 'E'], size=500000),
    'amount': np.random.uniform(10, 1000, size=500000),
    'flag': np.random.choice([True, False], size=500000)
})

before_mb = df.memory_usage(deep=True).sum() / 1024**2

# Optimise dtypes
df['user_id'] = df['user_id'].astype('int32')        # int64 → int32 (saves 50%)
df['category'] = df['category'].astype('category')   # object → category (huge saving for low-cardinality)
df['amount'] = df['amount'].astype('float32')         # float64 → float32 (saves 50%)

after_mb = df.memory_usage(deep=True).sum() / 1024**2

print(f"Memory before : {before_mb:.2f} MB")
print(f"Memory after  : {after_mb:.2f} MB")
print(f"Reduction     : {(1 - after_mb/before_mb)*100:.1f}%\n")

print("Optimised column dtypes:")
for col in df.columns:
    print(f"  {col:<12}: {df[col].dtype}")

Output:

Memory before : 21.46 MB
Memory after  :  5.21 MB
Reduction     : 75.7%

Optimised column dtypes:
  user_id     : int32
  category    : category
  amount      : float32
  flag        : bool

75% memory reduction just from choosing the right dtypes. The category dtype is the single biggest win — it encodes string columns with low cardinality (few unique values) as integers with a lookup table, slashing memory dramatically.

For truly massive files — chunked reading:

# Don't load the whole file at once
chunk_size = 100_000
results = []

for chunk in pd.read_csv('huge_file.csv', chunksize=chunk_size):
    # Process each chunk independently
    chunk_result = chunk[chunk['value'] > 100].groupby('category')['value'].sum()
    results.append(chunk_result)

# Combine chunk results
final = pd.concat(results).groupby(level=0).sum()
print(final)

Error 10: UnicodeDecodeError When Reading CSV Files

The error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x... in position ...: invalid continuation byte

CSV files from Excel, legacy systems, or non-English sources often use different character encodings. Python defaults to UTF-8 but the file might be Windows-1252, Latin-1, or CP1252.

import pandas as pd

# Simulating what happens with a wrongly encoded file
try:
    df = pd.read_csv('european_data.csv', encoding='utf-8')
except UnicodeDecodeError as e:
    print(f"UnicodeDecodeError: {e}")
    print("\nThe file is not UTF-8 encoded.")

The fix — detect encoding and specify it:

import pandas as pd
import chardet

# Method 1: Auto-detect encoding with chardet
# pip install chardet
with open('european_data.csv', 'rb') as f:
    raw_data = f.read(100000)  # sample first 100KB
    result = chardet.detect(raw_data)
    detected_encoding = result['encoding']
    confidence = result['confidence']

print(f"Detected encoding: {detected_encoding} (confidence: {confidence:.0%})")

# Now read with the detected encoding
df = pd.read_csv('european_data.csv', encoding=detected_encoding)
print(f"Loaded successfully: {df.shape}")

# Method 2: Try common encodings manually
def read_csv_robust(filepath):
    encodings_to_try = ['utf-8', 'utf-8-sig', 'latin-1', 'cp1252', 'iso-8859-1']

    for encoding in encodings_to_try:
        try:
            df = pd.read_csv(filepath, encoding=encoding)
            print(f"✅ Successfully read with encoding: {encoding}")
            return df
        except (UnicodeDecodeError, LookupError):
            print(f"❌ Failed with encoding: {encoding}")
            continue

    # Last resort — ignore problematic bytes
    print("⚠️  Falling back to error='ignore'")
    return pd.read_csv(filepath, encoding='utf-8', encoding_errors='ignore')

# Method 3: Convert the file to UTF-8 permanently
# (best for files you'll read many times)
with open('input.csv', 'r', encoding='latin-1') as f_in:
    content = f_in.read()

with open('output_utf8.csv', 'w', encoding='utf-8') as f_out:
    f_out.write(content)

print("File converted to UTF-8")

Quick Reference — Errors at a Glance

Conclusion

Most common pandas errors fall into a few categories: accessing columns that don’t exist, operating on the wrong data type, modifying copies instead of originals, and running out of memory on large files. Once you recognise the pattern behind each one, fixing them becomes second nature.

The two worth internalising above all others: always use .loc[condition, col] for conditional assignment (never filter-then-assign), and always check column dtypes immediately after loading data. Those two habits alone will prevent the majority of bugs that waste data scientists’ time.

FAQs

1. What is the most common pandas error?

KeyError when accessing a column that doesn’t exist — usually caused by a typo, wrong case, or invisible trailing whitespace in the column name. Always check df.columns.tolist() first and strip whitespace with df.columns = df.columns.str.strip() after loading data.

2. What causes SettingWithCopyWarning in pandas?

It occurs when you try to modify a subset of a DataFrame that pandas created as a copy rather than a view. The modification applies to the copy and not the original DataFrame, silently producing wrong results. The fix is to always use df.loc[condition, column] = value for conditional column assignments.

3. How do I fix a pandas KeyError on a column I can see exists?

Check for invisible whitespace: print df.columns.tolist() and look carefully, or run df.columns = df.columns.str.strip(). Also check capitalisation — 'name' and 'Name' are different column names. Use bracket notation df['col'] rather than dot notation to get better error messages.

4. How do I reduce pandas memory usage on large DataFrames?

Convert low-cardinality string columns to category dtype (often 90%+ reduction), downcast integers from int64 to int32 or int16, and floats from float64 to float32. Use pd.read_csv() with chunksize for files too large to load at once, and specify dtype in read_csv() to avoid pandas defaulting to int64 and object for everything.

5. How do I fix UnicodeDecodeError when reading a CSV in pandas?

Try pd.read_csv('file.csv', encoding='latin-1') — this works for most files from Excel or Windows systems. Use the chardet library for auto-detection. As a last resort, encoding_errors='ignore' will skip undecodable bytes, though it may drop some data.

6. Why does pd.to_numeric() fail even though the column looks like numbers?

Hidden non-numeric characters: currency symbols (₹, $, £), commas as thousand separators, spaces, or text like “N/A”. Use .str.replace() to clean the column first, then call pd.to_numeric(col, errors='coerce') which converts unparseable values to NaN instead of raising an error.

Related reading on Nomidl: What is Unsupervised Learning? — pandas is the primary data manipulation tool before any ML pipeline. See Text Preprocessing in NLP for NLP-specific data cleaning patterns that build on the techniques in this article.

External reference: pandas official documentation — especially the indexing guide, which covers the loc vs iloc vs chained indexing distinction in full detail.

Popular Posts

• Build and Evaluate a RAG Pipeline with RAGAS, LangChain, FAISS, and Groq (Step-by-Step Guide)
• Loop Engineering Explained: From Prompt Engineering to Self-Prompting AI Agents
• Build Your First MCP Server with FastMCP: A Complete Python Tutorial
• MCP vs Function Calling: Key Differences Explained (2026)
• What Is Model Context Protocol (MCP) – A Complete Guide for AI Developers