pandas read_csv() chunksize Parameter: Documentation and Examples

When working with datasets that exceed available RAM, calling pd.read_csv("large_file.csv")often terminates with a fatal MemoryError. The chunksize parameter in Pandas solves this by streaming the file in manageable row batches instead of attempting to parse the entire file into memory at once.

This documentation guide covers the chunksize parameter syntax, the TextFileReaderreturn object, memory-safe iteration and aggregation patterns, type consistency, and performance tuning. For the official parameter list and signature details, see the official pandas.read_csv documentation.

The quick answer

To read a large CSV file in chunks, pass an integer to chunksize and iterate over the resulting reader object:

import pandas as pd

# Open the file and iterate over chunks of up to 100,000 rows
with pd.read_csv("large_dataset.csv", chunksize=100_000) as reader:
    for i, chunk in enumerate(reader):
        print(f"Chunk {i}: {len(chunk):,} rows, {len(chunk.columns)} columns")

Inside the loop, chunk is a standard pd.DataFrame containing at most 100,000 rows. Peak memory during parsing is governed primarily by the active chunk rather than the entire file on disk. Total memory consumption in your script depends on row width, data types (especially strings and object columns), the transformations performed inside the loop, and whether any intermediate results are retained.

Parameter syntax and return behavior

In the official pandas API, the parameter signature is declared as:

pandas.read_csv(filepath_or_buffer, *, chunksize=None, iterator=False, ...)

• Type: int, optional. Accepts positive integers ($\ge 1$, such as10_000 or 50_000). Pandas strictly validates this parameter:
  • Non-positive integers: Passing zero or negative integers (such as 0 or-1) raises ValueError: 'chunksize' must be an integer >=1 (verified against pandas 3.0.6).
  • Non-integer types: Passing non-integer values—such as floating-point numbers (10.5), strings ("100"), or booleans (False or True)—also raisesValueError: 'chunksize' must be an integer >=1, rather than a TypeError.
• Default: None. When unset (and iterator=False), pandas eagerly loads the entire file into RAM and returns a single DataFrame.
• Return Type: When chunksize is provided, the function returns apandas.io.parsers.readers.TextFileReader object instead of a DataFrame.

The TextFileReader implements Python's iterator protocol (__iter__ and__next__), so it can be used directly in for loops, list comprehensions, or with the built-in next() function.

Why chunksize returns a TextFileReader, not a DataFrame

A Pandas DataFrame represents concrete 2D tabular data residing entirely in memory. If pd.read_csv("large.csv", chunksize=100_000) returned a DataFrame, it would either have to load the full dataset (defeating the purpose) or discard all rows after line 100,000 (which is the purpose of the nrows parameter).

By returning a TextFileReader, Pandas defers parsing until your code requests each slice of data. Rather than loading everything up front, the reader yields successive DataFramechunks on demand, supporting standard iteration in for loops as well as explicit batch retrieval with reader.get_chunk(). When your code finishes with a chunk and holds no remaining references or views, that DataFrame becomes eligible for garbage collection. Note that actual process memory release depends on Python's reference counts and whether the operating system memory allocator retains freed heap pages for subsequent allocations.

Minimal runnable iteration example

Here is a complete, minimal example demonstrating how to iterate over chunks and track global parsed-row offsets explicitly:

import pandas as pd

# Read in batches of 25,000 rows
total_rows_parsed = 0

with pd.read_csv("customers.csv", chunksize=25_000) as reader:
    for chunk_index, chunk in enumerate(reader):
        start_offset = total_rows_parsed
        end_offset = total_rows_parsed + len(chunk) - 1
        print(
            f"Chunk {chunk_index}: parsed rows {start_offset:,}–{end_offset:,} "
            f"(chunk shape: {chunk.shape})"
        )
        total_rows_parsed += len(chunk)

print(f"Finished processing all chunks. Total rows parsed: {total_rows_parsed:,}")

Notice that total_rows_parsed tracks the logical row count processed by Pandas, which differs from raw file line numbers whenever headers, blank lines, comments, or skipped rows are present. The final chunk may contain fewer rows than the specified chunksize if the total number of data rows is not an exact multiple of the batch size. Pandas handles this end-of-file condition automatically without raising an error.

Memory-safe chunk processing and aggregation

A common anti-pattern among developers new to chunksize is storing each chunk in a list and calling pd.concat() at the end:

# ANTI-PATTERN: Defeats the purpose of chunksize!
chunks = []
for chunk in pd.read_csv("huge_file.csv", chunksize=50_000):
    chunks.append(chunk)   # Memory still accumulates!

df = pd.concat(chunks)     # Requires ~2x peak memory during concatenation!

To preserve bounded memory, you must process and condense data inside the loop. Here are the three recommended architectural patterns:

Pattern 1: Running arithmetic and summary statistics

When calculating totals, counts, or averages, accumulate scalar metrics across chunks. Usingchunk["amount"].count() ensures that only valid, non-null entries are counted in the denominator, keeping the average calculation null-safe and mathematically aligned withsum():

import pandas as pd

total_sales = 0.0
valid_transaction_count = 0

with pd.read_csv("transactions.csv", chunksize=100_000) as reader:
    for chunk in reader:
        # sum() omits NaNs by default; count() tallies only non-null values
        total_sales += chunk["amount"].sum()
        valid_transaction_count += chunk["amount"].count()

average_order_value = (
    total_sales / valid_transaction_count if valid_transaction_count > 0 else 0.0
)

print(f"Total Sales: ${total_sales:,.2f}")
print(f"Valid Transactions: {valid_transaction_count:,}")
print(f"Average Order Value: ${average_order_value:,.2f}")

Pattern 2: Two-stage GroupBy across chunks

When grouping data by categories, aggregate within each chunk and sum the intermediate series:

import pandas as pd

# Running accumulator Series
category_totals = pd.Series(dtype="float64")

with pd.read_csv("transactions.csv", chunksize=100_000) as reader:
    for chunk in reader:
        # 1. Group within the chunk
        chunk_summary = chunk.groupby("category")["amount"].sum()
        # 2. Add to global accumulator, matching indexes and filling missing with 0
        category_totals = category_totals.add(chunk_summary, fill_value=0)

print(category_totals.sort_values(ascending=False))

Pattern 3: Stream filtering to an output CSV

When extracting a subset of a large file, filter each chunk and append matching rows to disk. Use our guide on appending to CSV files to ensure headers are written only once.

When appending with mode="a", remember that if the destination file already exists, new chunks will be appended onto the previous file's contents. If your pipeline is intended to generate a fresh export from scratch on each run, check for and remove any previous file explicitly (or write to a unique timestamped path) before entering the loop:

from pathlib import Path
import pandas as pd

output_path = Path("high_value_orders.csv")

# If starting a fresh export, remove any prior output explicitly.
# Otherwise, mode="a" will append new rows to existing file data.
if output_path.exists():
    output_path.unlink()

with pd.read_csv("all_orders.csv", chunksize=100_000) as reader:
    for chunk in reader:
        high_value = chunk[chunk["amount"] >= 500.0]
        if not high_value.empty:
            # Write header only when the destination file does not yet exist
            high_value.to_csv(
                output_path,
                mode="a",
                index=False,
                header=not output_path.exists(),
            )

Combining chunksize with usecols and dtype

Using chunksize alone limits row memory, but loading unused columns or relying on automatic type inference introduces avoidable overhead:

import pandas as pd

# Define schema and column filter upfront
columns_to_keep = ["order_id", "customer_id", "status", "total_price"]

dtypes = {
    "order_id": "int64",
    "customer_id": "int32",
    "status": "category",
    "total_price": "float32",
}

total_revenue = 0.0

with pd.read_csv(
    "orders_2026.csv",
    chunksize=50_000,
    usecols=columns_to_keep,
    dtype=dtypes,
) as reader:
    for chunk in reader:
        # Self-contained operation: sum verified numeric amounts
        total_revenue += chunk["total_price"].sum()

print(f"Total revenue: ${total_revenue:,.2f}")

Why usecols is critical for chunked reads

Passing usecols tells Pandas which fields to construct into the resulting DataFrame. Because CSV is a plain text delimited format, the parser still must scan the line and identify delimiters to locate column boundaries. However, usecols avoids constructing Series, arrays, and Python objects for the omitted columns, which substantially reduces the DataFrame memory footprint of each chunk. Actual memory savings depend on the count and data types of the excluded fields.

Why explicit dtype prevents chunk-splitting schema mismatches

When dtype is omitted, Pandas infers column types independently on each chunk based only on the rows in that batch. If a column contains only integers in early chunks (inferringint64) but later chunks encounter strings, floating-point numbers, or null values, those subsequent chunks may be typed as float64 or object. While the C parser sometimes raises a DtypeWarning, a warning is not guaranteed in every scenario, leaving you with mismatched data types across iterations. Supplying an explicit dtypedictionary enforces a uniform schema across all chunks. For more on diagnosing type conflicts, read our guide to fixing mixed dtypes in read_csv.

Choosing a chunk size and measuring memory

Selecting an appropriate chunksize involves balancing Python iteration overhead against memory consumption:

• Too small (e.g. chunksize=100): Python overhead dominates. Creating hundreds of thousands of tiny DataFrame objects creates significant garbage collection pauses and slows down overall execution.
• Too large (e.g. chunksize=2_000_000): Excessively large chunks risk exhausting RAM or triggering swap space, especially if your transformations copy columns or create temporary structures.
• A rough starting heuristic: There is no universal chunk size or memory guarantee. A 50,000-row chunk of five narrow integer columns consumes only a few megabytes, whereas the same number of rows across dozens of free-form text columns can easily exceed hundreds of megabytes. A range between 10,000 and 100,000 rows serves only as a rough starting heuristic; you should always benchmark your specific file schema, row widths, and downstream transformations.

How to estimate chunk memory

You can inspect the memory footprint of a small sample using chunk.memory_usage(deep=True).sum(). Keep in mind that deep=True estimates the memory consumed by the constructed DataFrame object itself—it is not a guarantee of peak process memory. Parser buffer allocations, temporary transformation objects, and variable-length text records later in the file will increase overall RAM usage.

import pandas as pd

# Inspect a 10,000 row probe chunk
probe = pd.read_csv("dataset.csv", nrows=10_000)
sample_bytes = probe.memory_usage(deep=True).sum()
print(f"Estimated DataFrame memory for 10,000 rows: {sample_bytes / 1024**2:.2f} MB")

# Rough estimate for a chosen target chunk memory (e.g. 50 MB - adjust for your system)
target_chunk_memory_bytes = 50 * 1024 * 1024
bytes_per_row = sample_bytes / 10_000
estimated_rows_for_target_chunk_memory = int(target_chunk_memory_bytes / bytes_per_row)
print(f"Estimated rows for 50 MB target: {estimated_rows_for_target_chunk_memory:,} rows")

The difference between chunksize and iterator=True

Both chunksize and iterator=True return a TextFileReader, but they serve slightly different iteration patterns:

With iterator=True, you can pull arbitrary slices dynamically:

import pandas as pd

reader = pd.read_csv("stream.csv", iterator=True)

# Fetch a tiny batch to validate headers and data
sample = reader.get_chunk(100)

# Fetch a larger batch for processing
batch_1 = reader.get_chunk(50_000)

# Close when finished
reader.close()

When using iterator=True without configuring chunksize, callingreader.get_chunk(size) requires passing an explicit integer size to control how many rows are retrieved. If reader.get_chunk() is invoked without arguments whenchunksize is unset, pandas falls back to reading all remaining rows in the file at once. Conversely, when both iterator=True and chunksize=10_000 are configured inread_csv(), reader.get_chunk() defaults to returning 10,000 rows when called without arguments.

Limitations, boundary splits, and common errors

While chunksize prevents loading entire files into memory, processing data in isolated blocks introduces structural constraints:

• Cross-chunk boundary splits and state management: Operations that depend on surrounding rows or global context cannot be performed naively on individual chunks:
  • Group-by aggregations: Associative metrics (such as sums, counts, min, and max) can be accumulated across chunks using two-stage aggregation (as shown in Pattern 2 above). However, non-associative calculations (such as global medians, quantiles, or distinct counts of high-cardinality entities) cannot be computed per-chunk without sorting the dataset or maintaining intermediate state.
  • Rolling windows and lag/lead: Calculations like df.rolling() or df.shift()require continuity across chunk borders. To calculate a 7-day rolling mean across chunks, your loop must retain the trailing rows from the prior chunk, prepend them to the start of the current chunk, execute the rolling window, and discard the prepended overlap before outputting results.
• Chunk index vs. physical file line numbers: By default (when no index_col is specified), pandas generates a contiguous RangeIndex across chunks reflecting the count of successfully parsed data rows (e.g. Chunk 0 has indices 0..9,999; Chunk 1 has indices10,000..19,999). However, this DataFrame index is not equivalent to physical line numbers in the source CSV file:
  • File structure offsets: Header rows, multi-line headers, skipped rows (skiprows), blank lines, and comment lines (e.g. comment="#") consume physical lines in the source file without consuming DataFrame row index numbers.
  • Malformed line filtering: When corrupt lines are dropped with on_bad_lines="skip", subsequent valid rows receive contiguous index values, further disconnecting DataFrame indices from source-file line positions.
  • Index continuation vs. restarts: If you pass an index_col, chunk indices are derived from column values rather than a sequential row counter. Conversely, if your pipeline callschunk.reset_index(drop=True) or filters rows inside the loop, indices will restart from0 or become non-contiguous within that chunk.
  • Global parsed-row tracking: If your application requires tracking global row positions across chunks, track an explicit counter (such as the total_rows_parsed counter demonstrated inSection 4) rather than assuming the DataFrame index corresponds directly to physical line numbers on disk.
• EmptyDataError: If you pass chunksize to an empty 0-byte file, pandas will immediately raise pd.errors.EmptyDataError: No columns to parse from file. Guard your pipeline by checking os.path.getsize(file) > 0 before opening.
• Unclosed file handles: If an exception occurs inside a standard for chunk in reader:loop, the file descriptor may remain open in memory until the interpreter runs garbage collection. Always wrap your iteration in a with pd.read_csv(...) as reader: context block.

Exporting processed DataFrames

Once your chunk processing is complete, you will typically have a condensed summary DataFrame, an aggregated metric table, or a cleaned dataset ready for export.

To convert your final DataFrame back into a clean, portable CSV file:

• Read our comprehensive tutorial on how to save a DataFrame to CSV for details on filepaths, verification, and compression.
• Remember to suppress the default row numbers by passing index=False, as explained in our pandas to CSV without index guide.
• Explore custom quoting, line terminators, and date formatting recipes in ourto_csv() options cookbook.
• If you are exploring read parameters or want a broader overview of file ingestion, review our pillar guide on how to read a CSV file with Pandas.