How to Append to a CSV File with Pandas

Why the default is to overwrite, not append

to_csv() opens files in mode="w". Truncating is the default because the common case — regenerating a report, re-running an export — wants a clean file every time. There is no prompt, no backup, and no merge: the previous contents are gone the moment the write starts. If two runs of your script should build on each other, the append has to be asked for explicitly.

The reverse case — deliberately replacing the file — is simply the default at work:df.to_csv("output.csv", index=False) overwrites. Both behaviours live side by side in this one argument, which is why the mode is the first decision to make.

The first append: write or check for the header

The tricky moment is the transition from "file does not exist yet" to "file exists". A fixed header=False loses column names entirely when the file is new; a fixedheader=True duplicates the header row on every subsequent run.

To make the append idempotent, determine whether to write the header by checking file existence and file size dynamically before calling to_csv():

from pathlib import Path
import pandas as pd

output_path = Path("telemetry.csv")

# True if the file does not exist OR exists but is completely empty (0 bytes)
write_header = not output_path.exists() or output_path.stat().st_size == 0

df.to_csv(output_path, mode="a", header=write_header, index=False)

Checking st_size == 0 catches a common real-world edge case: files created empty by a prior step (like touch, an initial shell redirection, or an interrupted process). Without the size check, output_path.exists() would evaluate to True, producing a file that starts with data rows and has no column names at all.

Key limitations of the idempotent check

• Concurrency and race conditions: If multiple processes, threads, or Celery workers append to the same file simultaneously, two workers can test not output_path.exists() at the same millisecond. Both will write headers, or worse, interleave partial rows. For concurrent writes, use OS-level file locking (such as portalocker or fcntl) or have each worker write a separate partition file.
• No schema validation: The header check only decides whether column titles are printed. It does not ensure that the new DataFrame matches the column names, count, or ordering already on disk.
• Encoding and delimiter consistency: Every append must match the original file's delimiter (sep), line terminators, and encoding. Appending UTF-8 data to an ISO-8859-1 file will corrupt the resulting document for future readers.

Appending safely: match the schema first

pandas writes rows, not relationships. It will happily append a frame with different columns, a different column order, or extra columns — producing a file that no standard reader can trust. Align the new data to the header already on disk before writing:

existing = pd.read_csv("output.csv", nrows=0).columns   # header only, no data read
df = df.reindex(columns=existing)                       # same columns, same order
df.to_csv("output.csv", mode="a", header=False, index=False)

The same rule applies to the separator and encoding: the append must use whatever the first write used, or the file ends up with mixed delimiters.

Try append mode in the converter

The tool opens with Append rows selected — the generated code readsmode="a" and the header checkbox is off, which is the state you want after the first write. Switch to Overwrite to see the default (mode="w") return, and open More options to bring the header back for the file's very first write. The preview here is a single session, so download the CSV and append to it again to see the behaviour against a real file.

Common mistakes

• Appending with the default mode. Forgetting mode="a" does not fail loudly — it overwrites the file with only the new rows.
• Leaving header=True on the second write. The column names are stamped again where the appended rows begin, and every later reader trips over them.
• Adding the index on append. index=True writes a fresh 0, 1, 2… sequence for the new rows, so the file gets a meaningless leading column that does not match the rows above it.
• Assuming pandas aligns columns for you. It does not — a frame with reordered or extra columns corrupts the file's structure silently. Reindex to the existing header first.
• Changing the separator between writes. The append is raw text; a different sep mid-file produces mixed delimiters.
• Appending from two processes at once. There is no locking — concurrent writers can interleave or corrupt rows. Write to separate files and concatenate later.

Frequently asked questions

Related guides

• How to save a Pandas DataFrame to a CSV file — the base df.to_csv() call this page builds on: paths, folders, encoding, verification.
• Pandas to CSV without index — why the index column exists and the four ways to keep it out of the file.
• Common Pandas to CSV examples — ten worked calls, including a minimal append, on the home page.