Merge Multiple CSV Files using Python I/O

It is common for applications to use CSV files for recording data such as performance values or execution logs. And, there could be more than one CSV file which can give you a meaningful outcome only after their consolidation. In this example, we are demonstrating how to merge multiple CSV files using basic Python File I/O without losing any data.

A CSV file, as the name suggests, combines multiple fields separated by commas. The first row contains the name or title of each column, and the remaining rows contain the actual data values.

Also Read: Multiple Ways to Merge Python Dictionaries

Merging Multiple CSV Files in Python using File I/O

Below is the complete code to perform the merging of CSV files. We used a CSV format which records the performance data for web services. While using this example, you may need to modify the HEADER variable (with the name of actual fields) as per the format of your CSV.

It is essential to understand the header fields we’ve used in this example. It includes the following:

• Timestamp – The time of a web service request
• Client IP – The IP address of the client system making the calls
• Web Service – The name of the web service used
• Status – The return code reflecting success or failure
• GOOD – The total no. of successful requests
• BAD – The total no. of failure calls

import os

csv_header = 'Timestamp,Client IP,Web Service,Status,Good,Bad'
csv_out = 'consolidated.csv'

csv_dir = os.getcwd()

dir_tree = os.walk(csv_dir)
for dirpath, dirnames, filenames in dir_tree:
   pass

csv_list = []
for file in filenames:
   if file.endswith('.csv'):
      csv_list.append(file)

csv_merge = open(csv_out, 'w')
csv_merge.write(csv_header)
csv_merge.write('\n')

for file in csv_list:
   csv_in = open(file)
   for line in csv_in:
      if line.startswith(csv_header):
         continue
      csv_merge.write(line)
   csv_in.close()
   csv_merge.close()
print('Verify consolidated CSV file : ' + csv_out)

The above Python example provided a fully runnable code to merge multiple CSV files. First, try to copy/paste and run this code. After that, you may extend or modify it as per your use case.

Also, Read – Pandas to Combine Multiple CSV Files

A Quick Wrap

In this example, we covered “How to Merge Multiple CSV Files in Python.” It doesn’t use any special Python package to combine the CSV files and can save you a lot of time from going through multiple CSV individually.

If you’ve learned something from this example, then care to share it with your colleagues. Also, connect to our social media accounts to receive timely updates.

Best,

TechBeamers