Skip to content
Home » csv

csv

COMBINE MULTIPLE CSV FILES FROM THE COMMAND LINE AND RETAIN FILENAMES

For this guide we will take thousands of csv files and combine them into One single file, which will be uploaded to a database. This file will have a new column inserted, where the originating filename will be added next to the row. In scraping sites, often my output files are separated and after I clean them up I need to combine them. This is how I do it.

On my Windows 10 computer
I will copy my CSV files over to a secondary drive, leaving the originals intact.
Then I will open a command prompt as an administrator and navigate to that drive. If the files are on my H: drive will navigate by typing
H:
Then press Enter

Now I will navigate to the “dataset” folder
cd dataset

I will then list the directory contents to ensure this holds my csv files
dir

Once I see all of my csv files I will enter the following:

for /f %a in ('dir /b *.csv') do for /f "tokens=*" %b in (%a) do echo %b,%a >> combined-date.csv

Then press Enter

This will create a file called combined-04132021.csv (which is today’s date)

And I can then upload that file to my database easily

COMBINE MULTIPLE CSV FILES FROM THE COMMAND LINE

For this guide we will take thousands of csv files and combine them into One single file, which will be uploaded to a database. In scraping sites, often my output files are separated and after i clean them up I need to combine them. This is how I do it.

On my Windows 10 computer
I will copy my CSV files over to a secondary drive, leaving the originals intact.
Then I will open a command prompt as an administrator and navigate to that drive. If the files are on my H: drive will navigate by typing
H:
Then press Enter

Now I will navigate to the “dataset” folder
cd dataset

I will then list the directory contents to ensure this holds my csv files
dir

Once I see all of my csv files I will enter the following
copy *.csv combined-date.csv
Then press Enter

This will create a file called combined-04132021.csv (which is today’s date)

And I can then upload that file to my database easily