topic Re: Importing multiple CSV files using Python in ArcGIS Pro Questions

Importing multiple CSV files using Python

fred7 — Mon, 16 May 2022 14:30:07 GMT

Hello!
I have 30 csv files with similar structure. I have used to import CSV files manually from Menu -> Map -> Add Data -> XY Point Data, where I choose the CSV file and press "Run."
(X Field and Y Field are recognized/filled automatically from CSV - latitude, longitude). Z field is left empty.

How can I do this in Python?
Here are some of my thoughts:

import arcpy in_data = ["C:\data\"] filter = "*.csv" z_value = "" arcpy.management.XYTableToPoint(in_data, filter,"longitude", "latitude",z_value,arcpy.SpatialReference(4759, 115700))

Throws an error - SyntaxError: EOL while scanning string literal (<string>, line 2)

Re: Importing multiple CSV files using Python

KimberlyGarbade — Mon, 16 May 2022 16:08:41 GMT

If I understand your question correctly something like this should work:

import arcpy import os import glob import pandas as pd #concatenate your files (lines 13-22) #Set your workspace where you are going to create the output feature class arcpy.env.workspace = r"D:\Test\ESRI Community Helper\ESRI Community Helper.gdb" #The path to your .csv files goes here os.chdir(r"D:\Test\CSVTestDirector") extension = 'csv' csvAllFiles = [i for i in glob.glob('*.{}'.format(extension))] #combine all files in the list combined_csv = pd.concat([pd.read_csv(f) for f in csvAllFiles]) #export to csv combined_csv.to_csv( "AllCSVs.csv", index=False, encoding='utf-8-sig') #Create feature class from concatinated csv files #Update your .csv file path, output file name, "X","Y" column names, and SpatialReference here arcpy.management.XYTableToPoint(r"D:\Test\CSVTestDirector\AllCSVs.csv", "outputCSV","X","Y","",arcpy.SpatialReference(4759, 115700))

Re: Importing multiple CSV files using Python

Brian_Wilson — Mon, 16 May 2022 16:24:43 GMT

Of course read @KimberlyGarbade's answer but also you might want to know the error you are getting is because you used backslashes in Line 2.

You could either put a "r" in front like this: r"C:\data\" or use regular slashes "C:/data/"

There is one more way that might confuse you -- "C:\\" the first \ says to give the next character special treatment. Often it's used for "\t" which means "insert a TAB"

Once you fix that and run it again, I bet you get an error about in_data being a list and it wants to be a string in line 6. You have to feed it one filename at a time like: in_data = r"C:\data\foo.csv"

and the second arg "fillter" needs to be the name of the output, it would be great if you could feed it a wildcard but pretty much nothing in arcpy is that convenient. You'd have to do something outside arcpy, @KimberlyGarbade did it a good way using "glob" which can take wildcards and return filenames.

@fred7 are you trying to generate one feature class as output or 30?

Re: Importing multiple CSV files using Python

Brian_Wilson — Mon, 16 May 2022 16:41:41 GMT

Some things you wrote are confusing especially to beginners, like this

csvAllFiles = [i for i in glob.glob('*.{}'.format(extension))]

This line says "use the glob function to look for files matching the pattern *.csv and return a list, then iterate over the list and make another list." so you could just write this

csvAllFiles = glob.glob("*.csv")

and again with line 13 it could just be

combined_csv = pd.concat(csvAllFiles)

You could combine them into one line but personally I llke to add a test, actually lots and lots of tests 🙂

after the "glob" line I would do

if len(csvAllFiles)==0: print("I did not find any CSV files! Quitting.") exit(0)

Re: Importing multiple CSV files using Python

fred7 — Mon, 16 May 2022 21:56:12 GMT

Thank you all for the answers! I am trying to get the script work.

I have csv files for 30 different days. Each day we have measured water surface parametes (salinity, temperature, turbidity, chlorophyll) in different points.
I have to make interpolation maps (Kriging) for each parameter every 30 days.

I was thinking that I use Python to import all 30 files as XY Point Data (and get points on map).

Then manually interpolate those points with Kriging (using one parameter eg salinity) for each 30 day.
After that use the same imported points and interpolate another parameter (eg temperature). And so on.

Re: Importing multiple CSV files using Python

fred7 — Tue, 17 May 2022 07:10:48 GMT

Hi again, I was able to import 30 csv files using Geoprocessing -> XY Table To Point -> right click "Batch."
I choosed all CSV files and pressed "Run" and got those points on map.