topic loop through a folder with text files in ArcGIS Pro Questions

loop through a folder with text files

MarkBalman — Thu, 30 Jan 2020 13:46:06 GMT

Hi all

I am stuck with how to loop through a folder containing numerous text files containing lat and long values. The idea is to open a text file, create an XY layer and then append the result into a layer within a geodatabase. I can do the following which takes a single text file, creates the xy layer and then appends this xy layer to the geodatabase layer:

# XYTableToPoint.py
# Description: Creates a point feature class from input table

# import system modules
import arcpy

# Set environment settings
arcpy.env.workspace = r"C:\Projects\points\test.gdb"

# Set the local variables
in_table = r"D:\points\split_data\file_1.txt"
out_feature_class = "file1"
x_coords = "field7"
y_coords = "field6"
z_coords = ""

# Make the XY event layer...
arcpy.management.XYTableToPoint(in_table, out_feature_class,x_coords, y_coords, z_coords,
arcpy.SpatialReference(4326))

# Print the total rows
print(arcpy.GetCount_management(out_feature_class))

arcpy.management.Append("file1", r"C:\Projects\points\test.gdb\all_Points", "NO_TEST", 'id "id" true true false 25 Text 0 0,First,#,file1,Field1,0,8000;individualCount "individualCount" true true false 15 Text 0 0,First,#,file1,Field2,0,8000;year "year" true true false 4 Long 0 0,First,#,file1,Field3,-1,-1;month "month" true true false 2 Short 0 0,First,#,file1,Field4,-1,-1;country "country" true true false 255 Text 0 0,First,#,file1,Field5,0,8000;decimalLatitude "decimalLatitude" true true false 8 Double 0 0,First,#,file1,Field6,-1,-1;decimalLongitude "decimalLongitude" true true false 8 Double 0 0,First,#,file1,Field7,-1,-1;Name "Name" true true false 100 Text 0 0,First,#,file1,Field8,0,8000', '', '')

This works fine but this would be tedious to do edit and rerun manually for each file (potentially over a 1000 files).

Any advice most welcome.

Mark

Re: loop through a folder with text files

George_Thompson — Thu, 30 Jan 2020 14:45:22 GMT

Not Python so to speak but another option would be to use the LocateXT functionality within ArcGIS Pro; What is LocateXT—ArcGIS Help | ArcGIS Desktop

That may be easier to adjust. There is also more functionality coming out with the next release at ArcGIS Pro 2.5.

Re: loop through a folder with text files

JoshuaBixby — Thu, 30 Jan 2020 16:51:20 GMT

Python has several ways to find TXT files on a file system: Find all files in a directory with extension .txt in Python - Stack Overflow

Re: loop through a folder with text files

LorneDmitruk2 — Thu, 30 Jan 2020 21:33:18 GMT

Use the arcpy.da.Walk function to loop through the files.

Re: loop through a folder with text files

DavidPike — Sun, 12 Dec 2021 02:38:28 GMT

import arcpy, os

# Set environment settings
arcpy.env.workspace = r"C:\Projects\points\test.gdb"

# Set the local variables
in_folder = whatever your folder is
merged_fc = name of final merged fc to be created in gdb

x_coords = "field7"
y_coords = "field6"
z_coords = ""


#grab all the text files and put them in  a list
text_file_list = [ ]

for file in os.listdir(in_folder):
    if ".txt" in file:
        text_file_list.append(file)

# Make the point feature classes
path_list = [ ]
for text_file in text_file_list:
    in_table = os.path.join(in_folder,text_file)
    name = text_file.strip(".txt")
    out_fc = os.path.join("in_memory",name)
    path_list.append(out_fc)
    arcpy.management.XYTableToPoint(in_table, out_fc,x_coords, y_coords, z_coords, arcpy.SpatialReference(4326))

arcpy.Merge(path_list,merged_fc)

arcpy.Delete_management("in_memory")

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: loop through a folder with text files

JoshuaBixby — Thu, 06 Feb 2020 23:15:06 GMT

Mark Balman‌, have any of the suggestions helped?

Re: loop through a folder with text files

MarkBalman — Fri, 07 Feb 2020 10:45:06 GMT

Hi Lorne

Thank you for the suggestion. Am quite new to python so have yet to try this method

Best regards

Mark

Re: loop through a folder with text files

MarkBalman — Fri, 07 Feb 2020 10:46:30 GMT

Hi Joshua

Thanks for the suggestions, will try these once I have more time to spend on the task

Best regards

Mark

Re: loop through a folder with text files

MarkBalman — Fri, 07 Feb 2020 10:50:08 GMT

Thank you David,

I tried this but got error message which I am trying to work out (will be trying again next week)

Best regards,

Mark

Re: loop through a folder with text files

MarkBalman — Fri, 07 Feb 2020 10:51:52 GMT

Thank you George,

Not tried Locate yet but will have a look when I have some spare time.

Best regards

Mark

Re: loop through a folder with text files

MarkBalman — Thu, 13 Feb 2020 09:24:43 GMT

Hi George, Locate looks interesting but sadly we don't have a license for it.

Best wishes,

Mark

Re: loop through a folder with text files

MarkBalman — Sun, 12 Dec 2021 02:38:31 GMT

Hi David

I finally managed to get this working. I needed to modify the following lines

out_fc = os.path.join("in_memory",name) this doesn't work so changed to 
out_fc = os.path.join(r"memory",name)
arcpy.Merge(path_list,merged_fc) this doesn't work so changed to 
arcpy.management.Merge(path_list,merged_fc)

Many thanks for your assistance!
Best,
Mark

Re: loop through a folder with text files

JoshuaBixby — Thu, 13 Feb 2020 15:01:53 GMT

Just for reference, the old style of calling geoprocessing tools like Merge was:

arcpy.Merge_management()

and the new style is:

arcpy.management.Merge()

Both styles have worked since the beginning, but Esri is now focusing/transitioning more to the "new" style, which is definitely more Pythonic.