Creating shapefile from ASCII file

[MarcEngelsma1]

Hi all,
I'm trying to convert an ASCII file to a point feature class. I've been busy with this little project on and off for a few weeks now but it seems I've run out of ideas finally :).

The idea behind the script below is that it will be implemented as a tool eventually and it has to take other ASCII files as input as well although they have the same structure as the example in the attachment. At least one of them is in the order of 185 MB, so quite large.

The code I have sofar works without errors, it creates a shapefile with the correct name, it adds the correct fieldnames and it creates the correct number of points from the ASCII file in the right location.
There are still two critical things lacking though before I can turn it into a tool:

1) While I figured out how to add the fields I can't seem to get the fieldtype right. Some fields should be of type String but others should be Double or Long. Somebody gave me the tip to use a try and except on the data to test if it can be made into an integer, if not it's text, otherwise it's a Double. I understand the logic behind it, I'm just struggling to implement it. How do you link the result for the try and except to the fields that are created?

2) The script now creates the correct points but the only data that is attached to them is the X,Y data in the shape attributes. Short of *manually* letting the script write each item in the list of the current line to the correct field (in the same way I write the X,Y geometry) is there a quick way of getting all the data from the ASCII to the table correctly? I feel there should be a way but I was not able to find it yet or come up with one.

Thanks in advance for any answers, I hope someone can help me out here.

Cheers,
Marc

#Script settings en voorbereiding
#Check out any necessary licenses

import arcpy, string, os, sys, linecache

arcpy.env.overwriteOutput =1

#Inputs
File = r"S:\GEOINFO\01_Basis_info\12_BAG\Adresseerbare object special\ADRESOBJECT_SPS_20130131.UVA2"

InName =  os.path.splitext(os.path.basename(File))[0]
print InName

OutDatabase = r"S:\GEOINFO\01_Basis_info\12_BAG\BAG.gdb\Adresseerbare_Object_Special"
print OutDatabase

OutShape = str(OutDatabase) + "\\" + str(InName)
print OutShape

InputASCII = file(File, "r")

# Create shapefile
arcpy.CreateFeatureclass_management(r"S:\GEOINFO\01_Basis_info\12_BAG\BAG.gdb\Adresseerbare_Object_Special", InName, "Point", "", "", "", "PROJCS['RD_New',GEOGCS['GCS_Amersfoort',DATUM['D_Amersfoort',SPHEROID['Bessel_1841',6377397.155,299.1528128]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Double_Stereographic'],PARAMETER['False_Easting',155000.0],PARAMETER['False_Northing',463000.0],PARAMETER['Central_Meridian',5.38763888888889],PARAMETER['Scale_Factor',0.9999079],PARAMETER['Latitude_Of_Origin',52.15616055555555],UNIT['Meter',1.0]]")

print "Shapefile created"

#Read and create columns from ASCII
Headerline = linecache.getline (File,4)

TextList = Headerline.split(';')
for Column in TextList:
    arcpy.AddField_management(OutShape, Column, "TEXT")

print "Field names added"

#Create points from ASCII
Icur = arcpy.InsertCursor(OutShape)
LineCorrect = 0
LineError = 0

for Line in InputASCII.readlines ()[4:]:
    try:
       Row = Icur.newRow()

       List = Line.split(";")
       Point = arcpy.Point((List[20]), (List[21]))
       print Point.X
       print Point.Y

       Row.setValue("Shape", Point)
       Icur.insertRow(Row)

       LineCorrect = LineCorrect + 1
       del Row

    except:
       LineError = LineError + 1
       del Row

del Icur

print "LineCorrect = " + str(LineCorrect)
print "LineError = " + str(LineError)

#-------------------------------------------------------------------------------
#Close files
InputASCII.close

[RhettZufelt]

Have not tried this, so don't know if it would work, but you might try to MakeXYeventLayer http://resources.arcgis.com/en/help/main/10.1/index.html#//00170000006z000000 .

Like I said, not tried in python, but in ArcGIS, this lets you select the x/y fields from a table, and creates a point event theme from it and carries all the attributes in the original table itself.
Then, just export the event theme to shapefile.

Just a thought,

R_

[MarcEngelsma1]

Have not tried this, so don't know if it would work, but you might try to MakeXYeventLayer http://resources.arcgis.com/en/help/main/10.1/index.html#//00170000006z000000 .

Like I said, not tried in python, but in ArcGIS, this lets you select the x/y fields from a table, and creates a point event theme from it and carries all the attributes in the original table itself.
Then, just export the event theme to shapefile.

Just a thought,

R_

Perhaps I should have mentioned it but that was our initial thought as well, however we ran into some problems with that as well, I can not get the tool to accept the input file for some reason. Here's what I tried on the file and as with the other question, all suggestions are welcome (code is below and the CSV file itself is also attached)

- The tool does not accept a semi-colon as separator. Exchanging this character for a comma solves this problem but if you want to keep the original file you have to make a copy of the file. That's already two extra steps.
- The first three lines have are additional data which is not needed in teh shapefile so they have to be deleted, otherwise the tool cannot find the right fieldnames (am I correct here?). I didn't get this to work in python yet so I did it manually but that should be possible I think.
- If I follow the example from the ESRI help file on this tool I still get the message that the file does not exist or is not accepted. this is also the case if I try to drag it into the tool outside of python.

Additionally I changed the file extension from the original .UVA2 to .CSV. Essentially it's still just a txt file so I don't think this is an issue.

import arcpy

FileIn = r"S:\GEOINFO\01_Basis_info\12_BAG\Adresseerbare object special\Test.csv"
OutLayer = r"S:\GEOINFO\01_Basis_info\12_BAG\Adresseerbare object special\OutLayer.lyr"
x_coords = "Xcoordinaat (RD)"
y_coords = "Ycoordinaat (RD)"

arcpy.MakeXYEventLayer_management (FileIn, x_coords, y_coords, OutLayer)

arcpy.CopyFeatures_management(OutLayer, r"S:\GEOINFO\01_Basis_info\12_BAG\Adresseerbare object special\Test.shp")

print "Finished"

[LindseyWood]

Just some input....
I am actually having a very similar issue but with a polygon set... things I have found....
1) If you create a feature class in a gdb using catalog you can push the IMPORT button to somewhere in that tool about the 4th box and it grabs all the csv's fields and info. Which is exactly what is needed with the csv not sure about the ";" in your ascii
but the create feature class in python tool does not support this?

2) for the fields making a copy is an extra step but from there I think you could do something like
fields= arcpy.ListFields(gdbTable)
        fieldinfo = arcpy.FieldInfo()
or fieldmapping.addTable(gdbTable)
either of these I think could be used for the correct information for adding each field. "Double" "text" etc it should be all there

then add your fields from there based on your list" I have not tried this yet as I am testing the join"


3) Then either write them out like you said which would take awhile... or do a join at the end with your copied table something with a matching id.... right now I am testing the join which works fine but takes a while... I am hoping for a faster way as well it seems like there should be all this overhead seems unnecessary.

4) Also you header has a ton of spaces in it which arc will not read that may be why your xy even layer is not working... maybe this best for you in that case you would need to do some clean up on the ascii anyway, I also have to make a copy prior to input .csv for removal of spaces and creating decimal degrees to work with....
HEADER....
Identificerende sleutel standplaats,Datum begin geldigheid mutatie,Datum einde geldigheid mutatie,

[RhettZufelt]

Marc,

This code seems to be working just fine:

import arcpy


# Local variables:
temp = "D:\\temp"
ExampleTest__csv = "C:\\Documents and Settings\\rkzufelt\\Desktop\\ExampleTest .csv"
ExampleTest_CopyRows = "in_memory\\ExampleTest_CopyRows"
ExampleTestLayer = "ExampleTestLayer"

# Process: Copy Rows
arcpy.CopyRows_management(ExampleTest__csv, ExampleTest_CopyRows, "")

# Process: Make XY Event Layer
arcpy.MakeXYEventLayer_management(ExampleTest__csv, "Xcoordinaat_RD_", "Ycoordinaat_RD_", ExampleTestLayer, "", "")

# Process: Feature Class To Shapefile (multiple)
arcpy.FeatureClassToShapefile_conversion("ExampleTestLayer", temp)

However, I need to make your csv file a "valid" input file first.

ESRI tools do not allow special characters in the column headings.  So, once I stripped out the spaces and parrenthesis from your header row, all works as expected.


It sounds like maybe you are stuck with a particular input filetype/format?  If so, you might put some code in there that reads into memory (if you don't want to make a separate copy), but "filters" the results before writing (like don't copy the first three lines, then filter the column heading similar to: input_line = input_line.replace('(','_').replace(')','_').replace(' ','') to strip the bad chars).

The above code isn't actually using the copyRows, the MakeXYEverntLayer works just fine on the "cleaned up" csv file, put the copyRows in there as an example of reading into arcpy's in_memory workspace.
In the above code, the CopyRows is making an object that isn't used, change the input to the MakeEventLayer tool to the output of the CopyRows in order to utilize the copy vs the original.


R_