Overwriting Underlying Shapefile Table with ArcPy in ArcGIS Pro

[NaseerMuhammad]

Hi all,

Background: I'm currently working in ArcGIS Pro 2.1.0 using a toolbox with a script attached. The goal of the script is to send a pandas data frame representation of a shapefile's (possibly a geodatabase in the future) table out to a server that does some manipulation on the rows and sometimes adding new fields, then read this updated table back into ArcGIS Pro with the script in the toolbox and update the visible map. 

Issue: So far I've been easily able to send the data out and read it back in, but I have trouble when updating the shapefile's current table. My initial idea was to:

• Create a new table based off the updated data from the server (using numpy array to table)
• Copy the current table of the shapefile (using table to table conversion)
• Compare the fields in the updated table to the copied current table's and add fields to table copy accordingly (add field management)
• Create a dictionary of the updated table
• Use an update cursor to update the copied current using a key to access values in the updated table's dictionary
• Since overwrite is enabled (arcpy.env.overwriteOutput = True), use table to table conversion to now update the actual current table of the shape file

At the last, actually overwriting the table, I always run into an issue. With the steps outline above, I usually run into an issue where the actual shapefile is deleted leaving just the dbf and related files, and removing the associated layer from the map. I instead tried to use an update cursor and ran into an issue with locks on files. And when trying to delete the actual dbf file and create it anew, delete management ran into this same lock file issue. 

I'm struggling to find anything online that's helping with my issue and I wanted to know, am I approaching updating a table based on new data the right way? 

I will attach code snippets clearly displaying which method was used to update the table. My biggest suspicion is that, at least for method two, add fields management is creating a lock file on the table and not removing it when finished adding the fields. This then interferes with the cursor creation when updating the original table later.

[DanPatterson_Retired · Accepted solution]

without looking at your zip file, I always fine using

ExtendTable—Data Access module | ArcGIS Desktop or 

numpy recfunctions (addjoin, etc)

numpy/recfunctions.py at master · numpy/numpy · GitHub 

useful.  Even quicker than having to do a join and since you can get pandas stuff out to its roots (numpy structured arrays), then using those tools will help.

I assume you have a key field you are transferring between arc* and array-world (it OBJECTID or OID or equivalent)

[NaseerMuhammad]

I'll try both of these out and see if they work. For the numpy recfunctions, after manipulating the arrays, how would I then update the table of the shapefile? My guess would be to use NumpyArrayToTable, but it apparently doesn't overwrite existing tables, would I need to delete the table first?

[DanPatterson_Retired]

nothing is going to replace the table because the dbf, shx and shp are all tied together, if the dbf is not 'perfect' then your shapefile will be corrupt.  You need to join and if you don't want the original fields,

Delete Field—Data Management toolbox | ArcGIS Desktop 

can be used to delete multiple fields from the original

[NaseerMuhammad]

Extend table did the trick, thank you so much! Not sure if I implemented it quite the way you intended, but it works for the issue I was facing. With extend table I was able to get around the lock happening on the table due to the AddField management method I was using before which then lets me use the update cursor on the table later with no issues.

[DanPatterson_Retired]

Glad it worked... the nice thing about Extend table is that the join is permanent and it saves the requirement of having to save out again to maintain the data.