topic Re: Arcpy Intersect_analysis - cannot use layers as input in a script in Geoprocessing Questions

Arcpy Intersect_analysis - cannot use layers as input in a script

KimOllivier — Tue, 05 May 2015 22:51:38 GMT

I prototyped a tool in ArcMap using a multiple intersect of a point featureclass with several polygon layers. Basically a multiple point-in-polygon operation. It is much faster now that partitioning is automatically invoked at 10.3 (or some earlier version).

To avoid all the extra polygon fields I turn off all the additional fields in the layer properties. I do get extra FID_<fc> fields that I did not ask for but these are easy enough to delete with a list = [f.name for f in arcpy.Listfields(fc,'FID_*')] in the delete tool.

But now I want to put this in a script... I take the tool as a snippet and put it in a stand-alone script. I need to define the layers, which I do using arcpy.management.MakeFeatureLayer() adding a Fieldinfo() object with all the fields hidden or visible. (the opposite of visible is..?)

Output:

the overlay produces a point layer BUT last polygon layer is repeated n times, the other (n-1) layers are missing, the process takes far too short a time and the result is nonsense, but with no error messages. I was hoping to use the rename facility in fieldinfo (it does not rename the alias so It needs an extra step) but since nothing works that was too much of a stretch. ie why got to the trouble of using fieldInfo anyway.

Intersect will properly populate FID_<fc_name> fields if I use the option 'ONLY_FID' on featureclasses, but then I have the tedious task of adding the fields and populating them myself. JoinField is not suitable for 2 million records. Maye MakeSQLQuery would be better if I had to go down that route, add indexes, build a complex sql expression. Oh, that won't work, because all the tables have to be in the same database, unlike spatialite.

The only workaround I can think of is to leave ALL the fields on with a list of featureclasses, not layers, and then run a humongous delete function afterwards instead of a tidy fieldinfo beforehand.

Re: Arcpy Intersect_analysis - cannot use layers as input in a script

DanPatterson_Retired — Tue, 05 May 2015 23:09:48 GMT

Kim

I am having trouble understanding this step

"Now the last layer is repeated n times, the process takes far too short a time and the result is nonsense, but with no error messages. I was hoping to use the rename facility in fieldinfo (it does not rename the alias so It needs an extra step) but since nothing works that was too much of a stretch."

can you elaborate a bit or show an example...

Re: Arcpy Intersect_analysis - cannot use layers as input in a script

KimOllivier — Wed, 06 May 2015 03:31:49 GMT

Thanks for the concern. I am certain that its a bug, and I am just grumpy and want to alert others when they hit the same thing.

Suppose I have an address point layer and I want to assign a suburb and postcode from some polygon layers.

D:/data/NZFireService/nzsuburb.gdb/nz_localities Keep fields: ['suburb_4th', 'major_name']

Fieldinfo: nz_localities SUBURB_4TH SUBURB_4TH VISIBLE #;MAJOR_NAME MAJOR_NAME VISIBLE #

List fields: [u'OBJECTID', u'Shape', u'POSTCODE', u'ROUND_NAME', u'MAIL_TOWN', u'Shape_Length', u'Shape_Area']

D:/data/NZPost/PostCode/PNF.gdb/PostCode Keep fields: ['postcode']

Fieldinfo: PostCode POSTCODE POSTCODE VISIBLE #

List fields: [u'OBJECTID', u'Shape', u'POSTCODE', u'ROUND_NAME', u'MAIL_TOWN', u'Shape_Length', u'Shape_Area']

Processing overlays ... ['e:/crs/presentgdb/roadadd.gdb/addhist', 'lay_nz_localities', 'lay_PostCode']

0:00:51.979000

[u'OBJECTID', u'SHAPE', u'FID_addhist', u'sad_id', u'house_number', u'rna_id', u'range_high', u'rcl_id', u'et_created', u'sufi', u'unofficial_flag', u'range_low', u'et_edited', u'status', u'roadtype', u'location', u'status_name', u'road_name', u'road_type', u'road_suffix', u'full_road_name', u'FID_PostCode', u'POSTCODE', u'ROUND_NAME', u'MAIL_TOWN', u'FID_PostCode_1', u'POSTCODE_1', u'ROUND_NAME_1', u'MAIL_TOWN_1']

The result is - No suburbs! Just the postcode fields twice. If I have three polygon layers they are added 3 times and so on.

My workaround was to use full featureclass paths with ALL and run a couple of list comprehensions afterwards to generate a large list of fields to delete. The time ballooned out to 2 hours.

I then ran the new AlterField command that lets me rename a field and the alias. The fieldInfo object appears to set a new alias but does not.

Re: Arcpy Intersect_analysis - cannot use layers as input in a script

curtvprice — Wed, 06 May 2015 05:15:38 GMT

It will properly populate FID_<fc_name> fields if I use the option 'ONLY_FID' on featureclasses, but then I have the tedious task of adding the fields and populating them myself.

I don't think it's that tedious if you use ONLY_FID, then add joins, then run Merge with a field map to get just the fields you want.

Re: Arcpy Intersect_analysis - cannot use layers as input in a script

KimOllivier — Wed, 06 May 2015 05:38:29 GMT

Add Joins? You must be only using a sample. That would fail for 2 million records in my experience. The reason I went back to the overlay operations is because partitioning has been implemented. Before that it wasn't an option.

Since merge is not part of the partition function I wonder how long it might take.

My complaint is why do I have to go to all this trouble since the help clearly says that layers are a valid input. This thread is to warn that they are not.

Making a fieldmap is very tricky if I want to automate it because I would need to automate building the field maps.

My script is already complicated. It has gone from a single line Intersect tool to 140 odd lines and its not a general tool even then.

Re: Arcpy Intersect_analysis - cannot use layers as input in a script

curtvprice — Wed, 06 May 2015 05:44:33 GMT

Two million is a lot of features. Yes, the new behind the scenes tiling is awesome because before they were doing that it was impossible do to an overlay of that size.

Merge is going to be a dramatically faster than deleting fields, but you're right, messing with field maps in Python can be a lot of work.