topic Re: Dissolve with arcpy.da.UpdateCursor in Python Questions

Dissolve with arcpy.da.UpdateCursor

PeterShoemark — Sun, 13 Sep 2015 22:50:55 GMT

Hi, can anyone give me some guidance as to how (or if it is possible) to use arcpy.da.updatecursor to loop through multiple polylines and dissolve them one by one ? (Im on Win7, arcgis 10.2.1, Python 2.7 and the data is in an Oracle 11g sde db)

The problem I have is numerous layers that I am trying to automate removal of self overlaps but that needs to occur by objectid not by layer and I also dont want to create another dataset from the one being updated so maybe need to start an edit session first??.

While Im not a wizard at this (or I may have been able to resolve it by now), I do understand enough to be able to adjust the code below to do some other maintenance functions.

# The "if (row[1] or 0) == 0: " code: "row[1] or 0" will return either the

# value of the outputField, or if that value is None (null), then it will return 0...

inputField = "SEVENTH_ID"

outputField = "ELEVENTH_ID"

fc = "REGION_46"

fields = [inputField, outputField]

with arcpy.da.UpdateCursor(fc, fields) as cursor:

for row in cursor:

if (row[1] or 0) == 0:

row[1] = row[0]

cursor.updateRow(row)

del cursor, row

I think what I am trying to do would look something like;

# Dissolve the self overlaps by row for REGION_46

# If the current row has no self overlaps, move onto the next row

inputField = "OBJECTID"

fc = "REGION_46"

fields = [inputField]

with arcpy.da.UpdateCursor(fc, fields) as cursor:

for row in cursor:

if (row[0] has self overlaps, use "DISSOLVE" to eliminate them

cursor.updateRow(row) then move onto the next row

del cursor, row

Anyone successfully acjieved this previously or can give me some useful guidance?

(Noting I have read the help menu and other resources numerous times and not been successful)

Cheers, Peter

Re: Dissolve with arcpy.da.UpdateCursor

WesMiller — Mon, 14 Sep 2015 13:07:33 GMT

Is this something you'll need to do more than once? A topology may be the better route.

Re: Dissolve with arcpy.da.UpdateCursor

JoshuaBixby — Mon, 14 Sep 2015 20:06:28 GMT

Can you provide an real-world example, or illustrative example, of a polyline and what the expected outcome would be?

Unfortunately, this is a case where the weaknesses of the ArcPy Geometry classes really show throw. Other Python geometry libraries/packages like GeoDjango and Shapely offer more functionality at the geometry level. For example, ArcPy Geometry classes don't offer a simple/is_simple functionality to tell whether a linestring is simple or complex, which would be helpful in this case at determining whether a line either crosses or overlaps itself.

I may be able to offer additional feedback if I have a specific example to work with.

Re: Dissolve with arcpy.da.UpdateCursor

PeterShoemark — Tue, 15 Sep 2015 01:30:48 GMT

Thanks @ Wes Miller, Coincidentally the fc is a line topolgy.

I have run 'export topology errors' and can see which lines have overlap by joining the original and export tables so now I want to loop through the original lines by objectid to dissolve each lines errors.

Re: Dissolve with arcpy.da.UpdateCursor

PeterShoemark — Tue, 15 Sep 2015 01:36:17 GMT

Thnaks @ Joshua Bixby, Im not sure how to provide an example so let me expalin long hand.

I have a line which has several self overlaps on it where the previous operator has snapped 'back and forth' at times.

I want to take the line and dissolve all the self overlaps of that line without either doing the dissolve by layer or creating a new fc or layer.

Each of my dat sets has many lines within and objectid is the field i want to individually dissolve by. hth.

Re: Dissolve with arcpy.da.UpdateCursor

JoshuaBixby — Tue, 15 Sep 2015 14:24:08 GMT

Load the feature class in question, select 1 of the problem features using the ArcMap GUI, and then run the following code in the interactive Python window:

print next(arcpy.da.SearchCursor(fc, "SHAPE@"))[0].WKT

Replace fc with the name of the feature class/layer with the selected feature. This should print out the WKT representation of the feature, which can be used to recreate the feature for testing.

Re: Dissolve with arcpy.da.UpdateCursor

PeterShoemark — Wed, 16 Sep 2015 00:54:10 GMT

Thanks Joshua Bixby some sample lines below to demonstrate.

OBJECTID 1 has self ovelap and I want to dissolve it (and go to next OBJECTID)

OBJECTID 1 MULTILINESTRING ((9700614.7831249982 4432945.8557500001, 9700614.7831249982 4427945.8557500001, 9700614.7831249982 4428945.8557500001))

OBJECTID 2 has self ovelap and I want to dissolve it (and go to next OBJECTID)

OBJECTID 2 MULTILINESTRING ((9702086.5310000032 4431846.6370000001, 9702086.5308749974 4427846.6367499996, 9702086.5308749974 4428846.6367499996))

OBJECTID 3 has NO self ovelap so no requirement to dissolve it (skip and go to next OBJECTID)

OBJECTID 3 MULTILINESTRING ((9703086.5310000032 4431846.6370000001, 9703086.5308749974 4427846.6367499996, 9704269.3477500007 4427852.6163749993))

One of my fcs has more than 2000 OBJECTIDs that have self overlap and I want to dissolve them individually by OBJECTID. (and from my research perhaps arcpy.da.updatecursor is the way to go as it could cycle through the OBJECTIDs ??) Cheers, Peter

Re: Dissolve with arcpy.da.UpdateCursor

JoshuaBixby — Sat, 11 Dec 2021 16:41:05 GMT

Thanks for providing the WKT representations, quite helpful.

A quick, or not so quick, aside before moving on. When talking about valid geometries and spatial relationships, the nuance and complexity of the topics are often lost or oversimplified in conversation. There are very specific definitions, and sometimes there are disagreements or discrepancies between different groups/companies/organizations when it comes to the meaning and implementation of those definitions.

Looking at the examples provided in the context of ArcPy Geometry classes:

>>> #OBJECTID 1
>>> l1 = arcpy.FromWKT('MULTILINESTRING((9700614.7831249982 4432945.8557500001,'
...                                     '9700614.7831249982 4427945.8557500001,'
...                                     '9700614.7831249982 4428945.8557500001))')
...                               
>>> l1.overlaps(l1)
False
>>> l1.crosses(l1)
False
>>> l1.touches(l1)
False
>>>
>>> #OBJECTID 2
>>> l2 = arcpy.FromWKT('MULTILINESTRING((9702086.5310000032 4431846.6370000001,'
...                                     '9702086.5308749974 4428846.6367499996,'
...                                     '9702086.5308749974 4428846.6367499996))')
...                               
>>> l2.overlaps(l2)
False
>>> l2.crosses(l2)
False
>>> l2.touches(l2)
False
>>>
>>> #OBJECTID 3
>>> l3 = arcpy.FromWKT('MULTILINESTRING((9703086.5310000032 4431846.6370000001,'
...                                     '9703086.5308749974 4427846.6367499996,'
...                                     '9704269.3477500007 4427852.6163749993))')
...                               
>>> l3.overlaps(l2)
False
>>> l3.crosses(l3)
False
>>> l3.touches(l3)
False
>>>

From one set of definitions, the examples don't overlap, cross, or even touch themselves, which makes it fairly difficult to identify the geometries that "self overlap." The ArcPy Geometry classes, unfortunately, do not have a self-overlap method or anything similar. For the Python implementations (Shapely, GeoDjango) of JTS/GEOS, the closest approximation to a self-overlap method for lines is is_simple:

>>> from shapely import wkt
>>> 
>>> #OBJECTID 1
>>> _l1 = wkt.loads('MULTILINESTRING((9700614.7831249982 4432945.8557500001,'
...                                  '9700614.7831249982 4427945.8557500001,'
...                                  '9700614.7831249982 4428945.8557500001))')
...                                 
>>> _l1.is_simple
False
>>> 
>>> #OBJECTID 2
>>> _l2 = wkt.loads('MULTILINESTRING((9702086.5310000032 4431846.6370000001,'
...                                 '9702086.5308749974 4428846.6367499996,'
...                                 '9702086.5308749974 4428846.6367499996))')
...                                 
>>> _l2.is_simple
True
>>> 
>>> #OBJECTID 3
>>> l3 = wkt.loads('MULTILINESTRING((9703086.5310000032 4431846.6370000001,'
...                                 '9703086.5308749974 4427846.6367499996,'
...                                 '9704269.3477500007 4427852.6163749993))')
...                                 
>>> _l3.is_simple
True
>>>

While it is clear that OBJECTID 1 is not simple because it "self overlaps," or folds back onto itself, it is less clear why OBJECTID 2 is simple. Since JTS/GEOS libraries are OGC compliant, consecutive equal vertices are allowed in a LineString. A basic simplify operation will remove not only redundant vertices but also some intermediate vertices if not necessary to the topology of the line.

>>> _lc = wkt.loads('LINESTRING(0 0, 1 0, 1 0, 2 0, 2 0, 3 0, 3 0, 3 0)')
>>> _lc.is_simple
True
>>> _lc.simplify(0).wkt
'LINESTRING (0 0, 3 0)'
>>>

Instead of continuing this already-too-long comment, I will wrap us this aside and post another comment with additional thoughts.

UPDATE: OBJECTID 2 in my code above is not the same as OBJECTID 2 provided by OP. I must have miscopied the WKT representation of OBJECTID 2 and created a line with duplicate end points. Instead of correcting OBJECTID 2, I will leave it as is to discuss the situation when a line has consecutive vertices that are equal.

Re: Dissolve with arcpy.da.UpdateCursor

JoshuaBixby — Sat, 11 Dec 2021 16:41:08 GMT

As I alluded to in my previous comment, identifying the lines with "self overlaps" is not easy using the ArcPy Geometry classes. One could, if so inclined, write his/her own Shamos-Hoey or Bentley–Ottmann algorithm to identify self-intersections with lines, but it might just be easier at that point to convert ArcPy geometry objects over to Shapely or GeoDjango geometry objects and run an is_simple test on them. Another option could be generating a list of problem lines first, using Esri topology tools for example, and then using that list to only process lines that you know have issues.

I can't say it will work in all situations, but one approach worth looking into is using the union() method of the ArcPy Geometry class.

>>> #OBJECTID 1
>>> l1.length
6000.0
>>> l1u = l1.union(l1)
>>> l1u.length
5000.0
>>> l1u.WKT
u'MULTILINESTRING ((9700614.7830810547 4427945.8558959961, '
                   '9700614.7830810547 4428945.8558959961,'
                   '9700614.7830810547 4432945.8558959961))'
>>>
>>> #OBJECTID 2
>>> l2.length
3000.0002500005094
>>> l2u = l2.union(l2)
>>> l2u.length
3000.0001831054788
>>> l2u.WKT
u'MULTILINESTRING ((9702086.5311279297 4431846.6370849609,'
                   '9702086.5308837891 4428846.6369018555))'
>>>
>>> #OBJECTID 3
>>> l3.length
5182.8322396370995
>>> l3u = l3.union(l3)
>>> l3u.length
5182.832314284238
>>> l3u.WKT
u'MULTILINESTRING ((9703086.5311279297 4431846.6370849609,'
                   '9703086.5308837891 4427846.6369018555,'
                   '9704269.3479003906 4427852.6165161133))'
>>>

As you can see, the union() method of the ArcPy Geometry class does some amount of geometry validating when it creates a new geometry.

For OBJECTID 1, the line was cleaned up by reordering the vertices so the line no longer folded back onto itself.
For OBJECTID 2, the duplicate vertex was simply removed.
For OBJECTID 3, the union operation effectively recreated the original line.

Re: Dissolve with arcpy.da.UpdateCursor

PeterShoemark — Thu, 17 Sep 2015 22:59:59 GMT

Thank you for the comprehensive and concise response Joshua. Joshua Bixby

While I hear what you are saying, let's not get hung up on "valid geometries and spatial relationships etc etc"

Suffice to say I can open the fc attribute table and manually select each row and run the stock Dissolve tool and achieve the precise desired outcome. What I dont want to do is spend the last 6 years of my working life doing that when I suspect there are more efficient methods out there that users can point me to.

So to my original question, and in light of your findings with Union, "can anyone give me some guidance as to how (or if it is possible) to use arcpy.da.updatecursor to loop through multiple polylines and dissolve (or UNION) them one by one? (addendum, I want to use this in a stand alone Python script) Cheers, Peter

Re: Dissolve with arcpy.da.UpdateCursor

NeilAyres — Sat, 11 Dec 2021 16:41:11 GMT

Well Joshua has effectively given you the answer.

Why not implement something like this :

with arcpy.da.UpdateCursor(fc, ["SHAPE@"]) as Cur:
    for row in Cur:
        origGeom = row[0]
        row[0] = origGeom.union(origGeom)
        Cur.updateRow(row)

Re: Dissolve with arcpy.da.UpdateCursor

PeterShoemark — Sun, 20 Sep 2015 23:30:32 GMT

Thanks for the response Neil. Unfortunately I still dont have a clear handle on the process.

My code has fc set and returns the fc value to the screen with 'print'.

My fc has values in it including polylines whihc self overlap by way of returning back on themselves.

If I add the code above I get an error mesaage in my log which says;

'NoneType' object has no attribute 'union'

In my limited experience I have received that message in other situations but due to there being no value. Have I misinterperated this? Cheers, Peter

Re: Dissolve with arcpy.da.UpdateCursor

JoshuaBixby — Sat, 11 Dec 2021 16:41:14 GMT

Your error message indicates that the cursor is returning a None object for a shape, which happens when there is a NULL record or empty geometry in the feature class. To avoid the error, just check a geometry object exists before trying to union it.

with arcpy.da.UpdateCursor(fc, ["SHAPE@"]) as cur: 
    for shape, in cur:
        if shape:
            shape = shape.union(shape)
            cur.updateRow(shape)

Re: Dissolve with arcpy.da.UpdateCursor

PeterShoemark — Mon, 21 Sep 2015 22:39:43 GMT

Joshua Bixby Thanks again Joshua. I tried the code above but received an error message " 'list' object has no attribute 'union' ". (There is also a small typo (a comma) but I saw that and removed it)

I took your comments on board and re-ran your code from the day before after manually removing the NULLs from my fc and that code then ran successfully and resolved the overlaps very nicely.

However, while I do have code that would loop through the fc and remove NULLs, there is other data in fileds that I need to keep and above all else i am trying not to create new data sets for a bunch of business reasons.

Apologies in advance for what may then be a beginners question, but should I be concatenating the 2 lots of code together or is one intended to replace the other?

Re: Dissolve with arcpy.da.UpdateCursor

JoshuaBixby — Tue, 22 Sep 2015 11:49:35 GMT

The "extra" comma in my code wasn't extra, it was purposeful to allow for list/sequence unpacking. Removing the comma changed the shape variable from a geometry object to a list, hence the error message. Try re-running the code as is and see if it works for you.

Re: Dissolve with arcpy.da.UpdateCursor

BruceHarold — Tue, 22 Sep 2015 15:00:12 GMT

Does the Repair Geometry geoprocessing tool not do the job for you?

The Data Interoperability extension can also repair geometries in place and has fine grained control over handling various issues.

Regards

Re: Dissolve with arcpy.da.UpdateCursor

JoshuaBixby — Sat, 11 Dec 2021 16:41:17 GMT

Check/Repair Geometry won't do anything for most of these cases, at least the examples provided, since they are valid geometries. A line that folds back onto itself isn't simple but it is valid, unless you are using SQL Server where Microsoft is the oddball with how they deal with these types of cases.

A quick example using the Check Geometry tool:

>>> SR = arcpy.SpatialReference(26915)
>>> ln = arcpy.FromWKT('LINESTRING(575000 5250000, 576000 5250000, 575500 5250000)')
>>> arcpy.CheckGeometry_management(ln, 'in_memory/ln_chk')
<Result 'in_memory\\ln_chk'>
>>> arcpy.GetCount_management("ln_chk")
<Result '0'>

Re: Dissolve with arcpy.da.UpdateCursor

PeterShoemark — Tue, 22 Sep 2015 23:34:48 GMT

If I leave the comma in, I get this error message; "argument must be sequence of values"

If I leave it out I get this error message; "'list' object has no attribute 'union'"

If I remove the nulls with a sledge hammer appraoch (my effort below), your previous code (row[0] = origGeom.union(origGeom) ) works successfully.

My effort works on calculating shape.len into a new field and running a delete where len < 1 (in reality 20 is generally my shortist length)

rows = arcpy.UpdateCursor(fc1)

for row in rows:

if row.LINE_LENGTH < 5:

rows.deleteRow(row)

Re: Dissolve with arcpy.da.UpdateCursor

JoshuaBixby — Sat, 11 Dec 2021 16:41:20 GMT

Try this:

with arcpy.da.UpdateCursor(fc, ["SHAPE@"]) as cur: 
    for shape, in cur: 
        if shape: 
            shape = shape.union(shape) 
            cur.updateRow([shape])

The error you get is likely from Line #5 or my earlier code. What is odd is that my earlier code worked when I tested on a feature class earlier today, but it generated the same error when I tried it tonight. I must have tinkered enough to change something. The code above here should work and not require removing NULLs or empty geometries.

Re: Dissolve with arcpy.da.UpdateCursor

PeterShoemark — Wed, 23 Sep 2015 09:38:27 GMT

Exactly what I needed Joshua Bixby. Thanks for your assistance. That code with some automated topology code I have written let me cycle through 20,000 records and validate them in very quick fashion. It was a winner.