topic Re: How to determine the changes between two polygon shapefiles of similar data in Spatial Data Science Questions

How to determine the changes between two polygon shapefiles of similar data

WhitneyNewcomb — Wed, 08 Nov 2017 19:51:40 GMT

The city I work for gets its parcel data from the county once a month so we have several years of almost the same data. I want to be able to determine where changes occur between every new month. So what are the changes between September to October, October to November, etc. And by changes I mean any change in geometry (like parcel splits or merges) or changes in attributes (especially changes in ownership).

I've contacted ESRI about this and they weren't able to do a whole lot for me.

Here's what I've tried so far:

I am able to find changes in geometry by using the Select by Location: Identical features and inverting the selection.

This works fine but does not get me changes in attributes.

I would use the Feature Compare tool but this does not work as it requires the same number of rows in both tables - which I don't have.

I also found an old arcgisscripting tool and converted it to arcpy and it worked with point datasets (but at a certain point in time the County changed how they did their parcel data and went to polygons only which the script does not work with).

So I'm at a loss for what to do now.

Does anyone have any advice for me?

Please let me know if there's any info I can give to help you help me.

-Whitney Newcomb

Re: How to determine the changes between two polygon shapefiles of similar data

MichaelVolz — Wed, 08 Nov 2017 19:54:46 GMT

Does the data contain any date/time fields that you can use to differentiate the data? Do you know if the county enforces attribute changes to be tracked by populating a date/time field (Editor Tracking would be useful in this scenario, but you would need to contact the county to see if they are implementing this procedure)?

Re: How to determine the changes between two polygon shapefiles of similar data

DanPatterson_Retired — Wed, 08 Nov 2017 19:57:15 GMT

for your last option... doesn't work with polygons....produce a centroid file of your polygons, then you can use that at least for the attributes

Re: How to determine the changes between two polygon shapefiles of similar data

WhitneyNewcomb — Wed, 08 Nov 2017 21:04:32 GMT

Unfortunately, no it doesn't. They don't even fill out their metadata. It's rather upsetting. Hah!

Re: How to determine the changes between two polygon shapefiles of similar data

WhitneyNewcomb — Wed, 08 Nov 2017 21:08:29 GMT

Dan,

That actually might work for me to some extent, but I'm worried that I'll have a hard time converting it back to a polygon file.

The script I found grabs the different fields of the two parent files and dumps them into one new file so that for every point in it there is (usually) actuallly two points stacked on top of each other. One from the earlier month and one from the later month.

Re: How to determine the changes between two polygon shapefiles of similar data

DanPatterson_Retired — Wed, 08 Nov 2017 21:51:27 GMT

you wouldn't need to convert the centroid file back to a polygon file... it would just be used to identify which record attributes have changed... Your polygon file would remain untouched

Re: How to determine the changes between two polygon shapefiles of similar data

NobbirAhmed — Wed, 08 Nov 2017 22:02:59 GMT

Could you try first converting the polygons to polylines and then using the polylines to compare? If you want you can use Polygon to Line under Data Management Toolbox, in Features toolset.

http://support.esri.com/en/technical-article/000007707

Re: How to determine the changes between two polygon shapefiles of similar data

WhitneyNewcomb — Wed, 08 Nov 2017 22:03:58 GMT

You're totally right.

Thank you Dan. You have been most helpful!

Re: How to determine the changes between two polygon shapefiles of similar data

DanLee — Wed, 08 Nov 2017 22:07:55 GMT

Hi Whitney,

We are developing tools to help solve conflation problems. There is a GP tool Detect Feature Changes which currently only works on line features, such as roads, parcel lines, etc. This tool detects spatial (geometry) and attribute differences between two sets of line features.

We are investigating polygon use cases. What you described seems a good case for us. Would you be able to share some sample data, two or three monthly data of a small area? If so, you can attach your data in email to dlee@esri.com.

In the meantime, I think it's a good idea to first exclude the spatially identical ones, as you did. For the remaining parcels, perhaps try Spatial Join Oct vs. Nov, and then Spatial Join Nov. vs Oct. That could give you 1:n relationships both ways to catch what's been split or combined.

Thanks,

Dan (Esri Conflation Team)

Re: How to determine the changes between two polygon shapefiles of similar data

WhitneyNewcomb — Wed, 08 Nov 2017 22:17:11 GMT

Hi Dan,

Let me just check with my higher-ups and make sure I can share the data. It shouldn't be a problem and I'll email you ASAP.

Also, Could you elaborate on how the spatial joins will show me these differences? Thank you.

Re: How to determine the changes between two polygon shapefiles of similar data

DanLee — Wed, 08 Nov 2017 22:37:10 GMT

Great! I look forward to your sample data. Thanks again for sharing the data and for your time.

Spatial Join tool allows you to join 1-to-1 or 1-to-many; you can choose one of many Match Options, such as INTERSECT, CONTAINS, HAVE_THEIR_CENTER_IN, etc. If a polygon in Oct data Is split into 3 polys in Nov. data, the tool will find the relationship. If you choose 1-to-1 join, you will get a Join Count field with the value 3. If 2 polys are merged, the two polys will join the same merged poly. There are other tools to analyze duplicates, e.g. Find Identical, Frequency.

Re: How to determine the changes between two polygon shapefiles of similar data

WhitneyNewcomb — Wed, 08 Nov 2017 22:47:56 GMT

Hi Nobbir,

This doesn't work for because attributes do not come along with the polylines. I really need to see the differences in attributes.

Re: How to determine the changes between two polygon shapefiles of similar data

LindaRadomile — Mon, 19 Feb 2018 16:17:42 GMT

Hi Dan,

Has there been any movement on this tool? We too are looking for a polygon comparison to highlight additions, removals, or modifications.

Re: How to determine the changes between two polygon shapefiles of similar data

DanLee — Mon, 19 Feb 2018 23:37:30 GMT

Hi Linda,

I would still like to ask for your data, if possible, to see the specific case. I did help Whitney to solve the problem he faced once I got his data; he actually had common attributes between the two sources, so Add Join followed by Select Layer By Attribute did the trick. Would you be able to share sample data? You can email to dlee@esri.com

Thanks,

Dan

Re: How to determine the changes between two polygon shapefiles of similar data

GuillermoMarinez1 — Thu, 19 Apr 2018 17:03:34 GMT

Hi, i have the same problem, i am going to implement the folowing process in model builder. When i finished the model i will share them.

looking for the solution i found an tool in datainteroperability that perform the "change detector" operation, if one have this extension can find the diferences between two featureclass.

Re: How to determine the changes between two polygon shapefiles of similar data

deleted-user-_ZI9nJL9TGxh — Tue, 03 Jul 2018 00:31:55 GMT

Hello Guillermo,

Might your model merit a script that searches the tables to compare for changes in the APN before it finds spatial changes? We are working on that angle. My colleague and I are using the Union tool on the two different files with a unqiue field for APNCURRENT and APNPRIOR . I'm working at putting it into a python script but new to Python.

LIsa Pierce

Re: How to determine the changes between two polygon shapefiles of similar data

NobbirAhmed — Tue, 03 Jul 2018 14:52:03 GMT

If you need help with Python scripting please shoot me a mail at nahmed@esri.com (also some sample data will be helpful)

Re: How to determine the changes between two polygon shapefiles of similar data

ChrisDonohue__GISP — Tue, 03 Jul 2018 17:08:20 GMT

I do this all the time for the City I work for to "true-up" the land base layers. Here's two processes that seem to work well in identifying changes:

Detecting geometry changes. For both the existing polygons and the new polygons, convert the polygons to lines. Then run Erase on the line files to subtract the New from the Existing. Finally, run Multipart to Singlepart on the result. The output of all this will be lines that show where the geometry differs. This can be helpful in knowing exactly what needs to be edited, which can be an advantage over the "Select By Location, Identical" process (which is another way to do this).
Detecting Attribute changes. If you have a common unique ID, an Attribute Join can be used to combine the two attribute tables. Some examples: APN, TaxID. Obviously will vary depending on your attribute fields. Once the two datasets are joined, Queries can be run for each field of interest to see if the attribute has changed. Generalized example: Table1.APN <> Table2.APN.

Of course, one can do much more complex processing, but that will depend on knowing more of the specifics of the data.

Chris Donohue, GISP

Re: How to determine the changes between two polygon shapefiles of similar data

JosephCornellisson — Fri, 31 May 2019 18:19:05 GMT

Dan

Has any work been done on this polygon comparison tool. I have a similar situation trying to compare two parcel data sets that have changed over time

thanks

Joe

Re: How to determine the changes between two polygon shapefiles of similar data

DanLee — Sat, 01 Jun 2019 19:17:56 GMT

Hi Joe,

Thank you for checking. The short answer is no, we have not developed this tool/option yet. Part of the reasons is that we are still researching on use cases, not only parcel polygons but also other features, in order to fully understand the spectrum of scenarios and to see what's in common for a tool design.

Would you be able to share your parcel data with me? Could you describe in what way the two data sets are different, e.g. are they mostly identical except the changed features; do they have common attributes/IDs, such as APN?

Thanks,

Dan