topic Re: Combining polygons by date fields in Geoprocessing Questions

Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Tue, 12 Aug 2014 14:54:57 GMT

I have an interesting problem. I have been asked to create a set of polygons using dates. Below is an example table showing the data. As you can see the table is sorted by start year.

What I have to do is generate a set of polygons each time the start year, month and day changes. So the first polygon will just contain the first row of data. The second polygon will contain both the first and second and so on.

There is also another element to this. Each row of data has an end year, month and day. If a row has an end year of 2300 then that row will stay in the output once it has been added. If there is an end year that does not equal 2300 then that row will not be included in future versions of the output.

So is there anyone who can help? I have about two hundred tables like the one above. Would like a model builder solution.

Thanks in anticipation.

Paul

Re: Combining polygons by date fields

XanderBakker — Tue, 12 Aug 2014 18:43:52 GMT

There is a point I don't really get; when the end date of a record (current date you want to process) is not equal to 2300, BUT it is higher than the current date, should it be removed from the output?

It seems more logic to me if for each unique date in the StartDate you use all the records for which the start date is before the current date and the end date is after the current date and use those records to create the polygon.

Anyway, I would probably use some Python code to:

create a unique sorted list of start dates
create a layer through the select by attributes
dissolve the polygons and create a new featureclass (or add it to an existing one if you want the new merged polygons to be stored in 1 featureclass)

Then there is still the issue of what to do with the attributes of the records you're merging...

If you're willing to post a piece of your data, I can have a look and see what code is needed.

Kind regards, Xander

Re: Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Wed, 13 Aug 2014 10:21:52 GMT

Thanks for the reply Xander,

To answer your first question if the end year is not equal to 2300 but is higher than the current date, then it should be removed when it's end date is less than any of the start dates.

Could you tell me how I upload a file so you can get to it?

I can discuss the attributes once you have the file.

Thanks

Paul

Re: Combining polygons by date fields

XanderBakker — Wed, 13 Aug 2014 15:32:02 GMT

ZIP the extract of your data (shapefile or FGBD), and use the "reply" option in this thread. In the editor in the upper right corner you will see a link "Use advanced editor", once in the advanced editor you will see in the lower right corner the possibility to attach a file.

Re: Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Wed, 13 Aug 2014 15:42:59 GMT

Hi Xander,

Here is the zip file containing the shapefile.

Many thanks

Paul

Re: Combining polygons by date fields

XanderBakker — Wed, 13 Aug 2014 18:58:01 GMT

Hi Paul,

I have included some Python code that created a new shapefile with the merged polygons. The attributes are still to be defined. In the code the processing date is assigned the the output date fields.

Prerequisites:

Create two new columns that hold the start and end date as date (not as text).
- You can do this by adding the fields "DateFrom" and "DateTo" as date fields
- Calculate the new fields and simply assign the field StartDate and EndDate to it

The reason to do this is to simplify the syntax of the where clause used in the script.

Configuration:

Change line 6 to point to your input shapefile
Change line 7 and 8 to hold the field names of the date fields (the newly added fields)
Change line 11 to point to a shapefile that will be created

The script will:

Create an empty output shapefile
Create a list of unique dates from the start date field
Loop through each date and create an expression (see lines 29 - 31). It will select all polygons with a start date less than of equal to the current date and where the end dates are after the current date
On lines 35 - 42 the polygons that match the query will be merged into a single (multipart) polygon
On line 46 the polygon is written to the output shapefile

import arcpy, os
from datetime import datetime
arcpy.env.overwriteOutput = True

# change path to datasource
fc = r"D:\Xander\GeoNet\MergePolsDate\Reserves_copy.shp"
fld_start = "DateFrom" # create date field and fill it with content of StartDate
fld_end = "DateTo" # create date field and fill it with content of EndDate

# output shape
fc_out = r"D:\Xander\GeoNet\MergePolsDate\Reserves_merged.shp"

# create empty output shapefile
ws_path, fc_out_name = os.path.split(fc_out)
arcpy.CreateFeatureclass_management(ws_path, fc_out_name,
    "POLYGON", fc, "SAME_AS_TEMPLATE", "SAME_AS_TEMPLATE", fc)

flds = ("SHAPE@", fld_start, fld_end)

# create a unique list of dates from start date
lst_dates = list(set([row[0] for row in arcpy.da.SearchCursor(fc, (fld_start))]))
lst_dates.sort()

# insert cursor to store features to new featureclass
with arcpy.da.InsertCursor(fc_out, (flds)) as curs_out:
    # loop through list of unique dates
    for date in lst_dates:
        # create an expression
        expression = "{0} <= date '{2}' AND {1} >= date '{2}'".format(
            arcpy.AddFieldDelimiters(fc, fld_start),
            arcpy.AddFieldDelimiters(fc, fld_end), date)
        i = 0
        # create the searchcursor with the expression
        with arcpy.da.SearchCursor(fc, flds, where_clause=expression) as curs:
            for row in curs:
                pol = row[0]
                if i == 0:
                    # first polygon is just the polygon"
                    polygon = pol
                else:
                    # next polygons are added with union
                    polygon = polygon.union(pol)
                i += 1
            # insert the feature into the output featureclass
            # TODO: manage attributes
            curs_out.insertRow((polygon, row[1], row[1]))

In case of any doubts, just post a reply.

Kind regards, Xander

Re: Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Thu, 14 Aug 2014 14:12:20 GMT

Hi Xander,

That worked.

Just need to include the name and date attributes in the output.

What is being asked of me is to combine polygons into financial years, please see the example below.

Hope this makes sense.

Thank you for your time on this.

Paul

Re: Combining polygons by date fields

XanderBakker — Thu, 14 Aug 2014 14:52:41 GMT

Hi Paul,

From your example I don't really understand the concept of a "financial year". To fill the EndDate columns with the value 31/21/2300 doesn't seem right, since polygons that expire before that date, but have an end date after the current processing date, are included too.

It is possible to add the name and create a new column "OutputName" that holds the name and a date range. What is not very clear to me is this:

For the first record in your Excel it seems that the current processing date is "31/03/1965". In the code I provided the first processing date will be "01/04/1962" which will create a name "Reserve A 01/04/1962 - 01/04/1962".
For the next record it will include those polygons for which the start date is before or on the current processing date ("07/06/1965") and the end date is on or after the current date. This will include also the first record, creating an overlap in range and an OutputName "Reserve A 01/04/1962 - 07/06/1965"

Since the first record in the sample shapefile has an end date of 31/21/2300 it will be included in all the features.

Maybe you can explain a little more on what you are trying to achieve?

Re: Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Thu, 14 Aug 2014 15:26:56 GMT

Hi Xander,

Apologies for this. Don't worry about the financial year, I can sort that out later.

Have a look at this example

So the first version of the reserve is live between 01/04/1962 until 06/06/1965 as on the 07/06/1965 the reserve then includes both row 2 and 3 and so on.

Hope that makes it easier.

Again thank you for your patience.

Paul

Re: Combining polygons by date fields

XanderBakker — Thu, 14 Aug 2014 16:17:12 GMT

Hi Paul,

I think you could try something like this:

import arcpy, os
from datetime import datetime, timedelta
arcpy.env.overwriteOutput = True

# change path to datasource
fc = r"D:\Xander\GeoNet\MergePolsDate\Reserves_copy.shp"
fld_start = "DateFrom" # create date field and fill it with content of StartDate
fld_end = "DateTo" # create date field and fill it with content of EndDate
fld_name = "Name"
fld_outname = "OutputName" # output name
len_outname = 75

# output shape
fc_out = r"D:\Xander\GeoNet\MergePolsDate\Reserves_merged3.shp"

# create empty output shapefile
ws_path, fc_out_name = os.path.split(fc_out)
arcpy.CreateFeatureclass_management(ws_path, fc_out_name,
    "POLYGON", fc, "SAME_AS_TEMPLATE", "SAME_AS_TEMPLATE", fc)

# add string field
arcpy.AddField_management(fc_out, fld_outname, "TEXT", "", "", len_outname)

flds = ("SHAPE@", fld_start, fld_end, fld_name)
flds_out = ("SHAPE@", fld_start, fld_name, fld_outname)

# create a unique list of dates from start date
lst_dates = list(set([row[0] for row in arcpy.da.SearchCursor(fc, (fld_start))]))
lst_dates.sort()

# insert cursor to store features to new featureclass
with arcpy.da.InsertCursor(fc_out, (flds_out)) as curs_out:
    # loop through list of unique dates
    for date in lst_dates:
        # create an expression
        expression = "{0} <= date '{2}' AND {1} >= date '{2}'".format(
            arcpy.AddFieldDelimiters(fc, fld_start),
            arcpy.AddFieldDelimiters(fc, fld_end), date)
        i = 0
        # create the searchcursor with the expression
        with arcpy.da.SearchCursor(fc, flds, where_clause=expression) as curs:
            for row in curs:
                pol = row[0]
                name = row[3] # take name of first polygon
                if i == 0:
                    # first polygon is just the polygon"
                    polygon = pol
                else:
                    # next polygons are added with union
                    polygon = polygon.union(pol)
                i += 1

            # insert the feature into the output featureclass
            datefrom = date
            i_date = lst_dates.index(date)
            if i_date + 1 > len(lst_dates) - 1:
                dateto = ""
                outname = "{0} {1} - {2}".format(name, datefrom.strftime('%Y/%m/%d'), dateto)
            else:
                dateto = lst_dates[i_date + 1] - timedelta(days=1)
                outname = "{0} {1} - {2}".format(name, datefrom.strftime('%Y/%m/%d'), dateto.strftime('%Y/%m/%d'))
            curs_out.insertRow((polygon, row[1], name, outname))

The code will create the output column with the format from your example

Kind regards, Xander

PS: If the answer responds you question mark the post as answered at the post that answered your question. If it was helpful, mark it as helpful. This way other forum users will be able to find useful content. More on GeoNet can be found here: GeoNet Help‌

Re: Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Fri, 15 Aug 2014 09:34:51 GMT

Hi Xander,

Thank you so much. It now does what I want it to do. You are a star.

Many thanks

Paul

Re: Combining polygons by date fields

DanPatterson_Retired — Fri, 15 Aug 2014 09:42:48 GMT

Please then, mark one of Xander's answers as correct...currently the thread is still open

Re: Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Fri, 15 Aug 2014 10:23:04 GMT

Hi Dan,

Sorry, how do I do that?

Thanks

Paul

Re: Combining polygons by date fields

DanPatterson_Retired — Fri, 15 Aug 2014 10:39:23 GMT

for each thread there should be a greenish star with test beside it saying "correct answer" click on it for the thread that answered your question and NOT this one

Re: Combining polygons by date fields

XanderBakker — Fri, 15 Aug 2014 12:45:32 GMT

I'm glad it worked for you. Good luck!

Re: Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Mon, 18 Aug 2014 15:19:44 GMT

Hi Xander,

Just one more thing. How can I use your script to run against multiple input shapefiles?

Thanks

Paul

Re: Combining polygons by date fields

XanderBakker — Mon, 18 Aug 2014 18:00:58 GMT

Hi Paul,

That is possible, but it might be necessary to know a little more on the folder structure where you have your input featureclasses and how you want to name the outputs.

Normally a arcpy.ListFeatureclasses() can be used to get a list of featureclasses in the current workspace. In case they are in multiple folders and sub folders the arcpy.da.Walk() is better. Based on the input name, you can create an outputname (like for instance inputname_merged).

The current script can be rewritten as a function to process a single featureclass. This function will be called from the loop through the input featureclasses.

Kind regards, Xander

Re: Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Tue, 19 Aug 2014 09:16:42 GMT

Hi Xander,

Thanks for the reply.

My shapefiles are currently in this folder C:\Users\PaulBritten\Desktop\tenure\All reserves

and the out put name would be as you suggest with the _merged at the end of the name.

I will admit that I am not a python expert at all and have no idea how to amend/update the python script with any new functionality. Therefore would you mind amending your script?

Many thanks and sorry to be such a pain.

Paul

Re: Combining polygons by date fields

XanderBakker — Tue, 19 Aug 2014 18:53:58 GMT

Hi Paul,

Don't worry, that what the forum is all about... to help one another.

Have a look at the code below. The majority is placed inside a function "MergePolygonsOnDate". The main function (lines 13 - 23) contains the logic for looping through your featureclasses. It will create a list of all the featureclasses in you workspace. Based on the input name it will create the new output names. On line 23 for each featureclass the "MergePolygonsOnDate" is called. I haven't tested the code, so if you run into problems, just post it and I'll have a look.

#-------------------------------------------------------------------------------
# Name:        MergePolygonsOnDate_v3.py
# Purpose:     Merge polygons on date range
#
# Author:      xbakker
#
# Created:     19/08/2014
#-------------------------------------------------------------------------------

import arcpy, os
from datetime import datetime, timedelta

def main():
    # specify the workspace
    ws = r"C:\Users\PaulBritten\Desktop\tenure\All reserves"
    arcpy.env.workspace = ws
    arcpy.env.overwriteOutput = True

    lst_fc = arcpy.ListFeatureClasses()
    for fc in lst_fc:
        name, ext = os.path.splitext(fc)
        fc_out = "{0}_merged{1}".format(name, ext)
        MergePolygonsOnDate(os.path.join(ws, fc), os.path.join(ws, fc_out))


def MergePolygonsOnDate(fc, fc_out):

    # change settings
    fld_start = "DateFrom" # create date field and fill it with content of StartDate
    fld_end = "DateTo" # create date field and fill it with content of EndDate
    fld_name = "Name"
    fld_outname = "OutputName" # output name
    len_outname = 75

    # create empty output shapefile
    ws_path, fc_out_name = os.path.split(fc_out)
    arcpy.CreateFeatureclass_management(ws_path, fc_out_name,
        "POLYGON", fc, "SAME_AS_TEMPLATE", "SAME_AS_TEMPLATE", fc)

    # add string field
    arcpy.AddField_management(fc_out, fld_outname, "TEXT", "", "", len_outname)

    flds = ("SHAPE@", fld_start, fld_end, fld_name)
    flds_out = ("SHAPE@", fld_start, fld_name, fld_outname)

    # create a unique list of dates from start date
    lst_dates = list(set([row[0] for row in arcpy.da.SearchCursor(fc, (fld_start))]))
    lst_dates.sort()

    # insert cursor to store features to new featureclass
    with arcpy.da.InsertCursor(fc_out, (flds_out)) as curs_out:
        # loop through list of unique dates
        for date in lst_dates:
            # create an expression
            expression = "{0} <= date '{2}' AND {1} >= date '{2}'".format(
                arcpy.AddFieldDelimiters(fc, fld_start),
                arcpy.AddFieldDelimiters(fc, fld_end), date)
            i = 0
            # create the searchcursor with the expression
            with arcpy.da.SearchCursor(fc, flds, where_clause=expression) as curs:
                for row in curs:
                    pol = row[0]
                    name = row[3] # take name of first polygon
                    if i == 0:
                        # first polygon is just the polygon"
                        polygon = pol
                    else:
                        # next polygons are added with union
                        polygon = polygon.union(pol)
                    i += 1

                # insert the feature into the output featureclass
                datefrom = date
                i_date = lst_dates.index(date)
                if i_date + 1 > len(lst_dates) - 1:
                    dateto = ""
                    outname = "{0} {1} - {2}".format(name, datefrom.strftime('%Y/%m/%d'), dateto)
                else:
                    dateto = lst_dates[i_date + 1] - timedelta(days=1)
                    outname = "{0} {1} - {2}".format(name, datefrom.strftime('%Y/%m/%d'), dateto.strftime('%Y/%m/%d'))
                curs_out.insertRow((polygon, row[1], name, outname))


if __name__ == '__main__':
    main()

Re: Combining polygons by date fields

deleted-user-_78mlgGtbbzB — Wed, 20 Aug 2014 11:13:11 GMT

Hi Xander,

That worked brilliantly.

Many thanks

Paul