topic Re: add time constraint to near tool in Python Questions

add time constraint to near tool

QiuhuaMa — Wed, 08 Apr 2015 15:30:32 GMT

Hi,

I am working on my dissertation right now and need to use Arcgis to calculate the distance from the nearest fire to each house.

Since fire is polygon and house is point, I just use near tool to calculate distance from the nearest fire to the house. However the fires I am interested in is the fire that burned within the past 7 years before the sale of the homes. So I need to exclude all fires that burned after the sale date of the home, and all fires that burned greater than seven years before the sale of the home.

Is there anyway to add a time constraint to near tool using python?

thanks,

chelsea

Re: add time constraint to near tool

JamesCrandall — Wed, 08 Apr 2015 15:37:45 GMT

Can you just apply a definition query on the fire layer that meets your requirements then run your tool? Or do you have a specific python script that needs modified to accomplish this?

Re: add time constraint to near tool

XanderBakker — Wed, 08 Apr 2015 17:23:19 GMT

I assume that the sale of each house can have a different date, right? In that case the definition query on the fires per house would be different. In that case you don't really need to use the Near tool, but you could use the geometries to determine the distance. Do you have a maximum search distance defined?

This would require a loop through the houses, determine the date of sale, define the where clause based on the date to use in the search cursor for the fire polygons, calculate the distance, keep track of the shortest distance and store this as a result probably as an attribute for the current house.

If that is what you are looking for, and maybe if you can share a small part of your data, we can create a script to get that result.

Re: add time constraint to near tool

QiuhuaMa — Wed, 08 Apr 2015 18:21:21 GMT

Xander, you are right! Each house has a different sale date. I didn't

define the maximum search distance.

What you said is exactly what I want to do! I tried to do this manually but

it took so much time!

I attached part of my house sale data and fire history data. Parcelid is

parcel id for each house. Ma_date_dt is sale date of each house.

For each fire, I have fire_day, fire_month and fire_year!

If a house is sold on 07/15/2013, I would like to find the nearest fire to

this house that burned between 2006 and 05/2013.

if a house is sold on 05/02/2011, I would like to find the nearest fire to

the house that burned between 2004 to 03/2011.

I would like to restrict the beginning time to year but the end time

to month/year.

thanks so much,

Chelsea

Re: add time constraint to near tool

QiuhuaMa — Wed, 08 Apr 2015 19:35:09 GMT

the end time of fire need to be 2 month earlier than house sale time since

most sale process took longer than 2 month.

Re: add time constraint to near tool

XanderBakker — Sun, 12 Dec 2021 05:45:22 GMT

OK, I had some fun with the data and this is what I obtained as result (houses drawn on distance to nearest fire):

Preparation:

I projected the Fire polygons to the same spatial reference as the houses

Change in the code below:

paths to input data on lines 7 and 8

Code:

import arcpy
from datetime import datetime, date, timedelta
from dateutil.relativedelta import relativedelta

def main():
    # input fc's
    fc_house = r"C:\Forum\DistFireHouse\shp\housesaledata.shp"
    fc_fire = r"C:\Forum\DistFireHouse\shp\firehistory_NAD_fips3002_feet.shp"

    # input field
    fld_date_sale = "ma_date_dt"

    # output fields
    fld_dist = "Dist2Fire"
    fld_fire_oid = "FireOID"

    # add fields
    addField(fc_house, fld_dist, "DOUBLE")
    addField(fc_house, fld_fire_oid, "LONG")

    # loop through houses
    cnt = 0
    flds = ("SHAPE@", fld_date_sale, fld_dist, fld_fire_oid)
    with arcpy.da.UpdateCursor(fc_house, flds) as curs:
        for row in curs:
            cnt += 1
            if cnt % 25 == 0:
                print "Processing house: {0}".format(cnt)
            pnt = row[0]
            date_sale = row[1]
            row[2], row[3] = getNearestFire(fc_fire, pnt, date_sale)
            curs.updateRow(row)


def addField(fc, fldname, fldtype):
    if len(arcpy.ListFields(fc, wild_card=fldname)) == 0:
        arcpy.AddField_management(fc, fldname, fldtype)

def getNearestFire(fc, pnt, date_sale):
    # input fields
    fld_year = "FIRE_YEAR"
    fld_month = "FIRE_MONTH"
    fld_day = "FIRE_DAY"

    # defaults
    min_dist = 999999
    oid_fire = -1

    flds = ("SHAPE@", "OID@", fld_year, fld_month, fld_day)
    with arcpy.da.SearchCursor(fc, flds) as curs:
        for row in curs:
            polygon = row[0]
            oid = row[1]
            year = int(row[2])
            try:
                month = int(row[3])
            except:
                month = 6
            try:
                day = int(row[4])
            except:
                day = 15

            if chkDate(date_sale, year, month, day):
                dist = pnt.distanceTo(polygon)
                if dist < min_dist:
                    min_dist = dist
                    oid_fire = oid

    return min_dist, oid_fire

def chkDate(date_sale, year_fire, month_fire, day_fire):
    date_fire = datetime(year=year_fire, month=month_fire, day=day_fire)
    date_2monthsbefore_sale = date_sale + relativedelta(months=-2)
    year_sale = date_sale.year
    test = (year_fire >= year_sale - 7) and (date_fire < date_2monthsbefore_sale)
##    if test:
##        print "date_sale: {0}".format(date_sale)
##        print "date_fire: {0}".format(date_fire)
##        print "date_2monthsbefore_sale: {0}".format(date_2monthsbefore_sale)
##        print "year_sale: {0}".format(year_sale)
##        print "{0}  <  {1}  <  {2}".format(year_sale-7, date_fire, date_2monthsbefore_sale)
    return test

if __name__ == '__main__':
    main()

Re: add time constraint to near tool

XanderBakker — Sun, 12 Dec 2021 05:45:25 GMT

Since it may be a little difficult to verify the result I threw in another piece of code to aid the verification process:

These are lines that connect each house with the nearest fire and add the date info to the lines for verification.

FireDate is the date of the fire
ma_date_dt is the date when the house was sold
YrSaleMn7 is the year of sale minus 7 years
SaleMin2m is the date two months before the house was sold
The FireOID can be used to join additional data from the fire polygons to the lines (use the FID of the fire polygons)
The parcelid provides the link to the houses.

What you need to do is change these lines in the code below:

reference the input shapefiles on line 10 and 11
define the output shapefile on line 23

Run the code below:

import arcpy
import os
from datetime import datetime, date, timedelta
from dateutil.relativedelta import relativedelta

def main():
    arcpy.env.overwriteOutput = True

    # input fc's
    fc_house = r"C:\Forum\DistFireHouse\shp\housesaledata.shp"
    fc_fire = r"C:\Forum\DistFireHouse\shp\firehistory_NAD_fips3002_feet.shp"


    # input field house
    fld_fire_oid = "FireOID"
    fld_parcel_id = "parcelid"
    fld_date_sale = "ma_date_dt"

    # input fields fires
    fld_oid_fire = arcpy.Describe(fc_fire).OIDFieldName

    # output fc
    fc_out = r"C:\Forum\DistFireHouse\shp\lines01.shp"

    # output fields (parcelid, double)
    fld_sale_min7 = "YrSaleMin7"
    fld_sale_2month = "SaleMin2m"
    fld_fire_date = "FireDate"
    fld_dist = "Dist2Fire"

    # create empty output fc
    sr = arcpy.Describe(fc_house).spatialReference
    shp_folder, shp_name = os.path.split(fc_out)
    arcpy.CreateFeatureclass_management(shp_folder, shp_name, "POLYLINE", spatial_reference=sr)

    # add output fields
    addField(fc_out, fld_parcel_id, "DOUBLE")
    addField(fc_out, fld_fire_oid, "LONG")
    addField(fc_out, fld_date_sale, "DATE")
    addField(fc_out, fld_fire_date, "DATE")
    addField(fc_out, fld_sale_min7, "INTEGER")
    addField(fc_out, fld_sale_2month, "DATE")
    addField(fc_out, fld_dist, "DOUBLE")

    # loop through houses
    cnt = 0
    flds_out = ("SHAPE@", fld_parcel_id, fld_fire_oid, fld_date_sale,
                fld_fire_date, fld_sale_min7, fld_sale_2month, fld_dist)
    flds_house = ("SHAPE@", fld_parcel_id, fld_date_sale, fld_fire_oid)
    with arcpy.da.InsertCursor(fc_out, flds_out) as curs_out:
        with arcpy.da.SearchCursor(fc_house, flds_house) as curs_in:
            for row_in in curs_in:
                cnt += 1
                if cnt % 25 == 0:
                    print "Processing connection: {0}".format(cnt)
                pnt1 = row_in[0]
                parcel_id = row_in[1]
                date_sale = row_in[2]
                fire_oid = row_in[3]

                # get fire details
                pnt2, date_fire = getDetailsFirePolygon(fc_fire, fld_oid_fire, fire_oid, pnt1)

                # create the lines
                line = arcpy.Polyline(arcpy.Array([pnt1.firstPoint, pnt2.firstPoint]), sr)

                date_2monthsbefore_sale = date_sale + relativedelta(months=-2)
                year_salemin7 = date_sale.year - 7

                row_out = (line, parcel_id, fire_oid, date_sale, date_fire,
                           year_salemin7, date_2monthsbefore_sale, line.length, )
                curs_out.insertRow(row_out)


def getDetailsFirePolygon(fc_fire, fld_oid_fire, fire_oid, pnt):
    fld_year = "FIRE_YEAR"
    fld_month = "FIRE_MONTH"
    fld_day = "FIRE_DAY"

    where = "{0} = {1}".format(arcpy.AddFieldDelimiters(fc_fire, fld_oid_fire), fire_oid)
    row = arcpy.da.SearchCursor(fc_fire, ("SHAPE@", fld_year, fld_month, fld_day), where_clause=where).next()
    polygon = row[0]
    year_fire = int(row[1])
    try:
        month_fire = int(row[2])
    except:
        month_fire = 6
    try:
        day_fire = int(row[3])
    except:
        day_fire = 15

    date_fire = datetime(year=year_fire, month=month_fire, day=day_fire)

    polyline = polygon.boundary()
    pnt2 = polyline.queryPointAndDistance(pnt, False)[0]

    del row
    return pnt2, date_fire

def addField(fc, fldname, fldtype):
    if len(arcpy.ListFields(fc, wild_card=fldname)) == 0:
        arcpy.AddField_management(fc, fldname, fldtype)

if __name__ == '__main__':
    main()

Add the result to ArcMap and define two relates on the lines featureclass:

one linking the lines to the houses using the parcelid fields
the other linking the lines to the fires using the Fireoid and FID fields

Re: add time constraint to near tool

QiuhuaMa — Thu, 09 Apr 2015 16:41:51 GMT

thanks so much, Xander!

I am a Econ student and don't have any programming skills. I tried read

your script but can only understand a few.

I will run your code and compare it to results that I got earlier to see

whether they are consistent.

Except for fire history in the past 7 years, I also would like to do fire

history in the past 10, 15 and 20 years.

It seems to me that I only need to change the code in the following 2 lines:

def chkDate(date_sale, year_fire, month_fire, day_fire):

date_fire = datetime(year=year_fire, month=month_fire, day=day_fire)

date_2monthsbefore_sale = date_sale + relativedelta(months=-2)

year_sale = date_sale.year

test = (year_fire >= year_sale - 7) and (date_fire <

date_2monthsbefore_sale)

if test:
print "date_sale:
".format(date_sale) ".format(date_fire)
print "date_2monthsbefore_sale:
".format(date_2monthsbefore_sale) ".format(year_sale)
print " < < ".format(year_sale-7, date_fire,

date_2monthsbefore_sale)

return test

if __name__ == '__main__':

main()

Am I right? Are there any other changes that I need to make?

thanks,

Chelsea

Re: add time constraint to near tool

XanderBakker — Thu, 09 Apr 2015 16:51:26 GMT

You are right. In fact the only change would be on line 76, where the number 7 should be replaced by the number of years you want to use. It is not necessary to change it on line 82 since it is switched of (commented), but if you uncomment the lines, you would have to adapt that number too.

BUT, also make sure that you point the result to different output fields or to a different copy of your input input houses. If you don't the new run will overwrite the data in the existing fields.

If you consider that an answers has been helpful you can mark an answer as such using the link "Helpful Yes | No" below each post.

Re: add time constraint to near tool

QiuhuaMa — Thu, 09 Apr 2015 17:14:52 GMT

thanks for the reminder about overwriting problem.

Actually I get confused starting at # loop through houses! Is this loop

still consistent with what you posted earlier? For each house, find all

fires in the past 7 years based on house sale date. Then calculate distance

of each fire to the house and store the shorted one!

Chelsea

Re: add time constraint to near tool

XanderBakker — Thu, 09 Apr 2015 18:54:54 GMT

If you are interested in understanding the code, I can give you some pointers, but this example may not be the best way of learning python. But, anyway, here goes:

Lines 1 to 3 do all the import outside the functions (the "def"s) so that all the functions can make use of them
The code actually starts at the very last line (86) calling the main def on line 5
As stated before lines 7 and 8 hold references to the input shapefiles
Line 11 has the name of the input date field in the house shapefile
Lines 14 and 15 have the names for the output fields to be added to input house shapefile (with the result of the analysis)
Lines 17 and 18 call a custom function addField (defined on line 35) which checks if the field already exists and if not (length of the list of fields with the provided name is 0) the field is added to the shapefile
Line 22 defines a counter variable cnt that will be incremented for each row in the house shapefile
- Lines 26-28 provide some information for the user on the progress (every 25th record it will write a message "Processing house: #" where # is the counter
Line 24 starts the update cursor on the houses and it take the list of fields defined on line 23
Line 25 will go through each row in the cursor
Line 29 reads the geometry, [0] refers to the first index of the list of field, in this case "SHAPE@"
Line 30 read the input date of sale of the house
Line 31 triggers the analysis. It calls a user defined function called "getNearestFire" defined on line 39 which will return two variables: distance and oid of fire. These will be written to the fields of the row (field [2] and [3]).
Line 39 the does the analysis of determining which fire is closest with based on a given date
Line 41-43 define the input fields that hold the date of the fire
Line 46 and 47 set some initial values for the minimum distance found and the corresponding oid
Line 49-51 start the cursor on the fires
It extracts the polygon, the oid and the fields for the date of fire
- note that the try, except lines (55-62) is for those cases that the month and day fields are empty
Next the chkDate function is called which is defined on line 72 and hold the logic for determining of the fire is in the date range that we are looking for
The chkDate function on line 72:
- creates a date object for the fire
- calculates the date two months before the house was sold
- defined the year of the sale
- and tests is the date is what we are looking for (returns True) or not (returns False)
On line 64 this result is used. If it is in the date range, the distance between the point and the polygon is calculated (this required the data to be in the same projection) on line 65
On line 66 it checks if the distance is less than the minimum distance, and if so it assigns it to the minimum distance and sets the oid to the oid_fire
On line 70 the minimum distance and oid of the corresponding fire is returned. which is read on line 31
On line 32, the row with the minimum distance and fire oid is stored.

That is basically it...

Kind regards, Xander

Re: add time constraint to near tool

XanderBakker — Fri, 10 Apr 2015 11:36:35 GMT

If you consider that a reply answered your question, please mark that reply as the correct answer. See: Mark a reply as the Correct Answer

Re: add time constraint to near tool

QiuhuaMa — Fri, 10 Apr 2015 21:15:33 GMT

I tried this code with another house sale data. This data has different

fields from the one I posted, but still has house id and sale date. I got

the following message:

Runtime error

Traceback (most recent call last):

File "", line 36, in addField

File "c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\__init__.py",

line 1075, in ListFields

return gp.listFields(dataset, wild_card, field_type)

File "c:\program files

(x86)\arcgis\desktop10.1\arcpy\arcpy\geoprocessing\_base.py", line 344, in

listFields

self._gp.ListFields(*gp_fixargs(args, True)))

IOError: "C:\Forum\DistFireHouse\shp\housesaledata.shp" does not exist

any idea what happened?

Chelsea

Re: add time constraint to near tool

QiuhuaMa — Fri, 10 Apr 2015 21:20:21 GMT

I just corrected this error: IOError:

"C:\Forum\DistFireHouse\shp\housesaledata.shp"

does not exist

but still got the message:

Runtime error

Traceback (most recent call last):

File "", line 74, in chkDate

File

"C:\Python27\ArcGIS10.1\lib\site-packages\dateutil\relativedelta.py", line

247, in __radd__

raise TypeError, "unsupported type for add operation"

TypeError: unsupported type for add operation

Re: add time constraint to near tool

XanderBakker — Fri, 10 Apr 2015 23:09:20 GMT

Glad to hear that you solved to error referring to the shapefile that was being referenced.

The other error on line 74 in the chkDate function refers to this line:

date_2monthsbefore_sale = date_sale + relativedelta(months=-2)

The only variable that could influence the error is the date_sale. Are you sure that the date field in the houses is of type Date? Could you check that all records have valid values?

Re: add time constraint to near tool

QiuhuaMa — Fri, 10 Apr 2015 23:20:57 GMT

Yeah, the problem is some houses don't have a valid sale date! After

deleting these data, it works.

thanks,

Chelsea

Re: add time constraint to near tool

XanderBakker — Fri, 10 Apr 2015 23:24:25 GMT

Glad it works. Better code would include methods to properly handle those cases...

Re: add time constraint to near tool

QiuhuaMa — Mon, 13 Apr 2015 20:15:42 GMT

Hi,

another way to examine the effect of fire on house price is to check number

and size of fire within a certain distance in the last 7 years.

I would like to count number of fires within 5km of a house in the past 7

years, then calculate their

average size!

Can this be done using python?

THANKS SO MUCH,

chelsea

2015年4月10日下午5:25于 "Xander Bakker" <geonet@esri.com>写道：

GeoNet
<http://jiveon.jivesoftware.com/wf/click?upn=Dg1s4x8le7Lmxv8KWGaqo8h7SGfRSMkw-2FpvHGF9-2FW3rK-2Bvs1kL9-2FnG6jjf2NZhrLDz0M-2BrY-2By9IaziQEKVk3Hg-3D-3D_cklnSfGWc0xtrUWz22hpfRmL-2BltIn2gVqEZxa2WpZ5WxnKfgSyyLU2hOb5I9HulWredg8bzBclHT-2Bssd3ileiOvLFQa-2FmBCtTaSqDGUHKu4XnPDXbl4Ut-2FMsay6tDfBzbVyLltMUwOlSIVVR-2BV5NMQcQfabq6bnsJ3h2e1w1qxNeHuvAFqXMl-2Fgn3Jq-2FWu3P1S-2FUEe4CHuJjCedZY-2BH76WvpBn6acAcWT4wYCAVAPnSBQ3MG2e01gVAgY-2FP54joxFx3SrBDvBPCJclBcl-2B3JuMZQtE3dW34Cm4Y7KaLDVIyvFkRba7LQTlzH6ZiELIuu> add
time constraint to near tool
reply from Xander Bakker
<http://jiveon.jivesoftware.com/wf/click?upn=Dg1s4x8le7Lmxv8KWGaqo7A4BXwO9PY1WvQ5cXCtK4-2FJIS3CK9JWidf-2FyI744iulp2nFUojYk8SjUF04OfNOXam6B1F7Qh0-2B9QYrDAA5RTM-3D_cklnSfGWc0xtrUWz22hpfRmL-2BltIn2gVqEZxa2WpZ5WxnKfgSyyLU2hOb5I9HulWredg8bzBclHT-2Bssd3ileiOvLFQa-2FmBCtTaSqDGUHKu4XnPDXbl4Ut-2FMsay6tDfBzYMfGzF0tzxGItU0XBAVRvIFK1Crre-2BQMAzijTGJvPtjjVYP9Y2zQdaiYRPZQVTXdLWrglYoGLlp7tj9o4J8kB32EMk-2BdykdvaaHciSVPu0coHhC9yF25LKcHvkBr1y2gAnZn7Nufsz3QlcJ-2Fmb-2BV9l9nGGoVsfujd2BJdCilzosqp8OjRNjyb4CuZDTW9M7-2F>
in Python - View the full discussion
<http://jiveon.jivesoftware.com/wf/click?upn=Dg1s4x8le7Lmxv8KWGaqo0VQjeJBYpG9HGC8QZBT-2FEx1rLkR8-2Fx-2FWdOrqbWRfYSj-2Bj-2F8IDaEgj29-2BmPV8A-2By0H03mtjLHTpxxgI2CO1cdwQ-3D_cklnSfGWc0xtrUWz22hpfRmL-2BltIn2gVqEZxa2WpZ5WxnKfgSyyLU2hOb5I9HulWredg8bzBclHT-2Bssd3ileiOvLFQa-2FmBCtTaSqDGUHKu4XnPDXbl4Ut-2FMsay6tDfBzmKBKWVY715nrdJfp-2FROf-2BiJjyGh0EpaBnitt8dOcfj8-2FIpvsZkweBZlgU8P46bbtoaUsygjnX-2Bj83goL3QHpLVW5CqZCUL-2Fv0eXXfNhrRuQ5TpEhowkCZPUIt9zUhX6lNio2ms1y3ULw3lXOtWDOOXPhz7pltNlXOR9jdHaCaabEZ8BTMt899G3wLZUdw7oA>

Re: add time constraint to near tool

XanderBakker — Mon, 13 Apr 2015 21:54:37 GMT

Yes, this can be done. The logic should go in the lines 65 - 70. Instead of checking the distance you could determine the number of fire, min, max and mean size and pass this back.

You would need to change to the way the result is read on line 31 and add the appropriate fields to store the results. Lines 13 to 23. It may be good to open a new discussion and add a link to this discussion, since this one is getting a little long...