topic Re: Rebuild Geocode Service in Python Questions

Rebuild Geocode Service

BlakeTerhune — Sun, 12 Dec 2021 07:01:31 GMT

Our organization hosts several geocode services from both regular and composite address locators. As we all should know, it is important to rebuild an address locator as changes are made to the reference data. Since the locator gets copied to the server, rebuilding the original locator is not sufficient so I've found the entire service needs to also be "rebuilt" (overwritten). We will create scheduled tasks that run a Python script to automate this process.

Having experienced the nightmare that occurs when an address locator gets "corrupted" and cannot be rebuilt, I have made the choice to simply recreate the locator each time instead of rebuilding it. The advantage of this method is that you aren't dead in the water should something tragic happen to the original locator because the specifications for creating it are in the script. Alternatively, you could just ensure the specifications of the locator are well documented in the script and just do a rebuild. Either way, the geocode service still needs to get republished with the new address locator reference data.

Here is a simplified process I use to rebuild a geocode service.

Create temporary directory where the address locator and service definition files will be staged. This is cleaned up automatically when the script ends.
Create address locator
Create and analyze service definition draft file for sharing the address locator
Modify Service Definition Draft file to overwrite the existing service.
Convert Service Definition Draft to a Service Definition file
Upload and publish service definition file as a service

To clarify, this was all tested on 10.2.2.

import arcpy
from contextlib import contextmanager
import datetime
import os
import shutil
import tempfile
import xml.dom.minidom as DOM  ## sddTypeReplacement()

def main():
    # Local variables
    agsconn = r"C:\test\MyAdminArcGISServerConnection.ags"
    sdeconn = r"C:\test\MySDEConnection.sde"
    my_fancy_source_data = os.path.join(sdeconn, "my_fancy_source_data")
    locator_name = "MY_FANCY_LOCATOR"

    with makeTempDir() as temp_dir:
        # Create address locator
        temp_locator = os.path.join(temp_dir, locator_name)
        field_info=(
            "'Feature ID' OBJECTID VISIBLE NONE;"
            "'*House Number' ADDR_NUM VISIBLE NONE;"
            "Side <None> VISIBLE NONE;"
            "'Prefix Direction' PRE_DIR VISIBLE NONE;"
            "'Prefix Type' <None> VISIBLE NONE;"
            "'*Street Name' NAME VISIBLE NONE;"
            "'Suffix Type' STREET_TYPE VISIBLE NONE;"
            "'Suffix Direction' SUF_DIR VISIBLE NONE;"
            "'City or Place' <None> VISIBLE NONE;"
            "'ZIP Code' <None> VISIBLE NONE;"
            "State <None> VISIBLE NONE;"
            "Longitude <None> VISIBLE NONE;"
            "Latitude <None> VISIBLE NONE;"
            "'Street ID' <None> VISIBLE NONE;"
            "'Min X value for extent' <None> VISIBLE NONE;"
            "'Max X value for extent' <None> VISIBLE NONE;"
            "'Min Y value for extent' <None> VISIBLE NONE;"
            "'Max Y value for extent' <None> VISIBLE NONE;"
            "'Additional Field' <None> VISIBLE NONE;"
            "'Altname JoinID' <None> VISIBLE NONE"
        )
        arcpy.CreateAddressLocator_geocoding(
            "US Address - Single House",  ## in_address_locator_style
            "'{}' 'Primary Table'".format(my_fancy_source_data),  ## in_reference_data
            field_info,  ## in_field_map
            temp_locator
        )

        # Create and analyze service definition draft file
        sddraft = os.path.join(temp_dir, locator_name+".sddraft")
        sdd_analyze_result = arcpy.CreateGeocodeSDDraft(
            temp_locator,  ## in_address_locator
            sddraft,  ## out_sddraft
            locator_name,  ## service_name
            "FROM_CONNECTION_FILE",  ##  server_type
            agsconn,  ## connection_file_path
            folder_name="Geocoders",
            summary="Created by Python script {}".format(
                datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
            )
        )

        # Inspect analyze result
        """result object of CreateGeocodeSDDraft is a dictionary of dictionaries structured like
            {
                'errors': {},
                'messages': {},
                'warnings': {
                    (u'Warning message', 9999): [],
                    (u'Warning message', 9999): []
                }
            }
        """
        ## Check for error messages to create the issues list.
        ## If there are no analyze errors, analyze_issues will be empty list []
        analyze_issues = sdd_analyze_result.get("errors").keys()
        ## Check for warning messages and add them to the issues list.
        ## Only include those that aren't in the ignore list.
        warn_ignore = [
            ('Missing Summary in Item Description', 97),
            ('Missing Tags in Item Description', 98),
            ('Locator will be copied to the server', 24044)
        ]
        analyze_issues += [
            warn for warn in sdd_analyze_result.get("warnings").keys()
            if warn not in warn_ignore
        ]

        if analyze_issues:
            raise Exception(
                "Analyzing the service definition draft revealed an issue.\n"
                "{}".format(analyze_issues)
            )
        else:
            # Modify Service Definition Draft file to overwrite the existing service
            sddTypeReplacement(sddraft)

            # Convert Service Definition Draft to a Service Definition file.
            ## Staging compiles all the necessary information needed to publish
            ## the GIS resource. If your data is not registered with the server,
            ## the data will be added when the Service Definition Draft file is staged.
            service_definition = os.path.join(temp_dir, locator_name+".sd")
            arcpy.StageService_server(sddraft, service_definition)

            # Upload and publish service definition file as a service
            arcpy.server.UploadServiceDefinition(service_definition, agsconn)
            return arcpy.GetMessages()
        ## end if analyze_issues
    ## end with temp_dir


def sddTypeReplacement(service_definition_draft):
    """Modifies a service definition draft file to overwrite an existing service
        instead of the defaut to create a new service.
        This function based on Esri example code for arcpy.UploadServiceDefinition_server()

        Requires module xml.dom.minidom as DOM
        Takes string input parameter of the path to a service definition draft file.

        I did test overwriting a service without this function and it seems to
        work fine. However, I kept it because Esri took the effort to post the
        code example in the documentation and there must be a good reason.
        Futher investigation may be helpful in understanding this.
    """
    xml_doc = DOM.parse(service_definition_draft)
    descriptions = xml_doc.getElementsByTagName("Type")
    for desc in descriptions:
        if desc.parentNode.tagName == "SVCManifest":
            if desc.hasChildNodes():
                desc.firstChild.data = "esriServiceDefinitionType_Replacement"
    with open(service_definition_draft, 'w') as f:
        xml_doc.writexml(f)


@contextmanager
def makeTempDir():
    """Creates a temporary folder and returns the full path name.
    Use in with statement to delete the folder and all contents on exit.
    Requires contextlib contextmanager, shutil, and tempfile modules.
    """
    temp_dir = tempfile.mkdtemp()
    try:
        yield temp_dir
    finally:
        shutil.rmtree(temp_dir)


# Script start
if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I will be also adapting this same process for rebuilding geocode services with a composite locator. As always, suggestions are welcome!

P.S.

When I got this vague error, I found it was caused by incorrect field names in the field info for the in_field_map parameter of arcpy.CreateAddressLocator_geocoding()

ERROR 000042: Failed to create the address locator.
Underlying DBMS error [ORA-00923: FROM keyword not found where expected]

Re: Rebuild Geocode Service

StefanChapkanski1 — Tue, 04 Jul 2017 16:32:41 GMT

Hello @Blake Terhune, I would like to share the solution that works for me.

So, I needed to publish composite address locator on AGS as a geocoding service and use it in AGOL as asset locator. My composite locator is created from several single field locators. I am doing this on AGS machine directly and I am using ArcMap. All versions are 10.3. The data source I am using to construct those locators is updated daily. To keep the search in AGOL up to date based on this service I must update it daily. The only solution for me was to find automated way. No manual operations. No custom code.

My solution is:

Create all the single field locators and combine them in composite one.
Publish the composite locator as geocoding service on AGS. AGS copy all the data.
At that point, my original locators become useless for me. So, I continued my work on a model in model builder with the locators copied on the server (Windows 2012 R2). The location you can find them is C:\arcgisserver\directories\arcgissystem\arcgisinput\<AGS Folder name>\<LocatorName.GeocodeServer>\extracted\v101. In the root of the folder I have the composite locator LOC and XML files. The folder contains also sub folders called 00, 01, 02 and so on depending how many simple locators participating in the composite one. In every one of those sub folders are located the simple locators’ files – LOC, XML and LOX.
Download ArcGIS Server Administration Toolkit - 10.1+ containing tool that allows me to stop or start a service within a model.
Create toolbox with 2 models inside. The first one is for stopping the geocoding service. The next one looks like this and it is intended to rebuild the simple locators and the to start the service. Note that I am not rebuilding the composite locator, because it is only a reference to the simple ones. I should mention some additional steps I did before compiling my models. I went to the AGS system directory (through ArcMap) checked the locators, and manually run rebuild function just to be sure everything is OK. I needed to make a change of the reference data location by pointing to the original GDB (registered in AGS).
Export those 2 models as python scripts.
Create a new task in Windows Task Scheduler that runs those 2 scripts daily.

Let me know if this approach works for you as well.

Re: Rebuild Geocode Service

JoeBorgione — Wed, 05 Jul 2017 00:26:31 GMT

A couple questions: In 10.3.1, I've never been able to rebuild a locator, and it was brought to my attention that it's a bug. I'm curious that model builder can do it for you; how is that? Also, if you are essentially over-writing a locator service, why do you need to stop and start it? I notice your fgdb is registered with AGS; if you are copying data to AGS, what is the advantage to having it registered?

I use model builder to to re-create locators (not rebuild) and then I re-create the composite they belong to, followed by manually publishing it, copying all data. None of my data sources are registered with AGS.

Stefan.Chapkanski‌

Re: Rebuild Geocode Service

StefanChapkanski1 — Wed, 05 Jul 2017 06:24:51 GMT

Joe... I've never experienced this bug, but as far I can find info about it Esri suggestion is to use Rebuild Address Locator gp tool. When you overwrite an existing service restarting is handled by AGS. Basically if your data source is not registered within AGS and you are trying to publish service based on that data then AGS needs to be able to use it. What happens then is described in here - Copying data to the server automatically when publishing. The advantage of having your DB and/or folders registered is that you are allowing AGS to have an access. In the other scenario AGS still needs that data in order to run your services and then copy all the necessary data. In other words you can update your registered data and the changes will be available in the service. More about it you can find here.

Re: Rebuild Geocode Service

BlakeTerhune — Wed, 05 Jul 2017 18:50:36 GMT

Stefan Chapkanski wrote:
Note that I am not rebuilding the composite locator, because it is only a reference to the simple ones. I should mention some additional steps I did before compiling my models. I went to the AGS system directory (through ArcMap) checked the locators, and manually run rebuild function just to be sure everything is OK. I needed to make a change of the reference data location by pointing to the original GDB (registered in AGS).

Apologies if I'm misunderstanding your process, but I don't think you can simply rebuild the underlying locators of a composite locator. You have to recreate the composite locator each time a participating locator is rebuilt.

Rebuild Address Locator—Help | ArcGIS Desktop

Composite address locators cannot be rebuilt using this tool. The best practice for managing composite locators is to create a composite locator using the Create Composite Address Locator tool in a geoprocessing model. In ModelBuilder, you can also chain the tool with the Create Address Locator tool to use its outputs as the inputs to the composite locator and manage the entire process of creating or updating a composite address locator.

Re: Rebuild Geocode Service

StefanChapkanski1 — Thu, 06 Jul 2017 06:43:20 GMT

Why don't you give it a try?

Creating a composite address locator—Help | ArcGIS for Desktop

The composite address locator stores only references to the participating address locators and geocode services; it does not contain the actual address information, indexes, and data of the individual locators.