Programmatically append data to existing utility network and establish connectivity

[JoelMa923]

Our client regularly receives CAD drawings of their water network, wonder if it is possible to programmatically import the features into their existing utility network running on ArcGIS Enterprise? 

Due to their firewall settings, we can only access the utility network via feature services (no direct DB connection), so we cannot use some of the GP tools such as the Import Association tool.  Using feature services, can the line and point features be imported into the existing utility network, and also establish all necessary connectivity such as associations, terminal connections programmatically?

[MikeMillerGIS]

Yes this is possible.

Features - You can use the Append GP tool, Insert Cursor or the Python API.  This will make the appropriate Apply Edits calls.  You might have to use the python API though, if you are defining the Global ID on the data and want to preserve them when you are inserting the records.  The UseGlobalIDs value needs to be set to True - https://developers.arcgis.com/rest/services-reference/enterprise/apply-edits-feature-service-.htm#GU...

If you want to load into a version, it gets a little more complicated, but can be done.  

Terminals Connections - These are just attributes on lines, nothing special to do to set these.

Associations - Through services, you will add associations via the association table and the apply edits capability.  You will insert rows just like they are features.  This table is hidden with you look at the rest page, but it is at layer id 500001.  To find all the UN hidden tables, you need to look at the json of the Utility Network layer in the feature service.

Controllers -  Same answer as associations, but 500001

I had this code laying around to delete features, figured you might find it useful and you can repurpose for the inserts.  This shows two different ways to edit feature services the most efficient way.

 def _delete_service_rows(self, layer):
         """ arcpy.DeleteRows against a feature service runs significantly faster inside an edit session, but at 2.9.x
         sends chunks of 1000, to overcome this, we can use an update cursor and edit session to control the chunk
         size """
        chunk_size = 250
        d = arcpy.Describe(layer)
        fs = os.path.dirname(d.catalogPath)  # cannot lru_cache a Layer object
        arcpy.SetProgressor(type='STEP', message=f'Removing data on {d.aliasName} - 0/{self.layer_count}', min_range=0, max_range=self.layer_count)
        with arcpy.da.UpdateCursor(layer, ['OID@']) as ucurs:
            edit = arcpy.da.Editor(fs)
            edit.startEditing(False, False)
            edit.startOperation()
            for i, row in enumerate(ucurs, 1):
                ucurs.deleteRow()
                if i % chunk_size == 0:
                    logger.debug(f'{i}: {datetime.datetime.now()}')
                    edit.stopOperation()
                    edit.stopEditing(True)
                    arcpy.SetProgressorLabel(f'Removing data on {d.aliasName} - {i}/{self.layer_count}')
                    arcpy.SetProgressorPosition(i)
                    edit.startEditing(False, False)
                    edit.startOperation()
            edit.stopOperation()
            edit.stopEditing(True)

    def _delete_service_rows_API(self, layer):
        """ using the python api to delete feature removes a series of queries and creates an optimized delete payload """
        from arcgis.features import FeatureLayer
        from arcgis.gis import GIS
        import arcpy
        import datetime
        chunk_size = 250
        import itertools
        gis = GIS("pro")
        def chunks(iterable, size):
            it = iter(iterable)
            chunk = list(itertools.islice(it, size))
            while chunk:
                yield chunk
                chunk = list(itertools.islice(it, size))
        d = arcpy.Describe(layer)
        with arcpy.da.SearchCursor(layer, ['OID@']) as curs:
            oids = list(curs)
        fl = FeatureLayer(d.catalogPath, gis=gis)
        arcpy.SetProgressor(type='STEP', message=f'Removing data on {d.aliasName} - 0/{self.layer_count}', min_range=0,
                            max_range=self.layer_count)
        for i, chunk_oids in enumerate(chunks(oids, chunk_size), 1):
            arcpy.SetProgressorLabel(f'Removing data on {d.aliasName} - {i * chunk_size}/{self.layer_count}')
            fl.edit_features(deletes=','.join([str(oid[0]) for oid in chunk_oids]))
            logger.debug(f'{i}: {datetime.datetime.now()}')