Load large sewer datasets into the UN

[TobeyKane-Seitz]

Hi all,

Has anyone run into issues loading large sewer datasets into the UN? I have 250,000 line features in an asset package that I am migrating into the UN. My goal is to migrate these features and to see that the Line - Snap to Manhole attribute rule is working properly, meaning that the pipe connections are generated with associations, and the line is offset from the manhole features. The sewer device and structure junctions were previously loaded into the database.

For now, I am loading the data into a fgdb UN.

I have tried the following

• Applying the asset package with data: this loads the line features, but the attribute rule does not fire. The lines are still connected to the manhole features
• Appending the data in small subsets: I have tried this in modelbuilder and just using the gp tool. It is extremely slow and seems to crash if I select more than 5,000 features at a time.

My network topology is disabled, and I've also disabled all unnecessary attribute rules to try to speed up the process. My source and target databases are in the same folder on my local machine.

Any help is appreciated!

[JohnAlsup]

When running Apply Asset Package where the UN properties are being applied and data is being loaded in a single execution, Attribute Rules do not fire, we do this for performance reasons.

I would suggest first using Apply Asset Package and not checking the Load Data option so that your UN properties are applied to the target UN and then re-running Apply Asset Package with the Load Data option checked against the same UN.  

However, a more scalable approach is to manipulate the data before loading in to the UN.  Relying on the Attribute rules will be very slow for loading that volume of data.

[TobeyKane-Seitz]

Thank you! As an update: I ran the apply asset package a first time to apply the UN properties, and then re-ran apply asset package with the load data option selected. The data loaded (very slowly, as expected) but the Line - Snap to Manhole rule did not fire. 

I'm relying on this attribute rule to take advantage of the manhole configuration in the UN. I'm wondering if there's a better way to run this rule after loading the data? 

[MikeMillerGIS]

Attribute rules are not added to the resulting layers until after append when your are applying the asset package.  This is as designed.  The sewer data model includes batch versions of those rules that can be rule after append.  If you want attribute rules to fire during the apply of the asset package, there is a trick you can use.

In ArcGIS Pro, open the python window

enter the following

import untools
untools.common.settings.RULES_BEFORE_APPEND = True

[TobeyKane-Seitz]

How do you recommend running the batch rules after applying the asset package? When I enable the batch rules and run the evaluate rules gp tool, I'm not seeing the expected results. 

[MikeMillerGIS]

My guess is the order of operations.  The lines probably were appended first, then the structure junctions.  You will need to adjust the append order.  The only way I know if doing that is using the AP Workspace.  You can set the untools to do a dry run mode, which only creates the ap workspace.  Then you can change the order of append in those files and run the main.py in a python command prompt.  

import untools
untools.common.settings.AP_DRY_RUN = True

As to why the batch rules did not work, I have no idea.  @AlexKabak you have any thoughts?