UN version 7, ArcGIS Pro 3.5.4 validation errors - a showstopper?

[DavidSDouglas]

We're planning to upgrade in Production, and have done it in our Test environment. We're having big problems with validation  and can't risk an upgrade in production without a fix. Has anyone one else experienced similar issues? I'm asking before contacting support with the local ESRI partner - it's easier if some else can confirm this issue.

We upgraded from ArcGIS Pro 3.3 and ArcGIS Server 11.1     to 3.5.4 and 11.5
Also Utility Networks upgraded from version 5 to 7 (that took 20 hours)

.NET 8.0 is installed on servers (previously 6.0)

Some functions are running faster than before, faster even than the production environment which has not yet been upgraded - it has most resources. The servers are Windows Server 2022 Datacenter Azure Edition 21H2. The SQL Server database is Microsoft SQL Azure (RTM) - 12.0.2000.8

We have divided Denmark into 27 large polygons and validate these each night using a Python script:

GPextent= ('{} {} {} {}'.format(x1, y1, x2, y2))
arcpy.un.ValidateNetworkTopology(serviceUrlProd,GPextent)

After upgrading, this takes about 30 minutes to run, about half the previous time - also compared to production.

However, interactive validation in ArcGIS Pro using the Validate function on the Utility Network tab times out after 5 minutes even with the smallest extent (a few metres - which took only a few seconds before, and in production). If I attempt to validate again within the next few minutes I get:

Edit operation failed. Message: Unable to complete operation.
Details: Serialization lock in progress [[Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Lock TimeoutX] [Serialization lock in progress - operation not allowed]

Eventually the lock is released - and validation will time out again on the next attempt. This has not been seen in the current production version,even if validation timed out after using too large an extent.

Using Validate Network Topology (Utility Network Tools), these small extents do validate - but it takes 30 minutes!

During editing, no dirty area appears after altering attributes. They do however appear if a feature is deleted.

[RobertKrisher]

Have you opened a case with support yet?

[DavidSDouglas]

Not yet, but will do soon. In the last major support cases we have been asked to perform many time -consuming tests (creating new features, services, changing versioning, exporting to file geodatabase, recording video of screen etc, but there was no resolution of the the problem. That problem was that after running the Export to CAD tool, or Flip Line, ArcGIS Pro began to call a  feature service repeatedly for no evident reason, taking 20 - or sometimes even 200 seconds per call! The only solution was to exit and reenter ArcGIS. Then, when we upgraded to 3.5.4 in Test, we saw that the bug was no longer present. We can't have been the only customer to experience this, and I was hoping that by posting our current problem here, someone might recognise it (even though it's only a few weeks since 3.5.4 was released). 

[PierreloupDucroix]

Hello,

Did you update the geodatabase version in addition to the UN version (using the Geodatabase Update tool)?

This resolved a significant performance issue we were experiencing in ArcGIS Pro (related to version posting, not network validation).

This could be a lead to investigate further.

[DavidSDouglas]

Yes, updating to the latest geodatabase version was a requirement for upgrading, now that objectid is 64 bit

[Thomas_Brown]

David, 

Please provide the incident number. I'll make sure the issue is escalated and you will receive the proper support to narrow down this issue.

There are a few issues at play here... first to understand why your validates are so slow (which are likely leading to the serialization lock issue). Assumption is the validate you are performing is in a version?

Thank you.

Tom

[DavidSDouglas]

I installed the 3.5.5 update, which has some UN Validation  bug fixes - but that didn't help. However, I noticed we hadn't installed the ArcGIS Server 11.5 Feature Services Security Patch (from October) on the servers, so I did that, and it has helped the interactive validation problem - we can now validate, but only pretty small extents, otherwise it times out. We also are unable to run prune branch history - it failed after 36 hours working on UN_8_EID_MAPPINGS (no idea what that is, but there's 230 million rows in the database table). Almost all the other tables were pruned. We also tried to remove versioning and re-establish it, but there's still the same massive numbers of rows in UN_8_EID_MAPPINGS. We're now planning to start again with a fresh copy of the production database in Test, and document every step of the way. Can it validate using 3.3.0 before upgrading? I'm sure it can, but we will document it. Then we will try to remove versioning - if that doesn't succeed it's not a 3.5.5 / 11.5 issue. If it does, try validation again.  It may be faster to do the database upgrade again after removing versioning has massively lowered the number of rows in the tables, with added RAM and CPU just like in Production. That's so we can estimate how long it will take when we upgrade in Production (our control room is manned 24/7, so taking say 20 hours to upgrade the database is a major inconvenience).