We are in the process of migrating the environment and workflows to branched versioning and need to convert several of our scripts that update 20, 50, 180,000+ records nightly to use the services workflow. I see posts talking about unregistering branched and performing edits via sql, but this workflow is not feasible to do every night.
I currently have the script to break up the 180k features into chunks of ~5,000 for each processor (16 of them) to work on. When I do an edit or adds (a single .edit() request containing the x number of features the process worked on), it times out and kills the whole script. The components in the environment are barely moving the cpu/memory needle and staying < 40% use. If I submit each feature as they are calculated to the edit, (even though the web service should be able to handle and queue multiple requests) the sql db will throw an occasional error:
Error: DBMS deadlock victim [[Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Transaction (Process ID 89) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.]
Does this get retried, or terminate the request?
One version of the mp workflow is to calculate all of the features, and as they are returned from the process, submit the result using one version. This bottlenecks the script and slows the speed from doing 8-10 features a second (via arcpy in the old script) to only doing 1-2 a second using arcgis. Another version is to have each process create its own version to work on (can't be shared due to locks). This results in 1 to 100+ versions to reconcile. This seems to be the only way I can get each process working on data without erroring from locks from the version manager- but if one fails, it takes the whole script with it.
What are the workflows, examples, documentation for performing edits like this? All workflows seem to assume that we only edit a handful of features and the process does not scale.
