Benchmark GP tool execution times against previous versions

[Bud]

Edited.

I came across an issue in ArcGIS Pro 2.9.12 where the Make Route Event Layer geoprocessing tool was much slower than previous versions (2.6.8).

Route Event Layers — Slow performance in ArcGIS Pro 2.9.12, but not in 2.6.8

That performance issue is a significant problem for my unit. We can't upgrade from 2.6.8 to 2.9.12 (or 3.2) because route event layers are unuseable in 2.9.12. (My organization will upgrade to 3.x eventually; that's beyond my control.)

I'm wondering if that kind of performance issue could have been caught by benchmarking GP tool execution times against previous versions of Pro. Could that be incorporated into existing automated Esri testing mechanisms? And could the results be released to paying customers as some sort of new documentation product?

For example, if that had been done, then it would have been clear that the Make Route Event Layer tool is significantly slower in 2.9.12 compared to previous versions. That would have raised a red flag [for Esri and for users planning to upgrade].

[SSWoodward]

Thanks for bringing up this important topic, @Bud . We do try our very best to address performance issues and understand it can be frustrating when things like this happen. 

The specific issue you've brought up was introduced in Pro 2.9.6 and 3.1 and has been addressed in the coming release of ArcGIS Pro 3.3

Since this Idea is in reference to internal testing practices and not an implementable feature in ArcGIS Pro, it will be closed.  

If you're looking for more information on guidelines for Ideas on the ESRI communities pages, you can refer to the submission guidelines at the link below.

ESRI Ideas Submission Guidelines  

[RTPL_AU]

@SSWoodward From a paying customer perspective if you already have internal testing processes that detect regressions, these regressions should be public and part of the release notes, and if not, this is still a valid idea and should be moved a more appropriate section such as "things that Esri could do better that isn't a product feature".   
Just closing an Idea related to our experience of Pro that is "not an implementable feature in ArcGIS Pro" does not seem very nice? 
Then pointing an active and experienced contributor to your guidelines comes across a bit harsh. 
So many Ideas are related to Pro performance and issues surrounding them, that you have to, by now, realise that you have a problem and that you have loyal customers trying to help. 

[Bud]

@SSWoodward Thanks for providing this info:

The specific issue you've brought up was introduced in Pro 2.9.6 and 3.1 and has been addressed in the coming release of ArcGIS Pro 3.3

That certainly helps. We will try upgrading to 2.9.5 instead of 2.9.12.

---

I like @RTPL_AU's comment:

...if you already have internal testing processes that detect regressions, these regressions should be public and part of the release notes...

---

@SSWoodward, if I were to create a new post as described below, would you close it?

Idea: Documentation — Provide GP tool execution time benchmarks, including comparison against previous Pro versions
Label: Documentation & Help

I'm open to input on the title/content of such an idea.

[MarcoBoeringa]

@RTPL_AU  wrote:@SSWoodward From a paying customer perspective if you already have internal testing processes that detect regressions, these regressions should be public and part of the release notes, and if not, this is still a valid idea and should be moved a more appropriate section such as "things that Esri could do better that isn't a product feature". 

I think I would even go a step further: regarding the nature of the bugs when there is a new software release, there are two clearly distinguishable ones in my opinion:

- 1) Bugs in newly added functionality that doesn't affect any of the existing functionality of ArcGIS and allows the current user base to continue using Pro unhindered as they did with the previous release, and to potentially upgrade to the latest release with the minor caveat of needing to avoid the affected functionality. While it might be a nuisance that some fancy new feature is not usable due to such an issue, such bugs in my opinion are relatively harmless and lower priority from the perspective of the user base (unless there is a risk that the bug causes major damage to e.g. an enterprise geodatabase, but such bugs are rare).

- 2) Regression bugs that directly affect the usability of existing functionality of ArcGIS, and make it impossible to use and / or upgrade to the latest release either due to a severe performance issue, or tools / functionality being truly broken and totally unusable.

In my opinion, the second type of bugs - regression bugs - is far worse than the first, and should always be top priority for fixing as they make it impossible for the existing user base to use or upgrade to the latest version while continuing their regular workflow.

---

The specific issue you've brought up was introduced in Pro 2.9.6 and 3.1 and has been addressed in the coming release of ArcGIS Pro 3.3

---

In my opinion, regression bugs should always be fixed in the next patch release, not in a major or minor release! That is, this bug should have been fixed in a 2.9.x or 3.1.x patch, not in Pro 3.3. Or, if the original fix was developed based on the 3.3 code base, it should have been back ported to all previous releases affected.

I realize time constraints may make it impossible to fix stuff in the next patch release, e.g. 2.9.7, but that still means the fix should have been part of 2.9.8 or so.

If you do not follow such practices, you may end up with a perpetually broken product, because regressions are never fixed in the release cycle as used by the existing user base.

[DaraBurlo]

@RTPL_AU We understand you are trying to help and we do appreciate that. However, ArcGIS Ideas is not the correct location for discussing performance issues. As stated, in the ArcGIS Ideas Submission Guidelines and Statuses :

4. Performance issues are typically complex and involve troubleshooting that may be specific to data, hardware, network, bandwidth, and so on, that are not generally applicable to all users.  While ideas related to performance are not strictly off limits, it will usually be more productive to work with Technical Support to troubleshoot performance concerns. 

Please work with Technical Support on your concerns regarding performance issues.

@Bud yes, in this case, your documentation idea would be closed. Due to the complexities mentioned above, we will not provide such information.  

@MarcoBoeringa We do our best to fix issues in timely manner. Decisions of what does and doesn't make it into a release are not taken lightly and are heavily weighed against many factors. It is not always possible to address issues immediately in the following release after it is found. Some issues, especially complex performance issues, like this one, can be quite gnarly and take a longer time to work through and resolve. 

[Bud]

Esri Case #03617239 - ENH: Provide execution time documentation for all GP tools in each ArcGIS Pro release

Hi Esri Support,

Could we submit this enhancement request to Esri Inc.?
Provide execution time documentation for all common GP tools in each ArcGIS Pro release

For example, a table in the online docs or a spreadsheet that is a list of generic tests/execution times for each geoprocessing tool. There would be a column for each version of Pro (going forward) so that we could easily assess if the version of Pro we want to upgrade to has any known GP tool performance issues.

I suspect that kind of testing is already done internally by Esri as part of regression testing for each release. I’m requesting that such information be released publically [or at least distributed to paying Esri customers].

More information in this closed idea: Benchmark GP tool execution times against previous version
Esri Ideas Managers: “Please work with Technical Support on your concerns regarding performance issues.”

Edit:

Support said

An Ideas post and an Enhancement request are essentially the same thing just logged through different systems. I could log the Enhancement request but I expect it would be closed with the same response from Esri Inc. as they have already stated that a Documentation Idea for this would not be implemented.

Case closed.

[RTPL_AU]

Hi @DaraBurlo 

I am not trying to help, I am trying to communicate the frustration I experience in dealing with Esri and the many, and complex, issues surrounding full adoption of Pro, near 9 years after launch. With Pro basically a rolling release, and with poorly documented release notes, it is really hard to balance the much needed new features with potential for high impact regressions in each release. Coupled with day 1 UI & UX issues still unresolved, and a lack of feature parity with ArcMap/Catalog makes it really difficult to navigate and advise on.

Back to the topic at hand.

We are not discussing performance issues but Esri being more transparent about 'things' they introduce that will impact paying customers' productivity.

The core of the idea is the implementation of functionality, either in ArcGIS Pro, or in related documentation, that exposes more information about new versions and potential benefits/regressions.

Yes, it is complex, but Esri, being the only developer, would have all the metrics on hand to describe benefits and/or regressions. If not you who? Paying customers via Ideas (that get closed) or support cases that can take months to resolve, especially for non-US customers?   

Lastly - after a prolonged support case related to lagginess of the Contents pane in Pro, the conclusive advice from US Support was to 'work slower'. Helpful. 

I understand that staff like you are under pressure to resolve ideas and triage as best you can to reduce the negative messaging around Pro but rather than spending time in forums Esri should just be more transparent and provide more objective, quantitative metrics that we can use the evaluate versions, point versions, or patches & fixes.

[Bud]

@RTPL_AU 

We are not discussing performance issues but Esri being more transparent about 'things' they introduce that will impact paying customers' productivity.

The core of the idea is the implementation of functionality, either in ArcGIS Pro, or in related documentation, that exposes more information about new versions and potential benefits/regressions.

Related post from @JoshuaBixby here:
New Geodatabase Features Add Functionality But Complicate Compatibility and Interoperability

[Bud]

I don't see how a request for additional documentation would be "too complex" and an "unimplementable feature". If Esri's testing/benchmarking hardware changes over the years, then that could be noted in the documentation to provide context. It's when a tool is orders of magnitude slower that we want to watch out for.

It's difficult to find a version of Pro that doesn't have show-stopper bugs. Each organization has its own specific must-haves. Documentation like this would help us find a version of Pro that will work for us.