Improve Pro Join Performance

[Tim_Ultee]

I've been working on migrating our workflows from ArcMap to ArcGIS Pro over the last year, and one consistent annoyance is the performance of the join tools in Pro. Joining csv files, tables, and feature classes to each other in ArcMap has always been a cinch and usually takes no more than 10 seconds. In Pro however, both the join and remove join(s) tools run multiple times slower than in ArcMap and often freeze the software as well.

I've experienced these issue from Pro 1.4 through 2.2.3 without seeing any improvement. In what is supposed to be newer software with improved performance, I don't understand why this straight-forward and essential task is such a hindrance.

[DuncanHornby]

100% agree I too find joining data much slower in Pro. I am often adding and removing joins for census data from different sources (csv/excel) and frustrated by its slowness and or freezing. 

[KoryKramer]

For those on this thread, is anybody able to provide data that shows a clear performance difference between joining in ArcMap and ArcGIS Pro?

Here is a test that I just did:

Could you provide similar comparisons and since you're seeing big discrepancies, share the data with us?

Thanks!

[KoryKramer]

Tim Ultee‌ Duncan Hornby‌ Reid Ellis‌ and Levi Cecil‌ Any examples that you're able to share that show poor performance when joining would make this idea actionable.

Please share details about number of records in each table, number of fields, type of join, time for join to complete in ArcMap and time in Pro.  Or, if you could just share example data that demonstrates the issue, that would be most helpful and allow us to look into this.  See my results above.

Thanks

[DuncanHornby]

Kory,

So I have a windows 10 machine, 16GB ram, with a solid state C:\ drive, the datasets are on the network.

In this example I am joining a file geodatabase table to a featureclass.

I am joining approximately 27,000 census rows to about 14,000 polygons. I start the join tool, the stop watch says it took about 19 seconds to run, but the view details pop up says 1.23 seconds!

[KoryKramer]

Thanks for the additional details, Duncan Hornby‌.

1. How many layers are in the open map?
2. Do many layers point to the same dataset that is input to your join?
3. How many maps have been opened in the project? Any maps with tons of layers?
4. Does the join still take this long when run in a new project with a new map?
5. Are you able to provide the data and your project for troubleshooting?
6. Can you try to run the exact same join in ModelBuilder and see if the tool completes faster? Tools run in MB use the main UI thread for execution, so there is not the overhead of background GPthread.

Thank you!

[DuncanHornby]

Kory,

1. How many layers are in the open map? -- this particular map has 14 vector layers, 2 rasters, all were turned off except 2
2. Do many layers point to the same dataset that is input to your join? - None
3. How many maps have been opened in the project? Any maps with tons of layers? - 23 maps in this project, that's what makes ArcPro great, like arcview 3.2, many maps/layouts in a single project!
4. Does the join still take this long when run in a new project with a new map? - Not tried it but can test this, if you want the data.
5. Are you able to provide the data and your project for troubleshooting? - Yes, it's all data off the Indian Census website so OK to distribute, I could zip it all up in a single geodatabase.
6. Can you try to run the exact same join in ModelBuilder and see if the tool completes faster? Tools run in MB use the main UI thread for execution, so there is not the overhead of background GPthread. - OK could try but not for next few days, other commitments

[DuncanHornby]

Hmm interesting! With the expectation you would want the data,

1. I copied the table and featureclass into a new file geodatabase (C:\).
2. Closed ArcPro, started a new session
3. Added a new map and added the the featureclass and table.
4. ran the join, it took less than 4 seconds using my stop watch
5. I repeated the whole process but from a geodatabase on the network, same run time.

This test would certainly point to the idea that it might have something to do with numbers of maps/layers in an ArcPro Map Document.

I would be very reluctant in splitting out a project into several map documents. I always thought when ESRI moved from ArcView to ArcMap I felt they had not understood how a great many users used their product and loosing the ability to have many maps/layouts in a single project was a real step backwards in productivity. But now ArcPro puts that all back in, which is great!

[DrewFlater]

Duncan Hornby‌ a project package of your original project would be the most useful in this case.

I am not sure what improvements can be made since moving the information model of large maps and layers between threads takes some time in Pro, but we will look at the case and optimize it as much as possible.

Like I mentioned, you may find running Add Join in ModelBuilder will be significantly faster, as in that environment tools always run in the main foreground thread, so it does not require the overhead of moving the map and layer information model between threads. Running in standalone Python will be the fastest environment, with zero overhead.

While your image only shows one open map, was it possible that you had closed other maps that were previously open in the same project session? If a map was never opened in a session, we shouldn't be loading the information model for that map and it's layers, which lessens the performance overhead of moving between threads. But if a map was ever opened in the session we still may be loading it's information. This is speculative at this point until I can troubleshoot further, aided by your project.

Thanks

Drew - Geoprocessing team

[DuncanHornby]

I am asking my superiors if I can share the entire project, data and maps. If I can share the entire project with you how do I send it to you?  You have hidden all your contact details on geonet and thus essentially un-contactable...

[KoryKramer]

Duncan, you can email to me at kkramer@esri.com if it is small enough.  If email isn't an option let me know (by email) and I will set up an ftp site to transfer the package.  Thanks!