idea Improve Pro Join Performance in ArcGIS Pro Ideas

Improve Pro Join Performance

Tim_Ultee — Mon, 18 Feb 2019 16:48:27 GMT

I've been working on migrating our workflows from ArcMap to ArcGIS Pro over the last year, and one consistent annoyance is the performance of the join tools in Pro. Joining csv files, tables, and feature classes to each other in ArcMap has always been a cinch and usually takes no more than 10 seconds. In Pro however, both the join and remove join(s) tools run multiple times slower than in ArcMap and often freeze the software as well.

I've experienced these issue from Pro 1.4 through 2.2.3 without seeing any improvement. In what is supposed to be newer software with improved performance, I don't understand why this straight-forward and essential task is such a hindrance.

Re: Improve Pro Join Performance

DuncanHornby — Tue, 19 Feb 2019 10:51:07 GMT

100% agree I too find joining data much slower in Pro. I am often adding and removing joins for census data from different sources (csv/excel) and frustrated by its slowness and or freezing.

Re: Improve Pro Join Performance

KoryKramer — Wed, 27 Feb 2019 20:28:14 GMT

For those on this thread, is anybody able to provide data that shows a clear performance difference between joining in ArcMap and ArcGIS Pro?

Here is a test that I just did:

Could you provide similar comparisons and since you're seeing big discrepancies, share the data with us?

Thanks!

Re: Improve Pro Join Performance

KoryKramer — Sat, 09 Mar 2019 13:28:35 GMT

Tim Ultee‌ Duncan Hornby‌ Reid Ellis‌ and Levi Cecil‌ Any examples that you're able to share that show poor performance when joining would make this idea actionable.

Please share details about number of records in each table, number of fields, type of join, time for join to complete in ArcMap and time in Pro. Or, if you could just share example data that demonstrates the issue, that would be most helpful and allow us to look into this. See my results above.

Thanks

Re: Improve Pro Join Performance

DuncanHornby — Mon, 11 Mar 2019 10:12:55 GMT

Kory,

So I have a windows 10 machine, 16GB ram, with a solid state C:\ drive, the datasets are on the network.

In this example I am joining a file geodatabase table to a featureclass.

I am joining approximately 27,000 census rows to about 14,000 polygons. I start the join tool, the stop watch says it took about 19 seconds to run, but the view details pop up says 1.23 seconds!

Re: Improve Pro Join Performance

KoryKramer — Mon, 11 Mar 2019 15:39:24 GMT

Thanks for the additional details, Duncan Hornby‌.

How many layers are in the open map?
Do many layers point to the same dataset that is input to your join?
How many maps have been opened in the project? Any maps with tons of layers?
Does the join still take this long when run in a new project with a new map?
Are you able to provide the data and your project for troubleshooting?
Can you try to run the exact same join in ModelBuilder and see if the tool completes faster? Tools run in MB use the main UI thread for execution, so there is not the overhead of background GPthread.

Thank you!

Re: Improve Pro Join Performance

DuncanHornby — Mon, 11 Mar 2019 16:27:42 GMT

Kory,

How many layers are in the open map? -- this particular map has 14 vector layers, 2 rasters, all were turned off except 2
Do many layers point to the same dataset that is input to your join? - None
How many maps have been opened in the project? Any maps with tons of layers? - 23 maps in this project, that's what makes ArcPro great, like arcview 3.2, many maps/layouts in a single project!
Does the join still take this long when run in a new project with a new map? - Not tried it but can test this, if you want the data.
Are you able to provide the data and your project for troubleshooting? - Yes, it's all data off the Indian Census website so OK to distribute, I could zip it all up in a single geodatabase.
Can you try to run the exact same join in ModelBuilder and see if the tool completes faster? Tools run in MB use the main UI thread for execution, so there is not the overhead of background GPthread. - OK could try but not for next few days, other commitments

Re: Improve Pro Join Performance

DuncanHornby — Mon, 11 Mar 2019 16:45:30 GMT

Hmm interesting! With the expectation you would want the data,

I copied the table and featureclass into a new file geodatabase (C:\).
Closed ArcPro, started a new session
Added a new map and added the the featureclass and table.
ran the join, it took less than 4 seconds using my stop watch
I repeated the whole process but from a geodatabase on the network, same run time.

This test would certainly point to the idea that it might have something to do with numbers of maps/layers in an ArcPro Map Document.

I would be very reluctant in splitting out a project into several map documents. I always thought when ESRI moved from ArcView to ArcMap I felt they had not understood how a great many users used their product and loosing the ability to have many maps/layouts in a single project was a real step backwards in productivity. But now ArcPro puts that all back in, which is great!

Re: Improve Pro Join Performance

DrewFlater — Mon, 11 Mar 2019 23:58:35 GMT

Duncan Hornby‌ a project package of your original project would be the most useful in this case.

I am not sure what improvements can be made since moving the information model of large maps and layers between threads takes some time in Pro, but we will look at the case and optimize it as much as possible.

Like I mentioned, you may find running Add Join in ModelBuilder will be significantly faster, as in that environment tools always run in the main foreground thread, so it does not require the overhead of moving the map and layer information model between threads. Running in standalone Python will be the fastest environment, with zero overhead.

While your image only shows one open map, was it possible that you had closed other maps that were previously open in the same project session? If a map was never opened in a session, we shouldn't be loading the information model for that map and it's layers, which lessens the performance overhead of moving between threads. But if a map was ever opened in the session we still may be loading it's information. This is speculative at this point until I can troubleshoot further, aided by your project.

Thanks

Drew - Geoprocessing team

Re: Improve Pro Join Performance

DuncanHornby — Tue, 12 Mar 2019 11:10:07 GMT

I am asking my superiors if I can share the entire project, data and maps. If I can share the entire project with you how do I send it to you? You have hidden all your contact details on geonet and thus essentially un-contactable...

Re: Improve Pro Join Performance

KoryKramer — Tue, 12 Mar 2019 12:51:15 GMT

Duncan, you can email to me at kkramer@esri.com if it is small enough. If email isn't an option let me know (by email) and I will set up an ftp site to transfer the package. Thanks!

Re: Improve Pro Join Performance

LarryMcEvoy — Tue, 26 Nov 2019 13:41:02 GMT

With SQL Server 2016 as a back end cross database joins are an absolute night mare speed is so slow in comparison to arc map 10.6. 20,000 geometry records joined to a view with cross database elements taking nearly 6x as long as arcmap, never mind the fact that i can run the same spatial query in SSMS and get a return in seconds. Yes there are workarounds of creating a table etc, but this is simple database stuff surely ESRI realize that customers need to interact with large custom built databases and want to be able to not have to duplicate data on a constant basis. Speed does improve when using a query layer but it seems to be a lottery on which polygon you will get the attributes from when clicking for a pop up.

Re: Improve Pro Join Performance

Tim_Ultee — Fri, 24 Jan 2020 22:00:43 GMT

I sent a project package to Kory on 4/23/2019 and Esri confirmed that they saw the join performance problem. Kory said he'd share it with the development team but I haven't heard anything about this issue since then.

Re: Improve Pro Join Performance

esrizarei — Wed, 20 Dec 2023 17:20:13 GMT

100% agree and on your side. every time I need to remove my joins as soon as I hit remove I can go for a coffee break and come back and still ArcGIS pro is struggling to remove the join. I used to do the same simple task in ArcMap in a blink of an eye. unfortunately our organization is pushing us to migrate to pro before even ESRI make necessary improvements in its top of the line software. I am getting calls from project manager why I have charged so much for this and that. I have to explain our software is not as fast as before. they cannot charge our wasted time to our clients though, that means my chargeable time will be ride off.

and to the ESRI guy who is asking for some example data. you don't need to wait you can try it yourself. just add a feature later and try to join and remove it later both in ArcMap and ArcGIS pro and you see what we are talking about.

Honestly I thought I am the only one is suffering and now I noticed I am not alone,

by the way this poor performance does not stop here, it is everywhere all over ArcGIS pro, everything and anything take more time to accomplish. Even a simple act of selecting a layer takes more time!!

wow!!!

this issue still exists in pro 3.2

Re: Improve Pro Join Performance

ErickFox — Fri, 08 Mar 2024 15:58:12 GMT

Experiencing this as well--removing joins takes forever. Currently sitting on a 30-minute (and counting) batch remove join process. It is crazy that we are still dealing with this issue and many other Pro performance issues 5 years and however many releases of Pro later from the original comment. Very frustrated right now.

Re: Improve Pro Join Performance

Bud — Sat, 09 Mar 2024 00:49:39 GMT

Perform join at the database level, not the application level, when the datasets are in the same workspace

Re: Improve Pro Join Performance

JamesCoyle — Thu, 23 Jan 2025 14:51:15 GMT

In working with New York State 6.2 million record SAM database, joins in ArcMap to tables of 2 million records for transfer of data often took seconds and transfer of 2,000,000 field values might take 2 minutes on not particularly robust laptop. I have had to divide this dataset into the 62 individual NY counties and still the joins get hung up in counties with less than 100,000 records in ArcGIS Pro. Obviously not an improvement.

Re: Improve Pro Join Performance

PeteFellows — Tue, 03 Feb 2026 14:15:36 GMT

OK just did a GEOID BLK GRP small text to text attribute join for a PORTION of VT. ArcMap seconds ArcGIS Pro crashed. I know I can convert to numeric but then I get SCINOT when downloading data table from census. Just let me use TEXT fields for small ones!