topic Re: Speeding up a join in Python Questions

Speeding up a join

BrooksBreece — Sun, 04 Mar 2018 14:38:12 GMT

What is a way to speed up a join (based on a feature ID) of a feature class (in a file gdb) and a table of summary statistics about features in the feature class? The table has a one-to-one relationship with the feature class. Split the feature class and table and use multiprocess?

Has the package multiprocess superseded multiprocessing?

Re: Speeding up a join

XanderBakker — Sun, 04 Mar 2018 15:23:17 GMT

How large is your data? How many records do you have and how many fields do you want to join?

Richard Fairhurst has a very good blog post about joins in python using dictionaries, which I recommend you to read: /blogs/richard_fairhurst/2014/11/08/turbo-charging-data-manipulation-with-python-cursors-and-dictionaries

Re: Speeding up a join

BrooksBreece — Sun, 04 Mar 2018 16:52:28 GMT

I have 10,000ish records and am joining 5 to 7 fields. The join takes 30 minutes, which is not horrible. But, I want to make it faster.

Re: Speeding up a join

XanderBakker — Sun, 04 Mar 2018 23:31:50 GMT

Just before you start coding there are some general recommendations to make things faster:

Is you data on a network drive, make a local copy
What is the data format? Best to load the data into a file geodatabase
The fields you join on, do you have attribute indexes defined on those fields?
obviously the computer specs also have influence over the speed you will obtain.

30 minutes for 10.000 records (both featureclass and table) is very slow. It should be possible to do this a lot faster.

Have a look at "Example 2 - Transfer of Multiple Field Values between Feature Classes where there is a 1:1 Match between Field Sets" within the section on "Using a Python Dictionary Built using a da SearchCursor to Replace a Join Connecting Two Feature Classes". That seems exactly what you are looking for.

If you run into problems please postback the code you are using and a sample of your data and we will have a look.

Re: Speeding up a join

ChrisRingo1 — Mon, 05 Mar 2018 21:35:33 GMT

Here is a pretty simple python join tool that ESRI created a couple years ago that just uses a cursor to do the joining. It only runs on geodatabase inputs, however. But it's very fast (at least compared the the standard JoinField tool). I had to modify it slightly, since it didn't recognize field of type "SHORT".