Speeding up a join

[BrooksBreece]

What is a way to speed up a join (based on a feature ID) of a feature class (in a file gdb) and a table of summary statistics about features in the feature class?  The table has a one-to-one relationship with the feature class.  Split the feature class and table and use multiprocess?

Has the package multiprocess superseded multiprocessing?

[XanderBakker · Accepted solution]

How large is your data? How many records do you have and how many fields do you want to join?

Richard Fairhurst has a very good blog post about joins in python using dictionaries, which I recommend you to read: /blogs/richard_fairhurst/2014/11/08/turbo-charging-data-manipulation-with-python-cursors-and-diction... 

[BrooksBreece]

I have 10,000ish records and am joining 5 to 7 fields.  The join takes 30 minutes, which is not horrible.  But, I want to make it faster.

[XanderBakker]

Just before you start coding there are some general recommendations to make things faster:

• Is you data on a network drive, make a local copy
• What is the data format? Best to load the data into a file geodatabase
• The fields you join on, do you have attribute indexes defined on those fields?
• obviously the computer specs also have influence over the speed you will obtain.

30 minutes for 10.000 records (both featureclass and table) is very slow. It should be possible to do this a lot faster.

Have a look at "Example 2 - Transfer of Multiple Field Values between Feature Classes where there is a 1:1 Match between Field Sets" within the section on "Using a Python Dictionary Built using a da SearchCursor to Replace a Join Connecting Two Feature Classes". That seems exactly what you are looking for.

If you run into problems please postback the code you are using and a sample of your data and we will have a look.

[ChrisRingo1]

Here is a pretty simple python join tool that ESRI created a couple years ago that just uses a cursor to do the joining. It only runs on geodatabase inputs, however.  But it's very fast (at least compared the the standard JoinField tool). I had to modify it slightly, since it didn't recognize field of type "SHORT".