topic Re: Better method than APPEND? in Data Management Questions

Better method than APPEND?

JimCousins — Tue, 18 Mar 2014 15:56:06 GMT

I am processing @ 5,000 small tables (Info tables). They are the result of an intersect and summary statistics for each element of a polygon grid.
I now wish to get all tabular information into a single table. I am using python with an iterator and Append_Management, but it is slow (about 1 hour per 1,000 tables). Is there a better way to accomplish this? All tables have the same schema, only different number of rows(1 to 12).
Regards,
Jim

Re: Better method than APPEND?

RobertBorchert — Tue, 18 Mar 2014 17:46:10 GMT

I think your stuck with what you are going. Actually an hour to append 1000 tables does not sound bad. Processing speed is dependent on size of the tables and the number of tables associated with the append. In this case 5000 tables is certainly a slow down factor, even if they are small.

Data loader is faster because your not in an edit session. However, setting it up would be cumbersom.

I am processing @ 5,000 small tables (Info tables). They are the result of an intersect and summary statistics for each element of a polygon grid.
I now wish to get all tabular information into a single table. I am using python with an iterator and Append_Management, but it is slow (about 1 hour per 1,000 tables). Is there a better way to accomplish this? All tables have the same schema, only different number of rows(1 to 12).
Regards,
Jim

Re: Better method than APPEND?

JimCousins — Tue, 18 Mar 2014 18:44:36 GMT

Thanks, Robert. It is helpful to know I have not overlooked the obvious.... Again.
Best Regards,
Jim

Re: Better method than APPEND?

MarkBoucher — Wed, 19 Mar 2014 22:53:00 GMT

Editing over a network is slow. If you are working over a network, copy the data to your local drive, edit it faster, then copy it back. Just a thought.

Re: Better method than APPEND?

RichardTruong — Sun, 06 Dec 2015 23:59:11 GMT

Hi Robert,

I tried Data Loader for my table in ArcServer, and I received an error "Cannot load simple table". Any suggestions are greatly appreciated.

Richard

Re: Better method than APPEND?

RobertBorchert — Tue, 08 Dec 2015 13:42:32 GMT

Since writing my last reply I then saw Mark Boucher answer and it reminds me of something I do.

I created a simple batch file to automatically back up the databases I work with on my computer and compress them automatically. I simply click the icon first thing every morning and it starts the programs I want every morning (email, browser, ArcGIS), compresses my active databases and backs them up to the network.

Editing on a personal computer, as Mark wrote, is significantly faster in our environment then working off a hard drive on a server somewhere