topic Re: Data Interoperability woes in Data Management Questions

Data Interoperability woes

LarryPhillips — Fri, 30 Apr 2010 13:52:57 GMT

I have been trying to find a way to run my Spatial ETL tools in smaller sets and need help. I work in a version and process 400000+ records in a single translation and that wrecks havoc on the server and the adds/deletes tables. Any advice on how to run these large data sets would be greatly appreciated.

Re: Data Interoperability woes

BruceHarold — Mon, 03 May 2010 14:59:57 GMT

Hello Furyk

Is it feasible to reduce the number of edits by first doing change detection with the Matcher or ChangeDetector transformers, then only making the edits you need? The idea is to not write a delete/add pair for records which do not change.

Regards

Re: Data Interoperability woes

LarryPhillips — Mon, 03 May 2010 15:18:09 GMT

Thank you for your reply, it points out how poor of a post I made. My problem comes from having many (400,000+) records needing updated. If this data were in a file geodatabase it would be fine because I can set the number of records to write in a transaction. The problem is I'm working in a version and can't run transactions. What I want to avoid is having to manually split the data into smaller sets and run one after the other instead of automating the whole process in Data Interoperability (DI).

Re: Data Interoperability woes

BruceHarold — Tue, 04 May 2010 14:05:56 GMT

Hi again

If your problem is performance of the system with large A and D tables, set your Geodatabase writer to reconcile and post after translation, then the records will be moved to the base table.

Regards