topic Re: Export delta tables? in Data Management Questions

Export delta tables?

Zeke — Fri, 21 Sep 2012 15:34:53 GMT

Is it possible to export delta tables? We're looking at a workflow to provide change only data to another entity on a weekly or monthly basis. If so, what info exactly is in the delta tables? All attributes of the feature created/changed, or something else? Not sure if the delta table keeps the old data or just new data; wondering if we could use the exported delta table with the updated feature to create a feature class to show before/after features. Currently on SDE 10.0 non-versioned, will be upgrading to 10.1 in a few months, versioning undecided. Thanks.

Re: Export delta tables?

KevinDunlop — Fri, 21 Sep 2012 15:52:09 GMT

I would look into replication. You could do a 1-way replication with that other organization you need to share with. This way your data is not changed but all changes from the last sync is captured and shared with the other organization.

Re: Export delta tables?

VinceAngelo — Fri, 21 Sep 2012 16:17:37 GMT

What sort of volume are you talking about? How many tables? Are you using ArcGIS
to edit the tables right now?

I've used a simple hashing function to identify row change, and built views to export
insert, update, and delete messages (as ASCII files) to synch data between two
air-gapped systems.

- V

Re: Export delta tables?

Zeke — Fri, 21 Sep 2012 16:59:04 GMT

Thanks Kevin, Vince. The basic scenario right now is that the county needs any new, updated or deleted city street and city boundary data, but they don't want all the data every time, just changes. They'd get this data weekly, possibly monthly. It shouldn't be a whole lot of data. We want the process to be as simple and quick as possible on our end.

Using ArcGIS to edit. Not familiar with replication, will look into it. I was thinking that just sending the delta tables would work, but then again, not really familiar with them in detail either.

Re: Export delta tables?

VinceAngelo — Fri, 21 Sep 2012 18:50:11 GMT

You can't send just the delta tables if you're not using versioned geodatabases (no delta
tables to send, for one, besides, not all the deltas make it into the base table).

Replication is very powerful, but adds a UUID field requirement to versioning. If the data
is being regenerated from an external source, replication could not distinguish unchanged
rows, while using a hashing function to pick the columns that matter could.

I had a daily job to evaluate the results of a 5 hour long data processing job. It took 5-7
minutes to detect changes, and another 5-10 minutes on the receiving end to import
not more than 50Mb of changes. I also had a feedback loop to transmit the state of the
receiving end, to make sure the two tables stayed in sync.

- V