David.Edmends

Help with moving a large dataset

Discussion created by David.Edmends on Jul 26, 2012
Latest reply on Jul 27, 2012 by ktuszynska-esristaff
Hi all,

I was wondering if anyone could help me with a strategy for moving a large sde database.

Our setup consists of two identical ArcSDE 10.0 (no sp) PostgreSQL 8.3 installs, the first machine is a setup to only do Geoprocessing the other on a ArcGIS server. Both machines are 64bit and have a minimum of 8GB or ram each.

My plan is to process monthly updates on the Geoprocessing machine then transfer it over to my ArcGIS Server. The intention is to transfer the whole database every couple of months and replace the old version 100%.

The database occupies about 350 GB of HDD space in PostgreSQL however is it around 40GB when I exported using SDEExport or pg_dump.

I have already tried running SDEExport, transferring the data and running SDEImport however I have found that after an hour or so the import failed with the following message.

[ATTACH=CONFIG]16407[/ATTACH]

One thing I did notice was that the import failed when PostgreSQL got up to using 2GB of RAM, does anyone else have a problem with PostgreSQL being allocated this amount of RAM?

The alternative method I have used was run the pg_dump command which I have never used before but with a little help from the internet I was able to run. Getting the data out was relatively simple however the pg_restore has been running now for 7 days and I am unsure that this method will be successful as I can see a number of error messages.

Does anyone have experience using either of the above methods and could offer me some advice?

The only other option I know of it to setup DB replication between the two servers however I have not done this before. Can anyone tell me if replicating a DB of this size is possible and maybe offer my any advice in setting up the process?

If you have any questions about the above or can offer advice your comments will be very welcome.

Kind Regards

David

Outcomes