How do you manage and update file geodatabase data?

[SarahNoakes1]

Hi,

We have a lot of data stored as file geodatabases.  We are serving this data via ArcGIS for Server 10.2.2 map services.

Our data is stored in a 'published' area on our network which users do not have write access to.  Users update their data locally and supply it to the GIS team and we then update the published data.

I am interested in the methodologies other organisations have come up with to manage and update data in this scenario.

The options I am exploring at the moment are to either:

a) in Windows Explorer copy and paste the folder dataset_name.gdb - directly overwriting the files seems to work and ArcGIS for Server picks up the changes to the data straight away. This does not seem a very robust solution though - I'm not sure how it would work with a large dataset, and I'm also concerned that if something went wrong we would end up with corrupt datasets

b) Stop ArcGIS for Server. Use Catalog (or Python scripting) to rename the current data then copy and paste in the new data.  This could be done at either the geodatabase or feature class level. Restart ArcGIS for Server.  This is more robust but I am keen to avoid unnecessary regular downtime.

How do you manage your data?  Do you have a better workflow?  I'm really interested in what other people are doing so please get involved in the discussion.

Thanks,

Sarah Noakes
Cornwall Council

[DavidColey]

Hi Sarah-

We use a combination of python and bat scripts to accomplish exactly what you describe for layers that experience tremendous I/O. thereby relieving our SQL SDE of large cpu loads.  Each python script loops through feature datasets in my staging fgdb's stored locally on my dev pc, deletes the desired fc's, then either copies or imports (depending on the data and relational dependencies) the same fc down from our sde.  The scripts then update spatial and attribute indices (if import method must be used).  Even for large datasets such as parcels and tax rolls, all of this takes less than 5 minutes.  I then use bat scripts to remove the fdgb directory and xcopy in the staging fgdbs, usually in less than one minute.  If map service must hold a lock on the publicaiton data, the arcgis utitlity scripts for stopping and starting the service is employed as part of a bat. 

This is performed nightly for some datasets (such as for property tax rolls), weekly for all others (such as streets - we empoly a weekly replica sync from our edit to production sde gdbs).
 
And forgive me for saying so, but this is a quite robust method and I have yet to experience a failure not caused by my own user error.  Please keep in mind that all shared directories must be setup with the appropriate permissions for all participating directories.
Thanks
David

[SarahNoakes1]

Hi David,

Thanks for the input.  That sounds like the sort of thing we are thinking of doing.  I think the only robust way to do the data updates is to pull the ArcGIS for Server services down briefly overnight, move the old data out to a backup folder and copy the updated data in.

At the moment we are mostly still using ArcIMS and we bring the software down every night for about seven minutes.  This is for two reasons - partly to do the data updates but also because ArcIMS benefits from regular restarts.  However it then takes over ten minutes to fully restart.

With ArcGIS for Server the software is considerably more robust, so we don't need to be restarting it every night and we don't want that much downtime.  I think a weekly update schedule with occasional ad hoc updates for critical data should suffice - as long as the data owners agree.

The other thing I haven't investigated yet is how to handle data updates when the underlying structure of the data changes i.e. new, deleted or renamed fields.  Presumably the map service would need refreshing following the update?

Regards,

Sarah.

[TysonJohnston]

I am curious as to why using versioned editing and/or distributed geodatabase is not an option?  We have had success using versioned editing on data edited only by staff in our building and feature service editing (a combination of versioned editing and distributed geodatabase) for anyone else.

[DavidColey]

Sure thing.  The only time that I must stop services in our workflows are when those services have to persist a shema lock, as is the case with mobile and feature editing.  But in those cases the mobile and feature services are being sourced directly to a version in our edit sde instances. 

I didn't realize you were still on IMS, but as far as schema changes go:  they should be planned and infrequent and when they do occur you will need to overwrite your map services.  Further, if you are that concerned about losing data, then perform some type backup prior to fdgb removal. 

As far as Tyson's comments are concerned, we have succefully employed  full versioning and replication (we replicate via historical archiving) workflows since 2007.  Don't get me wrong: over 95% of the data in our dynamic map services is sourced directly from our production sde,where data syncs occur weekly to production and data is never out of date-
Thanks
David

[SarahNoakes1]

We are still using ArcIMS with shapefiles - pulling the software down each night and copying shapefiles across at the file level has worked well for us for (many) years.

However we are trying to migrate to ArcGIS for Server, and that doesn't seem to perform too well with shapefiles, so we are also migrating to using file geodatabases as our default storage format - hence this thread.

Versioned editing would not be appropriate because we do not hold most of our data in SDE - we've only found that useful for large fairly static datasets, such as aerial imagery and basemapping.  Until recently shapefiles easily out performed SDE for small datasets.  I've briefly looked at distributed geodatabases and it might be an option, but I think that's probably overcomplicated for the type of data we are looking at.

I'm interested that you say you don't always have to stop the services - I've been playing on our test servers and I can overwrite a *.gdb folder in Windows Explorer when the data is in use and ArcGIS for Server picks the changes up straight away. However I can't delete a *.gdb folder that is in use.  Uisng Catalog I can rename a feature class within a fgdb and copy the new one in, but Server does not pick up the new data until the map service is restarted.

Cheers,

Sarah.

[SarahNoakes1]

Hi,

I have a further question.  The file geodatabases I am trying to update are also used by ArcGIS for Desktop users.  These users only have read access to the data, However, if one of them has left ArcGIS open overnight when the update is running some of the files within the fgdb are locked and it doesn't delete cleanly (even though the arcpy.Delete_management command doesn't report an error).

Have you come across this issue?  Have you any suggestions how to disconnect users so the fgdb can be deleted?

Thanks,

Sarah.

[MichaelVolz]

I would perform a truncate and append of the data from the source file gdb(s) to the file gdb(s) used for AGS services which bypasses any schema locks you would have.  The only data type that I had an issue with using truncate and append was annotation feature classes as I don't believe it supports an append operation.  I had to stop this service overwrite the feature class and start the service again which was automated with python as a scheduled task.

[DavidColey]

You could try setting up a shutdown in your licence manager for all desktop products.  We use FlexLM and it does just that.  And to your earlier question, no I don't always have to stop services, only if a schema lock is set to true. And yes, you will have to overwrite or restart the map service depending on the nature of the change.