Working with GIS data on Network Share

4001
14
Jump to solution
07-26-2019 06:50 AM
MalcolmMeyer2
Occasional Contributor II

Hey Ohio GIS community - what is your workflow for working with GIS data that is stored on a network share. I am not talking about data stored in an enterprise database or workgroup database. I am talking shapefiles, file geodatabases, or personal geodatabases. I have recently been told by Esri support, after encountering some data corruption, that editing file geodatabases off a shared drive is not recommended.

File geodatabases are not recommended to be stored on network drives, and two or more users simultaneously viewing/editing a feature class within the database can easily create a wide range of issues...I would suggest that you copy the file geodatabase onto a local drive on your machine, or onto a SQL Server express database...

see - https://community.esri.com/thread/221282-file-geodatabase-in-windows-networks-very-poor-performance


Has anyone found one of the three data storage types listed above more reliable than others? It looks like my only avenue until I get SQL Server Express up and running on the network would be - 

  1. Copy data to local drive
    1. This now means that I have one MXD linked to local data, and the viewer MXDs that are stored on the network linked to network data, unless I completely recreate the server file structure on my local machine, thus allowing relative paths to work for both MXDs.
  2. Make Edits
  3. Verify all Arc sessions are closed and copy the data back to the server
    1. Unless I do this daily, I have to then create a backup plan for my local data and MXDs (the server is already backed up)
    2. Scripting this will not work as I have to first verify all Arc sessions are closed and no lock files are present
1 Solution

Accepted Solutions
LanceCole
MVP Regular Contributor

Malcolm Meyer,

pGDB do have a MDB extension but they are a hybrid database for use in ESRI products.  You can open them in Access, but it is not supported and can corrupt the file if you are not careful when saving and making changes.  Not only is ESRI no longer developing and supporting pGDBs but Microsoft is no longer developing and supporting MDBs that the file is based upon.  It is a matter of time before these files will disappear or become unusable.  Furthermore and as previously stated, pGDB files absolutely do not support multi-user access.  Each user must have their own copy of the data. 

We we have hundreds of fGDB files saved on network shares that about 30 users access for read-only purposes.  The network shares are flagged read only for all authorized users with the exception a few users that have full access for editing.  This prevents lock files from being created on the fGDB files by users only accessing to read the data.  It also prevents accidental editing, deleting, etc.  We have never had a fGDB become corrupted but have had orphaned lock files from ArcGIS crashes (like that ever happens) while an editor is working on the file.  A quick delete of the lock files fixes the issue.

View solution in original post

14 Replies
ScottFierro2
Regular Contributor

I can say that I haven't worked with shapefiles or personal geodatabases (pGDB) in a long time. Primarily because I only work with shapefiles if I have a system that dictates its the only or sometimes easiest way to get data to that system. pGDB's I am sure there are maybe 3 workflows under which ESRI recommends them as the best practice to use. We strictly use File Geodatabases (fGDB) and I will do that even for something as simple as 1 data set. Storage is so cheap these days that any slight amount more it takes up is irrelevant and fGDB's have been extremely reliable with the ability to leverage a large amount of the geoprocessing (GP) tools we'd invoke against an enterprise geodatabse (eGDB).

For a variety of reasons we have a slew of fGDB's that live out on share networks that we use. I'd say this really becomes more of a workflows situation. Our fGDB's get used for odds and ends stuff and a lot for verification. They aren't used as production sources nor are they utilized for routine or scripted processes. It sounds like this may be production level, transactional type work you are doing. If that's the case and you don't have an eGDB then I'd agree with using SQL Express as the next best option and in your case until SQL Express is available then a fGDB is the best option.

ESRI is correct, fGDB's are not intended for multi-user editing type workflows. While you can have multiple users inside a fGDB at once you do run some risks of creating issues once those multiple users start touching the same data sets. I can't say I have ever heard that simply storing the fGDB on a share drive carries risks for corruption. The only theory I can provide on that would be based on a networking aspect. If you are mid edit or a GP is running and there's a significant network lag/spike issue or temporarily drops then I could see how this situation could cause orphaned processes that create issues or corruption. That's nothing that can'y be accounted for and addressed by doing something simple like a robocopy backup of your fGDB each night or if you don't have access to things like Windows Task Scheduler can be done via python or Model Builder options against an ArcServer. You are just essentially taking on some of the routine maintenance type tasks database administrator's (DBA) take on.

MalcolmMeyer2
Occasional Contributor II

For clarification, the current workflow only involves one editor and two viewers. Even with only one editor data corruption is occurring, likely due to internal network issues which are beyond my control. I did not think this was possible, but I think you're right on here - 

If you are mid edit or a GP is running and there's a significant network lag/spike issue or temporarily drops then I could see how this situation could cause orphaned processes that create issues or corruption
0 Kudos
LanceCole
MVP Regular Contributor

Malcolm‌,

You really only should consider File Geodatabases of the three options noted.  You can share a fGDB between multiple users, however, there really can and should only be one editor per feature class at any one time.  We have all our fGDBs on network drives as users are not allowed to store them on local drives for backup reasons should a local drive fail.  Personal geodatabases are not really supported any longer and on their way out.  Shapefiles are great for their intended purpose, portability between systems especially different platforms but not everyday use.  I am just as guilty, as most others, of creating a shapefile for a quick use that creeps in to a production workflow.  

Based upon your follow up post, fGDB would work well with one editor and others only viewing in a network environment even though not recommended.  You will still have issues on occasion where you have a lock condition occur requiring other users to exit ArcGIS to release the lock files.  You can also consider having a script run to make a copy of the fGBD daily that the users would access the copy and the editor would access the original.  These can both be on your network drives just different folders.  Once the MXDs are created, as long as the file names and paths remain the same,  users will not  notice nightly updates.

The best option you noted for small workgroups is Workgroup geodatabases  .  This runs on Microsoft SQL Server Express and can support up to 10 users.  A workgroup geodatabase can be setup on a server in under an hour and fGBDs/feature classes are literally drag and drop.

0 Kudos
MalcolmMeyer2
Occasional Contributor II

As I noted above, this is not recommended by Esri via Esri support - 

You can share a fGDB between multiple users, however, there really can and should only be one editor per feature class at any one time.

This is my current setup and is leading to data corruption. I am switching to SQL Server Express and will test that out, but that conversion is up to IT and is some unknown time in the future.

0 Kudos
AdamBourque
New Contributor III

Hey Malcolm, I am in Kentucky. I will say I like to create a file geodatabase and then convert that into a shapefile for intended uses such as transferring it over to QGIS. I think it's good to have a shapefile copy of the file geodatabases but fGDB are the way to go for many reasons such as collection and organization/processing data. 

0 Kudos
MalcolmMeyer2
Occasional Contributor II

Another clarification - I am wondering about shapefile vs FGDB vs PGDB in terms of their reliability and stability when used over a network share with multiple viewers and one editor. I realize this is not a true enterprise setup, but I inherited hundreds of shapefiles that are referenced via a network share. I do not want to convert all these to an Enterprise GDB (once we get this up and running) if I can avoid it. 

0 Kudos
LanceCole
MVP Regular Contributor

Do not use pGDBs.  They are not supported moving forward in Pro and possible future ArcMap versions.  Additionally, they absolutely do not support multi users in any way.  

0 Kudos
MalcolmMeyer2
Occasional Contributor II

That's fine, but doesn't really get to my question. I have a hunch that an Access database would be more reliable on a windows network share than an Esri File Geodatabase, because 1000x more people use MS Access than ArcGIS, therefore more testing, more bugs, etc., but that's just a hunch. I am posting this to get real-world experience using GIS data on network shares outside of an Enterprise Geodatabase.

0 Kudos
LanceCole
MVP Regular Contributor

Malcolm Meyer,

pGDB do have a MDB extension but they are a hybrid database for use in ESRI products.  You can open them in Access, but it is not supported and can corrupt the file if you are not careful when saving and making changes.  Not only is ESRI no longer developing and supporting pGDBs but Microsoft is no longer developing and supporting MDBs that the file is based upon.  It is a matter of time before these files will disappear or become unusable.  Furthermore and as previously stated, pGDB files absolutely do not support multi-user access.  Each user must have their own copy of the data. 

We we have hundreds of fGDB files saved on network shares that about 30 users access for read-only purposes.  The network shares are flagged read only for all authorized users with the exception a few users that have full access for editing.  This prevents lock files from being created on the fGDB files by users only accessing to read the data.  It also prevents accidental editing, deleting, etc.  We have never had a fGDB become corrupted but have had orphaned lock files from ArcGIS crashes (like that ever happens) while an editor is working on the file.  A quick delete of the lock files fixes the issue.