AnsweredAssumed Answered

File geodatabase in Windows networks very poor performance

Question asked by LstAB_Herner on Sep 14, 2018
Latest reply on Oct 29, 2019 by mboeringa2010

Like • Show 2 Likes2
Comment • 28

Hello,

I'll try to keep this short. I represent a large goverment organisation in Sweden. I am a GIS coordinator for the user side of the organisation and of my daily tasks is to recommend best practices, to keep performance up within the typical large organisation restraints (central SDE, network drives)

I have a benchmark script that does this:

1 - creates two file gdbs at the same location (arcpy.CreateFileGDB_management x 2)
2 - creates two empty featureclasses in one of them (arcpy.CreateFeatureclass_management x 2)
3 - exports both featureclasses with one command (arcpy.FeatureClassToGeodatabase_conversion(fcs))
4 - then exports them both again, this time one by one

When I do this on my local harddrive, i.e. my computer, it takes around 10 seconds all together to execute the script.

When I run the script on a network drive, it takes upwards of 8 to 10 minutes (!!!!!)

Where do I start? Any good resources I should read? I am not a windows network specialist and I have no idea why it is like this. Any feedback is greatly appreciated, thank you.

1 person has this question

Outcomes

28 Replies

JQuinn-esristaff Sep 14, 2018 10:39 AM
Mark Correct
Correct Answer
WIndows shares will always be slower than using a local drive and your network will determine how much slower. You've already benchmarked performance, but if you want to take ArcMap/Esri products out of it, you can use Python to create/delete files on the share or IoZone, which can provide benchmarking statistics for throughput on reading/writing to directories:

Iozone Filesystem Benchmark

You'll likely need to involve the organizations IT staff.
5 people found this helpful
Like • Show 0 Likes0
Actions
bixb0012 Sep 14, 2018 12:30 PM
Mark Correct
Correct Answer
I agree with Jonathan, it is best to look at performance outside of Esri applications first. It may be that your performance bottlenecks have little to do with Esri applications, or maybe it is just the opposite.

Are you running this on a local PC or in a centralized or virtualized environment like Citrix?
Like • Show 0 Likes0
Actions
- LstAB_Herner @ bixb0012 on Sep 17, 2018 4:30 AM
  Mark Correct
  Correct Answer
  Thank you for your feedback so far.
  Joshua, I am running on a physical laptop with a network connection. No Citrix or Virtual Machine.
  
  I've hacked together a new script that creates 100 files, writes some stuff in them, reads it again, and deletes the file. Our general IT infrastructure is set up as: A centralised IT department 300 miles away that has the physical servers. They organise the networks etc.
  K: - this is a DNS/DFS (?) pointer to a local server in my building. this is used to set up alias' to hide certain folders etc. Basically, so IT can better manage everyone.
  J: is a direct UNC mapping to the same physical local server (that K: points to)
  B: is my local harddrive.
  
  So the script repeats the same process X times and records the overall time, as shown below. All the random filenames and paths are pregenerated.
  
  This is my timing output:
  
  So, it appears that the delete operation is causing a lot of time. I am asuming this has to do with permissions and network control to the files. Does this give any of you really smart people an idea of where to start my search for the underlying problem?
  
  I have heard that the fgdb format is very "chatty" in terms of network utilization, and with the approach of ArcGIS Pro, where everything is designed to work in fgdbs, we are looking for official recommendations. But for now, I'll happily take inofficials ones!
  
  When I run my benchmark test against the UNC mapped server, I get better performance, but still not nearly as good as I would like. These are my results from averaging out several runs last year:
  Like • Show 0 Likes0
  Actions
  - bixb0012 @ LstAB_Herner on Sep 17, 2018 7:05 AM
    Mark Correct
    Correct Answer
    Although your tests are a start, the steps in your tests are not very representative of I/O patterns from ArcGIS software to a file geodatabase. That said, the results can still be used to glean some information. Looking at the results from J:, the times range from 3.4x slower (writing files) to 76.6x slower (reading files).
    
    The results from your ArcGIS tests show that J: is taking 15.4x longer than B:. Seeing that your ArcGIS tests will involve a mixture of creating, writing, and reading files on the file system, and those operations range from 3 to 75 times slower on J: than B:, I don't think seeing a 15x slower time to J: with your ArcGIS tests is that unexpected.
    
    I would say that what you are seeing in ArcGIS are symptoms and not the problems themselves. Optimizing SMB/CIFS performance is very involved, and it surely is outside the bounds of GIS, i.e., your IT department should not expect you (a GIS person) to be providing them direction on optimizing SMB/CIFS performance.
    
    If your J: drive is hosted on Windows Servers, you can start by reading Performance Tuning for SMB File Servers | Microsoft Docs
    3 people found this helpful
    Like • Show 0 Likes0
    Actions
    - LstAB_Herner @ bixb0012 on Sep 18, 2018 5:03 AM
      Mark Correct
      Correct Answer
      Joshua, thank you for a very helpful response. There is no doubt that I need IT's involvement, but in all honesty, it has been hard to get them to prioritize this 'niche' problem in a 10k person organisation, when they have to deal with all day-to-day support as well. I'm trying to gather enough input from here to help narrow down their search window. And yes, the solution is definitely beyond my skillset.
      
      I have two more questions Before I go to my IT department:
      
      1 - for ESRI / Jonathan Quinn: Are there any official recommendations from ESRI regarding setting up a large Enterprise Environment with file share servers, central SDEs, and a multiple users spread out geographically? We are your typical run-of-the-mill government organisation, using windows servers (R2012)? Any official recommendations and documentation on "best practice" etc would be greatly appreciated.
      
      2 - During my benchmark test, there is a clear pattern evolving:
      If exporting a featureclass by itself takes time t, then exporting them as a group ((arcpy.FeatureClassToGeodatabase_conversion([fc1,fc2]) takes 3 x t
      This to me sounds like that system is waiting for a lock in the file system to time out. If you have any input on what to look at more specifically (OpLocks,SMB file caches and other fancy terms I have come across..), it would be very helpful.
      
      Thanks!
      Like • Show 0 Likes0
      Actions
LstAB_Herner Sep 18, 2018 5:09 AM
Mark Correct
Correct Answer
And let me just add that the fgdb, with two featureclasses is ~150kb .. and it takes 8-10 minutes to move the data (two empty feature classes) over the network. Hence my suspicion on the network protocol and/or how fgbd are 'chatty' with their operations.
Like • Show 0 Likes0
Actions
- bixb0012 @ LstAB_Herner on Sep 18, 2018 6:54 AM
  Mark Correct
  Correct Answer
  I agree that 8-10 minutes for ~150 KB is excessive, even on a high-latency network connection. That said, troubleshooting issues like this takes some thorough knowledge of the GIS software, IT in general, and access to certain types of tools. Honestly, it could be something like antivirus software or some other monitoring software your organization has deployed that is hanging it up.
  
  When I come across these issues, I usually start by doing process monitoring/profiling, using something like Process Monitor - Windows Sysinternals | Microsoft Docs , to see both what ArcGIS is doing but also other processes on the computer. I would also ask the network operations group to do package captures to see where the delays or errors are happening at the network layer. I would ask the system administrators for both the SMB/CIFS server configurations as well as what warnings or errors they are seeing with your connections.
  
  You can start to see the picture, troubleshooting is more of an art than a science, and there are no special tools or sequences of steps that will lift the rug on the root cause. If the skills or access to systems doesn't exist within the GIS shop to troubleshoot this issue, then it becomes about selling your leadership to sell leadership above them that resources need to be devoted to improving this situation.
  
  Have you tried other tools besides Feature Class to Geodatabase? If passing a list of feature classes takes longer proportionally than passing a single feature class, what about looping over the list and passing them individually?
  Like • Show 0 Likes0
  Actions
  - LstAB_Herner @ bixb0012 on Sep 19, 2018 3:51 AM
    Mark Correct
    Correct Answer
    Thanks again Joshua! Yes, I have tried different tools, read and writing from geodatabases vs shapefiles etc. In the past, I had a a script that did some common geoprocesses, select, clip, definition query, export selection and projection on relatively large datasets. I had copies of the data set up in shapefiles, SDE environment, fgdb on the network to simply try to figure out what the issue is.
    
    performance testing, same operations, different datasources; SDE, fgdb, and shapefiles. H: here is the same as J: above, i.e. a UNC direct mapping to the server, where K: is a normal dfs redirect, or whatever the technical term is..
    
    The bottom line is: file geodatabase performance is absolutely horrible in our network. I am at a loss. I am simply trying to find out here if someone has "the magic word" that I can use with my IT-department (with some reference documentation), because somewhere, someone did something 'smart' in our network configuration that causes this. We are a standard run-of-the-mill organisation in a windows server R2 2012 environment, Win7 / Win10 laptops, centralized SDE, and network drives hosted on servers.
    
    In fact, if someone here in a similar network environment is interested in doing a benchmark test with my script I would greatly appreciate it. Contact me at: kristian.herner@lansstyrelsen.se
    
    Edit: In the above, the y-scale indicates times in seconds to perform the operations, "H:" is the same as J: above, a direct UNC mapping to the network drive.
    This script answers a simply question: "can you help me find all the school building in a certain municipality and deliver it with a different projection?" - so I 'Initialize' my workspace, use 'definition query' to get all the buildings with "school" attribute from a pool of 1 million+ buildings, I 'select by attribute' the municipality polygon, 'Clip' the these two files and 'project' the result to an outfile. I then use a da.cursor to 'control' the number of objects in the outfile, to verify the script ran succesfully. This operation took about 8 seconds with shapefiles on my local drive on average and about 6-7 minutes from geodatabases on my network drive.
    Like • Show 1 Like1
    Actions
    - bixb0012 @ LstAB_Herner on Sep 19, 2018 6:24 AM
      Mark Correct
      Correct Answer
      In the past, I had a a script that did some common geoprocesses, select, clip, definition query, export selection and projection on relatively large datasets.
      ...
      somewhere, someone did something 'smart' in our network configuration that causes this
      I read those two lines as saying you have done this same workflow in the past and it ran much faster. If so, how long ago did it change? What were the speeds, ball park, when it used to run better? Between now and then, has anything changed in your GIS software? New programs, versions, etc...?
      Like • Show 0 Likes0
      Actions
      - LstAB_Herner @ bixb0012 on Sep 19, 2018 6:56 AM
        Mark Correct
        Correct Answer
        Unfortunately, nothing changed during my time here - it has always been this bad (4+ years) but people have been using shapefiles in general and havent noticed too much (except the usual "GIS is really slow!" and you see 20 layers with broken datasources in their MXDs, etc...). To the best of my knowledge, it has always been like this.
        
        With ArcGIS Pro on the horizon where everything defaults to file geodatabases we need to identify the problem and solve/bypass it before we can even consider switching to Pro.
        Like • Show 0 Likes0
        Actions
        devlavigne @ LstAB_Herner on Nov 12, 2018 11:15 AM
        Mark Correct
        Correct Answer
        I'm with Kristian on this one. I have been trying to work with a geodatabase over a network and it has been absolutely excruciating. I have tried to copy small datasets between geodatabases on my network and it takes 30+ minutes. My productivity is so poor that I now work on two things at once as simply working in GIS would allow me to get anything done.
        Like • Show 0 Likes0
        Actions
devlavigne Nov 12, 2018 11:53 AM
Mark Correct
Correct Answer
I just posted two animated Gifs of me browsing directories in Pro on another thread and thought I should post them here too. Something is either not working in Pro, or I'm missing a necessary network protocol on my server adapter to make this function as one would expect.

1. Browsing a directory in ArcGIS Pro

2. Browsing the same directory in Windows
3 people found this helpful
Like • Show 3 Likes3
Actions
- Mark.Peacey@transpower.co.nz_Transpower @ devlavigne on Jun 25, 2019 2:31 PM
  Mark Correct
  Correct Answer
  We get the exact same experience here. We're moving to a DFSR path to a local server on the network but it doesn't look like that resolves it either. It seems there's something with Pro which slows down browsing and it's noticeably worse than ArcCatalog/Map.
  Like • Show 0 Likes0
  Actions
cozgis Jul 9, 2019 6:46 AM
Mark Correct
Correct Answer
I have a GB connection to my local network Windows Server which is in my same building, and in ArcMap 10.6 & now 10.7 FGDB editing over the network is very slow and buggy. Just had some data 'disappear'. My IT team wants to know if there is a minimum or recommended network read/write speed for using file geodatabases over a networked connection or mapped drive. Also, is this even a recommended practice - using a networked drive for GIS data stored in File Geodatabases but outside of a relational/Enterprise database?
Like • Show 0 Likes0
Actions
- bixb0012 @ cozgis on Jul 9, 2019 8:31 AM
  Mark Correct
  Correct Answer
  Responding to your last question, is it recommended? I can't recall ever seeing any documentation where Esri says it is "recommended." That said, I also haven't seen any documentation where they say it is unsupported. If hosting file geodatabases and other spatial data on network drives common? I would say yes. The organization I work for has its enterprise file system for GIS file-system data on a NetApp-backed network drive.
  
  Hosting GIS data on a network drive introduces many new levels of complexity versus just hosting data on "local" drives. There are network related issues, like latency, timeouts, etc... There are SMB configuration issues, both on the client and server.
  
  I have seldom heard of network data just "disappearing." Typically, there is either an error on the server side that corrupts it (which the IT shop should be able to track down), or someone else with access to that data accidentally deletes it.
  Like • Show 0 Likes0
  Actions
  - cozgis @ bixb0012 on Jul 9, 2019 8:49 AM
    Mark Correct
    Correct Answer
    So how do you find the performance of editing FGDB over the network? Is there any difference in performance over the network with Pro and Desktop? Finally, do you know your read/write speed to the network drive?
    Like • Show 0 Likes0
    Actions
    - bixb0012 @ cozgis on Jul 9, 2019 9:27 AM
      Mark Correct
      Correct Answer
      The "performance of editing FGDB," whether locally or on a network, depends on the workflow. Are you manually editing data using the GUI in an edit session? Are you using cursors to bulk update data outside of an edit session? There are so many variables at play that it is hard for someone to make general comments on finding the performance of editing FGDB data.
      
      One of the main metrics of storage performance is IOPS (input/ouput operations per sec), especially with a shared storage environment (SAN, NAS, etc...) where there are lots of competing read and write requests. Storage is an entire specialty within IT, and I am not knowledgeable enough to be giving advice.
      
      To answer your read/write question at a basic level, writing a single 1 GB file to our enterprise file system (1000 MB x 1) gets around 150 MB/s throughput, writing one hundred 10 MB files (10 MB x 100) sequentially gets around 110 MB/s, and writing one thousand 1 MB files (1 MB x 1000) sequentially gets around 45 MB/s. The read speeds are faster, but you get a sense of the scale we are dealing with.
      Like • Show 0 Likes0
      Actions
      - cozgis @ bixb0012 on Jul 9, 2019 10:18 AM
        Mark Correct
        Correct Answer
        I am getting around 115 MB/s read and 24 MB/s write with the default test in CrystalDiskMark. The write speeds are the bottleneck in our network for sure. I recall seeing a 100 MB/s read/write recommendation for GIS use at one point either on a forum or blog, but have not been able to find it again. If there was an Esri minimum requirement that I could show to my IT team that would be very helpful, albeit this would be subject to a wide variety of contingencies depending on the types of edits as you noted above. In my situation there are three people reading data and one person editing data.
        
        I have a hunch that I would get better write speeds using PostgreSQL on a VPS in the cloud. That is the route I am likely to go at this point, at least for testing, and if it proves successful it means either dropping Arc for editing or fork over 20k for Enterprise : (.
        Like • Show 0 Likes0
        Actions
        bixb0012 @ cozgis on Jul 9, 2019 11:10 AM
        Mark Correct
        Correct Answer
        Although this page is for ArcGIS Enterprise, I would argue the statements apply to ArcGIS Desktop as well: https://enterprise.arcgis.com/en/server/latest/install/windows/choosing-a-nas-device.htm.
        Immediate consistency
        Choose a storage device that enables files to be read immediately from any node in your site once an operation and corresponding write has completed. This disqualifies many types of distributed file systems, such as GFS and DFS.
        NFS shares must be configured to ensure consistent reads and avoid using stale data caches.
        
        Performance
        Choose a storage device that performs well while incurring volumes of small, random input/output (I/O.) Consider that read and write performance can greatly fluctuate depending on the characteristics of the I/O. This is an important distinction as ArcGIS Serverand Portal for ArcGIS operations (interactions with the configuration store, cached bundle tiles, and so on) follow this pattern.
        Often this means that a device that has been optimized for large, sequential reads and writes (as often occurs with imagery and video) is unsuitable for use with ArcGIS Enterprisecomponents.
        If your implemented file storage mechanism does not handle small, random I/O well, you may experience significantly increased response times, or even failures, especially for administrative operations applied in your site.
        Like • Show 0 Likes0
        Actions
        cozgis @ bixb0012 on Jul 9, 2019 12:01 PM
        Mark Correct
        Correct Answer
        I think that answers it. Random I/O is key. I will share with our IT and go from there. I'll report back if I have any updates.
        Like • Show 0 Likes0
        Actions
        LstAB_Herner @ cozgis on Sep 24, 2019 2:15 AM
        Mark Correct
        Correct Answer
        Malcolm, any updates on this?
        Like • Show 0 Likes0
        Actions
        cozgis @ LstAB_Herner on Oct 28, 2019 10:01 AM
        Mark Correct
        Correct Answer
        We have switched to running SQL Express on a shared networked server. There are some performance issues (initial selection for editing). I have not ported everyone over to the new data so I cannot really give a good review of this setup just yet.
        Like • Show 1 Like1
        Actions
        LstAB_Herner @ cozgis on Oct 29, 2019 1:15 AM
        Mark Correct
        Correct Answer
        Thank you Malcolm! My IT department has finally given me a network technician to investigate this together, and we will start testing soon to troubleshoot the issue(s). I have referred him to this thread as well, and I'll keep this thread updated accordingly! I hope we find the core issue soon!
        
        I'm a little surprised that ESRI hasnt investigated this themselves. When I first started this thread, I was expecting a simple sheet with "recommended network settings for working with file geodatabases in a network environment" page somewhere. Instead it has been several years and plenty of unsupported workarounds in our organisation workflow.
        Like • Show 0 Likes0
        Actions
        cozgis @ LstAB_Herner on Oct 29, 2019 4:31 AM
        Mark Correct
        Correct Answer
        The best advice I have heard on this if you are set on using FGDB is to set the file permissions of the file share at the server level in windows to read only for everyone who does not need write access. This prevents a lock file from being created when reading the data by read-only viewers and might not lead to the issues I was having.
        Like • Show 0 Likes0
        Actions
        mboeringa2010 @ LstAB_Herner on Oct 29, 2019 7:00 AM
        Mark Correct
        Correct Answer
        I think the closest to your "simple sheet" with "recommended network settings for working with file geodatabases in a network environment" is ESRI's System Design Strategies Wiki pages. Unfortunately, although still partially maintained and updated with e.g. recent 2019 benchmarks, ESRI still does not directly or clearly link to it:
        
        http://wiki.gis.com/wiki/index.php/System_Design_Strategies
        
        IMHO, they should put up a permanent "Recommended Content" link to the above System Design Strategies from e.g. the main Support page...:
        https://support.esri.com/en/
        It is listed under "Other Resources" as "wiki.GIS.com" but that is totally unhelpful in realizing how valuable this content is...
        Like • Show 0 Likes0
        Actions
  - cozgis @ bixb0012 on Jul 17, 2019 11:35 AM
    Mark Correct
    Correct Answer
    Just received this message from Esri Support -
    File geodatabases are not recommended to be stored on network drives, and two or more users simultaneously viewing/editing a feature class within the database can easily create a wide range of issues...I would suggest that you copy the file geodatabase onto a local drive on your machine, or onto a SQL Server express database...
    In my case we have two viewers and one editor (myself), and I was getting very strange behavior such as randomly deleted features when other features were deleted or changed.
    Like • Show 0 Likes0
    Actions
    - bixb0012 @ cozgis on Jul 17, 2019 7:15 PM
      Mark Correct
      Correct Answer
      Malcolm Meyer, did Esri Support offer a reference/citation for that quote or was it just an analyst's opinion in an e-mail? If I were you, I would push back on Esri Support, especially the vague hand waiving of "two or more users simultaneously viewing/editing a feature class...." According to Esri's own documentation, Types of geodatabases—ArcGIS Help | ArcGIS Desktop (which is written by product teams and not support), file geodatabases are for:
      Single user and small workgroups:many readers or one writer per feature dataset, stand-alone feature class, or table. Concurrent use of any specific file eventually degrades for large numbers of readers.
      Viewing and editing data are very different beasts, so lumping them together with "viewing/editing" is ridiculous. Also, the documentation might not define "large numbers of readers," but no one can reasonably state that two is a large number. And, it isn't possible to have more than 1 editor for a stand-alone feature class, so I don't know why Esri Support is implying you can.
      
      Where is Esri Support coming up with "file geodatabases are not recommended to be stored on network drives"? According to Paths explained: Absolute, relative, UNC, and URL—Help | ArcGIS Desktop :
      In ArcGIS, you can use a UNC path anywhere a path is requested. This is particularly advantageous for shared data on a local area network (LAN). Data can be stored on one computer and everyone with access to the computer can use the data, as long as the computer is not turned off or removed from the network.
      There is no qualifier that UNC paths don't work for file geodatabases. If Esri says they support them, they need to support them even if there are some performance trade-offs when storing geospatial data on UNC paths.
      2 people found this helpful
      Like • Show 0 Likes0
      Actions
      - cozgis @ bixb0012 on Jul 18, 2019 4:54 AM
        Mark Correct
        Correct Answer
        I was not given a reference anywhere, but I can for sure tell you that problems are occurring with using File Geodatabases from our file server, with two viewers and one editor. It would be nice if they came out and acknowledged that this setup is not recommended, not that it will not work in some cases, but that depending on the local network setup issues can arise, even with a small number of users. I would also argue that this statement -
        Concurrent use of any specific file eventually degrades for large numbers of readers
        is too vague, and should read more like "concurrent use of data stored in File Geodatabases can lead to data corruption and loss when accessed by readers and simultaneously edited by another user" or something like that. What does "degrade" even mean if not data corruption? Maybe that access will be slow? Clarification would be helpful here.
        
        Previously we used shapefiles, which create a lock in order to, among other things, prevent data corruption. We did not have any data corruption issues when we used shapefiles.
        
        In all cases, regardless of the setup, having more than one user accessing a feature class does create a schema lock which will not allow an editor or viewer to add or delete fields.
        Like • Show 0 Likes0
        Actions