topic Geodatabase extremely slow in Data Management Questions

Geodatabase extremely slow

TracyDash — Mon, 24 Feb 2014 14:26:02 GMT

Hi,

I have a huge and growing geodatabase (10,000 fields now) and it's connected to a basic entry and search form for my coworkers to enter data with. Both have gotten extremely slow lately and I'm not sure what to do. I've compacted the database and deleted excess data. Right now it takes a good three or four minutes to simply OPEN. Submitting information also takes a very long time. Any ideas as to how to make it faster??

Thanks!

Re: Geodatabase extremely slow

GISDev1 — Mon, 24 Feb 2014 14:46:33 GMT

We need more details. What geometries? Polygons, lines, points? What kind of extents? Where is the File GDB located? A network drive, local drive? What kind of network speed do you have if it's a network drive? Are you accessing the File GDB via the API at all or just in ArcMap/Catalog? Did you notice when the slowness started or was it just randomly 1 day?
Have you tried copy pasting all of the feature classes into a brand new File GDB? What version is the File GDB and what version are the clients? If you have multiple people editing, that should really be done in SDE, from what the Esri Documentation states.

Re: Geodatabase extremely slow

TracyDash — Mon, 24 Feb 2014 14:56:30 GMT

Sorry, should have been more specific.

-FGDB stored on fast server and accessed by a dozen people
-two point feature classes and five tables (one point class is the extremely large one)
-accessing the fgdb via the api and arcexplorer (although very few have explorer) and I'm the only one with arcmap/catalog
-slowness comes and goes: the program might be fine for a few weeks and then it's hit with a sudden slowness. however, it was slow two days last week and today

I've tried copying the actual GDB and using that one but it does no difference. Would copying feature classes into a new gdb yield a different result?

The FGDB doesn't need to be upgraded (arcmap is 10.1)

Re: Geodatabase extremely slow

JacobBoyle — Mon, 24 Feb 2014 17:16:38 GMT

Due to the nature of how Arc keeps tracks of changes(adds and deletes mostly), are you running a regular compact on the database from ArcCatalog?

Compacting the database regularly cleans up the underlying system tables and should vastly improve performance.

Re: Geodatabase extremely slow

TracyDash — Mon, 24 Feb 2014 17:57:52 GMT

Jacob,

Yes, I compact when I remember every few weeks. Just did it this morning. How often do you recommend compacting a large gdb?

Re: Geodatabase extremely slow

GISDev1 — Mon, 24 Feb 2014 18:08:15 GMT

So, it's a Network File GDB edited by a dozen people? I'd say the best answer is to switch to SDE. I bet it would make your life much easier as the admin.

Re: Geodatabase extremely slow

TimHayes — Mon, 24 Feb 2014 18:42:53 GMT

A File GDB used in this way is likely the cause of the slowness. I would switch to using a SQL Server Enterprise Geodatabase (SQL Server Express 2008 R2 and ArcGIS for Server 10.2.1 Enterprise Basic or Standard) with Versioning, maybe each user would have their own Version.

10,000 fields? Have you considered separating out these fields into different tables and relating them to each other? this will be easier to manage. From this you can generate Views (as long as you are using the Enterprise Geodatabase in SQL).

SQL Server Express 2008 R2 is a free download.

Re: Geodatabase extremely slow

TracyDash — Mon, 24 Feb 2014 19:32:21 GMT

I have thought about separating them into different tables...just not sure how to go about that yet.

But GIS Dev and Tim, thanks! All good advice. I'll definitely look into it tomorrow.

Re: Geodatabase extremely slow

MarcoBoeringa — Mon, 24 Feb 2014 20:30:53 GMT

Hi,

( I have a huge and growing geodatabase 10,000 fields now) and it's connected to a basic entry and search form for my coworkers to enter data with. Both have gotten extremely slow lately and I'm not sure what to do. I've compacted the database and deleted excess data. Right now it takes a good three or four minutes to simply OPEN. Submitting information also takes a very long time. Any ideas as to how to make it faster??

Thanks!

-two point feature classes and five tables (one point class is the extremely large one)

10,000 fields? Have you considered separating out these fields into different tables and relating them to each other? this will be easier to manage. From this you can generate Views (as long as you are using the Enterprise Geodatabase in SQL).

I have thought about separating them into different tables...just not sure how to go about that yet.

Tracy,

I can not even begin to fathom what enterprise data-model would require 10.000 fields to track attributes in just a handful of tables... Are you sure "fields" shouldn't be "records"?... How would you even handle and make sense of 10.000 attributes if you had to base some sort of decision or calculation on it?

Can you please give some hints as to the kind of data these "tables" and "fields" contain?

Shouldn't you possibly be pivoting the table to make records out of fields to make a sensible data-model of this?

Marco

Re: Geodatabase extremely slow

TracyDash — Tue, 25 Feb 2014 11:12:02 GMT

Marco

I meant records. Sorry.

Digitizing every job a surveying company has done for the last two decades. You have no idea how much data goes into that.

Re: Geodatabase extremely slow

MarcoBoeringa — Tue, 25 Feb 2014 20:17:35 GMT

Marco

I meant records. Sorry.

Digitizing every job a surveying company has done for the last two decades. You have no idea how much data goes into that.

Tracy, 10.000 (10k) records is nothing, this should be blazing fast in ArcGIS, and certainly not take minutes to open. There are spatial databases far larger than that around.

I do see a potential issue with the way the data is digitized. Are you using some sort of "stream-digitizing" mode, automatically generating points along lines on scanned maps? This can lead to vast amounts of vertices, which could explain bad performance. You may need to weed out excess vertices using generalization, for example using the Simplify Line tool. For more options, see the An overview of the Generalization toolset Help page.

I do appreciate the amount of work you are facing. I was involved in a major project here in the Netherlands regarding large scale (1:1000) photogrammetrically and survey derived datasets of the entire highway system of the Netherlands, with over 300+ object types recognized. The amount of data at these detailed large scales can be staggering...