topic File geodatabase is getting fat extremely in ArcGIS Enterprise Questions

File geodatabase is getting fat extremely

AhmadSALEH1 — Wed, 08 Feb 2017 11:58:05 GMT

Hi,

File geodatabase are getting fat extremely.

I have a FGD with a name “U.gdb”, I have noticed that this DB has a size of 10 G.B, despite the fact that my data is not that big in size, so I decided to copy the F.C’s inside the U.gdb to a newly created geodatabase with a name “U2.gdb” the size of the newly created db decreased immediately to 148 MB.

The same data are in both GDBs while the size is extremely different. Compact and Compress File Geodatabase Data fails to reduce the size of the geodatabase.

The source of this issue might be because I have an enterprise geodatabase (U.mdf) and regularly I used to delete all F.Cs from U.gdb and copy the mdf F.Cs to U.gdb in order to update my data.

What should I do to maintain the same gdb size and avoid the over growing size if the GDB.

Thanks

Ahmad

Re: File geodatabase is getting fat extremely

JoshuaBixby — Wed, 08 Feb 2017 13:50:39 GMT

Compact doesn't do anything? That seems odd. What is the size after running compact?

Not to ask the obvious, but have you observed the Tip in the Compact file and personal geodatabases documentation:

If data from the geodatabase is open for editing, it cannot be compacted. To compact the database, remove it from the map.

Also, what about running compact after you delete the feature classes and before repopulating the geodatabase, does that make a difference versus running compact after the geodatabase has been repopulated?

Re: File geodatabase is getting fat extremely

Asrujit_SenGupta — Wed, 08 Feb 2017 13:51:04 GMT

From the images, it does not seem that both the File Gdbs have the same amount of data in them.

U seems to have 2435 files as per your image, whereas U2 has only 116 files!

Analyze it more carefully and check what these are pointing to..

Re: File geodatabase is getting fat extremely

JayantaPoddar — Wed, 08 Feb 2017 15:08:43 GMT

Looking at the number of files as mentioned by Asrujit SenGupta‌, I think there could be a lot of orphaned files in the FGDB. Not sure what's causing these files to "stay alive" even after deletion of associated features class/tables from the FGDB.

Re: File geodatabase is getting fat extremely

AhmadSALEH1 — Wed, 08 Feb 2017 15:45:44 GMT

Thanks guys,

Joshua compact fails to reduce the size even after deleting the F.C then compress and copy again! The reduction in size is about 100 MB after applying the compact.

Asrujit your right dispite the fact that thy have the same number of F.Cs, U.gdb have a lot more files inside it when you show the file properties in windows!

Jayanta your right, orphaned files is stayed alive after deleting the F.Cs ! in my daily scenario, 3 editors are connected to an enterprise geodatabase, at the end of the day I delete the old files in the U.gdb and I copy the new F.Cs form the enterprise to U.gdb. After a year my geodatabase became this big!

It seems that deleting F.C inside the geodatabase keeps some orphaned files a live and not deleted. How this can be solved.

Thanks

Ahmad

Re: File geodatabase is getting fat extremely

Asrujit_SenGupta — Wed, 08 Feb 2017 15:53:24 GMT

You should think about Replication...which will save you the time spent behind deleting and copying the FCs everyday.

If Replication is not what you want, then why not simply delete the Old File gdb and create a new File gdb with same name everyday, then copy the FCs. Hardly a minutes work.

Re: File geodatabase is getting fat extremely

MichaelVolz — Wed, 08 Feb 2017 16:16:37 GMT

Going with Asrujit's 2nd suggestion (not replication), you could script this out and run it everyday as a scheduled task. One thing to note is if the data is being used by an ArcGIS Server mapservice as that could put locks on the data.

Re: File geodatabase is getting fat extremely

AhmadSALEH1 — Wed, 08 Feb 2017 16:31:44 GMT

Thanks again guys,

Asrujit, Replication is not thing that I want to do in my case, it needs a lot of efforts and time and its useless, imagine that it fails to reflect the edits if they were more than 1000 ! this is one of many limitations that it has.

Michael you got the point, I run arcgis server so the data is locked you can’t delete the geodatabase without turning off the Arcgis server, then my entire system (web mapping application, arcgis server) will be down. What should I consider else?

Thanks

Ahmad

Re: File geodatabase is getting fat extremely

Asrujit_SenGupta — Wed, 08 Feb 2017 16:37:32 GMT

The Bug that you are talking about, that has been fixed a long time back. Replication was the best solution to your scenario, however as I mentioned, if that is not what you are comfortable with, then go for other solutions.

How many records do the FCs have? If they are not too big, then you can automate Truncate of the FCs(basically deleting all records from the FCs in the File gdb) and then load the updated data from the sde FCs.

Re: File geodatabase is getting fat extremely

AhmadSALEH1 — Wed, 08 Feb 2017 16:58:58 GMT

Hi Asrujit,

Thanks for the prompt reply, glade to hear that this bug has been fixed, as I remember there was another 2 major issues with it.

My F.C has almost 100K records, I will try to Truncate of the FCs and see how it works, but I am still curious about the size issue I hope to find an answer let’s wait maybe one of esri’s developers can give a clear answer for this issue

Thanks

Ahmad

Re: File geodatabase is getting fat extremely

JoshuaBixby — Wed, 08 Feb 2017 17:13:06 GMT

asrujit_pb‌, your comment about truncating raises an interesting question, at least for me. With enterprise database systems (Oracle, SQL Server, Postgres, etc...), there are notable differences between truncate and delete. Truncate can remove large numbers of records much faster than delete, but it does so by bypassing some of the integrity mechanisms. With the simplified database engine in file geodatabases, I don't really have any idea what the differences are between using delete and truncate. I assume there are differences, but I have never seen them documented or discussed. I might have to experiment sometime....

Re: File geodatabase is getting fat extremely

Asrujit_SenGupta — Wed, 08 Feb 2017 17:27:01 GMT

Personally I have not observed any difference as such, other than the faster performance with Truncate Table tool.

Re: File geodatabase is getting fat extremely

MichaelVolz — Wed, 08 Feb 2017 17:30:18 GMT

Ahmad:

I would go with the truncate and append method to update your file gdb as that method bypasses locks from AGS mapservices. I have been going with this methodology for years with success.

I would make rename your existing excessively large file gdb (backup) and create a new clean file gdb that you could then use in the new truncate and append script tied to a scheduled task. This would clean-up your production file gdb and bypass AGS locks on the data at the same time. The scripting is quite minimal.

Re: File geodatabase is getting fat extremely

AhmadSALEH1 — Thu, 09 Feb 2017 06:17:24 GMT

Hi Michael,

truncate and append seems to be a good choice, I have tried them truncate is very fast, I will shift my work procedure to use truncate rather than delete and append rather than copy.

but I am also interested about the size of the geodatabase U.gdb why its getting fat? is the normal delete procedures are not enough to clear the geodatabase!

Thanks

Re: File geodatabase is getting fat extremely

Asrujit_SenGupta — Thu, 09 Feb 2017 06:24:53 GMT

Will suggest contacting Esri Tech Support, so that they can take a look at the concerned File gdb and analyze it. We here can just guess what is wrong with it without taking a look at the geodatabase itself!

Re: File geodatabase is getting fat extremely

NicholasGraf — Fri, 30 Jun 2017 16:08:15 GMT

I am noticing the same thing. We have a File Geodatabase that is used as part of data sharing between agencies, and is updated by a script nightly. What I have found is that even though I delete the previous feature class, the "record" stays in the file structure. Looking at the files within the gdb folder I see "a00000009.gdbtable", "a00000012.gdbtable", "a00000014.gdbtable" are all about the same size, but have a one day different in the date modified. So every night by database basically get new data added, but the old data lives in the shadows.
I have tried both compress, and compact and they make little to no difference.

Has this been reported to esri?

Re: File geodatabase is getting fat extremely

NicholasGraf — Mon, 10 Jul 2017 14:14:03 GMT

I have an update: I am using a script to update features in the FGBD that are used in services. I was not stopping the services, as they were set to not create a schema lock. Once I stopped all the services using the data, the FGDB remained small. There does NOT seem to be a way to reduce the size once the "inflation" has occurred, other than creating a new FGDB

Re: File geodatabase is getting fat extremely

AhmadSALEH1 — Tue, 11 Jul 2017 14:19:26 GMT

Nicholas,

I think this is the correct answer, I used to do the same as you deleting the F.C inside the geodatabase without stopping the services to avoid downtime for the services. it seems the the ArcMAP allows you to delete the F.Cs but they still alive in the background to feed the services.

Re: File geodatabase is getting fat extremely

MajdoleenO_A__Awadallah — Thu, 13 Jul 2017 05:39:32 GMT

Thank you Nicholas, and Ahmad,

Do you have any idea about the MDF itself? the issue is posted here by Jamal Numan: https://community.esri.com/message/678613-the-size-of-mdf-file-is-much-bigger-than-its-gdb-file

I will try your suggestion and get back to you.

Thank you

Best,

Majdoleen

Re: File geodatabase is getting fat extremely

JamalNUMAN — Fri, 14 Jul 2017 06:26:03 GMT

Hi All,

Do you mean that we need to stop the service in question before transferring the data from mdf for gdb? I think that this is a disadvantage which will cause inconveniences for users and minimize the chance to be 24 hours online with no downtime.

What do you think?