File geodatabase is getting fat extremely

8752
32
Jump to solution
02-08-2017 03:58 AM
AhmadSALEH1
Occasional Contributor III

Hi,

File geodatabase are getting fat extremely.

I have a FGD with a name “U.gdb”, I have noticed that this DB has a size of 10 G.B, despite the fact that my data is not that big in size, so I decided to copy the F.C’s inside the U.gdb to a newly created geodatabase with a name “U2.gdb” the size of the newly created db decreased immediately to 148 MB.

The same data  are in both GDBs while the size is extremely different. Compact and Compress File Geodatabase Data fails to reduce the size of the geodatabase.

 

The source of this issue might be because I have an enterprise geodatabase (U.mdf) and regularly I used to delete all F.Cs from U.gdb and copy the mdf F.Cs  to U.gdb in order to update my data.

 

What should I do to maintain the same gdb size and avoid the over growing size if the GDB.   

Thanks

Ahmad

Tags (2)
32 Replies
CalvinHarmin
New Contributor III

I'd also like to add that I've been having this problem at my local government, after discovering three of my common file geodatabases (fgdb) that we use to be our main "read-only" fgdb's became bloated to many gigabytes in size. They normally are 100-500MB when "normal". I did extensive testing, and found that...

-> Junk data is generated whether or not the fgdb is actively connected to published web services

-> Junk data is generated with Truncate & Append, CopyFeatures, and FeatureToFeature geoprocessing tools

-> Doesn't matter if it's from a python script or through ArcMap/Catalog GUI. 

-> It is not consistent -- the nightly overwrites generate junk data, but if I manually run overwrites immediately in succession, the amount of junk data produced is less. But if I wait some period of time, the amount of junk is greater. Sometimes junk data (bloated size) is created in one of the fgdb tables, as viewed in file explorer, but no extra files are actually created. Sometimes many new files are created. 

-> I opened a ticket with ESRI and they have no solution, they actually pointed to this thread as one of the few addressing the bug. 

-> My only solution is to rebuild the fgdb's periodically. 

-> Replication is not an answer for me because that requires all data to be coming from an SDE, my data comes from multiple sources, some of which are not SDE. 

-> I've also tested compressing. When overwriting using truncate/append on compressed data, I uncompressed, truncate/appended, then recompressed. Successive runs of this STILL produced bloated data. 

abdullahqasrawi
Occasional Contributor

Dear Calvin Harmin

Try the tool in the link below that posted by ahmed saleh, this tool will transfer the feature class from mdf to gdb, but you have to note that sde name is as same as gdb name.

I used the tool and i found that the size of gdb became normal.

https://community.esri.com/message/732714-re-a-generic-tool-to-convert-multiple-mdf-files-to-gdb?com...


See Screenshot below

CalvinHarmin
New Contributor III

Thanks, but I don't really see how that addresses any of these bloat issues. The bloat data comes from overwriting of the feature classes in a file geodatabase, repetitively, over time. Making a new fgdb and importing the layers will of course be the correct/normal size, so that's not a surprise. 

(edit: .MDS is personal geodatabase, .MDF is a SQL Server database file) From your comment I just want to clarify that SDE (enterprise geodatabase) is not the same as MDF (personal file geodatabase). It looks this process just creates a new fgdb (file geodatabase) as far as I can tell, rather than addressing the overwrite/bloat issue.

I've written a script to automate rebuilding my fgdb's when needed (when it gets ridiculously large at some point), so I can work around the issue but it's just a bit annoying. Not all of my data comes from an SDE source, and the bloat issue is not necessarily an SDE issue. It also occurs when I tested local fgdb to local fgdb.

Asrujit_SenGupta
MVP Regular Contributor

*.MDB is actually a Personal geodatabase (Access Database)

MDF is the datafile for a SQL Server database. The use of the term MDF is actually not correct in that other thread, as one cannot copy data from an MDF file directly to a File gdb. They are rather copying data from the SDE Connection(s) to a File GDB (FGDB).

CalvinHarmin
New Contributor III

Ah thank you, that makes more sense now why MDF was being referred to in that thread. I mixed up my file extensions, and was wondering why an access database would have much relevance there

JamalNUMAN
Legendary Contributor

At the end of the day, we are transferring data from mdf file to gdb file. The issue here is that the gdf gets really fat as data is deleted and re-copied from its corresponding mdf file (via the sde connection)

----------------------------------------
Jamal Numan
Geomolg Geoportal for Spatial Information
Ramallah, West Bank, Palestine
0 Kudos
CalenDaugherty
New Contributor III

Just wanted to chime in on my experience with this issue. Whenever I use a tools like Delete Features - Append, the file geodatabase bloats up in size. If I am running a Python script every night doing something like this, then the size can grow quickly. A fgdb that started as 20MB can grow to a GB or more. I don't notice much of a performance issue in ArcMap or ArcGIS Pro, but scripts running against the data can take progressively longer. Compact will shrink the fgdb by a bit, but not by much. It is the xxxx.gdbtable file size that grows . . . its like orphaned data is left behind each time. The only way I have been able to do repetitive processes without this occursing is to include a step to completely delete the features class, and then bring in the data, rather than doing delete features - append. I would be interested to hear if anyone has come across a solution to help with this.

JoshuaBixby
MVP Esteemed Contributor

Interesting that Compact doesn't work well for you, we have really good results using that tool to clean up and shrink the size of bloated FGDBs.

CalenDaugherty
New Contributor III

Thanks for the reply. The compact does work in some instances. It seems like when I try to use it with a fgdb that is a data source for a map service, there are issues. I know the map service must be stopped for compact to run . . . the compact will run successfully, but it almost as if even though the map service is stopped, something is still preventing compact from getting rid off all the bloated tables. I would have to do some more testing to pinpoint an exact scenario.

NathanHeickLACSD
Occasional Contributor III

Thanks for the thread!  I also have various publishing scripts that perform the basic concept of a truncate and load.  I did some research on the file geodatabases and I found a few things:

  1. If I turn on the Size attribute in ArcCatalog, the size of the file geodatabases matches the size of the data in the folder.
  2. If I add up the sizes of the objects in the file geodatabase, the sizes are generally significantly lower than the size of the geodatabase.
  3. If I run Compact on them, some of the geodatabases shrink significantly and the tables inside also shrink significantly.  However, others only shrink a moderate amount, if at all.
  4. If I copy the data to a new file geodatabase, I get even better results than Compact.  The tables become an appropriate size.
  5. I had the issues for scripts with map services locking the data and without.  I had the issue on a script that only called DeleteFeatures and loaded new data using an InsertCursor.  Therefore, I suspected that the problem is with DeleteFeatures.
  6. I looked in the problem file geodatabases and I noticed a lot of similarly sized .gdbtable files separated by a day going back several months.  Each run of the script appeared to be creating a new .gdbtable file for each table.
  7. I ran DeleteFeatures with Windows Explorer open.  The number of files in the problem FGDB would increase by one.  When I tried to test this again starting from scratch with a new file geodatabase and a GDB feature class, I could not reproduce it.
  8. I ran TruncateTable with Windows Explorer open.  The number of files did not change.

I would need more time to test, but I believe that DeleteFeatures may be creating a new empty geodatabase table and possibly leaving the old table in the FGDB folder.  Those files are building up over time.  It doesn't happen consistently, so I am not sure what the cause is.  I don't think it is service locking because it happened without a service.  It looks like replacing DeleteFeatures with TruncateTable will solve the problem.  Of course, I don't have hours and hours to troubleshoot this issue or work with Technical Support to identify the issue, so I might be off.  Still, try TruncateTable and look in your FGDBs and let me know if you're seeing similar behavior.  I suspect that it might be related to creating the FGDB in an older version of ArcMap and then running it in a newer version of arcpy.  I've even seen differences between ArcMap and arcpy for the same tools, so that might be it as well.  In addition, antivirus software may be in play here.  It has been interfering with python.exe on certain tools.

0 Kudos