topic Re: imported Feature Class size in Data Management Questions

imported Feature Class size

DanielSmith — Wed, 30 Oct 2013 18:46:58 GMT

Hello,

I have a contour feature class stored in a file geodatabase. when looking at the contents tab of the geodatabase in ArcCatalog i see that the size of the file is 4GB.

I am attempting to store this data in an SDE database (SQL Server Express 2008 R2, so a 10GB limit). When I import the data (either using feature class to feature class, right-click > Import, or creating a blank feature class and loading it) the mdf file bloats to 10 GB and it fails to import.

Why does this 4 GB file bloat and blow up my SDE database? is there a way i can import this contour data set into my SDE instance?

Any thoughts, suggestion, or guidance is greatly appreciated.

-Daniel

Re: imported Feature Class size

VinceAngelo — Wed, 30 Oct 2013 19:59:21 GMT

There are probably a number of things going on here:

Databases format differently than file geodatabase

Databases index differently than file geodatabase

File geodatabase supports compression

But first that are a number of issues to address:

Are you using an Enterprise ArcSDE in the SQL-Server Express?

What is the default storage format?

How is your coordinate reference (specifically xyscale) defined?

What is the physical size of the file geodatabase directory (according to Explorer)?

I'd expect a File Geodatabase to be faster (especially after you compress it) than any RDBMS-
based geodata, so I'm wondering why you are even on this path. Do you really need multi-user
editing on a contours layer?

- V

Re: imported Feature Class size

DanielSmith — Wed, 30 Oct 2013 20:25:41 GMT

There are probably a number of things going on here:

Databases format differently than file geodatabase

Databases index differently than file geodatabase

File geodatabase supports compression

yeah, i agree on the formatting, indexing, and compression. I did not expect a > 2x expansion though.

But first that are a number of issues to address:

Are you using an Enterprise ArcSDE in the SQL-Server Express?

What is the default storage format?

How is your coordinate reference (specifically xyscale) defined?

What is the physical size of the file geodatabase directory (according to Explorer)?

I'd expect a File Geodatabase to be faster (especially after you compress it) than any RDBMS-
based geodata, so I'm wondering why you are even on this path. Do you really need multi-user
editing on a contours layer?

- V

SQL Server Express

i have not modified the storage format for SDE so what ever the default would be. what is the best way to determine storage format?

Do you mean the resolution? its defined on import by the input feature class. additionally it has z coordinates stored in the geometry.

the entire GDB in explorer is ~8GB but it includes all the individual tiles that comprise the merged data set

Its less about the multi-user editing and more about using 1 way replication to subcontractors via geodata services. It is much faster in the file GBD you are correct there. maybe a better plan would be to host the static data from a file geodatabase and allow extraction of the data instead of using replication and keep the more dynamic data in SDE. There is some data that really wont be edited and some that will be actively edited hence the static/dynamic distinction.

Re: imported Feature Class size

VinceAngelo — Wed, 30 Oct 2013 21:19:12 GMT

Are you using Enterprise ArcSDE or Personal/Workgroup ArcSDE in the Express instance?

I think the default is now GEOMETRY storage, which would explain the significant storage
difference.

- V

Re: imported Feature Class size

DanielSmith — Wed, 30 Oct 2013 21:32:57 GMT

Are you using Enterprise ArcSDE or Personal/Workgroup ArcSDE in the Express instance?

SQL Server Express 2008 R2

I think the default is now GEOMETRY storage, which would explain the significant storage
difference.

this seems to be the case. i checked an imported data set and the shape data type is GEOMETRY. this matches the result of manually setting config keyword to GEOMETRY when creating a new blank feature class.

Re: imported Feature Class size

DanielSmith — Wed, 30 Oct 2013 21:57:27 GMT

I think the default is now GEOMETRY storage, which would explain the significant storage
difference.

just checked the db_tune table and this is the case.

Re: imported Feature Class size

VinceAngelo — Wed, 30 Oct 2013 22:11:23 GMT

Are you using Enterprise ArcSDE or Personal/Workgroup ArcSDE in the Express instance?

- V

Re: imported Feature Class size

DanielSmith — Wed, 30 Oct 2013 22:13:00 GMT

Are you using Enterprise ArcSDE or Personal/Workgroup ArcSDE in the Express instance?

- V

hahahah. sorry. It would be Enterprise, licensed through ArcGIS for Server.

Re: imported Feature Class size

VinceAngelo — Wed, 30 Oct 2013 22:22:09 GMT

You've got more administration options with Enterprise, which is why I was asking.

You can try changing the default storage type to SDEBINARY, though I don't know
what sort of storage difference it will make (Microsoft uses a compressed integer
representation as well).

Of more significance, with SDEBINARY you can back off the default submillimeter
coordinate resolution to a decimeter and probably reduce the feature storage
significantly (at a precision cost, of course). [You can make that change in
GEOMETRY storage as well,but it doesn't use the coordref to calculate the integer
representation, so that wouldn't have a storage impact.]

- V

Re: imported Feature Class size

DanielSmith — Wed, 30 Oct 2013 22:49:41 GMT

OK cool, I will check out SDEBINARY and mess with the resolution.

Thanks for your help

Re: imported Feature Class size

DanielSmith — Thu, 31 Oct 2013 00:01:12 GMT

V,

Thank you, that worked pretty well and the feature was loaded and has a much smaller impact on the database size.

the contour data that we have was generated by our vendor from a LiDAR point cloud. I'm gonna check on the accuracy of their collection to better justify the decreased resolution of the coordinate system. I am sure that they are no where near the default accuracy so reducing the resolution of the storage should just be a matter of striking a happy medium between collection precision, contour accuracy, and coordinate system resolution.

Re: imported Feature Class size

MarcoBoeringa — Thu, 31 Oct 2013 10:59:15 GMT

I have a contour feature class stored in a file geodatabase. when looking at the contents tab of the geodatabase in ArcCatalog i see that the size of the file is 4GB.

the contour data that we have was generated by our vendor from a LiDAR point cloud. I'm gonna check on the accuracy of their collection to better justify the decreased resolution of the coordinate system.

That is a rather huge vector Feature Class by any standard. You really have to wonder, especially if this contour data was auto-generated from a highly detailed raster DEM or LIDAR point cloud, if there isn't a huge amount of bloat in this dataset in terms of excess polyline vertices. Usually, with auto-generated contours, a generalization step is in order and can often reduce storage constraints considerably, while maintaining the basic quality of the dataset. There is often way to much detail and vertices added to the contours in the contour generation process.

Generalization will probably also mean you don't have to reduce storage precision, while still achieving a huge reduction in dataset size.

hahahah. sorry. It would be Enterprise, licensed through ArcGIS for Server.

I really think you should be looking at getting some sort of "enterprise" type license for your SQL Server as well. It is a waist of money to have an Enterprise license for ArcGIS for Server, while using SQL Server Express as the back-end.

Re: imported Feature Class size

DanielSmith — Thu, 31 Oct 2013 15:23:39 GMT

That is a rather huge vector Feature Class by any standard. You really have to wonder, especially if this contour data was auto-generated from a highly detailed raster DEM or LIDAR point cloud, if there isn't a huge amount of bloat in this dataset in terms of excess polyline vertices. Usually, with auto-generated contours, a generalization step is in order and can often reduce storage constraints considerably, while maintaining the basic quality of the dataset. There is often way to much detail and vertices added to the contours in the contour generation process.

Generalization will probably also mean you don't have to reduce storage precision, while still achieving a huge reduction in dataset size.

yes, i agree. I had tested generalization using bend simplification and point remove and the results were really great, bend simplify had what i considered to be better results. much faster and much smaller in the end.

unfortunitely the powers that be did not want the data manipulated from the what was delivered. i will have to bring this up with them again. Somewhere accuracy is going to be lost (from the original delivery) if they want it in SDE. may it be in the resolution of the coordinates or in the reduction of verticies.

Re: imported Feature Class size

MarcoBoeringa — Thu, 31 Oct 2013 19:51:28 GMT

unfortunitely the powers that be did not want the data manipulated from the what was delivered. i will have to bring this up with them again. Somewhere accuracy is going to be lost (from the original delivery) if they want it in SDE. may it be in the resolution of the coordinates or in the reduction of verticies.

I can think of no reason not to generalize contours from a (LIDAR) DEM or point cloud. If you want the accuracy of the original data, start using the original data...

I see little value in terms of analytics, or scientific analysis, in height contours putting such severe constraints on accuracy that generalization would be out of the order.

Height contours are usually just cartographic (but maybe there's the culprit, and your customer is a cartographic agency afraid for quality loss compared to traditional "hand-drawn" photogrammetric contours...)

If the latter is the case, it maybe of help to you to hear that here in the Netherlands, the major cartographic agency responsible for country wide topographic maps (the "Kadaster"), actually just managed a unique achievement, that even got them an ESRI "Special Achievement in GIS" award: implementation of a fully(!) automated cartographic generalization process based on a huge - 400+ models - custom build ArcGIS Modelbuilder / FME suite that creates 1:50.000 maps from 1:10.000 without human intervention.

This seems a world's first and quite a colossal achievement, especially since the 1:50.000 auto-generated map is fully replacing the existing "manually generalize" production line for 1:50.000. There is no other cartographic agency who seems to have achieved this yet, although many have research projects in this direction.

A Dutch article about this:
http://www.gdmc.nl/publications/2012/Automatische_generalisatie.pdf

And from an online publisher an English language article in "Cartography and Geographic Information Science":
http://www.tandfonline.com/doi/abs/10.1080/15230406.2013.824637#.UnK83-L27zs

Re: imported Feature Class size

DanielSmith — Thu, 31 Oct 2013 21:24:36 GMT

I can think of no reason not to generalize contours from a (LIDAR) DEM or point cloud. If you want the accuracy of the original data, start using the original data...

i could not agree more! I continually seem to deal with folks' inclination that GIS data, no matter the source, IS correctly and accurately reflecting reality and not a just a representation.

I see little value in terms of analytics, or scientific analysis, in height contours putting such severe constraints on accuracy that generalization would be out of the order.

Height contours are usually just cartographic (but maybe there's the culprit, and your customer is a cartographic agency afraid for quality loss compared to traditional "hand-drawn" photogrammetric contours...)

This is exactally the point i have tried to make repeatidly, though no where near as elegant. And no, the end users are not cartographers by any means. Rather more consultants.

If the latter is the case, it maybe of help to you to hear that here in the Netherlands, the major cartographic agency responsible for country wide topographic maps (the "Kadaster"), actually just managed a unique achievement, that even got them an ESRI "Special Achievement in GIS" award: implementation of a fully(!) automated cartographic generalization process based on a huge - 400+ models - custom build ArcGIS Modelbuilder / FME suite that creates 1:50.000 maps from 1:10.000 without human intervention.

This seems a world's first and quite a colossal achievement, especially since the 1:50.000 auto-generated map is fully replacing the existing "manually generalize" production line for 1:50.000. There is no other cartographic agency who seems to have achieved this yet, although many have research projects in this direction.

A Dutch article about this:
http://www.gdmc.nl/publications/2012/Automatische_generalisatie.pdf

And from an online publisher an English language article in "Cartography and Geographic Information Science":
http://www.tandfonline.com/doi/abs/10.1080/15230406.2013.824637#.UnK83-L27zs

So you're pretty much my new hero for thorough post replies now. If you were invloved in the above, kudos to you! That is a pretty amazing achievment.

Cheers,

Update: for anyone reading this far vangelo answered my question, but mboeringa2010 solved my problem! MANY THANKS TO BOTH OF THESE USERS.

Re: imported Feature Class size

MarcoBoeringa — Fri, 01 Nov 2013 06:40:20 GMT

So you're pretty much my new hero for thorough post replies now. If you were invloved in the above, kudos to you! That is a pretty amazing achievment.

No, I wasn't involved in this particular project. I just thought it worth mentioning, as I just discovered these articles myself (I wasn't aware of this ongoing project), and thought it worth mentioning as being quite a remarkable achievement.

I have in the past been involved in other major projects here in the Netherlands for the Ministry of Transport, including a country wide highly detailed (about 1:10.000 scale) traffic noise modelling program and database implemented using ArcGIS, geodatabase versioning and a custom 3D traffic noise modelling core build by our project partner.