Select to view content in your preferred language

GDB Performance Concerns

1456
7
08-17-2011 01:22 PM
Labels (1)
MichaelErlich
Emerging Contributor
We converted some shape files that cover 3 states in the U.S. from TeleAtlas data consisting of 11 layers (highway exits, institutions, landmarks, highways, railroad, streams, streets, parks, water, counties, and cities).  The zoom level of when to show each of the layers was optimized to where ArcMap could perform well (set to a lower minimum than we would probably use in production).  The shape files were converted to the geodatabase format (using feature class to geodatabase conversion tool).  After the mpk was created, it took 55 seconds to load the map from the time the application loaded to the time the maps were actually displayed.  To do a query using the QueryTask by searching for a city name in the streets layer, it took 40 seconds.

Are there other formats the data can be converted to for better performance, or does the control not handle large data-sets?

These tests were performed on a dual quad-core (2GHZ each) with 8GB RAM on an x64 machine.  Our typical use case is for this to run on client machines with the data locally with a single processor (1.2GHZ) with 1GB RAM.
0 Kudos
7 Replies
MichaelBranscomb
Esri Frequent Contributor
Hi,

RE: GDB Performance:

The File GDB generally offers excellent performance, particularly for larger datasets. As a side note - when creating Map Packages, if you choose not to reference the data (the default option) the packaging process will convert all your datasources to a File GDB anyway. To leave your data in it's original format, you should choose to reference the data.

My first suggestion would be to look at what user interaction is required with your 11 layers (highway exits, institutions, landmarks, highways, railroad, streams, streets, parks, water, counties, and cities). If any of these are relatively static layers and could constitute a more seamless basemap then you should consider creating a map to specifically use as a basemap and generate a Tile Package (TPK) from that map. Then, layers which a more dynamic and need to be dunamically rendered on each pan/zoom or are required to support user interaction (identify, query, editing) should be added to a separate map which is saved as a Map Package. The 55 seconds, which sounds a long time) is actually just the initial local map service initialization time and it's possible to spin up these local services behind the scenes and add them to the map when ready.

The following code, which you could put in the window constructor, will asynchronously kick off the LocalMapService initialization then once the service startup is complete the Action will execute which in this example creates a new LocalArcGISDynamicMapServiceLayer based on that LocalMapService and inserts it into the map (note the path below is relative to my app executable):

LocalMapService.GetServiceAsync(@"Maps_and_Data\OperationalLayers.mpk", (localMapService) =>
{
LocalArcGISDynamicMapServiceLayer localArcGISDynamicMapServiceLayer = new LocalArcGISDynamicMapServiceLayer()
{
Path = localMapService.Path,
ID = "SpecialEventPlanning",
ImageFormat = ArcGISDynamicMapServiceLayer.RestImageFormat.PNG32,
PanBuffer = PanBufferSize.Medium
};
MapControl.Layers.Insert(1, localArcGISDynamicMapServiceLayer);

RE: Query Performance:

I would expect it to be significantly faster than 40 seconds - do you have attribute indexes built on the fields that you are searching? Could you consider building an ArcGIS Locator for searching for street locations?

RE: Specs:

- Single CPU - is that multi core or single core? (You'll get more benefit from processors with multiple-cores / Hyper-threading, as well as multiple processors).
- 1 GB RAM - do you know how much free memory you'll have after the OS and other business apps are taken into account? 


Cheers

Mike
0 Kudos
MichaelErlich
Emerging Contributor
Converting to a tile pack was very fast on startup.  However, the main concern is that the lost functionality will include losing the ability to turn on/off layers at runtime, and changing the display colors of the layers at runtime (when supported).

Also, with the 55 second cold start time, my concern is that when we start including larger areas, the time will increase.  This test consisted of using 3 states.  However, to meet customer demands, we will need to increase any where from 10-48 states.  So, I'm guessing at that point that 55 seconds is going to look lightening fast.  Is this something being worked on to improve the performance for larger .mpk files?  Or, are these the metrics I can expect by release time?
0 Kudos
MichaelErlich
Emerging Contributor
Mike,

After playing around with the mpk's a little more, I see a lot of movement in the hard-drive size indicator when the application starts up.  This begs the question of what exactly is the runtime service doing on startup?  It appears that most of the time is spent uncompressing the mpk and possibly copying the contents somewhere (haven't investigated to that length).  Hence, it seems that this could be simplified drastically to improve performance by tracking a hash code (or some other parameter) and ignore repeating this processing on every startup if the data hasn't changed.


Another question with the tilepacks...  For our needs, we need to have the ability to turn layers on/off at runtime.  The performance bottlenecks (pending above questions) with mpk's concerns me.  I have confirmed that I can overlay a tilepack over another tilepack (as the unfilled areas are set to transparent).  With large datasets, what is the performance degradation with having many tilepacks loaded (each with a different layer)?  Wondering if you have any metrics available to cut back the 10's of hours that will be needed to create the tile packs with large mapsets for this test?  Or, any other suggestions on how to use multiple layers so that any can be turned on/off at will?  The 55 second startup time (more on larger mapsets) will not work in our situation and we must have better performance before this control can be considered adequate.

Thanks,
Mike
0 Kudos
MichaelErlich
Emerging Contributor
Another concern to add to the above is that if we have to go the route of creating several tile-packs, the size required for the files will be extremely large.  The tilepack for one city (approx. 7x14 square miles) with half of the production layers is 1.1GB.  Started a tilepack that includes all necessary zoom levels which includes 3 states.  So, I should know the size of that one in another day after processing.  But, I'm assuming that if we split layers to different tile packs multiplied by 5-20 states, we could be looking at large space requirements for the files?  Or, are there better ways to compress the data or control zoom levels?  What is the size of the full USA with the ESRI data?  Are there any tricks to keep space requirements down when tiling?
0 Kudos
MichaelBranscomb
Esri Frequent Contributor
Hi,

RE: MPK Unpacking:

The process follows your latter suggestion - it does not unpack if already unpacked and the data has not changed. The first time you access a Map Package via the ArcGIS Runtime (or other ArcGIS software) it will be unpacked to your user profile. The next time that same MPK is accessed the software will first check your user profile for the exploded Map Package and, if present, will confirm it is the same version/date. In this case the MPK will not be unpacked again. If the MPK being loaded by the Runtime is found to be a more recent version then it will be unpacked again. 

If you create a Map Package which includes the data (the default option) then the unpacking process will take marginally longer than if the Map Package just references the data. Once the MPK is unpacked the local map mapservice is then spun up. Currently this typically takes 5-15 seconds depending on the complexity of the map and the number of layers, but the benefit of the asynchronous API design is that this can be done in the background whilst your application is initializing other elements and/or the user can continue using the application.

RE: MPKs versus TPKs:

If you need to switch layers on and off at runtime then you likely need to use MPKs. We'll do some more investigation into your reported performance issues. Can you provide some information on:

- Size of data (e.g. total size of source File GDB)
- Total number of layers in map 
- Size of MPK
- Rendering styles used in map (e.g. cartographic or simple symbols, representations, maplex labelling)
- ... or perhaps you can share your map and a sample of the data?

Regarding tile packaging advice - the tiles are stored in the most optimal manner, using the compact cache format introduced in ArcGIS Server at v10, but you can specify the image format to provide some control over the cache size/quality - for more information and advice see http://resourcesbeta.arcgis.com/en/help/runtime-wpf/help/#/About_tile_packages/01700000004w000000/.

We greatly appreciate your feedback

Cheers

Mike
0 Kudos
MichaelErlich
Emerging Contributor
With the map pack in question, here are the specs you requested:
GDB Size: 2.86GB
11 Layers
MPK Size: 359 MB

I can provide the map data and projects for the data in question.  However, do the size, it is too large to upload on the forum.  Can I get an e-mail address so that I can provide ftp credentials to get the data from us?

Thanks,
Mike
0 Kudos
MichaelBranscomb
Esri Frequent Contributor
Hi,

Yes - please email me the FTP info at mbranscomb@esri.com.

Cheers

Mike
0 Kudos