topic Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers? in Qt Maps SDK Questions

Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

MKa — Thu, 09 Jul 2020 17:33:18 GMT

I have a product in production that lets my users take a subset of the data offline using generateOfflineMap. This process say, for my web map with 5 feature layers (filtered using definition query) and the tiled basemaps (filtered by area and scale considerations) takes almost 2 minutes to download what says 4.5mb in the estimate. This is unacceptable to my users. So I decided to dig into this and see why it takes so long. I took the overrides function so that I can remove the feature layers and tiled layers to see what is taking so long.

I assumed the basemap was what was taking so long, but to my surprise I removed all of the layers using the override functions and the download with the basemaps took under 10 secs, which I thought was very reasonable. So I realize it is the layers, so i turn them on one at a time (each by themselves), and that doesn't improve. It only improves when the are are all off.

What about these layers on my webmap is making them take so long to download? Each layer has a definition query that might break them down to include only 1 feature? So it isn't a data thing as far as I can tell. What am I missing.

Matt

also, I used this to remove some layers and get overrides to work. But this just helped in breaking down the problem. Hoping I can use the overrides to speed up by removing something or checking something.

Generate offline map (overrides) | ArcGIS for Developers

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

LukeSmallwood — Fri, 10 Jul 2020 08:04:26 GMT

Hi Matt, thanks for the info on your problem - which version of the SDK are you using?

Can you share any more details about the particular feature layers that you are using? For example:

are they all from the same Feature Service
what kind of definition query are you using
what options are you using for creating the replica (e.g. sync model, attachments directions, schema only etc.)
is this an editing workflow

It would also be interesting to test what would happen if you download a single layer without any definition query at all to see how fast that is.

Another approach which may be worth looking into (if you haven't already) is the Preplanned Workflow. This would allow you to define the areas for your offline map ahead of time so that users can simply download the required area and associated data. This is a lot quicker for the end user but does require you to know the work areas ahead of time.

Thanks,

Luke

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

MKa — Sun, 12 Dec 2021 16:42:09 GMT

I have tried everything I can think of, including what you have above.

are they all from the same Feature Service (no)
what kind of definition query are you using (the definition query set on the WebMap for each layer)

what options are you using for creating the replica (e.g. sync model, attachments directions, schema only etc.)

parameters.maxScale = zoomLevels[maxLevel]

parameters.minScale = zoomLevels[minLevel]

parameters.attachmentSyncDirection = Enums.AttachmentSyncDirectionNone;

parameters.returnLayerAttachmentOption = Enums.ReturnLayerAttachmentOptionNone

parameters.returnSchemaOnlyForEditableLayers = false;

is this an editing workflow (no)

I have also tried to remove the definition query from the layeroptions on each of my 5 layers. Then remove the useGeometry too. That didn't seem to do anything either.

layerOptions = geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(0)).layerOptions;
                for (i = 0; i < layerOptions.length; i++) {
                    layerOption.whereClause = "";
                    layerOption.queryOption = Enums.GenerateLayerQueryOptionAll  ;
                    layerOption.useGeometry = false
               }

Again, I turned off all 5 of my layers with this and it is fast, but when turning any one of them on, it becomes slow again. My tiles are not the speed issue, it is my layers, again this removes them in the overried section and without these options it is qucik.

geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(0)).layerOptions = []; 
geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(1)).layerOptions = [];
geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(2)).layerOptions = [];
geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(3)).layerOptions = [];
geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(4)).layerOptions = [];

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

LukeSmallwood — Mon, 13 Jul 2020 08:49:16 GMT

Thanks Mark,

Are your features services on AGOL or ArcGIS Enterprise?

If your workflow is strictly no editing and you don't even need to sync changes down to the clients you could try setting the SyncModel for the GenerateGeodatabaseParameters QML Type | ArcGIS for Developers to "None" (again using the overrides workflow). This would make your offline map a "snapshot" of the service without the overhead of creating and maintaining a replica and so on.

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

MKa — Mon, 13 Jul 2020 14:46:19 GMT

Luke,

My services are in our company ArcGIS Enterprise Portal.

I changed the SyncModel for all of my layers like this for each of them.

geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(0)).syncModel = Enums.SyncModelNone;

That did not increase my speed at all. I set the definition query to only have one record as well for each layer.

Matt

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

LukeSmallwood — Mon, 13 Jul 2020 14:53:09 GMT

Thanks for testing that out Matt - sorry this is not working well for you. It would also be good to know what kind of data store you are using and what kind of features you are trying to take offline.

Would it be possible for you to raise this with Esri support? It sounds like the issues you are seeing are on the feature service end so it would be good to loop them in to what you are seeing. They should also be able to give you specific advice for your installation.

Hope that helps,

Luke

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

MKa — Mon, 13 Jul 2020 18:32:52 GMT

Maybe this is a clue that can help us solve this issue. when I set the geometry to null on the layers it is fast, but of course no records are returned. But that seems to be the bottleneck. If this helps at all?

geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(0)).extent = null;

geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(1)).extent = null;

geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(2)).extent = null;

geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(3)).extent = null;

geodatabaseParamOverrides.value(geodatabaseParamOverrides.keyAt(4)).extent = null;

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

GaryMacDougall — Tue, 14 Jul 2020 00:25:56 GMT

Hi Matt

A few more questions / comments:

- Is this a host feature service in Enterprise? If not, what is the back end RDBMS

- What release of Enterprise are you using?

- How many features / how much data are you taking offline?

- Does it contain complex feature such as very detailed polygons?

- Are you familiar with running requests via the REST API? You mentioned that it was multiple feature services, would be interesting to see how much time each specific feature service takes to go offline.

thanks

Gary

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

MKa — Tue, 14 Jul 2020 00:51:53 GMT

Gary,

this is indeed hosted on enterprise 10.7.1 but the data is in sql database and not the esri data store.

I can get this to take almost two minutes for one or 1000 features. It doesn’t matter. The polygons are not that complex, and when I do individual feature layers on my web map one by one it doesn’t matter. I even created a new web map and added individual feature layers to check each one, but all did same thing and were 3 mi a minimum.

I run the simple code to take my web map offline and even provide an envelope with scale for the basemap and default parameters. The basemap by itself is very fast and not the problem.

this must have something to do with the rdbms being used instead of the data store. Which we do to update our date from other systems and still have it hosted and readily available.

amy suggestions would be helpful as the speed off going offline is the only thing holding our implementation of ESRI app studio app from being utilized company wide. I have to figure this out.

please assist.

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

GaryMacDougall — Tue, 14 Jul 2020 05:18:50 GMT

Hi Matt

You should be able to connect to the back end database with Pro or ArcMap. One thing to try is to update the statistics (analyze datasets GP tool) and rebuild indexes (rebuild indexes GP tool) on the datasets. Also, make sure there is enough memory resource on the database machine (make sure it's not maxed out). I assume the create replica is an async process - correct? If so, then the sync processing is performed by the synctools

Make sure these is enough RAM on the server to run these.

If you can include a SQL Server trace file, I can see if I can find someone to review and maybe detect where the time is being taken.

thanks

Gary

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

MKa — Tue, 14 Jul 2020 20:46:34 GMT

Are resources look just fine. And we ran the analysis and reindex and nothing changed. We will now try and get you the SQL Server trace file so we can see what is going on, where should I send it?

This is so strange, it must be in the packaging of the .geodatabase file, as i watch my offline map package folder being created, the tile files are created very quickly, then there is about a minute pause until the .geodatabase files then start to appear. I think we are getting closer to figuring this out.

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

GaryMacDougall — Wed, 15 Jul 2020 05:40:32 GMT

Hi Matt

Making arrangements for a location to send the logs - will let you know

thanks

Gary

Re: Offline Map Job generateOfflineMap - Speed Slow on Feature Layers?

MKa — Fri, 17 Jul 2020 15:10:17 GMT

I continue to work with Gary on this but wanted to give an update and ask for some help.

We discovered that at least in my case that through creating the replica using ArcGIS Rest Services that creating my offline replicas is twice as fast when I select the Synchronous option and NOT the async option. Is there a way for me to do this in code so that the create replica process works Synchronously?

Also, the number of columns in my table is also making my offline create process very slow, but I need the information in those columns (100+ columns). Any ideas on how to get those columns offline without making, what appears to be a slow schema create process.