topic Re: ArcGIS Web Map Viewer Max Record in ArcGIS Enterprise Portal Questions

ArcGIS Web Map Viewer Max Record

GIS123EXPERT — Tue, 01 Aug 2023 08:22:02 GMT

I am working on ArcGIS Enterprise system and I want to publish a layer consisting of 1.7 million points.

I publish this layer with ArcGIS Pro as a wfs service. I set the maximum records by server as 2000000 from the sharing settings.

After publishing the service, you can see the maximum number of records below.

When I open it with the viewer, all the records are not displayed and when I set the WFS OGC options on the Web Map Viewer, I can enter the maximum value of 30000.When I enter the value 30001, I get an error.

How can I see all my records as WFS?

Thanks..

Re: ArcGIS Web Map Viewer Max Record

A_Wyn_Jones — Tue, 01 Aug 2023 08:37:43 GMT

This seems to be a default value of 3000 to protect the performance of your consumer. Depending on where you consume these layers, Pro seems to have a way to update the max number of records returned from a WFS: https://pro.arcgis.com/en/pro-app/latest/help/data/services/use-wfs-services.htm#:~:text=Set%20the%20maximum%20features%20returned,is%20set%20to%203000%20features.

There is a specific requirement for COUNT in WFS 2.0.0 for this to work.

I'd suggest changing this property and then saying it as a layer file if you need people to see the 1.7 million points.

Re: ArcGIS Web Map Viewer Max Record

ChristopherCounsell — Tue, 01 Aug 2023 10:21:43 GMT

The default limit is 1,000.

https://enterprise.arcgis.com/en/server/latest/publish-services/windows/available-wfs-service-properties.htm#ESRI_SECTION1_7FEB1CEC36AD4E1282744DDE75102DC0

See the section here:

When the Enable maximum number of features property is checked, you can use this property to specify the maximum number of features to be returned. By default, the value is set to 1000.
Keep in mind that specifying a large number of features to be returned by the server can slow the performance of client applications and your GIS server.

Your server has a max limit of 200000. The client is limiting the number requested to 3000.

Reading the Map Viewer documentation they indicate the limit for Hosted is 3000. They don't specify a limit for referenced services that I can see, but going off the server documentation defaults, 3000 sounds like a good point.

https://enterprise.arcgis.com/en/portal/latest/use/ogc.htm

So... why do you want to try and render 1.7 million points? It'll throttle your service and your client web browsers. It'll also look horrendous. Typically you'd enable scale-driven symbology to avoid trying to render more than the limits. There's also lots of methods/symbology to display large quantities of data e.g. clustering and heat maps:

https://doc.arcgis.com/en/arcgis-online/reference/best-practices-high-density-data.htm

I'd strongly recommend NOT trying to mess with these limits. It'll just give you and your end users a bad time.

Re: ArcGIS Web Map Viewer Max Record

GIS123EXPERT — Tue, 01 Aug 2023 12:01:23 GMT

I made the settings you mentioned in the WFS 2.0 version I published.

However, these settings seem to affect only the ArcGIS Pro side. Users who will use the service want to access and edit over the web(Web Map Viewer), so I need to make this setting for the web side. For the web side, there is an error like in cc.png it comes into the mix.

Re: ArcGIS Web Map Viewer Max Record

GIS123EXPERT — Tue, 01 Aug 2023 12:05:35 GMT

A public institution wants to see all the building numbers in the same service and at one time. That's why I'm trying to publish 1.8 million data as WFS. As you say, exceeding the limits in terms of performance will cause slowness for both the publisher and the user, but because I need to publish the data, I am looking for various ways.

Re: ArcGIS Web Map Viewer Max Record

ChristopherCounsell — Tue, 01 Aug 2023 12:06:41 GMT

Pro's default is 3000 for performance reasons. Desktop graphics.

Web Map Viewer's default is 1000 for performance reasons. Client browser rendering.

Re: ArcGIS Web Map Viewer Max Record

GIS123EXPERT — Tue, 01 Aug 2023 12:11:50 GMT

Thank you for your support

Re: ArcGIS Web Map Viewer Max Record

ChristopherCounsell — Tue, 01 Aug 2023 12:22:18 GMT

"at one time"?

put a scale level on it and make it display when they zoom into a building?

Re: ArcGIS Web Map Viewer Max Record

berniejconnors — Tue, 01 Aug 2023 12:26:50 GMT

You said:

"A public institution wants to see all the building numbers in the same service and at one time"

IMHO, it would be better to solve this issue by using clustering or a heat map at smaller scales and then show all the points at larger scales. Nobody can visually discern 1.7 million points with labels on a single map. Your users don't always understand the limits of ArcGIS Server. It is part of our job to educate our users so they get what they need in a manner that is performant for them and for our servers.

I think Montgomery Scott said it best...

Cheers,

Bernie.

Re: ArcGIS Web Map Viewer Max Record

GIS123EXPERT — Tue, 01 Aug 2023 12:48:33 GMT

"at one time" is to see all the points when the map is opened. Even if they do not see all the points when the map is opened, as a result they should be able to see all the points when they zoom in on the map anywhere. However, since the service first filters all the returned data, not all points are visible on the map even when scaling.

Re: ArcGIS Web Map Viewer Max Record

berniejconnors — Tue, 01 Aug 2023 15:49:57 GMT

When the user zooms in their mapping client should be sending new requests to the feature service for the points that fall within the new map extent. With a smaller map extent the WFS service should be able to provide records for all visible points.

Re: ArcGIS Web Map Viewer Max Record

ChristopherCounsell — Wed, 02 Aug 2023 04:09:42 GMT

Thanks @GIS123EXPERT for clarifying.

You don't need to adjust the maxRecordCount here. That record limits the number of features that are returned as part of requests. If you try and load a map extent that exceeds the maxRecordCount of the service/client, it will only return that many features. If you zoom in on the map, the request will be made again, and if you zoom in far enough, the total number of features in the map will be below the max record count.

It is not a good idea to return all features at once. Your client (browser) and your server will have performance issues as it tries to return all the features and render them.

What you can do is use scale visibility ranges. This means the layer won't display on the map until the users zoom in. All 1.7 million points will be available for viewing. Just turn it off at a city/country/world level so that users don't try and request all of them at once. You want to choose a scale range that ideally only returns the max record count (1000) within the map extent.

https://www.esri.com/arcgis-blog/products/arcgis-online/mapping/using-scale-visibility-ranges-for-symbology-in-arcgis-online-web-maps/

Some web mapping applications also restrict the zoom levels, to help the user experience.

If you do want to visualize the buildings at the smaller scales such as a state or country, I would suggest swapping out individual feature rendering with one of the symbology methods mentioned earlier, like heat mapping or clustering. This allows users to visualize the building features in aggregate, then zoom in to see individual features at an appropriate scale.

https://doc.arcgis.com/en/arcgis-online/reference/best-practices-high-density-data.htm

Re: ArcGIS Web Map Viewer Max Record

nadja — Wed, 21 Aug 2024 06:05:24 GMT

hey @GIS123EXPERT , @ChristopherCounsell , @berniejconnors , @A_Wyn_Jones

we've just encountered aa similar issue. honestly, we dont care too much about how the map would look with that many points, for us it is far more important that an analysis is carried out with the entire dataset. it might be that the service will be slow, but there's certainly other ways to boost the performance. and even if not - ana anlysis without the entire dataset is simply pointless.

here's an idea which might help: https://community.esri.com/t5/arcgis-pro-ideas/when-stting-up-a-wfs-connection-make-it-possible/idi-p/1322091

Re: ArcGIS Web Map Viewer Max Record

berniejconnors — Thu, 22 Aug 2024 01:29:15 GMT

Nadja,

I am not an expert but here are some opinions:

If you need access to millions of data points why not just use desktop GIS to connect directly to your database? Using a map service or feature service to acess the data is just going to slow down the analysis.

If you must use a map service or feature service there are methods to send multiple requests to retrieve all of the required records. If the max record count is 2000 you can get 2000 records with the first request and then 2000 more with the second request, etc., etc., until your user has the entire dataset. I expect this will require the creation of a custom widget (that you will have to create) to allow your users to retreive all of the records via an ArcGIS map or feature service. Essentially, your users will be using a custom widget to access the service to retrieve an entire copy of the feature class and store it locally in a data structure on their own computer. This does not sound efficient. This is also the definition of "data scraping".

Perhaps it would be smarter to create a geoprocessing service that you host on your server that can directly access the entire feature class from your database.

Good luck.

Bernie.

Re: ArcGIS Web Map Viewer Max Record

nadja — Thu, 22 Aug 2024 04:50:53 GMT

@berniejconnors

well, I can't connect directly to the database, because it is not ours 😉 they provide just a WFS with which we have to do the analysis. so that's simply not an option.

also we need an option which does not require much programming or data scraping or anything. we need a straightforward method to access ALL the data which is offered by the data owner. currently we need to either ask what's the feature count or open the data in QGIS to see the number of features and then set the maximum number of features returned by the WFS accordingly. That's unnecessary.

and since it's not our data, we have no possible to change the way the data is served.

Re: ArcGIS Web Map Viewer Max Record

berniejconnors — Thu, 22 Aug 2024 12:26:00 GMT

Nadja,

Thanks for the clarification. I thought you were publishing the WFS. Without doing some research and testing I think your only option is to scrape the data from the WFS service and then do your analysis locally. You could also ask the data publisher if they also provide a download of the data. Here is a Google search with some promising results.

Good luck,

Bernie.