nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Increasing the ???maximum number of records returned by the server???,

JamalNUMAN

Increasing the ???maximum number of records returned by the server???,

If wondering what consequences we might have in case we increase the ???maximum number of records returned by the server??? from 1,000 (default) to 240,000

For example, in my case, I have a layer which contains about 240,000 records. I need to set the ???maximum number of records returned by the server??? to be 240,000 to make sure that all the records of the attribute table of the layer can be returned if a particular query is applied.

[ATTACH=CONFIG]30262[/ATTACH], [ATTACH=CONFIG]30263[/ATTACH]

Thank you

Best

Jamal

Clip_703.jpg

Clip_702.jpg

Lake_CountyIllinois_GIS_0-1749225348652.png

FrancTrigt_0-1749225323651.png

Vind meer berichten getagd met

Accepted answers

AntGiles

Jamal,

You will find the response time from the server really slow and your application will struggle to handle the results and probably crash,

Regards

Anthony

All comments

AntGiles

Jamal,

You will find the response time from the server really slow and your application will struggle to handle the results and probably crash,

Regards

Anthony

Anonymous User

Original User: nidhinkn

It will slow the performance of client applications consuming your map service, such as web browsers, and your GIS server.

LeoDonahue

Jamal,

If that query needs to return all the records, why not just maximize the efficiency of viewing that layer in your map in it's entirety? Is it a point layer or polygon?

LeoDonahue

For example, have you considered using a view to represent that query, rather than let the tool build the query?

http://resources.arcgis.com/en/help/main/10.2/002q/002q00000023000000.htm

Anonymous User

Original User: Jamal432@gmail.com

For example, have you considered using a view to represent that query, rather than let the tool build the query?

http://resources.arcgis.com/en/help/main/10.2/002q/002q00000023000000.htm

Many thanks guys for the help. your answers are very useful.

Sometimes, for particular query, the returned values are more than 1,000 where the end-user needs to export them as CSV file (from the web mapping application).

My current layer is parcel (polygon layer) which contains 240,000 records.

If performance is down for more than 1000 records returned, I would prefer to keep the number as is.

LeoDonahue

Jamal,

There are other ways to serve this layer efficiently. I would explore them before you listen to advice that says your site will "probably" crash, with really no evidence given.

At 10.2, if you are licensed, you can publish that parcel layer as it's own service, using it's own site - group of ArcGIS Servers. ( I think that is what they are calling them now ).

*edit: Clusters... is the word I was looking for.

Anonymous User

Original User: ad_giles@hotmail.com

The problem is not with the server being able to handle publishing services with a large amount of data it is with the browser being able to handle a large amount of features. I would pretty much guarentee that trying to display 240,000 features in the attribute table would cause the browser to stop responding, that is if the server can actually manage to return that number of features.

I have not tried to up the minimum features to that sort of figure as doing so would defenatly cause issues.

Regards

Anthony

Anonymous User

Original User: mboeringa2010

Many thanks guys for the help. your answers are very useful.

Sometimes, for particular query, the returned values are more than 1,000 where the end-user needs to export them as CSV file (from the web mapping application).

My current layer is parcel (polygon layer) which contains 240,000 records.

If performance is down for more than 1000 records returned, I would prefer to keep the number as is.

Jamal, while it probably is unwise to try to allow access to all 240.000 records directly through a web service, there is a huge difference between that 1000 current setting and the maximum 240.000 records of your layer. You might consider increasing it experimentally and incrementally in steps of a thousand records to see what is still acceptable. Maybe most of your users' requirements can be met by increasing it to say 5000 or 10.000 records, and you might discover this still works. But only testing can tell (or reviewing the detailed info and using the tools available in the ESRI System Design Strategies webpages, as the benchmarks for certain configurations listed there can be compared to your situation)

Of course, if current performance at 1000 records is already doubtful, than increasing it won't help you unless you take additional action to boost your sites performance, network or server hardware.

Anonymous User

Original User: ldonahue

The problem is not with the server being able to handle publishing services with a large amount of data it is with the browser being able to handle a large amount of features.

The browser? I disagree.

I have not tried to up the minimum features to that sort of figure as doing so would defenatly cause issues.

You should check before you make that statement.

On a poorly configured map service, running 10.0, and using the SQL hack, I can get 240,000 parcel attributes in about 35 seconds. No crashing.

[ATTACH=CONFIG]30299[/ATTACH]

240000.jpg

AntGiles

Leo,

Ok you have managed to query that amount of records via the rest endpoint but you are not returning any geometry and only one attribute per feature and that took 35 seconds which would not be that acceptable within a web application. Returning more attributes to display in a table and the geometry to draw the features on the map will cause the browser to struggle.

Regards

Anthony

Anonymous User

Original User: ldonahue

No. Not the browser.

You can not tweek your browser to make ArcGIS server return features or records faster than it does, based on the way it is configured. That is simply wrong advice.

And like I said, you could use a SQL View, index the fields you want to display, deploy a cluster to handle just this map service, etc, etc to make the response much faster. My example was running on a vm, with 4GB of RAM, max of 2 SOC processes, the parcel layer was in a map service with 60+ other layers and the fields were not indexed.

You said the browser would crash. I showed that is not the case. The issue is how long does Jamal want to wait for the response from ArcGIS Server and are there other ways to do the same thing which are faster.

My other question to Jamal would be, can you really see 240,000 polygons in your map, even if the request took 3 seconds? If you just want to return attributes about those polygons, there are faster ways of doing that.

LeoDonahue

Returning more attributes to display in a table and the geometry to draw the features on the map will cause the browser to struggle.

Does your browser struggle when it downloads the ESRI ISOs? That is a much larger file than 240,000 parcel polygons.

It is not the browser.

Anonymous User

Original User: ad_giles@hotmail.com

Leo,

I think we may have our wires crossed on what the we mean by the browser handling data, downloading an ISO is something completely different to having a responsive web mapping application. Have a look at the sample apps here and see if they handle massive amounts of features being drawn:

http://blogs.esri.com/esri/arcgis/2011/06/23/determining-limits-for-map-graphics/

Regards

Anthony

LeoDonahue

I also think Derek is wrong to say the browser has anything to do with it. He was talking about a JavaScript "app" that plots points. Apps, and the infrastructure behind it are the issue, not the browser.

Write an ASP or JSP web page that returns all fields of the parcel layer and all records into a datagrid or simply write it out to the browser. It's fast.

Derek later on talks about the difference between pushing 44KB vs 3MB over the wire (the network, not the browser), and how that impacts the JavaScript API processing that response. Recall, the JavaScript API makes calls across your network to ESRI's to process that output, everytime.

Things that slow you down:

Your ArcGIS server infrastruture
Your database infrastructure
Your network
Your web server
API's that access and process data

If Derek believes the browser is the problem in generating speedy output then we wouldn't be downloading ESRI media using our browsers.

LeoDonahue

The only way I will concede that the browser can be part of the problem is when it comes to sending large amounts of HTTP headers with every request. However, that rests solely on the API being used to query the data. It's not the browsers fault that it has to send that much info as part of the request.

http://www.chromium.org/spdy/spdy-whitepaper

https://developers.google.com/speed/docs/best-practices/request

http://www.websocket.org/quantum.html

Anonymous User

Original User: Jamal432@gmail.com

The only way I will concede that the browser can be part of the problem is when it comes to sending large amounts of HTTP headers with every request. However, that rests solely on the API being used to query the data. It's not the browsers fault that it has to send that much info as part of the request.

http://www.chromium.org/spdy/spdy-whitepaper

https://developers.google.com/speed/docs/best-practices/request

http://www.websocket.org/quantum.html

Mamy thanks guys for the very useful discussion,

As I set the �??maximum number of records returned by the server�?� to be 240,000, my Geocortex web mapping application fails to return any value as shown in the screenshot below.

[ATTACH=CONFIG]30303[/ATTACH], [ATTACH=CONFIG]30304[/ATTACH]

By the way, is there a way to know the maximum number of records a web mapping application can return?

Clip_732.jpg

Clip_731.jpg

LeoDonahue

Jamal,

It could be a number of factors, but not your browser... 😄

I re-tested a parcel layer of my own. I gave the service a min of 4 SOC processes and a max of 4 SOC processes. I set the maximum number of records to 240,000, and using the REST query inteface, I let it return everything, geometry included.

What I got was a 503 error, after several minutes of processing, which means the server is too busy to render a response in a timely fashion.

As you can see, the resources on the ArcGIS Server are considerable. But the processing takes too long so the webserver returns a 503 HTTP code. The lack of returning a result has nothing to do with the browser, unless you can modify the "Retry-After" request header. Your browser doesn't process the request, it simply renders it.

Your problem of wanting to display 240,000 parcels in a Geocortex web application could be related to these issues or other ones. It's unlikely that you will achieve displaying this many results in your web application in a timely fashion using a third party web app, unless you are willing to break up those 240,000 parcels into meaningful groups of some kind.

As Anthony originally said:
"You will find the response time from the server really slow and your application will struggle to handle the results and probably crash"

response time from the server really slow - agree.
your application will struggle to handle the results - disagree, that processing takes place on the ArcGIS Server.
and probably crash - hard to say, every application deals with issues in different ways. It appears Geocortex simply does not process your request.

Note in the third image how long each part of the request/response and content load takes.

I made this request directly to ArcGIS Server on port 8399 and did not route my request through a production web server.

[ATTACH=CONFIG]30319[/ATTACH][ATTACH=CONFIG]30316[/ATTACH][ATTACH=CONFIG]30318[/ATTACH]

notthebrowser_2.jpg

89587notthebrowser_3.jpg

notthebrowser_1.jpg

notthebrowser_3.jpg

89585notthebrowser_1.jpg

Anonymous User

Original User: ad_giles@hotmail.com

I think a lot of this will depend on what type of service GeoCortex consumes, if it is a map service or wms then I agree with Leo that the server will handle creating the image and should be capable unless the response times out. If GeoCortex takes a feature service then the features will be drawn client side in the browser and I would hazard a guess that this amount of features will cause problems in the browser, I know from experience that this is the case in a flex application even when we are talking 10000 - 20000 features.

Regards

Anthony

TanuHoque

Sometimes, for particular query, the returned values are more than 1,000 where the end-user needs to export them as CSV file (from the web mapping application).

I'd recommend to have a gp service which will execute the query, export to a CSV file on the server side and return it by url (or by value, if that is what you want).

I'd not recommend to convert a query result to CSV on the client side especially when a query returns a ton of records.

Anonymous User

Original User: Jamal432@gmail.com

I'd recommend to have a gp service which will execute the query, export to a CSV file on the server side and return it by url (or by value, if that is what you want).

I'd not recommend to convert a query result to CSV on the client side especially when a query returns a ton of records.

Many thanks guys for the help. Your input is very helpful

Best

Jamal

Doe mee aan het gesprek

Aangemelde leden kunnen berichten plaatsen, updates volgen en meer. Nieuw hier? Registreer een gratis account.

Snelle Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.