topic Re: Increasing the �??maximum number of records returned by the server�?�, in ArcGIS Enterprise Questions

Increasing the ???maximum number of records returned by the server???,

JamalNUMAN — Sun, 05 Jan 2014 19:06:33 GMT

Increasing the ???maximum number of records returned by the server???,

If wondering what consequences we might have in case we increase the ???maximum number of records returned by the server??? from 1,000 (default) to 240,000

For example, in my case, I have a layer which contains about 240,000 records. I need to set the ???maximum number of records returned by the server??? to be 240,000 to make sure that all the records of the attribute table of the layer can be returned if a particular query is applied.

[ATTACH=CONFIG]30262[/ATTACH], [ATTACH=CONFIG]30263[/ATTACH]

Thank you

Best

Jamal

Re: Increasing the �??maximum number of records returned by the server�?�,

AnthonyGiles — Sun, 05 Jan 2014 20:14:39 GMT

Jamal,

You will find the response time from the server really slow and your application will struggle to handle the results and probably crash,

Regards

Anthony

Re: Increasing the �??maximum number of records returned by the server�?�,

Anonymous User — Mon, 06 Jan 2014 06:20:08 GMT

Original User: nidhinkn

It will slow the performance of client applications consuming your map service, such as web browsers, and your GIS server.

Re: Increasing the �??maximum number of records returned by the server�?�,

LeoDonahue — Mon, 06 Jan 2014 13:42:42 GMT

Jamal,

If that query needs to return all the records, why not just maximize the efficiency of viewing that layer in your map in it's entirety? Is it a point layer or polygon?

Re: Increasing the �??maximum number of records returned by the server�?�,

LeoDonahue — Mon, 06 Jan 2014 13:56:50 GMT

For example, have you considered using a view to represent that query, rather than let the tool build the query?

http://resources.arcgis.com/en/help/main/10.2/002q/002q00000023000000.htm

Re: Increasing the �??maximum number of records returned by the server�?�,

Anonymous User — Mon, 06 Jan 2014 18:33:27 GMT

Original User: Jamal432@gmail.com

For example, have you considered using a view to represent that query, rather than let the tool build the query?

http://resources.arcgis.com/en/help/main/10.2/002q/002q00000023000000.htm

Many thanks guys for the help. your answers are very useful.

Sometimes, for particular query, the returned values are more than 1,000 where the end-user needs to export them as CSV file (from the web mapping application).

My current layer is parcel (polygon layer) which contains 240,000 records.

If performance is down for more than 1000 records returned, I would prefer to keep the number as is.

Re: Increasing the �??maximum number of records returned by the server�?�,

LeoDonahue — Mon, 06 Jan 2014 18:42:34 GMT

Jamal,

There are other ways to serve this layer efficiently. I would explore them before you listen to advice that says your site will "probably" crash, with really no evidence given.

At 10.2, if you are licensed, you can publish that parcel layer as it's own service, using it's own site - group of ArcGIS Servers. ( I think that is what they are calling them now ).

*edit: Clusters... is the word I was looking for.

Re: Increasing the �??maximum number of records returned by the server�?�,

Anonymous User — Mon, 06 Jan 2014 19:25:00 GMT

Original User: ad_giles@hotmail.com

The problem is not with the server being able to handle publishing services with a large amount of data it is with the browser being able to handle a large amount of features. I would pretty much guarentee that trying to display 240,000 features in the attribute table would cause the browser to stop responding, that is if the server can actually manage to return that number of features.

I have not tried to up the minimum features to that sort of figure as doing so would defenatly cause issues.

Regards

Anthony

Re: Increasing the �??maximum number of records returned by the server�?�,

Anonymous User — Mon, 06 Jan 2014 19:37:53 GMT

Original User: mboeringa2010

Many thanks guys for the help. your answers are very useful.

Sometimes, for particular query, the returned values are more than 1,000 where the end-user needs to export them as CSV file (from the web mapping application).

My current layer is parcel (polygon layer) which contains 240,000 records.

If performance is down for more than 1000 records returned, I would prefer to keep the number as is.

Jamal, while it probably is unwise to try to allow access to all 240.000 records directly through a web service, there is a huge difference between that 1000 current setting and the maximum 240.000 records of your layer. You might consider increasing it experimentally and incrementally in steps of a thousand records to see what is still acceptable. Maybe most of your users' requirements can be met by increasing it to say 5000 or 10.000 records, and you might discover this still works. But only testing can tell (or reviewing the detailed info and using the tools available in the ESRI System Design Strategies webpages, as the benchmarks for certain configurations listed there can be compared to your situation)

Of course, if current performance at 1000 records is already doubtful, than increasing it won't help you unless you take additional action to boost your sites performance, network or server hardware.

Re: Increasing the �??maximum number of records returned by the server�?�,

Anonymous User — Mon, 06 Jan 2014 19:55:02 GMT

Original User: ldonahue

The problem is not with the server being able to handle publishing services with a large amount of data it is with the browser being able to handle a large amount of features.

The browser? I disagree.

I have not tried to up the minimum features to that sort of figure as doing so would defenatly cause issues.

You should check before you make that statement.

On a poorly configured map service, running 10.0, and using the SQL hack, I can get 240,000 parcel attributes in about 35 seconds. No crashing.

[ATTACH=CONFIG]30299[/ATTACH]

Re: Increasing the �??maximum number of records returned by the server�?�,

AnthonyGiles — Mon, 06 Jan 2014 20:12:32 GMT

Leo,

Ok you have managed to query that amount of records via the rest endpoint but you are not returning any geometry and only one attribute per feature and that took 35 seconds which would not be that acceptable within a web application. Returning more attributes to display in a table and the geometry to draw the features on the map will cause the browser to struggle.

Regards

Anthony

Re: Increasing the �??maximum number of records returned by the server�?�,

Anonymous User — Mon, 06 Jan 2014 20:28:04 GMT

Original User: ldonahue

No. Not the browser.

You can not tweek your browser to make ArcGIS server return features or records faster than it does, based on the way it is configured. That is simply wrong advice.

And like I said, you could use a SQL View, index the fields you want to display, deploy a cluster to handle just this map service, etc, etc to make the response much faster. My example was running on a vm, with 4GB of RAM, max of 2 SOC processes, the parcel layer was in a map service with 60+ other layers and the fields were not indexed.

You said the browser would crash. I showed that is not the case. The issue is how long does Jamal want to wait for the response from ArcGIS Server and are there other ways to do the same thing which are faster.

My other question to Jamal would be, can you really see 240,000 polygons in your map, even if the request took 3 seconds? If you just want to return attributes about those polygons, there are faster ways of doing that.

Re: Increasing the �??maximum number of records returned by the server�?�,

LeoDonahue — Mon, 06 Jan 2014 20:29:45 GMT

Returning more attributes to display in a table and the geometry to draw the features on the map will cause the browser to struggle.

Does your browser struggle when it downloads the ESRI ISOs? That is a much larger file than 240,000 parcel polygons.

It is not the browser.

Re: Increasing the �??maximum number of records returned by the server�?�,

Anonymous User — Mon, 06 Jan 2014 20:42:48 GMT

Original User: ad_giles@hotmail.com

Leo,

I think we may have our wires crossed on what the we mean by the browser handling data, downloading an ISO is something completely different to having a responsive web mapping application. Have a look at the sample apps here and see if they handle massive amounts of features being drawn:

http://blogs.esri.com/esri/arcgis/2011/06/23/determining-limits-for-map-graphics/

Regards

Anthony

Re: Increasing the �??maximum number of records returned by the server�?�,

LeoDonahue — Mon, 06 Jan 2014 20:57:33 GMT

I also think Derek is wrong to say the browser has anything to do with it. He was talking about a JavaScript "app" that plots points. Apps, and the infrastructure behind it are the issue, not the browser.

Write an ASP or JSP web page that returns all fields of the parcel layer and all records into a datagrid or simply write it out to the browser. It's fast.

Derek later on talks about the difference between pushing 44KB vs 3MB over the wire (the network, not the browser), and how that impacts the JavaScript API processing that response. Recall, the JavaScript API makes calls across your network to ESRI's to process that output, everytime.

Things that slow you down:

Your ArcGIS server infrastruture

Your database infrastructure

Your network

Your web server

API's that access and process data

If Derek believes the browser is the problem in generating speedy output then we wouldn't be downloading ESRI media using our browsers.

Re: Increasing the �??maximum number of records returned by the server�?�,

LeoDonahue — Mon, 06 Jan 2014 21:08:15 GMT

The only way I will concede that the browser can be part of the problem is when it comes to sending large amounts of HTTP headers with every request. However, that rests solely on the API being used to query the data. It's not the browsers fault that it has to send that much info as part of the request.

http://www.chromium.org/spdy/spdy-whitepaper

https://developers.google.com/speed/docs/best-practices/request

http://www.websocket.org/quantum.html

Re: Increasing the �??maximum number of records returned by the server�?�,

Anonymous User — Tue, 07 Jan 2014 07:18:59 GMT

Original User: Jamal432@gmail.com

The only way I will concede that the browser can be part of the problem is when it comes to sending large amounts of HTTP headers with every request. However, that rests solely on the API being used to query the data. It's not the browsers fault that it has to send that much info as part of the request.

http://www.chromium.org/spdy/spdy-whitepaper

https://developers.google.com/speed/docs/best-practices/request

http://www.websocket.org/quantum.html

Mamy thanks guys for the very useful discussion,

As I set the �??maximum number of records returned by the server�?� to be 240,000, my Geocortex web mapping application fails to return any value as shown in the screenshot below.

[ATTACH=CONFIG]30303[/ATTACH], [ATTACH=CONFIG]30304[/ATTACH]

By the way, is there a way to know the maximum number of records a web mapping application can return?

Re: Increasing the �??maximum number of records returned by the server�?�,

LeoDonahue — Tue, 07 Jan 2014 13:31:36 GMT

Jamal,

It could be a number of factors, but not your browser... 😄

I re-tested a parcel layer of my own. I gave the service a min of 4 SOC processes and a max of 4 SOC processes. I set the maximum number of records to 240,000, and using the REST query inteface, I let it return everything, geometry included.

What I got was a 503 error, after several minutes of processing, which means the server is too busy to render a response in a timely fashion.

As you can see, the resources on the ArcGIS Server are considerable. But the processing takes too long so the webserver returns a 503 HTTP code. The lack of returning a result has nothing to do with the browser, unless you can modify the "Retry-After" request header. Your browser doesn't process the request, it simply renders it.

Your problem of wanting to display 240,000 parcels in a Geocortex web application could be related to these issues or other ones. It's unlikely that you will achieve displaying this many results in your web application in a timely fashion using a third party web app, unless you are willing to break up those 240,000 parcels into meaningful groups of some kind.

As Anthony originally said:
"You will find the response time from the server really slow and your application will struggle to handle the results and probably crash"

response time from the server really slow - agree.
your application will struggle to handle the results - disagree, that processing takes place on the ArcGIS Server.
and probably crash - hard to say, every application deals with issues in different ways. It appears Geocortex simply does not process your request.

Note in the third image how long each part of the request/response and content load takes.

I made this request directly to ArcGIS Server on port 8399 and did not route my request through a production web server.

[ATTACH=CONFIG]30319[/ATTACH][ATTACH=CONFIG]30316[/ATTACH][ATTACH=CONFIG]30318[/ATTACH]

Re: Increasing the �??maximum number of records returned by the server�?�,

Anonymous User — Tue, 07 Jan 2014 15:24:45 GMT

Original User: ad_giles@hotmail.com

I think a lot of this will depend on what type of service GeoCortex consumes, if it is a map service or wms then I agree with Leo that the server will handle creating the image and should be capable unless the response times out. If GeoCortex takes a feature service then the features will be drawn client side in the browser and I would hazard a guess that this amount of features will cause problems in the browser, I know from experience that this is the case in a flex application even when we are talking 10000 - 20000 features.

Regards

Anthony

Re: Increasing the �??maximum number of records returned by the server�?�,

TanuHoque — Wed, 08 Jan 2014 15:40:11 GMT

Sometimes, for particular query, the returned values are more than 1,000 where the end-user needs to export them as CSV file (from the web mapping application).

I'd recommend to have a gp service which will execute the query, export to a CSV file on the server side and return it by url (or by value, if that is what you want).

I'd not recommend to convert a query result to CSV on the client side especially when a query returns a ton of records.