topic Re: OpenData download problems in ArcGIS Hub Questions

OpenData download problems

deleted-user-Bkz8cHvtt5bs — Wed, 02 Dec 2015 21:02:11 GMT

There have been quite a few issues lately with downloads of data. I even tried to change most of the datasets taht are not versioned to be a manual cache so that I can ensure 1 good download on ESRI servers. Unfortunately, I've been receiveing a lot of complaints from my users, particularly about this dataset:

http://geodata.myfwc.com/datasets/68fbc06232164ce6b6b0d046e906b885_0

Can this be fixed on ESRI's end?

Re: OpenData download problems

DanielFenton1 — Wed, 02 Dec 2015 21:05:39 GMT

Hi Chris, we need to figure out why this query is failing or slow:

http://atoll.floridamarine.org/arcgis/rest/services/FWC_GIS/OpenData_Elevation/MapServer/0/query?outSR=4326&where=OBJECTID%20%3E=%2014001%20AND%20OBJECTID%3C=15000&f=json&outFields=*&geometry=&returnGeometry=true&geometryPrecision=10

Open Data cannot produce a download from this dataset if this query fails repeatedly.

My best guess is that the heap size on your server is set to low. That means this query is simply to big to hold in memory.

You could try lowering the max record count for this service or increasing the allowed heap size. Do either of those help that query complete?

Daniel

Re: OpenData download problems

deleted-user-Bkz8cHvtt5bs — Thu, 03 Dec 2015 16:12:17 GMT

Hi Daniel,

We are going to go through our OpenData services and reduce the max record counts for all of them to 1000 or 2000 records, but I wanted to verify that the big data download issue (i.e. the shapefiles only downloading approximately 1000 records) was resolved with the bootstrap update about 9 months ago. The original reason we upped the max record count was to work around the truncated download issue.

I’d also like to say that I’m unfamiliar with the concept of the “heap size” setting on server. I’ve researched it a little bit and found how to change it, but the ESRI help documentation simply stated that it should be changed to “an appropriate heap size.” Do you know of any documentation to determine what an “appropriate” heap size would be? For example, if our server is currently at the defaults of 64MB for the SOC heap size and 256MB for the server heap size, and we have about 40 services, should we quadruple the heap sizes or would doubling them suffice? Also, would it be appropriate to only up one of the heap sizes? Since, you’re suggesting to increase the server heap size, should we only increase the server heap size, or should we also increase the SOC heap size proportionally?

Re: OpenData download problems

DanielFenton1 — Thu, 03 Dec 2015 16:36:53 GMT

Hey Chris,

Lowering your max record counts to 1k to 2k does sound appropriate in general. As you have noted, the process of creating downloads has changed significantly since the early days of Open Data.

Unfortunately the documentation is a little weak around recommend heap sizes. How much total ram is available on your server instance(s)? On machines with large amounts of RAM you should be able to increase Server heap size quite a bit. Without seeing the actual error that is reported by your server instance it’s tough for me to tell you whether you need to increase Server, SOC or both.

The Server heap is the total amount of memory available to available to all services while SOC represents the amount available to individual services.

All that said, reducing the max record counts to more appropriate levels will reduce the need for ArcGIS Server to use heap space when serving requests.

Daniel

Re: OpenData download problems

deleted-user-Bkz8cHvtt5bs — Thu, 03 Dec 2015 16:41:41 GMT

Hi Daniel,

Thanks for this information. Currently we have 37 services (I think) on one of our servers named atoll (atoll.floridamarine.org). That server has 64GB of RAM available, of which we are currently only using 13%. Our current heap size settings are the default 64MB for SOC and 256MB for the server.

Re: OpenData download problems

DanielFenton1 — Thu, 03 Dec 2015 16:52:29 GMT

Wow, 64GB is a ton of RAM. You can safely increase your heap sizes quite a bit. I’m hesitant to give you specific numbers because I’m not an expert in ArcGIS Server. But I can certainly say the defaults were created for a machine with much less than 64GB of ram. Support may be able to point you to a better guideline.

Again, it may not be necessary at all if you lower the max record count to more appropriate values.

Re: OpenData download problems

deleted-user-Bkz8cHvtt5bs — Thu, 03 Dec 2015 16:56:41 GMT

Thanks Daniel,

I also see a number of my datasets are coming up with errors stating that the “Record count query failed exceeded timeout of 90000ms.” Would this also be resolved by increasing the heap time, or is this something else in the service that needs to be optimized?

Re: OpenData download problems

DanielFenton1 — Thu, 03 Dec 2015 17:00:17 GMT

It’s possible that those are related, but I really can’t say. I’d talk to Support and see if they can link you up with someone who specializes in Server Performance.

Re: OpenData download problems

DanielFenton1 — Thu, 03 Dec 2015 17:00:53 GMT

Out of curiosity, are you willing to post the service definition?

Re: OpenData download problems

deleted-user-Bkz8cHvtt5bs — Thu, 03 Dec 2015 17:53:28 GMT

Attached is the service definition file for the service that is throwing the “Record count query failed [because it] exceeded timeout of 90000ms” error. Please let me know if you see any glaring problems.

Thanks,

Chris

Re: OpenData download problems

deleted-user-Bkz8cHvtt5bs — Fri, 04 Dec 2015 14:48:39 GMT

After discussing this issue with other Florida State Government colleagues, it sounds like this is a common problem. They have taking this issue to Tech support and played with their heap size to no avail. Apparently this was logged as a bug (BUG-000091676) and forwarded to the Open Data team for review back at the beginning of November.

We will try to play with the heap size on our servers to see if this may resolve our issues, but if you have any more insight into another possible remedy, I'd be very interested.

Re: OpenData download problems

DanielFenton1 — Fri, 04 Dec 2015 16:03:21 GMT

Chris,

Now it looks like your network is blocking requests from the service that creates downloads. Can you check to see why you guys are blocking requests from “User-agent: featureservices-node”?

No downloads will be able to work if that is blocked.

Re: OpenData download problems

deleted-user-Bkz8cHvtt5bs — Fri, 04 Dec 2015 16:45:57 GMT

I can look into it, but I have no idea how to set that setting. If you have any insight into that setting please share. I’ll do some google searches in the meantime.

Re: OpenData download problems

DanielFenton1 — Fri, 04 Dec 2015 16:48:21 GMT

It’s not a setting on ArcGIS server. It would be something set at the network level. Perhaps there is an IT administrator you can speak to?

Re: OpenData download problems

deleted-user-Bkz8cHvtt5bs — Fri, 04 Dec 2015 17:40:33 GMT

Daniel,

Can you supply us with the full call request and the full error message that you’re receiving on your end? When I log into OpenData Data Manager, I’m not seeing this error anywhere. I can’t replicated what you’re seeing.

I don’t know if this will help you in your analysis of the issue, but the way our infrastructure is set up, we have a single server instance of ArcGIS Server sitting behind a reverse proxy which is in turn behind a firewall.

Re: OpenData download problems

DanielFenton1 — Fri, 04 Dec 2015 18:37:33 GMT

This is likely at the firewall level. The block is preventing traffic from our servers from reaching yours. It’s not something you are going to see in your data manager and it’s not easy for you to replicate.

What’s being blocked is requests with a header specifying the user-agent: “featureservices-node”.

When I run this test `curl -H 'User-agent: featureservices-node' http://atoll.floridamarine.org/arcgis/rest/services/FWC_GIS/OpenData_Elevation/MapServer/0` I get an empty reply from your server. Meaning that this request was blocked. I only see this when specifying that user agent.

Here is some information about user agents in general: https://en.wikipedia.org/wiki/User_agent

Re: OpenData download problems

deleted-user-Bkz8cHvtt5bs — Fri, 04 Dec 2015 18:41:10 GMT

Daniel,

I've talked with our IT Network staff and they said that we are not blocking anything on our side.

Also, I was thinking about it and the results have been pretty inconsistent. If what you're saying is true that our network is blocking some traffic due to this setting, wouldn't this affect all of our data downloads on OpenData? All of our services for OpenData are being populated from one server, but some datasets are downloading just fine.

Anyway, we're going to try to alter the heap size this afternoon and see how that affects the failed downloads issue. Hopefully we can come up with something else here soon.

Thanks again,

Chris

Re: OpenData download problems

DanielFenton1 — Fri, 04 Dec 2015 18:45:57 GMT

It’s being blocked somewhere. Sending curl requests without that header leads to the information returning just fine. With that header blocked I cannot test for other possible problems.

This is what I see when I send a request with that header:

$curl -H 'User-agent: featureservices-node' http://atoll.floridamarine.org/arcgis/rest/services/FWC_GIS/OpenData_Elevation/MapServer/0\?f\=json

curl: (52) Empty reply from server

And with a different one:

$curl -H 'User-agent: foo' http://atoll.floridamarine.org/arcgis/rest/services/FWC_GIS/OpenData_Elevation/MapServer/0\?f\=json

{"currentVersion":10.22,"id":0,"name":"Bathymetry of Select Lakes in Florida","type":"Feature Layer","description”:” ….

Re: OpenData download problems

DanielFenton1 — Fri, 04 Dec 2015 20:17:10 GMT

Hi Chris, I’ve been trying to debug someone else’s issue and it looks like the blocking maybe somehow be coming from my work network. Let me look into this.

Re: OpenData download problems

deleted-user-Bkz8cHvtt5bs — Fri, 04 Dec 2015 20:53:00 GMT

oh, ok Daniel. Good luck with that.

I just spent sometime with my coworker and we increased the heap size and restarted all of our services. The website seems to be working a little bit better now, but I'm going to let a week or two pass before I can make an official determination.

Thanks for your help. I'll let you know if i find anything else out.

Chris