nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

OpenData download problems

deleted-user-Bkz8cHvtt5bs

There have been quite a few issues lately with downloads of data. I even tried to change most of the datasets taht are not versioned to be a manual cache so that I can ensure 1 good download on ESRI servers. Unfortunately, I've been receiveing a lot of complaints from my users, particularly about this dataset:

http://geodata.myfwc.com/datasets/68fbc06232164ce6b6b0d046e906b885_0

Can this be fixed on ESRI's end?

Najděte více příspěvků označených jako

Accepted answers

DanielFenton1

Hi Chris, we need to figure out why this query is failing or slow:

http://atoll.floridamarine.org/arcgis/rest/services/FWC_GIS/OpenData_Elevation/MapServer/0/query?outSR=4326&where=OBJECTID >= 14001 AND OBJECTID<=15000&f=json&outFields=*&geometry=&returnGeometry=true&geometryPrecision=10

Open Data cannot produce a download from this dataset if this query fails repeatedly.

My best guess is that the heap size on your server is set to low. That means this query is simply to big to hold in memory.

You could try lowering the max record count for this service or increasing the allowed heap size. Do either of those help that query complete?

Daniel

All comments

DanielFenton1

Hi Chris, we need to figure out why this query is failing or slow:

Open Data cannot produce a download from this dataset if this query fails repeatedly.

My best guess is that the heap size on your server is set to low. That means this query is simply to big to hold in memory.

You could try lowering the max record count for this service or increasing the allowed heap size. Do either of those help that query complete?

Daniel

deleted-user-Bkz8cHvtt5bs

Hi Daniel,

We are going to go through our OpenData services and reduce the max record counts for all of them to 1000 or 2000 records, but I wanted to verify that the big data download issue (i.e. the shapefiles only downloading approximately 1000 records) was resolved with the bootstrap update about 9 months ago. The original reason we upped the max record count was to work around the truncated download issue.

I’d also like to say that I’m unfamiliar with the concept of the “heap size” setting on server. I’ve researched it a little bit and found how to change it, but the ESRI help documentation simply stated that it should be changed to “an appropriate heap size.” Do you know of any documentation to determine what an “appropriate” heap size would be? For example, if our server is currently at the defaults of 64MB for the SOC heap size and 256MB for the server heap size, and we have about 40 services, should we quadruple the heap sizes or would doubling them suffice? Also, would it be appropriate to only up one of the heap sizes? Since, you’re suggesting to increase the server heap size, should we only increase the server heap size, or should we also increase the SOC heap size proportionally?

DanielFenton1

Hey Chris,

Lowering your max record counts to 1k to 2k does sound appropriate in general. As you have noted, the process of creating downloads has changed significantly since the early days of Open Data.

Unfortunately the documentation is a little weak around recommend heap sizes. How much total ram is available on your server instance(s)? On machines with large amounts of RAM you should be able to increase Server heap size quite a bit. Without seeing the actual error that is reported by your server instance it’s tough for me to tell you whether you need to increase Server, SOC or both.

The Server heap is the total amount of memory available to available to all services while SOC represents the amount available to individual services.

All that said, reducing the max record counts to more appropriate levels will reduce the need for ArcGIS Server to use heap space when serving requests.

Daniel

deleted-user-Bkz8cHvtt5bs

Hi Daniel,

Thanks for this information. Currently we have 37 services (I think) on one of our servers named atoll (atoll.floridamarine.org). That server has 64GB of RAM available, of which we are currently only using 13%. Our current heap size settings are the default 64MB for SOC and 256MB for the server.

deleted-user-Bkz8cHvtt5bs

When we completed lowering the number of features downloaded to 1,000, increasing the heap size for both the SOC and the Server Heaps, cleared the caches, and determined which downloads should really be manual (the more static datasets) versus automatic (often updated), the download performance was greatly increased on our OpenData site.

Thanks for your help Daniel!

MalcolmJ

I've been having trouble with this service in particular. I don't know if its a similar problem to the others in this thread?

Dataset | MDC Open Data

The dataset has slightly over 100,000 records and is a dataset of LiDAR contours, so each record can have a lot of vertices. Are there too many features in this dataset to be able to download or is there a server setting which is causing the downloads to fail to generate?

Generally when I try download this, the generating download bar will go up to about 80-90%, then fail.

DanielFenton1

Hey Malcolm, sorry for the slow reply. For some reason I stopped getting Geonet notifications.

I'm looking into this dataset and will let you know what I find.

Daniel

Připojte se ke konverzaci

Přihlášení členové mohou přispívat, sledovat aktualizace a další. Jste tu noví? Zaregistrujte si bezplatný účet.

Rychlé odkazy

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.