An upgrade to ArcGIS Hub's hosted service download experience is coming in February!

[ThomasHervey1]

UPDATES

- 2024-06-25: Minor language changes to clarify that this release is for feature service items that are hosted in ArcGIS Online only.

---

Data downloads and interoperability are cornerstones of ArcGIS Hub's open data principles. To better support user workflows and reduce headaches by content manangers, the ArcGIS Hub team is pleased to announce an upgraded download experience starting with Features Services hosted in ArcGIS Online. In this article, we will explain what is changing, how you can take advantage of these features today, and what download features are coming next. 

We would love to hear from you, so please share your feedback in the comments and let us know what download and reuse features you would like to see next for ArcGIS Hub.

_________________________________________________________________________________

Table of Contents

• Table of Contents
• What's new with the upgraded download experience?
• How can I use the upgraded download experience?
• What else should I know?
• What else has changed?
• What's coming next?

_________________________________________________________________________________

What's new with the upgraded download experience?

On February 27th, 2024, Hub’s improved download experience for feature services hosted in ArcGIS Online, in beta during 2023, will become the default download experience for all users. Visitors to the content view of an item that uses a hosted feature service will automatically see the upgraded download experience. For details on what's changing and steps you can take to use the upgraded experience, keep reading. 

Once released, users will see:

• Additional download formats including File Geodatabase (FGDB), Feature Collection, Excel, SQLite Geodatabase, GeoPackage. 
• A progress indicator that tells users when the system is "processing" a request to generate or update a file, and the percentage completed. 
• Download speed increases for items with a large number of records and/or complex geometry. 
• Download reliability improvements for fewer errors and generic messages. 
• Filtering support for public and private content; use the attribute table filters or precise on-map draw tools to download the exact records you need.

How can you tell if your item is already using the upgraded download system? In the top right corner of the download panel, you will see a "Hosted Downloads" badge and additional formats.

Figure 1. Hosted download badge is visible in the top right corner of the download panel.

How can I use the upgraded download experience?

At this time the upgraded experience is only available on items that 1. references a feature service, and 2. that feature service is hosted in ArcGIS Online. These items most commonly created either through the ArcGIS Online "Content" tab > "New Item" action or through a workflow when publishing a hosted service from ArcGIS Pro. 

We encourage all content managers to try the upgraded download experience (in beta) today. Start by visiting an item's edit route in ArcGIS Hub by selecting the edit pencil. Then, select the download tab and enable both settings for 1. "Extract Capabilities" and 2. "Hosted Downloads."  

If neither the download tab or those settings are available then the item is not a hosted feature service. Also, if the "Export data" setting is already enabled on the item within the ArcGIS Online home application, then extract capability is already enabled for the hosted service. Once completed, simply select the 'X' in the top left to view the content; open the download panel to see the upgraded download experience.

 

Figure 2. Try out the upgraded download experience today by enabling "Extract Capabilities" and "Hosted Downloads" within an item's Content Details in ArcGIS Hub.

What else should I know?

As mentioned previously, the upgraded download experience is only available for hosted feature services. Due to faster update cycles and increased reliability and performance, we strongly encourage content managers to shift existing non-hosted services over to hosted services when possible. However, we recognize that this is not an easy process and may not be feasible for all customers. We're working closely with the ArcGIS Enterprise team to expand support for downloads so that in a future release non-hosted services will also benefit from the upgraded download experience.

What else has changed?

Technically hosted downloads use the "create replica" operation on a hosted feature service. As noted above, create replica offers more download formats as well as reliability and speed improvements. However, several features are no longer supported including: option to download a previous version, download file size metadata, and file names that match the item name. We're working with the feature service team to re-introduce these features.
_________________________________________________________________________________

What's coming next?

We know how important reuse and downloads are to you and your end users, which is why in the first half of 2024 the Hub team is focused on improvements to the download features. Here's what we're considering: 

• Scheduled download updates will allow content managers to set an update schedule and simply forget about manually checking for updated download files. Scheduled download updates are ideal if you have items that are updated very infrequently and so Hub shouldn't check the service for updates. Conversely, if you have frequently updated data, schedule a daily download update and your users won't have to wait for files to generate.
  
• Improved error messaging will give simple explainations to end users and detailed actions to content managers when a download error occurs. 
• Rearrangeable download formats will let customers showcase the most valuable formats at the top whether they are out-of-the-box supported formats like CSV or Shapefile, or the custom links you've added to your metadata distributions. 

Excited for more? Again we would love to hear from you, so please share your feedback in the comments on the experience and let us know what download and reuse feature's you would like to see next for ArcGIS Hub. 

[Ed_]

Hi Thomas, thank you for sharing, these are nice features 🙂

[jsarthur]

Thanks for the update and information! We have external users who like the KML download option and it has been removed with this update. Is it on the roadmap to add KML back as a download option or is that being phased out? Thanks!

[TomScavo]

I agree, the KML download option is a significant loss. A KML file always contains lat-long coordinates regardless of the source data. This is helpful (to me) since my applications require lat-long coordinates. I've tried using the GeoJSON download option as an alternative but apparently it does not convert the source data to lat-long coordinates. These two taken together make the new download options unusable for me. 

[KansasDASC]

@ThomasHervey1Thanks for this. We specifically are looking for a way to improve/speed up a 1 million record hosted feature layer's download capability in the Hub. Subsets of this layer work at around 800k records in about 15 minutes, but when trying the whole dataset, it just spins for hours and eventually just errors out. When I chatted with the Hub dev team at Dev Summit, they said that the option for Hosted Feature Downloads needs to be enabled and the Extract capability needs to be enabled. The REST service for this item shows the Extract option enabled in the JSON and the Hosted Downloads is enabled, but the error still persists.

I can't seem to also locate the screen you posted about with check boxes for either option. Where is that or how do I get to it?

[JosephRhodes2]

@jsarthur @TomScavo I happened to notice that disabling extract on the feature service forces the old download sidebar and brings back the KML download option, at least for now. Hopefully KML will be added back to the new download experience.

[VCGIvanBrown]

@ThomasHervey1, the new download experience for a hosted feature layer is an improvement for hosted feature layers; however, the experience with items served via ArcGIS Server map-service layers still has substantial issues (e.g., API errors in the download panel that can be cleared via an un-share/re-share, but then return later, and a problem when some ArcGIS Server - based items temporarily don't show up in a Hub site's search results but then start showing up again w/o any action being applied by us).

In some cases, serving a dataset as a hosted feature layer makes a lot of sense. In other cases, it doesn't. For example, we pause and weigh the cost (credits) and benefit for a large/complex dataset such as a state-wide layer with a lot of complex geometry (from among our set of more than 100 layers); in such a case, we are compelled to serve the dataset as an ArcGIS Server map-service layer.

Although we have transitioned some of our layers to hosted feature layers, ArcGIS Server map-service layers continue to be a major constituent of our Hub site.

The blog states:

Due to faster update cycles and increased reliability and performance, we strongly encourage content managers to shift existing non-hosted services over to hosted services when possible. However, we recognize that this is not an easy process and may not be feasible for all customers. We're working closely with the ArcGIS Enterprise team to expand support for downloads so that in a future release non-hosted services will also benefit from the upgraded download experience.

It seems like the Hub investment is more focused in the ArcGIS Online (hosted content) realm than wiring in legacy-like ArcGIS Server map-image resources. It would be good to know if the classic ArcGIS Server map-service approach is officially legacy tech that's being phased out--and if it is, a sunset date would be great for planning.

[TomScavo]

@JosephRhodes2 thanks for the note. I'd love to have the old download options back but I don't know how to "disable extract on the feature service"? Is this something a sysadmin has to do? (I'm just a lowly user)

Thanks,

Tom

[jsarthur]

@TomScavo You need permission to edit your ArcGIS Hub site to make this change. The extract capabilities setting is on the Content Details > Downloads page. 

[JosephRhodes2]

@TomScavo @jsarthur If you’re the item owner, you should be able to change this on the item settings page by unchecking “Allow others to export to different formats”. A couple of caveats though: 1. This takes away the option to export as file geodatabase, and 2. All exports will be in WKID 4326 (KML is always 4326 anyway, but this forces 4326 on shapefiles etc as well, ignoring the Local Projections setting).

[AndrigoSalvador]

I agree with @VCGIvanBrown. I use many services from Arcgis Enterprise/sde. @ThomasHervey1 now I try use the ArcGis Hub because Arcgis Sites  on our Portal don't show donwload options for services not hosted. But I am a data base administrator and I prefer see my tables and your relashionships. I have sistens thata depends this tables. So, I really hope that the Esri come back to the bases and see to Arcgis enterprises clients.. we are the base, we pay a lot for this software every year, and I don't wana pay more per credits every year. I live in Brazil, our city hall don't have many ressources. We pay in dolar. In Arc Gis Hub my feature classes not hosted spend a very long time for donwload and in my Portal (Arcgis Sites)I can't share services from my geotadabase.

I hope it is will better.

Thanks.

Thanks.

[Alex-Santos]

Hey thanks for the updates and information! For the coming items, like scheduled downloads, are those also only going to be available for hosted downloads? I think that will be very helpful for us to schedule items. It's exciting, but less so if these up and coming features are only for hosted downloads. We have moved some popular, and complex, datasets to hosted layers, but our footprint is still substantially with ArcGIS Server hosted services (like others above). We hear the recommendations, but we've made significant investments (based on EEAP engagements with Esri years ago) in AGS and we'd like to see improvements on that side too with ArcGIS Hub. Please don't leave ArcGIS enterprise users behind. Perhaps we need another engagement with Esri to figure out a better balance with AGS hosted layers and AGOL. What is that threshold of performance that we should consider utilizing AGOL hosted layers, while still having an AGS footprint.

[NinaRihn]

These are nice option for our customers!        I do agree about supporting more options for AGS referenced items.  If we host feature layers on AGOL, we have to manually update them, whereas AGS map services are dynamically referencing SDE and take care of themselves.     We could publish from SDE to Portal, but then in order to get to AGOL for open data portal, we would have to set up a collaboration to copy the data to AGOL, which doesn't work for SDE spatial views, which we use pretty frequently.     Also agree about bringing back KML.  This is allows my external customers to use our data in the ways THEY need.  

[chinelat_unicamp_arcgis]

We have a public visualization feature layer item that didn't have the “Export Data” option enabled, but now has the option to download the data opened after this new update. It doesn't respect the option that exists in the item. How do we close this option? We don't want this option

[SusanCarlson]

It seems that since the 13 August 2024 update that added the new download experience to layers that are not hosted, we've been having issues with the download to shapefile either taking an extremely long time or not finishing at all. On subsequent tries, some will immediately say a percent processing but never go past it. We've been directing customers to download based on map filter, but that is still very slow. I tried unsharing/resharing the data with the public and open data to no avail. Downloading to CSV seems to work well enough (still slow), but our customers are looking for shapefiles. (We'd really love to be able to download as FGDB from non-hosted data.)

[NinaRihn]

Hi @SusanCarlson, I wanted to let you know that I have an incident in with Esri about our ArcGIS Server-referenced services, and they have logged a bug BUG-000170515.     “Attempting to download ArcGIS Enterprise hosted services from ArcGIS Hub becomes stuck in the processing stage and not completes or returns an error.”     This bug status is currently "In Review".   

[SusanCarlson]

Thank you @NinaRihn for the bug information! The logged bug looks like it's just for Hosted data. Our data is not hosted so I may need to report an additional bug. We received more complaints about downloads today 😞

[NinaRihn]

@SusanCarlson yeah the way Esri words things using terms like "hosted" and "enterprise" often makes it confusing to read because we have used those terms differently in the past,  but what this bug referring to is services hosted on Enterprise, which Esri usually uses to call ArcGIS Server.        Then when we say "hosted" services,  we mean services hosted on AGOL, so the use of the word "hosted" confuses the bug language!                So long story short, my bug is specific to AGOL layers shared to our open data portal that are referencing ArcGIS Server services.

[SusanCarlson]

@NinaRihn Thank you for the clarification! Your bug report matches our issue. Gotta love Esri - let's call our product 'server' or 'enterprise' so internet searches will be so easy. 😄