topic Intermittently Unable to Create Hosted Feature Services in ArcGIS Enterprise Portal Questions

Intermittently Unable to Create Hosted Feature Services

RyanUthoff — Wed, 17 Apr 2024 14:16:21 GMT

I'm having an issue where we are intermittently unable to create hosted feature services using ArcGIS Enterprise 11.2 on Windows. This includes creating hosted feature services directly through the Portal UI as well as creating Survey123 forms.

The only error message I receive is: Create Service exception 'java.lang.Exception: Exception: Read timed out'

Survey123 explicitly gives that error message, while Portal gives a generic error message in the web page, but I can see the same error message when viewing the internet traffic. It is a read timed out error, but it usually fails within seconds of attempting to create a hosted feature service.

Esri is not producing any error messages in the Portal, Server, or Data Store logs when this error occurs. This one error message is literally the only error message I'm seeing.

This has happened to multiple people at various times with various hosted feature services/S123 forms, including Esri's template hosted feature services you can create through Portal.

I'm working with Esri support on this one, but does anyone else have any ideas on what this could be? We've already ruled out anti-virus/firewall and the ArcGIS Data Store successfully validates through Server Manager.

Re: Intermittently Unable to Create Hosted Feature Services

CodyPatterson — Wed, 17 Apr 2024 16:28:46 GMT

Hey @RyanUthoff

Are you using a Reverse Proxy at all? From the sound of it at a surface level, something isn't communicating correctly as it's timing out. I would possibly check Wireshark traffic when creating a hosted feature layer through this method along with possibly using Fiddler on the requester side, this will see requests go through a proxy. Along with this it may be a certificate issue, support hounded me on certificate issues with an Enterprise setup, and even though it didn't fix anything and was not the issue, it may fix something in your case!

Hope that helps!

Cody

Re: Intermittently Unable to Create Hosted Feature Services

RyanUthoff — Wed, 17 Apr 2024 17:07:01 GMT

Thanks for replying back! We're just using Esri's web adaptor and not any third party reverse proxies or anything like that. Support did have me attempt to create a hosted feature service on the Portal machine itself, using the local machine name URL which bypasses all the web adaptors and I continue to experience the same error. So I'm thinking with that troubleshooting step, we've at least ruled out anything on the web adaptor side of things. I didn't think to check the certificates, so I'll verify those. I think I've heard of certificate issues causing random issues, so I'll check that out. Thank you!

Re: Intermittently Unable to Create Hosted Feature Services

A_Wyn_Jones — Thu, 18 Apr 2024 09:11:57 GMT

Is the IPV6 network adapter enabled on the machine? There's a bug in 11.2 that breaks communications to Datastore if this adapter is disabled.

It's been discussed here with workarounds:https://community.esri.com/t5/publishing-and-managing-services-questions/publish-not-working-after-fresh-install/td-p/1370170/page/3

I wouldn't expect this to be intermittent in your case though, may help get to the bottom of it!

Re: Intermittently Unable to Create Hosted Feature Services

RyanUthoff — Thu, 18 Apr 2024 13:33:15 GMT

Thank you for the reply! Our IPV6 is disabled, but the behavior we're experiencing appears to be different than the behavior outlined in the geonet post you linked though. Also, we started experiencing this issue in 10.9.1, so as long as this bug is new to 11.2, then I don't think that would explain the issue we're experiencing.

Re: Intermittently Unable to Create Hosted Feature Services

BrittanyBurson — Tue, 04 Jun 2024 22:54:31 GMT

I am experiencing this in Enterprise 10.9.1. Anyone get around it?

Re: Intermittently Unable to Create Hosted Feature Services

RyanUthoff — Thu, 13 Jun 2024 16:17:34 GMT

Oh wow, I'm surprised someone else has this issue! I'm still working with Esri support. I highly highly highly recommend you create an Esri support case if you can. The more people who create cases about this issue, the more likely we'll get a resolution.

One thing that might be worth testing is if you can create hosted feature services directly through Portal. It might take a few attempts to receive an error though since sometimes it works, and sometimes it doesn't.

I'm curious if our environments are similar. By any chance do you have multiple ArcGIS Server machines in the same site?

Re: Intermittently Unable to Create Hosted Feature Services

BrittanyBurson — Thu, 06 Jun 2024 22:27:30 GMT

Hi @RyanUthoff you were right, I experienced the same issue creating a hosted layer within Portal. We opened a case with Esri as well. I don't know enough detail about our server architecture though, sorry.

Re: Intermittently Unable to Create Hosted Feature Services

DavidJantz — Tue, 11 Jun 2024 16:58:40 GMT

We do have multiple ArcGIS Server machines in the site. A DEV environment with a single ArcGIS Server machine does not have this issue. IPv6 is disabled as well. Did you try enabling that?

I do get the following error logged in Portal, but only at a DEBUG level:

Exception while computing compute certain flags in self resource (ex: supportsSceneServices). Likely cause is a badly configured system default account (system_publisher)
Exception: Read timed out
at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:834)
at com.esri.gw.account.GWAccounts.a(GWAccounts.java:2373)
at com.esri.gw.account.GWAccounts.getAccountProperties(GWAccounts.java:2488)
at com.esri.gw.account.AccountPropertiesHandler.getAccountProperties(AccountPropertiesHandler.java:60)
at com.esri.gw.account.AccountsRequestHandler.H(AccountsRequestHandler.java:5333)
at com.esri.gw.account.AccountsRequestHandler.service(AccountsRequestHandler.java:1743)
at com.esri.gw.DispatchServlet.service(DispatchServlet.java:159)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764).....

The Create Service exception 'java.lang.Exception: Exception: Read timed out' error in Portal seems to only be when publishing from an Esri template or from Survey123 Connect. Publishing from ArcGIS Pro, from an existing item or URL or even uploading a shapefile and creating a service from that in Portal works, although the above DEBUG message is still logged. Nothing logged in Server or Data Store.

Re: Intermittently Unable to Create Hosted Feature Services

Thomas_Puthusserry — Wed, 12 Jun 2024 16:10:29 GMT

@RyanUthoff

I had this issue and raised a ESRI Support tickets. The issues are two types:

1. Unable to create a hosted feature service (from template, or via ArcPro), unable to create a view layer from any hosted feature service, but possible to publish a hosted feature layer by uploading a gdb as zip file: View Layer could not be created. Try again - ArcGI... - Esri Community.

2. Unable to publish a hosted scene layer from ArcPro, but possible to add SLPK via portal and publish scene layer: Re: Your portal does not support publishing web sc... - Esri Community

Both issues seems related (in my case) and was sorted by following this: Error: ERROR 001369: Failed to Create the Service (esri.com), Specifically the issue was related to permission for arcgis service account to a specific folder C:\arcgisserver

Thomas

Re: Intermittently Unable to Create Hosted Feature Services

RyanUthoff — Wed, 12 Jun 2024 16:21:09 GMT

Thanks for the reply! We've already made sure that all the Esri directories have full permissions to the service account running server, so unfortunately that wasn't the issue.

Re: Intermittently Unable to Create Hosted Feature Services

AhmadAwada1 — Wed, 27 Nov 2024 13:34:09 GMT

Hi Ryan,

i am facing exactly the issue you described. Were you able to figure out a solution? If yes, please help!!

Re: Intermittently Unable to Create Hosted Feature Services

RyanUthoff — Wed, 27 Nov 2024 14:54:44 GMT

Nope. Have an Esri support case open for 7 months and counting. I know at least one other person has an Esri support case open as well, and they haven't had a resolution either.

Right now, we're getting towards the point of having to do a full uninstall and reinstall of the ArcGIS Data Store, and then restore it from a backup, because we're running out of other troubleshooting options.

My recommendation is to open an Esri support case. The more people we have reporting this issue, the better. It might demonstrate that Esri has a legitimate bug on their hands instead of a random one-off issue.

I'm actually hesitant to believe that the Data Store is the problem in the first place because what we found is that when we receive this error message, the requests aren't even making it to the IIS level. A successful publishing of a hosted feature service sends requests through IIS. However, an unsuccessful request does not. So, in order for the Data Store to be able to return an error, a request would need to be made to the Data Store first, which we would see in IIS. But, it's not doing that..... Of course, I don't know what all goes on behind the scenes, but that's my thought process on it.

Re: Intermittently Unable to Create Hosted Feature Services

Thomas_Puthusserry — Thu, 28 Nov 2024 15:02:43 GMT

@RyanUthoff Is this anything to do with your GIS server's config store location? I had a similar issue in our TEST env which had the config store / server directories located to Azure storage account. The publication of hosted feature service many times failed (but worked few times). In the end, after moving the config store and server directories to a fileshare VM the issue didn't recur. Just thought to highlight that.

Re: Intermittently Unable to Create Hosted Feature Services

RyanUthoff — Wed, 04 Dec 2024 08:43:39 GMT

It's possible, but I don't know for sure. We have two Enterprise environments with the config store in the exact same AWS FSx, but only one environment is having issues. The other is perfectly fine.

In our case, what we notice is that the requests aren't even making it to IIS. When normally creating hosted feature services, we can see requests being sent in the IIS logs. However, when I receive the error, I don't see anything in the IIS logs. Since all internet traffic and requests go through IIS/web adaptors, I'd think that if it was a problem with Portal/Server/Data Store, the request would be logged in IIS. In order for Portal/Server/Data Store to return an error, it needs to receive a request first, but it's not being logged in IIS.

So if my logic is correct, then the problem must be something at the web adaptor or IIS level. But I'm not sure where to look or troubleshoot any further.

Re: Intermittently Unable to Create Hosted Feature Services

RyanUthoff — Tue, 04 Mar 2025 15:52:05 GMT

For anyone following this post, the final resolution was to move our config-store to a local drive. After moving it to an EBS volume attached to the ArcGIS Server EC2 instance, we no longer had any issues with creating hosted feature services. Esri support did provide me with a bug number: BUG-000173561. If anyone else is also experiencing this issue, you can probably open a support case and reference the bug number.