Intermittently Unable to Create Hosted Feature Services

[RyanUthoff]

I'm having an issue where we are intermittently unable to create hosted feature services using ArcGIS Enterprise 11.2 on Windows. This includes creating hosted feature services directly through the Portal UI as well as creating Survey123 forms.

The only error message I receive is: Create Service exception 'java.lang.Exception: Exception: Read timed out'

Survey123 explicitly gives that error message, while Portal gives a generic error message in the web page, but I can see the same error message when viewing the internet traffic. It is a read timed out error, but it usually fails within seconds of attempting to create a hosted feature service.

Esri is not producing any error messages in the Portal, Server, or Data Store logs when this error occurs. This one error message is literally the only error message I'm seeing.

This has happened to multiple people at various times with various hosted feature services/S123 forms, including Esri's template hosted feature services you can create through Portal.

I'm working with Esri support on this one, but does anyone else have any ideas on what this could be? We've already ruled out anti-virus/firewall and the ArcGIS Data Store successfully validates through Server Manager.

[CodyPatterson]

Hey @RyanUthoff

Are you using a Reverse Proxy at all? From the sound of it at a surface level, something isn't communicating correctly as it's timing out. I would possibly check Wireshark traffic when creating a hosted feature layer through this method along with possibly using Fiddler on the requester side, this will see requests go through a proxy. Along with this it may be a certificate issue, support hounded me on certificate issues with an Enterprise setup, and even though it didn't fix anything and was not the issue, it may fix something in your case!

Hope that helps!

Cody

[RyanUthoff]

Thanks for replying back! We're just using Esri's web adaptor and not any third party reverse proxies or anything like that. Support did have me attempt to create a hosted feature service on the Portal machine itself, using the local machine name URL which bypasses all the web adaptors and I continue to experience the same error. So I'm thinking with that troubleshooting step, we've at least ruled out anything on the web adaptor side of things. I didn't think to check the certificates, so I'll verify those. I think I've heard of certificate issues causing random issues, so I'll check that out. Thank you!

[A_Wyn_Jones]

Is the IPV6 network adapter enabled on the machine? There's a bug in 11.2 that breaks communications to Datastore if this adapter is disabled.

It's been discussed here with workarounds:https://community.esri.com/t5/publishing-and-managing-services-questions/publish-not-working-after-f...

I wouldn't expect this to be intermittent in your case though, may help get to the bottom of it!

[RyanUthoff]

Thank you for the reply! Our IPV6 is disabled, but the behavior we're experiencing appears to be different than the behavior outlined in the geonet post you linked though. Also, we started experiencing this issue in 10.9.1, so as long as this bug is new to 11.2, then I don't think that would explain the issue we're experiencing.

[BrittanyBurson]

I am experiencing this in Enterprise 10.9.1. Anyone get around it?

[RyanUthoff]

Oh wow, I'm surprised someone else has this issue! I'm still working with Esri support. I highly highly highly recommend you create an Esri support case if you can. The more people who create cases about this issue, the more likely we'll get a resolution.

One thing that might be worth testing is if you can create hosted feature services directly through Portal. It might take a few attempts to receive an error though since sometimes it works, and sometimes it doesn't.

I'm curious if our environments are similar. By any chance do you have multiple ArcGIS Server machines in the same site?

[BrittanyBurson]

Hi @RyanUthoff you were right, I experienced the same issue creating a hosted layer within Portal. We opened a case with Esri as well. I don't know enough detail about our server architecture though, sorry.

[DavidJantz]

We do have multiple ArcGIS Server machines in the site. A DEV environment with a single ArcGIS Server machine does not have this issue. IPv6 is disabled as well. Did you try enabling that?

I do get the following error logged in Portal, but only at a DEBUG level:

Exception while computing compute certain flags in self resource (ex: supportsSceneServices). Likely cause is a badly configured system default account (system_publisher)

Exception: Read timed out
at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:834)
at com.esri.gw.account.GWAccounts.a(GWAccounts.java:2373)
at com.esri.gw.account.GWAccounts.getAccountProperties(GWAccounts.java:2488)
at com.esri.gw.account.AccountPropertiesHandler.getAccountProperties(AccountPropertiesHandler.java:60)
at com.esri.gw.account.AccountsRequestHandler.H(AccountsRequestHandler.java:5333)
at com.esri.gw.account.AccountsRequestHandler.service(AccountsRequestHandler.java:1743)
at com.esri.gw.DispatchServlet.service(DispatchServlet.java:159)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764).....

The Create Service exception 'java.lang.Exception: Exception: Read timed out' error in Portal seems to only be when publishing from an Esri template or from Survey123 Connect. Publishing from ArcGIS Pro, from an existing item or URL or even uploading a shapefile and creating a service from that in Portal works, although the above DEBUG message is still logged. Nothing logged in Server or Data Store.

[Thomas_Puthusserry]

@RyanUthoff  

I had this issue and raised a ESRI Support tickets. The issues are two types:

1. Unable to create a hosted feature service (from template, or via ArcPro), unable to create a view layer from any hosted feature service, but possible to publish a hosted feature layer by uploading a gdb as zip file: View Layer could not be created. Try again - ArcGI... - Esri Community.

2. Unable to publish a hosted scene layer from ArcPro, but possible to add SLPK via portal  and publish scene layer: Re: Your portal does not support publishing web sc... - Esri Community

Both issues seems related (in my case) and was sorted by following this: Error: ERROR 001369: Failed to Create the Service (esri.com), Specifically the issue was related to permission for arcgis service account to a specific folder C:\arcgisserver

Thomas