New dedicated Workflow Manager Server Machine giving checkStatus 500 error before creating workflow item?

[Amarz]

Hello,

I have a new 3 machine setup, Map01=ArcGIS Server (hosting), Map03=ArcGIS Server WFM Server, Map05=Portal for ArcGIS, all on Enterprise 11.2. Both ArcGIS Servers are federated with my Portal. Map01 as hosted, and Map03 as WFM. both are validating with green checks.

All are sharing a wildcard certificate, along with the selfsignedcertificates on the respective machine.

When I open the WorkFlow Manager App from Portal, I am greeted with this error in Inspections > Network before creating workflow item.

GET https://<DNS>.<domain>/arcgis/workflow/checkStatus 500 (Internal Server Error)

After I attempt to  , I also get an error in my .server/manager Logs saying:

No subject alternative DNS name matching <MAP01>.<DOMAIN>.LOCAL found

This is after confirming the wildcard cert, the portal cert, and the WFM server cert are on each others machines.

Any help would be appreciated.

[CodyPatterson]

Hey Amarz,

I was curious if when attempting to create the Workflow Item, if anything shows up in the server logs?

I have an open case with ESRI support for over two weeks now with no resolution in sight regarding a very similar issue, actually this is what it was started over.

Is the FQDN that the GET request is sending to equipped with a web adaptor named arcgis, or are you using the :port before it as https://my-company:6443/arcgis/workflow?

I've seen success in dealing with this myself by restarting the server machine, then portal, then workflow server machine and restarting it in that same order.

I have many other things support has requested I try that I can provide if that doesn't work, they may not have worked with me but you could have success!

[Amarz]

Sorry for the delay in responding.

The FQDN that the GET request is sending to is equipped with a webadaptor named arcgis, and we are not using the :port. 

The error log is as follows:

! @8439a21ga - Internal server error, for (POST) [/workflow/admin/createWorkflowItem?name=test] -> play.api.UnexpectedException: Unexpected exception[ConnectException: No subject alternative DNS name matching <MAP05>.<DOMAIN>.LOCAL found.] at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:358) Caused by: java.net.ConnectException: No subject alternative DNS name matching <MAP05>.<DOMAIN>.LOCAL found. at play.shaded.ahc.org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179) Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching CC05MAP05.CATHEDRALCITY.LOCAL found. at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching <MAP05>.<DOMAIN>.LOCAL found. at java.base/sun.security.util.HostnameChecker.matchDNS(Unknown Source)	

Yet, the Workflow Manager Server (Map03) has our wildcard set as web server, along with its tree (root, 2x intermediates) as well as the selfsigned from our Portal that has the DNS the matches in the error. 

[JFarmer]

Hi @Amarz 

A couple of things for you to check here.

• Does your privateportalURL include the FQDN <MAP01>.<DOMAIN>.<LOCAL>? It seems like Workflow Manager cannot reach the server using the DNS https://DNS.<domain>/arcgis as it needs to.
• What SAN values are in your certificates? 

This error looks less like a failure to import certificates in portal/server and more like a communication issue with FQDN / DNS. Particularly the possibility that the wildcard certificate doesn't have a SAN value for <MAP01>.<DOMAIN>.<LOCAL>. This isn't an issue in a base ArcGIS Enterprise deployment but it is for Workflow Manager. We need to be able to fully trust all the components in the system.

Jonathan

[Amarz]

Apologies for the delay in response.

The Private Portal url on the WFM (Map03) has the Portal Properties privateportalURL as "https://<map05>.<domain>.local:7443/arcgis"

For Certificates here is what I have on the machine:

MAP03 - gisworkflow
Certificate Name	Type	CN	SAN
wildcard (web server)	PrivateKeyEntry	*.domain	DNS: *.domain.gov  DNS: domain.gov
intermediate1	trustedCertEntry
intermediate2	trustedCertEntry
portal	trustedCertEntry	MAP05.<DOMAIN>.LOCAL	MAP05.<DOMAIN>.LOCAL
sectigo (root)	trustedCertEntry
selfsignedcertificate_map01	trustedCertEntry	MAP01.<DOMAIN>.LOCAL	MAP01.<DOMAIN>.LOCAL
selfsignedcertificate	PrivateKeyEntry	MAP03.<DOMAIN>.LOCAL	MAP03.<DOMAIN>.LOCAL

Each machine also has all the same certificates.

Our understanding was that the wildcard and the use of the DNS should make these separate machines be able to communicate, and if not, thats why the selfsigned from each machine is also present.

[Amarz · Accepted solution]

Thank you @CodyPatterson & @JFarmer  for helping me confirm that it is not an issue with the Certificates. I solved my issue by adjusting a few settings in my Portal Administration Directory.

Previously, my Portals Admin Url was using "https://machine.domain.local:7443/arcgis", I instead updated this to reflect the "https://DNS.domain.com:7443/arcgis" and this allowed me to create a new workflow, but with some errors. I then had to also add a new property in the Portal Admin Directory > System > Properties to assign the privatePortalURL that was being referenced in my Map01(federated hosting server) and my MAP03 (federated WFM sever):

{
"WebContextURL":"https://DNS.domain.com/portal",
"privatePortalURL":"https://DNS.domain.com/portal"
} 

After performing those steps, I am now able to Create / Delete / Update New Workflow Items.

Thanks again for pointing me away from the certificates.

If you can think of a reason why this is not a good approach to fixing the issue, please let me know as well!

I hope this helps.

[CodyPatterson]

Hey @Amarz,

Great job on figuring it out and I greatly appreciate you returning with this information, I am still having issues with my own Workflow server, so I will be giving this a shot to see if it resolves my issues that are currently on going. I'll let you know what I find!

[JFarmer]

Hi @Amarz ,

Thanks for providing this information and also what you did to get things working. I'm glad it's working now for you. However, I did want to provide some more information on a different route you could take and why the changes you did make addressed the issue for anyone who may think themselves to be in a similar situation.

According to the cert information you sent, it looks like ArcGIS Server and Portal know themselves as machine.local. They then each had a wildcard certification on their internal web server (not the external web server where the web adaptor is located) that contained a SAN which wasn't for machine.local. This seems to be the crux of the issue. While this setup may work during basic use and during setup, it won't for Workflow Manager. We look to ensure that the URL being reached has a certificate with a matching SAN value and that its intermediate and root components are either from a well-known certifying authority or that the intermediate and root are already imported into ArcGIS Server's trusted store. So, really, there are two options to achieve this:

• Add a SAN entry for .local in the existing wildcard
• Add another certificate for .local and swap the root and intermediate certs with the other enterprise components to establish trust

Your solution, while perfectly fine if it works for your needs, was more of a workaround. You've updated Server and Portal to now know themselves internally as the DNS and are routing all your traffic through the external web adaptor. And that's fine as long as you're OK with it and have planned for the existing traffic load that is now going through your external web server. If you keep it this way, I'd suggest updating the security config in each ArcGIS Server to set the privatePortalUrl to be this new URL as well since you are altering things post-federation.

Let me know if you have any questions.

Thanks!

Jonathan

[Amarz]

@JFarmer Thank you for your detailed response. It turns out the changes I made did impact another component of our Enterprise, that being the SSO with IWA. In order to maintain SSO, we have reverted to the Portal's Admin URL = <machine>.<domain>.local, and the privateportalURL = <https://<machine>.<domain>.local:7443/arcgis.

In response to:

• Add another certificate for .local and swap the root and intermediate certs with the other enterprise components to establish trust

---

Shouldn't the selfsignedcertificate_map03 (referenced in the table of a previous comment) accomplish this?

Each machine has the wildcard with its root and intermediates already, along with the portal certificate (default).

I just want it all to work...

[JFarmer]

Hi @Amarz 

Would you be able to work with Tech Support on this issue if you aren't already? It would be good for them to dig a little deeper and see what the issue is here. Feel free to post the case number here once you have one (or message me directly here on Esri Community). 

Thanks!

Jonathan