The curious case of an ArcGIS Pro "Add Content" ailment...
The Case:
Our team was recently asked to assist with troubleshooting an odd set of failures when adding data hosted on ArcGIS Online to ArcGIS Pro. When adding ArcGIS Online hosted feature services to the ArcGIS Pro project, the customer received error 499: Invalid token.
They were already signed into ArcGIS Online, so it didn't seem like there was an issue with the token. Pro was functioning correctly on every other host in this organization. Odd that this issue would be localized to one machine.
This mystery problem was blocking acceptance of a migration, as this particular host was leveraged with some additional integrations and customizations, and the host was purchased explicitly for its designated workflow.
Symptoms:
- ArcGIS Pro was slow to load
- Adding hosted feature services to a new ArcGIS Pro project failed with error "Authentication Token required (status code 499)
- Running previously successful scripting/geoprocessing tasks failed with ERROR 160228, "The user does not have permission to execute the operation.
The customer was logged into ArcGIS Online successfully, so unless something upstream was somehow mangling an authorization token, it seemed that there was something else in play.
Given the error, we needed to check to see if ArcGIS Pro was having an issue accessing an OAuth resource or perhaps some generateToken call was failing. To do that, we broke out Fiddler4.
While using Fiddler to monitor the web traffic between ArcGIS Pro and ArcGIS Online while adding content, we immediately noticed that none of the CRL/OSCP links were accessible - the error was HTTP 502 (bad gateway).
So what's OCSP and what's a CRL anyway?
OCSP is the Online Certificate Status Protocol. It's an alternative to CRL (Certificate Revocation Link).
OCSP and CRL are two of the most common ways web browsers use to check if a site’s certificate has been revoked. A CRL is a list containing serial numbers of all certificates that have been revoked by a Certificate Authority.
OCSP is a protocol used to discover the revocation status of a certificate and contains signatures that assert a certificate has not been revoked. This makes it a more effective and efficient validation process, as unlike CRL it does not require a list to be downloaded to discover the status of a certificate.
Why did this matter?
An HTTP 502 (bad gateway) blocking these URLs is suspect. OCSP and CRL links are expected to be accessible via plaintext HTTP. After all, if a certificate is compromised, how do you validate the status of the certificate using HTTPS? The Windows crypto API won't even try to connect to a CRL or OCSP link over HTTPS. So why were these resources blocked?
The answer was simple - the customer had included this host in a firewall policy that blocked all outbound plaintext HTTP - including CRL and OCSP.
Organizations sometimes misinterpret PCI/DSS compliance controls to mean communication using plaintext is prohibited, but that's not the case.
1.2.1 - Restrict inbound and outbound traffic to that which is necessary for the cardholder data environment, and specifically deny all other traffic.
1.3.4 - Do not allow unauthorized outbound traffic from the cardholder data environment to the Internet.
As long as there is a legitimate reason for this traffic (which there is - OCSP and CRL are served over plaintext), and it is limited to the specific systems and external addresses that are required (CRL and OCSP should always be on allow lists - don't miss this during your firewall log reviews) then there is no issue with sending traffic to a plaintext resource.
It seems like ArcGIS Pro didn't handle blocked connections to OCSP and CRL links well and threw a misleading error message when it could not get past the failed certificate revocation checks.
Resolution:
The customer's firewall team created and documented exceptions to allow outbound connections to:
- http://s.ss2.us/r.crl
- http://ocsp.rootg2.amazontrust.com/*
- http://crl.rootg2.amazontrust.com/rootg2.crl
- http://crl3.digicert.com/DigiCertGlobalRootG2.crl
- http://crl.rootca1.amazontrust.com/rootca1.crl
- http://ocsp.digicert.com/*
For good measure, they also added http://ctldl.windowsupdate.com to the allow list, since that's the resource Windows uses to check for Windows updates, including to update it's own list of root CA certs.
Meanwhile, we're working with support to improve our doc and error messaging/handling to make this a little more obvious, as we expect more and more users to run into this situation as they implement ZTA patterns.
References: