nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

All our hosted feature services broken this morning

Matt-Goodman

Global news outlets this morning are reporting massive interruption to the airline industry, banking, etc. due to a bug/failure of Microsoft's CrowdStrike software. Global tech outage disrupts industries, highlights online risks | Reuters

Coincidentally, none of our hosted feature services in Enterprise Portal for ArcGIS are working or available. Their data will not load, they cannot be added to a map, they cannot be overwritten, they cannot be accessed, except to the item 'details' page in Portal.

Are these two things related? I'm not sure. What would cause just the hosted feature services to break, while everything else in our Portal environment seems to work fine?

Encuentra más publicaciones etiquetadas con

Administration

Sharing and collaboration

Data and services

Accepted answers

Matt-Goodman

Finally resolved things...

The main/specific cause was the Crowdstrike update. Most of our servers were "blue screen of death'd" over night. The process to fix it, was to boot in safe mode and delete an offending file. For whatever reason, one GIS server was still in safe mode....may have been attended to and had a hiccup?

All comments

JoshuaBixby

Not to nitpick on semantics, but saying "Microsoft's Crowd Strike software" implies: 1) the software is called 'Crowd Strike', and 2) Microsoft either develops or owns the software. CrowdStrike is a company and not a software product, and the company is not owned by Microsoft. The affected software is Falcon Sensor, part of the CrowdStrike's Falcon multi-platform security product. The botched update causing the issue only impacts Microsoft Windows software, but that doesn't mean Microsoft has anything to do either directly or indirectly with what is happening.

It is impossible to say whether your hosted feature services are impacted by the CrowdStrike issue because no one in the community here knows your organization's technical architecture and software products.

CodyPatterson

Hey @Matt-Goodman

I would review with your internal IT to see if the enterprise servers are running with Crowdstrike. I'm not entirely sure the extent of the damage that happened due to the issue, but if it extended to your servers, you will end up having service downtime. Your IT department will be able to tell you if that's the anti-virus software that is currently being used.

If it is not the case that Crowdstrike is being used, I would recommend a server restart, and then a consultation with ESRI support to see if it may be another issue!

Hope that helps!

Cody

NicoleGamble1

Echoing Cody's advice, I would recommend to log a tech support ticket for assistance if you determine Crowdstrike isn't a factor.

Here's where I would normally start with this kind of issue:

-Does the relational ArcGIS Data Store validate successfully in ArcGIS Server Manager?

-Are there any related errors in the ArcGIS Server logs?

-Does restarting the ArcGIS Server service in windows resolve the issue?

-If yes, was there windows patching last night? Check if this patch applies to your version if so:
https://support.esri.com/en-us/patches-updates/2023/arcgis-server-hosted-services-restart-patch

Matt-Goodman

Thanks, we're still struggling with the issue, but I'm working with our smarter I.T./database/server folks.

We've restarted our Portal server and the Web Adapter, to no avail.
On the hosting server, in Server Manager, the Data Stores do not validate (Relational, Spatiotemporal, Tile Cache).

Next attempt: try rebooting all the various GIS servers? IDK, grasping at straws here....

ChelseaRozek

Is the web adapter on the same server as portal? When "restarting" it, did you restart the whole server or refresh the app pools or restart IIS?

What version of Enterprise are you on? Your situation reminded me of this thread: https://community.esri.com/t5/arcgis-enterprise-questions/arcgis-web-adaptor-11-1-app-pool-freezes/m-p/1291473#M36004 which I believe we encountered recently as well, but it's so infrequent it's hard to pin down

NicoleGamble1

If your Data Stores don't validate, I would focus on getting that fixed - you can check the ArcGIS Server logs to see if they have more details on why they aren't validating. You can also run the describedatastore command directly on the Data Store machine(s) to see if they are functional on that side, and consider restarting the ArcGIS Data Store service in windows.

Technical support would likely be able to guide you though troubleshooting the issues with your Data Stores so I'd recommend logging a ticket if you haven't already. Hope this helps!

Matt-Goodman

AprilChipman

We are having the same issue. Our hosted feature services aren't working, and we can't get to any of the surveys on the Survey123 site. We have rebooted the server and the Data Store validates. Should I try restarting the Data Store service on the server?

MattReynolds

We are seeing something similar after a forced reboot last night.

Describedatastore.bat returns:

Error encountered: No valid connection to ArcGIS Data Store configuration store established.
Caused by: Connection to localhost:9876 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

YoloCounty

Anyone else fixed this issue?

I have two separate Enterprise environments that are both still down after the Crowdstike fix was applied.Things I have done so far:

I read Matt's solution and double checked that none of my servers are still in Safe Mode.
Restarted all of my servers
Manually stopped and restarted all of my services
Applied patches (Check for ArcGIS Enterprise Updates)
I called Tech Support a couple of hours ago, but no call back yet.
Ran describedatastore.bat - no issues

I can't even login to portal admin or portal at all in either environment. I also can't login to server manager in either environment, so it seems to be a portal issue since both environments are federated. I can log in to server admin.

AprilChipman

We are getting the same error message. ESRI Tech Support said we needed to restart the machine and then restart the windows services in a particular order. No luck yet.

AprilChipman

Well, I'm not sure what fixed it, but our system is working again. Here are the steps I was sent by ESRI:

1) restart machine
2) stop/start the Windows service for Portal for ArcGIS
3) stop/start the Windows service for ArcGIS DataStore
4) wait 10 minutes or until Portal is running correctly
5) stop/start the Windows service for ArcGIS Server
6) validate DataStore from Server Manager
7) validate the federation from Portal

I didn't do step 1 since I had already done it today. I had also already tried restarting the Windows services, but maybe I just didn't start and stop the Windows services in the right order?

Or, maybe something else was magically fixed elsewhere and nothing I did resolved the issue, whatever the issue was?

I dunno, but it's working. I may call it a day and start my weekend early...

SimonJackson

Have applied the workaround from Crowdstrike/AWS to bring our environment back online. However, the ArcGIS Enterprise environment is not starting correctly. We get this in the logs:

"

The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.)."

"Failed to start the portal. The observer's beforeStart() function returned a failure."

Will be troubleshooting in more detail tomorrow and likely returning to a prior snapshot as suspect the BSOD would have been what corrupted things. But if anyone else has seen similar issues and knows of a workaround, please share.

MattReynolds

For me the problem turned out to have two components.

First there was a corrupted config file for the tomcat server in DataStore: C:\Program Files\ArcGIS\DataStore\framework\runtime\tomcat\conf\server.xml ESRI Tech support helped track this down. This prevented the embedded postgres from starting. This is a generic server conf file, you can grab a clean one from a fresh install or another datastore machine of the same version without Crowdstrike installed (what I did).

And second, the Crowdstrike “C-00000291*.sys” files needed to be deleted. Once I deleted that and restarted, port 2443 opened up. In Azure we had to take a snashot of the datastore's OS drive, make a disk from it, and mout it to antoher VM. Then delete the “C-00000291*.sys” files from it. Then detach the modified OS disk, and swap it back into the failed datastore VM.

My VM backup from 7/18 was corrupted (probably by Crowdstrike), so unless we fixed it, we would lose a day of fieldwork and editing.

I hope this helps you out. We all had a rough day yesterday.

-Matt

LeeIrminger2

Great post, thank you! Since Crowdstrike had already updated the “C-00000291*.sys” file - timestamp of 7:59 AM UTC - it did not have to be deleted after replacing the server.xml file. Falcon Content Update Remediation and Guidance Hub | CrowdStrike

Jon Emch

Hello folks! Adding my voice to say that this is a priority issue for Esri to revolve. If you find yourself in a position where services are not starting properly please log a tech support ticket. We are able to assist to get you back up and running.

LeeIrminger3

Hi Matt,

Have you had issues reaching the ArcGIS Data Store config page over 2443 after using that fix?

Únete a la Conversación

Los miembros registrados pueden publicar, seguir actualizaciones y más. ¿Nuevo aquí? Regístrate gratis.

Enlaces Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.