ArcGIS Server Log Severe questions

[LavanyaVasudevanNES]

Hello, 

I am seeing a weird server log issues on my ArcGIS Server Manager 11.3.  We recently enabled Organization only Sharing for a lot of our services to help reduce traffic and SOC counts that were crashing our servers.   We are in the process of fine tuning the number of services and grouping them so we have fewer dedicated instances.  I am seeing some weird errors and nothing online seems to help me with a solution.  Can anyone here help.  Below are the 4 errors within the last 24 hours that keep taking over our log files.  

Failed to obtain item from data store for given id: ()

Response already committed. Cannot forward to error page.

Unable to process request. java.lang.NullPointerException: Cannot invoke "com.esri.client.AGSResponse.toResponseJson()" because "agsResponse" is null

Error performing query operation Error handling service request :Could not service request.

[TimoT]

Hi @LavanyaVasudevanNES ,

Based on your note about switching many services from dedicated to shared instances, the types of errors you're seeing sound consistent with the shared instance pool being overloaded or non-optimally configured.

Just to narrow things down a bit:

• Are any of the errors in your Server Manager logs originating from the DynamicMappingHost service?
  If so, that’s a strong indicator as this service is directly related to the Shared Instance Pool.

Some follow-up questions that may help clarify the situation:

1. Is there any noticeable impact to users when these errors occur? (e.g. layers failing to load, slow performance, etc)
2. Roughly how many services have been configured to the Shared Instance Pool?
3. Are any of these services high-traffic, complex (e.g. with heavy querying), or tied to large data sources?
4. If you check Server Manager > Site > Settings, how many instances are currently allocated to the shared instance pool?
5. How many ArcGIS Server machines are part of the affected server site?
6. What does CPU and memory usage look like on the ArcGIS Server machine(s) when the errors occur?
7. How many ArcSOC.exe  processes are running during normal and peak times? (Instructions to check here)
8. Since you've mentioned dedicated vs shared instances, I’m assuming your services are not Hosted Feature Layers. If the services are referencing data from a geodatabase, have you noticed any resource or performance issues on the database server itself?

[LavanyaVasudevanNES]

Thankyou Tim. 

Yes, this one specific set up has over 300 layers and most of them are on the shared pool.  A lot of the errors are related to the DynamicMappingHost service.  We have set the min and max to 16 and it seems to be a stable.  Yes, rendering of the data seemed super slow for a lot of the users.  This is a single ArcGIS Server deployment.  There were about 97-110 ArcSOCs running whenever we tried to zoom in/pan in a map.  The thing that seemed to work for us to is to get the portal/server/datastore directions exempted from virus scanning.  The OpenJDK and the Antivirus scans were over 60% of the resources and ESRI tech suggested the folder exceptions to help.  

Next steps would be to go over the ESRI hardening guide in detail to make sure it is okay to set these AGE folders under exceptions.  We plan on consolidating some of the services and redo the publishing so that we don't need to have so many Shared Instances.  We noticed there were bulk services published within a folder, and syncing hasn't worked since 2023.  I am thinking this is a different issue but in the greater scheme of things possibly causing resources issue on the server.  

[SergioGalindo1]

Hi!

About your comment: "get the portal/server/datastore directions exempted from virus scanning"; do you mean add exceptions in each component (portal, server, datastore) so the local folders aren't scanned anymore by the antivirus software?

Thanks in advance.

[LavanyaVasudevanNES]

Yes that is what was suggested to us by ESRI Tech support.  Seems very counter intuitive to exempt all ArcGIS Folders from antivirus.   There has got to be a better way than this.  

[SergioGalindo1]

Thanks for your feedback!

Regarding your Antivirus, are you using Windows Defender, another product? Is your local firewall also disabled?

[LavanyaVasudevanNES]

Windows Defender and no firewall isnt disabled.   

[DougBrowning]

Checking in to see how this is going for you?  We started getting Failed to obtain item from data store for given id: () also.  The admins set us up to do everything as shared to start.  As we migrate to the new server more and more we started getting these and really poor performance.  We doubled the number of shared but still having issues.  Seems like moving anything that has issues to dedicated has helped.  But some still did crash out.  We have over 300 services now and have it set to 16 instances on a 8 core with 50 for each (which I think is the default).  Server is CPU under 30% and RAM under 40% used.  Its a VM. 

thanks for any info

[AndreaB_]

Hi all,

We are seeing "Failed to obtain item from data store for given id: ()" constantly in server logs. I'm using Enterprise 11.3 one machine deployment with Server, Portal, Data store all on one machine.  A lot are in shared pool but I've been moving some to Dedicated as I see errors.  A lot of the errors are related to the DynamicMappingHost service.  I have Shared Instance Settings Number of shared instances per machine set to 8. I'm running 4 core. Server CPU around 30% and RAM 57%. Also a VM.  We have 28 services, 18 in Shared pool. We've increased the Database Server resources a few times already because we had some concern there with the logs from SSMS.  There were 38 ArcSOCs running.

For those of you with insight - how does this set up sound? Is this error just a bug or actually telling us something?

**Edit - it is a bug! just found this https://support.esri.com/en-us/bug/publishing-from-arcgis-pro-335-to-arcgis-enterprise-suc-bug-00017... 

Thank you!

A

[DougBrowning]

I now confirmed it has been on our main older productions servers for a long time now.  No one has ever figured it out so just been ignoring it.

This is way before 3.3.5 too.