Error performing query operation?

[Claire_Peterson]

Any help would be appreciated...

we have a few hosted map services that sit on a local ArcGIS Server. Starting around the 14th of this month, a few of the services started to experience some issues and not load correctly in the web maps/web mapping applications and operations dashboard. Nothing has changed in the settings, etc so I am curious why this has started to occur.

In the server logs I keep received error messages like:

Error performing query operation Error handling service request :Processing request took longer than the usage timeout for service '**'.

Processing request took longer than the usage timeout for service '**'. Server request timed out. Check that the usage timeout is appropriately configured for such requests.   

Error performing query operation Error handling service request :java.lang.OutOfMemoryError: Java heap space

Instance of service '**' failed to process a request.

ArcGIS Data Store has detected an issue with db.

We have increased the heap sizes which has corrected the java heap space error message but that is about the only thing that has been solved.

What's bizarre though, is that we have another service that has editing enabled and sits in the same location as the other services but is working just fine (though it does not have as many records as the problem services)

Thank you for your time!

[George_Thompson]

Are there any errors in the Data Store logs?

[Claire_Peterson]

There are 4 log files under C:\arcgisdatastore\logs\machine_name\server but only 3 have messages

<Msg time="2019-02-19T09:43:21,798" type="WARNING" code="110130" source="Data Store" process="3596" thread="1" methodName="" machine=" " user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-19T09:43:57,762" type="WARNING" code="110130" source="Data Store" process="3596" thread="1" methodName="" machine=" " user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-19T09:44:33,713" type="WARNING" code="110130" source="Data Store" process="3596" thread="1" methodName="" machine=" " user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-19T09:44:33,713" type="SEVERE" code="110131" source="Data Store" process="3596" thread="1" methodName="" machine=" " user="" elapsed="">ArcGIS Data Store encountered too many problems. Failover may be invoked if standby is configured.</Msg>
<Msg time="2019-02-19T09:44:33,718" type="SEVERE" code="110653" source="Data Store" process="3596" thread="1" methodName="" machine=" " user="" elapsed="">Data store machine '**' has failed.</Msg>
<Msg time="2019-02-19T09:51:45,72" type="WARNING" code="110129" source="Data Store" process="3596" thread="1" methodName="" machine=" " user="" elapsed="">Attempt to start ArcGIS Data Store service failed. Cannot run program "C:\Program Files\ArcGIS\DataStore\bin\pkill.exe": CreateProcess error=19, The media is write protected</Msg>

<Msg time="2019-02-19T10:17:13,980" type="WARNING" code="110130" source="Data Store" process="2824" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-19T10:17:49,987" type="WARNING" code="110130" source="Data Store" process="2824" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>

<Msg time="2019-02-02T00:00:48,156" type="WARNING" code="110718" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">Error during relational data store vacuum/analyze maintenance on ' '. Failed to execute a SQL script.</Msg>
<Msg time="2019-02-14T00:01:34,443" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-15T09:13:35,595" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-15T09:14:44,168" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-18T13:43:35,191" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-18T13:48:34,188" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-18T13:49:35,673" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-18T15:34:36,659" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-18T15:35:13,941" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-18T15:52:35,720" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-18T15:53:13,237" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-18T15:53:50,126" type="WARNING" code="110130" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store has detected an issue with db.</Msg>
<Msg time="2019-02-18T15:53:50,128" type="SEVERE" code="110131" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">ArcGIS Data Store encountered too many problems. Failover may be invoked if standby is configured.</Msg>
<Msg time="2019-02-18T15:53:50,135" type="SEVERE" code="110653" source="Data Store" process="3164" thread="1" methodName="" machine="**" user="" elapsed="">Data store machine '**' has failed.</Msg>

[George_Thompson]

I would recommend that you contact Esri Technical support and work with them on this issue.

Did the DS machine run out of disk space?

[Claire_Peterson]

Unfortunately we have to go through someone here on campus in order to reach out to tech support so I am hoping it won't take too long as now all of our services are down on this machine.

That is a possibility. I just restarted the machine and it seems to be running a bit better but still timing out.