Workforce Not Loading Assignments/Workers

1652
7
05-09-2019 03:43 PM
by Anonymous User
Not applicable

We have a couple of workforce projects that are opening, but not loading any assignments or workers. The data is still in ArcGIS Online, but the "loading" message never goes away. When the assignments finally show up, we can't assign anyone as it shows no workers as part of the project. We've opened up the app from both the administrator account and other dispatcher accounts. Is anyone else experiencing these issues? Only started happening within the last 2 hours.

7 Replies
AndresCastillo
MVP Regular Contributor

Karen,

Did you find resolution to this?

We experience the same

Datastore hosted services on ArcGIS Enterprise goes down after a certain amount of requests are made on workforce hosted services.

Kelly Gerrow

Craig Gillgrass

Jeff Shaner

Aaron Pulver

Kylie Donia

Rickey Fite

Xander Bakker

Joshua Clifford

0 Kudos
by Anonymous User
Not applicable

Hi Andres, 

This issue sorted itself out after a few days, so I'm not actually sure what causes the issue - if it's an ESRI update/hosted feature service problem, some of the configurations made to the assignments layer, or a browser issue.

We still experience this issue every now and then, but not too frequently. When it does happen, we can use an incognito browser and that usually fixes the problem. 

What I have found causes issues is if you edit the assignments layer in ArcGIS Pro. Our workforce project has been buggy ever since we did this, so I would recommend editing the assignments in a Web App, Collector, through python, etc. instead. Here's what I mean:

Wouldn't recommend doing that!

by Anonymous User
Not applicable

Hi Andres,

Can you send an email to workforce4arcgis@esri.com with this info:

  • Enterprise version
    • Deployment configuration (cores, RAM, machines, ...)
  • Number of workers in the project
  • Number of assignments in the project
  • A har file of the requests causing issues - if you have them
  • Any applicable server or datastore logs

It will probably be easier to sort out the cause this way as it is likely project/environment specific.

AndresCastillo
MVP Regular Contributor

will do.

We have an escalated ticket with Esri Support, so I will include that email.

Thank you for your reply.

0 Kudos
AndresCastillo
MVP Regular Contributor

The solution seems to be to disabling the sync option in the hosted feature layer settings:

The analyst correlated the server and the postgresql logs with inserts into "location__d9188f26f2343f78951e16ec_locationtracking_evw" and the disconnection of the database. Having *_evw is an indicative that sync was enabled on some feature services.

Comment from Esri Support Analyst------------------------------------------------------------------------------------------------------------------------------------------------------------

Enabling sync on a hosted feature layer may increase the feature storage size.

So it is possible that there could be archived records in the table in the Data Store postgres database.

Please refer to the "Allow offline editing or collaboration" section here:

Manage hosted feature layers—Portal for ArcGIS | Documentation for ArcGIS Enterprise 

Disabling sync on the hosted feature layer would empty the archiving information from the table in the Data Store.

I suspect that this could attribute to the Data Store functioning properly today.

End comment------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I saw that AGO says Workforce Beta is available with offline support.

Eventually, the same should come with ArcGIS Enterprise.

Does our sync issue reveal that we need to increase the PostgreSQL datastore machine storage?

I believe we had plenty of space in that machine.

Is there a best practice to configure the datastore to handle sync enabled?

Sync would be a requirement for future offline workflows.

We are using soap ui/JMeter to load test the hosted feature services created by the workforce project using a query to the database:

https://community.esri.com/groups/technical-support/blog/2012/06/14/preparing-for-the-flood-will-you... 

First, get a temporary token for

https://{portal.domain.com}/{webadaptor}/admin/

from the Federated and Hosting ArcGIS Server:

https://{arcgisserver.domain.com}/{webadaptor}/sharing/rest/generateToken

Workers:

https://{portal.domain.com}/{webadaptor}/rest/services/Hosted/workers_{itemID}/FeatureServer/0/query?where=1=1&token={token}&f=json

Assignments:

https://{portal.domain.com}/{webadaptor}/rest/services/Hosted/assignments_{itemID}/FeatureServer/0/query?where=1=1&token={token}&f=json

Location:

(testing system to see if parsing long strings)
https://{portal.domain.com}/{webadaptor}/rest/services/Hosted/location_{itemID}/FeatureServer/0/query?where=1%3D1&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&distance=&units=esriSRUnit_Foot&relationParam=&outFields=&returnGeometry=true&maxAllowableOffset=&geometryPrecision=&outSR=&havingClause=&gdbVersion=&historicMoment=&returnDistinctValues=false&returnIdsOnly=false&returnCountOnly=false&returnExtentOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&multipatchOption=xyFootprint&resultOffset=&resultRecordCount=&returnTrueCurves=false&returnCentroid=false&sqlFormat=none&resultType=&datumTransformation=&token={token}&f=json

0 Kudos
AndresCastillo
MVP Regular Contributor

2 weeks later, the workforce project started causing issues again.

The Esri analyst said:

location__d9188f26f2343f78951e16ec_locationtracking_evw before crashing (could not receive data from client: An existing connection was forcibly closed by the remote host)
FATAL: no pg_hba.conf entry for host "::1", user "<username>", database "<dbInstanceName>", SSL off
the location tracking is inside this location service
I checked and did not see Sync under capabilities in the service rest
From the logs, this service may have archiving enabled on the database. This is what I would like to confirm in-house before making any suggestions on your end. 

I see logs with errors requesting the location service as well as many unauthorized access to the database under the couch logs.

Our reply:

Modify pg_hba.conf on Relational Datastore? 

Although we do not have the option to disable sync for location in the portal item page, in the admin endpoint for the location layer, we might be able to disable it by updating definition with the capabilities in json format, excluding the sync capability:
Esri reply:
Please find below instructions to update the service definition:

While logged in as an administrator, go to this location:  https://<serverurl>/webadaptor/rest/services/mysevice/FeatureServer?f=pjson and get the “capabilities”. E.g.:  "capabilities": "Query,Create,Update,Delete,Uploads,Editing,Sync",
Remove Sync from the list
Next go to this location https://<serverurl>/webadaptor/rest/admin/services/mysevice/FeatureServer/updateDefinition
Run the updateDefinition with the capabilities from step 2 (minus sync)  e.g.  { "capabilities": "Query,Create,Update,Delete,Uploads,Editing"}
0 Kudos
ahagopian_coj
Occasional Contributor

I know this topic is a bit old but did making the changes fix your sync issues?  We have been plagued with sync issues since moving our Workforce project into our Enterprise environment.

0 Kudos