topic Re: Workforce - Migrate Assignments to different Account in ArcGIS Workforce Questions

Workforce - Migrate Assignments to different Account

JonathanGroves — Thu, 19 Oct 2017 11:33:51 GMT

I look after Workforce for an Enterprise Client organisation.

They have just procured an on-prem instance of ArcGIS and I am migrating the Assignments from the old (ArcGIS Online Workforce) to new (private Workforce) instance.

I have created a new Project, set up the required Assignment Types and given the Users the right Roles.

I have exported all the Assignments from the old instance using the Python standalone script export_assignments_to_csv.

I am editing the produced CSV file to make it compatible with the create_assignments_from_csv Python standalone script so that I can use this to upload the Assignments from the ArcGIS Online to the private Workforce...

I have a bunch of Status field entries in my export, which I imagine reflect the Completed/In Progress/Paused etc status of each task - but the create_assignments documentation does not say that I can import these.

Please advise how I can import the status fields?

Or... is there a better way to achieve what I'm trying to do?

Thanks so much in advance for your help.

Jonathan

Re: Workforce - Migrate Assignments to different Account

CraigGillgrass — Fri, 20 Oct 2017 23:51:36 GMT

Hi Jonathan,

A couple things you're going to want to keep in mind when migrating a Workforce Project and it's data to another project:

Workforce uses a Coded Value Domain for storing the Assignment Types, so there may be gaps in the code for the Assignment Types. You'd see this in the new Assignments feature service when viewing the data.
For the Workers and Dispatchers; the same issue can arise; you may have gaps in the ObjectID field for each of the feature services could cause issues in the new project. You can also see this in the feature service data.

Can you go into a little more details on your comment about not being able to import the status fields? This should just work; the codes for the status fields should match up with the status field values in the new project.

Craig

Re: Workforce - Migrate Assignments to different Account

JonathanGroves — Sat, 21 Oct 2017 05:44:02 GMT

Hi Craig.

Thanks for your reply.

Yes, I’d noticed the coded type values: I have created dummy assignments containing each type with the type name as text in the description. As you note, the same issue arises with Workers but I have included all the variations in dummy assignments similarly. (This Project only has one Dispatcher.) Exporting these dummy assignments using the download to CSV script, then inspecting the CSV reveals the correlation between keys and values, so I can then search and replace in my input file with ease.

Regarding the Status field... the documentation for the create_assignments_from_csv script does not mention Status as one of the available fields. What column header text must I use?

The code in the create_assignments_from_csv.py script which handles passed parameters from the script invocation shows this:

if __name__ == "__main__":
    # Get all of the commandline arguments
    parser = argparse.ArgumentParser("Add Assignments to Workforce Project")
    parser.add_argument('-u', dest='username', help="The username to authenticate with", required=True)
    parser.add_argument('-p', dest='password', help="The password to authenticate with", required=True)
    parser.add_argument('-url', dest='org_url', help="The url of the org/portal to use", required=True)
    # Parameters for workforce
    parser.add_argument('-pid', dest='projectId', help="The id of the project to add assignments to", required=True)
    parser.add_argument('-xField', dest='xField', help="The field that contains the x SHAPE information", required=True)
    parser.add_argument('-yField', dest='yField', help="The field that contains the y SHAPE information", required=True)
    parser.add_argument('-assignmentTypeField', dest='assignmentTypeField',
                        help="The field that contains the assignmentType", required=True)
    parser.add_argument('-locationField', dest='locationField',
                        help="The field that contains the location", required=True)
    parser.add_argument('-dispatcherIdField', dest='dispatcherIdField',
                        help="The field that contains the dispatcherId")
    parser.add_argument('-descriptionField', dest='descriptionField', help="The field that contains the description")
    parser.add_argument('-priorityField', dest='priorityField', help="The field that contains the priority")
    parser.add_argument('-workOrderIdField', dest='workOrderIdField', help="The field that contains the workOrderId")
    parser.add_argument('-dueDateField', dest='dueDateField', help="The field that contains the dispatcherId")
    parser.add_argument('-attachmentFileField', dest='attachmentFileField',
                        help="The field that contains the file path to the attachment to upload")
parser.add_argument('-workerField', dest='workerField', help="The field that contains the workers username")
    parser.add_argument('-dateFormat', dest='dateFormat', default=r"%m/%d/%Y %H:%M:%S",
                        help="The format to use for the date (eg. '%m/%d/%Y %H:%M:%S'")
    parser.add_argument('-timezone', dest='timezone', default="UTC", help="The timezone for the assignments")
    parser.add_argument('-csvFile', dest='csvFile', help="The path/name of the csv file to read")
    parser.add_argument('-wkid', dest='wkid', help='The wkid that the x,y values are use', type=int, default=4326)
    parser.add_argument('-logFile', dest='logFile', help='The log file to use', required=True)
    args = parser.parse_args()

... so I don't see how I can ever pass to the script which fieldname to use for the status?

Much obliged.

Re: Workforce - Migrate Assignments to different Account

CraigGillgrass — Wed, 01 Nov 2017 18:39:48 GMT

Hi Jonathan,

I understand your point; the create_assignments_from_csv script was meant to import new assignments (and optionally assign them to a worker); it was not written to port assignments from an existing instance.

The script can be modified to import status values, unfortunately, this isn't an update we have planned for the scripts.

Craig

Re: Workforce - Migrate Assignments to different Account

JonathanGroves — Wed, 01 Nov 2017 18:46:10 GMT

Thanks, Craig. But according to the Issue on GitHub it doesn’t even import basic fields like DueDate - which would surely be needed for everyday use in a use case where Assignments need to be batch imported from almost any other system...

I take your point about not supporting migration - we’ve already accepted this can’t be done - but if we can’t import planned Assignments in any way so they are usable, my Client will have to abandon Workforce and build a bespoke system instead.

Kind Regards,

Jonathan.

Re: Workforce - Migrate Assignments to different Account

CraigGillgrass — Wed, 01 Nov 2017 18:58:25 GMT

Jonathan,

I agree that fields such as Due Date would be needed for everyday use when importing from another system. That is the goal behind the scripts.

We have updated the scripts to support 10.5.1; can you confirm that you're using the most recent version of the scripts? If you are, could you supply a sample csv file you're using to upload to a Workforce Project on a 10.5.1 portal?

thanks,

Craig

Re: Workforce - Migrate Assignments to different Account

CraigGillgrass — Wed, 01 Nov 2017 19:21:55 GMT

We just tagged a new release 1.2 for the scripts to clarify the updates.

Re: Workforce - Migrate Assignments to different Account

JonathanGroves — Wed, 01 Nov 2017 19:31:11 GMT

Thanks, Craig.

I am indeed using the latest scripts, at least they were the latest when I double-checked last week.

I’ll test again tomorrow and provide exact details of what I observe and the input data.

Kind Regards,

JG.

Re: Workforce - Migrate Assignments to different Account

JonathanGroves — Thu, 02 Nov 2017 10:22:33 GMT

Hi Craig.

Here's the bash script (create_assignments_onprem.sh) I use to run the Enterprise assignments upload from csv:

#!/bin/bash
# Onwave 2017
# Runs Python script: create_assignments_from_csv.py
# which imports WorkForce assignments from a csv file named inputfile.csv
# csv format thus:
# Header row: xField,yField,Type,Location,Dispatcher,Description,Priority,Work Order Id,Due date,attachment
# Body rows follow this layout, comma separated, no trailing spaces.
# Field descriptions:
#       xField          = x coord (longitude) as decimal value
#       yField          = y coord (latitude) as decimal value
#       Type            = type of assignment (must be an integer)
#       Location        = Description of location, i.e. address
#       Dispatcher      = Name of Dispatcher adding data (perhaps use version of schedule file?)
#       Priority        = Priority rating
#       Work Order Id   = Job identity
#       Due Date        = Date job must be completed by
#       attachment      = Filename of file to attach
#       worker          = name of worker
#
# This version updated to allow for On-Prem WorkForce - Oct 2017 - Jonathan Groves

# Build parameter string (q):
q="sudo python create_assignments_from_csv.py "
# csv filename:
q+="-csvFile \"./inputfile.csv\" "
# Username:
q+="-u \"Jonathan.Groves\" "
# Password:
q+="-p \"xxxx\" "
# url:
q+="-url \"https://xxxx\" "
# Project ID:
q+="-pid \"xxxx\" "
# xField - name of xField column:
q+="-xField \"xField\" "
# yField:
q+="-yField \"yField\" "
# Type:
q+="-assignmentTypeField \"Type\" "
# Location:
q+="-locationField \"Location\" "

# Description:
q+="-descriptionField \"Description\" "
# Priority:
q+="-priorityField \"Priority\" "
# Work order Id:
q+="-workOrderIdField \"Work Order Id\" "
# Dispatchers:
q+="-dispatcherIdField \"Dispatcher\" "
# due date:
q+="-dueDateField \"Due Date\" "
# attachment:
q+="-attachmentFileField \"attachment\" "
# wkid - spatial reference used by xField and yField
# NB defaults to 4236 (GCS_WGS_1984)
#q+="-wkid 102100 "
# Logfile:
q+="-logFile \"./log.txt\" "
# Worker:
q+="-workerField \"Worker\" "
# Timezone:
q+="-timezone \"US/Eastern\" "

# Debug:
#echo ">"$q"<"
eval $q

Here is the output when I run it:

./create_assignments_onprem.sh
[2017-11-02 10:05:36,356] [create_assignments_from_csv.py: 261 -                           main()]         [MainThread] [      root] [    INFO] Authenticating...
[2017-11-02 10:05:36,608] [create_assignments_from_csv.py: 345 -                       <module>()]         [MainThread] [      root] [CRITICAL] Exception detected
, script exiting
[2017-11-02 10:05:36,609] [create_assignments_from_csv.py: 346 -                       <module>()]         [MainThread] [      root] [CRITICAL] No JSON object cou
ld be decoded
[2017-11-02 10:05:36,612] [create_assignments_from_csv.py: 347 -                       <module>()]         [MainThread] [      root] [CRITICAL] Traceback (most re
cent call last): |   File "create_assignments_from_csv.py", line 343, in <module> |     main(args) |   File "create_assignments_from_csv.py", line 263, in main |
    token = workforcehelpers.get_token(args.org_url, args.username, args.password) |   File "/home/pi/arcgis/workforcehelpers.py", line 78, in get_token |     res
ponse = post(url, data) |   File "/home/pi/arcgis/workforcehelpers.py", line 40, in post |     response = requests.post(url, data, files=files).json() |   File "/
usr/local/lib/python2.7/dist-packages/requests/models.py", line 885, in json |     return complexjson.loads(self.text, **kwargs) |   File "/usr/lib/python2.7/json
/__init__.py", line 338, in loads |     return _default_decoder.decode(s) |   File "/usr/lib/python2.7/json/decoder.py", line 366, in decode |     obj, end = self
.raw_decode(s, idx=_w(s, 0).end()) |   File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode |     raise ValueError("No JSON object could be decoded"
) | ValueError: No JSON object could be decoded |

I successfully ran my original script on the Client's AGOL-based Workforce yesterday using exactly the same input file - here's that script (create_assignments_production.sh):

#!/bin/bash
# go.sh - Onwave 2017
# Runs Python script: create_assignments_from_csv.py
# which imports WorkForce assignments from a csv file named inputfile.csv
# csv format thus:
# Header row: xField,yField,Type,Location,Dispatcher,Description,Priority,Work Order Id,Due date,attachment
# Body rows follow this layout, comma separated, no trailing spaces.
# Field descriptions:
#       xField          = x coord (longitude) as decimal value
#       yField          = y coord (latitude) as decimal value
#       Type            = type of assignment (must be an integer)
#       Location        = Description of location, i.e. address
#       Dispatcher      = Name of Dispatcher adding data (perhaps use version of schedule file?)
#       Priority        = Priority rating
#       Work Order Id   = Job identity
#       Due Date        = Date job must be completed by
#       attachment      = Filename of file to attach
#       worker          = name of worker

# Build parameter string (q):
q="sudo python create_assignments_from_csv.py "
# csv filename:
q+="-csvFile \"./inputfile.csv\" "
# Username:
q+="-u \"Jonathan.Groves\" "
# Password:
q+="-p \"xxxx\" "
# url:
q+="-url \"https://xxxx.maps.arcgis.com\" "
# Project ID:
q+="-pid \"xxxx\" "
# xField - name of xField column:
q+="-xField \"xField\" "
# yField:
q+="-yField \"yField\" "
# Type:
q+="-assignmentTypeField \"Type\" "
# Location:
q+="-locationField \"Location\" "

# Debug:
#echo ">"$q"<"
eval $q

As you can see, the only differences between the two is the url and the PID.

Oh - here's a snippet of the csv file:

xField,yField,Type,Location,Dispatcher,Description,Priority,Work Order Id,Due Date,attachment,Worker
0.101466667,50.77328333,1,Seaford,1,Cut 6,3,USFD,10/23/2017 23:59:59,,eshgrass1
0.195244444,50.85660278,1,Upper Dicker,1,Cut 6,3,UUDK,10/30/2017 23:59:59,,eshgrass1
0.166230556,50.83036389,1,Berwick,1,Cut 6,3,UBRK,10/30/2017 23:59:59,,eshgrass1
0.156430556,50.80839167,1,Alfriston,1,Cut 6,3,UALF,10/30/2017 23:59:59,,eshgrass1
0.20045,50.819,1,Wilmington,1,Cut 6,3,UWLM,10/30/2017 23:59:59,,eshgrass1
0.204241667,50.764325,1,Friston,1,Cut 6,3,UFRS,10/31/2017 23:59:59,,eshgrass1

Sadly the Python script from your GitHub has no readily-available version information but, from what I can see in your GitHub, the script has not changed since I downloaded it.

Any help you can give will be most gratefully received.

Kind regards,

Jonathan

Re: Workforce - Migrate Assignments to different Account

CraigGillgrass — Thu, 02 Nov 2017 20:15:10 GMT

Hi Jonathan, thanks for sending this, we'll dig into it and get back to you. It will be next week, as we're finishing up a number of releases.

Craig

Re: Workforce - Migrate Assignments to different Account

CraigGillgrass — Fri, 10 Nov 2017 20:39:06 GMT

Jonathan, we're still looking at this - should have something next week.

Re: Workforce - Migrate Assignments to different Account

JonathanGroves — Fri, 10 Nov 2017 20:42:28 GMT

Thanks, Craig.

Kind Regards,

JG.

Re: Workforce - Migrate Assignments to different Account

Anonymous User — Sat, 16 Dec 2017 16:59:03 GMT

Jonathan,

Only the scripts that use the Python API for ArcGIS have been updated to work with 10.5.1 Portal. It was too much effort to maintain 3 different versions of the scripts and moving forward the Python API will continue to wrap many things such as querying/authentication/updating...

The main difference between online and portal, is the field names are lowercase due to the Postgres datastore. To use the Python API scripts, you'll have to install Conda. It appears that you're running this on a Raspberry Pi? You can install conda using a shell script from from https://repo.continuum.io/miniconda for earlier raspberry Pi's you'll need the arm v6 for newer versions, you'll need arm v7.

Additionally, we can look into adding the status field as an optional parameter to the import_assignments_from_csv.py script.