topic Re: GP Service Multiple Processes from MXD in ArcGIS Enterprise Questions

GP Service Multiple Processes from MXD

LloydBronn — Thu, 09 Mar 2017 16:43:08 GMT

I have a GP service that generates a PDF report for a country based on a click event. This report includes charts generated on the fly with matplotlib, sourced from rasters. It also includes symbolized raster images exported from an MXD. While testing, we've found that if two users try to run the service at the same time, it errors out for both users. The browser console.log displays: "unable to complete operation." The ArcGIS server logs displays: " MapDocObject: Unable to save. Check to make sure you have write access to the specified file and that there is enough space on the storage device to hold your document. Failed to execute." We don't get these errors when we run the process one at a time.I'm assuming that the MXD can only be accessed by one user at a time. Is there a way for the server to queue users if more than one are trying to access this process at the same time? Would saving multiple numbered copies of the MXD solve the problem?

Re: GP Service Multiple Processes from MXD

NeilAyres — Thu, 09 Mar 2017 16:58:25 GMT

Certainly the GP service can be used by multiple users. Each time the GP service executes a new job is created on the server. So no conflict there.

What do you mean by the same "mxd"? Are users actually opening the same file on a network somewhere?

Re: GP Service Multiple Processes from MXD

LloydBronn — Thu, 09 Mar 2017 17:53:02 GMT

The MXD and all shapefiles and rasters are housed locally on the server machine. The user clicks on a country on the web map. That country name is passed as an input parameter to the GP Python script tool. The Python script accesses the MXD, creates background outline mask layers for the country and exports symbolized raster images masked to the country. The background layers stay generic, but the script determines what the last country source was, updates the layers to the new country and saves the MXD. I think the problem is that the MXD can't be editied and saved with two or more countries at once. I'm thinking the solution is to create multiple copies of the MXD and have the script determine whether the first MXD is in use, if so, move to the next one. I just need to figure out the best method for that in Python.

Re: GP Service Multiple Processes from MXD

KevinHibma — Sat, 11 Dec 2021 14:01:53 GMT

Without seeing your code, I assume you're doing some sort of .save() on the MXD object. I could imagine that 2 people trying to work off and save the MXD would fail. Two people (2 instances) simply reading the same MXD should be fine.

If you're in fact doing a save, change it to a .saveAs() and write the output of this operation to your scratch.:

myNewMXD = os.path.join(arcpy.env.scratchFolder, "newMXD.mxd")
mxdObject.saveAs(myNewMXD)
#... carry on processing/exporting from this new MXD.‍‍‍‍‍‍‍‍‍‍‍‍
newMXDobj = arcpy.mapping.MapDocument(myNewMXD)‍‍‍‍

Re: GP Service Multiple Processes from MXD

LloydBronn — Sat, 11 Dec 2021 14:01:55 GMT

Two people can't use the same MXD, because they will be creating different background layers based on the countries they've chosen. How about this?

try: 
 mxd = arcpy.mapping.MapDocument("C:\\Website_Test\\Report_Tests\\MXDs\\Analysis_For_Reports_1.mxd")
except RuntimeError:
 for num in range(2,6):
 mxd = "C:\\Website_Test\\Report_Tests\\MXDs\\Analysis_For_Reports_" + num + ".mxd"‍‍‍‍‍‍‍‍‍‍

Re: GP Service Multiple Processes from MXD

KevinHibma — Thu, 09 Mar 2017 18:39:02 GMT

While that code is fine.... and nobody is stopping you from doing that.... I really dont think you need to go that route. (personally I wouldnt, but I understand at the end of the day we all just want something that works)

I can open 2 instances of ArcMap and reference the same MXD object in each. I only receive a runtime error when I do a save (that matches your original error message). Having 1-n of instances of a GP Service executing against 1 MXD is the same principal. Each instance gets its own space (folder) to execute against. This folder is unique and other executing instances have zero knowledge of it -- thus no write problems. This is why I suggest doing a saveAs. All instances make use of 1 MXD (your source), do the modifications to it in memory (meaning just acting on the mxd object) and then do the final saveAs to persist this information and then export.

So like I said to begin with, if you really want to go down the path of multiple MXDs using a try/export....I wont stop you. It's just more heavy handed and will be more of a maintenance problem if you want to do updates later.

OHH EDIT -- ya, with the multiple MXD approach, if you're letting the publishing process copy the data, this will fail. The publishing process doesnt know to traverse code to find all MXDs and copy them. You'd need to create a variable that references the folder those MXDs exist in. That way the publishing process copies everything inside the folder (all mxds). Alternative if you set a data store folder to this location, nothing will need to be copied and it should work.

Re: GP Service Multiple Processes from MXD

LloydBronn — Sat, 11 Dec 2021 14:01:58 GMT

(I posted this reply before I saw your edit)

I see what you're saying, but I'm just not sure that it would actually work. The sources in the background layers keep changing based on the current chosen country. If two people pick Brazil and Iceland at the same time, I can't see how it wouldn't fail. Maybe I'm wrong. Here is the definition that updates the background layers.

def update_background_layers(mxd,df,input_country):
    
    background_layers_list = ["cities", "country", "oceanout","whiteout", "six_largest_cities"]

    for background_layer in background_layers_list:
        
        for lyr in arcpy.mapping.ListLayers(mxd):
            if lyr.supports("DATASOURCE"):
                
                if background_layer == str(lyr.name):
                    old_source = os.path.dirname(lyr.dataSource)
                    new_source = "C:\\Website_Test\\Report_Tests\\Background_Layers\\" + input_country
             
                    lyr.findAndReplaceWorkspacePath(old_source, new_source, False)

    mxd.save()
    return()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: GP Service Multiple Processes from MXD

KevinHibma — Sat, 11 Dec 2021 14:02:01 GMT

I'm not setup to quickly test, but I'm 99% certain line 16 (save) is the problem. User A choosing ICELAND executes and has their own MXDObject. User B chooses Brazil and again they have their own MXDObject. These two objects exist in their own process (arcsoc.exe) independent from each other. It is only at that time of line 16: mxd.save() do worlds collide and one process tries to write over the other process.

So, if you were to slightly modify your code:

def update_background_layers(mxd,df,input_country):
    
    background_layers_list = ["cities", "country", "oceanout","whiteout", "six_largest_cities"]

    for background_layer in background_layers_list:
        
        for lyr in arcpy.mapping.ListLayers(mxd):
            if lyr.supports("DATASOURCE"):
                
                if background_layer == str(lyr.name):
                    old_source = os.path.dirname(lyr.dataSource)
                    new_source = "C:\\Website_Test\\Report_Tests\\Background_Layers\\" + input_country
             
                    lyr.findAndReplaceWorkspacePath(old_source, new_source, False)

    
    #mxd.save()
    tempMXD = os.path.join(arcpy.env.scratchFolder, "mxd2Export.mxd")
    mxd.saveACopy(tempMXD)
    # re-load that MXD up as a true MXD object, not just the path
    # because the next function wants the object
    newMXD = arcpy.mapping.MapDocument(tempMXD)
    return newMXD

# You'll need to change whatever called "update_background_layers" to accept the 
# new MXD bein returned. As you have it now, you're kind of relying on a "global" MXD
# so this function would spit out the new MXD and your code would have to be updated to
# handle that.‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: GP Service Multiple Processes from MXD

LloydBronn — Thu, 09 Mar 2017 19:04:11 GMT

OK, so tempMXD would be the original Analysis_For_Reports.mxd, and newMXD would be a copy of that?

Re: GP Service Multiple Processes from MXD

KevinHibma — Thu, 09 Mar 2017 19:07:31 GMT

Not exactly. They're actually the same thing.

One is just a path on disk (tempMXD) and the other is an arcpy.mapping-MapDocument (newMXD).

You could pass the tempMXD out of the function and handle this line somewhere else if you want:

newMXD = arcpy.mapping.MapDocument(tempMXD)

Re: GP Service Multiple Processes from MXD

LloydBronn — Thu, 09 Mar 2017 19:11:43 GMT

OK, so when I call update_background_layers(newMXD,df,input_country) I get a server error:

arcpy.mapping.MapDocument("C:\\Website_Test\\Report_Tests\\MXDs\\Analysis_For_Reports.mxd") File "c:\program files\arcgis\server\arcpy\arcpy\arcobjects\mixins.py", line 611, in __init__ assert (os.path.isfile(mxd) or (mxd.lower() == "current")), gp.getIDMessage(89004, "Invalid MXD filename") AssertionError: Invalid MXD filename. Failed to execute (CreateReport). Failed to execute (CreateReport).

Re: GP Service Multiple Processes from MXD

KevinHibma — Thu, 09 Mar 2017 19:32:35 GMT

That'll be from MY line #22 above. It sounds like for some reason it couldnt find the new MXD it saved. If this is coming from the service you'll need to manually inspect those directories and make sure something got created in the scratch directory. You may also need to add some AddMessages to see whats going where. Sorry, the code above is untested, but fundamentally I'm sure of what needs to happen/flow.

Re: GP Service Multiple Processes from MXD

LloydBronn — Sat, 11 Dec 2021 14:02:03 GMT

All the directories, data and MXD are local on the Server C drive. I had problems with the paths to the server system folder before, so I made everything local. Here is the definition. I confirmed that the MXD is copying to the RAM drive temp folder, but it still errors out when we try to make two reports simultaneously. I'm a little mixed up, should I pass in newMXD instead of mxd in the function call?

def update_background_layers(mxd,df,input_country):
 
 background_layers_list = ["cities", "country", "oceanout","whiteout", "six_largest_cities"]
for background_layer in background_layers_list:
 
 for lyr in arcpy.mapping.ListLayers(mxd):
 if lyr.supports("DATASOURCE"):
 
 if background_layer == str(lyr.name):
 old_source = os.path.dirname(lyr.dataSource)
 new_source = "C:\\Website_Test\\Report_Tests\\Background_Layers\\" + input_country
 
 lyr.findAndReplaceWorkspacePath(old_source, new_source, False)

 #mxd.save()
 tempMXD = os.path.join("R:\\temp", "Analysis_For_Reports.mxd")
 mxd.saveACopy(tempMXD)
 # re-load that MXD up as a true MXD object, not just the path
 # because the next function wants the object
 newMXD = arcpy.mapping.MapDocument(tempMXD)
 return newMXD


update_background_layers(mxd,df,input_country)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍