topic Re: Geoprocessing Service Runs But Doesn't Produce Output in ArcGIS Enterprise Questions

Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Sun, 12 Dec 2021 04:47:01 GMT

So I've created a geoprocessing Python script tool that I've published as a GP service. This script takes lat/lon coordinates, extracts data from a raster from these coordinates, outputs a .csv and creates an Excel chart from a .vbs script. I can run the Python script on our server from the "C:\arcgisserver\directories\arcgissystem\arcgisinput" path and it works. I'm trying to test the GP service (Execute Task (GET)) from a browser on our REST services page. It says it's successful, but it doesn't actually output the chart. It does create the .csv, but it doesn't run the .vbs script. Here is the result message:

{
 "results": [],
 "messages": [
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Executing (ChartTest): ChartTest 45 -122 Portland"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Start Time: Wed Jan 25 13:51:23 2017"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Executing (ChartTest): ChartTest 45 -122 Portland"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Start Time: Wed Jan 25 13:51:23 2017"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Running script ChartTest..."
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Completed script ChartTest..."
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Succeeded at Wed Jan 25 13:51:23 2017 (Elapsed Time: 0.70 seconds)"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Succeeded at Wed Jan 25 13:51:23 2017 (Elapsed Time: 0.71 seconds)"
  }
 ]
}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I had a lot of trouble publishing this GP service because the Python script creates a .vbs, and these types of scripts can't be copied to the server. I got around this by changing the ".vbs" reference to ".txt" and publishing. I changed it back to ".vbs" on the server. I'm not sure if the server is blocking the .vbs or what. There are no errors in the server logs.

Re: Geoprocessing Service Runs But Doesn't Produce Output

RebeccaStrauch__GISP — Mon, 30 Jan 2017 17:40:14 GMT

Lloyd, I marked this as Assumed Answered since I did not see a question above. It looks like you have a workaround or were you looking for some more alternatives? IF so, you can unmark the Assumed, and add you question....or change it to a discussion instead of a question.

Re: Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Mon, 30 Jan 2017 18:47:09 GMT

Rebecca,

Sorry, the question is, why is the last part of my tool not running? The results say it is, but the .vbs script is not being created and the chart is not being generated. This script tool runs to completion in IDLE on the ArcGIS server machine, but not from the REST services page online. The messages say "succeeded" after I run execute task (GET), but it only gets to the step where the .csv is created. It seems like the server is blocking the .vbs from running, but I'm not sure. I've been talking to tech support about this, but I haven't gotten an answer yet. We're thinking it may be a permissions issue, ie the user online doesn't have permission to run a script on our local server machine.

Re: Geoprocessing Service Runs But Doesn't Produce Output

JonathanQuinn — Mon, 30 Jan 2017 18:59:54 GMT

The "Completed script ChartTest" is printed after the .vbs script is run? Are you running Idle as the same user running the ArcGIS Server Windows service? I'd suggest you take a look at the Event Viewer to see if there are any errors there. Also, you can run ProcMon and set a filter for .vbs script to see if it's actually called on. I don't think it's a permission issue, as the .vbs script resides in the arcgisinput folder, which the ArcGIS Server account would have full access to. The online user doesn't need permission to that folder; the user needs permission to the service and the user running the service needs permissions to whatever is being processed within the service so running Idle as the same user that runs the ArcGIS Server Windows service may be a good test. Make sure you're not running Idle as an admin when you run the script, as I doubt the GP service would run with elevated permission.

Re: Geoprocessing Service Runs But Doesn't Produce Output

RebeccaStrauch__GISP — Mon, 30 Jan 2017 20:44:55 GMT

Lloyd, I removed the assumed....so treat as a question again. btw, how you do that is to mark one as correct and then unmark it right away.

Re: Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Mon, 30 Jan 2017 21:07:04 GMT

The .vbs doesn't reside in the arcgis server folder, it's written on the fly by the Python script and run. I have been running the IDLE from an admin account on the ArcGIS server machine.

Re: Geoprocessing Service Runs But Doesn't Produce Output

JonathanQuinn — Mon, 30 Jan 2017 21:13:40 GMT

Ah, I see. Does the .vbs file get created in the scratch directory? How are you constructing the path to the output .vbs file within the script?

Re: Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Sun, 12 Dec 2021 04:47:03 GMT

When the Python script tool is run it follows these steps:

1. Extract values from raster based on input lat/lon coordinates

2. Write these values to .csv

3. Write .vbs to create chart of values in Excel

4. Run .vbs

Both the .csv and .vbs are written to the C:\temp drive of the server machine. These files will be overwritten everytime the tool is run. When I run the script in IDLE, everything works. When I enter the parameters into the execute task form on the GP service page, it gets to step 2, and writes the .csv to the temp drive. It doesn't get past step 2.

Both the .vbs and .csv are created dynamically in Python modules like so:

vbs_filepath = "C:\\temp\\Build_raster_Charts.vbs"

csv_filepath = "C:\\temp\\ + "chart_data_for_"\
+ location_label + ".csv"‍‍‍‍

Re: Geoprocessing Service Runs But Doesn't Produce Output

JonathanQuinn — Mon, 30 Jan 2017 21:31:16 GMT

Is that the syntax within the unpublished script, or the published one? What happens if you construct the path using the os.path.join function and the scratch folder environment variable:

vbs_filepath = os.path.join(arcpy.env.scratchFolder,"Build_raster_Charts.vbs")

csv_filepath = os.path.join(arcpy.env.scratchFolder,"{0}.csv".format(local_label)

By doing this, you use the scratch workspace of the Server when running the GP service, (the arcgisjobs directory), to maintain unique job folders for each run of the tool that are cleaned up automatically by the software. Typically, the published script will be updated to use this directory during the publishing process. Also, using os.path.join avoids potential path problems when publishing GP services.

Re: Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Mon, 30 Jan 2017 21:39:09 GMT

OK. So would I define the scratch folder like so?

arcpy.env.scratchFolder = "C:\arcgisserver\directories\arcgisjobs"‍

There was not a scratch folder defined when I published the service

Re: Geoprocessing Service Runs But Doesn't Produce Output

JonathanQuinn — Sun, 12 Dec 2021 04:47:06 GMT

The scratch folder is set automatically by the software in both ArcMap and ArcGIS Server. In ArcMap it'd be set to C:\Users\<your_username>\Documents\ArcGIS\scratch by default, and on the Server it'll be set to C:\arcgisserver\directories\arcgisjobs\<service_name>\<unique_jobID>\scratch. It will create this folder whenever the service is run and put any intermediate or output data there. You don't need to overwrite it within your script, just use arcpy.env.scratchFolder to call on it

Ex.

vbs_filepath = os.path.join(arcpy.env.scratchFolder,"Build_raster_Charts.vbs")‍

arcpy.env.scratchWorkspace is the <unique_jobID> folder and arcpy.env.scratchGDB is the <unique_jobID>\scratch.gdb file geodatabase.

This may not specifically address the problem you're running into, but it's a best practice and can help to make sure that things are written to the jobs directory correctly. What you can also do is add a few more arcpy.AddMessage print statements so you get a better indication of how far along the script has progressed, for example:

arcpy.AddMessage("Creating .vbs file.")
...
arcpy.AddMessage("Created .vbs file.")‍‍‍

Re: Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Sun, 12 Dec 2021 04:47:09 GMT

I set the workspace and added a print statement, but the execute task page just returned the same results as above. Still no chart or .vbs script created.

arcpy.env.scratchWorkspace = "C:\arcgisserver\directories\arcgisjobs\scratch"


try: 
   os.system(raster_vbs_script_filepath)
 except:
    arcpy.AddMessage("Failed to create .vbs file.")‍‍‍‍‍‍‍

Re: Geoprocessing Service Runs But Doesn't Produce Output

JonathanQuinn — Tue, 31 Jan 2017 01:29:24 GMT

The arcpy.env.scratchFolder environment variable is set for you when you run the GP service. There's no need to set it to anything within your script. Simply construct the path to your output data using the arcpy.env.scratchFolder environment variable:

vbs_filepath = os.path.join(arcpy.env.scratchFolder,"Build_raster_Charts.vbs")

Do you see your "Failed to create .vbs file" message? Is there anything in the server logs? Try to set the logs to DEBUG.

Re: Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Tue, 31 Jan 2017 14:39:17 GMT

We did some digging last night and discovered that the script is stopping before it even opens Excel, after the csv is created. So it wasn't even getting to the vbs script. It stops at this line in the Python script.

Excel = xlApp.Workbooks.Open(spreadsheet_template_filepath)‍

I tried setting all the Excel files and vbs script to output paths to os.path.join(arcpy.env.scratchFolder,"file.xlsm") It DID work in IDLE when these paths are joined to the scratchFolder.

I also changed the paths for the Excel spreadsheet and vbs script to our wwwroot folder that's accessible from our web server. And I moved the Excel and vbs files to "C:\arcgisserver\directories\arcgissystem\arcgisinput\Excel_Files" .This didn't work either.

The crazy thing is, there is nothing in the server logs. No errors, or any messages at all. And no matter what we do, the execute task page returns the same message that the script ran successfully.

I found this thread, and I'm wondering if we need to add the Windows Script Host to the application pool on the IIS server?

When I open the registry editor, there is no "enable" key under the Windows Script Host settings.

Re: Geoprocessing Service Runs But Doesn't Produce Output

JonathanQuinn — Tue, 31 Jan 2017 19:33:32 GMT

If the Excel file isn't generated, then we should focus on that first before looking into running the .vbs script. What version are you using? If you're creating an .xls file, what if you use arcpy instead? I'm not sure if you use your other module due to a formatting thing.

What is the spreadsheet_template_filepath variable set to? I imagine the GP service would fail if Server couldn't access the file. Can you add a message for the Excel variable and see what it returns, (arcpy.AddMessage(Excel). It may not be the actual data but at least you can verify if it's opening the template Excel file or not. I wouldn't move anything within wwwroot, because then you introduce latency going through your web server.

Re: Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Sun, 12 Dec 2021 04:47:12 GMT

We're using Excel 2016 on the server.

The spreadsheet template variable is:

vbs_filepath = os.path.join(arcpy.env.scratchFolder,"Build_" + custom_suffix + "Charts.vbs")‍‍‍‍

It writes successfully to the server scratch folder when I run it in IDLE.

Here is the module that creates the spreadsheet from the csv. It uses a static Excel template "CFS_Monthly_chart_template.xlsm", which I've placed in the scratch folder. I put in print statements to confirm the location of the scratch folder. It's: "C:\arcgisserver\directories\arcgissystem\arcgisinput\Chart_Test.GPServer\extracted\v101\website_test\scratch"

We created a log file output in the Python script to try to find out where the error is. It prints to line #22 here and then prints the error message at line #65, so we verified that it's not even opening Excel from the Server.

def build_spreadsheet_for_charts(location_data_dict,chart_folder,custom_suffix,daily_analysis_tablename,data_source):

    ##chart_test_path = "C:\\Chart_Test"
    chart_test_path = r_chart_path

    foldername = chart_folder

    ##r_spreadsheet_filepath = chart_test_path + "\\" + "CFS_Charts.xlsm"
    r_spreadsheet_filepath = os.path.join(arcpy.env.scratchFolder,"CFS_Charts.xlsm")
    ##r_spreadsheet_filepath = excel_files_path + "\\" + "CFS_Charts.xlsm"

    ##spreadsheet_template_filepath = chart_test_path + "\\" + "CFS_Monthly_chart_template.xlsm"
    ##spreadsheet_template_filepath = "C:\\Chart_Test\\CFS_Monthly_chart_template.xlsm"
    spreadsheet_template_filepath = os.path.join(arcpy.env.scratchFolder,"CFS_Monthly_chart_template.xlsm")   

    xlApp = None
    try:
        xlApp = win32com.client.Dispatch("Excel.Application")
        
        xlApp.Visible = False

        output_log_file.write("Getting ready to open excel spreadsheeet template \n")

        Excel = xlApp.Workbooks.Open(spreadsheet_template_filepath)

        output_log_file.write("Successfully opened excel spreadsheeet template \n")

        location_label = ""
        
        starting_row = 2
        row = starting_row
        ## index for location_data_dict starts at 1 not 0 (dictionary uses input index not standard index for a list)
        for location_index in range(1,len(location_data_dict)+1):
            location_data_tuple = location_data_dict[location_index]

            country, name, lat_str, lon_str, wmo_str, elev_m_str = location_data_tuple
            
            if location_index > 1:  ## location index = 1  --> row = 5; location index = 2 --> row = 6 (not 7)
                row = starting_row + (location_index - 1)
                    
            Excel.Worksheets("Available_Stations").Range("A" + str(row)).Formula = wmo_str
            Excel.Worksheets("Available_Stations").Range("B" + str(row)).Formula = lat_str
            Excel.Worksheets("Available_Stations").Range("C" + str(row)).Formula = lon_str
            Excel.Worksheets("Available_Stations").Range("D" + str(row)).Formula = name  
            Excel.Worksheets("Available_Stations").Range("E" + str(row)).Formula = country
            Excel.Worksheets("Available_Stations").Range("F" + str(row)).Formula = elev_m_str
            
        Excel.Worksheets("Available_Stations").Range("J2").Formula = foldername
        Excel.Worksheets("Available_Stations").Range("L2").Formula = data_source

        ## End of Location For Loop
        print r_spreadsheet_filepath
        if exists(r_spreadsheet_filepath):
            os.remove(r_spreadsheet_filepath)
        
        xlApp.ActiveWorkbook.SaveAs(r_spreadsheet_filepath)

        xlApp.ActiveWorkbook.Close
        xlApp.Quit()

        del xlApp
    except:
        output_log_file.write("Error updating CFS Charts spreadsheet from template\n")
        output_log_file.close()
        print "Error updating CFS Charts spreadsheet from template"
        del xlApp
        sys.exit()

    return()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Geoprocessing Service Runs But Doesn't Produce Output

JonathanQuinn — Sun, 12 Dec 2021 04:47:15 GMT

You should print the traceback error, as there's no way to know why it failed if it's in your try: except block. Can you update line 65 to:

print "Error updating CFS Charts spreadsheet from template\n{0}".format(traceback.format_exc())

You can parse out the traceback error however you like, but it's important to return it. In this case, it may be best to remove the try: except block so you can return all of the messages within the GP service framework. Be sure to import the traceback module at the beginning of your script.

Re: Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Wed, 01 Feb 2017 15:56:05 GMT

I added this line, but I didn't get an error message. Yesterday I added our ArcGIS server user login to the DCOM registry for Excel, and to the Desktop folder permissions under the Windows system folders (32 and 64 bit). Now when I click execute task on the REST page, a log file is generated on the server. It claims that that whole process is completing. This is new. However, nothing is generated in the scratch folder on the server. It still runs perfectly in IDLE.

Here is the log file output. It never got past line 3 here before. This log file is identical, whether I run the script from IDLE or from the REST page.

1. starting to create csv data file
2. starting to build the spreadsheet
3. Getting ready to open excel spreadsheeet template
4. Successfully opened excel spreadsheeet template
5. starting to build vbs script
6. starting to execute vbs script
7. return value: 0 commandline_output:

I'm wondering if I have my scratch workspace set correctly? It's

arcpy.env.scratchWorkspace = "C:\arcgisserver\directories\arcgisjobs"

The scratch folder is under the arcgissystem\arcgisinputs folder.

I'm also wondering about the data store on the server. I shouldn't have to register the scratch folder should I?

Re: Geoprocessing Service Runs But Doesn't Produce Output

JonathanQuinn — Wed, 01 Feb 2017 16:57:35 GMT

Ahh, sorry, Lloyd, I meant to update it to arcpy.AddMessage(<message>). You'll never see print statements when running a GP tool or GP service. If you still have your try: except block, remote that and let errors come back directly through the GP service.

You don't need to set any scratch workspaces, folders or geodatabases. The default path for them is correct. For ease of use, simply use the default of arcpy.env.scratchFolder to construct the path to output data that exists in folders.

The scratch folder doesn't need to be registered on the Server, as it knows it can write to the directory already, (providing it's left at the default location). I wouldn't think that the raster data location is a problem, as it would fail earlier in the script.

Re: Geoprocessing Service Runs But Doesn't Produce Output

LloydBronn — Sun, 12 Dec 2021 04:47:18 GMT

OK, I moved all of my data back to the server folders they were originally extracted to when I published the GP service. I had commented out all the g_ESRI_variables and put everything in local folders to test it in IDLE. I reinstated these paths.

When I try to run the script in IDLE now, I get this, "Error finding C:\Users\ADMINI~1.MET\AppData\Local\Temp\2"

I added this at the start of the script to see what was going on:

scratch_folder = arcpy.env.scratchFolder

print(scratch_folder)

sys.exit()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

If I don't define the scratch workspace, it prints this:

C:\Users\ADMINI~1.MET\AppData\Local\Temp\2\scratch

If I set the workspace to arcpy.env.scratchWorkspace = "C:\arcgisserver\directories\arcgisjobs", it prints this:

C:\arcgisserver\directories\arcgissystem\arcgisinput\Chart_Test.GPServer\extracted\v101\website_test\scratch

When I first published the tool, it was set to g_ESRI_variable_1 = os.path.join(arcpy.env.packageWorkspace,u'temp'). When I set it back to this, it also prints:

C:\Users\ADMINI~1.MET\AppData\Local\Temp\2\scratch

So I'm confused. There seem to be two scratch folders,

Scratch Workspace: C: rcgisserver\directories rcgisjobs\scratch
Scratch Folder: C:\arcgisserver\directories\arcgissystem\arcgisinput\Chart_Test.GPServer\extracted\v101\website_test\scratch
‍‍