Using 3. party python in gp-service - error 000816

[LemvigKommune]

Hi
I'm trying to create a GP-service that uses the python openpyxl 3. party module.
When i'm running it from arcmap(Standard), it works like a charm, publich the result as gp-service on the server.
When i then sets up the flexviewer 3.0 using the gp-widget - i get an error 000816 tool is not valid, and nothing else.

import osimport arcpy import uuid import openpyxl   class Toolbox(object):     def __init__(self):         """Define the toolbox (the name of the toolbox is the name of the         .pyt file)."""         self.label = "Toolbox"         self.alias = ""           # List of tool classes associated with this toolbox         self.tools = [Tool]     class Tool(object):     def __init__(self):         """Define the tool (tool name is the name of the class)."""         self.label = "Tool"         self.description = ""         self.canRunInBackground = False       def getParameterInfo(self):         """Define parameter definitions"""           param0 = arcpy.Parameter(         displayName="FileName",         name="filename",         datatype="String",         parameterType="Required",         direction="Input")     param0.value = 'OpenPyXl'     params = [param0]         return params       def isLicensed(self):         """Set whether tool is licensed to execute."""         return True       def updateParameters(self, parameters):         """Modify the values and properties of parameters before internal         validation is performed.  This method is called whenever a parameter         has been changed."""         return       def updateMessages(self, parameters):         """Modify the messages created by internal validation for each tool         parameter.  This method is called after internal validation."""         return       def execute(self, parameters, messages):         """The source code of the tool."""     wb = openpyxl.Workbook(optimized_write=True)         ws = wb.create_sheet()      wb_file = r'C:\arcgisserver\directories\arcgisoutput\_ags_' + str(uuid.uuid4()) + '_' + parameters[0].value + '.xlsx' ##    arcpy.AddMessage(wb_file)         wb.save(wb_file)  

this is the python toolbox i'm trying to publish, it's quite simple, the  only thing it does is making a xlsx file in the   c:\arcgisserver\directories\arcgisoutput folder with the name  _ags_UUID_OpenPyXl.xlsx

The output folder is set up in the Datastore on the server.
I've tried to copy the openpyxl folder from the python folder to the same folder as the python toolbox, no luck at that.
I've tried to add a sys.path.append and the path to the openpyxl folder, no luck at that.

somebody please help me, my larger tool wont work before i've solved this problem.

[KevinHibma]

I see you cross posted this in another forum and got some suggestions. The main one I'd confirm is having the 64bit Python module installed on your server machine.
With that installed, can you start 64bit Python on that machine and access your tool that way? The answer to this will show if its a 3rd party module/tool issue, or an issue specific to the service.

C:\Python27\ArcGISx6410.1\python.exe
import arcpy
arcpy.ImportToolbox(r"<path to .pyt">
arcpy.<toolname>(params)

There might be some information in this help topic of use to you:
http://resources.arcgis.com/en/help/main/10.1/#/Authoring_geoprocessing_tasks_with_Python_scripts/00...

[LemvigKommune]

Hi, thanks, but i'm sure that i'm running 32 bit python, and the reason is that the first time i installed the server i installed the 64 bit python.
And i sure that i tried to make a gp-process on a 32 bit computer with arcmap basic installed, and then my 64 bit server would not run it, thats why i started over and installed 32 bit python.
But i'm willing to try to install the 64 bit version of python again, to see if this will work.
First thing monday morning i'm trying to reinstall the 64 bit python.
I'm crossing my fingers that my other gp-process still will run without problems.
But then I can't use the installed arcmap basic on the server to publish some of my scripts, because that will only run 32 bit python. will it not ???
I'll wait with the reinstall until i've gotten a response on this.
Thanks
Mads

[KevinHibma]

No - they're separate.
< 10.1 ONLY information >
ArcMap is 32bit and uses 32bit Python
ArcGIS Server is 64bit and uses 64bit Python

If both products are installed on a single machine, you have 32bit and 64bit Python. (if you let the installer do it). If you're managing your OWN python installs, then you need to install both the appropriate Python (32 or 64) as well as Numpy and Matplotlib (again, 32 or 64 or BOTH depending on what Arc* products you have installed).

Before publishing a GP Service, ArcMap uses the 32bit Python to execute your tool.
Consuming a GP Service, Server is using the 64bit Python to execute the tool.
This is why you need both the 32 and 64bit versions of the 3rd party module installed when everything is on one machine.

If you have both products installed and are using Python from command line (you open prompt and simply typed "python.exe"), then you could be using either 32 or 64bit. You need to be aware of which one you're executing against. Typically (bot not always) you're executing against the last Python version you installed. The best way to be sure is to launch the Python you want by explicitly referencing it:
C:\Python27\ArcGIS10.1\python.exe
or
C:\Python27\ArcGISx6410.1\python.exe

I hope this clears up the differing "bits"

[LemvigKommune]

Hi Kevin
Ok, I've been trying to get this to work most of sunday and monday as well.
Got the Python up and running in 64 bit (killed my other scripts), but I'll look into that later.
If I'm using the openpyxl module, all I get in ArcMap is a red cross over the toolbox.
I then tried to make a new toolbox like this.

import os
import arcpy
import sys
import xlwt

class Toolbox(object):
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = ""

        # List of tool classes associated with this toolbox
        self.tools = [Tool]


class Tool(object):
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Tool"
        self.description = ""
        self.canRunInBackground = False

    def getParameterInfo(self):
        """Define parameter definitions"""
        params = None
        return params

    def isLicensed(self):
        """Set whether tool is licensed to execute."""
        return True

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return

    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter.  This method is called after internal validation."""
        return

    def execute(self, parameters, messages):
        """The source code of the tool."""
        arcpy.AddMessage('Version : %s' % sys.version)
        arcpy.AddMessage('Path to module : %s' % os.path.dirname(xlwt.__file__))
        arcpy.AddMessage('End')

print 'Version : %s' % sys.version
print 'Path to module : %s' % os.path.dirname(xlwt.__file__)
print 'End'

Running this in the 64 bit idle I get this:
Version : 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)]
Path to module : c:\Python27\ArcGISx6410.1\lib\site-packages\xlwt-0.7-4-py2.7.egg\xlwt
End

When i run it from ArcMap I get this:
Version : 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)]
Path to module : C:\Python27\ArcGIS10.1\lib\site-packages\xlwt-0.7.4-py2.7.egg\xlwt
End

But when I then share the result as a Service, it never finishes, it keeps on running at 175kb.

This is my Pythonpath in the System Properties : C:\Python27\ArcGISx6410.1;C:\Python27\ArcGISx6410.1\DLLs;C:\Python27\ArcGISx6410.1\lib;C:\Python27\ArcGISx6410.1\lib\lib-tk;C:\Python27\ArcGISx6410.1\Scripts

The PythonDir in HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ESRI\Python10.1 is C:\Python27\
The installpath in HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\2.7\InstallPath is C:\Python27\ArcGIS10.1\

Is that correct ?

[KevinHibma]

Based on all your information, at this point it sounds like you have everything setup correctly for Python.
All of the returns you've printed make sense based on where you've called them from.

So at this point, you can run your toolbox locally, and publish it. You just cant consume it as a GP Service?
What do you mean by:

But when I then share the result as a Service, it never finishes, it keeps on running at 175kb.

If you navigate to:
C:\arcgisserver\directories\arcgissystem\arcgisinput\<your service>.GPServer\extracted\v101\<folder original .pyt lived in>\
You'll find your toolbox that ArcGIS Server uses.
Can you consume this in ArcMap?

[LemvigKommune]

Hi Kevin
This starts to give me a headache... so I uninstalled all ArcGIS and python, and removed all related to ArcGIS and Python in the registry.
Installed only the Server this time, so I'm only running 64bit on the server now, so far so good.

When I look in the registry there is only reference to python in these places :
HKEY_LOCAL_MACHINE\SOFTWARE\ESRI\Python10.1 = C:\Python27\
HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7\InstallPath = C:\Python27\ArcGISx6410.1\

The python (sys.version) is : 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)]
I installed the openpyxl module and the path to that on the server (os.path.dirname(openpyxl.__file__) is : C:\Python27\ArcGISx6410.1\lib\site-packages\openpyxl-1.5.8-py2.7.egg\openpyxl

Now I've made a python toolbox like this :

import arcpyimport os, sys
import datetime, time
import uuid
from arcpy import env
import openpyxl


class Toolbox(object):
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = ""


        # List of tool classes associated with this toolbox
        self.tools = [Tool]




class Tool(object):
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Tool"
        self.description = ""
        self.canRunInBackground = False


    def getParameterInfo(self):
        """Define parameter definitions"""
        selectOwners = arcpy.Parameter(
        displayName = 'This is test 1',
        name = 'owner_selected',
        datatype = 'Boolean',
        parameterType = 'Optional',
        direction = 'Input')


        selectResidents = arcpy.Parameter(
        displayName = 'This is test 2',
        name = 'residents_selected',
        datatype = 'Boolean',
        parameterType = 'Optional',
        direction = 'Input')


        params = [selectOwners, selectResidents]
        return params


    def isLicensed(self):
        """Set whether tool is licensed to execute."""
        return True


    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return


    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter.  This method is called after internal validation."""
        return


    def execute(self, parameters, messages):
        """The source code of the tool."""
        scriptPath = sys.path[0]
        outfolder = os.path.join(scriptPath, 'Output')
        arcpy.env.scratchWorkspace = scriptPath
        wb = openpyxl.Workbook()
        filename = '_ags_' + str(uuid.uuid4()) + '.xlsx'
        arcpy.AddMessage('Filename : %s' % os.path.join(outfolder, filename))
        wb.save(os.path.join(outfolder, filename))

Running this from the ArcMap on a remote computer gives me a file in the Output folder where the toolbox is located.
It's running and doing what it should do.
I'm then trying to publish the result as a Geoprocess on my reinstalled server, this is where it breaks down.
I'm setting up the parameters, the tools and the Item Description, running Analyze = nothing...
pressing Publish and it's running and running and running and running, I think you get it... It's been running for over 30 minutes and nothing.
[ATTACH=CONFIG]18305[/ATTACH]
As of now, I have to install the 32 bit python and the ArcMap again, so that my other gp-services will run again.
Do you have any ideas ?
I'm willing to show it live to you or one of your colleagues on the ESRI EU UC in Oslo this monday or tuesday, is that possible ??
Thanks

[KevinHibma · Accepted solution]

Ok.... I couldnt find a windows installer for this package...I did find this though:
http://www.lfd.uci.edu/~gohlke/pythonlibs/
I installed both of these (32 and 64bit versions.... the installer finds my the proper python and puts them in there)
Base-12.9.15.win-amd64-py2.7.*exe
Base-12.9.15.win32-py2.7.*exe

I then took your tool you provided above, created a PYT and ran it on Desktop.
(I had to make an "output" folder in the same directory as the PYT)
No problems.
I then published it to Server, no problems.
I then executed the service, no problems.

My best guess of what you're doing - you've installed the package differently? It doesnt look like theres a real installer for it. I'm not sure I can offer any advice on getting this particular package installed short of the way I did it.

Why is it taking 30+ minutes to publish? My guess - you have a LOT of files inside that output folder. Gigabytes and gigabytes. As per the publishing rules, if you reference a folder, we take the contents of that folder because we assume it contains required inputs. The folder I made was brand new and had nothing in it.
Could you clean the folder or point to an empty one and try again?

Also - in your tool it looks like you're trying to handle unique output names for your excel file. Any particular reason why? If not, I'd suggest letting server do it for you by creating the file in the scratch directories. The way you have it right now, you're setting the scratchWorkspace in the script. You can't do this in a GP Service. The framework handles the scratchworkspace, thus you should just inherit from it. Your tool on Desktop can work the same way - let ArcMap set the scracthWorkspace and have the tool inherit from it.

import arcpy import os, sys import datetime, time import openpyxl   class Toolbox(object):     def __init__(self):         """Define the toolbox (the name of the toolbox is the name of the         .pyt file)."""         self.label = "Toolbox"         self.alias = ""           # List of tool classes associated with this toolbox         self.tools = [Tool]     class Tool(object):     def __init__(self):         """Define the tool (tool name is the name of the class)."""         self.label = "Tool"         self.description = ""         self.canRunInBackground = False       def getParameterInfo(self):         """Define parameter definitions"""         selectOwners = arcpy.Parameter(         displayName = 'This is test 1',         name = 'owner_selected',         datatype = 'Boolean',         parameterType = 'Optional',         direction = 'Input')           selectResidents = arcpy.Parameter(         displayName = 'This is test 2',         name = 'residents_selected',         datatype = 'Boolean',         parameterType = 'Optional',         direction = 'Input')           params = [selectOwners, selectResidents]         return params       def isLicensed(self):         """Set whether tool is licensed to execute."""         return True       def updateParameters(self, parameters):         """Modify the values and properties of parameters before internal         validation is performed.  This method is called whenever a parameter         has been changed."""         return       def updateMessages(self, parameters):         """Modify the messages created by internal validation for each tool         parameter.  This method is called after internal validation."""         return       def execute(self, parameters, messages):         """The source code of the tool."""                 wb = openpyxl.Workbook()         filename = os.path.join(arcpy.env.scratchFolder, "myExcel.xlsx")         arcpy.AddMessage('Filename : {}'.format(filename))         wb.save(filename)

[LemvigKommune]

Hi Kevin
Well, back from EUC in Oslo and a well-deserved vacation...
I've got the IT-dep. to make me a new server.
I've then installed the desktop first, then the server, then the base-files with all the python modules you told about in another message, created a script like this:

import os
import arcpy
import openpyxl

class Toolbox(object):
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = ""

        # List of tool classes associated with this toolbox
        self.tools = [Tool]


class Tool(object):
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Tool"
        self.description = ""
        self.canRunInBackground = False

    def getParameterInfo(self):
        """Define parameter definitions"""
        param0 = arcpy.Parameter(
        displayName = 'Filename',
        name = 'filename',
        datatype = 'String',
        parameterType = 'Required',
        direction = 'Input')
        param0.value = 'TestFile'
        params = [param0]
        return params

    def isLicensed(self):
        """Set whether tool is licensed to execute."""
        return True

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return

    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter.  This method is called after internal validation."""
        return

    def execute(self, parameters, messages):
        """The source code of the tool."""
        fn = parameters[0].value
        wb = openpyxl.Workbook()
        filename = 'C:/arcgisserver/directories/arcgisoutput/' + fn + '.xlsx'
        arcpy.AddMessage('Filename : %s' % filename)
        wb.save(filename)
        return

And it works like a charm... All I have to do now i set up my server and move all my services from one server to another, and make my scripts run.

I hope for the best.

Thanks for all you help.

[KevinHibma]

Excellent, I'm glad its working for you and the Esri EUC sounds like it went well!