Community
Select to view content in your preferred language

Script works in ArcMap, won't publish to ArcServer

793
3
02-07-2013 04:42 AM
DavidChevrier
by
Frequent Contributor
‎02-07-2013 04:42 AM
Hi,  I wrote a python script, wrapped it in ModelBuilder and tried to publish it.  It just gets stuck in a loop, forcing me to cancel the publish.  It passed the analysis and all the data used in it is in verified folders in the ArcServer data sources.  I can't figure out what's going on.  I've tried publishing it from ArcMap on both my desktop computer and on the server itself, but get the same error.  The error is 001270 Consolidating the data failed.  I followed all steps listed in the help for that error and it didn't work.  I'm out of ideas, can someone look at this code and figure out why it won't publish?  My only guess is it has to do with some sort of temporary data, possible the %scratchFolder%, although unlikely.  My other thought is one of the import scripts is giving some errors. 

The code takes a feature class, email information, and the %scratchFolder% variable.  It creates a NobelTec file (which is just a simple text file in the open navigation format) and a csv of the feature class's table, then zips the two new files and the feature class together, and emails the end user.  Pretty simple and works fine from ArcMap.  Most of it is code from the ArcToolBox that I took and hard-coded some things and deleted unnecessary parts.   I exported the ModelBuilder code to python and will show that first, then show the python script tool itself.  (Note that I replaced email addresses, folder locations, and server info with dummy info, but have them written out on my copy.)  Any help would be GREATLY appreciated!

Due to text length issues, the python code for the script will be posted in the first reply.

Exported ModelBuilder code:
import arcpy
arcpy.ImportToolbox("//XXX.XXX.XXX.XXX/arcData/lab_data/FOLDER1/FOLDER2/MyToolBox10_1.tbx")

jobID = arcpy.GetParameterAsText(0)
if jobID == '#' or not jobID:
    jobID = "\\\\XXX.XXX.XXX.XXX\\arcData\\lab_data\\FOLDER1\\FOLDER2\\ToolData\\SaveAndEmail.gdb\\Stations"

Output_File_Name = arcpy.GetParameterAsText(1)
if Output_File_Name == '#' or not Output_File_Name:
    Output_File_Name = "BTS_2013_Spring"

Email_To = arcpy.GetParameterAsText(2)
if Email_To == '#' or not Email_To:
    Email_To = "EndUserEmail@WhereEver.com"

Email_Subject = arcpy.GetParameterAsText(3)
if Email_Subject == '#' or not Email_Subject:
    Email_Subject = "BTS_2013_Spring_Stations"

Email_Body = arcpy.GetParameterAsText(4)
if Email_Body == '#' or not Email_Body:
    Email_Body = "Here are your stations"

Worked = arcpy.GetParameterAsText(5)

Input_Stations = "%jobID%"
Scratch_Folder = "%scratchFolder%"

arcpy.gp.toolbox = "//XXX.XXX.XXX.XXX/arcData/lab_data/FOLDER1/FOLDER2/MyToolBox10_1.tbx";
arcpy.gp.CreateFilesZipAndEmail(Input_Stations, Scratch_Folder, Output_File_Name, Email_To, Email_Subject, Email_Body)
Tags (2)
0 Kudos
3 Replies
DavidChevrier
by
Frequent Contributor
‎02-07-2013 04:43 AM
The python script:
import arcpy
import ErrorUtils
import locale
import os
import smtplib
import SSDataObject
import SSUtilities
import string
import zipfile
locale.setlocale(locale.LC_ALL, '')

arcpy.SetLogHistory(False)

from email.MIMEMultipart import MIMEMultipart
from email.MIMEBase import MIMEBase
from email.MIMEText import MIMEText
from email.Utils import COMMASPACE, formatdate
from email import Encoders

inputSratchFolder = ""
 
 
def createNobelTecFile(inputNTfc, inputNTfileName):
    try:
        outputNTname = os.path.join(inputSratchFolder, (inputNTfileName + "_Nav.txt"))
  
        f = open(outputNTname,'w')
 
        rows = arcpy.SearchCursor(inputNTfc)
        for row in rows:
          label = row.getValue("LABEL")
          f.write('[' + label + ']\n')
          f.write('Type = Mark\n')

          lon = row.getValue("LON")
          lonDD = int(abs(lon))
          lonMM = float((float(abs(lon)) - lonDD) * 60.0)
          lonMMs = str(lonMM)
          lat = row.getValue("LAT")
          latDD = int(lat)
          latMM = float((float(lat) - latDD) * 60.0)
          latMMs = str(latMM)
   
          f.write('LatLon = ' + str(latDD) + ' ' + latMMs + ' N ' + str(lonDD) + ' ' + lonMMs + ' W\n')
          f.write('Color = 0x000000\n')

          testType = row.getValue("TYPE")
          if testType == "CTD Only":
            symbol = str(1)
          elif testType == "Bongo and CTD":
            symbol = str(9)
          elif testType == "Random Clam Station":
            symbol = str(1)
          elif testType == "Random BTS Station":
            symbol = str(1)
          elif testType == "Random BTS Station with Bongo":
            symbol = str(9)
          elif testType == "Random BTS Alt Station":
            symbol = str(1)
          elif testType == "Scallop - Fixed Station: Start":
            symbol = str(2)
          elif testType == "Scallop - Fixed Station: End":
            symbol = str(3)
          elif testType == "Scallop - Non-Random":
            symbol = str(4)
          elif testType == "Scallop - Non-Random > 40m":
            symbol = str(5)
          elif testType == "Random Scallop Station":
            symbol = str(1)
          elif testType == "Fixed Scallop Station":
            symbol = str(9)
          elif testType == "Random Shrimp Station":
            symbol = str(1)
          elif testType == "Fixed Shrimp Station":
            symbol = str(9)
          
          f.write('BmpIdx = ' + symbol + '\n')
          f.write('ExistsOutsideCollection = True\n')
          f.write('Locked = True\n')
          f.write('MarkRenamed = True\n')
        
        f.close()
        return outputNTname
    except:
        arcpy.AddMessage("Create NobelTec File Tool Failed")
        arcpy.AddError(arcpy.GetMessages(2))

  
  
  
def createChiefSciFile(inputCSfc, inputCSfileName):
    try:
        outputCSname = os.path.join(inputSratchFolder, (inputCSfileName + ".csv"))
        fieldList = [f.name for f in arcpy.ListFields(inputCSfc) if not f.name.startswith('Shape')]
        exportXYV(inputCSfc, fieldList, outputCSname)
        return outputCSname
    except:
        arcpy.AddMessage("Create Chief Scientist File Tool Failed")
        arcpy.AddError(arcpy.GetMessages(2))  


  

def exportXYV(inputFC, fieldList, outFile):
    ssdo = SSDataObject.SSDataObject(inputFC)
    inputFields = [ssdo.oidName, "SHAPE@XY"] + fieldList 
    cnt = SSUtilities.getCount(inputFC)

    badIDs = []
    badRecord = 0
    try:
        rows = arcpy.da.SearchCursor(ssdo.inputFC, inputFields)
    except:
        arcpy.AddIDMessage("ERROR", 204)
        raise SystemExit()
 
    delimiter = ","
    floatTypes = ["Single", "Double"]
    localeDict = {}
    for field in fieldList:
        fieldType = ssdo.allFields[field].type
        if fieldType in floatTypes:
            formatToken = "%f"
        else:
            formatToken = "%s"
        localeDict[field] = formatToken

    fo = SSUtilities.openFile(outFile, 'w')

    outRow = delimiter.join(fieldList)
    fo.write("%s\n" % outRow.encode('utf-8'))
    for row in rows:
        OID = row[0]
        badValues = row.count(None)
        badRow = badValues
        rowValues = []
        for ind, field in enumerate(fieldList):
            value = row[ind + 2]
            if value == "" or value == None:
                rowValues.append("NULL")
            else:
                formatValue = locale.format(localeDict[field], value)
                rowValues.append(formatValue)

        if badRow:
            badIDs.append(OID)
        
        outRow = delimiter.join(rowValues)
        fo.write("%s\n" % outRow.encode('utf-8'))
        
    del rows
    fo.close()

    badIDs = list(set(badIDs))
    badIDs.sort()
    badIDs = [ str(i) for i in badIDs ]
    
    bn = len(badIDs)
    if bn:
        err = ErrorUtils.reportBadRecords(cnt, bn, badIDs, label=ssdo.oidName, allowNULLs = True)
  
  

  
def zipUpFolder(inputFC, chiefSciFile, nobelTecFile):
    try: 
        outZipFile = os.path.join(inputSratchFolder, "Stations.zip")
        zip = zipfile.ZipFile(outZipFile, 'w', zipfile.ZIP_DEFLATED)
        zipws(zip, inputFC, chiefSciFile, nobelTecFile)
        zip.close()
        return outZipFile
    except RuntimeError:
        if os.path.exists(outZipFile):
            os.unlink(outZipFile)
        zip = zipfile.ZipFile(outZipFile, 'w', zipfile.ZIP_STORED)
        zipws(zip, inputFC, chiefSciFile, nobelTecFile)
        zip.close()
        arcpy.AddWarning(arcpy.GetIDMessage(86133))


  

def zipws(zip, fc, chiefSciZip, nobelTecZip):
    path = os.path.normpath(os.path.dirname(fc))
    chiefSciName = os.path.basename(chiefSciZip)
    nobelTecName = os.path.basename(nobelTecZip)
    
    rootLen = len(path)+len(os.sep)
    for (dirpath, dirnames, filenames) in os.walk(path):
        for file in filenames:
            if not file.endswith('.lock'):
                zip.write(os.path.join(dirpath, file), os.path.join("Stations.gdb", os.path.join(dirpath, file)[rootLen:]))
   
    zip.write(chiefSciZip, chiefSciName)
    zip.write(nobelTecZip, nobelTecName)

 
 
 
def send_mail(send_to, subject, text, f):
    try:
        msg = MIMEMultipart()
        msg['From'] = r'FirstName.LastName@MyEmail.com'
        msg['To'] = COMMASPACE.join(send_to)
        msg['CC'] = "FirstName.LastName@MyEmail.com"
        msg['Date'] = formatdate(localtime=True)
        msg['Subject'] = subject
        
        msg.attach( MIMEText(text) )
        part = MIMEBase('application', "zip")
        part.set_payload( open(f,"rb").read() )
        Encoders.encode_base64(part)
        part.add_header('Content-Disposition', 'attachment; filename="%s"' % os.path.basename(f))
        msg.attach(part)
            
        smtp = smtplib.SMTP("MYSERVER")
        smtp.sendmail(r'FirstName.LastName@MyEmail.com', send_to, msg.as_string())
        smtp.close()
    except:
        arcpy.AddMessage("Send Email Tool Failed")
        arcpy.AddError(arcpy.GetMessages(2))
  


  
if __name__ == '__main__':
    try:
        inputGDB = arcpy.GetParameterAsText(0)
        global inputSratchFolder
        inputSratchFolder = arcpy.GetParameterAsText(1)
        inputFileName = arcpy.GetParameterAsText(2)
        sendto = arcpy.GetParameterAsText(3).split(";")
        subject = arcpy.GetParameterAsText(4)
        text = arcpy.GetParameterAsText(5)
  
        outputNobelTecFile = createNobelTecFile(inputGDB, inputFileName)
  
        outputChiefSciFile = createChiefSciFile(inputGDB, inputFileName)
  
        outputZipFile = zipUpFolder(inputGDB, outputChiefSciFile, outputNobelTecFile)
   
        send_mail(sendto, subject, text, outputZipFile)
        arcpy.SetParameterAsText(5, "True")

    except:
        arcpy.SetParameterAsText(5, "False")
        arcpy.AddMessage("Create Files Zip and Email Tool Failed")
        arcpy.AddError(arcpy.GetMessages(2))
0 Kudos
DavidChevrier
by
Frequent Contributor
‎02-07-2013 09:07 AM
I've narrowed down the problem.  The creation of the txt file and the zipping and emailing of the feature class along with it work fine.  The part that is giving me the error is with the export to csv function.  In fact, by just adding these two lines, it will fail:

import SSDataObject
import SSUtilities

I don't have the solution yet, but may try to code without using those scripts...
0 Kudos
DavidChevrier
by
Frequent Contributor
‎02-08-2013 05:55 AM
I was correct and there were errors with esri's SSUtilities and/or SSDataObject scripts.  By removing them and adding the functionality they had into my script, it not only loaded successfully, but the analyze part also took seconds rather than 10 minutes.  I guess when the analyzer drills down into the scripts it takes awhile to run the tests.  Anyway, here is the new code in case people are curious.  Note that I also fixed the CC part of the email as that wasn't working correctly, although unrelated to esri.

import arcpy
import locale
import os
import smtplib
import string
import zipfile
locale.setlocale(locale.LC_ALL, '')
arcpy.SetLogHistory(False)

from email.MIMEMultipart import MIMEMultipart
from email.MIMEBase import MIMEBase
from email.MIMEText import MIMEText
from email.Utils import COMMASPACE, formatdate
from email import Encoders

inputSratchFolder = ""




def createNobelTecFile(inputNTfc, inputNTfileName):
    try:
        outputNTname = os.path.join(inputSratchFolder, (inputNTfileName + "_Nav.txt"))
  
        f = open(outputNTname,'w')
 
        rows = arcpy.SearchCursor(inputNTfc)
        for row in rows:
            label = row.getValue("LABEL")
            f.write('[' + label + ']\n')
            f.write('Type = Mark\n')

            lon = row.getValue("LON")
            lonDD = int(abs(lon))
            lonMM = float((float(abs(lon)) - lonDD) * 60.0)
            lonMMs = str(lonMM)
            lat = row.getValue("LAT")
            latDD = int(lat)
            latMM = float((float(lat) - latDD) * 60.0)
            latMMs = str(latMM)
   
            f.write('LatLon = ' + str(latDD) + ' ' + latMMs + ' N ' + str(lonDD) + ' ' + lonMMs + ' W\n')
            f.write('Color = 0x000000\n')

            testType = row.getValue("TYPE")
            if testType == "CTD Only":
              symbol = str(1)
            elif testType == "Bongo and CTD":
              symbol = str(9)
            elif testType == "Random Clam Station":
              symbol = str(1)
            elif testType == "Random BTS Station":
              symbol = str(1)
            elif testType == "Random BTS Station with Bongo":
              symbol = str(9)
            elif testType == "Random BTS Alt Station":
              symbol = str(1)
            elif testType == "Scallop - Fixed Station: Start":
              symbol = str(2)
            elif testType == "Scallop - Fixed Station: End":
              symbol = str(3)
            elif testType == "Scallop - Non-Random":
              symbol = str(4)
            elif testType == "Scallop - Non-Random > 40m":
              symbol = str(5)
            elif testType == "Random Scallop Station":
              symbol = str(1)
            elif testType == "Fixed Scallop Station":
              symbol = str(9)
            elif testType == "Random Shrimp Station":
              symbol = str(1)
            elif testType == "Fixed Shrimp Station":
              symbol = str(9)
          
            f.write('BmpIdx = ' + symbol + '\n')
            f.write('ExistsOutsideCollection = True\n')
            f.write('Locked = True\n')
            f.write('MarkRenamed = True\n')
        
        f.close()
        return outputNTname
    except:
        arcpy.SetParameterAsText(6, "False")
        arcpy.AddMessage("Create NobelTec File Tool Failed")
        arcpy.AddError(arcpy.GetMessages(2))


  
  
def createChiefSciFile(inputCSfc, inputCSfileName):
    try:
        outputCSname = os.path.join(inputSratchFolder, (inputCSfileName + ".csv"))
        fieldList = [f.name for f in arcpy.ListFields(inputCSfc) if not f.name.startswith('Shape')]
        exportXYV(inputCSfc, fieldList, outputCSname)
        return outputCSname
    except:
        arcpy.SetParameterAsText(6, "False")
        arcpy.AddMessage("Create Chief Scientist File Tool Failed")
        arcpy.AddError(arcpy.GetMessages(2))


  

def exportXYV(inputFC, fieldList, outFile):
    try:
        rows = arcpy.da.SearchCursor(inputFC, fieldList)
        delimiter = ","
        floatTypes = ["Single", "Double"]
        localeDict = {}
        fieldListObject = arcpy.ListFields(inputFC)   
        for field in fieldListObject:
            fieldType = field.type
            if fieldType in floatTypes:
                formatToken = "%f"
            else:
                formatToken = "%s"
            localeDict[field.name] = formatToken
  
        try:
            fo = open(outFile, 'w')
        except:
            arcpy.SetParameterAsText(6, "False")
            arcpy.AddIDMessage("ERROR", 210, fileName)
            raise SystemExit()
        
        outRow = delimiter.join(fieldList)
        fo.write("%s\n" % outRow.encode('utf-8'))
        for row in rows:
            rowValues = []
            for ind, field in enumerate(fieldList):
                value = row[ind]
                if value == "" or value == None:
                    rowValues.append("NULL")
                else:
                    formatValue = locale.format(localeDict[field], value)
                    rowValues.append(formatValue)

            outRow = delimiter.join(rowValues)
            fo.write("%s\n" % outRow.encode('utf-8'))
        
        del rows
        fo.close()

    except:
        arcpy.SetParameterAsText(6, "False")
        arcpy.AddMessage("Create Chief Scientist File Tool Failed")
        arcpy.AddError(arcpy.GetMessages(2))



  
def zipUpFolder(inputFC, chiefSciFile, nobelTecFile):
    try: 
        outZipFile = os.path.join(inputSratchFolder, "Stations.zip")
        zip = zipfile.ZipFile(outZipFile, 'w', zipfile.ZIP_DEFLATED)
        zipws(zip, inputFC, chiefSciFile, nobelTecFile)
        zip.close()
        return outZipFile
    except RuntimeError:
        if os.path.exists(outZipFile):
            os.unlink(outZipFile)
        zip = zipfile.ZipFile(outZipFile, 'w', zipfile.ZIP_STORED)
        zipws(zip, inputFC, chiefSciFile, nobelTecFile)
        zip.close()
        arcpy.AddWarning(arcpy.GetIDMessage(86133))


  

def zipws(zip, fc, chiefSciZip, nobelTecZip):
    path = os.path.normpath(os.path.dirname(fc))
    chiefSciName = os.path.basename(chiefSciZip)
    nobelTecName = os.path.basename(nobelTecZip)
    
    rootLen = len(path)+len(os.sep)
    for (dirpath, dirnames, filenames) in os.walk(path):
        for file in filenames:
            if not file.endswith('.lock'):
                zip.write(os.path.join(dirpath, file), os.path.join("Stations.gdb", os.path.join(dirpath, file)[rootLen:]))
   
    zip.write(chiefSciZip, chiefSciName)
    zip.write(nobelTecZip, nobelTecName)

 
 
 
def send_mail(send_to, subject, text, f):
    try:
        msg = MIMEMultipart()
        msg['From'] = "FirstName.LastName@MyEmail.com"
        msg['To'] = COMMASPACE.join(send_to)
        msg['CC'] = "FirstName.LastName@MyEmail.com"
        msg['Date'] = formatdate(localtime=True)
        msg['Subject'] = subject
        
        send_to = [send_to, "FirstName.LastName@MyEmail.com"]
        msg.attach( MIMEText(text) )
        part = MIMEBase('application', "zip")
        part.set_payload( open(f,"rb").read() )
        Encoders.encode_base64(part)
        part.add_header('Content-Disposition', 'attachment; filename="%s"' % os.path.basename(f))
        msg.attach(part)
            
        smtp = smtplib.SMTP("MyServer")
        smtp.sendmail("FirstName.LastName@MyEmail.com", send_to, msg.as_string())
        smtp.close()
    except:
        arcpy.SetParameterAsText(6, "False")
        arcpy.AddMessage("Send Email Tool Failed")
        arcpy.AddError(arcpy.GetMessages(2))



  
if __name__ == '__main__':
    try:
        inputGDB = arcpy.GetParameterAsText(0)
        global inputSratchFolder
        inputSratchFolder = arcpy.GetParameterAsText(1)
        inputFileName = arcpy.GetParameterAsText(2)
        inputSend = [arcpy.GetParameterAsText(3)]
        subject = arcpy.GetParameterAsText(4)
        text = arcpy.GetParameterAsText(5)
  
        outputNobelTecFile = createNobelTecFile(inputGDB, inputFileName)
  
        outputChiefSciFile = createChiefSciFile(inputGDB, inputFileName)
  
        outputZipFile = zipUpFolder(inputGDB, outputChiefSciFile, outputNobelTecFile)
        send_mail(inputSend, subject, text, outputZipFile)
        arcpy.SetParameterAsText(6, "True")

    except:
        arcpy.SetParameterAsText(6, "False")
        arcpy.AddMessage("Create Files Zip and Email Tool Failed")
        arcpy.AddError(arcpy.GetMessages(2))
0 Kudos