Welcome to the Landscape Layers forum

CharlieFrye · ‎07-05-2013

Welcome to the Landscape Maps and Data forum. This forum is for you to post questions and provide ideas for the ArcGIS for Landscape (beta).

About the Landscape Analysis Layers

These layers are available for use now! Visit the Landscape Layers Group on ArcGIS.com to learn more.

Feedback

Your feedback matters! What layers do you find most useful? How are you using them? Did you use any geoprocessing tools with these services and what was your experience? If you want to let us know, post your feedback here on this forum. Please let us know too, if you have issues or are experiencing problems with any of the layers. We cannot wait to hear from you!

CharlieFrye · ‎12-08-2015

I decided to split this in two, and put the feature services code in a separate comment.

Feature Services: This is a complete prototype, and I am not done working on it, but the code you need is in here, along with examples a variety of common tasks. There is also code in here for creating a hosted feature service via Python--which I got to work with only this one instance, so cannot realistically support it, given that it is almost two years old.

#-------------------------------------------------------------------------------

# Name: Supply and Demand Prototype

# Purpose: Demonstrate using online demographic service as basis for either

# supply or demand, then assign supply/demand index to input

# point features (business locations). This prototype will

# specifically use case of population over age 65 from esri

# demographic services with user input locations for nursing

# homes within the U.S.

#

# Goal is to find appropriate processing granualarity for using

# the demographic services.

#

# Additional challenges include a follow on use of Hotspot Analysis

# service, and switching measurement of access of supply to demand

# from Euclidian distance to using esri's road/route network

# services.

# Created: 11/10/2013

#-------------------------------------------------------------------------------

# Import python modules

import arcpy

import re

import copy

import os, sys

from arcpy import env

# For Http calls

import httplib, urllib, json

#Globals

refURL = 'www.arcgis.com'

FailType = 'None'

geomFineNess = 'Coarse' # 'Coarse','medium','refined'

inputFSURL = r"http://services.arcgis.com/ldUM9mnXaowg9XKm/ArcGIS/rest/services/Microbreweries_as_of_November_4th_2..." #supply

demandType = "2014 Alcoholic Beverages Away from Home - Beer & Ale: Average" # Field Name = X2008_A

newSvcName = 'BeerShortage'

NewItemName = 'Beer Shortage?'

# Demand

#http://demographics4.arcgis.com/arcgis/rest/services/USA_Consumer_Expenditures_2014/MapServer/4

# A function to generate a token given username, password and the adminURL.

def getToken(username, password, serverName, serverPort):

# Token URL is typically http://server[:port]/arcgis/admin/generateToken

tokenURL = "/sharing/generateToken"

params = urllib.urlencode({'username': username, 'password': password, 'client': 'referer','referer': refURL, 'expiration':'60','f': 'json'})

headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}

# Connect to URL and post parameters

if serverPort == "":

httpConn = httplib.HTTPSConnection(serverName)

else:

httpConn = httplib.HTTPConnection(serverName, serverPort)

httpConn.request("POST", tokenURL, params, headers)

# Read response

response = httpConn.getresponse()

if (response.status != 200):

httpConn.close()

FailType = 'Failed getting token; check connection and credentials'

print "Error while fetching tokens from admin URL. Please check the URL and try again."

return

else:

data = response.read()

httpConn.close()

# Check that data returned is not an error object

if not assertJsonSuccess(data):

return

# Extract the token from it

token = json.loads(data)

return token['token']

# A function that checks that the input JSON object

# is not an error object.

def assertJsonSuccess(data):

obj = json.loads(data)

if 'status' in obj and obj['status'] == "error":

print "Error: JSON object returns an error. " + str(obj)

return False

else:

return True

def main():

orgAdminURL = r"http://services1.arcgis.com/pf6KDbd8NVL1IUHa/ArcGIS/admin" #Where the results will be written

# ^Organization^

#

# Get from local content in the organization of the user specified below in the credentials

scratchWkspc = arcpy.env.scratchGDB #Local Processing Environment

arcpy.env.overwriteOutput = True

# ###### Access for Premium Demographic Content ######

userName = "" # Enter your credentials here

passWord = ""

demandServerName = "demographics4.arcgis.com" #Demand Service Server (Expenditures on Alcohol)

supplyServerName = "services.arcgis.com" #Supply Service Server

tokenServerName = "www.arcgis.com" #Portal that the servers authenticate against--Optional

serverPort = ""

#Get token via function

tokenString = (getToken(userName,passWord,tokenServerName,serverPort)).encode('ascii','ignore')

# 0=Layer ID

service = r"Microbreweries_as_of_November_4th_2013/FeatureServer/0/query"

serviceURL = "/ldUM9mnXaowg9XKm/arcgis/rest/services/" + service

headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}

# Get the input feature service and make a feature set, which is what GP tools can use as input

inputField = "FID"

where = '1=1'

query = "/0/query?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, inputField, tokenString)

inputFSURL = r"http://services.arcgis.com/ldUM9mnXaowg9XKm/ArcGIS/rest/services/Microbreweries_as_of_November_4th_2..."

fsURL = inputFSURL + query

# ###### Make the 'Feature Class' from the Service

inputSupplyFC = arcpy.FeatureSet()

inputSupplyFC.load(fsURL)

# ###### Get Spatial References #######

params2 = urllib.urlencode({'token': tokenString, 'f': 'JSON'})

httpConn = httplib.HTTPSConnection(supplyServerName)

httpConn.request("POST", inputFSURL+r"/0", params2, headers)

response = httpConn.getresponse()

httpReturnCode = response.status

if (httpReturnCode != 200):

httpConn.close()

print "Could not read service information. Code " + str(httpReturnCode) #403 = Forbidden, #498 = Bad Token

FailType = "Could not get demand Spatial Reference"

return

else:

data = response.read()

httpConn.close()

if not assertJsonSuccess(data):

return

dataObj = json.loads(data)

srvJSONSR = dataObj["extent"]["spatialReference"]

srvJSONExt = dataObj["extent"]

srvJSONGeom = dataObj["geometryType"]

inputSupplySR = arcpy.SpatialReference(int(srvJSONSR.get('wkid')))

# Make sure the supply feature are points

if srvJSONGeom != u'esriGeometryPoint':

print "Input must be point features"

FailType = "Wrong type of geometry, expected Points"

exit

forSR = r"USA_Consumer_Expenditures_2014/MapServer"

srSvcURL = "/arcgis/rest/services/" + forSR

params2 = urllib.urlencode({'token': tokenString, 'f': 'JSON'})

httpConn = httplib.HTTPSConnection(demandServerName)

httpConn.request("POST", srSvcURL, params2, headers)

response = httpConn.getresponse()

httpReturnCode = response.status

if (httpReturnCode != 200):

httpConn.close()

print "Could not read service information. Code " + str(httpReturnCode) #403 = Forbidden, #498 = Bad Token

FailType = "Could not get supply Spatial Reference"

return

else:

data = response.read()

httpConn.close()

if not assertJsonSuccess(data):

return

dataObj2 = json.loads(data)

srvJSONSR2 = dataObj2["spatialReference"]

demandSR = arcpy.SpatialReference(int(srvJSONSR2.get('wkid')))

# ###### Core Analysis Section ######

# ###### Get the D and the S in Li = (si/S) – (di /D) ######

# ###### Where D = total demand and S = total Supply ######

# ###### Where di = local demand and si = local Supply ######

# Basic process is 2 stage:

# Stage 1:

#

# Using the extent of the users's supply points, query the

# demand service to get the features we need. Then write

# those to a feature class in the scratch folder

#

# Stage 2:

# Spatial Join and summ, then calculate Li

xMin = float(srvJSONExt["xmin"])

xMax = float(srvJSONExt["xmax"])

yMin = float(srvJSONExt["ymin"])

yMax = float(srvJSONExt["ymax"])

# Iterate over service features based on the extent above

##'Coarse','medium','refined'

if geomFineNess == 'refined':

levelOfDetail = '15' #Layer Number of coarser generalized ZIP Boundaries

elif geomFineNess == 'medium':

levelOfDetail = '17'

else:

levelOfDetail = '20'

# Make a dictionary that contains the returned ZIPs, demand values, initial 0 for supply, and centroids

DemandFCName = r"\demandFeats"

newFCName = scratchWkspc + DemandFCName

if arcpy.Exists(newFCName):

arcpy.Delete_management(newFCName)

tempZIPs = arcpy.CreateFeatureclass_management(scratchWkspc, DemandFCName, "POLYGON", "", "DISABLED", "DISABLED", demandSR)

arcpy.AddField_management(tempZIPs, "ZIP", "TEXT", "", "", "16")

arcpy.AddField_management(tempZIPs, "Di", "LONG")

arcpy.AddField_management(tempZIPs, "Si", "LONG")

arcpy.AddField_management(tempZIPs, "Li", "DOUBLE")

service = r"USA_Consumer_Expenditures_2014/MapServer/"+levelOfDetail+"/query"

serviceURL = "/arcgis/rest/services/" + service

queryGeom = str(xMin) +',' + str(yMin) + ',' + str(xMax) + ',' + str (yMax)

zipParams = "token=" + tokenString + "&geometryType=esriGeometryEnvelope&geometry=" + queryGeom +"&inSR="+str(inputSupplySR.factoryCode)+"&spatialRel=esriSpatialRelIntersects&relationParam=" \

"&objectIds=&where=&time=&returnCountOnly=false&returnIdsOnly=false&returnGeometry=true&maxAllowableOffset=&outSR=&geometryPrecision=1"

if demandType == "2014 Alcoholic Beverages Away from Home - Beer & Ale: Average":

zipParams = zipParams + "&outFields=ID%2C+X2008_A&f=json"

## elif demandType == "Current Year Under Age 20":

## zipParams = zipParams + "token=" + tokenString + "&outFields=ID%2C+POP0_CY%2C+POP5_CY%2C+POP10_CY%2C+POP15_CY&f=json"

## elif demandType == "Current Year Over Age 18":

## zipParams = zipParams + "token=" + tokenString + "&outFields=ID%2C+POP18UP_CY&f=json"

## elif demandType == "Current Year Total Population":

## zipParams = zipParams + "token=" + tokenString + "&outFields=ID%2C+TOTPOP_CY&f=json"

httpConn = httplib.HTTPSConnection(demandServerName)

httpConn.request("POST", serviceURL, zipParams, headers)

response = httpConn.getresponse()

httpReturnCode = response.status

if (httpReturnCode != 200):

httpConn.close()

FailType = "Failed to return demand features -- try a geographically smaller area or demand dataset"

print "Could not read service information. Code " + str(httpReturnCode) + ". " + FailType #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!

return

else:

data = response.read()

httpConn.close()

if not assertJsonSuccess(data):

return

rows = arcpy.da.InsertCursor(tempZIPs,["ZIP","Di","SHAPE@"])

bigD = 0

dataObj3 = json.loads(data)

newData = copy.deepcopy(dataObj3["features"])

fcount = len(dataObj3["features"])

for fno in range(0, (fcount - 1)):

feat = dataObj3["features"][fno]

attribs = feat["attributes"]

geom = feat["geometry"]["rings"]

zipName = attribs["ID"]

##if zipName == '92314':

## print("stop")

del attribs["ID"]

attVals = attribs.values()

pop = 0

for aidx in attVals:

if aidx != None:

pop = pop + int(aidx)

bigD = bigD + pop

k = arcpy.Array()

for part in geom:

j = arcpy.Array()

for xy in part:

j.add(arcpy.Point(xy[0],xy[1]))

k.add(j)

rowNum = rows.insertRow( [zipName, pop, arcpy.Polygon(k)] )

del rows

supplyWithZips = scratchWkspc + r"\supplyWithZIPs"

if arcpy.Exists(supplyWithZips):

arcpy.Delete_management(supplyWithZips)

arcpy.Identity_analysis(inputSupplyFC,tempZIPs,supplyWithZips,"ALL","#","NO_RELATIONSHIPS")

supplySummary = scratchWkspc + r"\supplySummary"

if arcpy.Exists(supplySummary):

arcpy.Delete_management(supplySummary)

arcpy.Statistics_analysis(supplyWithZips,supplySummary,[[inputField, "SUM"]],"ZIP")

# make a dictionary of ZIPs and supply counts

rows = arcpy.SearchCursor(supplySummary)

bigS = 0 #101.2

supplies = dict()

for row in rows:

ZIPName = row.getValue("ZIP")

SupplyInZip = row.getValue("SUM_"+inputField)

if SupplyInZip != None:

bigS = bigS + SupplyInZip

else:

SupplyInZip = 0

supplies[ZIPName] = SupplyInZip

del rows, row

# now set the Si

rows = arcpy.da.UpdateCursor(tempZIPs,("*"))

for row in rows:

#Li = (si/S) – (di /D)

di = row[5]

zID = row[4]

si = 0

if supplies.has_key(zID) == 1:

si = supplies[zID]

row[6] = si

Li = float(di)/float(bigD) - float(si)/float(bigS)

row[7] = Li

rows.updateRow(row)

del rows, row

# Run Hot Spot Analysis (But this writes to my desktop, so

outHSA = r"E:\Data\SupplyAndDemand\TestInputs.gdb\demandZIPs_OptimizedHotSpotA"

arcpy.OptimizedHotSpotAnalysis_stats(tempZIPs, outHSA,"Li","COUNT_INCIDENTS_WITHIN_FISHNET_POLYGONS","#","#","#")

# Need to run tableJoin to add ZIP from tempZIPs to outHSA based on the SourceID in outHSA and ObjectID in tempZIPs

arcpy.JoinField_management(outHSA,"SOURCE_ID",tempZIPs,"OBJECTID","ZIP")

# In this version of this tool we will now write the result of hotspot to my hosted services.

# This will be in contrast to the next version which will be writing each intermediate output

# to my hosted services as well, and running successive processes based on that.

# ###### Check to see if the service we want to create already exists, if so bail. ************

orgChkSvcNameURL = "http://Landscape.maps.arcgis.com/sharing/rest/portals/YkVYBaX0zm7bsV3k/isServiceNameAvailable"

orgCheckNameParams = "name=" + newSvcName + "&type=Feature Service&f=json&token="+tokenString

httpConn = httplib.HTTPConnection("mappingteam.maps.arcgis.com")

httpConn.request("POST", orgChkSvcNameURL, orgCheckNameParams, headers)

response = httpConn.getresponse()

httpReturnCode = response.status

if (httpReturnCode != 200):

httpConn.close()

print "Could not read service information. Code " + str(httpReturnCode) #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!

FailType = "Failed because requested output Service Name was already in use"

return

else:

data = response.read()

httpConn.close()

if not assertJsonSuccess(data):

return

dataObj4 = json.loads(data)

canUseName = dataObj4["available"]

if canUseName:

canCreateSvc = 1

# ######## Create the new service ########

orgMakeSvcURL = "http://mappingteam.maps.arcgis.com/sharing/rest/content/users/cfrye_mt/createService"

orgMakeSvcParams = ('targetType=featureService&f=json&token=' + tokenString +

'&createparameters={"name":"' + newSvcName + '","currentVersion":10.11,"serviceDescription":"Supply and Demand Hotspots",' +

'"supportedQueryFormats":"JSON","maxRecordCount":5000,"capabilities":"Create,Delete,Query,Update,Editing",'+

'"description":"Locations of significant supply and demand based on ArcGIS hotspot analysis",'+

'"hasStaticData":true,"hasVersionedData":false,"units":"esriMeters","supportsDisconnectedEditing":false,'+

'"copyrightText":"Esri","allowGeometryUpdates":true,"syncEnabled":false,"tables":[],'+

'"spatialReference":'+str(demandSR.factoryCode)+',"initialExtent":'+ json.dumps(srvJSONExt) +',' + ##"fullExtent":' + json.dumps(srvJSONExt) + ##'+"size":2662400,

'"copyrightText":"Esri","syncEnabled":false,"tables":[],"editorTrackingInfo":'+

'{"enableEditorTracking":true,"enableOwnershipAccessControl":true,"allowOthersToUpdate":true,"allowOthersToDelete":true},'+

'"xssPreventionInfo":{"xssPreventionEnabled":true,"xssPreventionRule":"InputOnly","xssInputRule":"rejectInvalid"}}')

httpConn = httplib.HTTPConnection("mappingteam.maps.arcgis.com")

httpConn.request("POST", orgMakeSvcURL, orgMakeSvcParams, headers)

response = httpConn.getresponse()

httpReturnCode = response.status

if (httpReturnCode != 200):

httpConn.close()

print "Could not read service information. Code " + str(httpReturnCode) #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!

FailType = "Create Service Failed"

return

else:

data = response.read()

httpConn.close()

if not assertJsonSuccess(data):

return

dataObj5 = json.loads(data)

madeSvc = dataObj5["success"]

if madeSvc:

svcItemID = dataObj5["itemId"]

svcURL = dataObj5["serviceurl"]

encodedURL = dataObj5["encodedServiceURL"]

# ######## Do Add to Definition to complete the process ########

# this bit converts the ending of the service URL to what is needed to do AddToDefinition

xform1 = re.sub("rest","admin", svcURL)

add2SvcURL = re.sub("/FeatureServer",".FeatureServer", xform1) + "/addToDefinition"

add2SvcParams = ('addToDefinition={"layers":[{"currentVersion":10.11,"id":0,"name":"Hotspots for Li","type":"FeatureLayer","displayField":"Li",' +

'"description":"test","copyrightText":"esri","defaultVisibility":true,"relationships":[],"isDataVersioned":false,"supportsRollbackOnFailureParameter":true,"supportsStatistics":true,' +

'"supportsAdvancedQueries":true,"geometryType":"esriGeometryPolygon","minScale":0,"maxScale":0,"extent":' + json.dumps(srvJSONExt) +

',"drawingInfo":{"renderer":{"type":"uniqueValue","field1":"Gi_Bin","field2":"","field3":"","fieldDelimiter" : ", ","uniqueValueInfos":['+

'{"value":-3,"label":"-3","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[69,117,181,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},' +

'{"value":-2,"label":"-2","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[132,158,186,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},'+

'{"value":-1,"label":"-1","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[192,204,190,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},' +

'{"value":0,"label":"0","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[255,255,191,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},'+

'{"value":1,"label":"1","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[250,185,132,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},' +

'{"value":2,"label":"2","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[237,117,81,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},'+

'{"value":3,"label":"3","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[214,47,39,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}}]},' +

'"transparency":0,"labelingInfo":null},"allowGeometryUpdates":true,"hasAttachments":false,"htmlPopupType":"esriServerHTMLPopupTypeAsHTMLText","hasM":false,"hasZ":false,' +

'"objectIdField":"OBJECTID","globalIdField":"","typeIdField":"","fields":[' +

'{"name":"OBJECTID","type":"esriFieldTypeOID","alias":"OBJECTID","sqlType":"sqlTypeOther","nullable":false,"editable":false,"domain":null},'+

'{"name":"Li","type":"esriFieldTypeDouble","alias":"Li","sqlType":"sqlTypeOther","nullable":true,"editable":true,"domain":null,"defaultValue":0},' +

'{"name":"GiZScore","type":"esriFieldTypeDouble","alias":"Gi Z Score","sqlType":"sqlTypeOther","nullable":true,"editable":true,"domain":null,"defaultValue":0},'+

'{"name":"GiPValue","type":"esriFieldTypeDouble","alias":"Gi P Value","sqlType":"sqlTypeOther","nullable":true,"editable":true,"domain":null,"defaultValue":0},'+

'{"name":"Gi_Bin","type":"esriFieldTypeInteger","alias":"Gi Bin","sqlType":"sqlTypeOther","nullable":true,"editable":true,"domain":null,"defaultValue":0},'+

'{"name":"ZIP_ID","type":"esriFieldTypeString","alias":"ZIP Code","sqlType":"sqlTypeOther","nullable":true,"length":16,"editable":true,"domain":null,"defaultValue":null}'+

']'+

',"types":[],"templates":[{"name":"New Feature","description":"","drawingTool":"esriFeatureEditToolPolygon","prototype":{"attributes":{"Li":null,"GiZScore":null,"GiPValue":null,"Gi_Bin":null,"ZIP_ID":null}}}],"supportedQueryFormats":"JSON","hasStaticData":true,"maxRecordCount":5000,' +

'"capabilities":"Query,Editing,generateRenderer,Create,Update,Delete,applyEdits","adminLayerInfo":{"geometryField":{"name":"Shape","srid":'+str(demandSR.factoryCode)+'}}}]}&f=json&token=' + tokenString)

refURL2 = "http://mappingteam.maps.arcgis.com/home/content.html"

headers4Add2 = {"Host": "services.arcgis.com", "Connection":"keep-alive", "Origin": "http://services.arcgis.com", "Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL, "Accept-Language":"en-US,en;q=0.8"}

httpConn = httplib.HTTPConnection("services.arcgis.com")

httpConn.request("POST", add2SvcURL, add2SvcParams, headers4Add2)

response = httpConn.getresponse()

httpReturnCode = response.status

if (httpReturnCode != 200):

httpConn.close()

print "Could not read service information. Code " + str(httpReturnCode) #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!

FailType = "Add to Definition Failed"

return

else:

data = response.read()

httpConn.close()

if not assertJsonSuccess(data):

return

dataObj6 = json.loads(data)

if dataObj6.get("error") != None:

madeAdds = False

print(dataObj6)

return

else:

madeAdds = dataObj6["success"]

if madeAdds:

# ###### Update the Item ########

updateItemURL = "http://mappingteam.maps.arcgis.com/sharing/rest/content/users/" + userName + "/items/" + svcItemID + "/update"

textparams = ('{"layers":[{"id":0,"popupInfo":{"title":"Hotspots for Li: {Li}","fieldInfos":[{"fieldName":"OBJECTID","label":"OBJECTID","isEditable":false,"tooltip":"",' +

'"visible":true,"stringFieldOption":"textbox"},{"fieldName":"Li","label":"Li","isEditable":true,"tooltip":"","visible":true,"format":{"places":6,"digitSeparator":false},' +

'"stringFieldOption":"textbox"},{"fieldName":"GiZScore","label":"Gi Z Score","isEditable":true,"tooltip":"","visible":true,"format":{"places":2,"digitSeparator":false},' +

'"stringFieldOption":"textbox"},{"fieldName":"GiPValue","label":"Gi P Value","isEditable":true,"tooltip":"","visible":true,"format":{"places":2,"digitSeparator":false},' +

'"stringFieldOption":"textbox"},{"fieldName":"Gi_Bin","label":"Gi Bin","isEditable":true,"tooltip":"","visible":true,"format":{"places":0,"digitSeparator":false},'+

'"stringFieldOption":"textbox"},{"fieldName":"ZIP_ID","label":"ZIP Code","isEditable":true,"tooltip":"","visible":true,"stringFieldOption":"textbox"}],"description":null,'+

'"showAttachments":true,"mediaInfos":[]}}]}')

typeKeyWords = 'ArcGIS Server,Data,Feature Access,Feature Service,Service,Hosted Service'

updateItemParams = ('f=json&token=' + tokenString + '&title=' + NewItemName + '&description=Locations of significant supply and demand based on ArcGIS hotspot analysis&tags=supply&extent=&thumbnailURL=http://services.arcgisonline.com/ArcGIS/rest/services/World_Topo_Map/MapServer/export?&bboxSR=4326&f... typeKeyWords + '&text='+textparams)

httpConn = httplib.HTTPConnection("mappingteam.maps.arcgis.com")

httpConn.request("POST", updateItemURL, updateItemParams, headers)

response = httpConn.getresponse()

httpReturnCode = response.status

if (httpReturnCode != 200):

httpConn.close()

print "Could not read service information. Code " + str(httpReturnCode) #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!

FailType = "Udating new service's content item failed."

return

else:

data = response.read()

httpConn.close()

if not assertJsonSuccess(data):

return

dataObj7 = json.loads(data)

updateItem = dataObj7["success"]

if updateItem:

# ######## Build the JSON object containin the newFeatures from the hotspots attributes and

# ######## and geometry from the newData obtained earlier from the ZIPs

hsRows = arcpy.da.SearchCursor(outHSA,[u"Li",u"GiZScore",u"GiPValue",u"Gi_Bin",u"ZIP"])

hsVals = {}

for hsRow in hsRows:

hsVals[hsRow[4]] = {"Li":hsRow[0],"GiZScore":hsRow[1],"GiPValue":hsRow[2],"Gi_Bin":hsRow[3]}

newFeatures = []

for feats in newData:

geom = feats["geometry"]

geom["spatialReference"] = {"wkid":demandSR.factoryCode}

ZIP = feats["attributes"]["ID"]

if hsVals.get(ZIP) != None: #All the ZIPs??

newFeat = {"geometry":geom, "attributes":{"Li":hsVals.get(ZIP).get("Li"),

"GiZScore":hsVals.get(ZIP).get("GiZScore"),

"GiPValue":hsVals.get(ZIP).get("GiPValue"),

"Gi_Bin":hsVals.get(ZIP).get("Gi_Bin"),

"ZIP_ID":ZIP}}

newFeatures.append(newFeat)

newFeats = json.dumps(newFeatures)

applyEditsURL = svcURL + "/0/addFeatures"

uapplyEditsParams = ('f=json&rollbackOnFailure=false&features=' + newFeats + '&token=' + tokenString)

httpConn = httplib.HTTPConnection("services.arcgis.com")

httpConn.request("POST", applyEditsURL, uapplyEditsParams, headers4Add2)

response = httpConn.getresponse()

httpReturnCode = response.status

if (httpReturnCode != 200):

httpConn.close()

print "Could not read service information. Code " + str(httpReturnCode) #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!

FailType = "Unable to add Features"

return

else:

data = response.read()

httpConn.close()

if not assertJsonSuccess(data):

return

dataObj8 = json.loads(data)

addedFeatures = dataObj8['addResults'][0]['success']

if addedFeatures:

genRendererURL = re.sub("addToDefinition","0/updateDefinition", add2SvcURL)

finalDef = ('{"drawingInfo":{"renderer":{"type":"uniqueValue","field1":"Gi_Bin","defaultSymbol":null,"uniqueValueInfos":[' +

'{"value":"-3","symbol":{"color":[69,117,181,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Cold Spot 99% Confidence"},' +

'{"value":"-2","symbol":{"color":[132,158,186,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Cold Spot 95% Confidence"},' +

'{"value":"-1","symbol":{"color":[192,204,190,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Cold Spot 90% Confidence"},' +

'{"value":"0","symbol":{"color":[255,255,191,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Not Significant"},' +

'{"value":"1","symbol":{"color":[250,185,132,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Hot Spot 90% Confidence"},' +

'{"value":"2","symbol":{"color":[237,117,81,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Hot Spot 95% Confidence"},' +

'{"value":"3","symbol":{"color":[214,47,39,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Hot Spot 99% Confidence"}]}},' +

'"typeIdField":"Gi_Bin","types":[' +

'{"id":"-3","name":"-3","templates":[{"name":"-3","description":"","prototype":{"attributes":{"Gi_Bin":"-3","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +

'{"id":"-2","name":"-2","templates":[{"name":"-2","description":"","prototype":{"attributes":{"Gi_Bin":"-2","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +

'{"id":"-1","name":"-1","templates":[{"name":"-1","description":"","prototype":{"attributes":{"Gi_Bin":"-1","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +

'{"id":"0","name":"0","templates":[{"name":"0","description":"","prototype":{"attributes":{"Gi_Bin":"0","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +

'{"id":"1","name":"1","templates":[{"name":"1","description":"","prototype":{"attributes":{"Gi_Bin":"1","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +

'{"id":"2","name":"2","templates":[{"name":"2","description":"","prototype":{"attributes":{"Gi_Bin":"2","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +

'{"id":"3","name":"3","templates":[{"name":"3","description":"","prototype":{"attributes":{"Gi_Bin":"3","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]}],"templates":[]}')

genRendererParams = ('f=json&updateDefinition=' + finalDef + '&token=' + tokenString)

httpConn = httplib.HTTPConnection("services.arcgis.com")

httpConn.request("POST", genRendererURL, genRendererParams, headers4Add2)

response = httpConn.getresponse()

httpReturnCode = response.status

if (httpReturnCode != 200):

httpConn.close()

print "Could not read service information. Code " + str(httpReturnCode) #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!

FailType = "Unable to add Features"

return

else:

data = response.read()

httpConn.close()

if not assertJsonSuccess(data):

return

dataObj9 = json.loads(data)

print("Completed with: "+data)

else:

print("failed to add features.")

else:

print('could not update the content item.')

else:

print('Could not do Add to Definition')

else:

print("Could not Create Service: " + FailType)

else:

print("Could not use Service name: ")

# name taken cannot continue

if __name__ == '__main__':

main()

GregConiglio · ‎12-08-2015

Thanks Charlie!!!

Taking a look already at this...

Looking at "USA Roads" in the Landscape group, in this list it only appears to have a MapService not FeatureService

http://www.arcgis.com/home/group.html?owner=esri&title=Landscape%20Layers&sortField=title&sortOrder=...

though proceeding suggest that vector data is available

http://proceedings.esri.com/library/userconf/proc14/tech-workshops/tw_453.pdf

Would the REST endpoint need to be a feature service for your script to work?

CynthiaCastro1 · ‎02-03-2016

Greg, thanks for posting this. I had the same questions back in November, and have contacted Esri without much luck. My code, as it stands, works about 50% of the time. I cannot for the life of me figure out what I am doing differently to get it to work. My thoughts were it would help to already establish the connection to landscape1 and also be logged into ArcGIS online, but even with that it works intermittently. Perhaps it is a difference between using 10.2 and 10.3, or an internet connectivity issue? I wonder if the Esri server is getting thrown sometimes, causing an error. If you were able to figure this out, will you please check out my code and offer any pointers?

arcpy.ImportToolbox("https://landscape1.arcgis.com:443/arcgis/services/;Tools/ExtractData")

Boundary = "Boundary"

resultsC = arcpy.ExtractData_ExtractData(Choose_Landscape_Layer="NHDPlus V2.1 Seamless Catchments", Study_Area="Boundary")

resultsF = arcpy.ExtractData_ExtractData(Choose_Landscape_Layer="NHDPlus V2.1 Seamless Flowlines", Study_Area="Boundary")

And then I use zipfile to extract the resultsC and resultsF. Any thoughts on why this only works sometimes?

Thank you so much!

CharlieFrye · ‎02-03-2016

We've had sporadic issues with that server, though it is generally up. We have had intermittent issues with that tool, and frequently need to restart instances of the server to get it to working.

That said, there is also a limit on the number of features that can be be downloaded via the ExtractData tool, so use a small extent (something that would download only a few flowline features) to verify. The service should return a useful error message if the extent is too large, meaning too many features have been requested.

The feature counts limits are listed here: http://www.arcgis.com/home/item.html?id=06f9e2f5f7ee4efcae283ea389bded15

send me an email if the tool seems to be down, and we will have a look. (Ha! I didn't know I could put my email in a hyperlink here).

Thanks

GregConiglio · ‎12-08-2015

OK made progress! But stuck... I build the service request like this:

http://landscape1.arcgis.com/arcgis/rest/services/USA_Roads/MapServer/2/query?where=1=1&outFields=OB...

and get this error in py

Traceback (most recent call last):

File "L:\Tools\EnE_Python_Tools\Land.py", line 108, in <module>

inputSupplyFC.load(fsURL)

File "C:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\arcobjects\arcobjects.py", line 175, in load

return convertArcObjectToPythonObject(self._arc_object.Load(*gp_fixargs(args)))

RuntimeError: RecordSetObject: Cannot open table for Load

Here's the odd thing. If I paste that URL into my browser as is I get this JSON:

{"error":{"code":498,"message":"Invalid Token","details":[]}}

I don't feel like the token is invalid because if I just enter the base URL without the parameters, I get the ArcGIS REST login screen, enter the token string and it's accepted. Once I do this I can, in the same browser window, enter the same URL request above, and it works, returning the proper JSON!!!

CharlieFrye · ‎12-08-2015

Replying to both questions:

Actually the dynamic map services can work as feature services do; at least for these services. Not all dynamic map services serve feature classes, which is why they are not general supported like feature services in the various ArcGIS client interfaces.

The invalid token message can occur if you get a token from one portal and try to use it for another. This should not be a problem if your credentials are for a .arcgis.com organization account. But, if your credentials are for a different portal, then the token will not be valid. One way to test this is just run the code that generates the token, to get a new token, and substitute that into the URL pasted into the browser.

That written, since it may be useful, I tried this by getting my own token, and got the same error. So I went to the REST endpoint, which required that same token, and it worked. It looks like there is a problem with the USA Roads service, none of the layers are showing in the REST end point. I'll see what I can do.

GregConiglio · ‎12-08-2015

Thanks Charlie! I'll paste the token code though sounds like there could be bigger issues if you had the same problem.

For what it's worth I also just tried it on the "USA_Wetlands" map service, with the same resulting error...

Thanks for your attention on this!

CharlieFrye · ‎12-10-2015

We got this fixed. Sorry for the delay and inconvenience.

GregConiglio · ‎12-11-2015

Thanks Charlie! I gave it a try and same behavior? I must be doing something wrong.--- same error but when I manually apply the token that was generated I am able to successfully see returned JSON using the service request. Maybe it could be something that you originally had pointed to --- using incorrect URL's for token requests? And I am indeed using a valid AGO organizational account. Here is the full code with the what I think are relevant lines highlighted ... Thanks for looking!!

#Globals

refURL = 'www.arcgis.com'

FailType = 'None'

geomFineNess = 'Coarse' # 'Coarse','medium','refined'

inputFSURL = r"http://landscape1.arcgis.com/arcgis/rest/services/USA_Wetlands/MapServer" #supply

# A function to generate a token given username, password and the adminURL.

def getToken(username, password, serverName, serverPort):

# Token URL is typically http://server[:port]/arcgis/admin/generateToken

print "entering getToken"

tokenURL = "/sharing/generateToken"

params = urllib.urlencode({'username': username, 'password': password, 'client': 'referer','referer': refURL, 'expiration':'60','f': 'json'})

headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}

# Connect to URL and post parameters

if serverPort == "":

httpConn = httplib.HTTPSConnection(serverName)

else:

httpConn = httplib.HTTPConnection(serverName, serverPort)

print "request" + tokenURL + "/" + str(params) + "/" + str(headers)

httpConn.request("POST", tokenURL, params, headers)

print "request complete"

# Read response

response = httpConn.getresponse()

if (response.status != 200):

httpConn.close()

FailType = 'Failed getting token; check connection and credentials'

print "Error while fetching tokens from admin URL. Please check the URL and try again."

return

else:

data = response.read()

httpConn.close()

print "response:" + data

# Check that data returned is not an error object

if not assertJsonSuccess(data):

return

# Extract the token from it

token = json.loads(data)

print "TOKEN SUCCESS"

return token['token']

# A function that checks that the input JSON object

# is not an error object.

def assertJsonSuccess(data):

obj = json.loads(data)

if 'status' in obj and obj['status'] == "error":

print "Error: JSON object returns an error. " + str(obj)

return False

else:

return True

orgAdminURL = r"http://services1.arcgis.com/pf6KDbd8NVL1IUHa/ArcGIS/admin" #Where the results will be written

# ^Organization^

#

# Get from local content in the organization of the user specified below in the credentials

scratchWkspc = arcpy.env.scratchGDB #Local Processing Environment

arcpy.env.overwriteOutput = True

# ###### Access for Premium Demographic Content ######

userName = "" # Enter your credentials here

passWord = ""

demandServerName = "demographics4.arcgis.com" #Demand Service Server (Expenditures on Alcohol)

supplyServerName = "services.arcgis.com" #Supply Service Server

tokenServerName = "arcgis.com" #Portal that the servers authenticate against--Optional

serverPort = ""

#Get token via function

tokenString = (getToken("greg.coniglio","cheyenne1",tokenServerName,serverPort)).encode('ascii','ignore')

# 0=Layer ID

service = r"/USA_Wetlands/MapServer/0/query"

serviceURL = "http://landscape1.arcgis.com/arcgis/rest/services" + service

headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}

# Get the input feature service and make a feature set, which is what GP tools can use as input

inputField = "OBJECTID"

where = '1=1'

query = "/0/query?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, inputField, tokenString)

inputFSURL = r"landscape1.arcgis.com/arcgis/rest/services/USA_Wetlands/MapServer"

CharlieFrye · ‎12-14-2015

I just double checked, both the USA_Roads and USA Wetlands services are working properly. That can be tested by using a URL like: http://landscape1.arcgis.com/arcgis/rest/services/USA_Wetlands/MapServer/0/5

where the last digit is the ObjectID of a feature, which will return the attributes of that feature.

it looks like you may have an issue in your code, where "0/query" is at the end of your service variable, and also in your query variable.