Latest Contributions by CharlieFrye

I decided to split this in two, and put the feature services code in a separate comment. Feature Services: This is a complete prototype, and I am not done working on it, but the code you need is in here, along with examples a variety of common tasks.  There is also code in here for creating a hosted feature service via Python--which I got to work with only this one instance, so cannot realistically support it, given that it is almost two years old. #------------------------------------------------------------------------------- # Name:        Supply and Demand Prototype # Purpose:     Demonstrate using online demographic service as basis for either #              supply or demand, then assign supply/demand index to input #              point features (business locations).  This prototype will #              specifically use case of population over age 65 from esri #              demographic services with user input locations for nursing #              homes within the U.S. # #              Goal is to find appropriate processing granualarity for using #              the demographic services. # #              Additional challenges include a follow on use of Hotspot Analysis #              service, and switching measurement of access of supply to demand #              from Euclidian distance to using esri's road/route network #              services. # Created:     11/10/2013 #------------------------------------------------------------------------------- # Import python modules import arcpy import re import copy import os, sys from arcpy import env # For Http calls import httplib, urllib, json #Globals refURL = 'www.arcgis.com' FailType = 'None' geomFineNess = 'Coarse' # 'Coarse','medium','refined' inputFSURL = r"http://services.arcgis.com/ldUM9mnXaowg9XKm/ArcGIS/rest/services/Microbreweries_as_of_November_4th_2013/FeatureServer" #supply demandType = "2014 Alcoholic Beverages Away from Home - Beer & Ale: Average"  # Field Name = X2008_A newSvcName = 'BeerShortage' NewItemName = 'Beer Shortage?' # Demand #http://demographics4.arcgis.com/arcgis/rest/services/USA_Consumer_Expenditures_2014/MapServer/4 # A function to generate a token given username, password and the adminURL. def getToken(username, password, serverName, serverPort):     # Token URL is typically http://server[:port]/arcgis/admin/generateToken     tokenURL = "/sharing/generateToken"     params = urllib.urlencode({'username': username, 'password': password, 'client': 'referer','referer': refURL, 'expiration':'60','f': 'json'})     headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}     # Connect to URL and post parameters     if serverPort == "":         httpConn = httplib.HTTPSConnection(serverName)     else:         httpConn = httplib.HTTPConnection(serverName, serverPort)     httpConn.request("POST", tokenURL, params, headers)     # Read response     response = httpConn.getresponse()     if (response.status != 200):         httpConn.close()         FailType = 'Failed getting token; check connection and credentials'         print "Error while fetching tokens from admin URL. Please check the URL and try again."         return     else:         data = response.read()         httpConn.close()         # Check that data returned is not an error object         if not assertJsonSuccess(data):             return         # Extract the token from it         token = json.loads(data)         return token['token'] # A function that checks that the input JSON object #  is not an error object. def assertJsonSuccess(data):     obj = json.loads(data)     if 'status' in obj and obj['status'] == "error":         print "Error: JSON object returns an error. " + str(obj)         return False     else:         return True def main():     orgAdminURL = r"http://services1.arcgis.com/pf6KDbd8NVL1IUHa/ArcGIS/admin"  #Where the results will be written     #                                            ^Organization^     #     # Get from local content in the organization of the user specified below in the credentials     scratchWkspc = arcpy.env.scratchGDB  #Local Processing Environment     arcpy.env.overwriteOutput = True     #  ######   Access for Premium Demographic Content  ######     userName = "" # Enter your credentials here     passWord = ""     demandServerName = "demographics4.arcgis.com"   #Demand Service Server (Expenditures on Alcohol)     supplyServerName = "services.arcgis.com"  #Supply Service Server     tokenServerName = "www.arcgis.com" #Portal that the servers authenticate against--Optional     serverPort = ""     #Get token via function     tokenString = (getToken(userName,passWord,tokenServerName,serverPort)).encode('ascii','ignore')     #                                                            0=Layer ID     service = r"Microbreweries_as_of_November_4th_2013/FeatureServer/0/query"     serviceURL = "/ldUM9mnXaowg9XKm/arcgis/rest/services/" + service     headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}     # Get the input feature service and make a feature set, which is what GP tools can use as input     inputField = "FID"     where = '1=1'     query = "/0/query?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, inputField, tokenString)     inputFSURL = r"http://services.arcgis.com/ldUM9mnXaowg9XKm/ArcGIS/rest/services/Microbreweries_as_of_November_4th_2013/FeatureServer"     fsURL = inputFSURL + query     # ###### Make the 'Feature Class' from the Service     inputSupplyFC = arcpy.FeatureSet()     inputSupplyFC.load(fsURL)     # ###### Get Spatial References #######     params2 = urllib.urlencode({'token': tokenString, 'f': 'JSON'})     httpConn = httplib.HTTPSConnection(supplyServerName)     httpConn.request("POST", inputFSURL+r"/0", params2, headers)     response = httpConn.getresponse()     httpReturnCode = response.status     if (httpReturnCode != 200):         httpConn.close()         print "Could not read service information. Code " + str(httpReturnCode)  #403 = Forbidden, #498 = Bad Token         FailType = "Could not get demand Spatial Reference"         return     else:         data = response.read()         httpConn.close()         if not assertJsonSuccess(data):             return     dataObj = json.loads(data)     srvJSONSR = dataObj["extent"]["spatialReference"]     srvJSONExt = dataObj["extent"]     srvJSONGeom = dataObj["geometryType"]     inputSupplySR = arcpy.SpatialReference(int(srvJSONSR.get('wkid')))     # Make sure the supply feature are points     if srvJSONGeom != u'esriGeometryPoint':         print "Input must be point features"         FailType = "Wrong type of geometry, expected Points"         exit     forSR = r"USA_Consumer_Expenditures_2014/MapServer"     srSvcURL = "/arcgis/rest/services/" + forSR     params2 = urllib.urlencode({'token': tokenString, 'f': 'JSON'})     httpConn = httplib.HTTPSConnection(demandServerName)     httpConn.request("POST", srSvcURL, params2, headers)     response = httpConn.getresponse()     httpReturnCode = response.status     if (httpReturnCode != 200):         httpConn.close()         print "Could not read service information. Code " + str(httpReturnCode)  #403 = Forbidden, #498 = Bad Token         FailType = "Could not get supply Spatial Reference"         return     else:         data = response.read()         httpConn.close()         if not assertJsonSuccess(data):             return     dataObj2 = json.loads(data)     srvJSONSR2 = dataObj2["spatialReference"]     demandSR = arcpy.SpatialReference(int(srvJSONSR2.get('wkid')))     # ######            Core Analysis Section               ######     # ######  Get the D and the S in Li = (si/S) – (di /D)  ######     # ######  Where D = total demand and S = total Supply   ######     # ###### Where di = local demand and si = local Supply  ######     # Basic process is 2 stage:     # Stage 1:     #     # Using the extent of the users's supply points, query the     # demand service to get the features we need. Then write     # those to a feature class in the scratch folder     #     # Stage 2:     # Spatial Join and summ, then calculate Li     xMin = float(srvJSONExt["xmin"])     xMax = float(srvJSONExt["xmax"])     yMin = float(srvJSONExt["ymin"])     yMax = float(srvJSONExt["ymax"])     # Iterate over service features based on the extent above     ##'Coarse','medium','refined'     if geomFineNess == 'refined':         levelOfDetail = '15'  #Layer Number of coarser generalized ZIP Boundaries     elif geomFineNess == 'medium':         levelOfDetail = '17'     else:         levelOfDetail = '20'     # Make a dictionary that contains the returned ZIPs, demand values, initial 0 for supply, and centroids     DemandFCName = r"\demandFeats"     newFCName = scratchWkspc + DemandFCName     if arcpy.Exists(newFCName):         arcpy.Delete_management(newFCName)     tempZIPs = arcpy.CreateFeatureclass_management(scratchWkspc, DemandFCName, "POLYGON", "", "DISABLED", "DISABLED", demandSR)     arcpy.AddField_management(tempZIPs, "ZIP", "TEXT", "", "", "16")     arcpy.AddField_management(tempZIPs, "Di", "LONG")     arcpy.AddField_management(tempZIPs, "Si", "LONG")     arcpy.AddField_management(tempZIPs, "Li", "DOUBLE")     service = r"USA_Consumer_Expenditures_2014/MapServer/"+levelOfDetail+"/query"     serviceURL = "/arcgis/rest/services/" + service     queryGeom = str(xMin) +',' + str(yMin) + ',' + str(xMax) + ',' + str (yMax)     zipParams = "token=" + tokenString + "&geometryType=esriGeometryEnvelope&geometry=" + queryGeom +"&inSR="+str(inputSupplySR.factoryCode)+"&spatialRel=esriSpatialRelIntersects&relationParam=" \                                                 "&objectIds=&where=&time=&returnCountOnly=false&returnIdsOnly=false&returnGeometry=true&maxAllowableOffset=&outSR=&geometryPrecision=1"     if demandType == "2014 Alcoholic Beverages Away from Home - Beer & Ale: Average":         zipParams = zipParams + "&outFields=ID%2C+X2008_A&f=json" ##    elif demandType == "Current Year Under Age 20": ##        zipParams = zipParams + "token=" + tokenString + "&outFields=ID%2C+POP0_CY%2C+POP5_CY%2C+POP10_CY%2C+POP15_CY&f=json" ##    elif demandType == "Current Year Over Age 18": ##        zipParams = zipParams + "token=" + tokenString + "&outFields=ID%2C+POP18UP_CY&f=json" ##    elif demandType == "Current Year Total Population": ##        zipParams = zipParams + "token=" + tokenString + "&outFields=ID%2C+TOTPOP_CY&f=json"     httpConn = httplib.HTTPSConnection(demandServerName)     httpConn.request("POST", serviceURL, zipParams, headers)     response = httpConn.getresponse()     httpReturnCode = response.status     if (httpReturnCode != 200):         httpConn.close()         FailType = "Failed to return demand features -- try a geographically smaller area or demand dataset"         print "Could not read service information. Code " + str(httpReturnCode) + ". " + FailType  #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!         return     else:         data = response.read()         httpConn.close()         if not assertJsonSuccess(data):             return     rows = arcpy.da.InsertCursor(tempZIPs,["ZIP","Di","SHAPE@"])     bigD = 0     dataObj3 = json.loads(data)     newData = copy.deepcopy(dataObj3["features"])     fcount =  len(dataObj3["features"])     for fno in range(0, (fcount - 1)):         feat = dataObj3["features"][fno]         attribs = feat["attributes"]         geom = feat["geometry"]["rings"]         zipName = attribs["ID"]         ##if zipName == '92314':         ##    print("stop")         del attribs["ID"]         attVals = attribs.values()         pop = 0         for aidx in attVals:             if aidx != None:                 pop = pop + int(aidx)         bigD = bigD + pop         k = arcpy.Array()         for part in geom:             j = arcpy.Array()             for xy in part:                 j.add(arcpy.Point(xy[0],xy[1]))             k.add(j)         rowNum = rows.insertRow( [zipName, pop, arcpy.Polygon(k)] )     del rows     supplyWithZips = scratchWkspc + r"\supplyWithZIPs"     if arcpy.Exists(supplyWithZips):         arcpy.Delete_management(supplyWithZips)     arcpy.Identity_analysis(inputSupplyFC,tempZIPs,supplyWithZips,"ALL","#","NO_RELATIONSHIPS")     supplySummary = scratchWkspc + r"\supplySummary"     if arcpy.Exists(supplySummary):         arcpy.Delete_management(supplySummary)     arcpy.Statistics_analysis(supplyWithZips,supplySummary,[[inputField, "SUM"]],"ZIP")     # make a dictionary of ZIPs and supply counts     rows = arcpy.SearchCursor(supplySummary)     bigS = 0  #101.2     supplies = dict()     for row in rows:         ZIPName = row.getValue("ZIP")         SupplyInZip = row.getValue("SUM_"+inputField)         if SupplyInZip != None:             bigS = bigS + SupplyInZip         else:             SupplyInZip = 0         supplies[ZIPName] = SupplyInZip     del rows, row     # now set the Si     rows = arcpy.da.UpdateCursor(tempZIPs,("*"))     for row in rows:         #Li = (si/S) – (di /D)         di = row[5]         zID = row[4]         si = 0         if supplies.has_key(zID) == 1:             si = supplies[zID]         row[6] = si         Li = float(di)/float(bigD) - float(si)/float(bigS)         row[7] = Li         rows.updateRow(row)     del rows, row     # Run Hot Spot Analysis (But this writes to my desktop, so     outHSA = r"E:\Data\SupplyAndDemand\TestInputs.gdb\demandZIPs_OptimizedHotSpotA"     arcpy.OptimizedHotSpotAnalysis_stats(tempZIPs, outHSA,"Li","COUNT_INCIDENTS_WITHIN_FISHNET_POLYGONS","#","#","#")     # Need to run tableJoin to add ZIP from tempZIPs to outHSA based on the SourceID in outHSA and ObjectID in tempZIPs     arcpy.JoinField_management(outHSA,"SOURCE_ID",tempZIPs,"OBJECTID","ZIP")     # In this version of this tool we will now write the result of hotspot to my hosted services.     # This will be in contrast to the next version which will be writing each intermediate output     # to my hosted services as well, and running successive processes based on that.     # ###### Check to see if the service we want to create already exists, if so bail. ************     orgChkSvcNameURL = "http://Landscape.maps.arcgis.com/sharing/rest/portals/YkVYBaX0zm7bsV3k/isServiceNameAvailable"     orgCheckNameParams = "name=" + newSvcName + "&type=Feature Service&f=json&token="+tokenString     httpConn = httplib.HTTPConnection("mappingteam.maps.arcgis.com")     httpConn.request("POST", orgChkSvcNameURL, orgCheckNameParams, headers)     response = httpConn.getresponse()     httpReturnCode = response.status     if (httpReturnCode != 200):         httpConn.close()         print "Could not read service information. Code " + str(httpReturnCode)  #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!         FailType = "Failed because requested output Service Name was already in use"         return     else:         data = response.read()         httpConn.close()         if not assertJsonSuccess(data):             return     dataObj4 = json.loads(data)     canUseName = dataObj4["available"]     if canUseName:         canCreateSvc = 1         # ######## Create the new service ########         orgMakeSvcURL = "http://mappingteam.maps.arcgis.com/sharing/rest/content/users/cfrye_mt/createService"         orgMakeSvcParams = ('targetType=featureService&f=json&token=' + tokenString +                             '&createparameters={"name":"' + newSvcName + '","currentVersion":10.11,"serviceDescription":"Supply and Demand Hotspots",' +                             '"supportedQueryFormats":"JSON","maxRecordCount":5000,"capabilities":"Create,Delete,Query,Update,Editing",'+                             '"description":"Locations of significant supply and demand based on ArcGIS hotspot analysis",'+                             '"hasStaticData":true,"hasVersionedData":false,"units":"esriMeters","supportsDisconnectedEditing":false,'+                             '"copyrightText":"Esri","allowGeometryUpdates":true,"syncEnabled":false,"tables":[],'+                             '"spatialReference":'+str(demandSR.factoryCode)+',"initialExtent":'+ json.dumps(srvJSONExt) +',' + ##"fullExtent":' + json.dumps(srvJSONExt) + ##'+"size":2662400,                             '"copyrightText":"Esri","syncEnabled":false,"tables":[],"editorTrackingInfo":'+                             '{"enableEditorTracking":true,"enableOwnershipAccessControl":true,"allowOthersToUpdate":true,"allowOthersToDelete":true},'+                             '"xssPreventionInfo":{"xssPreventionEnabled":true,"xssPreventionRule":"InputOnly","xssInputRule":"rejectInvalid"}}')         httpConn = httplib.HTTPConnection("mappingteam.maps.arcgis.com")         httpConn.request("POST", orgMakeSvcURL, orgMakeSvcParams, headers)         response = httpConn.getresponse()         httpReturnCode = response.status         if (httpReturnCode != 200):             httpConn.close()             print "Could not read service information. Code " + str(httpReturnCode)  #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!             FailType = "Create Service Failed"             return         else:             data = response.read()             httpConn.close()             if not assertJsonSuccess(data):                 return         dataObj5 = json.loads(data)         madeSvc = dataObj5["success"]         if madeSvc:             svcItemID = dataObj5["itemId"]             svcURL = dataObj5["serviceurl"]             encodedURL = dataObj5["encodedServiceURL"]             # ######## Do Add to Definition to complete the process ########             # this bit converts the ending of the service URL to what is needed to do AddToDefinition             xform1 = re.sub("rest","admin", svcURL)             add2SvcURL = re.sub("/FeatureServer",".FeatureServer", xform1) + "/addToDefinition"             add2SvcParams = ('addToDefinition={"layers":[{"currentVersion":10.11,"id":0,"name":"Hotspots for Li","type":"FeatureLayer","displayField":"Li",' +                              '"description":"test","copyrightText":"esri","defaultVisibility":true,"relationships":[],"isDataVersioned":false,"supportsRollbackOnFailureParameter":true,"supportsStatistics":true,' +                              '"supportsAdvancedQueries":true,"geometryType":"esriGeometryPolygon","minScale":0,"maxScale":0,"extent":' + json.dumps(srvJSONExt) +                              ',"drawingInfo":{"renderer":{"type":"uniqueValue","field1":"Gi_Bin","field2":"","field3":"","fieldDelimiter" : ", ","uniqueValueInfos":['+                              '{"value":-3,"label":"-3","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[69,117,181,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},' +                              '{"value":-2,"label":"-2","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[132,158,186,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},'+                              '{"value":-1,"label":"-1","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[192,204,190,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},' +                              '{"value":0,"label":"0","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[255,255,191,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},'+                              '{"value":1,"label":"1","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[250,185,132,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},' +                              '{"value":2,"label":"2","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[237,117,81,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}},'+                              '{"value":3,"label":"3","description":"","symbol":{"type":"esriSFS","style":"esriSFSSolid","color":[214,47,39,255],"outline":{"type":"esriSLS","style":"esriSLSSolid","color":[110,110,110,255],"width":0.375}}}]},' +                              '"transparency":0,"labelingInfo":null},"allowGeometryUpdates":true,"hasAttachments":false,"htmlPopupType":"esriServerHTMLPopupTypeAsHTMLText","hasM":false,"hasZ":false,' +                              '"objectIdField":"OBJECTID","globalIdField":"","typeIdField":"","fields":[' +                              '{"name":"OBJECTID","type":"esriFieldTypeOID","alias":"OBJECTID","sqlType":"sqlTypeOther","nullable":false,"editable":false,"domain":null},'+                              '{"name":"Li","type":"esriFieldTypeDouble","alias":"Li","sqlType":"sqlTypeOther","nullable":true,"editable":true,"domain":null,"defaultValue":0},' +                              '{"name":"GiZScore","type":"esriFieldTypeDouble","alias":"Gi Z Score","sqlType":"sqlTypeOther","nullable":true,"editable":true,"domain":null,"defaultValue":0},'+                              '{"name":"GiPValue","type":"esriFieldTypeDouble","alias":"Gi P Value","sqlType":"sqlTypeOther","nullable":true,"editable":true,"domain":null,"defaultValue":0},'+                              '{"name":"Gi_Bin","type":"esriFieldTypeInteger","alias":"Gi Bin","sqlType":"sqlTypeOther","nullable":true,"editable":true,"domain":null,"defaultValue":0},'+                              '{"name":"ZIP_ID","type":"esriFieldTypeString","alias":"ZIP Code","sqlType":"sqlTypeOther","nullable":true,"length":16,"editable":true,"domain":null,"defaultValue":null}'+                              ']'+                              ',"types":[],"templates":[{"name":"New Feature","description":"","drawingTool":"esriFeatureEditToolPolygon","prototype":{"attributes":{"Li":null,"GiZScore":null,"GiPValue":null,"Gi_Bin":null,"ZIP_ID":null}}}],"supportedQueryFormats":"JSON","hasStaticData":true,"maxRecordCount":5000,' +                              '"capabilities":"Query,Editing,generateRenderer,Create,Update,Delete,applyEdits","adminLayerInfo":{"geometryField":{"name":"Shape","srid":'+str(demandSR.factoryCode)+'}}}]}&f=json&token=' + tokenString)             refURL2 = "http://mappingteam.maps.arcgis.com/home/content.html"             headers4Add2 = {"Host": "services.arcgis.com", "Connection":"keep-alive", "Origin": "http://services.arcgis.com", "Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL, "Accept-Language":"en-US,en;q=0.8"}             httpConn = httplib.HTTPConnection("services.arcgis.com")             httpConn.request("POST", add2SvcURL, add2SvcParams, headers4Add2)             response = httpConn.getresponse()             httpReturnCode = response.status             if (httpReturnCode != 200):                 httpConn.close()                 print "Could not read service information. Code " + str(httpReturnCode)  #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!                 FailType = "Add to Definition Failed"                 return             else:                 data = response.read()                 httpConn.close()                 if not assertJsonSuccess(data):                     return             dataObj6 = json.loads(data)             if dataObj6.get("error") != None:                 madeAdds = False                 print(dataObj6)                 return             else:                 madeAdds = dataObj6["success"]             if madeAdds:                 # ###### Update the Item ########                 updateItemURL = "http://mappingteam.maps.arcgis.com/sharing/rest/content/users/" + userName + "/items/" + svcItemID + "/update"                 textparams = ('{"layers":[{"id":0,"popupInfo":{"title":"Hotspots for Li: {Li}","fieldInfos":[{"fieldName":"OBJECTID","label":"OBJECTID","isEditable":false,"tooltip":"",' +                      '"visible":true,"stringFieldOption":"textbox"},{"fieldName":"Li","label":"Li","isEditable":true,"tooltip":"","visible":true,"format":{"places":6,"digitSeparator":false},' +                      '"stringFieldOption":"textbox"},{"fieldName":"GiZScore","label":"Gi Z Score","isEditable":true,"tooltip":"","visible":true,"format":{"places":2,"digitSeparator":false},' +                      '"stringFieldOption":"textbox"},{"fieldName":"GiPValue","label":"Gi P Value","isEditable":true,"tooltip":"","visible":true,"format":{"places":2,"digitSeparator":false},' +                      '"stringFieldOption":"textbox"},{"fieldName":"Gi_Bin","label":"Gi Bin","isEditable":true,"tooltip":"","visible":true,"format":{"places":0,"digitSeparator":false},'+                      '"stringFieldOption":"textbox"},{"fieldName":"ZIP_ID","label":"ZIP Code","isEditable":true,"tooltip":"","visible":true,"stringFieldOption":"textbox"}],"description":null,'+                      '"showAttachments":true,"mediaInfos":[]}}]}')                 typeKeyWords = 'ArcGIS Server,Data,Feature Access,Feature Service,Service,Hosted Service'                 updateItemParams = ('f=json&token=' + tokenString + '&title=' + NewItemName + '&description=Locations of significant supply and demand based on ArcGIS hotspot analysis&tags=supply&extent=&thumbnailURL=http://services.arcgisonline.com/ArcGIS/rest/services/World_Topo_Map/MapServer/export?&bboxSR=4326&format=png24&f=image&bbox=-118.539,33.061,-114.753,35.021&typeKeywords='+ typeKeyWords + '&text='+textparams)                 httpConn = httplib.HTTPConnection("mappingteam.maps.arcgis.com")                 httpConn.request("POST", updateItemURL, updateItemParams, headers)                 response = httpConn.getresponse()                 httpReturnCode = response.status                 if (httpReturnCode != 200):                     httpConn.close()                     print "Could not read service information. Code " + str(httpReturnCode)  #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!                     FailType = "Udating new service's content item failed."                     return                 else:                     data = response.read()                     httpConn.close()                     if not assertJsonSuccess(data):                         return                 dataObj7 = json.loads(data)                 updateItem = dataObj7["success"]                 if updateItem:                     # ######## Build the JSON object containin the newFeatures from the hotspots attributes and                     # ######## and geometry from the newData obtained earlier from the ZIPs                     hsRows = arcpy.da.SearchCursor(outHSA,[u"Li",u"GiZScore",u"GiPValue",u"Gi_Bin",u"ZIP"])                     hsVals = {}                     for hsRow in hsRows:                         hsVals[hsRow[4]] = {"Li":hsRow[0],"GiZScore":hsRow[1],"GiPValue":hsRow[2],"Gi_Bin":hsRow[3]}                     newFeatures = []                     for feats in newData:                         geom = feats["geometry"]                         geom["spatialReference"]  = {"wkid":demandSR.factoryCode}                         ZIP = feats["attributes"]["ID"]                         if hsVals.get(ZIP) != None:  #All the ZIPs??                             newFeat = {"geometry":geom, "attributes":{"Li":hsVals.get(ZIP).get("Li"),                                                                       "GiZScore":hsVals.get(ZIP).get("GiZScore"),                                                                       "GiPValue":hsVals.get(ZIP).get("GiPValue"),                                                                       "Gi_Bin":hsVals.get(ZIP).get("Gi_Bin"),                                                                       "ZIP_ID":ZIP}}                             newFeatures.append(newFeat)                     newFeats = json.dumps(newFeatures)                     applyEditsURL = svcURL + "/0/addFeatures"                     uapplyEditsParams = ('f=json&rollbackOnFailure=false&features=' + newFeats + '&token=' + tokenString)                     httpConn = httplib.HTTPConnection("services.arcgis.com")                     httpConn.request("POST", applyEditsURL, uapplyEditsParams, headers4Add2)                     response = httpConn.getresponse()                     httpReturnCode = response.status                     if (httpReturnCode != 200):                         httpConn.close()                         print "Could not read service information. Code " + str(httpReturnCode)  #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!                         FailType = "Unable to add Features"                         return                     else:                         data = response.read()                         httpConn.close()                         if not assertJsonSuccess(data):                             return                     dataObj8 = json.loads(data)                     addedFeatures = dataObj8['addResults'][0]['success']                     if addedFeatures:                         genRendererURL = re.sub("addToDefinition","0/updateDefinition", add2SvcURL)                         finalDef = ('{"drawingInfo":{"renderer":{"type":"uniqueValue","field1":"Gi_Bin","defaultSymbol":null,"uniqueValueInfos":[' +                         '{"value":"-3","symbol":{"color":[69,117,181,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Cold Spot 99% Confidence"},' +                         '{"value":"-2","symbol":{"color":[132,158,186,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Cold Spot 95% Confidence"},' +                         '{"value":"-1","symbol":{"color":[192,204,190,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Cold Spot 90% Confidence"},' +                         '{"value":"0","symbol":{"color":[255,255,191,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Not Significant"},' +                         '{"value":"1","symbol":{"color":[250,185,132,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Hot Spot 90% Confidence"},' +                         '{"value":"2","symbol":{"color":[237,117,81,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Hot Spot 95% Confidence"},' +                         '{"value":"3","symbol":{"color":[214,47,39,255],"outline":{"color":[110,110,110,255],"width":0.75,"type":"esriSLS","style":"esriSLSSolid"},"type":"esriSFS","style":"esriSFSSolid"},"label":"Hot Spot 99% Confidence"}]}},' +                         '"typeIdField":"Gi_Bin","types":[' +                         '{"id":"-3","name":"-3","templates":[{"name":"-3","description":"","prototype":{"attributes":{"Gi_Bin":"-3","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +                         '{"id":"-2","name":"-2","templates":[{"name":"-2","description":"","prototype":{"attributes":{"Gi_Bin":"-2","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +                         '{"id":"-1","name":"-1","templates":[{"name":"-1","description":"","prototype":{"attributes":{"Gi_Bin":"-1","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +                         '{"id":"0","name":"0","templates":[{"name":"0","description":"","prototype":{"attributes":{"Gi_Bin":"0","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +                         '{"id":"1","name":"1","templates":[{"name":"1","description":"","prototype":{"attributes":{"Gi_Bin":"1","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +                         '{"id":"2","name":"2","templates":[{"name":"2","description":"","prototype":{"attributes":{"Gi_Bin":"2","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]},' +                         '{"id":"3","name":"3","templates":[{"name":"3","description":"","prototype":{"attributes":{"Gi_Bin":"3","Li":null,"GiZScore":null,"GiPValue":null,"ZIP_ID":null}}}]}],"templates":[]}')                         genRendererParams = ('f=json&updateDefinition=' + finalDef + '&token=' + tokenString)                         httpConn = httplib.HTTPConnection("services.arcgis.com")                         httpConn.request("POST", genRendererURL, genRendererParams, headers4Add2)                         response = httpConn.getresponse()                         httpReturnCode = response.status                         if (httpReturnCode != 200):                             httpConn.close()                             print "Could not read service information. Code " + str(httpReturnCode)  #403 = Forbidden, #498 = Bad Token, #504 Gateway timeout - it was too big!                             FailType = "Unable to add Features"                             return                         else:                             data = response.read()                             httpConn.close()                             if not assertJsonSuccess(data):                                 return                         dataObj9 = json.loads(data)                         print("Completed with: "+data)                     else:                         print("failed to add features.")                 else:                     print('could not update the content item.')             else:                 print('Could not do Add to Definition')         else:             print("Could not Create Service:  " + FailType)     else:         print("Could not use Service name:  ")         # name taken cannot continue if __name__ == '__main__':     main() ... View more

Hi Greg,  Below are examples for doing what you need. Depending on whether you are accessing image services or feature and dynamic map services, the workflow differs. Image services, which requires using the MakeImageServerLayer tool: #------------------------------------------------------------------------------- # Name:        Access Living Atlas Image Service Example ArcPy Python Script # Purpose:     Show how to us image services in scripting # Created:     07/15/2015 #------------------------------------------------------------------------------- # Import arcpy module import arcpy arcpy.CheckOutExtension("Spatial") from arcpy import env from arcpy.sa import * import os.path # set geoprocessing environment settings # including workspace, extent (from existing study area dataset) and coordinate system env.overwriteOutput = True env.workspace = r"D:\TestFs" env.extent = r"D:\TestFs\ImageLayerByPolygon.gdb\studyArea" env.outputCoordinateSystem = arcpy.SpatialReference(54009)  #using WKID for Mollweide, for area-based analysis # create connection file to esri landscape server from above parametersservername = "landscape7" userName = ""  #supply your own ArcGIS Online Organization account credentials passWord = "" servername = "landscape7" d = arcpy.mapping.CreateGISServerConnectionFile("USE_GIS_SERVICES", arcpy.env.scratchFolder,                                             servername + ".ags", "http://" + servername + ".arcgis.com/arcgis/rest/services",                                             "ARCGIS_SERVER", '', '', userName, passWord, "SAVE_USERNAME") # set up image layer from the landscape server connection, and get its spatial reference imageservicename = "World_Landforms_Improved_Hammond_Method" imageService = os.path.join(arcpy.env.scratchFolder, servername, imageservicename + ".ImageServer") # Create the Image Service Layer inputImageLayer = arcpy.MakeImageServerLayer_management(imageService,"imageLayer", "") # Example of using in a subsequent process outputrastername = "Landforms" output = r"D:\TestFs\Results.gdb" pr = arcpy.CopyRaster_management (inputImageLayer, output+"//"+ outputrastername) ... View more

Hi Madeline, I’ve enclosed two scripts (not sure how GeoNet will handle enclosures)… The GeoenrichPoly script uses landscape image services by summarizing them within a polygon. The SupplyDemand script shows how to use feature services, and represents a better formalization of coding practices by separating the getToken assertJsonSuccess functions. It took about 12 weeks to figure all this out while working with the development team here in Redlands. The point is that this is fairly advanced Python coding, particularly using the HTTPLib, and may take a few days to work through. Both are provided as is, with the expectation that you will have a username and password from an organization account to use in the variables that are currently defined as: userName = "" & passWord = "". Let me know if you have questions, Charlie ... View more

Good news on both fronts.  There is new content in the landscape layers, however it has not been announced yet, and I'd rather not do that here. I will be updating the PDF, likely in early April, as we have some additional content that will be done in the next week or two as well. On the Python question, you can use Python to access any public service (meaning if you know the URL to the ArcGIS server folder, it's public).  The main consideration is some services are authenticated, and therefore require a token to access.  Then it is only a matter of what kind of service is being accessed.  I will contact you separately to share this. ... View more

Here is a link to a tabular listing of Esri Landscape Layers as of today; noting there are plans to add more soon:  Landscape Services in Living Atlas.pdf - Box Also note there is a link at the bottom of the page to the ArcGIS Online group that contains all of these layers. Each layer in the list is hyperlinked to a corresponding ArcGIS Online content item, or to an ArcGIS Online search result when there are multiple layers that match the name.  For instance the latter scenario would include when we have an image service (raster) and a dynamic map service (vector) for the same topic. The descriptions in the content items introduce the layer, explain a little bit about the characteristics of the underlying data that are relevant for ArcGIS and for landscape applications.  For instance, it was often necessary to process the data in order to transform it into a form that would be immediately useful in ArcGIS.  Please let us know if we've failed to provide anything critical to successfully using any of these layers. ... View more

Thanks Ian--I've been writing VBA lately... ... View more

nameStr = row[0] if nameStr[0:3] == "***" then:    # This is if you wanted to just strip the ***    newName = nameStr.replace("***", None)    row[0] = newName    # set the string to nothing    row[0] = ""    cursor.update(row) ... View more

Replace the contents of the for block with: nameStr = row[0] if nameStr[0:3] == "***" then:    newName = nameStr.replace("***", None)    row[0] = newName    cursor.update(row) ... View more

Similarly, you could add five spaces "     ".  Using that notion, you could more flexibly manage columns of text within a label by introducing the best number of spaces to ensure alignment within columns based on the length of the string in the first column. ... View more

Unfortunately, we have an issue that we have not found a resolution to, which is that when the Extract Landscape Source Data geoprocessing service generates an error and sends the message you got, instead of the more informative messages we programmed into the tool. That we are still working on. In terms of getting you further along quickly, first, try a small area (a very small area, like 1 square mile), and verify that the tool is working at all for you (it should be as we ran into this a little over a week ago and verified that it was running properly when a sufficiently small area was chosen).  Generally have a look at the content item for the Extract Landscape Source Data service to get a better idea of how much data can be downloaded, in terms of feature counts. For soils, because the polygons are often rather complex, we support a maximum of 12,500 features per download, which is going to be a fairly small area. The easiest way to approach this is to split your study area into polygons that are sufficiently small and run the Extract service for each area and then merge the results. Editorially speaking, yes we totally agree with you that this could and should be better. We learned a lot trying to do this (make it possible to download a subset of 37.5 million features in a feature class). For now this is what we can deliver and support, but are working on better ways and as soon as we have one we will deliver it--luckily we are releasing our software more frequently, and we will evaluate each release to find opportunities to improve. ... View more

Yes it is, though it is important to access the image service via a connection file that you create in the python code.  Here is the essential snippet of code to use:     arcpy.mapping.CreateGISServerConnectionFile("USE_GIS_SERVICES", arcpy.env.scratchFolder,                                                 servername + ".ags", "http://" + servername + ".arcgis.com/arcgis/services",                                                 "ARCGIS_SERVER", '', '', userName, passWord, "SAVE_USERNAME")     # set up image layer from the landscape server connection, and get its spatial reference     imageService = os.path.join(arcpy.env.scratchFolder, servername, imageservicename + ".ImageServer")     inputImageLayer = arcpy.MakeImageServerLayer_management(imageService,"imageLayer") Once you have that inputImageLayer, you can use it to get properties via the arcpy.describe function: rastSpatRef = arcpy.Describe(inputImageLayer).spatialReference arcpy.env.cellSize = arcpy.Describe(inputImageLayer).meanCellWidth theCellSize = arcpy.Describe(inputImageLayer).meanCellWidth From there you can make a rasterLayer: arcpy.MakeRasterLayer_management(inputImageLayer, "temp_raster") And that can be used in many Geoprocessing functions: arcpy.sa.Con("junk2g","temp_raster") ... View more

While it is true that the REST endpoint HTML pages have been disabled for Esri's demographics services, landscape services and others, it is possible to access the information using the ArcGIS APIs, including Python as JSON rather than HTML.  Here is how using Python: Let's access this information for the USA Median Age service. The basic process is that you first need to get a token which will be used to authenticate you as a user when making the request for the service information.  Then the service information can be requested.  This example set of code gets the token and uses it to return the service's JSON and specifically get the spatial reference from the service. # Import python modules import arcpy # For Http calls import httplib, urllib, json #Globals refURL = 'ArcGIS' AGOL_org_User_Name = " " AGOL_passWord = " " def getToken(username, password, serverName, serverPort):     # A function to generate a token given username, password and the adminURL.     # Token URL is typically http://server[:port]/arcgis/admin/generateToken     tokenURL = "/sharing/generateToken"     params = urllib.urlencode({'username': username, 'password': password, 'client': 'referer','referer': refURL, 'expiration':'60','f': 'json'})     headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}     # Connect to URL and post parameters     if serverPort == "":         httpConn = httplib.HTTPSConnection(serverName)     else:         httpConn = httplib.HTTPConnection(serverName, serverPort)     httpConn.request("POST", tokenURL, params, headers)     # Read response     response = httpConn.getresponse()     if (response.status != 200):         httpConn.close()         FailType = 'Failed getting token; check connection and credentials'         print "Error while fetching tokens from admin URL. Please check the URL and try again."         return     else:         data = response.read()         httpConn.close()         # Check that data returned is not an error object         if not assertJsonSuccess(data):             return         # Extract the token from it         token = json.loads(data)         return token['token'] def assertJsonSuccess(data):     # A function that checks that the input JSON object     #  is not an error object.     obj = json.loads(data)     if 'status' in obj and obj['status'] == "error":         print "Error: JSON object returns an error. " + str(obj)         return False     else:         return True def main():     #  ######   Access for Premium Demographic Content  ######     # Get a Token     demogServerName = "demographics1.arcgis.com"     tokenServerName = "www.arcgis.com"     serverPort = ""     tokenString = (getToken(AGOL_org_User_Name,AGOL_passWord,tokenServerName,serverPort)).encode('ascii','ignore')     # Get the Spatial Reference object from the demographic service from Esri     demogServerName = "demographics1.arcgis.com"     headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}     srSvcURL = "/arcgis/rest/services/USA_Demographics_and_Boundaries/MapServer"     params2 = urllib.urlencode({'token': tokenString, 'f': 'JSON'})     httpConn = httplib.HTTPSConnection(demogServerName)     httpConn.request("POST", srSvcURL, params2, headers)     response = httpConn.getresponse()     httpReturnCode = response.status     if (httpReturnCode != 200):         httpConn.close()         print "Could not read service information. Code " + str(httpReturnCode)  #403 = Forbidden, #498 = Bad Token         FailType = "Could not get supply Spatial Reference"         return     else:         data = response.read()         httpConn.close()     if not assertJsonSuccess(data):         return     dataObj2 = json.loads(data)     srvJSONSR2 = dataObj2["spatialReference"]     demandSR = arcpy.SpatialReference(int(srvJSONSR2.get('wkid'))) if __name__ == '__main__':     main() The Result, if you print the dataObj2 variable is: {u'capabilities': u'Data,Map,Query', u'copyrightText': u'Esri, US Census Bureau, Navteq', u'currentVersion': 10.2, u'description': u'This thematic map contains over 2000 demographic variables from Esri Updated Demographics for 2013, including projections for 2018, and the United States 2010 Census, ready to be symbolized in ten different levels of geography. Two views are available. The default view depicts States at greater than 15m scale, Counties at 1.5m to 15m scale, Zip Codes at 500k to 1.5m scale, Census Tracts at 150k to 500k scale, and Census Block Groups at less than 150k scale. The second view allows the ten separate geography levels to be turned on or off individually, each including the full set of variables. The ten levels of geography represented are State, County, Tract, Block Group, Zip Code, Place, County Subdivision, Congressional District, CBSA and DMA.\n\nFor more information on this map, including our terms of use, visit us online at http://goto.arcgisonline.com/demographics/USA_Demographics_and_Boundaries', u'documentInfo': {u'AntialiasingMode': u'Fastest',                    u'Author': u'Esri',                    u'Category': u'',                    u'Comments': u'This thematic map contains over 2000 demographic variables from Esri Updated Demographics for 2013, including projections for 2018, and the United States 2010 Census, ready to be symbolized in ten different levels of geography. Two views are available. The default view depicts States at greater than 15m scale, Counties at 1.5m to 15m scale, Zip Codes at 500k to 1.5m scale, Census Tracts at 150k to 500k scale, and Census Block Groups at less than 150k scale. The second view allows the ten separate geography levels to be turned on or off individually, each including the full set of variables. The ten levels of geography represented are State, County, Tract, Block Group, Zip Code, Place, County Subdivision, Congressional District, CBSA and DMA.\n\nFor more information on this map, including our terms of use, visit us online at http://goto.arcgisonline.com/demographics/USA_Demographics_and_Boundaries',                    u'Keywords': u'polygon, demographics, census, state, county, tract, block group, Zip Code, postal, CBSA, DMA, place, county subdivision, congressional district, population, income, household, age, race, male, female',                    u'Subject': u'USA Demographics and Boundaries includes over 2000 demographic variables from the 2010 Census ready to symbolize in ten different geographies.',                    u'TextAntialiasingMode': u'Force',                    u'Title': u'USA Demographics and Boundaries'}, u'exportTilesAllowed': False, u'fullExtent': {u'spatialReference': {u'latestWkid': 3857, u'wkid': 102100},                  u'xmax': -7452828.103399999,                  u'xmin': -19942589.5526,                  u'ymax': 11537127.463399999,                  u'ymin': 2144435.340399999}, u'initialExtent': {u'spatialReference': {u'latestWkid': 3857,                                           u'wkid': 102100},                     u'xmax': -6427885.287170573,                     u'xmin': -20967532.368829425,                     u'ymax': 12006762.069549998,                     u'ymin': 8371850.2991352845}, u'layers': [{u'defaultVisibility': True,               u'id': 0,               u'maxScale': 0,               u'minScale': 0,               u'name': u'Demographics and Boundaries',               u'parentLayerId': -1,               u'subLayerIds': [1, 9, 10, 11, 12, 16]},              {u'defaultVisibility': True,               u'id': 1,               u'maxScale': 0,               u'minScale': 0,               u'name': u'State and County Boundaries',               u'parentLayerId': 0,               u'subLayerIds': [2, 3]},              {u'defaultVisibility': True,               u'id': 2,               u'maxScale': 0,               u'minScale': 1500000,               u'name': u'County Boundaries',               u'parentLayerId': 1,               u'subLayerIds': None},              {u'defaultVisibility': True, ## Shortened layers element to fit in post u'mapName': u'Layers', u'maxImageHeight': 4096, u'maxImageWidth': 4096, u'maxRecordCount': 3500, u'maxScale': 0, u'minScale': 0, u'serviceDescription': u'', u'singleFusedMapCache': False, u'spatialReference': {u'latestWkid': 3857, u'wkid': 102100}, u'supportedExtensions': u'KmlServer', u'supportedImageFormatTypes': u'PNG32,PNG24,PNG,JPG,DIB,TIFF,EMF,PS,PDF,GIF,SVG,SVGZ,BMP', u'supportedQueryFormats': u'JSON, AMF', u'supportsDynamicLayers': True, u'tables': [], u'units': u'esriMeters'} ... View more

I was able to make this work with a little help from our development team. Here is the example URL: http://services.myserver.com/ERmEceOGq5cHrItq/ArcGIS/admin/services/example1.FeatureServer/addToDefinition The blue portion is determined based on your organization. The best way to obtain what you'll need to include is to use either your browser's debug tools or something like Fiddler to capture the results of the Create Service button that is in your account's My Contents tab in the upper right. That will produce the local equivalent of the URL you'll need. You should also have created a feature service in ArcMap and shared it in order to provide yourself a "template" to use (rather than inventing the JSON object containing the parameters). Make sure the layer has the appropriate schema and then assign the symbols in the fashion you'd like them. Use that service when using the Create Service button, for the "an existing feature service" option. One other trick that is useful to know is once you've usedCreate service to get started, you can get to the REST services folder by first clicking on the REST URL in the new services's content item page and then clicking on the Services link in the upper left. Then change that URL to look like by changing "rest" to "admin": http://services.myserver.com/ERmEceOGq5cHrItq/ArcGIS/admin/services By first going through the Services Folder, you will set your browser up to manage your credentials (provided your ArcGIS Online credentials own the service) in order to get to the admin page for your service. If you do not own the service, you will not be able to access that page. ... View more

Yikes--we do have at least two meanings for "admin" in the situations you're asking about.  First, and the one I was referring to with respect to the groups question is your ArcGIS Online account in your organization is assigned to have the administrator role.  Second, is when creating the ArcGIS Server Admin connection in ArcCatalog, the credentials that are provided are for the server administrator (a user role defined on the server).  However, when creating a User connection, the credentials are for an ArcGIS online account, where we are internally checking to see if it is an account in an organization, a public account, or if no credentials are supplied, it is considered anonymous access. ... View more