Latest Contributions by GregConiglio

Thanks Charlie -- oh Fiddler, I didn't think of that!  Keep me posted on the load method. That makes sense as I noted twice it DID work .. and then failed after re-running after what I believe were no additional changes in the code Your help with this is extremely appreciated Charlie!  Happy Holidays!! ... View more

Thanks Charlie!  I spent some time getting the code streamlined (a lot of unused code, and duplicate variables as you noted).  It worked a few time!!  it is back to the same error and I am at an impasse why.  When it was working for you, did you request JSON as the output?  THat seems to be an issue.  The script fails with the "Cannot load" error still, presumably because of a lack of proper response / error.  I do these steps: 1. Run the code Ipasted below).  I get the standard error "RuntimeError: RecordSetObject: Cannot open table for Load" 2. In my python output I print the URL we use.  Example here.  I copy and paste this into a browser http://landscape1.arcgis.com/arcgis/rest/services/USA_Roads/MapServer/0/query?returnGeometry=true&f=json&inSR=4326&geome… {"error":{"code":498,"message":"Invalid Token","details":[]}} is the result in the browser. 3. if I remove the "&f=json" argument on the URL, I get a different result:  I am redirected to the ESRI Web API Rest Login Page (which redirects using a URL-safe version of my URL).  It's a little odd because the token argument is in the URL, but I from here I can copy the token string (the same one from above) into my text box and ESRI accepts it. 4. If I repeat step 2 and paste into the same browser instance, the desired JSON result is returned! (Example- {"displayFieldName":"fullname","fieldAliases":{"fullname":"Full Name"},"geometryType":"esriGeometryPolyline","spatialReference":{"wkid":102100,"latestWkid":3857},"fields":[{"name":"fullname","type":"esriFieldTypeString","alias":"Full Name","length":100}],"features":[{"attributes":{"fullname":"New York State Throughway "},"geometry":{"paths":[[[-8734554.8257999998,5309850.4296000004],[-8734553.2673000004,5309851.1904999986],[-8733750.6537999995,5310086.1526999995],[-8732985.3322999999,5310311.2289000005],[-8732888.8182999995,5310339.3828999996],[-8732505.6566000003,5310451.6950000003],[-8731923.9010000005,5310622.7531000003],[-8731833.7322000004,5310648.0164000019],[-8731491.5361000001,5310749.9833000004],[-8731394.4655000009,5310777.3775999993],[-8731240.3992999997,5310823.1871000007],[-8731130.6382999998,5310862.6048000008],[-8731041.6940000001,5310900.0442000031],[-8730938.1668999996,5310948.8981999978],[-8730887.9617999997,5310974.3144999966],[-8730830.9661999997,5311006.5797000006],[-8730806.3646000009,5311020.5816000029],[-8730713.7467999998,5311076.7415999994],[-8730643.0589000005,5311122.7048000023],[-8730529.2904000003,5311207.3263999969],[-8730500.2359999996,5311230.6127000004],[-8730472.4060999993,5311253.1379999965],[-8730182.3074999992,5311481.1340999976],[-8730059.2994999997,5311581.1314999983],[-8729903.8975000009,5311703.1997999996],[-8729871.1695000008,5311728.9226000011],[-8729773.2083999999,5311802.2863000035],[-8729771.9838999994,5311803.1996000037],[-8729645.1909999996,5311904.5702999979],[-8729389.2675000001,5312105.945100002],[-8729340.0643000007,5312143.6938000023],[-8729217.5014999993,5312239.740699999],[-8729133.0099999998,5312310.5208000019],[-8729054.0844999999,5312372.1683000028],[-8728857.0490000006,5312522.5596999973],[-8728842.2434999999,5312533.9760999978],[-8728833.0040000007,5312540.9782000035],[-8728831.000 ..... ) CODE is below.  A few thoughts... * Could there be a specific issue with the service retruning JSON results? Charlie did you request JSON when it worked? * Do I need to do more to make the request URL safe? * Does the same page handle token requests from all *.esri.com domains?  Should I be requesting the token from a different ESRI REST GenerateToken page if there are more than one? THANKS!!!!! # Originally based on script from Charlie Frye - ESRI - 12-8-2015 # Import python modules import arcpy import re import copy import os, sys from arcpy import env # For Http calls import httplib, urllib, json #Globals refURL = 'www.arcgis.com' # A function to generate a token given username, password and the adminURL. def getToken(username, password, serverName, serverPort):     # Token URL is typically http://server[:port]/arcgis/admin/generateToken     print "entering getToken"     tokenURL = "/sharing/rest/generateToken"     params = urllib.urlencode({'username': username, 'password': password, 'client': 'referer','referer': refURL, 'expiration':'600','f': 'json'})     headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}   # Connect to URL and post parameters     if serverPort == "":         httpConn = httplib.HTTPSConnection(serverName)     else:         httpConn = httplib.HTTPConnection(serverName, serverPort)     print "request" + tokenURL + "/" + str(params) + "/" + str(headers)     httpConn.request("POST", tokenURL, params, headers)     print "request complete"         # Read response     response = httpConn.getresponse()     if (response.status != 200):         httpConn.close()         FailType = 'Failed getting token; check connection and credentials'         print "Error while fetching tokens from admin URL. Please check the URL and try again."         return     else:         data = response.read()         httpConn.close()         print "response:" + data         # Check that data returned is not an error object         if not assertJsonSuccess(data):             return         # Extract the token from it         token = json.loads(data)         print "TOKEN SUCCESS"         return token['token']     # A function that checks that the input JSON object #  is not an error object. def assertJsonSuccess(data):     obj = json.loads(data)     if 'status' in obj and obj['status'] == "error":         print "Error: JSON object returns an error. " + str(obj)         return False     else:         return True scratchWkspc = arcpy.env.scratchGDB  #Local Processing Environment arcpy.env.overwriteOutput = True serverPort = "" #Get token via function tokenString = (getToken("XXXX","XXXXX","www.arcgis.com",serverPort)).encode('ascii','ignore') print print "GOT TOKEN STRING" print tokenString print # Get the input feature service and make a feature set, which is what GP tools can use as input #i#nputField = "OBJECTID" #where = '1=1' fsURL = "http://landscape1.arcgis.com/arcgis/rest/services/USA_Roads/MapServer/0/query?returnGeometry=true&f=json&inSR=4326&geometryType=esriGeometryEnvelope&geometry=-80,41,-77,44&token="+tokenString print fsURL print # ###### Make the 'Feature Class' from the Service inputSupplyFC = arcpy.FeatureSet() inputSupplyFC.load(fsURL) inputSupplyFC.save(r"L:\temp\greg2.shp") ... View more

Thanks Charlie!  I gave it a try and same behavior?  I must be doing something wrong.---  same error but when I manually apply the token that was generated I am able to successfully see returned JSON using the service request.  Maybe it could be something that you originally had pointed to --- using incorrect URL's for token requests?  And I am indeed using a valid AGO organizational account.  Here is the full code with the what I think are relevant lines highlighted ... Thanks for looking!! #Globals refURL = 'www.arcgis.com' FailType = 'None' geomFineNess = 'Coarse' # 'Coarse','medium','refined' inputFSURL = r"http://landscape1.arcgis.com/arcgis/rest/services/USA_Wetlands/MapServer" #supply # A function to generate a token given username, password and the adminURL. def getToken(username, password, serverName, serverPort):     # Token URL is typically http://server[:port]/arcgis/admin/generateToken     print "entering getToken"     tokenURL = "/sharing/generateToken"     params = urllib.urlencode({'username': username, 'password': password, 'client': 'referer','referer': refURL, 'expiration':'60','f': 'json'})     headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL}   # Connect to URL and post parameters     if serverPort == "":         httpConn = httplib.HTTPSConnection(serverName)     else:         httpConn = httplib.HTTPConnection(serverName, serverPort)     print "request" + tokenURL + "/" + str(params) + "/" + str(headers)     httpConn.request("POST", tokenURL, params, headers)     print "request complete"         # Read response     response = httpConn.getresponse()     if (response.status != 200):         httpConn.close()         FailType = 'Failed getting token; check connection and credentials'         print "Error while fetching tokens from admin URL. Please check the URL and try again."         return     else:         data = response.read()         httpConn.close()         print "response:" + data         # Check that data returned is not an error object         if not assertJsonSuccess(data):             return         # Extract the token from it         token = json.loads(data)         print "TOKEN SUCCESS"         return token['token']     # A function that checks that the input JSON object #  is not an error object. def assertJsonSuccess(data):     obj = json.loads(data)     if 'status' in obj and obj['status'] == "error":         print "Error: JSON object returns an error. " + str(obj)         return False     else:         return True orgAdminURL = r"http://services1.arcgis.com/pf6KDbd8NVL1IUHa/ArcGIS/admin"  #Where the results will be written #                                            ^Organization^ # # Get from local content in the organization of the user specified below in the credentials scratchWkspc = arcpy.env.scratchGDB  #Local Processing Environment arcpy.env.overwriteOutput = True #  ######   Access for Premium Demographic Content  ###### userName = "" # Enter your credentials here passWord = "" demandServerName = "demographics4.arcgis.com"   #Demand Service Server (Expenditures on Alcohol) supplyServerName = "services.arcgis.com"  #Supply Service Server tokenServerName = "arcgis.com" #Portal that the servers authenticate against--Optional serverPort = "" #Get token via function tokenString = (getToken("greg.coniglio","cheyenne1",tokenServerName,serverPort)).encode('ascii','ignore') #                                                            0=Layer ID service = r"/USA_Wetlands/MapServer/0/query" serviceURL = "http://landscape1.arcgis.com/arcgis/rest/services" + service headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain",'referer': refURL} # Get the input feature service and make a feature set, which is what GP tools can use as input inputField = "OBJECTID" where = '1=1' query = "/0/query?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, inputField, tokenString) inputFSURL = r"landscape1.arcgis.com/arcgis/rest/services/USA_Wetlands/MapServer" ... View more

Thanks Charlie!  I'll paste the token code though sounds like there could be bigger issues if you had the same problem. For what it's worth I also just tried it on the "USA_Wetlands" map service, with the same resulting error... Thanks for your attention on this! ... View more

OK made progress!  But stuck... I build the service request like this: http://landscape1.arcgis.com/arcgis/rest/services/USA_Roads/MapServer/2/query?where=1=1&outFields=OBJECTID&returnGeometry=true&f=json&token=0xPcZDxbnmW2ZEjyqErnESo9GSxGyrikM9uorERJ8mffDh010P7qwxoYID4LG58V4O0BwxeQcdrDkFEq8Adu14rYY-TwCPy6GXVGDs2FubbuzUHa1Z4nLXvVwc4-Hy_U and get this error in py Traceback (most recent call last):   File "L:\Tools\EnE_Python_Tools\Land.py", line 108, in <module>     inputSupplyFC.load(fsURL)   File "C:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\arcobjects\arcobjects.py", line 175, in load     return convertArcObjectToPythonObject(self._arc_object.Load(*gp_fixargs(args))) RuntimeError: RecordSetObject: Cannot open table for Load Here's the odd thing.  If I paste that URL into my browser as is I get this JSON: {"error":{"code":498,"message":"Invalid Token","details":[]}} I don't feel like the token is invalid because if I just enter the base URL without the parameters, I get the ArcGIS REST login screen, enter the token string and it's accepted.  Once I do this I can, in the same browser window, enter the same URL request above, and it works, returning the proper JSON!!! ... View more

Thanks Charlie!!! Taking a look already at this... Looking at "USA Roads" in the Landscape group, in this list it only appears to have a MapService not FeatureService http://www.arcgis.com/home/group.html?owner=esri&title=Landscape%20Layers&sortField=title&sortOrder=asc&start=141&q= though proceeding suggest that vector data is available http://proceedings.esri.com/library/userconf/proc14/tech-workshops/tw_453.pdf Would the REST endpoint need to be a feature service for your script to work? ... View more

Hello Charlie!  This group of Landscape Layers is great!  I have a few questions in regards to automation.  I've come across the "Extract Landscape Source Data" tool here available with the Mapping Service: http://www.arcgis.com/home/item.html?id=06f9e2f5f7ee4efcae283ea389bded15 I would love to automate this and develop a python script to iterate through and extract a series of desired layers. I created a simple python script but it fails because credentials (My AGO Organizational account) is not included anywhere: # Import arcpy module import arcpy # Load required toolboxes arcpy.ImportToolbox("http://landscape1.arcgis.com/arcgis/services;Tools/ExtractData") # Local variables: StudyArea = "L:\\temp\\StudyAreas.gdb\\StudyArea" The following error occurs because I believe, of the lack of credentials Traceback (most recent call last):   File "L:\Tools\EnE_Python_Tools\Land.py", line 13, in <module>     arcpy.ImportToolbox("http://landscape1.arcgis.com/arcgis/Tools/ExtractData")   File "C:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\__init__.py", line 124, in ImportToolbox     return import_toolbox(input_file, module_name)   File "C:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\toolbox_code.py", line 434, in import_toolbox     toolbox = gp.createObject("Toolbox", tbxfile)   File "C:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\geoprocessing\_base.py", line 379, in createObject     self._gp.CreateObject(*gp_fixargs(args, True))) IOError: The toolbox file http://landscape1.arcgis.com/arcgis/Tools/ExtractData was not found. 1. How would I in python supply these credentials. 2. I note some more detailed scripts you've offered on this site such as the Supply Demand Script.  That script seems to generate an access token and specfically iterates through items in the Landscape dataset as a Feature Cursor.  Not sure if the latter would work as in this case Extract Data does exactly what I need (depending on the feature limits).  I would want to actually clip to my polygon extent (unless I modified something like the Supply Demand tool to extract features, to get around the limit, then executed the clip locally after the fact. Any thoughts?  Thanks! ... View more

Thanks for the posts!  I am aware of building forms in these ways * Using Domains in our Geodatabase to allow a user to choose from a pick list * You can change minor things such as field labeling etc. using Attribute table aliases This has worked well though we are eager to see more form building capability like you'd have in ArcPad and other products for the same reasons others above stated. I ask cause I've seen sites like this one:  Welcome to iFormBuilder, GIS Professionals! That seem to have really cool forms attached to Collector.... Or is this an iOS SDK app that just looks a lot like Collector?  Thanks!! ... View more

Thanks David!  I wasn't aware of this.  The thing is though we are using Collector on an iPad.  I hope the bug NIM101060 doesn't extend beyond just Androids now? ... View more

You can embed a set of credentials on a token secured service by going through My Content > Add Item. Then you can add the feature service URL and the dialog will update to ask you if you want to save credentials on that layer and not prompt the users of the service or have it prompt always. Then add it to the web map, save it and when you open it up in the app you shouldn't see a prompt if configured that way. Russ Question:  More and more lately, I am getting an error that says "Valid Service Credentials Not Supplied".  I am positive I am entering an appropriate password.  And I know I can get to this from the internet because I can enter the REST URL from the web, ente rthe same password and see the endpoint page.  This same password worked all the time previously, and even earlier in the week. I noticed a few other posts on ESRI forums as of late with the same issue.  Any thoughts? Thanks! ... View more