Latest Contributions by ThomasColson

Yeah...it  might be a while. This is in a dev environment, I'm actually gearing up for another TS case, and validating 10.4 to reproduce the "other" issue I'm having in 10.3.1. ... View more

In Dekstop 10.3.1 File->Add Data->Add Query layer: Use this query: SELECT        dbo.RARE_VIEW_LOCATIONS.LOC_NAME, dbo.RARE_VIEW_LOCATIONS.LOC_NOTES, dbo.RARE_VIEW_LOCATIONS.WATERSHED,                           dbo.RARE_VIEW_LOCATIONS.COUNTY, dbo.RARE_VIEW_LOCATIONS.STATE, dbo.RARE_VIEW_LOCATIONS.LOCATIONDESCRIPTION,                           dbo.RARE_VIEW_LOCATIONS.LOCATIONNOTES, dbo.RARE_VIEW_LOCATIONS.FCSubtype, dbo.RARE_VIEW_LOCATIONS.SPECCOM,                           dbo.RARE_VIEW_LOCATIONS.Location_Type, dbo.RARE_tbl_Events.Start_Date, dbo.RARE_tbl_Events.Observation_NOTES, dbo.RARE_tbl_Events.EditDate,                           dbo.RARE_tbl_Events.Event_ID, dbo.RARE_tlu_Species.taxaGroup, dbo.RARE_tlu_Species.Genus, dbo.RARE_tlu_Species.Species,                           dbo.RARE_tlu_Species.Species_GUID, dbo.RARE_VIEW_LOCATIONS.LOCATION_ID, dbo.RAREPOINT.SHAPE, dbo.RAREPOINT.GIS_LOCATION_ID FROM            dbo.RARE_tlu_Species INNER JOIN                          dbo.RARE_tbl_Events ON dbo.RARE_tlu_Species.Species_ID = dbo.RARE_tbl_Events.Species_ID INNER JOIN                          dbo.RARE_VIEW_LOCATIONS ON dbo.RARE_tbl_Events.Location_ID = dbo.RARE_VIEW_LOCATIONS.LOCATION_ID INNER JOIN                          dbo.RAREPOINT ON dbo.RARE_VIEW_LOCATIONS.OBJECTID = dbo.RAREPOINT.OBJECTID WHERE        (dbo.RARE_tlu_Species.Species_GUID = 'abcd1234') Publish to SVR 10.3.1 as a map service with dynamic workspace enabled. Works. Publish to SVR 10.4 as a map service with dynamic workspace enabled. Works. In ArcMap, instead of a query layer, make a view in SQL and add the view to TOC. Publish to SVR 10.3.1 as a map service with dynamic workspace enabled. Works. Publish to SVR 10.4 as a map service with dynamic workspace enabled. Fails Failed to create the service.: Updating the server connection string for layer RARE.dbo.Spatial_View_Hellbender failed. Attempted connection string was SERVER=AASABBBB;INSTANCE="DSID=4ad33bec-ac5e-43b6-984a-ddfc1f78df98";DBCLIENT=sqlserver;DB_CONNECTION_PROPERTIES=AASABBBB;DATABASE=RARE;VERSION=dbo.DEFAULT;AUTHENTICATION_MODE=OSA. Table name is RARE.DBO.%Spatial_View_Hellbender. Please verify the data exists on the server.. Error executing tool. PublishServiceDefinition Job ID: j6fcd5f1eea7b45829411fed0e4178aaa : ERROR 001487: Failed to update the published service with the server-side data location. Please see the server's log for more details. (less) ERROR 001369: Failed to create the service. Failed to execute (Publish Service Definition). In all cases: SQL is 2014 Enterprise, feature class is Geography Storage type non-versioned, SDE 10.3.1, Windows Auth ONLY, service acccount(s) have correct permission to the tables and views. Since publishing a SQL view works in SVR 10.3.1, and doesn't work in 10.4, I'm leaning towards there's nothing wrong with my data and something wrong with 10.4. Publishing a "regular" table (feature class) works in 10.4 as a map/feature service. ... View more

Love the last line! I'm getting this on a SQL View, which publishes fine on 10.3.1, but fails to drizzle the swizzle on 10.4. ... View more

I'm betting you can connect to your SDE database just fine, but the ArcGIS server cannot. Try installing the ODBC drivers for your database on the ArcGIS Server, e.g., if using SQL, SQL Native Client whatever version. ... View more

How  many services are you pushing? How many instances of each service? What OS? Are you using a web adapter? Https? IWA or GIS Token? Image Service cached? There are about 2 dozen architecture factors that seriously impact performance, but, all that aside, with just two "cores", I'm surprised you can even get ArcGIS Server to start. On a VM, I had to push it to 16 cores to see any stability. What type if hard drive? Is it a shared VHD where dozens of other Vm's are using the same HDD? You should choose clustering only as a last resort, and I really emphasize "last resort" (e.g ESRI tech support told you there is no other way your site is going to work unless in a cluster). Clustering means you have a shared config store, so every other round-robin request for your image service has to go over the network via SMB to the config store. Get rid of the cluster, and add the RAM/CPU resources to the single instance. Or....run two sites, one on each VM, one for vector data, and a dedicated one for image. But by all means, avoid clustering. ... View more

I believe that the solution was to grant local users full control on the ArcGIS, MCS,  and Python install directories, a solution that has solved numerous ArcMap problems. Different organizations have different IT security policies, so a non-adm ArcMap user on an a "normal" computer would likely have a far different experience than a non-adm ArcMap user on tightly-locked down workstation, with everything else (hardware, OS, software installed) being the same. ... View more

Updating Hosted Feature Services in ArcGIS Online | ArcGIS Blog walks you through how to update an AGOL service. In my situation, my SDE is behind what is arguably the most restrictive firewall in the history of the internet, so I use this to refresh my AGOl services nightly, for a dozen-ish feature services I have on there. Note this is one-way, changes on the SDE go to AGOL, I don't have editing enables on the AGOL side (but there are scripts that will do it both ways). You could easily fire this every 15 minutes with Windows Task Scheduler. Another architecture you may want to consider is CartoDB. Cartodb can auto sync with many proprietary formats and render those data with a very lightweight map viewer such as leaflet. I have on particular project where due to the size of the data (terabytes), frequency of update, and cost, CartoDB and a java map viewer was the only way to go. # Import system modules import urllib, urllib2, json import sys, os import requests import arcpy import ConfigParser from xml.etree import ElementTree as ET class AGOLHandler(object):          def __init__(self, username, password, serviceName, folderName):         self.username = username         self.password = password         self.serviceName = serviceName         self.token, self.http = self.getToken(username, password)         self.itemID = self.findItem("Feature Service")         self.SDitemID = self.findItem("Service Definition")         self.folderName = folderName         self.folderID = self.findFolder()            def getToken(self, username, password, exp=60):                referer = "http://www.arcgis.com/"         query_dict = {'username': username,                       'password': password,                       'expiration': str(exp),                       'client': 'referer',                       'referer': referer,                       'f': 'json'}                query_string = urllib.urlencode(query_dict)         url = "https://www.arcgis.com/sharing/rest/generateToken"                token = json.loads(urllib.urlopen(url + "?f=json", query_string).read())                if "token" not in token:             print token['error']             sys.exit()         else:             httpPrefix = "http://www.arcgis.com/sharing/rest"             if token['ssl'] == True:                 httpPrefix = "https://www.arcgis.com/sharing/rest"                            return token['token'], httpPrefix                def findItem(self, findType):         #         # Find the itemID of whats being updated         #               searchURL = self.http + "/search"                query_dict = {'f': 'json',                       'token': self.token,                       'q': "title:\""+ self.serviceName + "\"AND owner:\"" + self.username + "\" AND type:\"" + findType + "\""}                  jsonResponse = sendAGOLReq(searchURL, query_dict)                if jsonResponse['total'] == 0:             print "\nCould not find a service to update. Check the service name in the settings.ini"             sys.exit()         else:             print("found {} : {}").format(findType, jsonResponse['results'][0]["id"])                  return jsonResponse['results'][0]["id"]     def findFolder(self):         #         # Find the ID of the folder containing the service         #         if self.folderName == "None":             return ""                findURL = self.http + "/content/users/{}".format(self.username)         query_dict = {'f': 'json',                       'num': 1,                       'token': self.token}         jsonResponse = sendAGOLReq(findURL, query_dict)         for folder in jsonResponse['folders']:             if folder['title'] == self.folderName:                 return folder['id']                print "\nCould not find the specified folder name provided in the settings.ini"         print "-- If your content is in the root folder, change the folder name to 'None'"         sys.exit()            def urlopen(url, data=None):     # monkey-patch URLOPEN     referer = "http://www.arcgis.com/"     req = urllib2.Request(url)     req.add_header('Referer', referer)     if data:         response = urllib2.urlopen(req, data)     else:         response = urllib2.urlopen(req)     return response def makeSD(MXD, serviceName, tempDir, outputSD, maxRecords):     #     # create a draft SD and modify the properties to overwrite an existing FS     #          arcpy.env.overwriteOutput = True     # All paths are built by joining names to the tempPath     SDdraft = os.path.join(tempDir, "tempdraft.sddraft")     newSDdraft = os.path.join(tempDir, "updatedDraft.sddraft")          arcpy.mapping.CreateMapSDDraft(MXD, SDdraft, serviceName, "MY_HOSTED_SERVICES")        # Read the contents of the original SDDraft into an xml parser     doc = ET.parse(SDdraft)        root_elem = doc.getroot()     if root_elem.tag != "SVCManifest":         raise ValueError("Root tag is incorrect. Is {} a .sddraft file?".format(SDDraft))        # The following 6 code pieces modify the SDDraft from a new MapService     # with caching capabilities to a FeatureService with Query,Create,     # Update,Delete,Uploads,Editing capabilities as well as the ability to set the max     # records on the service.     # The first two lines (commented out) are no longer necessary as the FS     # is now being deleted and re-published, not truly overwritten as is the     # case when publishing from Desktop.     # The last three pieces change Map to Feature Service, disable caching     # and set appropriate capabilities. You can customize the capabilities by     # removing items.     # Note you cannot disable Query from a Feature Service.        #doc.find("./Type").text = "esriServiceDefinitionType_Replacement"     #doc.find("./State").text = "esriSDState_Published"        # Change service type from map service to feature service     for config in doc.findall("./Configurations/SVCConfiguration/TypeName"):         if config.text == "MapServer":             config.text = "FeatureServer"        #Turn off caching     for prop in doc.findall("./Configurations/SVCConfiguration/Definition/" +                                 "ConfigurationProperties/PropertyArray/" +                                 "PropertySetProperty"):         if prop.find("Key").text == 'isCached':             prop.find("Value").text = "false"         if prop.find("Key").text == 'maxRecordCount':             prop.find("Value").text = maxRecords        # Turn on feature access capabilities     for prop in doc.findall("./Configurations/SVCConfiguration/Definition/Info/PropertyArray/PropertySetProperty"):         if prop.find("Key").text == 'WebCapabilities':             prop.find("Value").text = "Query,Extract"     # Add the namespaces which get stripped, back into the .SD       root_elem.attrib["xmlns:typens"] = 'http://www.esri.com/schemas/ArcGIS/10.1'     root_elem.attrib["xmlns:xs"] ='http://www.w3.org/2001/XMLSchema'     # Write the new draft to disk     with open(newSDdraft, 'w') as f:         doc.write(f, 'utf-8')            # Analyze the service     analysis = arcpy.mapping.AnalyzeForSD(newSDdraft)        if analysis['errors'] == {}:         # Stage the service         arcpy.StageService_server(newSDdraft, outputSD)         print "Created {}".format(outputSD)                else:         # If the sddraft analysis contained errors, display them and quit.         print analysis['errors']         sys.exit()          def upload(fileName, tags, description):     #     # Overwrite the SD on AGOL with the new SD.     # This method uses 3rd party module: requests     #        updateURL = agol.http+'/content/users/{}/{}/items/{}/update'.format(agol.username, agol.folderID, agol.SDitemID)            filesUp = {"file": open(fileName, 'rb')}        url = updateURL + "?f=json&token="+agol.token+ \         "&filename="+fileName+ \         "&type=Service Definition"\         "&title="+agol.serviceName+ \         "&tags="+tags+\         "&description="+description            response = requests.post(url, files=filesUp);       itemPartJSON = json.loads(response.text)        if "success" in itemPartJSON:         itemPartID = itemPartJSON['id']         print("updated SD:  {}").format(itemPartID)         return True     else:         print "\n.sd file not uploaded. Check the errors and try again.\n"         print itemPartJSON         sys.exit()             def publish():     #     # Publish the existing SD on AGOL (it will be turned into a Feature Service)     #        publishURL = agol.http+'/content/users/{}/publish'.format(agol.username)        query_dict = {'itemID': agol.SDitemID,               'filetype': 'serviceDefinition',               'overwrite': 'true',               'f': 'json',               'token': agol.token}          jsonResponse = sendAGOLReq(publishURL, query_dict)                print("successfully updated...{}...").format(jsonResponse['services'])        return jsonResponse['services'][0]['serviceItemId']    def enableSharing(newItemID, everyone, orgs, groups):     #     # Share an item with everyone, the organization and/or groups     #     shareURL = agol.http+'/content/users/{}/{}/items/{}/share'.format(agol.username, agol.folderID, newItemID)     if groups == None:         groups = ''        query_dict = {'f': 'json',                   'everyone' : everyone,                   'org' : orgs,                   'groups' : groups,                   'token': agol.token}          jsonResponse = sendAGOLReq(shareURL, query_dict)        print("successfully shared...{}...").format(jsonResponse['itemId'])            def sendAGOLReq(URL, query_dict):     #     # Helper function which takes a URL and a dictionary and sends the request     #        query_string = urllib.urlencode(query_dict)          jsonResponse = urllib.urlopen(URL, urllib.urlencode(query_dict))     jsonOuput = json.loads(jsonResponse.read())        wordTest = ["success", "results", "services", "notSharedWith", "folders"]     if any(word in jsonOuput for word in wordTest):         return jsonOuput       else:         print "\nfailed:"         print jsonOuput         sys.exit()           if __name__ == "__main__":     #     # start     #        print "Starting Feature Service publish process"        # Find and gather settings from the ini file     localPath = sys.path[0]     settingsFile = os.path.join(localPath, "settings.ini")     if os.path.isfile(settingsFile):         config = ConfigParser.ConfigParser()         config.read(settingsFile)     else:         print "INI file not found. \nMake sure a valid 'settings.ini' file exists in the same directory as this script."         sys.exit()        # AGOL Credentials     inputUsername = config.get( 'AGOL', 'USER')     inputPswd = config.get('AGOL', 'PASS')     # FS values     MXD = config.get('FS_INFO', 'MXD')     serviceName = config.get('FS_INFO', 'SERVICENAME')     folderName = config.get('FS_INFO', 'FOLDERNAME')     tags = config.get('FS_INFO', 'TAGS')     description = config.get('FS_INFO', 'DESCRIPTION')     maxRecords = config.get('FS_INFO', 'MAXRECORDS')        # Share FS to: everyone, org, groups     shared = config.get('FS_SHARE', 'SHARE')     everyone = config.get('FS_SHARE', 'EVERYONE')     orgs = config.get('FS_SHARE', 'ORG')     groups = config.get('FS_SHARE', 'GROUPS')  #Groups are by ID. Multiple groups comma separated           # create a temp directory under the script       tempDir = os.path.join(localPath, "tempDir")     if not os.path.isdir(tempDir):         os.mkdir(tempDir)     finalSD = os.path.join(tempDir, serviceName + ".sd")     #initialize AGOLHandler class     agol = AGOLHandler(inputUsername, inputPswd, serviceName, folderName)        # Turn map document into .SD file for uploading     makeSD(MXD, serviceName, tempDir, finalSD, maxRecords)        # overwrite the existing .SD on arcgis.com        if upload(finalSD, tags, description):                # publish the sd which was just uploaded         newItemID = publish()                # share the item         if shared:             enableSharing(newItemID, everyone, orgs, groups)                    print "\nfinished." and here's the ini file: [FS_INFO] SERVICENAME = GRSM_FISH_3_PASS_PT FOLDERNAME = Aqua MXD = C:\PRODUCTION\GRSM_3_PASS\3_PASS_WORKING\Maps\ArcGIS\GRSMAGOL_FISH_3_PASS_PT.mxd TAGS = IANDM, Water, Water Quality, HEM, Aquatic Resources, Surface Water Dynamics, Fishes, Vital Signs, Biodiversity Database, Freshwater Communities, Hydrology Event Management, Inventory And Monitoring, Fish, Hydrology, NHD, ATBI, 3-Pass, Electroshocking, Microfish DESCRIPTION  = These data show the location (only) of Fish 3-Pass Electroshocking study and monitoring sites within the park. Electoshocking utilizes an electrical generator which emits a non-lethal electrical current through the water. This electrical shock is employed to either stun the fish or encourages them to swim towards the electrical field for survey collection. MAXRECORDS = 3000 [FS_SHARE] SHARE = True EVERYONE = true ORG = true GROUPS = blah blah blah [AGOL] USER = sasquatch ... View more

Unlimited. ... View more

80% confident SQL is your bottleneck. Live GIS data from SQL will not stand up to "high-demand", which I define as more than 10 users, unless you're running SQL on a clustered, high-availability instance. Not likely you have 100k$ lying around, so you're best bet is to look at what sort of storage format you're using for the feature classes, index tune where/if you can, and/or move some of the data into views or hourly-updated FGDB's. USGS National Map serves 1000's of requests per hour (minute?), but sourced from FGDB. ... View more

What is the architecture? Is this pulling from SQL, a remote data service, or local FGDB? Any chance you could put the same data on AGOL and run the same query? Is this going through HTTPS? Windows Authentication? I've gotten better performance by siloing the site (putting Web Adapter on it's own server). ... View more

...and as a side-note: Jmeter is difficult, if not impossible, to configure to test against ArcGIS server that has a https cert on the web adapter and the GIS server. You might need to turn those things off, but be sure to test the http requests as they pass through the web adapter: if you test against the GIS server (Apache) but are using a web adapater, you won't see the performance hit that using a WA incurs. ... View more

I've had good luck with Jmeter. Instead of the 1=1 query plan, I'd add a csv element pre-populated with various zoom and extent levels, a wide range, and randomize zoom and extent with each query. Further more, I'd add a pan request. In this manner, you'll more accurately test what a real user does: Access the initial map view, then zoom and pan around. I've found rapid panning is what puts the most load on REST requests. To get really "real", add an identify query here and there. Zoom, pan, then click on something. ... View more