topic Re: Help designing load test in ArcGIS Enterprise Questions

Help designing load test

SteveCole — Tue, 23 Feb 2016 18:55:12 GMT

Our organization is experiencing issues with a service during periods of high demand. The service supports a flood warning system so those periods of high demand are sporadic so I'm looking towards using a load test with Apache JMeter to help us evaluate what we can do to better support our needs. I'm using this blog post and tutorial to help guide me and I've installed JMeter and have a test case set up. Since I'm new to this, I'm not sure if my parameters really mirror the real world so I'm posting to get some more experienced eyes on this.

Anyways, instead of a a map image export like in the tutorial, I'm just performing a simple "1=1" query on our particular map service. Our web map builds our stream gage layer on the fly at page load using this query based on a table in this particular service.

Below are a couple of screenshots from Google Analytics real-time view during our most recent events:

Based on this, I created a test with a thread of 40 users, a ramp up period of 20 seconds, and a loop count of 3. Does this sound reasonable, given the real-world examples shown above? Am I on the right track of way off?

Thanks!

Steve

Re: Help designing load test

ThomasColson — Wed, 24 Feb 2016 14:53:47 GMT

I've had good luck with Jmeter. Instead of the 1=1 query plan, I'd add a csv element pre-populated with various zoom and extent levels, a wide range, and randomize zoom and extent with each query. Further more, I'd add a pan request. In this manner, you'll more accurately test what a real user does: Access the initial map view, then zoom and pan around. I've found rapid panning is what puts the most load on REST requests. To get really "real", add an identify query here and there. Zoom, pan, then click on something.

Re: Help designing load test

ThomasColson — Wed, 24 Feb 2016 15:00:25 GMT

...and as a side-note: Jmeter is difficult, if not impossible, to configure to test against ArcGIS server that has a https cert on the web adapter and the GIS server. You might need to turn those things off, but be sure to test the http requests as they pass through the web adapter: if you test against the GIS server (Apache) but are using a web adapater, you won't see the performance hit that using a WA incurs.

Re: Help designing load test

SteveCole — Wed, 24 Feb 2016 16:20:21 GMT

Thanks for the response, Thomas. Unfortunately the "1=1" query *IS* the issue we're experiencing. If the query returns an empty array of results, the layer of stream gages doesn't get constructed properly (it's done client side) and that's what people are complaining about. The table getting query'd has anywhere from 2800-3400 records in it at any given time. The other two layers in the map are incredibly small (point layer of 16 points & county boundary line features). I ran my JMeter load test 100 times yesterday and only received correct results 51% of the time (results were identical for AGS 10.1 and AGS 10.3).

I'm kinda convinced we don't know what we're doing to support high load services (using the ESRI Web Adapter, no cluster of AGS servers, etc) but, to quote the adage, "that's not my department". As the developer, I just get the blame that the application doesn't work.

Re: Help designing load test

ThomasColson — Wed, 24 Feb 2016 16:33:57 GMT

What is the architecture? Is this pulling from SQL, a remote data service, or local FGDB? Any chance you could put the same data on AGOL and run the same query? Is this going through HTTPS? Windows Authentication? I've gotten better performance by siloing the site (putting Web Adapter on it's own server).

Re: Help designing load test

SteveCole — Wed, 24 Feb 2016 16:41:08 GMT

I'm getting outside my knowledge since I'm not responsible for the backend stuff but it's in our organizations Enterprise SDE in SqlServer 2012.

It's not going through HTTPS.

We do use Windows Authentication

I suspect that the web adapter is NOT silo'd. I'd bet money we just ran with the ESRI installation guide when setting things up.

Re: Help designing load test

ThomasColson — Wed, 24 Feb 2016 16:54:02 GMT

80% confident SQL is your bottleneck. Live GIS data from SQL will not stand up to "high-demand", which I define as more than 10 users, unless you're running SQL on a clustered, high-availability instance. Not likely you have 100k$ lying around, so you're best bet is to look at what sort of storage format you're using for the feature classes, index tune where/if you can, and/or move some of the data into views or hourly-updated FGDB's. USGS National Map serves 1000's of requests per hour (minute?), but sourced from FGDB.

Re: Help designing load test

SteveCole — Wed, 24 Feb 2016 17:49:45 GMT

That's interesting. The stream gage data is stored natively in an MS Access database (old gage software which is on its last legs officially and unofficially). Due to some permission conditions the gage software has with its corresponding database, we've set up a system where we use a python script to transfer data from the MS Access db into an SDE table. I suppose we could alter that process and have it dump the information into a FGDB and publish off of that.

Re: Help designing load test

SteveCole — Wed, 24 Feb 2016 18:08:37 GMT

I forgot to add a key piece of information. The data transfer from MS Access to SDE occurs every 10,15 minutes. During the transfer the table in SDE is purged and then the Access records are inserted. We've been getting erratic run times on the script. Initially, it was completing in 4 seconds but somehow it grows to 1-2 minutes. During that time, of course, the SDE table has no records and results in no data for folks who happen to load/re-load the page at that time.

Re: Help designing load test

MichaelVolz — Wed, 24 Feb 2016 18:44:45 GMT

I do not think you would want to use a file gdb for your stream gauge data. My organization tried to setup a road closure mapservice that would update every 5 minutes to the registered file gdb from another database that was capturing the data from a MS Access application. The mapservice was OK for some time, but the constant refreshing of data (OBJECTIDs were climbing rapidly) caused a severe degradation of performance. In the end someone scripted the data to be updated on AGOL every 5 minutes which resolved the performance issue.

As such, maybe you could make use of AGOL's scalability to solve your performance issue as an alternative to your current setup.

Re: Help designing load test

SteveCole — Wed, 24 Feb 2016 19:05:27 GMT

That's excellent feedback, Michael, thank you. It's pretty disconcerting that, as customers, you spend an arm & a leg for enterprise level products and ESRI still manages to find a way to force you to spend more money on something like AGOL.

I like dealing with web mapping less and less each day. When's Librarian coming back into vogue?..

Re: Help designing load test

SteveCole — Wed, 24 Feb 2016 20:29:15 GMT

Michael Volz Your option to post to AGOL might be something our organization needs to pursue. Would you be willing to share that script (stripped of specific server/account references, of course)? We could take the discussion off the board, if you're so inclined.

Steve

Re: Help designing load test

MichaelVolz — Wed, 24 Feb 2016 20:36:04 GMT

I would be inclined to take the discussion off the board because it is not my script that works with AGOL. I tried to script the update process for my organization's ArcGIS Server, but I was never able to overcome the degradation of the mapservice. The person in my organization who created the Access application was the person who created the script to update the data in AGOL, so I would need to talk to that person.

In the meantime I would search ESRI documentation and the geonet to find code (I would recommend python) to upload the stream gauge data to AGOL every 15 minutes. I think this is a pretty common workflow for endusers who have AGOL but do not have ArcGIS Server.

Re: Help designing load test

SteveCole — Wed, 24 Feb 2016 21:49:33 GMT

Understandable.

I don't do much Python but I did develop our script that transfers the MS Access records over into an SDE table. I think I'm solid with connecting with Access and the records in it but extremely shaky with AGOL. We, as an organization, haven't embraced AGOL at all so any scripting to interact with AGOL is completely foreign to me. It just sounds like the project you describe is real similar to the process we use for our Gage application.

Re: Help designing load test

ThomasColson — Fri, 10 Dec 2021 20:41:40 GMT

Updating Hosted Feature Services in ArcGIS Online | ArcGIS Blog walks you through how to update an AGOL service. In my situation, my SDE is behind what is arguably the most restrictive firewall in the history of the internet, so I use this to refresh my AGOl services nightly, for a dozen-ish feature services I have on there. Note this is one-way, changes on the SDE go to AGOL, I don't have editing enables on the AGOL side (but there are scripts that will do it both ways). You could easily fire this every 15 minutes with Windows Task Scheduler.

Another architecture you may want to consider is CartoDB. Cartodb can auto sync with many proprietary formats and render those data with a very lightweight map viewer such as leaflet. I have on particular project where due to the size of the data (terabytes), frequency of update, and cost, CartoDB and a java map viewer was the only way to go.

# Import system modules
import urllib, urllib2, json
import sys, os
import requests
import arcpy
import ConfigParser
from xml.etree import ElementTree as ET


class AGOLHandler(object):   
   
    def __init__(self, username, password, serviceName, folderName):
        self.username = username
        self.password = password
        self.serviceName = serviceName
        self.token, self.http = self.getToken(username, password)
        self.itemID = self.findItem("Feature Service")
        self.SDitemID = self.findItem("Service Definition")
        self.folderName = folderName
        self.folderID = self.findFolder()
       
    def getToken(self, username, password, exp=60):
       
        referer = "http://www.arcgis.com/"
        query_dict = {'username': username,
                      'password': password,
                      'expiration': str(exp),
                      'client': 'referer',
                      'referer': referer,
                      'f': 'json'} 
       
        query_string = urllib.urlencode(query_dict)
        url = "https://www.arcgis.com/sharing/rest/generateToken"
       
        token = json.loads(urllib.urlopen(url + "?f=json", query_string).read())
       
        if "token" not in token:
            print token['error']
            sys.exit()
        else:
            httpPrefix = "http://www.arcgis.com/sharing/rest"
            if token['ssl'] == True:
                httpPrefix = "https://www.arcgis.com/sharing/rest"
               
            return token['token'], httpPrefix
           
    def findItem(self, findType):
        #
        # Find the itemID of whats being updated
        #       
        searchURL = self.http + "/search"
       
        query_dict = {'f': 'json',
                      'token': self.token,
                      'q': "title:\""+ self.serviceName + "\"AND owner:\"" + self.username + "\" AND type:\"" + findType + "\""}   
       
        jsonResponse = sendAGOLReq(searchURL, query_dict)
       
        if jsonResponse['total'] == 0:
            print "\nCould not find a service to update. Check the service name in the settings.ini"
            sys.exit()
        else:
            print("found {} : {}").format(findType, jsonResponse['results'][0]["id"])   
       
        return jsonResponse['results'][0]["id"]


    def findFolder(self):
        #
        # Find the ID of the folder containing the service
        #


        if self.folderName == "None":
            return ""
       
        findURL = self.http + "/content/users/{}".format(self.username)


        query_dict = {'f': 'json',
                      'num': 1,
                      'token': self.token}


        jsonResponse = sendAGOLReq(findURL, query_dict)


        for folder in jsonResponse['folders']:
            if folder['title'] == self.folderName:
                return folder['id']
       
        print "\nCould not find the specified folder name provided in the settings.ini"
        print "-- If your content is in the root folder, change the folder name to 'None'"
        sys.exit()
           


def urlopen(url, data=None):
    # monkey-patch URLOPEN
    referer = "http://www.arcgis.com/"
    req = urllib2.Request(url)
    req.add_header('Referer', referer)


    if data:
        response = urllib2.urlopen(req, data)
    else:
        response = urllib2.urlopen(req)


    return response




def makeSD(MXD, serviceName, tempDir, outputSD, maxRecords):
    #
    # create a draft SD and modify the properties to overwrite an existing FS
    #   
   
    arcpy.env.overwriteOutput = True
    # All paths are built by joining names to the tempPath
    SDdraft = os.path.join(tempDir, "tempdraft.sddraft")
    newSDdraft = os.path.join(tempDir, "updatedDraft.sddraft")   
   
    arcpy.mapping.CreateMapSDDraft(MXD, SDdraft, serviceName, "MY_HOSTED_SERVICES")
   
    # Read the contents of the original SDDraft into an xml parser
    doc = ET.parse(SDdraft) 
   
    root_elem = doc.getroot()
    if root_elem.tag != "SVCManifest":
        raise ValueError("Root tag is incorrect. Is {} a .sddraft file?".format(SDDraft))
   
    # The following 6 code pieces modify the SDDraft from a new MapService
    # with caching capabilities to a FeatureService with Query,Create,
    # Update,Delete,Uploads,Editing capabilities as well as the ability to set the max
    # records on the service.
    # The first two lines (commented out) are no longer necessary as the FS
    # is now being deleted and re-published, not truly overwritten as is the
    # case when publishing from Desktop.
    # The last three pieces change Map to Feature Service, disable caching
    # and set appropriate capabilities. You can customize the capabilities by
    # removing items.
    # Note you cannot disable Query from a Feature Service.
   
    #doc.find("./Type").text = "esriServiceDefinitionType_Replacement"
    #doc.find("./State").text = "esriSDState_Published"
   
    # Change service type from map service to feature service
    for config in doc.findall("./Configurations/SVCConfiguration/TypeName"):
        if config.text == "MapServer":
            config.text = "FeatureServer"

   
    #Turn off caching
    for prop in doc.findall("./Configurations/SVCConfiguration/Definition/" +
                                "ConfigurationProperties/PropertyArray/" +
                                "PropertySetProperty"):
        if prop.find("Key").text == 'isCached':
            prop.find("Value").text = "false"
        if prop.find("Key").text == 'maxRecordCount':
            prop.find("Value").text = maxRecords
   
    # Turn on feature access capabilities
    for prop in doc.findall("./Configurations/SVCConfiguration/Definition/Info/PropertyArray/PropertySetProperty"):
        if prop.find("Key").text == 'WebCapabilities':
            prop.find("Value").text = "Query,Extract"


    # Add the namespaces which get stripped, back into the .SD   
    root_elem.attrib["xmlns:typens"] = 'http://www.esri.com/schemas/ArcGIS/10.1'
    root_elem.attrib["xmlns:xs"] ='http://www.w3.org/2001/XMLSchema'


    # Write the new draft to disk
    with open(newSDdraft, 'w') as f:
        doc.write(f, 'utf-8')
       
    # Analyze the service
    analysis = arcpy.mapping.AnalyzeForSD(newSDdraft)
   
    if analysis['errors'] == {}:
        # Stage the service
        arcpy.StageService_server(newSDdraft, outputSD)
        print "Created {}".format(outputSD)
           
    else:
        # If the sddraft analysis contained errors, display them and quit.
        print analysis['errors']
        sys.exit()

         
def upload(fileName, tags, description):
    #
    # Overwrite the SD on AGOL with the new SD.
    # This method uses 3rd party module: requests
    #
   
    updateURL = agol.http+'/content/users/{}/{}/items/{}/update'.format(agol.username, agol.folderID, agol.SDitemID)
       
    filesUp = {"file": open(fileName, 'rb')}
   
    url = updateURL + "?f=json&token="+agol.token+ \
        "&filename="+fileName+ \
        "&type=Service Definition"\
        "&title="+agol.serviceName+ \
        "&tags="+tags+\
        "&description="+description
       
    response = requests.post(url, files=filesUp);   
    itemPartJSON = json.loads(response.text)
   
    if "success" in itemPartJSON:
        itemPartID = itemPartJSON['id']
        print("updated SD:  {}").format(itemPartID)
        return True
    else:
        print "\n.sd file not uploaded. Check the errors and try again.\n" 
        print itemPartJSON
        sys.exit()       
   
   
def publish():
    #
    # Publish the existing SD on AGOL (it will be turned into a Feature Service)
    #
   
    publishURL = agol.http+'/content/users/{}/publish'.format(agol.username)
   
    query_dict = {'itemID': agol.SDitemID,
              'filetype': 'serviceDefinition',
              'overwrite': 'true',
              'f': 'json',
              'token': agol.token}   
   
    jsonResponse = sendAGOLReq(publishURL, query_dict)
           
    print("successfully updated...{}...").format(jsonResponse['services'])
   
    return jsonResponse['services'][0]['serviceItemId']
   


def enableSharing(newItemID, everyone, orgs, groups):
    #
    # Share an item with everyone, the organization and/or groups
    #


    shareURL = agol.http+'/content/users/{}/{}/items/{}/share'.format(agol.username, agol.folderID, newItemID)


    if groups == None:
        groups = ''
   
    query_dict = {'f': 'json',
                  'everyone' : everyone,
                  'org' : orgs,
                  'groups' : groups,
                  'token': agol.token}   
   
    jsonResponse = sendAGOLReq(shareURL, query_dict)
   
    print("successfully shared...{}...").format(jsonResponse['itemId'])   
   
   
   
def sendAGOLReq(URL, query_dict):
    #
    # Helper function which takes a URL and a dictionary and sends the request
    #
   
    query_string = urllib.urlencode(query_dict)   
   
    jsonResponse = urllib.urlopen(URL, urllib.urlencode(query_dict))
    jsonOuput = json.loads(jsonResponse.read())
   
    wordTest = ["success", "results", "services", "notSharedWith", "folders"]
    if any(word in jsonOuput for word in wordTest):
        return jsonOuput   
    else:
        print "\nfailed:"
        print jsonOuput
        sys.exit()
       
   
if __name__ == "__main__":
    #
    # start
    #
   
    print "Starting Feature Service publish process"
   
    # Find and gather settings from the ini file
    localPath = sys.path[0]
    settingsFile = os.path.join(localPath, "settings.ini")


    if os.path.isfile(settingsFile):
        config = ConfigParser.ConfigParser()
        config.read(settingsFile)
    else:
        print "INI file not found. \nMake sure a valid 'settings.ini' file exists in the same directory as this script."
        sys.exit()
   
    # AGOL Credentials
    inputUsername = config.get( 'AGOL', 'USER')
    inputPswd = config.get('AGOL', 'PASS')


    # FS values
    MXD = config.get('FS_INFO', 'MXD')
    serviceName = config.get('FS_INFO', 'SERVICENAME') 
    folderName = config.get('FS_INFO', 'FOLDERNAME')
    tags = config.get('FS_INFO', 'TAGS')
    description = config.get('FS_INFO', 'DESCRIPTION')
    maxRecords = config.get('FS_INFO', 'MAXRECORDS')
   
    # Share FS to: everyone, org, groups
    shared = config.get('FS_SHARE', 'SHARE')
    everyone = config.get('FS_SHARE', 'EVERYONE')
    orgs = config.get('FS_SHARE', 'ORG')
    groups = config.get('FS_SHARE', 'GROUPS')  #Groups are by ID. Multiple groups comma separated
   
   
    # create a temp directory under the script   
    tempDir = os.path.join(localPath, "tempDir")
    if not os.path.isdir(tempDir):
        os.mkdir(tempDir) 
    finalSD = os.path.join(tempDir, serviceName + ".sd") 


    #initialize AGOLHandler class
    agol = AGOLHandler(inputUsername, inputPswd, serviceName, folderName)
   
    # Turn map document into .SD file for uploading
    makeSD(MXD, serviceName, tempDir, finalSD, maxRecords)
   
    # overwrite the existing .SD on arcgis.com
   
    if upload(finalSD, tags, description):
       
        # publish the sd which was just uploaded
        newItemID = publish()
       
        # share the item
        if shared:
            enableSharing(newItemID, everyone, orgs, groups)
           
        print "\nfinished."

and here's the ini file:

[FS_INFO]
SERVICENAME = GRSM_FISH_3_PASS_PT
FOLDERNAME = Aqua
MXD = C:\PRODUCTION\GRSM_3_PASS\3_PASS_WORKING\Maps\ArcGIS\GRSMAGOL_FISH_3_PASS_PT.mxd
TAGS = IANDM, Water, Water Quality, HEM, Aquatic Resources, Surface Water Dynamics, Fishes, Vital Signs, Biodiversity Database, Freshwater Communities, Hydrology Event Management, Inventory And Monitoring, Fish, Hydrology, NHD, ATBI, 3-Pass, Electroshocking, Microfish
DESCRIPTION  = These data show the location (only) of Fish 3-Pass Electroshocking study and monitoring sites within the park. Electoshocking utilizes an electrical generator which emits a non-lethal electrical current through the water. This electrical shock is employed to either stun the fish or encourages them to swim towards the electrical field for survey collection.
MAXRECORDS = 3000


[FS_SHARE]
SHARE = True
EVERYONE = true
ORG = true
GROUPS = blah blah blah


[AGOL]
USER = sasquatch

Re: Help designing load test

MichaelVolz — Thu, 03 Mar 2016 17:15:07 GMT

Steve:

In another thread you said "Sorry, no, I haven't. Part of me isn't happy that I'm doing someone else's job for them and the other part of me got busy with other things. Plus it's not currently flooding."

Such is the life of a GIS worker in an enterprise environment.

Re: Help designing load test

SteveCole — Thu, 03 Mar 2016 17:24:06 GMT

Our organization has had a "wild west" type environment for many years. It's great if you have initiative and want to try things but it's not so great in times like this. I've learned a lot but I don't like the possibility of getting my reputation sullied by things that are not in my control.