topic Re: A method to check how our services are healthy in ArcGIS Enterprise Questions

A method to check how our services are healthy

rawansaleh1 — Sun, 09 Jul 2017 06:33:41 GMT

I couldn’t figure out if there is a method to check how our services are healthy.

For example, I need a tool to send a notification for me if one of my services are stopped suddenly or any failure happened to my services.

I need any method to make a periodic check on my services.

Any suggestion for that?

Best,

Rawan

Re: A method to check how our services are healthy

DerekLaw — Sun, 09 Jul 2017 14:26:52 GMT

Hi Rawan,

I would start by checking the ArcGIS Server logs,

About specifying server log settings—ArcGIS Server Administration (Windows) | ArcGIS Enterprise

Hope this helps,

Re: A method to check how our services are healthy

XanderBakker — Sun, 09 Jul 2017 15:56:48 GMT

Additionally, there is System Monitor offered through professional services and be sure to check the UC plenary tomorrow for news on this topic.

Re: A method to check how our services are healthy

rawansaleh1 — Sun, 09 Jul 2017 16:51:11 GMT

Dear Derek,

Thanks for your answer. I always check my argis server logs to ensure that the system is fine.

But what I exactly looking for here that if my system get failure or one of my services have a problem I need an a way to get me informed about that.

Is that possible?

Best,

Rawan

Re: A method to check how our services are healthy

rawansaleh1 — Sun, 09 Jul 2017 16:53:03 GMT

Dear Xander,

thanks for your input. i will follow up on that.

Best,

Rawan

Re: A method to check how our services are healthy

mfcallahan — Sun, 09 Jul 2017 17:44:22 GMT

I haven't done this specifically for my ArcGIS Server instances, but when I want to monitor services I have running, I like to use Windows Task Scheduler. I usually create a small console app or script that will make a test query to a service, and if there is not a response or the response is not as unexpected, it will send out an alert email. So for example I have an application running on my web server that depends on a few external APIs and database connections to run. I have a scheduled task running every 5 minutes (on a different machine than the server) and it will make sure each of the APIs is responding and the database connections are OK. If everything is OK, the task will end without notification. If anything reported an error, I send an alert email out to the dev team so we are aware of the poor health status of the application right away and can take immediate action.

Re: A method to check how our services are healthy

rawansaleh1 — Mon, 10 Jul 2017 10:12:26 GMT

Dear Matthew Callahan,

Thanks for your input. May you please share this script with me to examine it.

Best,

Rawan

Re: A method to check how our services are healthy

JoshuaBixby — Mon, 10 Jul 2017 13:52:17 GMT

System Monitor is not part of ArcGIS, per se. As Xander points out, you get access to it through Esri Professional Services since they are the ones that develop it. For System Monitor, you pay a consulting fee to have them give you the software and help get it setup and working. If you are willing to pay for a tool, I suggest you do an internet search on tools for monitoring ArcGIS Server and GIS services. Although there are not a lot of tools out there, there are at least a couple nice packages I have seen.

If you search GeoNet, you will find multiple cases where users ask/suggest that Esri package System Monitor with ArcGIS Server. Given the cost of ArcGIS Server products, I think providing a robust monitoring tool with it isn't too much to ask, but Esri obviously thinks differently.

Re: A method to check how our services are healthy

XanderBakker — Mon, 10 Jul 2017 14:24:37 GMT

I will share link with more info after the announcement at the plenary.

Edit: Not much information available yet, but have a look at: http://go.esri.com/monitor

Re: A method to check how our services are healthy

JonathanQuinn — Mon, 10 Jul 2017 18:47:48 GMT

You can write a python script to loop through all services and query them. By querying, you're making sure that both the service and data are accessible. If the query fails or doesn't return any features, then you can be alerted that there's a problem with the service. You should also query the logs to determine if any services are reporting errors.

Re: A method to check how our services are healthy

JoshuaBixby — Mon, 10 Jul 2017 23:03:47 GMT

There is an ArcGIS Monitor demo site floating around out there. I am not sure I should be sharing it, but Google doesn't mind sharing.

When I logged into the ArcGIS Monitor demo site, the version number is 3.0.250, which makes me think this is simply a rebranding of System Monitor than a new product. The latest version of System Monitor is 3.x.

The ArcGIS Monitor splash page on esri.com doesn't say "Professional Services," but I am guessing that is who reaches out to you after you fill out the contact information.

Re: A method to check how our services are healthy

RebeccaStrauch__GISP — Mon, 10 Jul 2017 23:13:16 GMT

Interesting page....

Oversee Your Enterprise GIS Usage and Performance
At Esri, we want you to get the most out of your investment in GIS and IT infrastructure. Soon we will offer ArcGIS Monitor, a tool uniquely tailored to audit the health of your ArcGIS implementations. ArcGIS Monitor will show you insightful information about your system usage and performance, while ensuring that Esri can support you throughout the lifecycle of your GIS. Sign up to get the latest news on how Esri can help you improve your system operation and reduce administration costs.

But on hitting submit, I get

Please correct the errors below:

with no errors listed. Maybe it didn't like that I selected "expert" as my "level of Experience using GIS" Which I thought was a strange question anyway.

Re: A method to check how our services are healthy

XanderBakker — Tue, 11 Jul 2017 00:27:28 GMT

That's odd. I did not have that option when I submitted the form... maybe only those worthy of being an expert are presented with that option...

Re: A method to check how our services are healthy

XanderBakker — Sat, 11 Dec 2021 13:35:27 GMT

Here is an example script that I have used in the past to simplify the information in the log files. It will only register 1 record for an error that occurred several times during the same day and write out the start and end date time and the number of times the error occurred. It is far from being perfect, but might be a start.

def main():
    import os
    import datetime
    import codecs

    # Define date range for service logfile to analyze
    min_date = datetime.datetime(2017, 7, 1).date() 
    max_date = datetime.datetime(2017, 7, 10).date() 
    today = datetime.date.today()

    log_folder = r'C:\GeoNet\ServiceHealth\txt'
    lst_servers = ['dev_server' , 'test_server', 'prod_server']  # fill in you server names
    atts_service = ['time', 'type', 'code', 'target', 'machine', 'process', 'thread']

    # what types of mesages do you want to process
    # lst_errores = ['DEBUG', 'FINE', 'INFO', 'SEVERE', 'VERBOSE', 'WARNING']
    lst_errores = [u'SEVERE', u'WARNING']

    # process each server
    for svr in lst_servers:
        path = r'\\{0}\arcgisserver\logs\{1}\services'.format(svr, svr.upper())

        # create a filename for each server
        file_name = 'service_errors_{0}_{1}_{2}.txt'.format(svr, min_date, max_date)
        log_file = os.path.join(log_folder, file_name)
        print log_file

        dct = {}
        # process the lofile for each server
        for (path, dirs, files) in os.walk(path):
            for fname in files:
                if fname.endswith('.log'):
                    filepath = os.path.join(path, fname)
                    dt = datetime.datetime.fromtimestamp(os.stat(filepath).st_mtime).date()
                    # only process those files with a datestamo with the indicated date range
                    if all([dt >= min_date, dt <= max_date]):
                        print " - ", filepath, ' (', round(os.path.getsize(filepath)/1024, 0), ' KB)'
                        with open(filepath, 'r') as f:
                            for line in f:
                                line = line.replace('\n', '')
                                if '<Msg' in line:
                                    err = line[line.find('>')+1:line.find('<',line.find('>'))]
                                    lst = []
                                    for att in atts_service:
                                        lst.append(getAttrService(line, att))
                                    lst.append(err)
                                    daytime = lst[0]
                                    err_dt = getDateTime(daytime)
                                    day = daytime[:10]
                                    ids = svr + ' ' + day + ' ' + err

                                    dct = updateInfo(dct, ids, err_dt, lst)

        # write the file
        with codecs.open(log_file, 'w', encoding='utf-8') as f:
            # header
            header = ['first', 'last', 'count', 'type', 'code', 'target', 'machine',  'process', 'thread', 'error']
            f.write('\t'.join(header)+'\r\n')
            for ids, info in dct.items():
                lst = [to_unicode_or_burst(info[1]), to_unicode_or_burst(info[2]), to_unicode_or_burst(info[3])]
                for a in info[0][1:]:
                    lst.append(to_unicode_or_burst(a))
                if lst[3] in lst_errores:
                    f.write(u'\t'.join([unicode(a) for a in lst])+'\r\n')


def to_unicode_or_burst(obj, encoding='utf-8'):
    if isinstance(obj, basestring):
        if not isinstance(obj, unicode):
            try:
                obj = unicode(obj, encoding)
            except Exception as e:
                print e
    return obj

def getDateTime(daytime):
    from datetime import datetime
    lst = daytime.split(',')
    dts = lst[0]
    dts = dts.replace('T', ' ')
    return datetime.strptime(dts, '%Y-%m-%d %H:%M:%S')

def updateInfo(dct, ids, err_dt, lst):
    if ids in dct:
        info = dct[ids]
        first_dt = info[1]
        last_dt = info[2]
        cnt = info[3]
        if err_dt < first_dt: first_dt = err_dt
        if err_dt > last_dt: last_dt = err_dt
        cnt += 1
        info[1] = first_dt
        info[2] = last_dt
        info[3] = cnt
        dct[ids] = info
    else:
        info = [lst, err_dt, err_dt, 1]
        dct[ids] = info
    return dct

def getAttrService(line, att):
    s = "{0}='".format(att)
    i1 = line.find(s)
    i2 = line.find("'", i1+len(s))
    return line[i1+len(s):i2]

def getAttrServer(line, att):
    s = '{0}="'.format(att)
    i1 = line.find(s)
    i2 = line.find('"', i1+len(s))
    return line[i1+len(s):i2]


if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: A method to check how our services are healthy

XanderBakker — Sat, 11 Dec 2021 13:35:31 GMT

And here an example of crawling through the service folders. Haven't used it for a while...:

import json
import urllib2

def main():
    import codecs
    import sys
    import datetime
    import os

    # edit these settings
    log_folder = r'C:\GeoNet\ServiceHealth\txt'
    dct_srvr = {"DEV 10.4.1": "https://myDEVservername",
                "TEST 10.4.1": "https://myTESTservername",
                "PROD 10.4.1": "https://myPRODservername"}

    for a, s in dct_srvr.items():
        baseUrl = "{0}/arcgis/rest/services".format(s)
        svr = s.replace('https://', '')

        file_name = 'REST_services_{0}_{1}.txt'.format(svr, datetime.date.today())
        log_file = os.path.join(log_folder, file_name)
        print log_file

        with codecs.open(log_file, 'w', encoding='utf-8') as f:
            f.write(u"DateTime\tEnvironment\tServiceName\tServiceType\tConnection\tLayername\tRESTcode\r\n")
            getCatalog(baseUrl, f, a)



def getCatalog(baseUrl, f, ambiente):
    try:
        catalog = json.load(urllib2.urlopen(baseUrl + "/" + "?f=json"))
        if "error" in catalog:
            return
        services = catalog['services']
        for service in services:
            response = json.load(urllib2.urlopen("{0}/{1}/{2}?f=json".format(baseUrl, service['name'], service['type'])))
            URL = "{0}/{1}/{2}".format(baseUrl, service['name'], service['type'])
            writeData(f, to_unicode_or_burst(URL), to_unicode_or_burst(ambiente), to_unicode_or_burst(service['name']), to_unicode_or_burst(service['type']), u'ERROR' if "error" in response else u'SUCCESS')

        folders = catalog['folders']
        for folderName in folders:
            try:
                catalog = json.load(urllib2.urlopen(baseUrl + "/" + folderName + "?f=json"))
                if "error" in catalog:
                    return
                services = catalog['services']
                for service in services:
                    response = json.load(urllib2.urlopen("{0}/{1}/{2}?f=json".format(baseUrl, service['name'], service['type'])))
                    URL = "{0}/{1}/{2}".format(baseUrl, service['name'], service['type'])
                    writeData(f, to_unicode_or_burst(URL), to_unicode_or_burst(ambiente), to_unicode_or_burst(service['name']), to_unicode_or_burst(service['type']), u'ERROR' if "error" in response else u'SUCCESS')

            except Exception as e:
                f.write(u"{0}\t{1}\t{2}\t{3}\t{4}\n".format(getDateTime(), to_unicode_or_burst(ambiente), to_unicode_or_burst(e), to_unicode_or_burst(folderName), u"ERROR"))
    except Exception as e:
        f.write(u"{0}\t{1}\t{2}\t{3}\t{4}\n".format(getDateTime(), to_unicode_or_burst(ambiente), to_unicode_or_burst(e), u"ROOT", u"ERROR"))


def to_unicode_or_burst(obj, encoding='utf-8'):
    if isinstance(obj, basestring):
        if not isinstance(obj, unicode):
            try:
                obj = unicode(obj, encoding)
            except Exception as e:
                print e
    return obj

def getDateTime():
    from datetime import datetime
    return datetime.now().strftime('%Y-%m-%d %H:%M:%S')

def getServiceProperties(URL, prop):
    fURL = URL + "?f=json"
    openURL = urllib2.urlopen(fURL, '').read()
    outJson = json.loads(openURL)
    if prop in outJson:
        return outJson[prop]
    else:
        return ""

def writeData(f, URL, ambiente, service_name, service_type, status):
    prop = "layers"
    layers = getServiceProperties(URL, prop)
    for lyr in layers:
        try:
            rest_code = lyr["id"]
        except:
            rest_code = "-1"
        try:
            layer_name = lyr["name"]
        except:
            layer_name = "utf8-error"

        f.write(u"{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\r\n".format(getDateTime(), ambiente, service_name, service_type, status, layer_name, rest_code))


if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: A method to check how our services are healthy

RebeccaStrauch__GISP — Tue, 11 Jul 2017 01:25:39 GMT

Maybe it is still only available internally. The other choices were "Novice user in GIS" and "Some Background in GIS". strange question....strange choices. Looking forward to seeing it. I'm heading down to the UC on Weds so maybe there will be a demo available by then.

Re: A method to check how our services are healthy

Anonymous User — Tue, 11 Jul 2017 08:20:31 GMT

Hello Rawan,

I have an script which will fulfill the necessary task for you.

Thanks,

Avinash Patel

Re: A method to check how our services are healthy

rawansaleh1 — Wed, 12 Jul 2017 10:57:34 GMT

Dear Xander, Rebecca, Joshua

Thanks for sharing the information about professional services.

Re: A method to check how our services are healthy

rawansaleh1 — Wed, 12 Jul 2017 11:02:43 GMT

Thanks Jonthan,

We hope that ESRI provide us such a tool (not out from the box) to help the users in tracking their system.

Best,

Rawan

Re: A method to check how our services are healthy

rawansaleh1 — Wed, 12 Jul 2017 11:04:44 GMT

Thanks Xander for sharing the above python, that’s will be helpful

I will try it to see if this works with me or not.

Best,

Rawan