how default python logging is handled in arcgis ?

[SchoppMatthieu]

Hello I have some difficulties with log files

The problem is that running it from ArcGIS Toolbox interface doesn't give me the same result than running it from python IDLE.

In python IDLE, a new logFile is created each time.
When executed from ArcGIS toolbox interface, the logFile is only created once and kept in memory. Each time you run the script logs are written in the first logfile (created the first time the script is executed) rather than creating a new one.

Please have a look at the sample script attached in my second post

[SchoppMatthieu]

I have attached the sample of the script with its toolbox.

That would be great if someone could have a quick look.

You should notice that running it from python IDLE doesn't give the same result than running it from the toolbox (graphic interface).
Running it from the toolbox graphic interface doesn't recreate the log file.

I'm probably doing something wrong but I can see what it is.

Thanks

[StacyRendall1]

This script worked perfectly for me, but the scratch directory is different when running from within ArcMap or ArcCatalog. If you look closely your run from IDLE places the file here:
[INDENT]C:\Users\ADMINI~1\AppData\Local\Temp\2\scratch\[/INDENT]

while running from Arc it is placed here (emphasis placed on difference):
[INDENT]C:\Users\ADMINI~1\AppData\Local\Temp\2\arc4BFD\scratch\[/INDENT]


For me there was even a difference between running from ArcMap and running from ArcCatalog.

[SchoppMatthieu]

Thank you for having a look at it.
This is surprising that it is not working for me.

This is the problem I can see :

There are two 'print' 'addMessage' in the script

- The name and directory of the logfile I want to create
- The actual log handlers base file

arcpy.AddMessage("my log file name    : "+str(outputFileLog))
print ("my log file name    : "+str(outputFileLog))

and

    for handler in logger.handlers:    
        print ("my log file handler : "+handler.baseFilename)

When I run this from python IDLE my two prints appear in the console and they match each other, which is what I want :

>>> 
my log file name    : C:\Users\ADMINI~1\AppData\Local\Temp\2\scratch\WebMap_f4ccc4d1-00c3-11e3-835e-123142ff4c42.log
I'm getting here
my log file handler : C:\Users\ADMINI~1\AppData\Local\Temp\2\scratch\WebMap_f4ccc4d1-00c3-11e3-835e-123142ff4c42.log
>>> ================================ RESTART ================================
>>> 
my log file name    : C:\Users\ADMINI~1\AppData\Local\Temp\2\scratch\WebMap_0e69ed9e-00c4-11e3-840d-123142ff4c42.log
I'm getting here
my log file handler : C:\Users\ADMINI~1\AppData\Local\Temp\2\scratch\WebMap_0e69ed9e-00c4-11e3-840d-123142ff4c42.log
>>> 

It means a new log file is created each time.

However each time I run it from the toolbox graphic interface, a new file handler pointing to the correct logfile is created but previous ones are not released and logs are written in all of them (see attachment).

When I publish it to the server, not even new logs are created, the geoprocessing service just keep writing in the first log generated the first time the service started.

Could it be related to the way logs are managed in ArcGIS, I saw this in the ArcGIS Server documentation that looks really similar to the problem I'm experiencing :

[HTML]By default, the log files are written to <installation_location>\server\user\log on every machine. Each time ArcGIS Server restarts, new log files are created, and the server will continue to write messages to those log files until they reach the maximum log size.[/HTML]

Here is my code :

import arcpy, uuid, os, logging
from arcpy import env

env.overwriteOutput = True

outputDir = arcpy.env.scratchFolder
outputName = 'WebMap_{}'.format(str(uuid.uuid1()))
outputLogName = outputName + r".log"
outputFileLog = os.path.join(outputDir, outputLogName)


arcpy.AddMessage("my log file name    : "+str(outputFileLog))
print ("my log file name    : "+str(outputFileLog))

def createLogger(logFile):

    arcpy.AddMessage("I'm getting here")
    print ("I'm getting here")

    logger = logging.getLogger(__name__)
    logger.setLevel(logging.INFO)

    for handler in logger.handlers:
        arcpy.AddMessage("my log file name    : "+handler.baseFilename)
        print ("my log file handler : "+handler.baseFilename)
        handler=[]
        

    handler = logging.FileHandler(logFile)
    handler.setLevel(logging.INFO)

    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)
    logger.addHandler(handler)

    for handler in logger.handlers:
        arcpy.AddMessage("my log file name    : "+handler.baseFilename)
        print ("my log file handler : "+handler.baseFilename)

        
    return logger


logger = createLogger(outputFileLog)

logger.info("I'm created")

del logger

I'm afraid I will have to write logs on a plain text file...

[SchoppMatthieu]

I found something else interesting about writing logs in a text file.

http://support.esri.com/es/knowledgebase/techarticles/detail/38445

Why not using python logging ? Is it possible it get in conflict with ArcGIS logging configuration ?

[KevinHibma]

I'm not 100% sure what you're trying to accomplish, but I'll explain how scratch works when using arcpy.env.scratchFolder like you are.
When you're in the app (ArcMap, ArcCatalog, etc), the scratchFolder will always be a folder called "scratch" in the directory of that applications scratchworkspace environment. So you can control this location by setting the scratchworkspace environment.
If a scratchworkspace has not been set, the location defaults to the user who is running the application temp directory. One of the things we did at 10.1 was introduce a temp folder with a unique id, usually arc#### inside the temp directory. This assists with clean up. As your screen shots are pointing there, I assume you're running your tools from ArcCatalog - as by default Catalog doesn't usually set a scratchworkspace and the scratchFolder environment is failing over into a known location.

If you run your scrips from Python, ie. NOT inside the application, they make use of whoever is running it's temp directory. Basically the same as above, except no unique arc####. Again, if you explicitly set the scratchworkspace in the script, the scratchFolder will get created/used there.

With ArcGIS Server, every execution of a GP Service happens in a new folder, designated by a GUID. In each of these folders a new scratch directory gets created. It is here your output gets created.

I hope by knowing the rules of "Scratch" it'll hep you move forward.

These help links offer a little more info:
http://blogs.esri.com/esri/arcgis/2012/10/19/the-new-scratchgdb-and-scratchfolder-environments/
http://resources.arcgis.com/en/help/main/10.2/index.html#//001w00000046000000

[SchoppMatthieu · Accepted solution]

Hello khibma, I hope you didn't spend too much time writing your answer, I have absolutely no problem with the scratch directory.

The problem comes from log files that are not released when executed from the toolbox.
I solved it by using a text file instead of using 'logging' facilities.