Understanding .dat-i and .dat-u from server statistics

StanleyBrown — Wed, 26 Jan 2022 15:01:14 GMT

tl;dr What is the format of .dat-i and .dat-u files in arggisssystem/statistics?

I'm trying to analyse usage patterns for our web services so I can optimise resource allocation (for example, increasing/decreasing running instances). I've tried a few methods that didn't quite work:

The statistics report tool built into ARCGIS server manager is not suitable for this task because it can only produce pretty limited statistical outputs.
The export function that is built-in only produces a few data points at a time which makes it a very slow way of extracting the data that sits behind it. (Approx 3 hours to get a sample of 20 .csv's)
The API works, but only seems able to retrieve total figures. It doesn't seem to have a parameter that lets me specify a date range for the data.

So I've found the place where the statistics are actually saved:

Location of files ^. Although at first menacing, both the .dat-i and the .dat-u files turned out to be structured like csv's.

Sample .dat-i ^

Sample .dat-u ^

So it seems I have managed to find all the statistics I could ever want in a simple machine-readable format that is well suited for statistical analysis. Hurrah! However, I'm not quite sure what most of the fields actually are.

The first field in both is an epoch timestamp in milliseconds. The second is the address of each service.

Then the .dat-i files have 4 other fields:

The number 60
The number 0.000
Either 1.000, 2.000, or 3.000
Either 0.000, 30.000, 58.000 or 60.000

Wheras, the dat-u files have 7 other fields:

An integer, mostly under 100
0, 1, 2 or 3
0
0
An integer, mostly under 100, mostly the same value as field 1.
A 3 decimal place value, mostly under 1
A 3 decimal place value, typically higher than field 7.

I have some of my own guesses for some of these, but it would be really helpful if any of you either know for sure what they are, have good guesses, or can point me to the documentation for this. Also, what is the difference between the dat.i and the dat.u files? Thanks for reading!

Re: Understanding .dat-i and .dat-u from server statistics

berniejconnors — Wed, 26 Jan 2022 19:05:22 GMT

Stanley - what about the System Log Parser?

https://community.esri.com/t5/implementing-arcgis-blog/system-log-parser-statistics-and-service/ba-p/886389

Bernie.

Re: Understanding .dat-i and .dat-u from server statistics

StanleyBrown — Mon, 31 Jan 2022 13:45:02 GMT

Thanks Bernie!

That's working wonders, I knew there had to be an easier way. Thanks so much for pointing it out.

Stanley

topic Re: Understanding .dat-i and .dat-u from server statistics in ArcGIS Enterprise Questions

Understanding .dat-i and .dat-u from server statistics

Re: Understanding .dat-i and .dat-u from server statistics

Re: Understanding .dat-i and .dat-u from server statistics