topic Re: How much geoprocessing history is kept in item description? in Data Management Questions

How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 15:58:28 GMT

I often turn to geoprocessing history in an item's description to remind myself of the steps that were taken to produce a particular data layer. A problem I keep running into though, is that the geoprocessing history just stops at a certain date in the past, and doesn't show the most recent history, which is usually the crucial part I want to examine. Is there a way to change this? Are these more recent geoprocessing steps hidden away somewhere else?

Re: How much geoprocessing history is kept in item description?

DanPatterson_Retired — Wed, 24 Jun 2015 16:05:53 GMT

I have never seen it not keep things close to the present?

I have set my ...

Geoprocessing, Geoprocessing options

Log geoprocessing operations to a log file ..... toggle off

Results Management

Keep results younger than .... I have mine set to 2 weeks ... there is a 'never delete' option

much might be gleaned from finding out where stuff is deleted when you want it really cleared out...see

How to remove geoprocessing history in ArcGIS 10.2.2 for Desktop? - Geographic Information Systems Stack Exchange

Re: How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 17:05:49 GMT

Just to clarify, I'm not talking about the results window, but rather the metadata for a data layer.

Re: How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 17:25:33 GMT

So I looked in the History folder described in the Stack Exchange thread. I see a long list of XML files there, but what are they? How do I view them, and, once viewed, how would I associate them with a particular feature class? It's the recent history that doesn't seem to be retained, so setting a longer time in geoprocessing options won't help.

Re: How much geoprocessing history is kept in item description?

DanPatterson_Retired — Wed, 24 Jun 2015 18:36:22 GMT

don't know, except for shapefiles, the xml associated with it may contain the stuff you need and can be viewed in a browser. For example...

C:\Users\ME\AppData\Roaming\ESRI\Desktop10.3\ArcToolbox\History

When I need details, I log details through a geoprocessing log and/or my own notes in *.txt format. I actually prefer notes since you can comment in them...details which aren't going to be kept in any other logging system.

For critical files. I keep temporal backups whose time-spacing is the amount of time I don't mind wasting if things go south. I have done this since you had to span a zip file across floppy disks. Today with usb drives and other external drives, it is easy.

Re: How much geoprocessing history is kept in item description?

XanderBakker — Wed, 24 Jun 2015 18:59:11 GMT

There is some explanation here: ArcGIS Help (10.2, 10.2.1, and 10.2.2)

If I look at the folder I see the history per version since installation until last usage (today for 10.3).

Re: How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 19:25:17 GMT

Thanks for your comment. So, the help file you linked says:

"Each geoprocessing tool that creates new output data or updates the input data will add metadata about the execution of the geoprocessing tool, which includes the tool name, its location, and the parameters used. This behavior can be disabled in desktop applications by un-checking the Log geoprocessing operations to a log file option found in the Geoprocessing > Geoprocessing Options dialog box. In a Python script, logging of geoprocessing history can be disabled using the SetLogHistory function.

Metadata for a dataset is viewed by right-clicking a dataset in the Catalog window or ArcCatalog and choosing Item Description."

You can see I do have that box checked in geoprocessing options:

I performed a test, a simple export of an FC to a shapefile, just now. Here is the item description window for the output, showing results only up to October of last year:

Do you see what I'm saying? I'm wondering if there is some kind of limit on the length of the geoprocessing history.

Re: How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 19:29:49 GMT

I keep notes too. That won't help if you make a mistake, or don't do what you thought you did. I second guess myself ALL the time, so the history for each data layer can be a really useful way of going back and seeing if you messed up in some way. It can also CYA when questions about data come up (or not ), months after the fact. I can't find any documented reason why this history doesn't extend all the way to the present, for some data. Maybe it's a bug of some kind?

Re: How much geoprocessing history is kept in item description?

DanPatterson_Retired — Wed, 24 Jun 2015 21:04:28 GMT

Using version 10.3.x apparently there was a change at version 10.0 and 'stuff' prior to that was kept in a different format...then it blathered on about doing this and that to see the old stuff...

As my eyes glazed over, I stumbled upon this link...I am sure there is an equivalent one, if you explore the online arctoolbox help tree for a similarly named topic

"http://desktop.arcgis.com/en/desktop/latest/manage-data/metadata/creating-and-managing-fgdc-metadata.htm"

Creating and managing FGDC metadata—Help | ArcGIS for Desktop

Re: How much geoprocessing history is kept in item description?

XanderBakker — Wed, 24 Jun 2015 21:05:18 GMT

I just did a small test with a featureclass created earlier today. I exported it to a new fgdb and added two fields (one using the toolbox and the other manually) and filled it with some data (one using the toolbox and the other do a normal field calculation).

The item description shows that it was created from the other input (creation of the field, spatial index) and the additional of the field through the toolbox and the calculation of both fields. So only the manually added field has not been logged.

See attachment.

Re: How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 21:11:16 GMT

Right, that is the expected behavior. Problem is, that's not what's happening for me, particularly with datasets that have a very loooooong history. See my previous attachments.

Re: How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 21:15:19 GMT

Well I suppose I could try upgrading the metadata. We only switched to 10.3 last week though. Prior to that we had been using 10.2 since whenever that came out (years). In my screenshot example, the metadata stopped circa October 2014. It seems unlikely to be related.

Re: How much geoprocessing history is kept in item description?

DanPatterson_Retired — Wed, 24 Jun 2015 21:19:34 GMT

When did 10.0 come out?

About that time? and

Were you working on those files using 9.3?

That is the only clue that I have that there "was change in the wind" between 9.3 and 10 and perhaps you need to update the metadata handler to look into the past

Re: How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 21:33:14 GMT

October of last year? Not quite... 10.0 came out in 2010 .

Re: How much geoprocessing history is kept in item description?

DanPatterson_Retired — Wed, 24 Jun 2015 21:37:30 GMT

Then you should try updating the metadata handler as I suggested regardless of when you started using 10. It is worth a shot and you can always switch back

Re: How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 22:06:41 GMT

Ok, I upgraded to the ArcGIS Metadata for this test layer. No dice. Same result after I exported the file again:

Re: How much geoprocessing history is kept in item description?

DanPatterson_Retired — Wed, 24 Jun 2015 22:31:38 GMT

hmmmm what does the

Include in lineage when exporting metadata No

line mean? Should this be Yes if you want it to be included for future reference? Does that mean that it won't be included. I guess there must be some place that you can toggle it on and off . I wonder if the history does get exported when you make copies via exporting...I can't check since I don't have any files that are over a year old that I have done 'stuff' to.

Re: How much geoprocessing history is kept in item description?

SepheFox — Wed, 24 Jun 2015 22:47:26 GMT

I don't know, but it's not editable, at least in the metadata. For the edification of all, I did a clip operation on my test file. Results still stop at that last 2014-10-09 process in the resulting feature class.

Are you sure you don't have metadata over a year old? The geprocessing history carries forward from the original layers, which incidentally seems like part of the problem. Perhaps the Geoprocessing history can only be so long? You would think, if that was the case, it would delete the older entries first. No one needs a record of all the permission changes that happened to the SDE version over the years.

Re: How much geoprocessing history is kept in item description?

AmyKnight — Tue, 29 Sep 2015 19:03:51 GMT

I wonder if this problem ever got fully resolved, or if you are able to see gp history in metadata for new files & operations? We are having a problem with any geoprocessing history being written to the metadata in 10.3. I read other threads related to this problem in 10.2 and we conform with all the settings suggested there, i.e. disable background processing, make sure log geoprocessing operations is checked on. We cannot get any geoprocessing operations to record in the metadata and have tried upgrading to/from various versions.

The odd thing is the geoprocessing history did appear in ArcGIS metadata for a file GDB after I compacted it today. But none for feature classes or rasters created as a result of geoprocessing operation. We need this function and are stumped.

Thanks to anyone who can shed light on this.