topic Re: Append (Data Management) failing with ultra large datasets in Geoprocessing Questions

Append (Data Management) failing with ultra large datasets

MarcoBoeringa — Thu, 10 Aug 2017 22:51:34 GMT

Hi all,

I am currently attempting to Append a couple of point datasets with a total of 370M points in three feature classes. All datasets reside in File Geodatatabases (three separate ones), and I am appending to one of the existing feature classes.

The data resides on a 2TB SSD with plenty of free space (>1TB). The machine runs ArcMap 10.5.1 and has 16GB RAM and Windows 10 as OS.

Now I have successfully done this with datasets with up to well over 100M records as final result dataset on Windows 7 machine, but I am now getting consistent failures with these. The error number is the not very useful generic 999998: Unexpected error type error and happens somewhere half way, so probably in the range of 100-200M records having been appended.

Clearly, with so many records, I might potentially run out of some resource, but it certainly isn't disk space.

Has anyone else been successful in appending feature classes with over 100M records each and well over 300M total, and especially also on Windows 10?

Re: Append (Data Management) failing with ultra large datasets

DanPatterson_Retired — Fri, 11 Aug 2017 00:01:35 GMT

Can you test in Pro? The tool is the same, but the memory limits imposed by ArcMap aren't there.

Also, have you tried the Append one at a time or are you doing them all at once?

Re: Append (Data Management) failing with ultra large datasets

MarcoBoeringa — Fri, 11 Aug 2017 09:04:31 GMT

Hi Dan,

Seems you were right. Pro succeeded in appending the 370M records into one feature class. However, I never saw RAM usage of Pro exceed more than just over 1GB, so I am a bit at loss at why ArcMap fails. Seems there is another limitation not so much related to total RAM, but to the 32 bit nature of ArcMap?

Anyway, I am now also attempting this one other time from ArcCatalog instead of ArcMap. I saw another GeoNet discussion where it was suggested that running tools from ArcCatalog instead of ArcMap made a difference in some cases, with tools run from the catalog window in ArcMap failing but not in ArcCatalog.

Re: Append (Data Management) failing with ultra large datasets

DanPatterson_Retired — Fri, 11 Aug 2017 09:41:57 GMT

Good find... I assume that you are running ArcCatalog separately and not from within ArcMap, if so, you have removed the memory overhead of ArcMap