topic Re: Exporting large geodatase tables produces corrupt files? in Data Management Questions

Exporting large geodatase tables produces corrupt files?

RobertTrotter — Mon, 27 Jan 2014 18:07:39 GMT

I am attempting to export the contents of a geodatabase table which includes four attributes/columns, and ~ 225 million records. Unfortunatly, (in addition to being very slow, ~ 8 hours), the resulting output file is incomplete/corrupted. I am able to read records in the file up to a point, beyond which the records are empty (though the file is still structured as though the total number of records is present). I have tried exporting the table in smaller pieces, but with the same result. The location in the file at which the records ends varies with each export (the maximum so far is ~64 million, the minimum has been ~28 million). Does anyone know if:

1) There is an inherent limit in the number of records being exported (note, they are being exported to a text file, and are not being loaded into memory).

2) Are there alternate ways of exporting tables of this size?

Thanks!
Talbot

Re: Exporting large geodatase tables produces corrupt files?

WilliamCraft — Mon, 27 Jan 2014 19:42:37 GMT

Do you have a 32-bit OS or a 64-bit OS? I seem to recall a file size limit where corruption eventually occurs on a 32-bit system. How large is the final output once the process completes?

Re: Exporting large geodatase tables produces corrupt files?

VinceAngelo — Mon, 27 Jan 2014 20:36:10 GMT

Eight hours isn't a particularly long time to complete the export of a quarter-billion rows.

We would need details on how you are going about the export to provide advice on other
possibilities. Be sure to include the exact version of ArcGIS, the exact RDBMS source, and
the code you are using for export.

- V

Re: Exporting large geodatase tables produces corrupt files?

NimaNattagh — Thu, 01 Nov 2018 03:51:53 GMT

Robert , did you get the issue resolved?

Re: Exporting large geodatase tables produces corrupt files?

RobertTrotter — Wed, 21 Nov 2018 20:32:10 GMT

Hi All, I did not find a resolution to the corruption problem, but I changed to exporting data in smaller sets, this seems to have sped the process up, and avoids the overflow problem (assuming that is what caused the problem). Thanks!

Re: Exporting large geodatase tables produces corrupt files?

RobertTrotter — Fri, 09 Jul 2021 20:30:30 GMT

As an update (this is an old thread, but in case it's ever useful): Subsetting the process by breaking it up did seem to resolve the issue, so I am assuming at this point that the problem was in the architecture of the operating system (i.e. exceeding memory or the limits of a 64-bit system).