topic Re: Missing Data After Merging Two Large Point Datasets (4000000 Records Each) in Geoprocessing Questions

Missing Data After Merging Two Large Point Datasets (4000000 Records Each)

KwekuBrown — Fri, 07 Nov 2014 14:05:57 GMT

Hello everyone,

I recently merged two point datasets (each having about 400000 records). I realized that I lost some data after the merge even though the attribute table count of records shows that I have the complete set. Is there a limit to the number of records involved in a merge?

I also tried the 'append' option and I get the same results with some records missing after the process.

Does anyone know what is going on and how to work around this issue?

Thank you

Kweku Brown

Clemson University

Re: Missing Data After Merging Two Large Point Datasets (4000000 Records Each)

DanPatterson_Retired — Fri, 07 Nov 2014 15:13:26 GMT

What type of container are you using? Check this link then you can explore each type to see if there are limits on the number of rows

Re: Missing Data After Merging Two Large Point Datasets (4000000 Records Each)

KwekuBrown — Fri, 07 Nov 2014 20:45:28 GMT

Hey Dan,

Thanks for your response. The two files are shapefiles, so they are not in any kind of geodatabase.

I had not thought about converting them to a feature class and doing the merge to see if it works.

Thanks for the idea. I will explore that and let you know if it worked.

Kweku

Re: Missing Data After Merging Two Large Point Datasets (4000000 Records Each)

StevenGraf1 — Fri, 07 Nov 2014 20:49:56 GMT

I have a suspicious your shapefile has grown too large. See this article.

There is a limitation on shapefile component file size, not on the shapefile size. All component files of a shapefile are limited to 2GB each.

Accordingly, the .dbf cannot exceed 2GB, and the .shp cannot exceed 2GB. These are the only component files that are likely to approach the 2GB limit.

Therefore, the total size of a shapefile (the full set of component files) can exceed 2GB.

Re: Missing Data After Merging Two Large Point Datasets (4000000 Records Each)

KwekuBrown — Fri, 07 Nov 2014 21:09:53 GMT

Hello Dan,

I imported the two shapefiles as feature classes into a file geodatabase. The merge worked this time. It looks like I am not missing any data.

Thanks again for pointing me in the right direction

Kweku

Re: Missing Data After Merging Two Large Point Datasets (4000000 Records Each)

KwekuBrown — Fri, 07 Nov 2014 21:13:46 GMT

Hello Stephen,

Thank you for your response. I just realized what you are saying might be the issue. I checked the DBF file after the merge and it was just over 2GB thats why some of the data went missing. Stan, in an earlier response suggested geodatabases so I did that. I converted the shapefiles to feature classes and did the merge and it worked.

Thanks for your input. Its good to know that I don't have to worry about missing data although I have to redo some steps in my analysis.

Regards

Kweku