Missing Data After Merging Two Large Point Datasets (4000000 Records Each)

2892
5
Jump to solution
11-07-2014 06:05 AM
KwekuBrown
New Contributor

Hello everyone,

I recently merged two point datasets (each having about 400000 records). I realized that I lost some data after the merge even though the attribute table count of records shows that I have the complete set. Is there a limit to the number of records involved in a merge?

 

I also tried the 'append' option and I get the same results with some records missing after the process.

 

Does anyone know what is going on and how to work around this issue?

 

Thank you

 

Kweku Brown

Clemson University

0 Kudos
1 Solution

Accepted Solutions
StevenGraf1
Regular Contributor

I have a suspicious your shapefile has grown too large.  See this article.

There is a limitation on shapefile component file size, not on the shapefile size. All component files of a shapefile are limited to 2GB each.

Accordingly, the .dbf cannot exceed 2GB, and the .shp cannot exceed 2GB. These are the only component files that are likely to approach the 2GB limit.

Therefore, the total size of a shapefile (the full set of component files) can exceed 2GB.

View solution in original post

0 Kudos
5 Replies
DanPatterson_Retired
MVP Esteemed Contributor

What type of container are you using?  Check this link    then you can explore each type to see if there are limits on the number of rows

KwekuBrown
New Contributor

Hey Dan,

Thanks for your response. The two files are shapefiles, so they are not in any kind of geodatabase.

I had not thought about converting them to a feature class and doing the merge to see if it works.

Thanks for the idea. I will explore that and let you know if it worked.

Kweku

0 Kudos
KwekuBrown
New Contributor

Hello Dan,

I imported the two shapefiles as feature classes into a file geodatabase. The merge worked this time. It looks like I am not missing any data.

Thanks again for pointing me in the right direction

Kweku

0 Kudos
StevenGraf1
Regular Contributor

I have a suspicious your shapefile has grown too large.  See this article.

There is a limitation on shapefile component file size, not on the shapefile size. All component files of a shapefile are limited to 2GB each.

Accordingly, the .dbf cannot exceed 2GB, and the .shp cannot exceed 2GB. These are the only component files that are likely to approach the 2GB limit.

Therefore, the total size of a shapefile (the full set of component files) can exceed 2GB.

View solution in original post

0 Kudos
KwekuBrown
New Contributor

Hello Stephen,

Thank you for your response. I just realized what you are saying might be the issue. I checked the DBF file after the merge and it was just over 2GB thats why some of the data went missing. Stan, in an earlier response suggested geodatabases so I did that. I converted the shapefiles to feature classes and did the merge and it worked.

Thanks for your input. Its good to know that I don't have to worry about missing data although I have to redo some steps in my analysis.

Regards

Kweku

0 Kudos