nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Missing Data After Merging Two Large Point Datasets (4000000 Records Each)

KwekuBrown

Hello everyone,

I recently merged two point datasets (each having about 400000 records). I realized that I lost some data after the merge even though the attribute table count of records shows that I have the complete set. Is there a limit to the number of records involved in a merge?

I also tried the 'append' option and I get the same results with some records missing after the process.

Does anyone know what is going on and how to work around this issue?

Thank you

Kweku Brown

Clemson University

Find more posts tagged with

Accepted answers

StevenGraf1

I have a suspicious your shapefile has grown too large. See this article.

There is a limitation on shapefile component file size, not on the shapefile size. All component files of a shapefile are limited to 2GB each.

Accordingly, the .dbf cannot exceed 2GB, and the .shp cannot exceed 2GB. These are the only component files that are likely to approach the 2GB limit.

Therefore, the total size of a shapefile (the full set of component files) can exceed 2GB.

All comments

KwekuBrown

Hello Stephen,

Thank you for your response. I just realized what you are saying might be the issue. I checked the DBF file after the merge and it was just over 2GB thats why some of the data went missing. Stan, in an earlier response suggested geodatabases so I did that. I converted the shapefiles to feature classes and did the merge and it worked.

Thanks for your input. Its good to know that I don't have to worry about missing data although I have to redo some steps in my analysis.

Regards

Kweku

KwekuBrown

Hello Dan,

I imported the two shapefiles as feature classes into a file geodatabase. The merge worked this time. It looks like I am not missing any data.

Thanks again for pointing me in the right direction

Kweku

StevenGraf1

I have a suspicious your shapefile has grown too large. See this article.

There is a limitation on shapefile component file size, not on the shapefile size. All component files of a shapefile are limited to 2GB each.

Accordingly, the .dbf cannot exceed 2GB, and the .shp cannot exceed 2GB. These are the only component files that are likely to approach the 2GB limit.

Therefore, the total size of a shapefile (the full set of component files) can exceed 2GB.

KwekuBrown

Hey Dan,

Thanks for your response. The two files are shapefiles, so they are not in any kind of geodatabase.

I had not thought about converting them to a feature class and doing the merge to see if it works.

Thanks for the idea. I will explore that and let you know if it worked.

Kweku

DanPatterson_Retired

What type of container are you using? Check this link then you can explore each type to see if there are limits on the number of rows

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.