Big Data Management with Arcmap desktop, best ways to handle huge files

699
3
01-16-2019 10:16 PM
OrkhanAbdullayev
New Contributor III

Sometimes I am getting text files that contain only XYZ (easting, northing, and height) with 10-15GB size (points or polylines). If I need to interpolate that file or create fishnet, adjust Z value of data or etc. in ArcMap it stops or taking forever. My comp specs I7,32GB RAM, working on SSD, GeForce GTX 1080, 4.20GHZ.

Please, if you have any suggestions let me know, thank you for your attention.

0 Kudos
3 Replies
XanderBakker
Esri Esteemed Contributor

Sounds a lot like you have a point cloud (similar to LAS) that you want to process. If you can only obtain the file as a text file with XYZ and it is not available as LAS file you should try and convert it into LAS. This will allow you to handle the data much easier and create DTM/DSM quickly. 

There are several way to do this. For instance using FME workbench (if you have the data operability extension you should have access to this) as shown here: How to convert Point Clouds from XYZ to LAS - YouTube 

The way I have translated the XYZ text files to LAS in the past is using the ArcGIS LAStools toolbox for LiDAR processing | rapidlasso GmbH and the TXT2LAS tool.

0 Kudos
OrkhanAbdullayev
New Contributor III

Thank you, I will give a try and will let you know about is useful that mentioned way with my data (as you said it is point cloud)

0 Kudos
XanderBakker
Esri Esteemed Contributor

Another comment on the your configuration. Although you have a lot of RAM available, ArcMap is a 32 bit program that can only access a small portion of the available memory. If you have access to ArcGIS Pro, this would be better