topic Re: Performance issues - can't (or barely can) reproject 300 million points in ArcGIS in Geoprocessing Questions https://community.esri.com/t5/geoprocessing-questions/performance-issues-can-t-or-barely-can-reproject/m-p/66017#M2355 <HTML><HEAD></HEAD><BODY><SPAN>Next time, use gdalwarp, it does all of that together and the -multi flag uses parallel processing.</SPAN></BODY></HTML> Thu, 05 Jan 2012 21:52:03 GMT KyleShannon 2012-01-05T21:52:03Z Performance issues - can't (or barely can) reproject 300 million points in ArcGIS10 https://community.esri.com/t5/geoprocessing-questions/performance-issues-can-t-or-barely-can-reproject/m-p/66016#M2354 <HTML><HEAD></HEAD><BODY><SPAN>I have a pretty big raster that's currently gridded in NAD83 State Plane feet that I wanted to re-grid as accurately as possible in the corresponding NAD83 UTM zone, so I exported the grid as points to a geodatabase, and reprojected from State Plane to UTM into a shapefile. The point file is 306,660,952 points, 16 bit integer values.</SPAN><BR /><BR /><SPAN>I started this about four hours ago. geoprocessor went through the % complete bar pretty quickly, but it's been stuck at 100% and it WAS slowly writing output for the last 3.5 hours or more... but in the last few minutes it appears that it stuck at 100% and stopped writing output. The file sizes are:</SPAN><BR /><BR /><SPAN>.dbf - 1,268,832 KB</SPAN><BR /><SPAN>.shp - 2,089,840 KB</SPAN><BR /><SPAN>.shx - 597,098 KB</SPAN><BR /><BR /><SPAN>Oh. It just finished.</SPAN><BR /><BR /><SPAN>Anyway, I quickly became frustrated with the write speed, so I started to perform the same task in parallel using gdal (1.8) and cs2cs (4.7.1). In a fraction of the time (about 43 minutes), I was able to export the same raster to an XYZ file using gdal_translate (1.8), and convert it to UTM using CS2CS (4.7.1). on the same machine. The write speed of Arc, and the cpu overhead, didn't change appreciably while I was doing this.</SPAN><BR /><BR /><SPAN>For the gdal/cs2csthe raster started out as a 211 MB geotif, exported to XYZ as a 5.75 GB XYZ (integer feet), and transformed into a 8.89 GB UTM xyz.</SPAN><BR /><BR /><SPAN>Food for thought.</SPAN><BR /><BR /><SPAN>(WIN7x64 Dual 6-core CPUs (Xeon X5650 @ 2.67 GHz)/ 12 GB RAM / using different 2 TB SATA 3 read/write drives)</SPAN></BODY></HTML> Fri, 11 Feb 2011 02:39:17 GMT https://community.esri.com/t5/geoprocessing-questions/performance-issues-can-t-or-barely-can-reproject/m-p/66016#M2354 AndyRitchie 2011-02-11T02:39:17Z Re: Performance issues - can't (or barely can) reproject 300 million points in ArcGIS https://community.esri.com/t5/geoprocessing-questions/performance-issues-can-t-or-barely-can-reproject/m-p/66017#M2355 <HTML><HEAD></HEAD><BODY><SPAN>Next time, use gdalwarp, it does all of that together and the -multi flag uses parallel processing.</SPAN></BODY></HTML> Thu, 05 Jan 2012 21:52:03 GMT https://community.esri.com/t5/geoprocessing-questions/performance-issues-can-t-or-barely-can-reproject/m-p/66017#M2355 KyleShannon 2012-01-05T21:52:03Z Re: Performance issues - can't (or barely can) reproject 300 million points in ArcGIS https://community.esri.com/t5/geoprocessing-questions/performance-issues-can-t-or-barely-can-reproject/m-p/66018#M2356 <HTML><HEAD></HEAD><BODY><SPAN>No doubt gdal is faster for many things, but...</SPAN><BR /><BR /><SPAN>1) Why not just project it in raster format? Since there is no datum transformation, I don't (think?) you would&nbsp; get any meaningfully added precision by converting to point format and projecting that.</SPAN><BR /><BR /><SPAN>2) You listed the .shp file size as being 2,089,840 MB, which is basically the 32-bit 2.1GB file size limit... Are you sure ALL your points made it to the shapefile?</SPAN></BODY></HTML> Thu, 05 Jan 2012 23:21:08 GMT https://community.esri.com/t5/geoprocessing-questions/performance-issues-can-t-or-barely-can-reproject/m-p/66018#M2356 ChrisSnyder 2012-01-05T23:21:08Z