topic Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)' in Python Questions

arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

RYokochi — Sat, 13 Mar 2021 21:56:50 GMT

I have 570 ascii files with headers (example shown below). About 400 of them opens as ascii or I can convert to raster using arcpy 'AsciiToRaster'. But I have problem with coastal cells containing relatively large fraction of 'Nodata_value' cells.

I first thought it could be the format of nodata_values, but some files contain smaller number of nodata-values works, so this is probably not the case. The coastal cells I can open does not seem to have exact matching coastal line with the baseman. The coordinate system of the ascii data is JGD2011.

Anyone could advise what the problem is and how I could fix?

-----

NCOLS 225
NROWS 150
XLLCORNER 130.3
YLLCORNER 32.675
CELLSIZE 5.555555333335368e-05
NODATA_VALUE -9999

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

DanPatterson — Sat, 13 Mar 2021 23:20:38 GMT

Is there an error message?

How are you opening them?

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

RYokochi — Sun, 14 Mar 2021 01:57:00 GMT

Thank you for asking!

The error message is: Error 010067: Error in executing grid expression

Failed to execute (ASCIIToRaster)

I tried both ways (from python and from ArcGISPro Tool).

I attach the file I cannot open. Ascii files are not supported so it was zipped.

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

DanPatterson — Sun, 14 Mar 2021 02:10:50 GMT

Sounds like you aren't specifying a *.tif as the output format and/or your destination workspace has spaces in it.

It you are doing this through a script ensure that you are using "raw" encoding for your path, make your path short and simple and use *tif files as the output raster type

Also, no matter how I try every line has a line break after every number so it doesn't comply with the header, so compare the "worked" vs the "didn't work"

NCOLS 225
NROWS 150
XLLCORNER 130.125
YLLCORNER 32.683333333
CELLSIZE 5.555555999999721e-05
NODATA_VALUE -9999
3.40
15.10
17.20
21.80
27.30
29.20
28.10
27.30
28.60
30.10
31.30
30.30
27.90
24.40
21.20
18.20
15.10
12.00

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

RYokochi — Sun, 14 Mar 2021 04:59:09 GMT

Thank you for checking!!!
I processed the files that worked in the exact same way, and a large fraction of them get converted to rasters. The files that worked is attached this time. The ones that does NOT work are along the coast lines with a lot of -9999. But everything else is the same (or at least to me it seems).

I also attach a screenshot of the map that contains all the patches that worked. As you see, the ones that did not work are all by the coast and have smaller file size (a lot of -9999). Some of the working files do have -9999 but only sparsely.

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

Luke_Pinner — Wed, 17 Mar 2021 22:09:14 GMT

Your A01-20-DEM5A.asc is corrupt. The header specifies NCOLS 225 and NROWS 150 which means you should have exactly 33750 (225x150) values in your asc file, but there are only 16063 values.

If I pad out the file so there are 33750 values, then the file will convert (but obviously is missing lots of actual data).

See below (I've used black to represent -9999, blue for data values and green for the padding values that I added just to demonstrate).

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

Luke_Pinner — Wed, 17 Mar 2021 11:45:56 GMT

@DanPatterson wrote:

Also, no matter how I try every line has a line break after every number so it doesn't comply with the header, so compare the "worked" vs the "didn't work"

The use of space or newline separators doesn't matter as long as the number of values = NCOLS x NROWS.

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

RYokochi — Thu, 18 Mar 2021 04:44:08 GMT

Thank you so much for pointing this out. I converted the original file provided as xml (attached) and it does say 225x150, but you are correct, and the number of values is lower than anticipated in the original file, too.

It seems that the file with more -9999 have smaller number of values. I was looking at wrong problem.

As I tried more things, it turned out that some of the ascii files can open directly in ArcGISPro if I wait for a long time (like 30 min or 1 hour). I also noticed that the cells with its LLcorner in water seems to be the ones that are problematic, sometimes showing up as a strangely cut mosaic.

This problem turned out to be caused by the original file and not the ArcGIS, so I close this for now. But I will post if I ever find what to do with these files in case someone else works on Japanese DEM database:
https://fgd.gsi.go.jp/download/mapGis.php?tab=dem

P.S. I am new here. If anything missing, please let me know.

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

RYokochi — Thu, 18 Mar 2021 05:06:17 GMT

Going back to the .xml file, I realized that there is

<gml:startPoint>16 78</gml:startPoint>

I suspect that the data starts from there [col,row]=[16,78]

This will not make exactly 16063, but closer than 33750.

The files that work do not always have 33750, and I suspect that this was the reason.

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

Luke_Pinner — Thu, 18 Mar 2021 08:57:22 GMT

Have a look at "demtool" for converting Japanese GML DEMs:

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

RYokochi — Mon, 22 Mar 2021 04:02:39 GMT

Here is a summary so people get consolidated answers from the question without following the exchanges:

Why didn't arcpy convert my ascii file?: Because the number of data was insufficient for the given column x row

Why was the number of data insufficient?: Because a fraction of the data over the ocean was omitted. The original data in xml file did contain this information as 'start point' but I was not aware.

The ascii file was generated from an .xml file from Japanese database at https://fgd.gsi.go.jp/download/mapGis.php?tab=dem

Rather than translating these oneself, there two programs freely available [both also informed by Luke_Pinner]:

A software from: https://www.ecoris.co.jp/contents/demtool.html

A Python script from: https://gist.github.com/minorua/4993166.

I used the Python script to generate the map I wanted (attached).

Thank you, Luke_Pinner!!!

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

Luke_Pinner — Thu, 18 Mar 2021 21:34:13 GMT

One more option which works with ArcGIS Pros arcgispro-py3 python environment

Save and run the fgddem.py script from https://gist.github.com/minorua/4993166.

D:\Temp\jpgis> activate arcgispro-py3 (arcgispro-py3) D:\Temp\jpgis> python fgddem.py No input files selected === Usage === python fgddem.py [-replace_nodata_by_zero] [-f format] [-out_dir output_directory] [-q] [-v] src_files* src_files: The source file name(s). JPGIS(GML) DEM zip/xml files. (arcgispro-py3) PS D:\Temp\jpgis> python fgddem.py -f GTIFF -out_dir . FG-GML-4930-01-20-DEM5A-20161001.xml translating FG-GML-4930-01-20-DEM5A-20161001.xml completed

Re: arcpy AsciiToRaster cannot convert ascii files that contain a lot of 'nodata (-9999)'

RYokochi — Mon, 22 Mar 2021 02:05:20 GMT

This is my favorite! Thank you again for the solution #3!