topic Re: How to get terrain properties? in Python Questions

How do I check, with python, if a Terrain Model has pyramids or data sources?

JohnLay — Tue, 26 May 2015 11:54:46 GMT

Hello all,

This question has gotten wildly off topic, so I edited the heading to reflect more clearly what I'm trying to ask.

I'm hoping someone can point me in the right direction.

I've written a script (still in development) that creates terrains from LiDAR. They are HUGE terrains, so the process from going from LAS to Multipoint can take up to 30 hours. I've hit a hick-up that I'm trying to pinpoint that occurs sometime after the multipoint creation, but before the building of the terrain.

The code I've written works fine process by process, so I'm not sure what the issue is--it may have been a network thing. I'm trying to troubleshoot it. The problem is the time it takes to run from the beginning until it breaks.

I've added if not arcpy.Exists statements for the elements that I can, but I'm stuck on how to find the properties for the terrain so that I can skip those steps if they've alread been done. For instance, if the terrain already has pyramids defined how do I check for that? Or if / what layers have already been added to the terrain?

I've spent the last 1.5 hours doing a Google search and I've got nothing except for arcpy.mapping: "There are a few specialized layers and datasets that don't fall into one of these three categories: annotation subclasses, dimension features, network datasets, terrain datasets, topology datasets, and so on." but no way to handle these specialized layers.

Suggestions?

Thanks for any help you may provide,

John

Message was edited by: John Lay

Re: How to get terrain properties?

larryzhang — Tue, 26 May 2015 12:35:35 GMT

J.,

First of all, really hard to instruct you for the solid solution to your issue, in particular, without some knowledge about your LiDAR data;

Secondly, advisable to do the following research before taking too much time on programming:

take a look at ArcGIS Help 10.1 and accept some advices from there;
browse your LiDAR data and its metadata, including data vendor, data deliverable standard & format ...

Re: How to get terrain properties?

JohnLay — Tue, 26 May 2015 14:18:53 GMT

The LiDAR is not the issue. The LiDAR LAS are v1.3. I need to create terrain models for each of 20 counties to match 20 other counties that were created by a contractor. Each terrain model must contain layers that are specific to each county, including hydrology and clipping boundaries. I'm not building rasters.

The process is:

LAS to Multipoint

Create Terrain

Add pyramids to terrain

add layers to terrain (hydro line, clip bounds, multipoints (embedded))

change terrain resolution (for hydro breaklines)

build terrains.

The script failed at add layers to terrain for some unknown reason. As i said previously, separating this bit of code and running as a single process worked just fine.

Instead of starting from scratch every time (last time took 18 hours to get to add layers to terrain), I want to start where the script broke: e.g. the add layers to terrain.

In order to do so, however, I need the script to be able to identify that the terrain already has pyramids.

Additionally, since I tested the bit a script that failed previously and the terrain now also has the proper layers added, I need the script to check for that too.

I have been unable to find anywhere on line that addresses how to read the properties of the terrain model.

Re: How to get terrain properties?

larryzhang — Wed, 27 May 2015 04:13:09 GMT

To get accurate DEM from LAS cloud points, the first thing is to get GROUND points data. Pls refer to http://sweb.cityu.edu.hk/hongbofu/doc/Deforestation_crv08.pdf

As an example, with ERDAS Point Cloud Tools, the tool 'Filter' (and 'Classify') offer this type of workflow to extract /filter certain class (like buildings, trees, Ground, water, etc) from LAS 1.2 or higher...

Re: How to get terrain properties?

JohnLay — Wed, 27 May 2015 10:34:30 GMT

Larry,

I do not need help building a terrain model. I know how to do that. I have no idea where you are coming up with this. I have asked a very specific question and you are providing answers to anything but. Go back and read, carefully, my previous posts. (No fair editing out your responses)

I am fully aware of what class our ground is. I know the multipoints should only be ground points for a surface model, not all classes. I know how to change the resolution and tolerances once the terrain is built. I do not need a lesson in LAS properties or LiDAR,

You are are making assumptions about imaginary problems.

Let me repeat, I do not need to know how to build a terrain.

What I need to know is how to check, with python, to see if the terrain model (that, yes, is contained within the feature dataset inside a file geodatabase) has pyramids. If the terrain has pyramids I need the python script to skip the process of adding pyramids. I then need to check to see if the terrain model has data sources. If the terrain has data sources, I need the python script to skip that process. And so on.

The problem I am having, and the ONLY problem I am having, is finding a way to have python check a terrain model for various properties. At this point in the script process, I don't care about the LiDAR or ground points. Their job is done. All I care about is whether or not pyramids or data sources have been added to the terrain model. That's it. Nothing more.

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

LukeWebb — Wed, 27 May 2015 15:45:26 GMT

I know nothing about terrain files sorry, but I do know for a jpg, if you add it into ArcMap and build pyramids, it will create a file on the file system disk with a .ovr extension to hold the pyramids. Perhaps you could look and see if this file exists using standard python OS. (It wont show up in any Arcpy listing data as far as I know)

E.g:

image.jpg

image.jpg.ovr

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

JohnLay — Wed, 27 May 2015 16:30:38 GMT

Unfortunately, Terrains are stored in geodatabases.

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

JohnLay — Thu, 28 May 2015 10:08:19 GMT

The more I scour the web for a solution to this problem, the more I'm beginning to believe that this will not be solved with python alone but with the comtypes module and ArcObjects. I know nothing about either.

If someone knows a good learning source, I'd appreciate it.

Esri's ArcObjects Resource Center is not very helpful.

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

ToddBlanchette — Thu, 28 May 2015 11:27:18 GMT

Hi John,

I did a lot of research on this last night as your question really intrigued me. Something as simple as a click in ArcMap should have a similar option in arcpy to access that information. Like you, I imagine, I found absolutely nothing that led me to believe this is possible in arcpy.

I'm suggesting a long shot here - and have not tested this theory to see if this actually works. It appears that the pyramid info is stored in a separate system table within the geodatabase. This table obviously isn't visible at the GIS level, but if you could send a SQL query through to the database and query whether the table or a specific entity in the table exists (I know, not being very helpful here) then you might be able to use that return as a check prior to running any further in your workflow. You might need to do some digging in your DBMS back-end to figure out what table/object is created with pyramids as I don't have the specific answer for this, just that when I researched it led me to this fact.

There should be some python modules out there that allow you to access your geodatabase's system tables and query them. I realize this isn't an answer, but hopefully it leads you to one.

Cheers,

Todd

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

JohnLay — Thu, 28 May 2015 12:03:04 GMT

Thanks Todd. We are on the same page. What little I could find is that the "guts" of the geodatabase are not visible to python but are to ArcObjects.

There are a number of references to a gentleman named Mark Cederholm who did some work a couple years back (9.3 & 10.0) with the comtypes module that can access ArcObjects. He was working with comtypes 0.6.2 found at sourceforce. Others have continued to update his Snippets module for successive releases. But Comtypes is now at 1.1.1 if you search Python.org, and I've found no acknowledgment of this new comtypes version whenever anyone speaks of Mr. Cederholm.

I've looked at Esri's ArcObjects Recource Center and I believe I have found the functions to do what I want to do, but I never learned ArcObjects and code examples to study and learn from are hiding from me if they exist in the recourse center. It appears that ITerrain Interface has "Properties Members" for DataSource and PyramidLevel. But like I said, I know nothing about ArcObjects.

You don't know of any tutorials or learning sites that I can get a crash course in ArcObjects do you? How 'bout Comtypes?

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

LukeWebb — Thu, 28 May 2015 12:06:53 GMT

Are you able to try storing it inside an SQL Server, according to this help I gather that if so, for any terrain dataset which has pyramids, a table should be made with the name:

DTM_<ID>_MRFC

The ID is specific to each terrain dataset.

Can you see if you have these tables, then you just need to find the table that has the terrain dataset names and IDs in to associate the two. Try the query:

select * from GDB_EXTENSIONDATASETS

ArcGIS Server for the Microsoft .NET Framework 9.3 Help

These tables wont be visible in ArcCatalog, you will have to connect using SQL Server client. You may need to be an admin as they may be owned by SDE not your user.

Seperately, this link may also be of use: ESRI File Geodatabase (FileGDB)

If you have OGR setup as I do, this command worked to retrieve the features using SQL from my table 'a':

ogrinfo -dialect FileGDB -sql "select * from a" "pathto\Geodatabase.gdb"

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

JohnLay — Thu, 28 May 2015 12:46:54 GMT

No, unfortunately they are being stored in file geodatabases so that they can be distributed.

The ESRI File Geodatabase (FileGDB) driver looks interesting, however. I'm off to figure out how it works. Thanks!

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

XanderBakker — Sat, 11 Dec 2021 16:34:59 GMT

When trying to get properties of an object I would normally first try with arcpy.Describe(yourTerrain). Reading up on the Describe function, I cannot find anything that is specific for terrain datasets.

import arcpy
terrain = r"D:\Xander\GeoNet\Terrain\gdb\test.gdb\TerrainData\testTerrain"
desc = arcpy.Describe(terrain)

# dataset properties
print desc.datasetType
print desc.extent
print desc.spatialReference.name
print desc.ZExtent

# object
print desc.baseName
print desc.catalogPath
print desc.children
print desc.dataElementType
print desc.extension
print desc.file
print desc.name
print desc.path

I also tried to use the arcpy.GetCount_management, but this cannot be used for terrain datasets.

In case the Terrain has just been created it will have None as extent and also None spatial reference, but having a spatial reference and an extent does not mean that the terrain dataset has data.

What you could do is creating the pyramids in a Try Except statement. If a level exists, you will not be able to add it.

Same with building the terrain, if there is not data in it it will generate the error (which you can catch):

ERROR 999999: Error executing function.

The Terrain definition is incomplete or invalid.

Failed to execute (BuildTerrain).

It is something that is lacking in arcpy. There should be some option to handle this when using the Describe function on a terrain dataset.

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

ToddBlanchette — Sat, 11 Dec 2021 16:35:02 GMT

This made for a pretty interesting project this morning. I think I finally may be on to something lol.

The following python code invokes the GDAL/OGR methods (click here if you don't have these installed).
Researching File GDB schema breakdown and with Luke Webb's confirmation above, we know we're looking for the system table called "TerrainDataset_1_MRFC" - or realistically "*MRFC", which tells us that the terrain dataset has pyramids.
Using this search parameter, I loop through all the objects in the File GDB and find this table, I then extract its name and append it to a list.
You can now use the names in this list to compare to your known dataset names to see if their pyramids have been created or not (ie: if the name shows up in the list, your pyramids "SHOULD" be good, or at the very least created at one time.

** I'm not sure if this technique will return pyramid tables from other rasters. I would set this up so that it searches specifically for the datasets you are looking for and not a blanket search on all datasets like I did. ***

# standard imports
import sys

# import OGR
from osgeo import ogr

# use OGR specific exceptions
ogr.UseExceptions()

# get the driver
driver = ogr.GetDriverByName("OpenFileGDB")

# opening the FileGDB
gdb_path = r"D:\terrain\terrain.gdb"
try:
    gdb = driver.Open(gdb_path, 0)
except Exception, e:
    print e
    sys.exit()

# list to store layers'names
featsClassList = []

# parsing layers by index
for featsClass_idx in range(gdb.GetLayerCount()):
    featsClass = gdb.GetLayerByIndex(featsClass_idx)
    terrainFCs = featsClass.GetName()
    suffix = "MRFC"
    if terrainFCs.endswith(suffix):
        featsClassList.append(terrainFCs.split('_', 1)[0])

# sorting
featsClassList.sort()

# printing
print "The following Datasets have pyramids. Catch these results and compare to your list:"
for featsClass in featsClassList:
    print featsClass

# clean close
del gdb

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

JeffreySwain — Fri, 29 May 2015 12:35:41 GMT

Right now John, the functionality to do that is not available, but should be available in a future release.

Re: How do I check, with python, if a Terrain Model has pyramids or data sources?

MarcSwartz — Tue, 19 Nov 2019 13:34:56 GMT

Mr Swain,

It is now November of 2019, has the functionality been made available yet?