How do I check, with python, if a Terrain Model has pyramids or data sources?

4169
15
05-26-2015 04:54 AM
JohnLay
Occasional Contributor

Hello all,

This question has gotten wildly off topic, so I edited the heading to reflect more clearly what I'm trying to ask.

I'm hoping someone can point me in the right direction.

I've written a script (still in development) that creates terrains from LiDAR. They are HUGE terrains, so the process from going from LAS to Multipoint can take up to 30 hours. I've hit a hick-up that I'm trying to pinpoint that occurs sometime after the multipoint creation, but before the building of the terrain.

The code I've written works fine process by process, so I'm not sure what the issue is--it may have been a network thing. I'm trying to troubleshoot it. The problem is the time it takes to run from the beginning until it breaks.

I've added if not arcpy.Exists statements for the elements that I can, but I'm stuck on how to find the properties for the terrain so that I can skip those steps if they've alread been done. For instance, if the terrain already has pyramids defined how do I check for that? Or if / what layers have already been added to the terrain?

I've spent the last 1.5 hours doing a Google search and I've got nothing except for arcpy.mapping: "There are a few specialized layers and datasets that don't fall into one of these three categories: annotation subclasses, dimension features, network datasets, terrain datasets, topology datasets, and so on." but no way to handle these specialized layers.

Suggestions?

Thanks for any help you may provide,

John

Message was edited by: John Lay

15 Replies
larryzhang
Regular Contributor

J.,

First of all, really hard to instruct you for the solid solution to your issue, in particular, without some knowledge about your LiDAR data;

Secondly, advisable to do the following research before taking too much time on programming:

  1. take a look at ArcGIS Help 10.1   and accept some advices from there;
  2. browse your LiDAR data and its metadata, including data vendor, data deliverable standard & format ...
0 Kudos
JohnLay
Occasional Contributor

The LiDAR is not the issue. The LiDAR LAS are v1.3. I need to create terrain models for each of 20 counties  to match 20 other counties that were created by a contractor. Each terrain model must contain layers that are specific to each county, including hydrology and clipping boundaries. I'm not building rasters.

The process is:

LAS to Multipoint

Create Terrain

Add pyramids to terrain

add layers to terrain (hydro line, clip bounds, multipoints (embedded))

change terrain resolution (for hydro breaklines)

build terrains.

The script failed at add layers to terrain for some unknown reason. As i said previously, separating this bit of code and running as a single process worked just fine.

Instead of starting from scratch every time (last time took 18 hours to get to add layers to terrain), I want to start where the script broke: e.g. the add layers to terrain.

In order to do so, however, I need the script to be able to identify that the terrain already has pyramids.

Additionally, since I tested the bit a script that failed previously and the terrain now also has the proper layers added, I need the script to check for that too.

I have been unable to find anywhere on line that addresses how to read the properties of the terrain model.

larryzhang
Regular Contributor

To get accurate DEM from LAS cloud points, the first thing is to get GROUND points data. Pls refer to http://sweb.cityu.edu.hk/hongbofu/doc/Deforestation_crv08.pdf

As an example, with ERDAS Point Cloud Tools,  the tool 'Filter' (and 'Classify') offer this type of workflow to extract /filter certain class (like buildings, trees, Ground, water, etc) from LAS 1.2 or higher...

0 Kudos
JohnLay
Occasional Contributor

Larry,

I do not need help building a terrain model. I know how to do that. I have no idea where you are coming up with this. I have asked a very specific question and you are providing answers to anything but. Go back and read, carefully, my previous posts. (No fair editing out your responses)

I am fully aware of what class our ground is. I know the multipoints should only be ground points for a surface model, not all classes. I know how to change the resolution and tolerances once the terrain is built.  I do not need a lesson in LAS properties or LiDAR,

You are are making assumptions about imaginary problems.

Let me repeat, I do not need to know how to build a terrain.

What I need to know is how to check, with python, to see if the terrain model (that, yes, is contained within the feature dataset inside a file geodatabase) has pyramids. If the terrain has pyramids I need the python script to skip the process of adding pyramids. I then need to check to see if the terrain model has data sources. If the terrain has data sources, I need the python script to skip that process. And so on.

The problem I am having, and the ONLY problem I am having, is finding a way to have python check a terrain model for various properties. At this point in the script process, I don't care about the LiDAR or ground points. Their job is done. All I care about is whether or not pyramids or data sources have been added to the terrain model. That's it. Nothing more.

LukeWebb
Occasional Contributor III

I know nothing about terrain files sorry, but I do know for a jpg, if you add it into ArcMap and build pyramids, it will create a file on the file system disk with a .ovr extension to hold the pyramids. Perhaps you could look and see if this file exists using standard python OS. (It wont show up in any Arcpy listing data as far as I know)

E.g:

image.jpg

image.jpg.ovr

JohnLay
Occasional Contributor

Unfortunately, Terrains are stored in geodatabases.

0 Kudos
JohnLay
Occasional Contributor

The more I scour the web for a solution to this problem, the more I'm beginning to believe that this will not be solved with python alone but with the comtypes module and ArcObjects. I know nothing about either.

If someone knows a good learning source, I'd appreciate it.

Esri's ArcObjects Resource Center is not very helpful.

0 Kudos
ToddBlanchette
Occasional Contributor II

Hi John,

I did a lot of research on this last night as your question really intrigued me.  Something as simple as a click in ArcMap should have a similar option in arcpy to access that information.  Like you, I imagine, I found absolutely nothing that led me to believe this is possible in arcpy.

I'm suggesting a long shot here - and have not tested this theory to see if this actually works.  It appears that the pyramid info is stored in a separate system table within the geodatabase.  This table obviously isn't visible at the GIS level, but if you could send a SQL query through to the database and query whether the table or a specific entity in the table exists (I know, not being very helpful here) then you might be able to use that return as a check prior to running any further in your workflow.  You might need to do some digging in your DBMS back-end to figure out what table/object is created with pyramids as I don't have the specific answer for this, just that when I researched it led me to this fact.

There should be some python modules out there that allow you to access your geodatabase's system tables and query them.  I realize this isn't an answer, but hopefully it leads you to one.

Cheers,

Todd

JohnLay
Occasional Contributor

Thanks Todd. We are on the same page. What little I could find is that the "guts" of the geodatabase are not visible to python but are to ArcObjects.

There are a number of references to a gentleman named Mark Cederholm who did some work a couple years back (9.3 & 10.0) with the comtypes module that can access ArcObjects. He was working with comtypes 0.6.2 found at sourceforce. Others have continued to update his Snippets module for successive releases. But Comtypes is now at 1.1.1 if you search Python.org, and I've found no acknowledgment of this new comtypes version whenever anyone speaks of Mr. Cederholm.

I've looked at Esri's ArcObjects Recource Center and I believe I have found the functions to do what I want to do, but I never learned ArcObjects and code examples to study and learn from are hiding from me if they exist in the recourse center. It appears that ITerrain Interface has "Properties Members" for DataSource and PyramidLevel. But like I said, I know nothing about ArcObjects.

You don't know of any tutorials or learning sites that I can get a crash course in ArcObjects do you? How 'bout Comtypes?