topic Re: [PYTHON] .gdb import - spatial envelope settings not working in ArcGIS CityEngine Questions

[PYTHON] .gdb import - spatial envelope settings not working

MarekDekys — Wed, 24 May 2017 14:47:04 GMT

Hi,

I'm trying to import .gdb (file geodatabase) file using python

while everything works fine, I'm having trouble defining custom envelope for my import to crop an area out from big .gdb file:

no matter what I put into [] it always loads all 490 objects:

setImportQueryAndEnvelope("OBJECTID < 491", [7157.656, 4162.500,274845.500,5143901.000] ,"NORTH_WEST")

but I'm not sure if the syntax is correct because manual suggests:

@param env: The spatial envelope with syntax (xMin, xMax, yMin, yMax). (default = None). [sequence of float]

while in dialog there are options for width, height and X-Offset and Y-Offset

I tried the documentation's syntax to achieve 1x1 km crop using :

[7157.656, 7257.656, 4162.500, 4262.500]

(the 2nd and 4th values are just incremented by 1000 - to correspond with manual's syntax of xMin, xMax, yMin, yMax ) but the output is always the same..

I also tried to put all zeros in dialog window and it correctly loaded 0 objects

but when I try the same thing in code using [0,0,0,0] it still loads all 490 objects and it should load 0 objects instead

any help appreciated.

thanks,

Mark

Re: [PYTHON] .gdb import - spatial envelope settings not working

IanMurray — Wed, 24 May 2017 16:34:50 GMT

"@param env: The spatial envelope with syntax (xMin, xMax, yMin, yMax). (default = None). [sequence of float]"

Have you tried entering the values as a tuple instead of a list in the parameters? Not sure it makes a difference, but the example does show it as a tuple not a list.

Re: [PYTHON] .gdb import - spatial envelope settings not working

MarekDekys — Thu, 25 May 2017 08:35:42 GMT

Hi Ian, thanks for a quick reply

I tried tuple with zeros using code below (hope it's tuple) to force it not to import anything (to check that it works) but looks like it doesn't work because it still exports everything instead of empty layer with 0 objects

settings.setImportQueryAndEnvelope("OBJECTID > 0", (0.0,0.0,0.0,0.0) ,"NORTH_WEST")

Not sure what they mean by a [sequence of float] - but I found another location in manual where it uses [sequence of float] and it is defined like a [val,val,val,val] but maybe i'm mistaken:

setAttributeLayerExtents(self, layer, extents)

@param extents: An list of four floating point values. [sequence of float]

ce.setAttributeLayerExtents(l, [0,20,3000,4000])

any help would be appreciated

Re: [PYTHON] .gdb import - spatial envelope settings not working

IanMurray — Thu, 25 May 2017 13:30:13 GMT

Have you made sure to not only set the envelope and query, but make sure it is being applied in the settings?

Based on your screenshot there is a Boolean to actually use the query and envelope values you put in, which I would imagine equates to getUseSelectionQueryAndSpatialEnvelope() method mentioned in the help. You are probably writing your envelope parameters correctly, its just not being applied and you need to set that value to True.

It would be nice if there was better help examples on that documention link for FGDB settings for CE import.

Re: [PYTHON] .gdb import - spatial envelope settings not working

MarekDekys — Thu, 25 May 2017 14:11:57 GMT

Yes I am using settings.setUseSelectionQueryAndSpatialEnvelope(True). see the code below: (same as in screenshot above)

the sql selection query works fine (i can set OBJECTID < 101 to import only first 100 objects.. etc..) but it ignores the rest :

from scripting import *
input_location = "_input_buildings_gdb/test_small.gdb"
ce = CE()# get a CityEngine instance
if __name__ == '__main__':
settings = FGDBImportSettings()
settings.setImportAndMapAttributes(True)
settings.setUseSelectionQueryAndSpatialEnvelope(True)
settings.setImportQueryAndEnvelope("OBJECTID > 0", (0.0,0.0,0.0,0.0) ,"NORTH_WEST")
ce.importFile(ce.toFSPath(input_location), settings)

Re: [PYTHON] .gdb import - spatial envelope settings not working

IanMurray — Thu, 25 May 2017 14:35:20 GMT

Apparently its too early and my reading comprehension is not what it needs to be...

TFuchs-esristaff‌

Re: [PYTHON] .gdb import - spatial envelope settings not working

DavidWasserman — Thu, 25 May 2017 21:34:23 GMT

Hi Marek,

On the GIS Side, would it be too much to perhaps just depend on the SQL functionality? I mean if you want to define zones for import, you could just have features within a grid boundary (or census block), be specified by a SQL query? Sometimes I just make the edits on the GIS side to get a smooth pipeline into CE.

Can you share the exact projection of the data you are importing?

Also on the python side, I might be missing something, but does the result change if you specify the specific file path/name of the FC you are trying to import using the setFile setting? (I don't use this function a lot, so it might be a silly question).

setFile(self, stringValue)Sets File field. @param stringValue: the new value. [str]

David

Re: [PYTHON] .gdb import - spatial envelope settings not working

MarekDekys — Fri, 26 May 2017 08:09:15 GMT

Hi David,

If I can use some SQL query to select only objects that have only certain cooridnates (that fit into boundig box) like:

SELECT * WHERE "OBJECTID > 0 AND X > 0 AND X<100 AND Y>0 AND Y<100" that would be great.

but the problem is I don't know if this is possible and the code above doesn't work for me because fields X,Y are not defined as a table (unlike OBJECTID which is defined as a field/table) and there are no fields/tables defined in gdb that contain information about position

I'm using UTM 32N projection

I have also rewritten the code to use getFile/setFile from what I understood from your reply but it works and acts the same as the code before:

from scripting import *
input_location = "_input_buildings_gdb/test_small.gdb"
ce = CE()# get a CityEngine instance
if __name__ == '__main__':
settings = FGDBImportSettings()
settings.setImportAndMapAttributes(True)
settings.setUseSelectionQueryAndSpatialEnvelope(True)
settings.setFile(ce.toFSPath(input_location))
settings.setImportQueryAndEnvelope("OBJECTID > 0" , (0.0,0.0,0.0,0.0) ,"NORTH_WEST")
ce.importFile(settings.getFile(),settings)

If anybody can share any code example about setting spatial envelope via setImportQueryAndEnvelope I would be grateful

Re: [PYTHON] .gdb import - spatial envelope settings not working

deleted-user-Q-Ku9J1BlYDK — Fri, 26 May 2017 22:20:04 GMT

Hi Marek,

I was thinking more along the lines of creating a fish net or selecting specifically the data you want to import and adding a field that you can use to flag for it. So you would select the relevant study area in GIS, add an attribute associated with it and then use a SQL query like "INSIDE_STUDY_AREA=1" based on your new field. This is not using the spatial envelope but might be a way to achieve the same thing. You could also use a fish net if you need to import areas in small batches (spatial join the fish net IDs to your data for example and then select based the fish net ID).

CE can be odd on some projections. Are you providing spatial envelopes appropriate to your projection? I try to use web mercator (auxiliary sphere) when importing into CE once if the data is not getting imported as I thought it should. It might change the result.

I thought the setFile/getFile paths need to be the full feature class path. In ArcPy, a specific feature class for example is set up as : "FolderPath/MyFGDB.gdb/SomeFeatureClass". I was curious if maybe this was a necessary setting for spatial envelop but it was just a guess.

I will try to test this on my end later next week if no one replies Marek.

David

Re: [PYTHON] .gdb import - spatial envelope settings not working

MarekDekys — Mon, 29 May 2017 08:39:40 GMT

Hi David,

thanks for in-depth reply. I agree that a possible workaround would be by adding flags to gdb and then use SQL to cut into regular grid/fishnet but unfortunatelly I don't have access to any good GIS tool from esri.. I only have global mapper which has problems processing .gdb

about setting up correct projection - There is no option in dialog to set the projection nor is in the documentation about

FGDBImportSettings() so I believe it will reproject it to the current projection of the scene (which works perfectly I have .gdb files in LV95 and it reprojects them correctly into WGS84 UTM 32N) - the problem is that I cannot force it to import 0 objects if I set spatial envelope like (0,0,0,0) to confirm that this command is working - It always imports everything just like it is ignoring this command

not sure what you mean about setFile/getFile. can you provide me with a code sample? (but I think this is not the case ) I don't know is CE uses ArcPy or it is just a regular python..

I have other workaround by processing everything entirely in CE (without need of other GIS tools that I don't have) which I will use if everything else fails but it is a slower proces for milions of files..

I was just hoping that this was only a small issue (in syntax misunderstanding in setImportQueryAndEnvelope command ) that can be solved by somebody providing me a correct code sample but it turns out it won't be so easy as it seems like nobody is using this command.. or it is a bug so I will report it to esri if nobody answers about the correct syntax of this command

Re: [PYTHON] .gdb import - spatial envelope settings not working

ThomasFuchs — Mon, 29 May 2017 16:06:40 GMT

Thank you for reporting this issue. I can reproduce it and need to work with a developer on this.
At this time it is unclear if it can be resolved by changing the command alone. I'll keep you posted.

Re: [PYTHON] .gdb import - spatial envelope settings not working

ThomasFuchs — Tue, 30 May 2017 14:15:27 GMT

Update:
Sadly, I have to confirm that there is a bug in the spacial envelope. Any entered values are not set. The development team is working on a fix.
At this point only the SQL queries can be used.

Re: [PYTHON] .gdb import - spatial envelope settings not working

DavidWasserman — Wed, 31 May 2017 02:35:28 GMT

Hi Thomas,

Thanks for confirming that this is a bug. It sounds like a SQL fish net is the only alternative. The only other alternatives is can think of is using OSGEO command line to potentially process the data before hand or transform it into a shapefile for editing in Global Mapper (which also is an imperfect process).
Marek if you can't do a GIS fix, your alternative might be the best approach. Thanks for reporting this.
David

Re: [PYTHON] .gdb import - spatial envelope settings not working

MarekDekys — Wed, 31 May 2017 08:28:47 GMT

Thanks all,

that's bad news that this is a bug, I will have to recreate spatial envelope in python in CE then. And doing that I found another issue -> https://community.esri.com/message/690436-no-more-handles-swt-error-while-batch-processing-gdb-in-python

do you have any estimation when it will be fixed?

thanks,

Mark

Re: [PYTHON] .gdb import - spatial envelope settings not working

ThomasFuchs — Thu, 26 Oct 2017 12:13:17 GMT

Good News! The issue has been resolved in CityEngine 2017.1

This version is scheduled to be released in November.

Re: [PYTHON] .gdb import - spatial envelope settings not working

DavidWasserman — Thu, 26 Oct 2017 14:10:59 GMT

Great news Thomas!