POST
|
No that's not really a must have. I just wanted to make it possible to use layers with selections as input. Also running it from ArcScene but without using layers as input seems to work fine. Well i think i just have to get along with that.
... View more
07-31-2014
12:22 AM
|
0
|
0
|
237
|
POST
|
After a few test runs i seem to have found the cause of the problem. Well not really the cause of the problem but at least the case were it can occur. As i mentioned in my last post the only difference between starting it from comandline or from ArcScene is that i use Layers as input if i start it from ArcScene. So i made a few runs from inside ArcGis with only "true files" as input and no error occured so far! So the problem seems to be: Error in "MakeFeatureLayer" on layer while multiprocessing in python. (Well also on raster layer and "CopyFeatures/Raster" also throws ErrorExecutingFunction sometimes)
... View more
07-30-2014
01:50 AM
|
0
|
0
|
237
|
POST
|
It runs without background processing. I tried it with now, but i dont get my console output anymore, so its hard to tell if any error occurs. Maybe i can write to a log file, but i dont think enableing background processing would better anything 😕 But that was a good idea background processing often screws up everything
... View more
07-30-2014
12:47 AM
|
0
|
0
|
237
|
POST
|
Hello again, thanks for the participation so far. Sadly my problem still persists. I still think that ArcScene is causing it somehow but not that simultaneous file access from ArcScene is the cause anymore. I inserted a retry mechanism that trys the data access two additional times after two 10 seconds breaks, but the data access fails everytime. I looked into the folder of the gdb and could not find any abnormal lock files there. One sr.lock for every data source and every process and one rd.lock everytime data is acually read. Furthermore, it looks like this problem only occures if the script is started out of ArcScene, not if ArcScene is just running like i thought before, but i am not sure about that, because there is no way to reliable reproduce the error. So starting the script out of a command line window seems to work correctly. I also noticed that in addition to the "Cannot open file" and "Dataset does not exist or is not supported" errors, sometimes a "Error executing function" error is thrown. One more point. Runing the script out of ArcScene with only one worker process seems to work correctly, again this is only an asumption, but it is not the case that there is always one worker thread which is fine. I had one test with 4 worker processes and 2 of them crashed during the handling of their first task and the other 2 during their second. I am pretty frustrated right now and i am running out of ideas. The only solution for me now seems to be to only use the script out of the command line. As i am writing this it comes to my mind that i am using layers as input if i run the script from inside ArcScene which is not the case if i run it from command line, so maybe there is a problem with using "MakeFeatureLayer" on layer? Sorry for any language errors.
... View more
07-29-2014
06:24 AM
|
0
|
2
|
763
|
POST
|
Hello Joshua and Chris, first of all thanks for your answers.
What happens if you set maxtasksperchild=1 when instantiating the pool?
It happens while the workers work on the first set of tasks so this won't help Also, how big are the data sets you are working with? Would it be possible for each child process to make an in-memory copy of the data set, or possibly a temporary copy on disk? Well the MakeFeatureLayer function calls are for exactly this purpose. Calling some function that have to read the same data synchronously failed in the past so i came up with this. (This shouldn't have happened, too, cause all these functions also just need reading access to the data) I can try to make a copy "in_memory" rather than a layer. Try using threads instead of processes and see what happens. I could try this, but it would not be a solution to the problem cause i implemented multiprocessing to make use of multiple processor cores. Multithreading doesn't offer that as far as i know in the Python implementation ArcGIS uses. One workaround i have in mind is to catch the exception earlier and let the process try the data access once more. Also i ran the script as a standalone python script (with ArcGIS completly closed) for an hour without an error wich is a clue for my "ArcScene running is occasionally accessing the data in the same moment i try to use MakeFeatureLayer on it" theory to be correct. Is there a chance that someone from esri stumbles upon this thread and give an answer on the expected behavior of the "MakeFeatureLayer" function or do i have to contact tech support directly?
... View more
07-28-2014
01:06 AM
|
0
|
1
|
763
|
POST
|
Hello Joshua, no it runs as a script tool and out of process. so this is a edited version of the example from the esri blog post i used. (my code is much more complex now but i can explain my problem better with that snipplet)
import multiprocessing
import numpy
import arcpy
def worker(ranges):
i, j = ranges[ 0 ], ranges[ 1 ]
lyr = arcpy.management.MakeFeatureLayer(
Input_Point_Features ,
"point_layer{0}".format(i) ,
"OBJECTID >= {0} AND OBJECTID <= {1}".format(i, j) )
result _table = calculatesomething(lyr)
result_array = arcpy.da.TableToNumPyArray(result_table, [ "*" ])
arcpy.management.Delete(gn_table)
return result_array
# End generate_near_table function
def main():
ranges = somegeneratetvalues
pool = multiprocessing.Pool()
result_arrays = pool. map (worker, ranges)
# Concatenate the resulting arrays and create an output table
# reporting any identical records.
result_array = numpy.concatenate(result_arrays,axis = 0 )
arcpy.da.NumPyArrayToTable(result_array, "mytargetlocation" )
# Synchronize the main process with the job processes to
# Ensure proper cleanup.
pool.close()
pool.join()
# End main
if __name__ = = '__main__' :
main()
The problem is: MakeFeatureLayer fails if two or more processes try to run it on the same data at the same time.If i run it with two processes and insert a few seconds waiting time for one of them everything is fine. But every example i could find on the internet used MakeFeatureLayer and multiprocessing equaly to the code above and none of them used some sort of mutual exclusion. (I'm using multiprocessing.Lock right now to fix the problem and it works most of the time. Still sometimes it fails. I think the cause of this is ArcScene running and occasionally accessing the data in the same moment i try to use MakeFeatureLayer on it, but i'm not sure about it)
... View more
07-24-2014
07:22 AM
|
0
|
6
|
763
|
POST
|
Hello everyone, --- Update ------------------------------- See later posts for specific problem description. i try to use python multiprocessing to do some processing on chunks of a feature class. I changed the third example from this esri post: Multiprocessing with ArcGIS – Approaches and Considerations (Part 1) | ArcGIS Blog If i set the poolsize to 1 it runs correctly but if i use more than one process "MakeFeatureLayer" throws a "Cannot open file" or "Dataset does not exist or is not supported" error most of the time for every process but the first one. (I use "MakeFeatureLayer" on three different Files in every process and it fails at one of them for sure) It seems to be an error related to accesing the same file at the same time out of different processes, because inserting a waiting time befor starting the "MakeFeatureLayer" methods seems to work. Because this problem is not mentioned in the esri post or somewhere else in the www i think it must be my mistake but i just do the same thing like esri in their third example of the mentioned post 😕 The only thing mentioned in the blogpost is that you cannot use multiprocessing to update data in a gdb but "MakeFeatureLayer" needs only read access, doesn't it?
def worker(ranges):
i = ranges[0]
j = ranges[1]
try:
arcpy.AddMessage("started processing thread ")
arcpy.AddMessage(ranges)
arcpy.CheckOutExtension("3D")
## time.sleep(1*(ranges[2]-1))
samples=arcpy.MakeFeatureLayer_management(Input_Point_Features, "point_layer{0}".format(i), "OBJECTID >= {0} AND OBJECTID <= {1}".format(i, j))
anyone some suggestions? thx
... View more
07-21-2014
03:42 AM
|
0
|
14
|
2186
|
POST
|
Hello everyone, is it possible to completly hide (parts of) features that are behind multipatches in ArcScene? Currently if you zoom in pretty much to a multipatch it hides the features behind it. But if you zoom out all point or line features get visible regardless of whether the multipatch is in front of them or not. An example: If i look at a building i dont want to see the lines of the street net behind it. Maybe this is just a little option in the settings, but i couldn't find it there or a solution in the forum .. i mean .. "GeoNet". Thx for your help.
... View more
07-15-2014
06:26 AM
|
0
|
0
|
1484
|
POST
|
Hi everyone, i found a little hack to get it going. I save a .lyr file of a layer that has the desired 3D-Attributes (no base height, take z values from features) and created a arcpy.mapping.Layer out of it in Python. Then i use replaceDataSource() to set the data source to my generated polygon class and then use the layer file as input for "Layer 3D to Feature Class". I still need to work on some invalid data errors if this is used with an Iterator, but basically it does what it should.
... View more
07-11-2014
01:48 AM
|
1
|
0
|
720
|
POST
|
Hello Jeffrey. First of all thanks for your reply. Do you know another way to transform 3D polygones to multipatches? Cause i want to use them as input for the Intersect 3D tool wich only accepts multipatches. My final goal is what i described in this Thread Distribute points on building facades
... View more
07-08-2014
05:37 AM
|
0
|
1
|
720
|
POST
|
Hello everyone, i try to generate multipatches from multiple rectangle polygons with z values using the "Layer 3D to Feature Class" tool. The layer containing the polygons was generated in the same model earlyer. The output is correct if i run the model inside modelbuilder with "add to display" turned on on the polygon layer, but not if not. Furthermore the output seems to be correct if the polygon layer was added to the display at least once. It does not have to be in the current scene. I guess the cause of this is, that the 3D properties of a layer, wich are used by "Layer 3D to Feature Class", are generated the first time it is displayed, not when it is generated. Does anyone have some ideas how to overcome this? Thanks for every answer.
... View more
07-02-2014
05:32 AM
|
0
|
4
|
7284
|
POST
|
Hello everyone, does somebody know a way to distribute points over building facades in ArcGIS similiar to the intervisibility 3D Analyst / CityEngine demo that you should find here: http://video.esri.com/watch/1662/3d-analyst-feature-and-volumetric-analysis? (Because the site was not reachable when i wrote the message, i am not sure if the link is correct.) In the demo they use CityEngine to generate these sample points. I tried to do the same by importing my shape file into CityEngine but i didn't find the panel option in the inspector. (maybe pretty naive but i dont know much about CityEngine :D) My best idea is to do several intersections with rectangles with different z values to get line features representing the outline of the building on different heights. Then i use the generate random points tool with a high max points parameter to generate nearly evenly distributed points on the lines. (I cant get it to work in ModelBuilder though because of a problem with the the "Layer 3D to Feature Class" tool when running it on a layer that was not added to the display before. For this, I will probably open another thread.) Does anyone have some ideas? Thanks for every answer.
... View more
07-02-2014
05:00 AM
|
0
|
0
|
3481
|
Title | Kudos | Posted |
---|---|---|
1 | 07-11-2014 01:48 AM |
Online Status |
Offline
|
Date Last Visited |
11-11-2020
02:23 AM
|