Latest Contributions by RichardFairhurst

As my post said, I never really tested that code.  However, I see which line of code needs to change to avoid the error you are getting.  I embedded the polyline geometry within a tuple in the dictionary value, so I need to use a tuple index so that the geometry will be assigned to the polyline variable and not a tuple with the geometry inside of it.  Line 25 of my code needs to change from: polyline = valueDict[line]‍‍ to:             polyline = valueDict[line][0]‍ ... View more

I finally got the Export Training Data for Deep Learning tool to work using the Classified Tiles metadata output after Sanjeet Mathew at Esri support suggested changing the NODATA value of my aerial input raster from 255,255,255 to -1,-1,-1.  With that setting the tool finally created the full set of 900 classified tiles I was expecting based on the tile size and stride size I specified. A note should definitely be added to the Export Training Data for Deep Learning tool help about the various requirements of the tool for the Classified Tiles metadata output.  It requires setting an impossible NODATA value of -1,-1,-1 for the input image and only using a label raster that has properties showing it is an 8-bit unsigned, thematic raster with a NODATA value that does not match any pixel of the label raster.  Also if the label raster contains any NODATA pixels they must be reclassified as 0 using the Reclassify tool. The output image tiles still look a little off on the barren land, but they look basically fine where the buildings exist, so I am not too concerned about that for the extraction of building footprints.  I might be a little concerned if I was classifying all portions of the aerial with a land use classification. The tool does perform faster than my custom tool, so I will use it.  I will have to create a fishnet that matches the size and locations of the output classified tiles I intend to create to be able to avoid generating tiles that do not contain any buildings. Sandeep: I was able to train a model with the Classified Tiles my custom tool generated.  I only used the 582 tiles created within the aerial image I have been showing in this post.  However, I actually potentially have 1,498 more similar aerials that cover the 800K+ building polygons that could be used to generate training chips.  Are there any guidelines I should follow for creating a training set that is large enough to use for ultimately extracting all of those buildings and any new future buildings that will show up in the 2020 aerial image I expect to have next year?  Is there a rule of thumb for coming up with a percentage or sampling of my buildings that my training data should cover?  Is there a maximum limit to the number of training chips I should use if I attempt to train the model using a single GPU with 8 GB VRAM that can only handle a batch size of 3 based on a chip size of 500x500 pixels?  For now that is a limit I have to work within, which I expect can only allow me to work with a dataset that is only suitable for demonstrating the proof of concept of these techniques on my own data.  Your comments may help me justify having my organization ultimately grant me access to Image Server and many more GPUs through a virtual machine. ... View more

That is interesting that a da cursor wouldn't work with your SQL-Server set up, since I have used da cursors with my SQL-Server set up, but I assume there are a variety of ways to configure SQL-Server and that your set up and mine could be different is some significant way.  Anyway, I am glad there is support for controlling SQL-Server using commands that are more native to that database that are working for you.  It did seem like the SDE component was the likely culprit behind your problem. ... View more

Your code is both using the mxd variable and deleting the mxd variable within the for loop.  That will cause an error if the steps in the loop are executed more than one time.  Repeating steps more than once is kind of the point of using a for loop, so make sure you only delete variables within a for loop that are created inside the for loop, otherwise you should dedent the del statement to remove it from the for loop and only execute it once after the for loop has been completed. On the surface, everything seems to be OK as far as declaring the cursor, updating the 7th field of the current row and updating the row using the cursor.  That assumes that the map_created field is a numeric field and not a string field or some other type of field.  However, since you are performing this edit within an Editor session I assume the data is versioned.  Does this data use any other advanced settings that require an Editor session, such as it's a geometric network or anything else? ... View more

You say that: countDict2018 is in the form of: {'84006': 3,'84009': 14,'84020': 37,...} That means an example of the key is '84006' and an example of the value is 3.  So the value is an integer and all you need to do to access it is use: countDict2018['84006'].  You only add an index value like  countDict2018['84006'][0] when the dictionary value contains a list or tuple (which was assumed in my sample blog code since I was normally storing values in the dictionary from multiple fields).  So the code you wrote would only work  if: countDict2018 is in the form of: {'84006': [3],'84009': [14],'84020': [37],...} or countDict2018 is in the form of: {'84006': (3),'84009': (14),'84020': (37),...} ... View more

No tool will create polygons from polylines that form open loops, so you are going to have to close the polyline loops first.  You may want to first make a copy of your lines as a backup, since the tool I would recommend using will modify the input lines directly rather than creating a new output feature class.  Assuming you have a Standard or Advanced license, you should try the Extend Line tool using the default Extend to Extensions rule.  You would need to measure the length of some of the largest gaps that you think should close to set the length value.  You would need to examine the results and decide if you should manually fix the set of gaps this tool may not fix or retry the tool with a larger length value.  Once the lines create closed loops where you expect them you can use the Feature to Polygon tool. ... View more

The steps you suggested output nothing, because I cannot get it to output any tile with buildings ever.  The Export tool does not work for me, even if it works for you and everyone else.  I won't be using your tool any more now that I have built my own tool.  I am not interested in your suggestions for your Export tool anymore, since that tool is just a waste of my time and I no longer have any use for it (at least when outputting Classified Tiles). I am now exclusively focused to doing the model training.  That is all I ever really cared about in the first place, not doing data preparation over and over.  Deep Learning needs to start working for me and stop making me work for it. ... View more

I have been working with Esri support staff, including members of the Deep Learning team in Redlands.  So far Esri staff has not been able to reproduce my problem.  However, from my perspective that doesn't matter.  All that matters to me is that I have not been able to reproduce Esri's successful use of the tool, and I am not going to wait for Esri to solve my problem. So I have created my own version of the tool that does work for me.  It outputs all of the image and label files I want, as well as the json, emd, map.txt and stats.txt files.  Because I have full access to all of the internal behaviors of my tool I can adapt it to any workflow scenario I can imagine.  My tool also is more forgiving than your tool, since it only has to work for me and I know my inputs meet your requirements without having to do any manual steps like setting the raster to thematic, especially since all of the tiles get output as Generic rasters by your tool anyway. I hope I can get the Esri tool to work, in case it indicates some other problems with my installation, but at least now I can move beyond the data preparation stage and finally start doing some actual deep learning modeling. ... View more

I went ahead and tried using the Clip tool in my script and the performance actually pretty good.  To create a set of 900 image tiles and 900 label tiles with 500 x 500 pixels per tile from two original 15,000 x 15,000 pixel rasters took 5 minutes and 11 seconds.  That is a rate of about 5.79 tiles per second.  That seems acceptable.  If I limit the output to only tiles that contain buildings that can speed up the process even more.  For example, for the sample image I have been using I would only need 582 image tiles and 582 label tiles to cover all of the tiles that contain buildings, which is only about 64.7% of all of the possible images.  I can generate that set of tiles in 3 minutes and 16 seconds (a rate of about 5.94 tiles per second). So, here is the code I currently have that will generate the set of image and label tiles that contain buildings from my sample rasters: from time import strftime print( "Start script: " + strftime("%Y-%m-%d %H:%M:%S")) import arcpy import numpy filein = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\Reclass_BUILDING_LABEL.tif" imagein = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\BUILDING_AREA_1080.tif" fileout = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\output\labels\BUILDING_AREA_1080.tif" imageout = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\output\images\BUILDING_AREA_1080.tif" blocksize = 500 arcpy.env.outputCoordinateSystem = filein arcpy.env.cellSize = filein myRaster = arcpy.Raster(filein) blockno = 0 for x in range(0, myRaster.width, blocksize): for y in range(0, myRaster.height, blocksize): # Lower left coordinate of block (in map units) mx = myRaster.extent.XMin + x * myRaster.meanCellWidth my = myRaster.extent.YMin + y * myRaster.meanCellHeight # Upper right coordinate of block (in cells) lx = min([x + blocksize, myRaster.width]) ly = min([y + blocksize, myRaster.height]) # noting that (x, y) is the lower left coordinate (in cells) # Extract data block myData = arcpy.RasterToNumPyArray(myRaster, arcpy.Point(mx, my), lx-x, ly-y) if 1 in myData: # Convert data block back to raster myRasterBlock = arcpy.NumPyArrayToRaster(myData, arcpy.Point(mx, my), myRaster.meanCellWidth, myRaster.meanCellHeight) # Save on disk temporarily as 'filename_#.ext' filetemp = ('_%i.' % blockno).join(fileout.rsplit('.',1)) imagetemp = ('_%i.' % blockno).join(imageout.rsplit('.',1)) myRasterBlock.save(filetemp) myextent = '{} {} {} {}'.format(mx, my, mx + blocksize * myRaster.meanCellWidth, my + blocksize * myRaster.meanCellHeight) arcpy.Clip_management(imagein, myextent, imagetemp) # Maintain a list of saved temporary files blockno += 1 print("Made " + str(blockno) + " images") # Release raster objects from memory del myRasterBlock del myRaster print( "Finished script: " + strftime("%Y-%m-%d %H:%M:%S")) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ All that remains is to write code for tracking the statistics I need to generate the json, emd, map.txt and stats.txt files and make revisions that would replace all of the hard-coded values with parameters that could be set at run time. Below is a comparison of the tiles that the Esri Export Training Data for Deep Learning tool was producing vs. the tiles my tool is producing: The Esri Export Training Data for Deep Learning Tool output.  None of the buildings in the original image are contained in any of the tiles.  Also, the colors in the tiles have changed slightly compared to the original image. . My version of the Export Training Data for Deep Learning Tool output.  The image colors match the original and all buildings or portions of buildings in the original image are covered by a tile. ... View more

I have started investigating developing my own tool.to replace the Export Training Data for Deep Learning.  Example 2 in the NumpyArrayToRaster help is very close to what I want to do and it works well for my classified raster.  In fact I can modify the code to only output chips that contain a code of 1 for buildings to get only the set of chips that contain a building, which is the exact opposite output from the one I am getting from the Esri tool and what I really want.  The performance is also great and can extract 900 tiles with 500 x 500 pixels from a 15,000 x 15,000 pixel classified raster in under 2 minutes.  I am also confident I can create the json, emd, map.txt and stats.txt files that the Esri tool also creates using the numpy array to generate the necessary statistics. import arcpy import numpy filein = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\Reclass_BUILDING_LABEL.tif" fileout = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\output\labels\BUILDING_AREA_1080.tif" blocksize = 500 arcpy.env.outputCoordinateSystem = filein arcpy.env.cellSize = filein myRaster = arcpy.Raster(filein) blockno = 0 for x in range(0, myRaster.width, blocksize): for y in range(0, myRaster.height, blocksize): # Lower left coordinate of block (in map units) mx = myRaster.extent.XMin + x * myRaster.meanCellWidth my = myRaster.extent.YMin + y * myRaster.meanCellHeight # Upper right coordinate of block (in cells) lx = min([x + blocksize, myRaster.width]) ly = min([y + blocksize, myRaster.height]) # noting that (x, y) is the lower left coordinate (in cells) # Extract data block myData = arcpy.RasterToNumPyArray(myRaster, arcpy.Point(mx, my), lx-x, ly-y) if 1 in myData: # Convert data block back to raster myRasterBlock = arcpy.NumPyArrayToRaster(myData, arcpy.Point(mx, my), myRaster.meanCellWidth, myRaster.meanCellHeight) # Save on disk as 'filename_#.ext' filetemp = ('_%i.' % blockno).join(fileout.rsplit('.',1)) myRasterBlock.save(filetemp) # Maintain a list of saved temporary files blockno += 1 print("Made " + str(blockno) + " images") # Release raster objects from memory del myRasterBlock del myRaster print("done")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ The output of the classified raster chip is shown below overlaying the original aerial using 30% transparency and it looks great (the raster matches the original building footprint polygon outlines): I would like to use the same code with my aerials, since the performance is great.  However, the same code does not work well for processing my aerial image.  Even though the code runs and is very fast, the output of the NumpyArrayToRaster looks bad.  Even though it is in the correct location and has the correct cell size, it looks nothing like a photo anymore.  I made sure the input and output format were both tif, but the output looks bad as shown below where the output chip is overlaying the original aerial at 20% transparency: The output raster retains just a suggestion of outlines of the features contained in the original aerial that are aligned correctly with the original aerial, but clearly this output is no longer formatted as a aerial.  I could use the Clip Raster tool as an alternative to do this step in the process and that would allow me to create a tool that could replace the Esri tool, but I believe the performance of that tool would definitely suffer relative to the performance of the Esri tool. Dan, do you have any ideas on processes that would quickly clip aerials to generate a set of aerial image chip files and their world files at a minimum?  Are there any alternative python modules I should look into, or is there some way to use RasterToNumpyArray and NumpyArrayToRaster to input and output an aerial that ends up with a usable aerial image?  I didn't see any Google search results or discussions on GeoNet or stackExchange about using these numpy methods with aerials or satellite images without transforming them.  Any ideas on alternate topics I could search?  Thanks.. ... View more

Based on the classified tile samples shown in the notebook, it seems clear that you are getting tiles that have buildings in them.  It seems like the tiles I am getting are not at all what I should expect.  In fact it looks like the tiles I am getting are exactly the opposite of the set of tiles that I want.  I don't need any of those tiles, but I need all of the tiles that were not generated. At this point I am thinking of developing my own script to replace the Export Training Data for Deep Learning tool to output the set of tiles that I believe I need.  I am thinking that I would need to use the RasterToNumPyArray and NumPyArrayToRaster to be able to achieve an acceptable level of performance, since I know the Clip Raster tool is very slow.  Dan, can you comment on the possibility and practicality of using NumPy to take the raster inputs shown above and output a set of 500 x 500 classified tiles that includes buildings rather than excludes them?  The tool won't need to handle every possible input and output like the Esri tool.  It just needs to work for the raster inputs outlined in my previous post based on Sandeep's procedure (pixel depth, cell size, bands, etc). ... View more

I tried again to follow your notebook example more exactly.  I used ObjectID rather than the classValue field as input to the Polygon To Raster tool: I then used the Reclassify tool to assign all buildings to 1 and NODATA to 0 (I had deleted the background feature so my ObjectIDs started at 2 rather than 1). I ran the Export Data tool based on the settings you showed. I changed the layer to Thematic. The Raster and its properties look like this: When I ran the Export Training Data using that label raster I get the same set of tiles that have no buildings in them like I did in the previous post: If I change the NODATA property of the label raster to 0 in Catalog then the background disappears when I bring the raster into ArcGIS Pro, but when I use that as my input the Export Training Data tool doesn't output any classified tiles at all. So is this the best output I am going to get with the Export Training Data tool and will these tiles work if I tried to do the other steps in the notebook? ... View more

Sandeep: I tried to follow your instructions.  I am using ArcGIS PRo 2.4.1.  First I used the Identity tool to create a background polygon surrounding my building footprint polygons so that I could classify the entire feature extent.  I assigned a classValue of 1 to the building footprint polygons and a classValue of 2 to the background polygon.  I used the Polygon to Raster tool to output a raster of the buildings and background.  I changed the raster format from Generic to Thematic in the output and the NODATA value from 3 to 0, but noticed the output was a 2-bit pixel-depth output, not an 8-bit unsigned pixel depth output.  So I used the Copy Raster tool to make a copy that was 8-bit unsigned, and verified that the layer format was Thematic and the NODATA value was 0.  I verified that both my aerial image and my label raster have identical Extents and Spatial References and that all of the pixels are assigned a value in the label raster. Here is a picture of how the aerial image and the label image appear in the map Table of Contents legend of ArcGIS Pro 2.4.1: Here is a picture of the label raster and the raster properties of the label raster in Catalog: Here is a picture of the aerial and its Catalog properties: Here are the Export Training Data for Deep Learning tool settings: The output of the tool has no image chips with buildings, only image chips without any buildings: Here are the label tiles: The output is identical when I check the Output No Feature Tiles option on the Export tool.  The output is also identical if I copy the aerial and label rasters into the ArcGIS Pro project File Geodatabase and use those rasters as inputs for the tool. Why is the Export tool failing to output any image chips with buildings in them?  Is this output what I should expect?  What do the output tiles look like for the rasters shown in your notebook? ... View more

With ArcMap 10.6 the Export Training Data tool has a Start Index parameter that will overwrite the existing tiles if you specify a starting index that overlaps any of the indexes used by previous tiles.  If you are using that version of the tool you have to manually set the Start Index to a number that comes after the last existing tile index to append tiles and not overwrite tiles. I believe the tool automatically never overwrites and always appends only if you are using the Export Training Data tool at ArcMap 10.7 or ArcGIS Pro 2.4.  The ArcMap 10.7 and ArcGIS Pro 2.4 versions of the tool dropped the Start Index parameter and apparently starts at 0 if there are no existing tiles in a directory or else it starts indexing tiles after the last used tile index to avoid overwriting the outputs.  I think the notebook requires or assumes that you are using the ArcMap 10.7 and ArcGIS Pro 2.4 version of the tool.  This version of the tool also creates different output files that are needed by the learn.ai module that the ArcMap 10.6 version did not create, so I am pretty sure you have to use the ArcMap 10.7 and ArcGIS Pro 2.4 version of the tool to follow the notebook. ... View more

Sandeep: Your blog makes it seem like it is a simple task to create a classified raster from building footprint polygons that the Export Training Data tool will accept. Extracting building footprints We used the existing building footprints as training data to train another deep learning model for extracting building footprints. This time, the model that we had to train was a pixel classification model, wherein each pixel belonging to a building needed to be labelled as such. Pixel classification for extracting buildings Exporting training data ArcGIS Pro has support for exporting training samples in the Classified Tiles format for training such models. All we had to do was convert the existing building footprint layer to a classified raster and feed it into the tool. So it is clear that I need to create a classified raster and use the Classified Tiles output from the Export Training Data tool to Extract Building Footprints.  Your blog makes it sound like you and your team think it should be obvious to everyone what steps they need to follow to create a classified raster from a building footprint layer, but I assure you that it is not.  There is nothing obvious about it and I have wasted days searching the ArcGIS Pro and ArcMap help documents for anything that creates an identical output that works with the Export Training Data tool.  All of my outputs that resemble yours are rejected by the tool as not being Thematic.  Is there any user out there who has a clue what actually needs to be done to complete this step? ... View more