Latest Contributions by RichardFairhurst

The angles created by the Linear Referencing layer are Arithmetic Bearings, not Geographic Bearings.  Geographic Bearings start with 0 degrees pointing North and the degrees increase in a clockwise direction, but Arithmetic Bearings start with 0 degrees pointing East and the degrees increase in a counterclockwise direction.  You have your layer set to show Geographic Bearings and need to change it to show Arithmetic Bearings.  Change the setting on the Symbology tab by pressing the Advanced button and choosing the Rotation... option.  Then choose your angle field and the Arithmetic Rotation Style and press OK for all the dialogs. \ Anyway, I just noticed that the Bearing Distance to Line tool expects degrees to be Geographic Bearings.  To convert Arithmetic Bearings to Geographic Bearings create a new double field and use this Python calculation (where %LOC_ANGLE% is the Arithmetic angle field you want to convert): (450 - %LOC_ANGLE%) % 360 I am sorry for the numerous steps needed just to convert the outputs of one tool for use as an input for another tool.  The Geoprocessing Tool approach tends to run into these kinds of issues where the tools don't easily integrate with each other.  A Python script could streamline the process and eliminate several steps, but my instructions are just meant to get the job done.  Once the process is proven to work you can invest time in learning ways to optimizing the steps if you will need to do this process repeatedly. ... View more

The output of step 10 should be a table, because the Bearing Distance to Line tool in Step 11 expects a table as its main input. ... View more

Is the table an Excel file or csv file or any table that lacks an ObjectID?  If it is then relates and joined record selections are prohibited.  See the help on the Fundamentals of Objectid fields.  You have to convert Excel files and csv files to either a geodatabase table that has ObjectIDs or a dbf table that has FIDs to be able to create a relate or do joined record selections.  The table conversion probably should occur before you use the XY Table to Point Layer tool for all tables involved in your relates or joins or for the point layer table you could export the point layer created by the tool to transform the in-memory layer the tool creates (which may lack a proper ObjectID if the table source is a csv file) into a permanent on-disk geodatabase point feature class (which will automatically generate an ObjectID field).  The Table to Table tool can be used to do the conversion of a csv file into a geodatabase table. ... View more

Here is how I would set it up in ArcMap (I won't use Pro).  The Route Identifier Field for both the Input Route Features and the Event Table Properties is the "FID_segment_original" field.  The "halfway" field is the Measure field.  The "offset_dist" field is the Offset Field.  It appears that the compliment angle option changes the Angle field output rather than creating a second field with the compliment angle, so you may have to create a new field for the compliment angle and calculate it.  I would need to see examples of the output you get to give you the field calculator formula but basically it would involve adding or subtracting 180 degrees and correcting values that are lower than 0 degrees or higher than 360 degrees. ... View more

The cell size is 0.5 feet height and width per pixel.  That is the resolution of my aerial.  The maximum area of my aerial I can work with from my Image Service is 15,000x15,000 pixel, because it has a restriction on the number of columns and rows I can download in order to change the NODATA values of my aerials.  I was using 500x500 pixels for my chips, because it exactly divides my downloaded aerials into whole chips and at the size the chip will normally contain one or more whole buildings and bigger portions of large buildings. Smaller size chips like 375x375 pixels (56.25% of a 500x500 pixel chip) or 300x300 pixels (36% of a 500x500 pixel chip) would also exactly divide my downloaded aerials into whole chips, but of course the number of whole buildings in each chip will be reduced and more chips will only contain partial buildings.  However, if I create a fishnet I can also do an Identity with my building polygons.  Then I should be able to statistically analyze those results so that the majority of chips I use for training contain whole buildings or all portions of large buildings that won't fit within a single chip. I don't really understand what the graph of learning rates means or how different learning rate numbers will affect the model training.  Can you provide a little more explanation of what adjustments to the learning rate does and what aspects of the learning rate graph I really need to focus on? ... View more

As my post said, I never really tested that code.  However, I see which line of code needs to change to avoid the error you are getting.  I embedded the polyline geometry within a tuple in the dictionary value, so I need to use a tuple index so that the geometry will be assigned to the polyline variable and not a tuple with the geometry inside of it.  Line 25 of my code needs to change from: polyline = valueDict[line]‍‍ to:             polyline = valueDict[line][0]‍ ... View more

I finally got the Export Training Data for Deep Learning tool to work using the Classified Tiles metadata output after Sanjeet Mathew at Esri support suggested changing the NODATA value of my aerial input raster from 255,255,255 to -1,-1,-1.  With that setting the tool finally created the full set of 900 classified tiles I was expecting based on the tile size and stride size I specified. A note should definitely be added to the Export Training Data for Deep Learning tool help about the various requirements of the tool for the Classified Tiles metadata output.  It requires setting an impossible NODATA value of -1,-1,-1 for the input image and only using a label raster that has properties showing it is an 8-bit unsigned, thematic raster with a NODATA value that does not match any pixel of the label raster.  Also if the label raster contains any NODATA pixels they must be reclassified as 0 using the Reclassify tool. The output image tiles still look a little off on the barren land, but they look basically fine where the buildings exist, so I am not too concerned about that for the extraction of building footprints.  I might be a little concerned if I was classifying all portions of the aerial with a land use classification. The tool does perform faster than my custom tool, so I will use it.  I will have to create a fishnet that matches the size and locations of the output classified tiles I intend to create to be able to avoid generating tiles that do not contain any buildings. Sandeep: I was able to train a model with the Classified Tiles my custom tool generated.  I only used the 582 tiles created within the aerial image I have been showing in this post.  However, I actually potentially have 1,498 more similar aerials that cover the 800K+ building polygons that could be used to generate training chips.  Are there any guidelines I should follow for creating a training set that is large enough to use for ultimately extracting all of those buildings and any new future buildings that will show up in the 2020 aerial image I expect to have next year?  Is there a rule of thumb for coming up with a percentage or sampling of my buildings that my training data should cover?  Is there a maximum limit to the number of training chips I should use if I attempt to train the model using a single GPU with 8 GB VRAM that can only handle a batch size of 3 based on a chip size of 500x500 pixels?  For now that is a limit I have to work within, which I expect can only allow me to work with a dataset that is only suitable for demonstrating the proof of concept of these techniques on my own data.  Your comments may help me justify having my organization ultimately grant me access to Image Server and many more GPUs through a virtual machine. ... View more

That is interesting that a da cursor wouldn't work with your SQL-Server set up, since I have used da cursors with my SQL-Server set up, but I assume there are a variety of ways to configure SQL-Server and that your set up and mine could be different is some significant way.  Anyway, I am glad there is support for controlling SQL-Server using commands that are more native to that database that are working for you.  It did seem like the SDE component was the likely culprit behind your problem. ... View more

Your code is both using the mxd variable and deleting the mxd variable within the for loop.  That will cause an error if the steps in the loop are executed more than one time.  Repeating steps more than once is kind of the point of using a for loop, so make sure you only delete variables within a for loop that are created inside the for loop, otherwise you should dedent the del statement to remove it from the for loop and only execute it once after the for loop has been completed. On the surface, everything seems to be OK as far as declaring the cursor, updating the 7th field of the current row and updating the row using the cursor.  That assumes that the map_created field is a numeric field and not a string field or some other type of field.  However, since you are performing this edit within an Editor session I assume the data is versioned.  Does this data use any other advanced settings that require an Editor session, such as it's a geometric network or anything else? ... View more

You say that: countDict2018 is in the form of: {'84006': 3,'84009': 14,'84020': 37,...} That means an example of the key is '84006' and an example of the value is 3.  So the value is an integer and all you need to do to access it is use: countDict2018['84006'].  You only add an index value like  countDict2018['84006'][0] when the dictionary value contains a list or tuple (which was assumed in my sample blog code since I was normally storing values in the dictionary from multiple fields).  So the code you wrote would only work  if: countDict2018 is in the form of: {'84006': [3],'84009': [14],'84020': [37],...} or countDict2018 is in the form of: {'84006': (3),'84009': (14),'84020': (37),...} ... View more

No tool will create polygons from polylines that form open loops, so you are going to have to close the polyline loops first.  You may want to first make a copy of your lines as a backup, since the tool I would recommend using will modify the input lines directly rather than creating a new output feature class.  Assuming you have a Standard or Advanced license, you should try the Extend Line tool using the default Extend to Extensions rule.  You would need to measure the length of some of the largest gaps that you think should close to set the length value.  You would need to examine the results and decide if you should manually fix the set of gaps this tool may not fix or retry the tool with a larger length value.  Once the lines create closed loops where you expect them you can use the Feature to Polygon tool. ... View more

The steps you suggested output nothing, because I cannot get it to output any tile with buildings ever.  The Export tool does not work for me, even if it works for you and everyone else.  I won't be using your tool any more now that I have built my own tool.  I am not interested in your suggestions for your Export tool anymore, since that tool is just a waste of my time and I no longer have any use for it (at least when outputting Classified Tiles). I am now exclusively focused to doing the model training.  That is all I ever really cared about in the first place, not doing data preparation over and over.  Deep Learning needs to start working for me and stop making me work for it. ... View more

I have been working with Esri support staff, including members of the Deep Learning team in Redlands.  So far Esri staff has not been able to reproduce my problem.  However, from my perspective that doesn't matter.  All that matters to me is that I have not been able to reproduce Esri's successful use of the tool, and I am not going to wait for Esri to solve my problem. So I have created my own version of the tool that does work for me.  It outputs all of the image and label files I want, as well as the json, emd, map.txt and stats.txt files.  Because I have full access to all of the internal behaviors of my tool I can adapt it to any workflow scenario I can imagine.  My tool also is more forgiving than your tool, since it only has to work for me and I know my inputs meet your requirements without having to do any manual steps like setting the raster to thematic, especially since all of the tiles get output as Generic rasters by your tool anyway. I hope I can get the Esri tool to work, in case it indicates some other problems with my installation, but at least now I can move beyond the data preparation stage and finally start doing some actual deep learning modeling. ... View more

I went ahead and tried using the Clip tool in my script and the performance actually pretty good.  To create a set of 900 image tiles and 900 label tiles with 500 x 500 pixels per tile from two original 15,000 x 15,000 pixel rasters took 5 minutes and 11 seconds.  That is a rate of about 5.79 tiles per second.  That seems acceptable.  If I limit the output to only tiles that contain buildings that can speed up the process even more.  For example, for the sample image I have been using I would only need 582 image tiles and 582 label tiles to cover all of the tiles that contain buildings, which is only about 64.7% of all of the possible images.  I can generate that set of tiles in 3 minutes and 16 seconds (a rate of about 5.94 tiles per second). So, here is the code I currently have that will generate the set of image and label tiles that contain buildings from my sample rasters: from time import strftime print( "Start script: " + strftime("%Y-%m-%d %H:%M:%S")) import arcpy import numpy filein = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\Reclass_BUILDING_LABEL.tif" imagein = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\BUILDING_AREA_1080.tif" fileout = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\output\labels\BUILDING_AREA_1080.tif" imageout = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\output\images\BUILDING_AREA_1080.tif" blocksize = 500 arcpy.env.outputCoordinateSystem = filein arcpy.env.cellSize = filein myRaster = arcpy.Raster(filein) blockno = 0 for x in range(0, myRaster.width, blocksize): for y in range(0, myRaster.height, blocksize): # Lower left coordinate of block (in map units) mx = myRaster.extent.XMin + x * myRaster.meanCellWidth my = myRaster.extent.YMin + y * myRaster.meanCellHeight # Upper right coordinate of block (in cells) lx = min([x + blocksize, myRaster.width]) ly = min([y + blocksize, myRaster.height]) # noting that (x, y) is the lower left coordinate (in cells) # Extract data block myData = arcpy.RasterToNumPyArray(myRaster, arcpy.Point(mx, my), lx-x, ly-y) if 1 in myData: # Convert data block back to raster myRasterBlock = arcpy.NumPyArrayToRaster(myData, arcpy.Point(mx, my), myRaster.meanCellWidth, myRaster.meanCellHeight) # Save on disk temporarily as 'filename_#.ext' filetemp = ('_%i.' % blockno).join(fileout.rsplit('.',1)) imagetemp = ('_%i.' % blockno).join(imageout.rsplit('.',1)) myRasterBlock.save(filetemp) myextent = '{} {} {} {}'.format(mx, my, mx + blocksize * myRaster.meanCellWidth, my + blocksize * myRaster.meanCellHeight) arcpy.Clip_management(imagein, myextent, imagetemp) # Maintain a list of saved temporary files blockno += 1 print("Made " + str(blockno) + " images") # Release raster objects from memory del myRasterBlock del myRaster print( "Finished script: " + strftime("%Y-%m-%d %H:%M:%S")) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ All that remains is to write code for tracking the statistics I need to generate the json, emd, map.txt and stats.txt files and make revisions that would replace all of the hard-coded values with parameters that could be set at run time. Below is a comparison of the tiles that the Esri Export Training Data for Deep Learning tool was producing vs. the tiles my tool is producing: The Esri Export Training Data for Deep Learning Tool output.  None of the buildings in the original image are contained in any of the tiles.  Also, the colors in the tiles have changed slightly compared to the original image. . My version of the Export Training Data for Deep Learning Tool output.  The image colors match the original and all buildings or portions of buildings in the original image are covered by a tile. ... View more

I have started investigating developing my own tool.to replace the Export Training Data for Deep Learning.  Example 2 in the NumpyArrayToRaster help is very close to what I want to do and it works well for my classified raster.  In fact I can modify the code to only output chips that contain a code of 1 for buildings to get only the set of chips that contain a building, which is the exact opposite output from the one I am getting from the Esri tool and what I really want.  The performance is also great and can extract 900 tiles with 500 x 500 pixels from a 15,000 x 15,000 pixel classified raster in under 2 minutes.  I am also confident I can create the json, emd, map.txt and stats.txt files that the Esri tool also creates using the numpy array to generate the necessary statistics. import arcpy import numpy filein = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\Reclass_BUILDING_LABEL.tif" fileout = r"C:\Users\rfairhur\Documents\Jupyter Notebooks\arcgis-python-api-master\Imagery\Riverside\output\labels\BUILDING_AREA_1080.tif" blocksize = 500 arcpy.env.outputCoordinateSystem = filein arcpy.env.cellSize = filein myRaster = arcpy.Raster(filein) blockno = 0 for x in range(0, myRaster.width, blocksize): for y in range(0, myRaster.height, blocksize): # Lower left coordinate of block (in map units) mx = myRaster.extent.XMin + x * myRaster.meanCellWidth my = myRaster.extent.YMin + y * myRaster.meanCellHeight # Upper right coordinate of block (in cells) lx = min([x + blocksize, myRaster.width]) ly = min([y + blocksize, myRaster.height]) # noting that (x, y) is the lower left coordinate (in cells) # Extract data block myData = arcpy.RasterToNumPyArray(myRaster, arcpy.Point(mx, my), lx-x, ly-y) if 1 in myData: # Convert data block back to raster myRasterBlock = arcpy.NumPyArrayToRaster(myData, arcpy.Point(mx, my), myRaster.meanCellWidth, myRaster.meanCellHeight) # Save on disk as 'filename_#.ext' filetemp = ('_%i.' % blockno).join(fileout.rsplit('.',1)) myRasterBlock.save(filetemp) # Maintain a list of saved temporary files blockno += 1 print("Made " + str(blockno) + " images") # Release raster objects from memory del myRasterBlock del myRaster print("done")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ The output of the classified raster chip is shown below overlaying the original aerial using 30% transparency and it looks great (the raster matches the original building footprint polygon outlines): I would like to use the same code with my aerials, since the performance is great.  However, the same code does not work well for processing my aerial image.  Even though the code runs and is very fast, the output of the NumpyArrayToRaster looks bad.  Even though it is in the correct location and has the correct cell size, it looks nothing like a photo anymore.  I made sure the input and output format were both tif, but the output looks bad as shown below where the output chip is overlaying the original aerial at 20% transparency: The output raster retains just a suggestion of outlines of the features contained in the original aerial that are aligned correctly with the original aerial, but clearly this output is no longer formatted as a aerial.  I could use the Clip Raster tool as an alternative to do this step in the process and that would allow me to create a tool that could replace the Esri tool, but I believe the performance of that tool would definitely suffer relative to the performance of the Esri tool. Dan, do you have any ideas on processes that would quickly clip aerials to generate a set of aerial image chip files and their world files at a minimum?  Are there any alternative python modules I should look into, or is there some way to use RasterToNumpyArray and NumpyArrayToRaster to input and output an aerial that ends up with a usable aerial image?  I didn't see any Google search results or discussions on GeoNet or stackExchange about using these numpy methods with aerials or satellite images without transforming them.  Any ideas on alternate topics I could search?  Thanks.. ... View more