Latest Contributions by RichardFairhurst

Are you using ArcMap Desktop or ArcGIS Pro?  You keep saying ArcMap, but posted in the ArcGIS Pro content group.  I will assume you are using ArcMap Desktop for the moment, since I am more familiar with troubleshooting that application than I am with ArcGIS Pro. Normally, you should first assume that strange behaviors are directly created by the specific map document you are working with.  This is especially true if you have been using this map document over the course of several ArcMap version upgrades.  Have you tried opening a brand new map and adding data to it directly from disk?  If your data displays normally in the new map while doing activities you have describe then your saved map is either configured wrong or corrupt.  If this is ArcMap you should try MXD Doctor.  If that doesn't work, you should then try copying the layers of your corrupt map and pasting them into a brand new map.  If that solves your problem kill the original map and start over with the fresh one.  If the problem starts occurring as a result of the copy paste only, then the problem is definitely in the way the layers are configured or with the data.  You may need to rebuild the map completely from scratch and only reference the original map during the rebuild process to replicate it without copy pasting anything. If you rebuild from scratch you may discover several problems in your configuration.  Configuration problems could include invalid label or definition query expressions, possibly caused by schema changes in your source data, or corrupted data.  You may need to run the Repair Geometry tool on your edit feature classes. If the problem occurs when you start with a completely new map and directly add data to it from disk, your default map template may be corrupt and should be deleted and regenerated.  If regenerating the map template fails to resolve the problem, then your software install may be corrupt. This does not exhaust all of the possible causes of your problem, but this is the normal order of the things I try to troubleshoot this kind of problem. ... View more

I tried running the code from the Light-Weight RefineNet (in PyTorch) Github project.  I was able to run the notebooks without a problem using the pretrained models.  However, when I tried to run the model training script I was unable to complete the first epoch, because it used up all of my GPU memory.  I am using NVidia Quadro P4000 with 8 GB VRam.  I thought the batch size was defaulting to 1, but it was actually set to 6 or higher.  I appears I am able to run the model after setting the batch size to 5 or less. ... View more

You cannot do this in one step with geoprocessing tools.  You should select the records with values that you want dissolved and dissolve them.  Then select all of the records with Null values in your original parcels and use the Append tool with the NO TEST option to insert them into the Dissolve output. ... View more

What really got me interested in deep learning was the Microsoft Building Footprints and really my goal is to essentially recreate their process so that I can apply it to more recent aerial photos.  They applied Semantic Segmentation like the project in my previous post, but they achieved much better results.  Apparently it is crucial to incorporate RefineNet upsampling layers described in this paper in the code to achieve resolutions that make it possible to extract individual building footprints when buildings cluster near each other.  The RefineNet paper included a link to the MATLAB code they used and their github page included a link to a PyTorch implementation of RefineNet. Sadly I have not found any code published by Microsoft related to the footprints they created, so I am stuck trying to build up enough of a knowledge base to move beyond just a conceptual understanding of what they did toward my own practical applications. ... View more

I have been trying to run code from the last link I posted, but the code was developed by Microsoft and is promoting the use of Azure.  I am having problems setting up the Azure workspace, and I don't really want to risk getting charged for storing data that is purely experimental at first.  I am frustrated that most of my problems are centered around licencing and access requirements of proprietary software or the set up of online services that may require payment and can't even begin running the code that actually processes any imagery or does any deep learning. I found yet another GitHub project for extracting building footprints described by the video here.  This code was developed by a graduate student.  He limited himself to using 3-band imagery, which is the kind of imagery I have.  I am not sure how easy this code is to run, but it seems to include a command line interface, which seems nice.  Hopefully it avoids software that isn't open source and online resources that have to be paid for, but I haven't verified that yet. ... View more

I just found this link for the github code for a deep learning project that extracts building footprints from satellite imagery.  An overview description of the the project is here.  I have not tried it or really explored the code yet, but at least initially this seems to be much closer to the kind of code I need.  The code was designed for use in Linux, so I expect it will need some tweaking to run in Windows. ... View more

Dan: I found this video on image segmentation using UNet to detect cell nuclei in images, which shows the basic principles of a modeling approach that I think could be adapted to extract building footprints from aerials.  I was able to get the code to work after pip installing a few site packages (opencv-python, tensorflow and tensorflow-gpu) and some NVidia developer software for GPU acceleration (CUDA 10.0 and CUDNN 7.6.2.24).   The division of the training/test data into tiles or chips was not done by the video example code, so I still have to deal with developing my own routines for preparing training data from much larger rasters, since the arcgis.learn.export_training_data() method esri has created requires access rights I don't have.  My starting data is similar to the esri sample, since I have a raster covering a much bigger extent than my area of interest, and a polygon layer of building footprints within my area of interest that optionally could be converted to a classified raster.  export = learn.export_training_data(input_raster=naip_input_layer, output_location=samplefolder, input_class_data=label_layer.url, chip_format="PNG", tile_size={"x":400,"y":400}, stride_size={"x":0,"y":0}, metadata_format="Classified_Tiles", context={"startIndex": 0, "exportAllTiles": False, "cellSize": 2}, gis = gis)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ The export_training_data method parameters suggest that this tool is very similar to the Split Raster tool.  I don't have experience using the Split Raster tool either, but it looks like the main difference in the parameters seems to be the metadata_format that outputs Classified Tiles.  I wish I could see a sample of the output of the export_training_data method that I could compare to the Split Raster output so that I could determine what, if any, additional processing is done beyond what the Split Raster tool does. The video example seems to handle training and testing of the model fairly well, however, it does not deal with creating a final model output from a new raster and it seems best suited to processing separate photos that do not have to be reassembled into a single image at the end, so I would also have to figure out how to accomplish that.  The esri example seems to have enclosed the final classification process in a black box method.  I assume that method tiles the new raster and combines the tiles at the end to create a final classified raster covering the original raster extent. Anyway, I would appreciate your thoughts on the assumptions I am making and any suggestions you may have that might help me create code or apply other techniques so that I could design a process of my own that might work for my needs. ... View more

Dan: Thanks for the links and comments.  The answer to "what do I want to use 'IT' for?' initially is orthophoto image analysis.  The value of AI was proven to me by the building footprints Microsoft released.  The inclusion of those 800K shapes overnight in my source data has allowed me to quickly analyze and dramatically improve the positional accuracy of my address points, parcels, land use tracking cases, general plan, zoning, cities, etc in a very short amount of time single-handed, which was never possible before.. But the Microsoft data is already getting out of date and I am certain that AI is the only practical way to make the maintenance of this layer possible for my jurisdiction and the 29 cities we encompass. I want to be part of the ability to create and maintain data in near real-time, with ever increasing resolution that is comprehensible and integrated across multiple objects in a variety of formats at scales both large and small for my jurisdiction.  Currently my jurisdiction is most lacking in its ability to extract useful information from orthophoto imagery that goes beyond making it a map background, ... View more

Well it looks like even if I could access the data that I personally can't really run any of the deep learning code.  Virtually all of the tools involved in model training require full access to Image Server.  While I believe my organization has Image Server, I personally don't have rights to access it. I find some of the documentation in the notebook and elsewhere to be misleading when it says that code like this can be run in ArcGIS Pro, since it looks like none of it really is run directly by Pro. Anyway, I feared that esri was blowing smoke at the conference saying that deep learning is coming to ArcGIS, when in reality it looks like it is only available to an extremely small percentage of their users and only an extremely small percentage of them will ever have any interest in tackling the learning curve.  I estimate that 99% of the people that attended their deep learning presentations at the UC wasted their time, since they will never have the rights to use deep learning with the way esri is currently deploying it. My frustration is that while the potential of deep learning is obvious to me, I didn't get enough information at the UC and can't get enough information from esri web articles to even intelligently talk to my organization about all of the licenses and server set ups we really would need to even attempt to use it.  I feel like there is no way for me to know if what we already have is enough or if we can justify the cost of upgrading our enterprise agreement and systems to make deep learning even possible. ... View more

After doing some searching it looks like ArcPy handles local data and arcgis.GIS handles webmap data and the two don't really mix.  So while I figured out that I can load a local layer into a Jupyter notebook by using arcpy.MakeRasterLayer_management, I can't display it in a webmap.  It looks like Spatially Enabled Dataframes can handle both local and online data together, but it seems to be more of an environment for manipulating feature class data in tabular format than it is for visualizing rasters on a map.  Anyway, it does look like the code would have to undergo a major rewrite to work with a mixture of local and online data, if it is even possible. I also tried to use an anonymous login to ArcGIS Online, but that only caused more errors.  The security issue seems to be that the data is housed in an online location that is requiring https security and my organization's setup is not compliant with the ssl certification protocols required to access the data.  It is a little frustrating that they didn't just post this data like all the other esri services that I can access through Portal. My best bet may be to publish the image data I downloaded from the Conservancy as an Image Service through my organization, although I am not completely sure even that will work. ... View more

I contacted esri tech support and they determined that the image is published, but it is being published under an organization operated by a division of esri rather than under the normal main esri organization, and my organization's security doesn't recognize that division and is blocking me from accessing the data.  So I will have to talk to my AGOL administrator to see if he either can grant me rights to see data published by that esri division or can move the data as a service under my organization. It appears that the notebook is expecting an image service and the tech person did not think it can be served from my local machine without a substantial rewrite of the code, but she said she would talk to some of the Python specialists to see if they can suggest any options. Anyway, I will post back if I make any progress.  I also will see if any of the other deep learning notebooks has a different way of setting up the training data that might work for me. ... View more

Dan wants you to show examples (pictures) of results you got using any method you choose where you think the results are wrong.  We won't make the judgement of what is right or wrong using any method, since our level of tolerance for error and yours could be very different and also as the complexity of this problem increases there could be multiple solutions that are technically right that you might reject as wrong.  Also, showing examples may lead to methods designed specifically for those special situations, if there are enough of them to make it worth the effort. ... View more

The Github link you referenced only has the Jupyter Notebook as far as I can see, not any image file.  I did change change directories to the location of the notebook I downloaded before launching it.  There is no image in that directory.  There was a data subdirectory, but the only image is called percipitation.tif without any classification labeling.  I searched all downloaded subdirectories for a file name that contained the word Kent and found nothing. Anyway, I have downloaded the Kent classified image from the Chesapeake Conservancy land cover projectand it looks like the image shown by the notebook. However, the way the code is written it is looking at ArcGIS Online, not the local directory where the notebook is located.  Anyway, I would think that there would be a way to create a layer from a file on my local hard drive, I just am not having success searching the ArcGIS Pro Python documentation for it. ... View more

I agree with Joshua that the Minimum Bounding Geometry tool may be a good starting point for analyzing your parcels, since you could compare the area of the parcel to the area of the minimum bounding geometry to separate parcels that are basically rectangular from parcels that are not.  Most likely it will be much easier to develop a process that can be successfully validated for parcels that are rectangular than for parcels that are more complex.  Separating parcels into different classes like rectangular and non-rectangular is critical to being able to build up a multi-tiered automated process that might ultimately be capable of handling the majority of parcels correctly. ... View more