Latest Contributions by HugoBouckaert1

Hi  We have to process big data and for that Geoanalytics server seems to be a good solution because it supports a clustered environment. The Geoanalytics server supports arcgis libraries which themselves can make use of clustered computing (just like pyspark). However, it does not support arcpy. At the moment all my scripts are written using arcpy, using an ArcGIS Pro desktop environment and Jupyter notebooks.  The scripts are quite complex and use a lot of arcpy functions.  Before we invest in implementing a Geoanalytics server, we need to know if my scripts will run only using  the arcgis libraries without access to the arcpy libraries. I know that arcgis is web-focused and is a wrapper for the REST api, and I am happy to translate what I have written using arcpy into arcgis. I also know the two are meant to be complementary to each other rather than "like for like", but nevertheless, for the scripts, I need equivalent functionality in arcgis that is currently available in arcpy.  So my questions are:  1. Is there a URL with an "equivalence list" and "exclusion list" ? For example there is an append procedure in both, but I could not find a NEAR function in arcgis (could be there of course, but this is just an example).  Going through the documentation function by function comparing arcgis to arcpy is a huge task, so a page that lists equivalent functions would be very helpful. Similarly a page that lists functions supported in arcpy that are not in arcgis would be very useful.   2. Can we use the Geoanalytics server stand-alone without making reference to ArcGIS Online or a local implementation of Portal? 3. Are File geodatabases supported? I watched a couple of videos about Geoanalytics server and the datastores it can use. Although shapefiles are mentioned File geodatabases were not in the list of supported datasets.  Only being able to use shapefiles would be a limitation, especially because the the lenght limit of column names.  Any help will be greatly appreciated.  Thanks Hugo     ... View more

Hi  I have several arcpy scripts that run on my desktop but take days to complete. So far I have spatially gridded the data and then used parallel processing to process the spatial grids simultaneously. We are looking at a better solution however, because creating spatial grids creates its own problems, with features "cut in half", and if when kept in two grids, resulting in duplication.  Does the GeoAnalytics server support arcpy? It does not look like it, but if it does, this might be a solution. Ideally we are looking at running this code in an Azure cloud environment. In Azure, you can make use of clusters (multiple VMs) which can be scaled up to do some powerful processing. However, in Azure only pyspark libraries are supported for multiprocessing. When you use arcpy, you have to install an anaconda environment (e.g. the environment that comes with ArcGIS Pro), but any code you run cannot use Azure clustered resources and so you are back to being limited to using a single server.  Are there any solutions out there to run python code with the arcpy libraries at scale? The compute resources needed are huge, so a single server, even if very powerful, might not suffice. Note also that python by default runs on a single core, so that a single powerful machine even with 64 cores will not speed up the process because only a single core will be used.  This is why I spatially gridded the data and sent each grid to a separate core, but as I said, we would like to get away from that solution.    Any ideas or help would be most welcome.  Thanks Hugo            ... View more

Hi  I have line features representing overhead power lines, and for these power lines, I have height attribute values at three different locations: the start of the line, the end of the line and (using a distance attribute from start) the point of maximal sag of the line between start and end points. I do not want to create straight lines between start point, sag point and end point but a curved line of best fit between the three points. Note that the "sag" point is not always midway between the start and end point. I need to do this in a python program as I have to calculate this for a large volume of lines, so the editing "arc" tool is not really something I can use. I looked into trying to fit an ellipsoid between the points but that is no good as this would mean I have one ellipsoid between start point and sag point and another ellipsoid between sag point and end point. I need a single curve that fits all three points. I tried the spline tool, which creates a raster surface from points (which means I can perhaps get back from raster to more points), but 3 points is not enough to create a raster surface. I also tried non spatial tools such as scipy interpolate (fitting a spline between sets of np.array values), but here also, I only have 3 points which is not sufficient to do the interpolation and create the "spline" line. Anyone knows how to create a curve of best fit between three point locations? In my case these points are z (height) values so either a curved "best fit" line is calculated directly or intermediate points are calculated from which I  create a new line feature. However, in that case I also would have to find the X and Y coordinates of the newly created (intermediate) z height values.  Difficult problem, I know, I am just wondering if someone has worked this out, or can help me in any way.  Thanks very much.  Hugo       ... View more

Hi  I would like to have a map (utilised by a webapp) that caters for more than one use, because the background layers will be the same for all users. The use includes editing, but editors of layer A should not be able to edit (and see) layer B, and vice versa. I can put editors of layer A in one group and editors of layer B in another, but my understanding is that the map (and webapp item) have to be part of both groups for any of the people in the groups to even open the map or webapp. Is it possible that the same map (and webapp)  is used by different groups whereby one group can edit layer A but cannot see (and edit) layer B, and the other group can edit layer B but not see (and edit) layer A? Perhaps there is a mechanism via views, but would views allow editing? Thanks Hugo       ... View more

Hi  I have been trying to delete a field from a feature layer. I have tried "delete_from_definition" and I get a success response, and yet the field (column) is still there afterwards:  1. I have a layer object, named updater_layer (it is definitely a feature layer object, I have checked its properties).   2. I create the JSON that defines the column:  status_remove = { "name" : "STATUS", "type" : "esriFieldTypeString", "alias" : "Status", "sqlType" : "sqlTypeOther", "length" : 35, "nullable" : 'true', "editable" : 'true', "domain" : 'null', "defaultValue" : 'null' } 3. I use: updater_layer.manager.delete_from_definition(status_remove) {'success': True} but when I look at the properties i.e.  updater_layer.manager.properties The STATUS field is still there.  How do I remove a field from a feature layer in ArcGIS Online using the arcgis.gis API?  Thanks Hugo   ... View more

Hi  Is it at all possible to create a feature layer in ArcGIS Online from a GeoDataFrame? If you cannot create a feature layer directly from a GeoDataFrame, is it possible to populate an existing feature layer from a GeoDataFrame (provided the fields match of course).  I tried all sorts of methods to create a feature layer directly using GeoJSON as the intermediary  - no luck. So then I tried to insert values (attributes and geometry) from a GeoDataFrame into an existing feature layer in AGOL, using GeoDataFrame.to_dict and then trying to add that dictionary to the feature layer using edit_features(adds=<my dictionary created from the GeoDataFrame>). No luck there either.  Anyone knows how this can be done?  Thanks Hugo  ... View more

Hi  At the moment I have a python script to insert, update or delete features from a feature layer. This works well, but I have to upload the feature class containing the new records first to AGOL and publish it as a feature layer. This is because in order to update an existing feature layer, I have to create a deepcopy from the layer containing the new records, like so: Say my layer (with the new records) is named "updater_layer". I do a query on it, using a where clause : updater_layer_query = updater_layer.query(where=where_clause) # I get the features:  update_query_features = updater_layer_query.features # and create a deepcopy which I can use to update an existing layer new_features = deepcopy(update_query_features)   But to get those deepcopy records, I first have to upload and publish a new layer with new records to AGOL.  I am wondering if there is a way to do this on the desktop so that I end up with a deepcopy list from the feature class on the desktop, without having to upload the feature class first to AGOL as a feature layer. That deepcopy (derived from the feature class on the desktop) can then directly be used to update an existing feature layer in AGOL. I looked but it does not seem possible to create a deepcopy from a feature class. Is there a way to do this, perhaps via a JSON representation of the feature class or perhaps by using a searchcursor on the feature class?    Thanks Hugo  ... View more

Hi  Items in ArcGIS Online can have their extent defined. What would be very useful is a tool that uses a polygon as input, looks for the extents of all items in a particular folder of ArcGIS Online, and returns the items if their extent intersects with the input polygon. This would be very useful for non-spatial items in AGOL such as PDFs, Excel spreadsheets and the like, because then these documents can be searched for based on the geographical area they are relevant to. For example a PDF about a national park can have its extent defined within the park area, so that it only will be returned when the input polygon is coincident with the park area.     Of course in a map you can search for items "only within the map area" but that means that for every search you have to create a map with a particular extent.  In addition the items returned are limited to those that can be added to the map such as feature layers. In a web application there are widgets that return items found within a drawn polygon, but these are limited to items already in the map that is sourced by the application, not PDFs for example.   I am wondering if it is possible to write a python script to select items that way. What I am not sure about is how to get the extents of items. And of course ideally the input polygon would be drawn on an AGOL map, which requires a JavaScript widget. At the momemt I cannot make widgets published in AGS available to AGOL, so for now perhaps something written using the arcgis.gis API would be useful i.e. something that can be used in Jupyter. Anybody has attempted something like this? Any hints how to go about it? Thanks Hugo   ... View more