POST
|
I'm currently busy with setting up a Household Income Survey for a project that we have in Dar es Salaam. I have created clusters of house locations with the help of Curtis Price based on the following post https://community.esri.com/thread/195972-how-to-group-points-into-optimal-clusters-proximity-and-timeframe My next challenge is that I'd like to create an optimal path for each cluster by creating the shortest path between each point for each cluster so that the shortest path is depicted for the surveyors to take to eliminate time wasted and the possibility of not completing any of the household surveys.. What approach or methodology could I follow to create the shortest polyline joining the points for each cluster separately that would depict the best path to take to complete the Household Income survey for each cluster separately. We have allocated 24 devices to 24 surveyors that will carry out the survey for each cluster over a period of eight days. Study Area: 24 Clusters Study Area Group1: Optimal Path Example
... View more
06-03-2017
01:22 AM
|
0
|
6
|
1859
|
POST
|
Hi Curtis The following has worked perfectly, with a few minor manual adjustments to the clusters. Thanks so much for the following. Same group for some reason, not sure why.
... View more
06-01-2017
12:23 PM
|
0
|
1
|
1306
|
POST
|
I' have extracted 2000 random building footprints within my study area and need to group the following points into optimal clusters to divide the study area for the 24 surveyors (x24 Tablets). We have 8 days to complete the survey and have roughly determined that it would take 45 minutes to survey a house, including walking from the previous house. The house locations are spread over the study area and have high and low clusters, which need to be taken into account. Any suggestions or methodologies that I could use to generate the clusters to group the points to split the study area up for the surveyors will be appreciated. I've tried used the Group Analysis tool within ArcGIS with little to no success as I have no attributes to use for any commonality between the points.
... View more
06-01-2017
08:39 AM
|
0
|
3
|
5613
|
POST
|
I'm trying to determine a initial method of determining a heat map based on the weighted distribution of bulk infrastructure within a study area (country study area). I have bulk infrastructure for the following infrastructure: electricity (high voltage to local household connections (polylines + points features) water (bulk to household connections (polylines + points features) sanitation (bulk to household connects (polylines + point features)) I'm trying to get an initial understanding of areas that have adequate infrastructure versus areas that don't, purely based on coverage at this stage as I have very little data regarding capacity. Any suggestions or approaches that I could use would be appreciated.
... View more
05-08-2017
01:23 PM
|
0
|
0
|
338
|
POST
|
Hi Dan \ Joshua As promised I've attached my updated module based on your suggestions as well as those that I received from my Stack Overflow post: '''
Created on March 12, 2017
Create a new ArcHydro Schema
File Geodatabase and Rasters
Folder
@author: PeterW
'''
# import site-packages and modules
import re
from pathlib import Path
import arcpy
import argparse
# set environment settings
arcpy.env.overwriteOutput = True
def rasters_directory(workspace):
"""Create rasters folder directory
if it doens't already exist"""
model_name = Path(workspace).name
layers_name = re.sub(r"\D+", "Layers", model_name)
layers_folder = Path(workspace, layers_name)
if layers_folder.exists():
arcpy.AddMessage("Rasters folder: {0} exists".format(layers_name))
else:
layers_folder.mkdir(parents=True)
arcpy.AddMessage("Rasters folder {0} created".format(layers_name))
def fgdb_schema(workspace, schema, dem):
"""Create file geodatabase using XML
schema and set coordinate system based
on input DEM if it doesn't already exist"""
model_name = Path(workspace).name
fgdb = "{0}.gdb".format(model_name)
if arcpy.Exists(str(Path(workspace, fgdb))):
arcpy.AddMessage("{0} file geodatabase exists".format(fgdb))
else:
new_fgdb = arcpy.CreateFileGDB_management(str(workspace), fgdb)
import_type = "SCHEMA_ONLY"
config_keyword = "DEFAULTS"
arcpy.AddMessage("New {0} file geodatabase created".format(fgdb))
arcpy.ImportXMLWorkspaceDocument_management(new_fgdb, schema,
import_type,
config_keyword)
arcpy.AddMessage("ArcHydro schema imported")
projection = arcpy.Describe(dem).spatialReference
projection_name = projection.PCSName
feature_dataset = Path(workspace, fgdb, "Layers")
arcpy.DefineProjection_management(str(feature_dataset),
projection)
arcpy.AddMessage("Changed projection to {0}".format(projection_name))
def model_schema(workspace, schema, dem):
"""Create model schema: rasters folder
and file geodatabase"""
rasters_directory(workspace)
fgdb_schema(schema, dem, workspace)
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Create a ArcHydro schema')
parser.add_argument('--workspace', metavar='path', required=True,
help='the path to workspace')
parser.add_argument('--schema', metavar='path', required=True,
help='path to schema')
parser.add_argument('--dem', metavar='path', required=True,
help='path to dem')
args = parser.parse_args()
model_schema(workspace=args.workspace, schema=args.schema, dem=args.dem)
... View more
03-13-2017
12:27 PM
|
1
|
0
|
14918
|
POST
|
I'm currently busy re-writing the following Python Module: ArcHydro Pre-processing I have just realised that ArcHydro doesn't recognize the arcpy.env.workspace or the ArcHydroTools.SetTargetLocations("HydroConfig", "Layers", "RasterWorkspace", "VectorWorkspace") '''
Created on March 13, 2017
ArcHydro Main Model
@author: PeterW
'''
# import system modules and site-packages
from pathlib import Path
import arcpy
import ArcHydroTools
# set arguments
rasters_workspace = r"E:\Projects\2016\G113386\ArcHydro\Model01\Layers01"
dem = r"E:\Projects\2016\G113386\DEM\raw"
fgdb_workspace = r"E:\Projects\2016\G113386\ArcHydro\Model01\Model01.gdb"
# check out extensions and set environment settings
arcpy.CheckOutExtension("Spatial")
arcpy.env.overwriteOutput = True
def rasters_main(rasters_workspace, dem):
""" Create Rasters as part of ArcHydro Pre-processing"""
# set raster workspace
arcpy.env.workspace = rasters_workspace
# set ArcHyro Target Locations for Raster and Vector Datasets
ArcHydroTools.SetTargetLocations("HydroConfig", "Layers", rasters_workspace, fgdb_workspace)
# create Fil Sinks raster
arcpy.AddMessage("Processing Fill Sinks")
ArcHydroTools.FillSinks(dem, "fil")
# create Flow Direction raster
rasters_main(rasters_workspace, dem) Python Module: arcpy.env.workspace and ArcHydroTools.SetTargetLocations ignored by ArcHydro Processing Fill Sinks
Traceback (most recent call last):
File "E:\Python\Eclipse\2015\arcpy64bit\src\archydro\archydro_main_test.py", line 34, in <module>
rasters_main(rasters_workspace, dem)
File "E:\Python\Eclipse\2015\arcpy64bit\src\archydro\archydro_main_test.py", line 31, in rasters_main
ArcHydroTools.FillSinks(dem, "fil")
File "C:\Program Files\ESRI\WaterUtils\ArcHydro\bin\ArcHydroTools.py", line 6805, in FillSinks
raise e
arcgisscripting.ExecuteError: You need to set either the geoprocessing Workspace or the Arc Hydro Raster Location before running the tool.
Failed to execute (FillSinks).
Error Message: I'm currently using ArcGIS 10.3.1 and ArcHydroTools 10.3.0.78
... View more
03-13-2017
09:18 AM
|
0
|
1
|
1328
|
POST
|
I have written a Python module that handles the pre-processing and ArcHydro. As part of the pre-processing it generates a set of rasters that are saved in a separate directory. The following rasters are then used as input to generate Feature Classes that are then saved in a File Geodatabase in a separate directory. I have split the following module into two functions, one for the rasters and the other for the Feature Classes. I then have a main function below that calls both the functions to pre-process the rasters and feature classes. In the first function I have set the workspace environment to the rasters directory that works well. The problem that I have is within the second function where I need to access the rasters as input but need to alter the output so that the Feature Classes are saved into the File Geodatabase without having to specify the output path manually for each Feature Class. I have attached below the directory structure of the rasters folder as well as the File Geodatabase where the Feature Classes are saved, as well as my current code. Any advice in how to handle separate input and output workspace environments without having to specify the paths manually will be appreciated. '''
Created on March 13, 2017
ArcHydro Main Model
@author: PeterW
'''
# import system modules and site-packages
from pathlib import Path
import arcpy
import ArcHydroTools
# check out extensions and set environment settings
arcpy.CheckOutExtension("Spatial")
arcpy.env.overwriteOutput = True
def rasters_main(rasters_workspace, dem):
""" Create Rasters as part of ArcHydro Pre-processing"""
# set current workspace
arcpy.env.workspace = rasters_workspace
# create Fil Sinks raster
arcpy.AddMessage("Processing Fill Sinks")
ArcHydroTools.FillSinks(dem, "fil")
# create Flow Direction raster
arcpy.AddMessage("Processing Flow Direction")
ArcHydroTools.FlowDirection("fil", "fdr")
# create Flow Accumulation raster
arcpy.AddMessage("Processing Flow Accumulation")
ArcHydroTools.FlowAccumulation("fdr", "fac")
# create Stream Threshold raster
arcpy.AddMessage("Processing Flow Stream Threshold")
maxcellsResult = arcpy.GetRasterProperties_management("fac",
"MAXIMUM")
maxcells = maxcellsResult.getOutput(0)
arcpy.AddMessage("{} Maximum Cells".format(maxcells))
stream_threshold_numcells = (int(maxcells)*0.03125/100)
arcpy.AddMessage("{} Stream Threshold".format(stream_threshold_numcells))
# Create Stream Definition raster
arcpy.AddMessage("Processing Stream Definition")
ArcHydroTools.StreamDefinition("fac", stream_threshold_numcells, "str")
# Create Stream Link raster
arcpy.AddMessage("Processing Stream Segmentation")
ArcHydroTools.StreamSegmentation("str", "fdr", "strlnk")
# Create Catchment Grid raster
arcpy.AddMessage("Processing Catchment Grid Delineation")
ArcHydroTools.CatchmentGridDelineation("fdr", "strlnk",
"cat")
def vector_main(fgdb_workspace):
"""Create Feature Classes as part of
ArcHydro Pre-processing"""
# Create Catchment
catchments = "{0}\\{1}".format(Path(fgdb_workspace, "Layers"), "Catchment") # repeating output workspace
arcpy.AddMessage("Processing Catchment Polygon")
ArcHydroTools.CatchmentPolyProcessing("cat", catchments)
# Create DrainageLine
drainageline = "{0}\\{1}".format(Path(fgdb_workspace, "Layers"),
"DrainageLine") # repeating output workspace
arcpy.AddMessage("Processing DrainageLine")
ArcHydroTools.DrainageLineProcessing("strlnk", "fdr",
drainageline)
# Create AdjointCatchment
adjointcatchment = "{0}\\{1}".format(Path(fgdb_workspace, "Layers"),
"AdjointCatchment") # repeating output workspace
arcpy.AddMessage("Processing Adjoint Catchment")
ArcHydroTools.AdjointCatchment(drainageline, catchments,
adjointcatchment)
arcpy.CheckInExtension("Spatial")
def main(rasters_workspace, dem, fgdb_workspace):
rasters_main(rasters_workspace, dem)
vector_main(fgdb_workspace)
if __name__ == "__main__":
main(rasters_workspace = r"E:\Projects\2016\01_Bertrand_Small_Projects\G113268\ArcHydro\Model03\Layers03",
dem = r"E:\Projects\2016\01_Bertrand_Small_Projects\G113268\ArcHydro\DEM2\raw",
fgdb_workspace = r"E:\Projects\2016\01_Bertrand_Small_Projects\G113268\ArcHydro\Model03\Model03.gdb")
... View more
03-13-2017
06:14 AM
|
0
|
0
|
1035
|
POST
|
Hi Joshua, Thanks so much for your suggestions, I'll have a look at the following links that you have provided and post my completed code afterwards.
... View more
03-12-2017
12:01 PM
|
0
|
0
|
14922
|
POST
|
Thanks Dan for all your help, I'll post my code once completed, that it might help others.
... View more
03-12-2017
07:41 AM
|
0
|
1
|
14918
|
POST
|
Hi Dan Thanks for the following, how would I set up the input arguments that need to be passed into my main() function. The reason I'm asking is that I'm calling two functions that both use the same input argument workspace.
... View more
03-12-2017
07:09 AM
|
0
|
3
|
14918
|
POST
|
I have recently started learning more about Python Packages and Modules. I'm currently busy updating my existing modules so that that can be run as script or imported as a module into my other code. I'm not sure how to construct my input arguments within my module and pass them to the main() function within my module. I've have written my my main() function and called it under if __name__ == '__main__' passing the input arguments. The inputs are currently hard coded to show what I'm trying to achieve. Any help in how to correctly construct my input arguments that the user will pass, which will then be passed onto the main function will be appreciated. As mentioned I'm trying to be able to use the following as a script when used directly or imported as a module into my other code and run from there. If I import it as a module would I call the main() function when importing it? Is the following structure correct in how I have written the following? Any advice is appreciated. '''
Created on March 12, 2017
Create a new ArcHydro Schema
File Geodatabase and Rasters
Folder
@author: PeterW
'''
# import site-packages and modules
import re
from pathlib import Path
import arcpy
# set environment settings
arcpy.env.overwriteOutput = True
def archydro_rasters_folder(workspace):
"""Create rasters folder directory
if it doens't already exist"""
model_name = Path(workspace).name
layers_name = re.sub(r"\D+", "Layers", model_name)
layers_folder = Path(workspace, layers_name)
if layers_folder.exists():
arcpy.AddMessage("Rasters folder: {0} exists".format(layers_name))
else:
layers_folder.mkdir(parents=True)
arcpy.AddMessage("Rasters folder {0} created".format(layers_name))
def archydro_fgdb_schema(schema, dem, workspace):
"""Create file geodatabase using XML
schema and set coordinate system based
on input DEM if it doesn't already exist"""
model_name = Path(workspace).name
fgdb = "{0}.gdb".format(model_name)
if arcpy.Exists(str(Path(workspace, fgdb))):
arcpy.AddMessage("{0} file geodatabase exists".format(fgdb))
else:
new_fgdb = arcpy.CreateFileGDB_management(str(workspace), fgdb)
import_type = "SCHEMA_ONLY"
config_keyword = "DEFAULTS"
arcpy.AddMessage("New {0} file geodatabase created".format(fgdb))
arcpy.ImportXMLWorkspaceDocument_management(new_fgdb, schema,
import_type,
config_keyword)
arcpy.AddMessage("ArcHydro schema imported")
projection = arcpy.Describe(dem).spatialReference
projection_name = projection.PCSName
feature_dataset = Path(workspace, fgdb, "Layers")
arcpy.DefineProjection_management(str(feature_dataset),
projection)
arcpy.AddMessage("Changed projection to {0}".format(projection_name))
def main(workspace, dem, schema):
"""main function to create rasters folder
and file geodatabase"""
archydro_rasters_folder(workspace)
archydro_fgdb_schema(schema, dem, workspace)
if __name__ == '__main__':
main(workspace = r"E:\Projects\2016\01_Bertrand_Small_Projects\G113268\ArcHydro\Model04",
dem = r"E:\Projects\2016\01_Bertrand_Small_Projects\G113268\ArcHydro\DEM2\raw",
schema = r"E:\Python\Masters\Schema\ESRI_UC12\ModelBuilder\Schema\Model01.xml")
... View more
03-12-2017
05:48 AM
|
0
|
7
|
38805
|
POST
|
from fnmatch import fnmatch, filter
from os.path import isdir, join
from shutil import copytree
def include_patterns(*patterns):
"""Factory function that can be used with copytree() ignore parameter.
Arguments define a sequence of glob-style patterns
that are used to specify what files to NOT ignore.
Creates and returns a function that determines this for each directory
in the file hierarchy rooted at the source directory when used with
shutil.copytree().
"""
def _ignore_patterns(path, names):
keep = set(name for pattern in patterns
for name in filter(names, pattern))
ignore = set(name for name in names
if name not in keep and not isdir(join(path, name)))
return ignore
return _ignore_patterns
# sample usage
copytree(src_directory, dst_directory,
ignore=include_patterns('*.dwg', '*.dxf')) I have found the following Python Factory Function on Stack Overflow that uses Python shutil.copytree to copy the directory and files based on the inverse of the ignore argument. It adds all file types that are not part of the include_patterns("*.dwg", "*.dxf"). The problem is that it's copying directories where there are no CAD files and resulting in empty directories. The link to the Stack Overflow post is: post Solution: I've found a very simple solution of removing the empty directories with some amendments. '''
Created on 27 Feb 2017
Extract CAD files
from within all
directories
@author: PeterW
'''
# import site-packages and modules
from fnmatch import filter
from os.path import isdir, join
from shutil import copytree, rmtree
from os import listdir, rmdir, walk
import errno
# set arguments
input_folder = r"H:\Tanzania"
output_folder = r"E:\CAD"
# create ignore patterns dynamically based on include patterns
def include_patterns(*patterns):
def _ignore_patterns(path, names):
keep = set(name for pattern in patterns for name in filter(names, pattern))
ignore = set(name for name in names if name not in keep and not isdir(join(path, name)))
return ignore
return _ignore_patterns
# copy directory structure and files based on pattern
def copytree_files(input_folder, output_folder):
if isdir(output_folder):
rmtree(output_folder)
copytree(input_folder, output_folder, ignore=include_patterns("*.dwg", "*.dxf"))
copytree_files(input_folder, output_folder)
# remove empty directories
def remove_empty_dirs(output_folder):
dirs = [x[0] for x in walk(output_folder, topdown=False)]
for dir in dirs:
try:
rmdir(dir)
except Exception as e:
if e.errno == errno.ENOTEMPTY:
print("Directory: {0} not empty".format(dir))
remove_empty_dirs(output_folder)
I found an additional solution that could work from a solution provided on Stack Overflow, but due to time constraints I wasn't able to get it to work: How do I copy an entire directory of files into an existing directory using Python? Mital Vora provided the following solution: def copytree(src, dst, symlinks=False, ignore=None):
if not os.path.exists(dst):
os.makedirs(dst)
for item in os.listdir(src):
s = os.path.join(src, item)
d = os.path.join(dst, item)
if os.path.isdir(s):
copytree(s, d, symlinks, ignore)
else:
if not os.path.exists(d) or os.stat(s).st_mtime - os.stat(d).st_mtime > 1:
shutil.copy2(s, d) The idea of the following was to create a list of directories that contain CAD drawings thereby eliminating the need of removing empty directories afterwards: Incomplete Solution: '''
Created on 27 Feb 2017
Extract CAD files
from within all
directories
@author: PeterW
'''
# import site-packages and modules
from fnmatch import fnmatch, filter
import os
from os.path import isdir, join
from shutil import copytree, copy2
# set arguments
input_folder = r"H:\Tanzania\fromDropbox\Infrastructure"
output_folder = r"E:\CAD"
def directories_list(input_folder):
dirs_list = []
for root, dirs, files in os.walk(input_folder):
if any(file.endswith('.dwg') or file.endswith('.dxf') for file in files):
dirs_list.append(os.path.abspath(root))
return dirs_list
dirs_list = directories_list(input_folder)
# function
def include_patterns(*patterns):
def _ignore_patterns(path, names):
keep = set(name for pattern in patterns for name in filter(names, pattern))
ignore = set(name for name in names if name not in keep and not isdir(join(path, name)))
return ignore
return _ignore_patterns
# relook at the following, its not working as expected
# due to copytree limitation of existing directory exists
# after first loop
def copytree_files(dirs_list, output_folder):
if not os.path.exists(output_folder):
os.makedirs(output_folder)
for dir in dirs_list:
for item in os.listdir(dir):
s = os.path.join(dir, item)
d = os.path.join(output_folder, item)
if os.path.isdir(s):
copytree(s, d, ignore=include_patterns('*.dwg', "*.dxf"))
else:
if not os.path.exists(d) or os.stat(s).st_mtime - os.stat(d).st_mtime > 1:
copy2(s, d)
copytree_files(dirs_list, input_folder)
... View more
02-27-2017
08:47 AM
|
0
|
3
|
8409
|
POST
|
I'm currently developing a Python Module to reproject shapefiles and CAD drawings (DWG, DXF) that have known and unknown coordinate systems. The code below deals with the known coordinate systems of the shapefiles of which there are 1500 shapefiles and 3600 CAD drawings. I realize that there are limitations in being able to identify the correct coordinate system of shapefiles\CAD drawings with unknown coordinate systems. I found the following two articles: Geographic and Vertical Transformation Tables Projected Coordinate System Tables Projected Coordinate System Tables: Datum Arc 1960 Projected Coordinate System Tables: Datum WGS 84 I was hoping that by comparing the spatial extent of the shapefiles\CAD drawings with the extent of the coordinate systems (i.e. Arc_1960_UTM_Zone_37S, WGS_1984_UTM_Zone_37S) I would be able to determine the correct coordinate system. The first problem is that the extent is in decimal degrees and for some reason or other they are in WGS 84. Projected Coordinate System Tables: values are based upon WGS84 If anyone has been able to come up with a workflow of identifying unknown coordinate systems based on the spatial extent of the shapefiles\feature classes\CAD drawings i'd truly appreciate any advice. '''
Created on 03 Feb 2017
@author: PeterW
'''
# import site-packages and modules
import arcpy
import glob2
import os
from pathlib import Path
# set arguments
input_folder = Path(r"H:\Tanzania2")
output_folder = r"H:\gdb"
# set environment settings
arcpy.env.overwriteOutput = True
# output coordinate system WGS_1984_UTM_Zone_37S \ 32737
out_coord = arcpy.SpatialReference(32737)
# function to create new geodatabase
def fgdb(input_folder, output_folder):
gdb = "{0}.gdb".format(input_folder.parts[-1])
arcpy.env.workspace = output_folder
arcpy.AddMessage("Creating {0}".format(gdb))
if arcpy.Exists(gdb):
arcpy.Delete_management(gdb)
arcpy.CreateFileGDB_management(output_folder, gdb)
else:
arcpy.CreateFileGDB_management(output_folder, gdb)
out_gdb = "{0}\\{1}".format(output_folder, gdb)
return out_gdb
out_gdb = fgdb(input_folder, output_folder)
# function to determine projection transformations
def projections(shp):
spatial_ref = arcpy.Describe(shp).spatialReference
trans = ""
if spatial_ref.type != "Unknown" and "Cape" in spatial_ref.GCS.datumName:
trans = "Cape_To_WGS_1984_2"
if spatial_ref.type != "Unknown" and "Arc" in spatial_ref.GCS.datumName:
trans = "Arc_1960_To_WGS_1984_3"
return spatial_ref, trans
# function iterate over shapefiles, reproject to single coordinate system
def reproject_shp(out_gdb, input_folder, out_coord):
# return shapefiles in Directory
arcpy.env.workspace = out_gdb
shp_pattern = "{0}\\**\\*{1}".format(input_folder, "*.shp")
for filename in glob2.iglob(shp_pattern):
try:
shp_records = int(arcpy.GetCount_management(filename).getOutput(0))
if shp_records > 0:
sr, transf = projections(filename)
if sr.type != "Unknown":
fcs_name = arcpy.ValidateTableName(str(Path(filename).stem))
out_dataset = "{0}\\{1}".format(out_gdb, fcs_name.title())
arcpy.AddMessage("Projecting {0}".format(fcs_name))
arcpy.Project_management(filename, out_dataset, out_coord, transform_method=transf)
except (RuntimeError, arcpy.ExecuteError) as e:
print(e)
reproject_shp(out_gdb, input_folder, out_coord)
... View more
02-09-2017
10:58 AM
|
1
|
2
|
2027
|
POST
|
Hi Mody Thanks for the following, I'll go through the following and let you know if I come right.
... View more
02-01-2017
12:06 AM
|
0
|
0
|
366
|
POST
|
I have three mosaic datasets that I've added into a newly created mosaic dataset. I periodically have to update "MDS_Ettiene_2000" by adding additional image tiles to the mosaic dataset. Currently I'm having to stop the image service, add the tiles to the original "MDS_Ettiene_2000" mosaic dataset, remove it from the grouped mosaic dataset and re add it once I've added the new images, rebuild the overviews and finally republish the image service. Can anyone suggest a better workflow that will allow me to automate (Python Module) the republishing of the following image service each time I need to add additional image tiles.
... View more
01-30-2017
11:37 PM
|
0
|
2
|
1110
|
Title | Kudos | Posted |
---|---|---|
3 | 01-16-2012 02:34 AM | |
1 | 05-07-2016 03:04 AM | |
1 | 02-01-2016 11:48 AM | |
1 | 01-30-2016 05:06 AM | |
1 | 04-21-2016 02:03 PM |
Online Status |
Offline
|
Date Last Visited |
03-04-2021
12:50 PM
|