Latest Contributions by MarcoBoeringa

John, unless the tool was specifically designed to take advantage of multiple cores, as witness-able by a "Parallel Processing Factor" environment setting mentioned in the ArcGIS Pro tool Help under the Environments heading, you will never see all cores fly. You also need to be aware that some processes simply can not reasonably be split up, and may never have a simple "Parallel Processing Factor" setting. Of course, if you are able to cut up your dataset in smaller pieces, and you have ArcMap installed, you could launch multiple ArcMap sessions and have each of those process a partial dataset. Unfortunately, although I think ESRI is working on it as many people complained about this, it is currently not possible to launch multiple ArcGIS Pro sessions, you can only have one instance of ArcGIS Pro. So, if you want to test your machine to the max, look in the ArcGIS Pro Help for the topics about Parallel Processing, and see which tools support it. It is senseless to try just any tool, because most of the hundreds of tools in ArcGIS won't support it. Some of the tools supporting it are in the Analysis toolbox (e.g. Clip): Analysis toolbox history—Help | ArcGIS Desktop  Most of the tools of the Geostatistical Analyst support it: Parallel processing with multiple CPUs—ArcGIS Pro | ArcGIS Desktop  Probably also multiple tools in the Spatial Analyst toolboxes. And if you have ArcMap installed - because this tool is not yet available for Pro - and have any interest in OpenStreetMap data, the latest toolbox of the ArcGIS Editor for OpenStreetMap has an OSM File Loader (Load only) tool for importing your data into a File Geodatabase, which can take advantage of multiple cores. Don't forget to set the parallel processing factor to the number of available cores though, e.g. in your case 18 under the "Environment" settings. You can download e.g. a country wide extract from Geofabrik that can serve as input for the tool. Make sure to download an .osm.bz2 zipped file, because that contains a zipped OSM XML file that is the type of input needed for the tool. Note that Windows can not unzip these by default, a program like 7-Zip can, and you will need to unzip the file before you can use it as input for the tool. This last tool will definitely make your machine run for its money... It is the tool I have used frequently with great results and with multiple cores flying. Note that this import tool goes through multiple stages, three lengthy ones of them using multiple cores, inbetween are processes that are necessarily single core (e.g. an Append operation on database tables). EDIT: Note that this tool also needs the Scratch Workspace from the "Environment" settings properly set. I would definitely recommend you to set the scratch workspace to the fastest SSD storage you have available, especially when trying to import with 18 cores. If you would do that using a traditional hard drive as Scratch, you might run into a serious IO bottleneck (and likely prematurely ruin the disk in the longer run). EDIT2: Also note that the initial step of the OSM File Loader (Load only) tool is to break up the original OSM XML file in smaller pieces for the processing on each core. This process is single core and can take time on large (e.g. country size) extracts. Only after breaking up the original XML file, will it start doing parallel processing. ... View more

As Dan_Patterson‌ already pointed out, this tool does not (yet) support the parallel processing factor environment setting, so it can't take advantage of multi-core CPUs. The fact that your processor has next to none activity, is probably caused by Windows only showing the combined activity of all cores. Since you have 12 cores on a single CPU, the CPU's usage is likely less than 10%, but that single core used for the actual task running at 100%. This is one of those cases where "less is more". It is likely a moderate 6 core CPU with higher clock frequency, could do the job faster. Although I haven't had that much need for the particular tools supporting the parallel processing, I have frequently used a single tool that did support it, and yes, all 4 cores of my machine were used to the fullest, so the CPU usage at 100% and each individual core at 25% of that total. Additionally, individually spawned Python processes for each core used up quite some memory. As to memory: well, most people actually find it desirable if the process uses less RAM. And it may actually be that your particular dataset and process flow, simply doesn't require that amount of RAM to execute, due to dataset characteristics and the way it is programmed. Lastly, there can be an IO bottleneck, if you have a slow network connection to the location where you access the original data or store the result. ... View more

@Daniel Matranga, You are not actually giving any real information here. Since you are outputting to a raster format (which I wasn't), the export may be highly influenced by the actual dpi resolution you choose. A 1200 dpi export is likely to cost 16 times longer on export than a 300 dpi export is (1200/300 * 1200/300). So: what resolution do you use? Additionally, transparency may add significantly as well. I haven't yet tried this seriously in Pro, but in ArcMap, adding transparent vector layers above a raster datasets essentially caused many extra redraw cycles, as ArcMap needed to rasterize the vector data and probably merge resulting pixel values to create the final output map. In ArcMap, I have seen dramatic increases in export times when using transparency. Also, make sure you use appropriate field indexes on all of your layers if you use Definition Queries to select subsets of your data, or if you have complex SQL Queries for you label settings. ... View more

Maxim, One other last question: did ESRI make changes to the text halo rendering and export output going from Pro 1.2 to 1.3 (or in one of the previous releases)? Because I now remembered having looked in detail at some of the PDF output from Pro in one of the previous versions (maybe 1.1 or so), and specifically text halos did also render poorly with jagged edges at lower (e.g. 300 dpi) resolutions in PDF output. There was a marked improvement in halo quality when using the higher dpi settings. Now, with ArcGIS Pro 1.3, I don't see this, the text halos look already smooth at 300 dpi. ... View more

Hi Maxim, Yes, I am aware of the differences in the display pipeline of ArcMap and Pro. I already explained the reason I have been exporting to such a high dpi resolution (despite 100% vector output): it primarily signficantly improves text halo rendering in the PDF output of ArcMap. Without it, the halos will suffer from poor quality in the ArcMap output. Like you explained, and as I read already before, ArcGIS Pro does not suffer from the same issue. I nonetheless thought it good to ask you to keep the same high dpi value in both applications, to avoid introducing yet another variable in the mix. Of course, no-one in his right mind is going to use 3600 dpi for exporting to an A0 layout containing raster data as well. That certainly won't fit in 6MB output file size... The highest I usually export with raster data included, is some 750 dpi at most. It is good to read by the way, that you get similar "1 minute" benchmarks for Pro 1.3. It seems to confirm a potential or likely edge over ArcMap. ... View more

Maxim (and others here in the thread), I just managed to install ArcGIS Pro 1.3. This is really preliminary as a first test, but it seems there is a huge difference: Test results on the A0 topographic map layout that I made available to Maxim: - ArcMap 10.4.1: 2 minutes export time for a 3600 dpi output 100% vector PDF - ArcGIS Pro 1.2: 10 minutes export time for a 3600 dpi output 100% vector PDF, so 5x longer - ArcGIS Pro 1.3: <2 minutes export time for a 3600 dpi output 100% vector PDF So, if I get similar results on other PDF exports, it seems Pro 1.3 is now on-par or even slightly faster than ArcMap. EDIT: a second test of the same layout even seems to indicate an export time of <1 minute in Pro 1.3! ... View more

I am not to surprised, I have had pretty terrible export times for PDF too in ArcGIS Pro. Exports that would take maybe 10-15 minutes in ArcMap, can easily take one or two hours in ArcGIS Pro. I have had a couple of really complex topographic map style layouts even taking up to something like 6-10 hours to export (imported from ArcMap mxd documents by using the option for that in Pro). ArcMap never took even remotely that long for exactly the same layouts . I really hope future versions of Pro improve on this, because it makes exporting a real pain in ArcGIS Pro, and that is not even considering automated batch exporting via something like Data Driven Pages. This is all on a true 4-core Core i5 desktop with 16GB RAM. ... View more

@Bruce Harold: The WFS connection I tried to access is on this address: https://geodata.nationaalgeoregister.nl/digitaaltopografischbestand/wfs It is a governmental extremely high detail +/-1:1000-1:5000 scale map of the entire highway system in the Netherlands. The WFS is by default limited to a max request size of 15k features according to the Dutch metadata page. As said, when I enter this URL in the ArcGIS Pro WFS To Feature Class tool, it does recognize the layers ("punten", "lijnen" en "vlakken" for points, lines and polygons in Dutch) in Select Feature Type To Extract parameter, but can't convert them. NOTE(!): data is almost certainly in the Dutch coordinate system, listed as "RD New" under Projected Coordinate Systems / Europe / National Grids in ArcGIS Pro (and ArcMap). I would recommend to zoom in on a ESRI Basemap near e.g. Amsterdam, look for a nice junction, and then set the coordinate system to RD New and set an appropriate transformation if needed for e.g. the basemap (not needed for the WFS as most like RD New), ArcGIS has one. Subsequently set the Environment for the WFS To Feature Class tool to "Current Extent", to avoid downloading an excessive amount of data. This is what I did, I certainly did not attempt to download the entire Netherlands, just a single highway junction, which should be well below the 15k features download limit. ... View more

Looking at this ArcGIS Pro Help page: Connect to a GIS server—ArcGIS Pro | ArcGIS for Desktop It is at least clear that directly connecting a WFS is not (yet) supported. There is no mention of WFS servers there, just WMS and TWMS as OGC options. This page though, does reference WFS, but shows you must convert the data to a Feature Class first before being able to use it (which the data operability extension maybe also does?): An overview of the From WFS toolset—Conversion toolbox | ArcGIS for Desktop It is in the Conversion toolbox. I just gave it a try with a WFS link I picked from the internet, but although it correctly sees the available layers of the service, it failed when trying to do the actual conversion. Don't forget to set the Environments as well, it probably defaults to current extent, but you don't want to start loading a ton of WFS data for a service with detailed data covering a large area. ... View more

Reading your post, you are working in "Layout View" in ArcMap. In Layout View, you can make objects "active" or "inactive" by setting the Draft mode. Select e.g. the data frame, and right-click it. Then choose Properties / Frame TAB. Now unselect the Draft mode checkbox in the dialog and click OK. The data frame will become entirely blue and no longer show the true contents. This allows you to work quickly without the burden of intermittent re-draws of the data frame. Note that you can also set other objects, e.g. the legend, to "draft mode". ... View more

Hi Dan, Well, I am even more baffled as to what is going on. The code definitely worked in 10.3.x and below. But now, I am even more confused, because when I attempt to run the arcpy.SaveToLayerFile_management tool directly by either opening the tool from ArcToolbox, or by typing the command in the Python window of ArcMap, it just works... no issue at all! I have no clue why the same code, without making any changes at all but only upgrading ArcGIS from 10.3.1 to 10.4.1 (I also previously tried 10.4 as you noted in the older post), makes the code go bust upon running it in the new version. I have attempted many things by now, and verified the objects / variables going into the command, but don't see anything wrong up to now. ... View more

;-( My 'solution' breaks the ability to save a layer file with relative paths, as is possible with the arcpy.SaveToLayerFile_management tool. I need this... The lyr.save() and lyr.saveACopy() methods seem to completely ignore current map document settings for relative or absolute paths. ... View more

I have solved it by switching to the lyr.saveACopy() method of a layer. I still think this is a bug introduced in the 10.4.x line of ArcGIS for Desktop. The arcpy.SaveToLayerFile_management tool should be callable from arcpy, and worked without issue prior to 10.4.x. ... View more