Latest Contributions by MarcoBoeringa

‎06-05-2025

Since this other thread about ArcGIS Monitor suggests the Datastore and associated database is just a regular PostgreSQL installation, the logical option to monitor your Datastore health would be any of the available (commercial) software packages for monitoring PostgreSQL database health. E.g. pgAdmin would be a logical choice. DBeaver is another option, and there others out there if you do a bit of searching. Although both of these suggested options are more about database management, and not so much about alerting or warning you when something catastrophic is about to happen. There are other options out there though, that do more of a kind of monitoring job if you do a bit of internet searching.

‎05-29-2025

Well, since you are writing about using an API, that essentially already means programming... I guess you could use the "Python Client for Google Maps Services" and ArcPy to write your own implementation as Python script to get access from within Pro: https://github.com/googlemaps/google-maps-services-python

‎05-20-2025

@TanuHoque Thanks, I overlooked the setting, and hadn't really expected it to reside in the "Map and Scene" section of the Options. I expected this somewhere in "General"/"Geoprocessing" or maybe "User Interface". But at least I am glad to see it implemented! Now I need to get a horrid bug in Pro 3.4+ fixed that has plagued and totally blocked my workflow ever since the Pro 3.4 upgrade, and forced me to stick to Pro 3.3 to get anything done, so I will unfortunately not be able to take advantage of it anytime soon...

‎05-20-2025

Honestly, that hardly sound like anything needing "special configuration". It is a tiny database by all measures nowadays. As said, I have run a PostgreSQL database with >2.5 B records on a single server. Secondly, any PostgreSQL configuration is also tightly coupled with the actual hardware specs. E.g. something like the 'shared_buffers' setting is almost always recommended to set as a percentage of your RAM, so unless you already know or share your configuration here, there is no sensible recommendation. That said, the three things that really will make a difference: - Do no run from HDD, use NVMe attached PCIe SSDs. It is hardly a question nowadays, but up to maybe five years ago, I still regularily read about people putting their database on HDD, which kills performance compared to modern NVMe SSDs. - Switch off 'synchronous_commit' in the postgresql.conf file. This is probably the single most effective configuration you can make if you do not need it (no replication to standby servers) to enhance performance on writes / UPDATEs. - Set an appropriate 'shared_buffers' setting to give PostgreSQL breathing room in RAM. All other settings are likely to only contribute minor, especially in a minor use case like yours.

‎05-20-2025

@TanuHoque Is this really implemented in Pro 3.5, or is this still "In Product Plan" and will be implemented in 3.6, because I do not see such an option to switch between "Clause" and "SQL" mode application wide in Pro 3.5? I assume it will be available somewhere in the "Project/Options", but in Pro 3.5, there still isn't such a setting as far as I can see it. Or am I missing something obvious?

‎05-14-2025

I am not sure what you mean. If the hosted database is based on PostgreSQL, there is no "magic spell" to make it better than any other PostgreSQL installation. In the distant past, ArcSDE - one of the backbones of ESRI database technology - supported the 'RAW' spatial storage, which, due to being directly convertible to the ESRI internal geometry representation in applications, was generally regarded as a more performant option over databases 'native' geometry implementations for ESRI software (although ESRI never really made much noise about this or admitted to it). However, 'RAW' is past in favor of 'native' or ESRI ST_Geometry geometry storage in all databases AFAIK. If with your remark, you are just referring to PostgreSQL configuration as done with e.g. the 'postgresql.conf' file, I suggest you start delving into the wonderful world of PostgreSQL configuration and read up about the subject. There is definitely stuff you need to adjust in there, as the default settings are not suited for large (Post-)GIS databases, but just the absolute minimum to get a server running on minimal hardware.

‎05-13-2025

Since the error states a "time out", has anything changed lately in your configuration to bog down your server, and maybe cause such a time out? Since you already run this for years, possibly on the same hardware, your hardware may no longer be able to fully keep up, and cause time out on some requests. Certainly, if your database and usage has grown considerably over the years, maybe the server is running against its limits. Have you checked that?

‎05-13-2025

It depends on what type of work you are going to do with the database, and your current connection to your local database, and so what your expectations are. E.g. if you just want to do an occasional update of a limited number of records and / or view the data, or if you need fast low latency access because you are going to re-write millions of records on a regular basis. Remote databases can have a very large latency (the time to do a round trip from you local machine to the server and back), which can kill the performance if you need to fire thousands or even millions of requests to it. In case you run a remote database and need low latency access, you may need a virtual machine running ArcGIS Pro in the same data center or area, so as to avoid the cost of the round trip. E.g. I personally run a local PostgreSQL database filled with the entire planet's worth of OpenStreetMap data on a professional dual CPU 44 core HP Z840 workstation. In the most outrages configuration, I filled it with Facebooks now defunct "Daylight distribution" of OpenStreetMap, which integrated an additional >1 Billion buildings of Google "Open Buildings", causing the largest table to exceed >2.4 Billion records. Yes, you read that right, billions! I subsequently process and update hundreds of millions of records in derived tables. I certainly wouldn't want to do this against a remote database with latencies running in the 100's of milliseconds. It would take ages. Running this all locally against PCIe NVMe drives, I am capable of updating hundreds of millions of records per hour from ArcGIS Pro (although batched in sets of tens of thousands to reduce the number of requests). This would be unthinkable against a remote database, unless running as suggested above, a VM with Pro in the same data center as the remote database. Finally, running a local database, although a burden, can learn you a lot about the proper configuration and running of a PostgreSQL database and server, knowledge that you may not gain with a remote database and its fully pre-installed configuration (which may or may not suite your use case or be adjustable to your taste or needs).

‎05-10-2025

The CVE seems to concern only one specific library regarding Avro format, which doesn't seem present in the Pro install (see my listing below which slightly differs from yours but does not show a file name with 'avro'). These found modules are different ones, and as far as I can tell not involved in the CVE. I guess the affected module is called simply 'parquet-avro-<VERSION>.jar', but I didn't see the actual full filename listed in the CVE.

‎05-02-2025

Those ping latencies do not look to good, although expected across larger distances with machines in different regions. My local network setup, which is of course not comparable as being best case, has <1ms latency on ping to my VM running PostgreSQL. I think this explains a lot of your issue, and George's comment about running ArcGIS Enterprise and Pro on the same VNet as PostgreSQL: https://innerjoin.bit.io/the-distance-dilemma-measuring-and-mitigating-postgres-network-latency-76f6cd1a6c57

‎04-28-2025

Of course, OpenStreetMap was already updated a year ago 😉, but you likely already know that: Relatie: ‪Blue Heron Lake‬ (‪12908‬) | OpenStreetMap

‎04-23-2025

@_____ wrote: "The bugs we can report and many of them get fixed with the next update." I don't believe that for one second. That hasn't been my experience, or any other persons experience that I know. In fact, it's a meme online how terrible Esri is at doing bug fixes in a timely manner. I do think this quote hits on one important aspect of the issue raised here: ESRI rarely if ever backports bugfixes to the previous major release. This causes major issues, as it means that, even if a bugfix is introduced "in the next release" (which I agree with you is definitely not a given, and lots of recognized bugs are only fixed after maybe two or three major releases - which is 2-3 years), it may well end up in a version of ArcGIS with a new bug affecting the same functionality. If that sounds improbable: I have a tool that heavily relies on the Maplex label engine. When ESRI made some major changes to Maplex in ArcMap 10.3 and if I remember well Pro 1.4, I was confronted with unusable labeling results. Reporting back confirmed the issue, and a bugfix was introduced in the next major release, only to introduce a new major issue! I finally had to wait until ArcMap 10.6 and Pro 1.6 for the Maplex labelling engine to be fully sorted out and give proper results... Best software practices regarding bugfixes require fixes to be backported to previous releases, at the very least the last major release cycle. A fix for an issue in Pro 3.4 should not require upgrading to Pro 3.5, but also be in the Pro 3.4.x cycle. If you don't do that, you end up with a potentially perpetually broken product, as each new release also inevitably introduces new bugs, and the version users actually use (often the last major one), never gets fixed properly to the extent of being really usable. I really think if ESRI sorted out the way they handle these issues and dramatically reduce their bugfix backlog, both ESRI support employees and users would see a dramatic drop in the time needed to deal with issues, as rather than seeing the same - unfixed - issue pop-up for the tenth time in the support process, support employees and users would only primarily have to deal with true new issues. There is so much wasted time in the current process, that ultimately also holds back developers and waists their, as some repeated issues will ultimately make it on their plate as well.

‎03-25-2025

@RyJohnsen wrote: I'm on windows 11. I initially used TreadPoolExecutor, but I'm not I/O bound and the GIL caused it to be much slower than ProcessPoolExecutor. Yes, it all depends on how much actual CPU work versus IO you are doing whether ThreadPoolExecutor or ProcessPoolExecutor is the best solution (and available resources in terms of RAM etc., although that starts to become a pretty mood discussion on modern power desktops, that usually have plenty of resources). However, in my experience, it is pretty hard to become purely CPU and not IO bound. You really need to do significant work to be CPU bound or limited.

‎03-25-2025

Actually, for Python, the maximum number of workers that can be launched on Windows is 61 according to the documentation for concurrent.futures: https://docs.python.org/3/library/concurrent.futures.html By the way, if you don't want irritating pop-ups of command windows during execution of processes, you can add the code below, which will change the python executable and prevent pop-ups and execute like if it was a thread (although process): import sys multiprocessing.set_executable(os.path.join(sys.exec_prefix,'pythonw.exe'))

‎03-21-2025

Can't help you with the specific error message that you receive, but a couple of remarks: - If you are still running Windows 10, you may not be able to go beyond 64 processes due to limitations with Windows Processor Groups: https://bitsum.com/general/the-64-core-threshold-processor-groups-and-windows/ It is still not entirely clear to me personally, if and how Windows 11 handles this and if it allows true unlimited process numbers. There were changes though to this aspect of Windows, but I am still on Windows 10 so can't verify it. - Do you really need processes? If you connect to a database, the database may well turn a threaded application into something close to a processes based multi-processing solution, however allowing you to use threads in your Python application. E.g. in the screenshot below, I am using a concurrent.futures.ThreadPoolExecutor to execute up to 44 threads with SQL statements to generalize data on the database using PostGIS commands. As you can see from the inset of the remote desktop to the server, this pushes the PostgreSQL database to a full 100% CPU usage, without processes.

Online Status	Offline
Date Last Visited	a month ago

My Ideas

Latest Contributions by MarcoBoeringa

Re: ArcGIS Monitor 2023.2 - No option to add Datastore as component

Re: Can Google Maps API be used in ArcGIS Pro?

Re: Application wide setting / option to switch between "Clause" and "SQL" mode for expressions

Re: Is there a real different between hosted and local Postgres geodatabase

Re: Application wide setting / option to switch between "Clause" and "SQL" mode for expressions

Re: Is there a real different between hosted and local Postgres geodatabase

Re: ArcGIS Server Service randomly returns a "Invalid Token"

Re: Is there a real different between hosted and local Postgres geodatabase

Re: Apache Parquet < 1.15.1 Remote Code Execution (CVE-2025-30065) in ArcGIS Pro

Re: How slow is too slow when using ArcGIS Enterprise with a PostgreSQL backend?

Re: How do I report a geographic name change for ESRI basemaps?

Re: Why is Pro so terrible after all these years?

Re: Arcpy concurrent.futures license issues

Re: Arcpy concurrent.futures license issues

Re: Arcpy concurrent.futures license issues

Re: Hinduja Swiss (Switzerland) IT-Professional-Ho...

Re: Issues with Editors using the new threading op...

Re: Issues with Editors using the new threading op...

Re: Issues with Editors using the new threading op...

Re: Issues with Editors using the new threading op...