Latest Contributions by AmirBar-Maor

Deploying the Parcel Fabric on the Cloud Why deploy the parcel fabric on the cloud? ArcGIS Pro, ArcGIS Enterprise and your DBMS can all be deployed on commercial and private clouds. Amazon Web Services (AWS) and Microsoft Azure are the most common cloud platforms. The motivation to use cloud instances is usually driven by the IT department. These are the arguments to deploy on the cloud: Cost saving will vary depending on the chosen instance types and the comparable on‑premises infrastructure. In reality, cost savings is not guaranteed. Easy and rapid deployment: no need to purchase and install any hardware. A process that can take months and even years Reduce IT resources: some organizations do not have, or want to reduce their IT resources Security and resilience: cloud providers have dedicated teams and experts that detect vulnerabilities. They also keep downtime to a minimum and offer fail-over solutions. Integration with other business systems that are cloud based Performance – this can prove valuable when the workforce is distributed or works from home. But pandemics never happen – right?! When not to deploy it on the cloud? Regulations: when the data must reside within the physical borders of a specific country.  Integration: when you have to integrate to other systems that reside on premise. Why and How? Many customers were asking for a recommended configuration and had performance concerns, so we wanted to test it for ourselves and provide a reference. To keep it real we use real data that contains around 1.5 million parcels.  In order to test the parcel fabric performance, we made sure: Attribute Rules – only use the attribute rules that are created when you create a new parcel fabric. We want to eliminate attribute rules from impacting the results. Parcel Types – too many parcel types can impact performance. This data has 4 parcel types, which is very reasonable. Virtual Users – we examined the types of workflows performed over a year, and divided them by the number of editors and working days. Each workflow was decomposed to the steps and server requests. We now mimic any number of users. By increasing the number of virtual users we are able to see which component of the system breaks first (where is the bottle neck). Is it the DBMS, or ArcGIS Server? REST API only – the requests are not sent from ArcGIS Pro, instead, we use special software that sends the requests. This guarantees that we are only testing the REST APIs (parcel fabric, versioning) without the overhead of the client(drawing, labeling…) Appels to Appels – all tests use exactly the same data. It allows us to compare the results between different DBMS, different deployments and different software releases. Keep it Real – we use real data, real workflows, real world instance sizes because we want to get real insights and provide good reference for you to use. Questions These are the type of questions we were curious to answer: How would performance on the cloud be compared to our on-premise deployment? Would any DBMS be faster than the other? Would any cloud be faster than the other? How would ArcGIS Pro perform when deployed on the cloud? What performance ‘price’ would you pay if you were to keep the ArcGIS Pro client on-premise and use the cloud for your enterprise environment? Is performance getting better or worse between releases? Choosing the right instances and Architecture A common mistake is to choose the cheapest instance, then wonder why performance is bad. So we consulted the pros and chose instances that customers will likely choose. Those are not the top tier, but the minimum that are likely to perform well. We also made sure to select the comparable instances between AWS and Azure to make sure we are comparing ‘apples to apples’. These are the Instances we used in 2022: Instances  \ Cloud AWS  Azure  ArcGIS Enterprise (Base deployment)  m6i.2xlarge 8 vCPUs x 32 GiB  80 GB Disk  esri Help link  Standard_D8as_v5   8 vCPUs x 32 GiB  80 GB Disk  DBMS:  MS SQL Server  Or   PostgreSQL  m6i.2xlarge   8 vCPUs x 32 GiB  250 GB Disk Standard_D8as_v5   8 vCPU core x 32 GiB  250 GB Disk ArcGIS Pro* g4dn.xlarge 1 GPU x 4 vCPU x 16 GiB  80 GB Disk Standard_NV4as_v4 1 GPU x 4 vCPU x 28 GiB  80 GB Disk  Blog: ArcGIS Pro on the Azure NVv4-series Performance Test harness  m5a.2xlarge   8 CPUs x 32 GiB   100 GB Disk  Windows VM Standard_D8as_v5  8 CPUs x 32 GiB   100 GB Disk  Windows VM   * The mentioned instances for Pro were the used in 2022 and they approach their end of life. For Azure, in 2026 we recommend using the NCasT4_v3 series. For AWS the G4 instances are currently recommended. You can find more useful information in this blog by @RyanDanzey    Our Configuration Assumptions we made in our tests: No High Availability  Dedicated to the editors  - there are no web apps competing with the editors, for example. All machines use MS Windows 10 OS  We installed ArcGIS Enterprise Base deployment on a single instance. For DBMS installed PostgreSQL 13 and MS SQL Server 2019 on an instances (EC2). Testing Results Some of you might want to know which DBMS outperformed the other in a specific cloud, others might know which DBMS they will use but want to know which cloud outperformed. Both clouds outperformed our on-premise physical servers test environment. MS SQL Server 16% faster on the cloud compared to on-premise deployment. Surprise: SQL Server was 29% slower on Azure compared to AWS (average response time) PostgreSQL In general, faster compared to SQL Server. Some APIs were much faster. 60% slower on Azure compared to AWS What would happen when you run ArcGIS Pro locally (on-premise) against ArcGIS Enterprise that is deployed to the cloud?  We saw an additional price of 50 ms for all requests. Depending on your network, internet speed and location relative to the cloud, expect to pay a price if you run ArcGIS Pro outside of the cloud. For that reason, we recommend to collocate the ArcGIS Pro clients on the cloud as well, with appropriate instances that support GPU. Since we repeat the test with each release, on both clouds, with 2 different DBMS, you can choose the columns you want to compare by downloading the attached Excel file. To get an impression of the performance you can watch this video:   Repeating the tests with newer versions of ArcGIS Enterprise We have been repeating our tests on premise and on the cloud with newer versions of ArcGIS Enterprise. We no focus on comparing all the permutations but make sure that performance does not degrade and that memory leaks are not introduced. We have also been adding new REST API methods. Moving a capability from the ArcGIS Pro client to the server improves performance for methods that process a lot of data. It also allow other clients (like web) to make use of the capability and support automation that does not require the ArcGIS Pro client, for example: using the server Python API. Database as a Service You can create an enterprise geodatabase on many relational databases but only the most common one have been certified by the parcel fabric team: MS SQL Server, PostgreSQL and Oracle. Why? We want to focus our testing on the most commonly used databases and adjust the list based on customer demand, not just because we can. As more and more customers move their IT infrastructure to the cloud, we have been testing common configurations: MS SQL Server and PostgreSQL on Azure and AWS. We shared the result in this presentation (minute 56). In those tests the database was install on an 'instance' with dedicated memory and virtual CPUs. You still need to install the software and a DBA to maintain the database. Cloud providers also offer 'database as a service' (DBaaS). You can access and use the database software without handling infrastructure setup, installation, or maintenance. The cloud provider manages provisioning, backups, security, and updates. Amazon (AWS) offers PostgreSQL as RDS and Aurora. We wanted to compare: PostgreSQL installed on an EC2 instance (AKA the 'baseline'). Installed on windows OS. RDS Aurora Previous tests (see above) showed that using PostgreSQL on AWS installed on an instance was the fastest. But there were rumors that using RDS is even twice as fast and that Aurora is 3 times as fast – we wanted to put that to the test. ArcGIS Enterprise specifications: version 12.0 installed on Windows 2022, with a m5a.4xlarge instance, 16 CPUs, 64 GB of RAM, and 16 instances of ArcGIS Server (ArcSocs). In all the 3 options the DBMS was using r6i.4xlarge with 16 CPUs, and 126 GB RAM. RDS and Aurora always use PostgreSQL native geometry (PostGIS), so we made sure to use it on the EC2 instance as well. Our results:   PostgreSQL installed on an instance (baseline) RDS Aurora Total requests in 1 hour (throughput) 241950 277834 (15%) 285273 (18%) Requests per second 65.5 75 (15%) 77 (18%) Average response time (ms) 557 292 (-48%) 251 (-55%)   During the tests ArcGIS Server CPU usage was between 73%-77%, memory usage between 42%-45%. Conclusions: RDS and Aurora are slightly faster but far from being 2-3 times faster. The difference between RDS and Aurora, from performance perspective, is minimal.   Let's Break It! We use specialized software to test the parcel fabric REST APIs performance called JMeter. It allows us to catch regressions, memory leaks and even stress load the system with any amount of virtual users to see what breaks first: the database or the server? How do you break the DBMS? We increased the number of virtual users from ?? to 90, reduced the delays between requests, increased the ArcGIS Enterprise to m5a.12xlarge (48 CPU, 189 GB RAM, 60 ArcSOCs) Results: we couldn't break it. The limiting factor for the installed PostgreSQL server is that is only accepts a 100 connections by default. That is a setting that a DBA can change. The RDS and Aurora support up to 5000 connections by default, and both showed a throughput of ~158K requests for a 15 minute run and could handle 270 requests per second with an average response time of 301 milliseconds. Impressive! Why Share it? The most important reason is that you have been asking for it, and as we see more organizations moving to the cloud we would like you to avoid common mistakes and make educated decisions. We share these insights in this post and we hope you find it useful. If you have any questions or requests please leave a comment below. Acknowledgements I would like to thank the experts who carried out these tests and provide these insights - Hector Martinez @HectorMartinez23  - Ken Galliher @KenGalliher1  - Chris Zemp @ChrisZemp  - Jason Camerano @JasonCamerano  - Ryan Danzey @RyanDanzey  Resources: Working from Home: ArcGIS Pro…Cloud hosted virtualization - https://www.esri.com/arcgis-blog/products/arcgis-pro/sharing-collaboration/working-from-home-arcgis-procloud-hosted-virtualization/ ... View more

"Necessity is the mother of all inventions" and the Parel Fabric keeps evolving fueled by your ideas. Join us in this meetup to express your ideas or vote on others. Register here: https://www.meetup.com/esri-parcel-fabric-meet-up/events/313329871/   ... View more