Latest Contributions by ArchitSrivastava

@JoshuaBixby  Completely reasonable point. I’m not saying the geodatabase type by itself is the cause. I’ve just noticed in a few environments that services backed by a centralized SDE tended to look more consistent across nodes, while file-based data deployments sometimes showed more variation. Either way, agreed it’s worth validating with a couple targeted tests. On your question about the full REST call: yes, the REST call I meant is the basic layer resource endpoint: https://<cname>/arcgis/rest/services/General/Data/MapServer/86?f=json&token=<token>  For troubleshooting, it also helps to hit each node directly and compare headers: https://<machine1>:6443/arcgis/rest/services/General/Data/MapServer/86?f=json&token=<token>  https://<machine2>:6443/arcgis/rest/services/General/Data/MapServer/86?f=json&token=<token>  Also, just to be clear: keeping the token stable during testing is simply to avoid changing the URL between requests (general HTTP caching behavior). I couldn’t find specific Esri documentation describing how ArcGIS Server generates ETags or whether the token is included as an input. ... View more

@NicolasGIS  Thanks for the added screenshots, they help a lot. What you’re seeing is expected in a multi-machine ArcGIS Server site, especially when you access it through ArcGIS Web Adaptor, since Web Adaptor distributes requests round-robin across the machines. So the same URL can hit different nodes on refresh. https://enterprise.arcgis.com/en/web-adaptor/latest/install/iis/multiple-machine-deployment-with-arcgis-web-adaptor.htm  ETag + Cache-Control + client-side caching all work together: the browser decides whether it can reuse a cached response based on Cache-Control: max-age, and once that expires it revalidates using ETag (If-None-Match). If ETags differ across machines, you can see mixed 200 and 304 results even for identical responses. https://enterprise.arcgis.com/en/server/latest/publish-services/windows/improve-map-service-display-performance.htm  On cacheControlMaxAge: Esri documents that this value is returned as max-age by the Export operation and the Tile resource. That matches your screenshots where tile/export-style requests honor max-age, while the layer metadata endpoint (.../MapServer/86?f=json) can still return "max-age=0". https://developers.arcgis.com/rest/enterprise-administration/server/editservice/  https://enterprise.arcgis.com/en/server/latest/publish-services/linux/setup-caches-for-client-consumption.htm  Your point about SDE vs file geodatabase is valid. In real deployments, services backed by SDE often behave more consistently across nodes (centralized backend), while file geodatabases can show more node-to-node differences depending on how the data is deployed (replicated per machine vs shared storage), which can show up as different ETags. Additionally, I agree it’s worth logging an Idea. Multi-machine behavior being more transparent client-side is a reasonable expectation, and even if it’s “by design,” it’s still a good enhancement request. https://community.esri.com/t5/arcgis-ideas/ct-p/arcgis-ideas  ... View more

Hello @JoshuaBixby , This understanding comes primarily from hands-on experience managing multiple ArcGIS Enterprise environments over time. I’ve seen this behavior across several multi-machine ArcGIS Server sites, configured with Web Adaptors and load balancers, using different browsers and client applications. It’s not based on a single observation, but repeated testing while troubleshooting client-side caching and REST performance. That said, this behavior is not driven by ArcGIS Server alone. It’s the interaction between ArcGIS Server headers, the Web Adaptor’s request routing, and browser caching behavior that produces what you see. The cacheControlMaxAge property works in combination with how the browser honors caching and validation headers. If someone wants to reproduce and validate this, two simple test scenarios help isolate the behavior (assuming a Web Adaptor–based site): Test scenario 1: Browser-side caching behavior (isolate browser caching and ETag handling.) Disable browser caching completely (for example, using Chrome DevTools). Reference (Follow the one with 2120 upvotes): https://stackoverflow.com/questions/5690269/disabling-chrome-cache-for-website-development  Open a REST endpoint such as a layer or export request. Send the same request multiple times. Observe that responses consistently return 200, regardless of machine routing. This confirms the browser cache is a contributing factor. Test scenario 2: ArcGIS Server caching behavior (observe the effect of ArcGIS Server cache headers.) Disable client-side caching for the service by setting cacheControlMaxAge to 0. Reference (official Esri documentation or you can control this from Service confirguration in ArcGIS Server Manager): https://pro.arcgis.com/en/pro-app/3.5/help/sharing/overview/configure-web-tile-layer.htm#:~:text=Optionally%2C%20check%20the,10.5%20or%20earlier.  Make sure browser caching is enabled. Send repeated REST requests through the Web Adaptor. Observe that caching behavior changes and ETag validation is no longer leveraged in the same way. In a Web Adaptor setup, requests are distributed in a round-robin manner by design. When caching is enabled, hitting the same machine with a valid ETag may return 304, while hitting another machine may return 200. This is expected given machine-specific ETag generation and client-side cache validation. Different environments, browsers, and request patterns can show slightly different results, which may explain why behavior appears inconsistent across sites. Keep in mind that changing these caching properties can have a direct impact on service performance. Disabling or reducing client-side caching may cause the service to fetch data or tiles on every request instead of reusing cached responses, which can increase load on ArcGIS Server and lead to slower response times. ... View more

Hello @NicolasGIS, Yes, as per my understanding, this is expected behavior in a multi-machine ArcGIS Server site. How ArcGIS Server uses ETags ArcGIS Server generates ETag headers per machine. Even when the REST response payload is identical, different machines return different ETags. In a load-balanced setup, this leads to alternating 200 and 304 responses. When accessing services through the Web Adaptor, this is also seems to be by design. The Web Adaptor distributes requests in a round-robin manner across ArcGIS Server machines. If a request is routed to the same machine where the cached response is still valid and the ETag has not expired, the server returns 304 Not Modified. If it is routed to a different machine, the ETag does not match and a fresh 200 response is returned. When a request is repeated and the maximum age of the cache has not expired, the browser uses the cached response without sending the request to the server. If the maximum age has expired, the browser must send the request to the server and set an IF-NONE-MATCH header with an associated ETag value corresponding to the response in its cache. ArcGIS Server evaluates the request and uses the ETag value to determine if the response has changed. If the response from the server is different than the copy on the browser, the server sends a new response to the browser. If the response is identical to the copy on the browser, the server alerts the browser to continue to use the response in its cache. (Quoted from Document)  How to influence caching behavior Rather than relying on ETags, Esri recommends controlling client-side caching using the cacheControlMaxAge property. This defines how long (in seconds) a response can be cached before revalidation, providing predictable behavior in load-balanced environments. It’s best to test different cacheControlMaxAge values to determine what works best for your usage patterns. For more details, see the official ArcGIS Enterprise documentation on setting up caches for client consumption:  https://enterprise.arcgis.com/en/server/latest/publish-services/linux/setup-caches-for-client-consumption.htm#ESRI_SECTION1_C8C2AA1672F34194817DD3BC793F52F5:~:text=a%20map%20service.-,Enable%20clients%20to%20cache%20locally,-When%20displaying%20cached  cacheControlMaxAge property details : https://enterprise.arcgis.com/en/server/latest/publish-services/linux/setup-caches-for-client-consumption.htm#ESRI_SECTION1_C8C2AA1672F34194817DD3BC793F52F5:~:text=Define%20the-,cacheControlMaxAge%20property,-When%20clients%20send  Hope it helps! ... View more

Hello @CodyPatterson, As you mentioned, the environment uses "standard "arcgis" service account", there should not be a lot of impact after the admin’s personal domain account is disabled. Portal, Server, and Data Store rely on the service account, not on the user account. However, it would be great to run a  couple of checks though (as I have encountered such scenario before): Check the Authentication Method If Portal uses IWA/Active DirectoryIf admin's domain account exists as a Portal login identity, Verify: The admin's user is not the only Portal Administrator Make sure another AD user or AD group already has admin privileges.                  If his account gets disabled and and that is the only admin, the environment would be locked b. If Portal uses built-in accounts Then all good, nothing to worry 2. Verify all core Windows Service : On every machine (Portal, Server, Data Store etc.) confirm that services run under "arcgis", not under personal domain account (you mentioned it's arcgis but worth checking once) 3. Check Portal item Ownership : If the domain account ever logged Portal with his domain account and created: Hosted Service, maps or apps, Groups, Database Connections (These would get harder to manage after the domain account is disabled) 4. Check permission on Shared Directories : Check that the "arcgis" service account, not his domain account, has permissions to: ArcGIS Server config-store, Server directories, Portal content directory, Any shared certificate folders, Shared file-based data sources. If his domain account has any unique permissions, replace them with the "arcgis" account or and AD group and test things once. 5. Scheduled Tasks, Scripts, and Automation (these are tricky ones to find as well) : If the AD user created, "Windows task Scheduler jobs, Python Scripts, batch jobs and Backups running under his user. Change the run-as to "arcgis" or any other AD account. 6. Check Database Connections: Check registered database connections, make sure none are registered with his AD account. Additionally, as @DEWright_CA , you can always reach out to ESRI support in case of any issues post the admin's AD account is disabled, they are great and guiding you through those kinds of events if you need. Hope it helps! ... View more

Hello @NarayanHamde , To better understand what’s happening, could you clarify a couple of things: What version of ArcGIS Enterprise are you using? How large are the multipatch layers or the SLPKs you’re working with (in GB)? When you say some layers fail with waittimeout, does this show up in the ArcGIS Server logs at the SEVERE or WARNING level? It would be great, if can share the below details as well: Server Logs – Set logs to DEBUG temporarily and review what happens at the moment publishing fails. This can confirm if it’s a timeout, memory, or cache issue.(Be sure to switch back to SEVERE or WARNING afterward, since keeping logs at DEBUG can quickly consume storage.) Machine Resources – Large 3D publishing jobs can require a lot of CPU/RAM. Do you know how many cores and memory your ArcGIS Server site has available? Sharing these details will really help in narrowing down the root cause of the issue. ... View more

Hello @vaishalikulkarni00 , I wanted to provide a quick update. I have not been able to identify any viable workarounds for this issue so far. However, I’m continuing to explore possible solutions. I’ll keep you informed of any progress. ... View more

@vaishalikulkarni00 , It does seem to be a different behavior than expected. However, I think that the configuration being on the Admin page is correct and the documentation for 11.1 does say the same, kindly find the details below: Create a geoprocessing service webhook: https://enterprise.arcgis.com/en/portal/11.1/administer/windows/create-webhooks.htm#ESRI_SECTION1_A94ED6E140A4426A8EC72C9A03786C5F  Create a feature service webhook: https://enterprise.arcgis.com/en/portal/11.1/administer/windows/create-webhooks.htm#ESRI_SECTION1_A3A3B5AE99F446C7B48ABE051E79F0C6  Where as the documentation for 11.4 suggests different workflow: Configure service webhooks: https://enterprise.arcgis.com/en/portal/latest/administer/windows/create-webhooks.htm#ESRI_SECTION1_A47832EBBC414E7C8250FFB95CCF000E  My guess is that  with new releases the behavior has been made user friendly as expected.  Additionally, I think we should not be looking for "Create Webhook" option on the REST Admin API.  I would try to look for a way which can allow you to create a Webhook with the custom role configuration and keep you posted if I find anything. Furthermore, I think reaching out to ESRI support at this point would also be a good idea. Hope it helps! ... View more

Hello @vaishalikulkarni00 , As the Admin is able to create Webhook, I would also assume that the SD (Service Definition) is OK. However, let's try to cover all our bases and try the following: Try by updating some other capability as "Sync" would require other checks on the data configuration end, can you try disabling "editing" capability or pick any other example from the below document: https://developers.arcgis.com/rest/services-reference/enterprise/update-definition-feature-service/   Try publishing a "Hosted Feature Service" with a very small data and try the Webhook creation workflow on that. Check the "Sharing" setting of the service (Shouldn't matter as you are the owner but still a check wouldn't do harm) Check how the "Custom Role" was created as I use the default publisher role as base and then just add "webhooks" capability (as shown in the video) maybe some required capability is missing from the role. In the meantime, I will try to search more on this and share details. Hope it helps! ... View more

Hello @vaishalikulkarni00, I ran some tests on my side by configuring a similar role on version 11.1 and 11.3 (with and without Azure AD config) both and followed the workflow mentioned in the document below: Create (Webhooks) | ArcGIS REST APIs | ArcGIS Developers Key point being update the URL to Rest Admin API, I am sharing a sample below: REST API URL: https://sampleserver6.arcgisonline.com/arcgis/rest/services/CommercialDamageAssessment/FeatureServer  ArcGIS REST Admin URL (added the bold section) https://sampleserver6.arcgisonline.com/arcgis/rest/admin/services/CommercialDamageAssessment/FeatureServer  The above URL should allow you to reach to "Update Definition" option and then follow the documentation mentioned above. Furthermore, I also noticed the following: If you are owner of the "Feature Layer" (Hosted and Referenced both): No redirect occurs If the item is owned by another user (Hosted and Referenced both): Redirect occurs and tries to generate token Hence, I think that the redirect has nothing to do with the "Azure AD" configuration. It is to validate the access of the item (assumption). Furthermore, can you try taking the ownership of the item or try doing the workflow on an item owned by your username? Please let me know how it goes, in the meantime I will try to find any other workaround. Hope it help! ... View more

Hello @vaishalikulkarni00 , I assume that your assigned role is a built-in role, not a custom one, and by default, the Webhooks capability is only available to the Administrator role. Please refer to the documentation below for more details: Full Document: Privileges granted to roles—Portal for ArcGIS | Documentation for ArcGIS Enterprise Webhooks Section (Geoprocessing):  https://enterprise.arcgis.com/en/portal/11.1/administer/windows/privileges-for-roles-orgs.htm#ESRI_SECTION1_C30D73392D964D51A8B606128A8A6E8F:~:text=is%20turned%20off.-,Webhooks,-Geoprocessing  Webhooks Section (Organization webhooks) :  https://enterprise.arcgis.com/en/portal/11.1/administer/windows/privileges-for-roles-orgs.htm#ESRI_SECTION1_C30D73392D964D51A8B606128A8A6E8F:~:text=Organization%20webhooks  Depending on the type of webhook you want to create, I recommend configuring a custom role, as assigning the default Administrator role may not be feasible for the Organization Admin. For Geoprocessing service webhooks: assign the following privileges to the custom role. For Organization webhooks: assign the following privileges to the custom role.   Hope it helps! ... View more

Hello @CarstenHogertz , As per my understanding, we should use "<ipVM1>:6443/arcgis" as utilized for registering the first ArcGIS Data Store machine as it is recommended in the documentation. For more details: https://enterprise.arcgis.com/en/portal/latest/administer/windows/add-standby-machine.htm#ESRI_SECTION1_91C0037FB8454EAFBB298CD4A0A07356:~:text=Use%20the%20same%20GIS%20Server%20site%20as%20you%20did%20when%20configuring%20the%20other%20machine%20or%20machines%20in%20the%20same%20data%20store%20for%20this%20ArcGIS%20Enterprise%20deployment.  Hope it helps! ... View more

Hello @JohnDeLucio1, It seems that the "Network Analysis" service is not able to find some files within the "ArcToolbox" which allows ArcGIS Server to execute these tools. However, prior to getting into the details of logs and workflows on the application side, can you confirm the following: Detailed workflow on how the service was published. If it's possible for you to try and publish a test "Network Analysis" using the steps mentioned in below document: Publish Routing Services utility—ArcGIS Server | Documentation for ArcGIS Enterprise Hope it helps! ... View more

Hello @SaurabhUpadhyaya , Your question covers a broad range of factors and points. Capacity planning can get quite detailed, and there are many variables at play. I would suggest starting with a few key questions and refining the setup based on what we observe during testing. Instead of jumping straight into complex planning, we can start small, gather insights, and scale accordingly. If it would be up to me, I would start with the following key questions: How much computing power (CPU, RAM) we need on AWS for ArcGIS Server? How many users can work simultaneously without performance issues? How much load can PostgreSQL RDS handle efficiently? We can utilize the following to estimate and answer to the above queries ESRI’s Capacity Planning Tool (CPT) for predicting system requirements. As you mentioned AWS, "AWS Cloudwatch" to monitor ArcGIS Server and database performance. JMeter for simulating real-world load and identifying bottlenecks We would also need to figure out the following to size the system properly and to decide AWS instance size. How many requests does the system handle? i.e. "Request Volume" If 50 users interact with the system at once, And each user makes 20 requests in one session, That’s 1,000 requests per hour (Transactions Per Hour - TPH). What are the most resource-heavy operations (which you have as functionalities to application)? Feature Service: Displays ~2 million points.  Search and Buffer: These queries hit the database the hardest. Export to PDF: Consumes a lot of CPU and memory. What can help in understanding ArcGIS Server logs → To see which requests take the longest which will allow us to identify slow requests PostgreSQL slow query logs → To find heavy database operations and allows us to pinpoint database bottlenecks JMeter load tests → Allows us to simulate concurrent users and test response time AWS CloudWatch→ Track CPU, memory, and database performance. You can also setup some "Optimization Strategies" beforehand ArcGIS Server Optimization Feature Service Performance Adjust Service Instances: Set Min 2, Max 5 instances, but monitor usage. As you may need to tweak these Enable Shared Instances: Reduces memory use for infrequent queries. Use Filtering: Only load relevant data (don’t send all 2M points to users). Would recommend using "Filtering" (as you mentioned that the application is based on ArcGIS JS API) (For more :Introduction to query and filter | Overview | ArcGIS Maps SDK for JavaScript 4.32 | Esri Developer" Search & Buffer Optimization Index spatial columns in PostgreSQL using GIST indexes for faster queries. Optimize queries: Use SQL EXPLAIN ANALYZE to check query efficiency.   Export to PDF Run exports in a separate ArcGIS instance to avoid slowing down others. Use asynchronous processing to handle large PDF exports. AWS Infrastructure Optimization Concurrent Users Suggested EC2 Instance 50 Users `m6i.xlarge` (4 vCPU, 16GB RAM) 100 Users `m6i.2xlarge` (8 vCPU, 32GB RAM) 200+ Users `m6i.4xlarge` (16 vCPU, 64GB RAM)   Cautiously monitor all the traffic by using AWS CloudWatch to track CPU and memory usage. If CPU is consistently over 80%, increase EC2 size or add more servers. (Remember always start with the small instance type and then move up) PostgreSQL RDS Performance Use PgBouncer → A connection pooler to handle many users efficiently. For more: pgbouncer/README.md at master · pgbouncer/pgbouncer · GitHub Use Read Replicas → Separate heavy search queries from main transactions. For more: Working with read replicas for Amazon RDS for PostgreSQL - Amazon Relational Database Service Monitor Queries → Check slow query logs to fix bottlenecks. For more: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.PostgreSQL.Query_Logging.html#USER_LogAccess.Concepts.PostgreSQL.Query_Logging.using  Testing and Scaling (use these as suggestions): Baseline Testing Deploy a basic setup. Use ArcGIS Server logs and CloudWatch to measure usage. Load Testing Use JMeter to simulate 50, 100, and 200 users. Measure response time for searches, feature loading, and PDF exports. For more details on utilization of JMeter: Performance Testing with Apache JMeter (An Introdu... - Esri Community  Scaling Plan If CPU is above 80%, increase EC2 size. If database is slow, upgrade RDS or add read replicas. If requests queue up, increase ArcGIS Server instances. Furthermore, these are just baseline recommendations based on standard performance expectations. The actual system behavior depends on multiple factors like data complexity, network latency, and real-world user interaction. I would suggest starting with these guidelines, monitoring system performance, and then adjusting based on real findings. Additionally, let me know your thoughts on this or if you want to dive into any specific concerns Hope it helps! ... View more