idea Keeping consistent layer ids when publishing a service in ArcGIS Pro Ideas

Keeping consistent layer ids when publishing a service

stevegourley — Mon, 18 Dec 2023 23:31:44 GMT

Layer id values in pro projects are assigned when a layer is added to a project. I don't understand the rules for how the values update when new layers are added, removed, and reordered in the project. But these id's cause issues for publishers who don't understand the importance that they stay consistent.

When a service is published to arcgis online, the layer is accessed by it's layer id. Depending on how a publisher updates that data, the layer id can change. These change happen without the publisher knowing it caused a breaking change for consumers.

As a consumer, it is akin to whack-a-mole trying to keep up with some publishers as the id seems to change on every publish. Some days, they create a new project, add the data, and overwrite the service creating a 0 index. Other days it seems they create a project with a bunch of data and publish creating a layer id of 4. They publish again and it's 6.

It's very difficult to trust these publishers and ultimately that boils down to the platform and the tools coming up short.

The best place to educate publishers that the id's will change is during the analyze step of publishing. I am aware that some customers do not enjoy the amount of information displayed there now which can lead to it being ignored. But where else could it be displayed? What other hook is available? Are there any optional analyze tools that can be triggered by the responsible people who are interested in this information?

My idea is to create an analyzer that can spot id's changing and allow publishers to read about the subject and assign an id to a layer to avoid the breaking change. It could link to a thoughtful article explaining the nuances of changing id's and yada yada.

If this doesn't fit into the required analyze step, then I propose creating an optional analyze step or a way to enable "advanced" analysis. I think you get the idea.

Re: Keeping consistent layer ids when publishing a service - Status changed to: Needs Clarification

TanuHoque — Mon, 18 Dec 2023 23:44:44 GMT

@stevegourley

did you try this option? https://pro.arcgis.com/en/pro-app/latest/help/sharing/overview/assign-layer-ids.htm

is it not working for you?

thanks

Re: Keeping consistent layer ids when publishing a service

stevegourley — Mon, 18 Dec 2023 23:47:38 GMT

@TanuHoque I am a consumer in this instance. I understand you can assign layer id's but many publishers do not or are not. Why is that? What would prompt them to read that article when making updates? Maybe they aren't using the same map to make updates etc. So while I appreciate the link to the document, I'm not the one who needs to read it - the publishers are.

Re: Keeping consistent layer ids when publishing a service

JonahLay — Tue, 19 Dec 2023 00:01:07 GMT

Hi @stevegourley,

To add to @TanuHoque response, analyzer 00374: Unique numeric IDs are not assigned was implemented in ArcGIS Pro 2.9 to encourage users to manually assign layer IDs. Are you not seeing this error?

You can learn more about this error and how to resolve it by reading the following blog article:

What does error 00374 mean and what should I do?

Best,

Jonah

Re: Keeping consistent layer ids when publishing a service

RandyCasey — Tue, 19 Dec 2023 00:15:20 GMT

@stevegourley when you publish a service the first time from ArcGIS Pro, it applies a zero-based, unique ID (non-repeated) index to every layer, starting from the top of the table of contents to the bottom, then continues on for each table after all the layers have been indexed. So, a TOC with 5 layer and 2 tables will have an index starting from 0 and ending with 6 (the first layer being 0 and the last table being 6). After the service is published, that index schema is static and does not change, but the map in which the service was published from may continue to be developed, with layers removed, added again, or replaced entirely. Each time this is done, ArcGIS Pro will add a new index number on that layer/table in the table of contents, in order to maintain a unique ID on each. So, for instance, if a layer is duplicated for any reason, the newer layer will have a higher index number than the original layer. If the map owner deletes the old layer and replaces it with the new layer and does not go in and manually renumber each layer/table index to match the previous indexing schema, the new index schema will overwrite the old one and that will change all of the service endpoint indexes. So, if the old index schema was 0,1, 2, 3 ,4, 5, 6; the new one may be 0, 1, 2, 7, 4, 5, 6.

I agree that this behavior is not highlighted enough. My personal thought is, absent an organizational workflow that requires checking indexes before publishing; ESRI should check the index stored in the service definition and notify the user if the existing layers have different indexes than the incoming layers/tables. It could even offer to reindex the TOC for the user, so long as layer names match the previous definition.

Re: Keeping consistent layer ids when publishing a service

stevegourley — Tue, 19 Dec 2023 01:04:36 GMT

Thanks @JonahLay but I think you are missing the point. Publishers are ignoring that on the first publish or not, it doesn't really matter. If they followed that analyzer and assigned id's, if they don't use the same map/project to update the data, they fall into the same pit of despair without any notice. It doesn't matter when they first publish a service because there is no contract to follow, no link created on the web for people to consume. The issue appears when publishers *update* the data and inadvertently create new ids for layers. This is when the URLs change and the contract is broken. All references are 404. It is my experience that they don't know they did it or why it happened.

@RandyCasey Thank you for describing the indexing algorithm in more depth. I agree 💯💯💯 esri should check the index in the service and notify the user to take action. Reindexing the project to fix any further issues is an interesting idea. This helps people do the right thing.

Re: Keeping consistent layer ids when publishing a service

OliverAdam — Tue, 19 Dec 2023 07:56:37 GMT

To ensure consistent layer IDs when publishing a service:

Plan Ahead:
- Establish a structured naming convention for layers before service publication, akin to arranging elements in "little alchemy" for consistent outcomes.
Avoid Modifications:
- Refrain from altering or reordering layers post-publication, mirroring the stability required maintain consistent layer IDs.
Version Control:
- Implement version control or backups when modifications are necessary, akin to recording steps in "little alchemy" to trace changes without affecting layer IDs.
Documentation:
- Document any adjustments made to layers, similar to documenting combinations in "little alchemy," ensuring clear references for future updates.
Testing:
- Post modification, thoroughly test the service to ensure functionality and retain consistent layer IDs, akin to testing different element combinations in "little alchemy" for desired results.
Communication:
- Communicate any alterations in layer structure or IDs to involved parties, aligning with transparency to maintain coherence across the service.

Re: Keeping consistent layer ids when publishing a service

GIS_Spellblade — Tue, 19 Dec 2023 15:35:12 GMT

As an additional confounding factor, if a user selects auto-assign IDs from within the sharing dialogue, those IDs are not permanently affixed to the APRX project file-- --I think that this contributes to the user errors, as a re-ordering of the layers that might otherwise not create new index IDs, would, if a user continually checks the "auto assign IDs" box within the sharing dialogue.

layer_a (0)
layer_b (1)
layer_c (2)

is not preserved within the APRX file even when a user saves after publishing a service. If those same layers are moved, and/or an additional layer added:

layer_b (0)
layer_a (1)
layer_c (2)
layer_d (3)

If the auto-assign function was honored within the project, the above rearrange and layer addition would not affect a production publishing service:

layer_b (1)
layer_a (0)
layer_c (2)
layer_d (3)

And I 100% agree with @stevegourley if a user was alerted to the fact that they might be pushing a bad change it could prevent some of those updates from happening. Obviously a centralized, RFC-controlled process is the most preferable option but those kind of strict controls are often not possible in geospatial programs of the following types: balanced, federated, hub and spoke, and decentralized. In my experience, most organizations fall outside of a fully centralized program and would benefit from this quality of life reminder.

https://resources.esri.ca/news-and-updates/how-to-organize-your-geospatial-talent

Re: Keeping consistent layer ids when publishing a service - Status changed to: Open

SusanTran — Tue, 19 Dec 2023 23:03:43 GMT

Re: Keeping consistent layer ids when publishing a service

TanuHoque — Tue, 19 Dec 2023 23:11:03 GMT

Thank you all for your comment to clarify the issue.

I just wanted to add a quick note since I see some discussions about how ids get assigned to a layer in Pro.

By default ArcGIS Pro does not assign any numeric ID (for the purpose of getting used in a service) to a layer.

You must have 'Allow assignment of unique numeric IDs for sharing web layers' options checked on the map's Properties page.

When this option is checked on, Pro assigns IDs in some sequential order (starting from 0) to existing layers. As you add new layers, it will assign new ids to those layers as well. These are more or less sequential. These IDs are stored with layers themselves. You can see (and modify if needed) the assigned ID in the layer's Properties page's General tab. Therefore change those layers order in the Contents pane won't impact these IDs.

And when you publish this as a service, these IDs get honored and you can access REST resources for those layers using that ID.

If the above check box on a map's Properties page is unchecked, and you publish/share it as a service, then auto-generated sequential IDs get assigned to all layers and tables of a map during the startup of the service.

I hope this helps.

Re: Keeping consistent layer ids when publishing a service

stevegourley — Tue, 19 Dec 2023 23:12:58 GMT

@OliverAdam Thank you for outlining your process. My agency also has an automated, repeatable process that does not violate the contracts we create with our customers. That's where my and your responsibility ends. ArcGIS Online is a lot bigger than you and I. Your process is not relevant to this conversation since you cannot force your process on to others publishing to ArcGIS online. You cannot make every publisher read your suggestion and perform those tasks. The common denominator is that we are all publishing with ArcGIS Pro content. That is the only place publishers can be informed to make the right decision. That is where this idea is aimed.

Re: Keeping consistent layer ids when publishing a service

stevegourley — Tue, 19 Dec 2023 23:17:23 GMT

Thank you for clarifying @TanuHoque. There does seem to be a lot of uncertainty around how the id's function.

Let's try to not get distracted with how the id's are created and focus our conversation on how to improve the product to help guide people to make less mistakes that cause breaking changes with ArcGIS Online services.

Re: Keeping consistent layer ids when publishing a service

BillFox — Wed, 20 Dec 2023 21:31:19 GMT

are you describing this behavior?

Re: Keeping consistent layer ids when publishing a service

RandyCasey — Wed, 20 Dec 2023 22:26:51 GMT

@BillFox I do not believe those are being identified here. This has to do specifically with the layer index of map services, which is derived from the map layer index of the layers when a service is published from ArcGIS Pro. If that map layer index is changed at any point after the first publish and the map is published again as a service, the layer indices of those specific REST endpoints get changed as well. This can and will cause issues, as applications, automation scripts, etc., will be pointed to a specific REST endpoint (ex., https://myfictitousdomain.com/arcgis/rest/services/fakeserverfolder/fakeservice/mapserver/3) but because the map index was changed in ArcGIS Pro and was published with the change intact, the REST endpoint has now changed (ex., https://myfictitousdomain.com/arcgis/rest/services/fakeserverfolder/fakeservice/mapserver/8 ), thus breaking the preexisting URL connections causing the domino effect of all applications, scripts, etc., to break as well. This is the specific behavior that this post is trying to call attention to and would like to address.

Re: Keeping consistent layer ids when publishing a service

BillFox — Wed, 20 Dec 2023 23:02:14 GMT

Hi Randy,

just curious, this issue seems to bleed back to at least 2015 here:

https://community.esri.com/t5/arcgis-web-appbuilder-questions/with-portal-for-arcgis-10-3-1-has-anyone-else/td-p/369664/page/2

Re: Keeping consistent layer ids when publishing a service

stevegourley — Wed, 20 Dec 2023 23:43:33 GMT

@BillFox This idea is to help fix the root issue and your issues are related to this but downstream. I think if esri can help solve this problem, the issues you highlight start to happen less and less.

Re: Keeping consistent layer ids when publishing a service

RandyCasey — Thu, 21 Dec 2023 00:02:55 GMT

@BillFox Yes and no. The original problem of, if a layer is moved in the TOC of the map after publishing, and is republished with those moves, there is no alert that you have changed the layer indexing, and it no longer matches the service definition has taken a new twist. In ArcGIS Pro 2.5x (I think), ESRI introduced a new feature, where you can now manually assign layer indexes regardless of where it is placed in the TOC (so long as you turn on the functionality). This is great if you want to position your layers wherever you want on the TOC and not have to deal with messing up your service indexes, but where this dovetails, is ArcGIS Pro will assign new indexes whenever new layers or tables are added to the Map. So, if you duplicate a layer, and replace an existing layer with the duplicate, it will have a much higher index number, as it will get assigned the next index in sequence. Repeat this enough times, and you could have all new indexes in your map. Republish that map in that state, and you have now broken every REST endpoint that previously existed for those layers. And still, just like back in 10.3x, there is no alert advising the user that they are about to republish with a non-matching index of the service definition.

Re: Keeping consistent layer ids when publishing a service

stevegourley — Thu, 29 Feb 2024 18:37:22 GMT

When I open the Item Details of the layer in the map, it results in an error, and it's pointing to ...Video_Coverage/MapServer/0
However, when opening the layer from the REST service, the actual ID number is 2 (...Video_Coverage/MapServer/2)
I don't know what happened. An easy fix is to just remove the version on the map, and add it again - that works fine, but then some of the functionality in the app is lost (namely, when you start a video, it no longer zooms to the road).

Here's another example of someone changing id's without knowing it and breaking an application

Re: Keeping consistent layer ids when publishing a service - Status changed to: Under Consideration

SusanTran — Mon, 28 Oct 2024 21:20:31 GMT

Re: Keeping consistent layer ids when publishing a service - Status changed to: In Product Plan

SusanTran — Fri, 06 Dec 2024 23:25:58 GMT