Keeping consistent layer ids when publishing a service

[stevegourley]

Layer id values in pro projects are assigned when a layer is added to a project. I don't understand the rules for how the values update when new layers are added, removed, and reordered in the project. But these id's cause issues for publishers who don't understand the importance that they stay consistent. 

When a service is published to arcgis online, the layer is accessed by it's layer id. Depending on how a publisher updates that data, the layer id can change. These change happen without the publisher knowing it caused a breaking change for consumers.

As a consumer, it is akin to whack-a-mole trying to keep up with some publishers as the id seems to change on every publish. Some days, they create a new project, add the data, and overwrite the service creating a 0 index. Other days it seems they create a project with a bunch of data and publish creating a layer id of 4. They publish again and it's 6.

It's very difficult to trust these publishers and ultimately that boils down to the platform and the tools coming up short.

The best place to educate publishers that the id's will change is during the analyze step of publishing. I am aware that some customers do not enjoy the amount of information displayed there now which can lead to it being ignored. But where else could it be displayed? What other hook is available? Are there any optional analyze tools that can be triggered by the responsible people who are interested in this information?

My idea is to create an analyzer that can spot id's changing and allow publishers to read about the subject and assign an id to a layer to avoid the breaking change. It could link to a thoughtful article explaining the nuances of changing id's and yada yada.

If this doesn't fit into the required analyze step, then I propose creating an optional analyze step or a way to enable "advanced" analysis. I think you get the idea.

[TanuHoque]

@stevegourley 

did you try this option? https://pro.arcgis.com/en/pro-app/latest/help/sharing/overview/assign-layer-ids.htm

is it not working for you?

thanks

[stevegourley]

@TanuHoque I am a consumer in this instance. I understand you can assign layer id's but many publishers do not or are not. Why is that? What would prompt them to read that article when making updates? Maybe they aren't using the same map to make updates etc. So while I appreciate the link to the document, I'm not the one who needs to read it - the publishers are.

[JonahLay]

Hi @stevegourley,

To add to @TanuHoque response, analyzer 00374: Unique numeric IDs are not assigned was implemented in ArcGIS Pro 2.9 to encourage users to manually assign layer IDs. Are you not seeing this error?

You can learn more about this error and how to resolve it by reading the following blog article:

What does error 00374 mean and what should I do?

Best,

Jonah

[RandyCasey]

@stevegourley when you publish a service the first time from ArcGIS Pro, it applies a zero-based, unique ID (non-repeated) index to every layer, starting from the top of the table of contents to the bottom, then continues on for each table after all the layers have been indexed. So, a TOC with 5 layer and 2 tables will have an index starting from 0 and ending with 6 (the first layer being 0 and the last table being 6). After the service is published, that index schema is static and does not change, but the map in which the service was published from may continue to be developed, with layers removed, added again, or replaced entirely. Each time this is done, ArcGIS Pro will add a new index number on that layer/table in the table of contents, in order to maintain a unique ID on each. So, for instance, if a layer is duplicated for any reason, the newer layer will have a higher index number than the original layer. If the map owner deletes the old layer and replaces it with the new layer and does not go in and manually renumber each layer/table index to match the previous indexing schema, the new index schema will overwrite the old one and that will change all of the service endpoint indexes. So, if the old index schema was 0,1, 2, 3 ,4, 5, 6; the new one may be 0, 1, 2, 7, 4, 5, 6. 

I agree that this behavior is not highlighted enough. My personal thought is, absent an organizational workflow that requires checking indexes before publishing; ESRI should check the index stored in the service definition and notify the user if the existing layers have different indexes than the incoming layers/tables. It could even offer to reindex the TOC for the user, so long as layer names match the previous definition.

[stevegourley]

Thanks @JonahLay but I think you are missing the point. Publishers are ignoring that on the first publish or not, it doesn't really matter. If they followed that analyzer and assigned id's, if they don't use the same map/project to update the data, they fall into the same pit of despair without any notice. It doesn't matter when they first publish a service because there is no contract to follow, no link created on the web for people to consume. The issue appears when publishers *update* the data and inadvertently create new ids for layers. This is when the URLs change and the contract is broken. All references are 404. It is my experience that they don't know they did it or why it happened.

@RandyCasey Thank you for describing the indexing algorithm in more depth. I agree 💯💯💯 esri should check the index in the service and notify the user to take action. Reindexing the project to fix any further issues is an interesting idea. This helps people do the right thing.

[OliverAdam]

To ensure consistent layer IDs when publishing a service:

1. Plan Ahead:

   • Establish a structured naming convention for layers before service publication, akin to arranging elements in "little alchemy" for consistent outcomes.

2. Avoid Modifications:

   • Refrain from altering or reordering layers post-publication, mirroring the stability required maintain consistent layer IDs.

3. Version Control:

   • Implement version control or backups when modifications are necessary, akin to recording steps in "little alchemy" to trace changes without affecting layer IDs.

4. Documentation:

   • Document any adjustments made to layers, similar to documenting combinations in "little alchemy," ensuring clear references for future updates.

5. Testing:

   • Post modification, thoroughly test the service to ensure functionality and retain consistent layer IDs, akin to testing different element combinations in "little alchemy" for desired results.

6. Communication:

   • Communicate any alterations in layer structure or IDs to involved parties, aligning with transparency to maintain coherence across the service.

[GIS_Spellblade]

As an additional confounding factor, if a user selects auto-assign IDs from within the sharing dialogue, those IDs are not permanently affixed to the APRX project file-- --I think that this contributes to the user errors, as a re-ordering of the layers that might otherwise not create new index IDs, would, if a user continually checks the "auto assign IDs" box within the sharing dialogue.

layer_a (0)
layer_b (1)
layer_c (2)

is not preserved within the APRX file even when a user saves after publishing a service. If those same layers are moved, and/or an additional layer added:

layer_b (0)
layer_a (1)
layer_c (2)
layer_d (3)

If the auto-assign function was honored within the project, the above rearrange and layer addition would not affect a production publishing service:

layer_b (1)
layer_a (0)
layer_c (2)
layer_d (3)

And I 100% agree with @stevegourley if a user was alerted to the fact that they might be pushing a bad change it could prevent some of those updates from happening. Obviously a centralized, RFC-controlled process is the most preferable option but those kind of strict controls are often not possible in geospatial programs of the following types: balanced, federated, hub and spoke, and decentralized. In my experience, most organizations fall outside of a fully centralized program and would benefit from this quality of life reminder.

https://resources.esri.ca/news-and-updates/how-to-organize-your-geospatial-talent

[TanuHoque]

Thank you all for your comment to clarify the issue.

I just wanted to add a quick note since I see some discussions about how ids get assigned to a layer in Pro.

By default ArcGIS Pro does not assign any numeric ID (for the purpose of getting used in a service) to a layer.

You must have 'Allow assignment of unique numeric IDs for sharing web layers' options checked on the map's Properties page.

When this option is checked on, Pro assigns IDs in some sequential order (starting from 0) to existing layers. As you add new layers, it will assign new ids to those layers as well. These are more or less sequential. These IDs are stored with layers themselves. You can see (and modify if needed) the assigned ID in the layer's Properties page's General tab. Therefore change those layers order in the Contents pane won't impact these IDs.

And when you publish this as a service, these IDs get honored and you can access REST resources for those layers using that ID. 

If the above check box on a map's Properties page is unchecked, and you publish/share it as a service, then auto-generated sequential IDs get assigned to all layers and tables of a map during the startup of the service.

I hope this helps.

[stevegourley]

@OliverAdam Thank you for outlining your process. My agency also has an automated, repeatable process that does not violate the contracts we create with our customers. That's where my and your responsibility ends. ArcGIS Online is a lot bigger than you and I. Your process is not relevant to this conversation since you cannot force your process on to others publishing to ArcGIS online. You cannot make every publisher read your suggestion and perform those tasks. The common denominator is that we are all publishing with ArcGIS Pro content. That is the only place publishers can be informed to make the right decision. That is where this idea is aimed.