Search not finding datasets

681
12
Jump to solution
12-18-2018 09:16 AM
KevinHibma-EsriCA
Esri Contributor

Hi,

Over the past couple days I've noticed my Hub (open data) site is not finding some content. 

For example, we have an Item for public swimming pools that comes from an external ArcGIS Server. The arcgis.com item was updated in the summer.

No amount of "swim, pool, public" will find the item. Although in the search text which auto populates, the terms matching this item are presented.

Even stranger - we have some hard coded links to the pools item, and they return a hub 404 error when clicked.

If I go and touch the item in arcgis.com (edit a field and push save)...it instantly becomes searchable in the hub site again.

I've tried refreshing the cache in the admin and we have the new beta search API turned on.

Any reason this might be happening?

Tags (2)
0 Kudos
1 Solution

Accepted Solutions
DanielFenton1
Regular Contributor

Hey Kevin, we've identified the root cause of this issue and should have a fix out today or tomorrow. It's a bit complicated but I'll try to explain step by step:

1. The first time we tried to index those items our process failed

2. The second time we tried it succeeded

3. Bug: There was still a record of the failed document in the system

4. Bug: Our nightly reindexing process tried to reindex the failed document

5. Bug: The process above thought layers associated with the failed document were outdated and deleted them

We're going to fix these related bugs and make sure your items get back into the index.

To explain the other phenomena:

Although in the search text which auto populates, the terms matching this item are presented."

- Autocomplete is still based on our older index. The switchover to the new index will be completed shortly. Autocomplete suggestions will then be in sync with documents that are on your site

"If I go and touch the item in arcgis.com (edit a field and push save)...it instantly becomes searchable in the hub site again."

- When you edit a public item, our system gets a notification to index the item. That indexing process makes your item visible. However it was still subject to the process overnight and would have been deleted within 24 hours.

"Even stranger - we have some hard coded links to the pools item, and they return a hub 404 error when clicked."
The same index that powers search also serves content. So if something is not visible in search (with the exception of autocomplete described above), it won't be linkable either.

I'm sorry you hit these bugs, but I am very thankful you brought it to our attention. This was a particularly tricky and insidious one so I am grateful to have caught it during the beta period.

Thank you,

Daniel Fenton

Search Engineering Lead | ArcGIS Hub

View solution in original post

12 Replies
DanielFenton1
Regular Contributor

Hi Kevin,

Can you please give me an Item ID for something that has gone missing? I can check our logs.

Thanks,

Daniel Fenton

0 Kudos
KevinHibma-EsriCA
Esri Contributor

Thanks Daniel,

7a66e4301c60460c97f0d7c2830fb855 and 0a7a8432fa9f4193adc23bc338ef97a7 are two items we noticed that have since been "fixed" with the save.

This one:  13092822f69143b695bdb916357d421b is currently not searchable.... I wont "fix" it for now.

0 Kudos
DanielFenton1
Regular Contributor

Hey Kevin, we've identified the root cause of this issue and should have a fix out today or tomorrow. It's a bit complicated but I'll try to explain step by step:

1. The first time we tried to index those items our process failed

2. The second time we tried it succeeded

3. Bug: There was still a record of the failed document in the system

4. Bug: Our nightly reindexing process tried to reindex the failed document

5. Bug: The process above thought layers associated with the failed document were outdated and deleted them

We're going to fix these related bugs and make sure your items get back into the index.

To explain the other phenomena:

Although in the search text which auto populates, the terms matching this item are presented."

- Autocomplete is still based on our older index. The switchover to the new index will be completed shortly. Autocomplete suggestions will then be in sync with documents that are on your site

"If I go and touch the item in arcgis.com (edit a field and push save)...it instantly becomes searchable in the hub site again."

- When you edit a public item, our system gets a notification to index the item. That indexing process makes your item visible. However it was still subject to the process overnight and would have been deleted within 24 hours.

"Even stranger - we have some hard coded links to the pools item, and they return a hub 404 error when clicked."
The same index that powers search also serves content. So if something is not visible in search (with the exception of autocomplete described above), it won't be linkable either.

I'm sorry you hit these bugs, but I am very thankful you brought it to our attention. This was a particularly tricky and insidious one so I am grateful to have caught it during the beta period.

Thank you,

Daniel Fenton

Search Engineering Lead | ArcGIS Hub

KevinHibma-EsriCA
Esri Contributor

No problem, Daniel. We're still in beta here as well (just a few things before we can launch). Thanks for jumping on these issues so quickly.

0 Kudos
DanielFenton1
Regular Contributor

Kevin, we've pushed fixes to the issues identified above. If you can give me some group Ids I can manually reindex your data.

Daniel

0 Kudos
KevinHibma-EsriCA
Esri Contributor

Thanks, Daniel.

If it's just the groups that are registered with open data, we only have 2:

16518f6b76d344deb4c0fca60606b32d

fe03c586cee84c08a205f81ebe8c4d35

Appreciate your quick work on this, especially getting into the holidays!

0 Kudos
StefanieWenker
Occasional Contributor

Hi Daniel, I am having this problem again. I have features service layers from ArcGIS Server and they intermittently won't show up. Our Open Data page is : opendata.kelowna.ca

Two layers I notice have been having trouble:

1. Sensitive Infill Housing - https://kelowna.maps.arcgis.com/home/item.html?id=370e2e9f88a54fe08724f1cb85360013

2. Building Outlines 2020 - https://kelowna.maps.arcgis.com/home/item.html?id=3f0743a55a454ac5b6631b6c45a0c9b5

They are shared with the public and shared with opendata groups. 

0 Kudos
KevinHibma-EsriCA
Esri Contributor

Another thing that may or may not be related? When trying to perform a data query on the wading pools, it sets in invalid query.

For example:  https://hub.arcgis.com/datasets/7a66e4301c60460c97f0d7c2830fb855_11/data?where=ACCESSIBLE%20%3D%20%2... 

If you load this link. Clear the filter on accessibility. Add the filter back for accessibility, chose either yes or no and click off. The URL updates to the above and the table / map goes blank. I don't have a whole lot of datasets with domains - the behavior of these seems inconsistent. 

0 Kudos
DanielFenton1
Regular Contributor

Kevin,

I don't think this one is related but we'll take a look at it nonetheless.

Thanks,

Daniel Fenton

0 Kudos