topic Re: Search not finding datasets in ArcGIS Hub Questions

Search not finding datasets

Anonymous User — Tue, 18 Dec 2018 17:16:29 GMT

Hi,

Over the past couple days I've noticed my Hub (open data) site is not finding some content.

For example, we have an Item for public swimming pools that comes from an external ArcGIS Server. The arcgis.com item was updated in the summer.

No amount of "swim, pool, public" will find the item. Although in the search text which auto populates, the terms matching this item are presented.

Even stranger - we have some hard coded links to the pools item, and they return a hub 404 error when clicked.

If I go and touch the item in arcgis.com (edit a field and push save)...it instantly becomes searchable in the hub site again.

I've tried refreshing the cache in the admin and we have the new beta search API turned on.

Any reason this might be happening?

Re: Search not finding datasets

DanielFenton1 — Tue, 18 Dec 2018 17:20:56 GMT

Hi Kevin,

Can you please give me an Item ID for something that has gone missing? I can check our logs.

Thanks,

Daniel Fenton

Re: Search not finding datasets

Anonymous User — Tue, 18 Dec 2018 17:26:26 GMT

Thanks Daniel,

7a66e4301c60460c97f0d7c2830fb855 and 0a7a8432fa9f4193adc23bc338ef97a7 are two items we noticed that have since been "fixed" with the save.

This one: 13092822f69143b695bdb916357d421b is currently not searchable.... I wont "fix" it for now.

Re: Search not finding datasets

Anonymous User — Tue, 18 Dec 2018 17:57:06 GMT

Another thing that may or may not be related? When trying to perform a data query on the wading pools, it sets in invalid query.

For example: https://hub.arcgis.com/datasets/7a66e4301c60460c97f0d7c2830fb855_11/data?where=ACCESSIBLE%20%3D%20%27undefined%27

If you load this link. Clear the filter on accessibility. Add the filter back for accessibility, chose either yes or no and click off. The URL updates to the above and the table / map goes blank. I don't have a whole lot of datasets with domains - the behavior of these seems inconsistent.

Re: Search not finding datasets

DanielFenton1 — Wed, 19 Dec 2018 13:37:56 GMT

Hey Kevin, we've identified the root cause of this issue and should have a fix out today or tomorrow. It's a bit complicated but I'll try to explain step by step:

1. The first time we tried to index those items our process failed

2. The second time we tried it succeeded

3. Bug: There was still a record of the failed document in the system

4. Bug: Our nightly reindexing process tried to reindex the failed document

5. Bug: The process above thought layers associated with the failed document were outdated and deleted them

We're going to fix these related bugs and make sure your items get back into the index.

To explain the other phenomena:

" Although in the search text which auto populates, the terms matching this item are presented."

- Autocomplete is still based on our older index. The switchover to the new index will be completed shortly. Autocomplete suggestions will then be in sync with documents that are on your site

"If I go and touch the item in arcgis.com (edit a field and push save)...it instantly becomes searchable in the hub site again."

- When you edit a public item, our system gets a notification to index the item. That indexing process makes your item visible. However it was still subject to the process overnight and would have been deleted within 24 hours.

"Even stranger - we have some hard coded links to the pools item, and they return a hub 404 error when clicked."
The same index that powers search also serves content. So if something is not visible in search (with the exception of autocomplete described above), it won't be linkable either.

I'm sorry you hit these bugs, but I am very thankful you brought it to our attention. This was a particularly tricky and insidious one so I am grateful to have caught it during the beta period.

Thank you,

Daniel Fenton

Search Engineering Lead | ArcGIS Hub

Re: Search not finding datasets

DanielFenton1 — Wed, 19 Dec 2018 13:38:25 GMT

Kevin,

I don't think this one is related but we'll take a look at it nonetheless.

Thanks,

Daniel Fenton

Re: Search not finding datasets

Anonymous User — Wed, 19 Dec 2018 13:50:53 GMT

No problem, Daniel. We're still in beta here as well (just a few things before we can launch). Thanks for jumping on these issues so quickly.

Re: Search not finding datasets

DanielFenton1 — Thu, 20 Dec 2018 20:20:10 GMT

Kevin, we've pushed fixes to the issues identified above. If you can give me some group Ids I can manually reindex your data.

Daniel

Re: Search not finding datasets

Anonymous User — Thu, 20 Dec 2018 20:29:08 GMT

Thanks, Daniel.

If it's just the groups that are registered with open data, we only have 2:

16518f6b76d344deb4c0fca60606b32d

fe03c586cee84c08a205f81ebe8c4d35

Appreciate your quick work on this, especially getting into the holidays!

Re: Search not finding datasets

StefanieWenker — Thu, 10 Jun 2021 14:50:59 GMT

Hi Daniel, I am having this problem again. I have features service layers from ArcGIS Server and they intermittently won't show up. Our Open Data page is : opendata.kelowna.ca

Two layers I notice have been having trouble:

1. Sensitive Infill Housing - https://kelowna.maps.arcgis.com/home/item.html?id=370e2e9f88a54fe08724f1cb85360013

2. Building Outlines 2020 - https://kelowna.maps.arcgis.com/home/item.html?id=3f0743a55a454ac5b6631b6c45a0c9b5

They are shared with the public and shared with opendata groups.

Re: Search not finding datasets

bfindlay — Fri, 03 Sep 2021 18:25:46 GMT

Hello,

Similar to Kevin and Stefanie, we have also been noticing our searchable items in Hub (open data) seem not to be indexing correctly, or the index keeps overwriting. This is exactly the same problem Kevin detailed above when he said:

"If I go and touch the item in arcgis.com (edit a field and push save)...it instantly becomes searchable in the hub site again."

We have the same situation here. If I go in and modify the item (e.g. un-share and re-share OR remove the tag and add the tag back) the item get's re-indexed and becomes available via the search but until I do this users cannot find these items shared with our open data group and tagged appropriately. This only seems to happen on some of our items. So for example if a user searches for Wastewater layers they may get 10 results, when really they should have like 18 results.

https://data-roseville.opendata.arcgis.com/search?tags=wastewater

So while Wastewater Main come up in the search:https://www.arcgis.com/home/item.html?id=b0985cd06bfe45a6877f68a7bf294ec6 others like Wastewater Abandoned Point do not come up: https://www.arcgis.com/home/item.html?id=562d208c28334f24ae50b840b1d2a51c

The other ids for our wastewater layers that need re-indexing include:

Wastewater Manhole: 8ebdf9221cb442d391a4f0fce53b8810
Wastewater service line: 4a7466c870a841469c555e349dbbcaeb
Wastewater Private Main: 8a3f094f2ccb41ac8d49548eca9e9c35
Wastewater System Valve:71abc616f5c64c6083eb870ec11bfeb4
Wastewater Basin: 6fa923b0f79f47d98ef56c05aad521bd
Wastewater Cleanout: 9fc52206745742739e307e33150d4206

All items are shared publicly and tagged the same, and I can find absolutely no differences between the details of items that come back in the search vs. those that don't. Even if I go in and make the changes so they all show up in the search, I know they will disappear again as I have done this a few times in the last year or so it is kind of frustrating.

Any advice on how we can correct this problem would be most appreciated.

Thanks,

Brad Findlay

Re: Search not finding datasets

StefanieWenker — Fri, 10 Sep 2021 21:57:31 GMT

Hi Brad,

I was eventually able to figure out a fix for the issue on our end. I was sharing the items using my Admin Account, but my account wasn't actually a member of the open data groups. I figured as an Admin I could just share items as I wanted... but that turned out not to be the case. Once I made myself a member and shared items, they did become available.

Not sure if it will fix your issue, but may be worth looking to see if the account you are using is a member of the open data groups.

Re: Search not finding datasets

bfindlay — Tue, 14 Sep 2021 00:06:18 GMT

Hi Stefanie,

Thanks for your reply, but unfortunately that doesn't appear to be the issue we are experiencing. Our City account owns and is the only member of our Open Data group and that is also the account that owns the items.