topic Re: Portal 10.5 Index Mismatch (search) in ArcGIS Enterprise Portal Questions

Portal 10.5 Index Mismatch (search)

FraserHand1 — Wed, 29 Mar 2017 01:49:17 GMT

Hi,

Has anyone come across a situation where the Portal Search index drops? This mean things like symbology, web app templates etc start to vanish from Portal. Note that doing a reindex still does not result in a full match. I have tried the go from an empty index and reindex from here:

http://support.esri.com/technical-article/000012663

and have run the indexer manually on the portal machine from here

C:\ArcGIS\Portal\tools\indexer>Indexer.bat -f -i F:\arcgisportal\index

Still end up with a mismatch on the Search Index

Note watching the index output I see stuff like

INFO: Query returned 7167 records.

0 [WARN] GWContent: - 'extent' parameter in item card is invalid

977 [WARN] GWContent: - 'extent' parameter in item card is invalid

2480 [WARN] GWContent: - 'extent' parameter in item card is invalid

6991 [WARN] GWContent: - 'extent' parameter in item card is invalid

which looks like the items not being indexed? I had though maybe it was permissions but procmon didn't turn anything up.

Running ArcGIS enterprise 10.5

Portal is single machine with federated Server.

Thanks,

Fraser

Re: Portal 10.5 Index Mismatch (search)

MichaelRobb — Mon, 14 Aug 2017 22:35:49 GMT

I have received this as well.

Unknown cause or unknown fix.

Re: Portal 10.5 Index Mismatch (search)

BingLu — Wed, 06 Dec 2017 17:18:33 GMT

We ran into the same problem. Last week our ArcGIS Portal's server ran out of space, which caused some index issues. So we had to use the "empty index" folder to regenerate the index. After that process was complete, for the "search" index, "Count in store" was 5797, and "Count in index" was 5790. But all other indexes match.

Any suggestions to fix this issue would be appreciated

Re: Portal 10.5 Index Mismatch (search)

PhilipHughes — Wed, 16 May 2018 05:53:24 GMT

Did you ever resolve this, have the same problem at 10.5.1

Re: Portal 10.5 Index Mismatch (search)

FraserHand1 — Wed, 16 May 2018 06:34:26 GMT

Hi,

Yes I did - the first issue we had was the index service port was blocked on the second portal server. The index still did drop and bounce around though. The best way I found to fix it was to shut down the secondary portal server, rerun a full index and check, and then bring the second portal back up. This was 10.5 though.

Thanks,

Fraser

Re: Portal 10.5 Index Mismatch (search)

KierenTinning2 — Mon, 04 Jun 2018 18:02:51 GMT

Fraser,

Our ports are open, but indexes are out of sync. Did you shut down the entire server or just the service? We have shut down the service and it didn't correct the issue.

Running portal 10.5.1

Thanks,

Kieren

Re: Portal 10.5 Index Mismatch (search)

RobertDriessen2 — Thu, 28 Jun 2018 04:12:55 GMT

Is there any further information on this issue. Reindex seems to stop with search count in store set to 7529 and search count in index set to 7503.

Have started reindex again with same result. Can anyone please confirm that the users' count has no direct relationship with the number of users. We don't have 64 users. I assume that means 64 user indexes?

We have a pair of portal VMs so I will attempt Frasers' solution by shutting down the secondary server.

Many of the items in our portal became orphans (users could not see them). However they were visible in the groups but not manageable. After indexing they also did not appear in the groups. Hopefully indexing will fix this.

Thanks, Rob

Re: Portal 10.5 Index Mismatch (search)

JonathanQuinn — Thu, 28 Jun 2018 16:00:17 GMT

What version are you using? Before 10.6 or 10.6.1, the index returned system accounts and groups, so the counts are a bit inflated.

If you're at 10.5.1, there's a bug that causes a "split-brain" problem, where both portal machines can think the component that controls the index is the master node. This causes index synchronization problems.

Re: Portal 10.5 Index Mismatch (search)

RobertDriessen2 — Thu, 28 Jun 2018 23:37:06 GMT

Hi Jonathan, thanks fir your quick response.

We are at 10.5.1. Stopping the secondary portal service, as suggested above, did not help so I still have an index problem. Sounds like upgrading to 10.6.1 will be the solution.

Incidentally, shutting down the secondary virtual machine rendered the portal inoperative.I have since read elsewhere (one of your posts) that this is expected behavior. That is, both machines need to be up for the portal to work in high availability setup however the portal service does not need to be running on one of the machines. Am I understanding that correctly? Is there any documentation on this?

Thanks again, Rob

Re: Portal 10.5 Index Mismatch (search)

JonathanQuinn — Thu, 28 Jun 2018 23:45:20 GMT

I don't think 10.6.1 will fix that problem if there's an issue with the underlying index schema or any other files. I'll follow up with a message.

You should definitely be able to shut down either machine and still have Portal function. If you shut down the primary, the standby should be promote to primary. If you shut down the standby, the primary should still function. If you start the VM, does everything work? If so, can you sign into Portaladmin and check the Machines list to see which machine is primary? If the machine you shut down is, but the standby is never promoted, then it's likely a recovery.done file exists in the db directory of the standby. That tells the internal DB that it already has failed over, so nothing happens.

In my other post, I may have been referring to shutdown order, which can certainly be a problem depending on how you shut down the machines. For example, if you shut down standby, shut down primary, and start standby, the standby won't assume the role of primary as it hasn't been able to get the latest data from the primary.

Re: Portal 10.5 Index Mismatch (search)

RobertDriessen2 — Fri, 29 Jun 2018 01:16:34 GMT

Thanks again Jon

The site had a problem when one of the VM's was down. Worked OK if it was up and portal service not running. However, can't get into site at all now as portal is not starting up. So no, at this point, when I start the VM the portal does not work at all.

I was not aware of the shutdown order (is this documented?) It could be the cause of the current problem. Our primary machine has a name like portal1 and the secondary machine has a name like portal2. Portal1 was set up as the primary site and Portal2 was added to it. To compound this Portal2 can become the primary and you need to check the admin folder to confirm. It seems random but I understand that if the primary fails then the secondary becomes primary and remains that way.

Thank you for your direct email, I have responded to those requests.

Your help is very much appreciated

Rob

Re: Portal 10.5 Index Mismatch (search)

SusmitaDuncan — Wed, 01 Aug 2018 15:23:56 GMT

Any update on getting the re-index to run successfully by anyone? I also have a 10.5.1 HA Portal setup, and have been trying to get the WebGISDR to run (because after a month we are already getting the walarchive file growth issue), but WebGISDR tool is erroring out. We have a ticket open and one of the items we are checking is if the Index is in sync, which it's not just like everyone else in this thread. Ours is more out of sync and I can't get the numbers to change.

Index Status

Type	Count in store	Count in index
users	64	64
groups	327	327
search	7522	45

Going to try the workflow of stopping the standby portal services (can't do a shutdown on the VM), but I'm wondering if anyone is also then not able to run the webgisdr tool because of the index mismatch? This is a problem since Portal will self corrupt with those walarchive files growing. We are planning on upgrading to 10.6.1 shortly after we run the upgrade through our dev portal, but I know the walarchive issue is not fixed, and it sounds like this index mismatch is also not fixed. Starting to consider disabling HA and put the Portal content directory back on the local machine to see if that resolves the index issue.

Re: Portal 10.5 Index Mismatch (search)

RobertDriessen2 — Wed, 01 Aug 2018 22:36:22 GMT

Hi Susmita

I have tried everything to solve this problem and have not found one. This has included reverting to a single server (removing secondary portal from site) and trying to reindex. I gained the impression from ESRI that upgrading might not fix the problem. We were in the position where we could recreate the site so we did that. That meant losing all content, settings etc so not something you would do lightly. Of course that worked (new indexes). Unfortunately the problem has come back - very frustrating - so if you fix it then it might come back. So I would like to know what the impact is of the mismatch. Is it in fact a problem? And if so, what is the fix?

Sorry I can't be more helpful but thought some feedback on my experience would be useful.

Re: Portal 10.5 Index Mismatch (search)

BradKiep — Thu, 30 Aug 2018 01:07:45 GMT

Hi Jonathan,

We have a 10.5.1 HA Enterprise and our Portal machines both have the recovery.done file in the db directory. I deleted it out of the Standby machine. I stopped the Portal service on that server to prove that machine was the Standby and our Portal continued to work normally. When I started the Portal service again, recovery.done was recreated in the db directory again. Do you know of any reason the recovery.done file would keep getting created on the Standby even though there was no failover issue with the Primary? Both servers continue to have recovery.done unless I manually delete it from the Standby.

Any help is greatly appreciated. We need our Portal to be HA.

Thanks,

Brad

Re: Portal 10.5 Index Mismatch (search)

ChelseaRozek — Thu, 30 Aug 2018 13:03:12 GMT

Upgrading to 10.6.1 fixed our mismatched index.

Re: Portal 10.5 Index Mismatch (search)

JonathanQuinn — Fri, 31 Aug 2018 17:55:03 GMT

Does the standby have a recovery.conf file? On primary, it's ok if there's a recovery.done. On Standby, you should definitely see a recovery.conf.

Re: Portal 10.5 Index Mismatch (search)

BradKiep — Fri, 31 Aug 2018 19:45:09 GMT

The primary has .done. The standby has .done and .conf. Will the .done on

the standby prevent it from promoting to primary if there is a failover

event?

The other day, I manually deleted the .done file on the standby and stopped

the portal service. Our enterprise continued to work as expected with the

primary. When i started the portal service again on the standby the .done

file was recreated. We are trying to find a time to test a failover by

stopping the portal service on the primary. We have users 24/7 so we have

to schedule any down time in advance.

Is there any documentation on how the failover works? How long will it

take to promote the standby to the primary? I want to make sure when we

test we are waiting long enough and we understand the results.

Thank you for your help. I was confused to learn that HA Portal worked as

a Primary/Standby system. I thought it worked more like our ArcGIS Server

machines that work in tandem to create HA. The diagram for configuring HA

<http://enterprise.arcgis.com/en/portal/latest/administer/windows/configuring-a-highly-available-portal.htm>

the help shows the non-Primary portal server as a secondary server and not

just a standby. Any help is greatly appreciated.

Thanks,

Brad

Re: Portal 10.5 Index Mismatch (search)

HarroldSompotan — Thu, 13 Sep 2018 20:56:23 GMT

Just wanted to share with everyone on this post, that we actually logged a Software Defect on this behaviour on version for Portal 10.5.1 in Highly Available environment.

BUG-000107551: Index counts are incorrect in a highly available Portal for ArcGIS deployment in a failover situation, if node 1 fails and then comes back up.

This is actually fixed in version 10.6.X

@Chelsea Rozek Glad the Upgrade worked for you.

Starting in 10.6 for upgrading portal we have implemented the Post-Upgrade process:

Navigate to Portaladmin end point: https://machine.com/portal/portaladmin/
Click System > Indexer > Reindex.
Click the Mode drop-down list, and select Full.
Click Reindex. This step will complete the upgrade of your portal. Depending on the number of users and volume of content in your portal, it will take some time for the reindex to complete. Do not interrupt the reindex process. You can monitor the indexing status by opening a new browser window (or tab), navigating to System > Indexer > Index Status, and refreshing the page. When the store and index counts are equal, the reindex and upgrade is complete.
Install and Configure the Web Adaptors.

Re: Portal 10.5 Index Mismatch (search)

BobWheeler — Thu, 01 Nov 2018 19:40:44 GMT

We didn't have issues until we upgraded from 10.5.1 to 10.6.1 (HA). We've reindexed several times without success. The reindex 'ends' in an error and the counts are way off for Search. esri suggested running a repair install on Portal.

Re: Portal 10.5 Index Mismatch (search)

JonathanQuinn — Thu, 01 Nov 2018 20:57:59 GMT

It's unlikely that a repair will work, as it's not the install directory that's the problem, but the index directory, (C:\arcgisportal\index, for example).