Portal 10.5 Index Mismatch (search)

5096
28
03-28-2017 06:49 PM
FraserHand1
New Contributor III

Hi,

Has anyone come across a situation where the Portal Search index drops? This mean things like symbology, web app templates etc start to vanish from Portal. Note that doing a reindex still does not result in a full match. I have tried the go from an empty index and reindex from here:

http://support.esri.com/technical-article/000012663

and have run the indexer manually on the portal machine from here

C:\ArcGIS\Portal\tools\indexer>Indexer.bat -f -i F:\arcgisportal\index

Still end up with a mismatch on the Search Index 

Note watching the index output I see stuff like

INFO: Query returned 7167 records.

0 [WARN] GWContent: - 'extent' parameter in item card is invalid

977 [WARN] GWContent: - 'extent' parameter in item card is invalid

2480 [WARN] GWContent: - 'extent' parameter in item card is invalid

6991 [WARN] GWContent: - 'extent' parameter in item card is invalid

which looks like the items not being indexed? I had though maybe it was permissions but procmon didn't turn anything up.

Running ArcGIS enterprise 10.5

Portal is single machine with federated Server.

Thanks,

Fraser

Tags (3)
28 Replies
RobertDriessen2
New Contributor III

Thanks again Jon

The site had a problem when one of the VM's was down.  Worked OK if it was up and portal service not running.  However, can't get into site at all now as portal is not starting up. So no, at this point, when I start the VM the portal does not work at all.

I was not aware of the shutdown order (is this documented?)  It could be the cause of the current problem.  Our  primary machine has a name like portal1 and the secondary machine has a name like portal2.  Portal1 was set up as the primary site and Portal2 was added to it.  To compound this Portal2 can become the primary and you need to check the admin folder to confirm.  It seems random but I understand that if the primary fails then the secondary becomes primary and remains that way.  

Thank you for your direct email, I have responded to those requests.

Your help is very much appreciated

Rob

0 Kudos
BradKiep
New Contributor III

Hi Jonathan,

We have a 10.5.1 HA Enterprise and our Portal machines both have the recovery.done file in the db directory.  I deleted it out of the Standby machine.  I stopped the Portal service on that server to prove that machine was the Standby and our Portal continued to work normally.  When I started the Portal service again, recovery.done was recreated in the db directory again.  Do you know of any reason the recovery.done file would keep getting created on the Standby even though there was no failover issue with the Primary?  Both servers continue to have recovery.done unless I manually delete it from the Standby.

Any help is greatly appreciated.  We need our Portal to be HA.

Thanks,

Brad

0 Kudos
JonathanQuinn
Esri Frequent Contributor

Does the standby have a recovery.conf file? On primary, it's ok if there's a recovery.done. On Standby, you should definitely see a recovery.conf.

0 Kudos
BradKiep
New Contributor III

The primary has .done. The standby has .done and .conf. Will the .done on

the standby prevent it from promoting to primary if there is a failover

event?

The other day, I manually deleted the .done file on the standby and stopped

the portal service. Our enterprise continued to work as expected with the

primary. When i started the portal service again on the standby the .done

file was recreated. We are trying to find a time to test a failover by

stopping the portal service on the primary. We have users 24/7 so we have

to schedule any down time in advance.

Is there any documentation on how the failover works? How long will it

take to promote the standby to the primary? I want to make sure when we

test we are waiting long enough and we understand the results.

Thank you for your help. I was confused to learn that HA Portal worked as

a Primary/Standby system. I thought it worked more like our ArcGIS Server

machines that work in tandem to create HA. The diagram for configuring HA

<http://enterprise.arcgis.com/en/portal/latest/administer/windows/configuring-a-highly-available-portal.htm>

in

the help shows the non-Primary portal server as a secondary server and not

just a standby. Any help is greatly appreciated.

Thanks,

Brad

0 Kudos
SusmitaDuncan
New Contributor II

Any update on getting the re-index to run successfully by anyone? I also have a 10.5.1 HA Portal setup, and have been trying to get the WebGISDR to run (because after a month we are already getting the walarchive file growth issue), but WebGISDR tool is erroring out. We have a ticket open and one of the items we are checking is if the Index is in sync, which it's not just like everyone else in this thread. Ours is more out of sync and I can't get the numbers to change.

Index Status

 

Type

Count in store

Count in index

users

64

64

groups

327

327

search

7522

45


Going to try the workflow of stopping the standby portal services (can't do a shutdown on the VM), but I'm wondering if anyone is also then not able to run the webgisdr tool because of the index mismatch? This is a problem since Portal will self corrupt with those walarchive files growing. We are planning on upgrading to 10.6.1 shortly after we run the upgrade through our dev portal, but I know the walarchive issue is not fixed, and it sounds like this index mismatch is also not fixed. Starting to consider disabling HA and put the Portal content directory back on the local machine to see if that resolves the index issue.

RobertDriessen2
New Contributor III

Hi Susmita

I have tried everything to solve this problem and have not found one.  This has included reverting to a single server (removing secondary portal from site) and trying to reindex.  I gained the impression from ESRI that upgrading might not fix the problem.  We were in the position where we could recreate the site so we did that.  That meant losing all content, settings etc so not something you would do lightly. Of course that worked (new indexes).  Unfortunately the problem has come back - very frustrating - so if you fix it then it might come back.  So I would like to know what the impact is of the mismatch.  Is it in fact a problem?   And if so, what is the fix?

Sorry I can't be more helpful but thought some feedback on my experience would be useful.

0 Kudos
ChelseaRozek
MVP Regular Contributor

Upgrading to 10.6.1 fixed our mismatched index.

0 Kudos
HarroldSompotan
Esri Contributor

Just wanted to share with everyone on this post, that we actually logged a Software Defect on this behaviour on version for Portal 10.5.1 in Highly Available environment. 

BUG-000107551: Index counts are incorrect in a highly available Portal for ArcGIS deployment in a failover situation, if node 1 fails and then comes back up.

This is actually fixed in version 10.6.X 

@Chelsea Rozek Glad the Upgrade worked for you. 

Starting in 10.6 for upgrading portal we have implemented the Post-Upgrade process:

    1. Navigate to Portaladmin end point: https://machine.com/portal/portaladmin/ 
    2. Click System > Indexer > Reindex.
    3. Click the Mode drop-down list, and select Full.
    4. Click Reindex. This step will complete the upgrade of your portal. Depending on the number of users and volume of content in your portal, it will take some time for the reindex to complete. Do not interrupt the reindex process. You can monitor the indexing status by opening a new browser window (or tab), navigating to System > Indexer > Index Status, and refreshing the page. When the store and index counts are equal, the reindex and upgrade is complete.
    5. Install and Configure the Web Adaptors. 
BobWheeler
Occasional Contributor

We didn't have issues until we upgraded from 10.5.1 to 10.6.1 (HA).  We've reindexed several times without success.  The reindex 'ends' in an error and the counts are way off for Search.  esri suggested running a repair install on Portal.  

0 Kudos
JonathanQuinn
Esri Frequent Contributor

It's unlikely that a repair will work, as it's not the install directory that's the problem, but the index directory, (C:\arcgisportal\index, for example).