Adding Machine to ArcGIS Server, Geocoder now fails to return results

[JoshStreufert]

We added a machine to an existing ArcGIS Server installation.  Everything works fine, except our custom geocoder isnt returning results to downstream apps.  I've checked the logs and Machine 2 gets the request and starts the Geocoder Server Object, and Machine 1 states that the request was successfully processed but the down stream app just spins with no results.

Here's an example log file from yesterday.

Level	Time	Message	Source	Machine	Code
FINE	Jul 15, 2019, 10:42:51 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 10:35:23 AM	REST request successfully processed. Response size is 180 characters.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100004
INFO	Jul 15, 2019, 10:35:23 AM	Reverse geocoding result: Shape=IPoint(779244.448357, 282277.204118), WKID: 102629&Street='3411 GRANBERRY'&Loc_name='siteaddressU_l'	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	70024
VERBOSE	Jul 15, 2019, 10:35:23 AM	Reverse geocoding location: IPoint(779257.962181, 282303.170134), WKID: 102629	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	70041
FINE	Jul 15, 2019, 10:35:23 AM	REST request received. Request size is 445 characters.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100001
FINE	Jul 15, 2019, 10:34:52 AM	REST request successfully processed. Response size is 146 characters.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100004
VERBOSE	Jul 15, 2019, 10:34:52 AM	Reverse geocoding: nothing found	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	70024
VERBOSE	Jul 15, 2019, 10:34:52 AM	Reverse geocoding location: IPoint(778940.025667, 282412.460811), WKID: 102629	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	70041
FINE	Jul 15, 2019, 10:34:52 AM	REST request received. Request size is 447 characters.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100001
FINE	Jul 15, 2019, 10:31:51 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 10:20:51 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 10:09:51 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 9:54:33 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 9:43:33 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 9:13:43 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 9:02:43 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 8:51:43 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 8:33:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 8:22:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 8:11:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 8:00:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 7:49:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 7:38:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 7:27:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 7:16:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 7:05:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 6:54:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 6:43:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 6:41:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 6:39:47 AM	REST request successfully processed. Response size is 76 characters.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100004
FINE	Jul 15, 2019, 6:39:47 AM	REST request received. Request size is 36 characters.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100001
FINE	Jul 15, 2019, 6:36:29 AM	REST request successfully processed. Response size is 6153 characters.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100004
FINE	Jul 15, 2019, 6:36:29 AM	REST request received. Request size is 2 characters.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100001
FINE	Jul 15, 2019, 6:34:13 AM	Finished GeocodeServer object construction.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	70002
FINE	Jul 15, 2019, 6:34:12 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 6:34:12 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	70001
FINE	Jul 15, 2019, 6:32:16 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 6:21:15 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 6:19:27 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 6:19:24 AM	Binary request successfully processed. Response size is 757 bytes.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100004
FINE	Jul 15, 2019, 6:19:24 AM	Binary request received. Request size is 173 bytes.	Production/Production_CompositeLocator.GeocodeServer	CCARCW01P	100003
FINE	Jul 15, 2019, 6:10:16 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001
FINE	Jul 15, 2019, 5:49:56 AM	Starting GeocodeServer object construction...	Production/Production_CompositeLocator.GeocodeServer	CCARCW02P	70001

[BasWest1]

Have you had any luck resolving this issue? I'm having the same problem and so far haven't been able to find a solution.

[BWSCIM]

Bas,  When push came to shove we just removed the 2nd machine. As soon as we did that, the Geocoder worked as expected.

[JacobSnyder1]

Hi Josh,

Is the account that is running the ArcGIS Server Windows service on each machine a domain account or a local account?

Could we try running the Configure ArcGIS Server Account utility on both machines using the existing ArcGIS Server account, config-store, and directories? Running this utility with the existing configurations will ensure that the ArcGIS Server account has permissions to all file locations that it needs to in order to operate successfully.

Thank you,

Jacob