Unable to publish map service with data in enterprise database using Server 10.4

ThomasPuthusserry · ‎05-24-2016

I have upgraded ArcGIS Server 10.3.1 to 10.4 and facing issues with publishing map services with data stored in enterprise databases.

The server is hosted in Amazon AWS with following infrastructure and works fine:

Windows 2012 R2

Server 10.3.1

PostgreSQL 9.3 (in the same server - localhost) with SDE enabled

PostgreSQL as AWS RDS with SDE enabled

However, when the server upgraded to 10.4, the service with data in enterprise databases (both localhost and AWS RDS) which used to publish are no longer working. The error is

The ArcServer log file shows the following details

Error executing tool. PublishServiceDefinition Job ID: j5a9a52478ffc49d19328e5ceefbb609f : ERROR 001487: Failed to update the published service with the server-side data location. Please see the server's log for more details. ERROR 001369: Failed to create the service. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer

Failed to create the service.: Updating the server connection string for layer sdedb1.web.%gwstatmaster_staging Events failed. Attempted connection string was ENCRYPTED_PASSWORD=00022e68334e624d44786b57627653553542393435702b3845413d3d2a00;SERVER=xxxxxxx.c11tt3rfoa7s.eu-west-1.rds.amazonaws.com;INSTANCE="DSID=a8960749-e49c-41db-a095-a8a257a693b4";DBCLIENT=postgresql;DB_CONNECTION_PROPERTIES=xxxxxx.c11tt3rfoa7s.eu-west-1.rds.amazonaws.com;DATABASE=sdedb1;USER=web;VERSION=sde.DEFAULT;AUTHENTICATION_MODE=DBMS. Table name is sdedb1.web.%gwstatmaster_staging. Please verify the data exists on the server. System/PublishingTools.GPServer

If I uninstall 10.4 and reinstall 10.3.1 on the same server, everything back to normal and working fine.

Has anyone else experienced the same issue?

Thanks and any suggestions much appreciated

Thomas

TrevorHart2 · ‎10-24-2016

Been banging my head for months with this problem.

I've got one problematic site that was upgraded from 10.4 to 10.4.1 and suddenly stopped publishing services with the same error as the original question above.

All existing services ran fine and if I published manually (as per the suggestion above) it worked fine too.

The service publishing patch did not solve the problem.

With this problematic site I noticed;

1) The data sources would not validate (just hangs) from Server Manager.

2) The lack of the dsconnections.lst file under "...\arcgisserver\config-store\data" (compared to a healthy 10.4.1 site).

WORD OF CAUTION; Messing with the config directory was a last resort before building the site again. This approach is not recommended nor supported. Seek proper support from Esri first before you try any of this. Also make sure you have backups as fiddling can corrupt your site.

To resolve I;

1) Shut down my service on both sites (ie broken and healthy)

2) On the broken site I deleted the contents from config-store\data

3) Copied over the contents of config-store\data from the healthy site to the broken site

4) Started up the broken site and the healthy site

5) IMPORTANT, I used ArcCatalog to delete all of the data stores from the broken site (EGDB and File Shares)

6) IMPORTANT, I used ArcCatalog to recreate all of my data stores (they would not recreate from Server Manager)

7) The data sources were then able to be validated

😎 The dsconnections.lst files timestamp was updated

After that I was able to publish (and swizzle) successfully. Your results may vary but this is what worked for me.

UPDATE

Actually it looks like validating the data sources from ArcCatalog (on the same machine as ArcGIS Server) generates/updates the dsconnections.lst file. As a test I stopped the service, renamed the lst files, restarted the service and then validated the data sources using ArcCatalog. The files were regenerated.

There are two locations to check for dsconnections.lst are

...\ArcGIS\Server\DatabaseSupport

and

...\arcgisserver\config-store\data

If these files are missing try validating the data sources from ArcCatalog first before fiddling with your config.

GIS_Solutions · ‎01-26-2017

I met with the same problem, after a long time reviewing the source of the problem.

I noticed that your alias is the same in a few areas.

After checking and fixing the nicknames, there was no problem.

MichaelVolz · ‎05-05-2017

Does anyone know if this problem occurs in 10.5 as my organization plans to upgrade from 10.3.1 to 10.5 where 10.4.x will be completely bypassed?

JonathanQuinn · ‎05-05-2017

If the problem is BUG-000096129, this is fixed at 10.5. It shouldn't matter what version you upgrade from, as long as you get to 10.5.

LanceGoens · ‎08-03-2017

We have 10.5.1 and are seeing the Swizzle failed error...

BrianBolduc2 · ‎08-03-2017

try from a different database. We have about 10 sql databases on our server. One of them was giving us this error. We are currenlty upgrading to 10.5.1 and our main database has this issue but the others work fine. 10.3.1 ArcGis Server will publish fine but 10.5.1 throws the swizzle error. But once I pointed to a different database it published fine. I think it is something jacked up in the sde tables or something.

TrevorHart2 · ‎08-03-2017

I'd really suggest you try validating the data sources from Server Manager as well as ArcCatalog.

Search for dsconnections.lst and view the contents in Notepad. Usually there is a broken connection in there!

LanceGoens · ‎08-03-2017

First, the data sources validate fine in both ArcCatalog as well as the ArcGIS Server Manager site.

We did notice that the log message, after publishing failure, contained an Instance parameter set to some "DSID=<GUID>". When we opened the dsconnections.lst file, we are seeing the Instance parameter showing a value of "sde\:oracle11g\:<instance name>.<server name>" - the <instance name>.<server name> is what we have in our tnsnames.ora file fyi - again, all validations work. That line in the lst file was prefixed with the same GUID - it's like it's replacing the Instance parameter value with the guid... not sure how or why or if it even should (swizzle service problem?)???

Also, it should be noted that we do not have the 11g client installed as shown in the lst file, it's the 12c client. We have 2 virtual machines. One for Desktop (ArcGIS 10.2.1 and 32-bit 12c Oracle client - no 64 bit client on this machine) and another for Server (ArcGIS Server 10.5.1 and 64-bit 12c Oracle client).

MichaelVolz · ‎08-04-2017

Lance:

I have done a good deal of testing with SDE connections with the Oracle 12c client. In the past for my organization when Oracle 11g was the standard, when we created connections just the server name from the tnsnames.ora file was added for the instance (e.g. sdex), but when the Connection Properties were viewed the instance name appeared as sde:oracle11g:sdex. I thought this was no big deal since the Oracle 11g Client was being used. Now that my organization is migrating to Oracle 12c, I am finding that even on a computer where the Oracle 11g Client has never been installed, sde:oracle11g: is automatically being added to the Instance (This appears to be hard-coded as I have searched the Registry and cannot find any references to Oracle 11g on the machine).

In addition if I change the Instance to sde:oracle12c:sdex, I get a broken connection from the Oracle 12c Client to the Oracle 12c database. Even though the default sde:oracle11g: connections work now, I am concerned that at some future date (maybe when Oracle 13c becomes the standard), all the SDE connections will be broken (for my organization this could be tens of thousands of SDE connections in thousands of mxds).

ANRGIS · ‎06-07-2017

Validating our data stores in ArcGIS Server Manager resolved the issue for us.