Geodata Service with 2-way Replication

[mpboyle]

I'm trying to setup a geodata service and use it as a means for 2-way geodatabase replication.  I don't have any problem creating the geodata service from a versioned SDE geodatabase.  However, I haven't been able to create a replica geodatabase and I'm not entirely sure why.

My goal is to publish a geodata service from a SDE gdb which other entities can use to create a 2-way replica and we can sync our edits back and forth...so as to have a common dataset for editing.  Unless I'm mistaken, this should be doable...?

I have the following questions about the workflow and was wondering if anyone has experience using a similar approach and could explain their methods?

1) When publishing the geodata service, do you use a versioned connection or a connection to the DEFAULT version?  Are there any permissions needed by the user the service is published from other than being able to edit the dataset?

2) What tool in ArcToolbox do you use to create the replica (Create Replica or Create Replica From Server)?  Or is there an alternate workflow for creating the replica through ArcMap?

3) Are there any necessary permissions within SDE that I'm overlooking?

Environment:

1) SDE = 10.1

2) ArcGIS Server = 10.3

3) RDBMS = SQL Server 2008 R2

Thanks in advance!

[mpboyle]

William,

What I have setup is more of a test environment.  What I have are two SDE geodatabases and two geodata services (one for each SDE gdb).  In my test environment, one SDE gdb is meant to represent the 'county' gdb and the other is meant to represent the 'city' gdb.

I have scripts setup that will sync changes, analyze datasets, compress, rebuild indexes, and analyze datasets once again on each SDE gdb.

So...for example, if I'm synchronizing changes to my 'county' gdb, then the I use a database connection file to the 'county' gdb as gdb1 and a ArcGIS Server connection file to the 'city' geodata service as gdb2.  What this would do is sync changes in both directions between my 'county' gdb and 'city' gdb using the geodata service as my remote connection to the 'city' gdb.  On the flip-side when I sync changes and run the gdb admin tools on the 'city' gdb, you basically just flip-flop gdb1 and gdb2...where gdb1 would be the database connection file to the 'city' gdb and gdb2 would be the ArcGIS Server connection to the 'county' geodata service.

I run these scripts on a daily basis and while we haven't been testing thoroughly, they haven't failed, and when I have tested by entering data into the 'county' gdb and then syncing back to the 'city' geodata service, it has been successful...likewise if I edit data in the 'city' gdb and then sync back to the 'county' geodata service.

Both the geodata services are within a secured folder on our test ArcGIS Server site.  Both SDE gdbs are versioned.

[WilliamWang]

Matthew,

Thank you for your detailed explanation.

We, too, have been testing with lots of frustrations. We have not been successful in running sync without having another Geodata Service (GDS). Meaning, 1 replica with 2 GDS. We weren't able to get sync to work both direction until the "child" GDB users publish a 1 GDS for the "parent" GDB users to absorb. An esri support analyst was able to duplicate the problem and agreed with our fining and it was noted as a bug. However, base on your description, you are running just fine. BTW, your county to/from city model is very much similar to ours.

What method did you use to bring your data over to the "child" GDB? We've been using Create Replica from Server GP tool rather than using the Distributed GDB toolbar in ArcMap.

Do you publish a map service along side with your GDS? We don't since we create our service from ArcCatalog.

As to my versioned question earlier, we've begun testing it recently and we found the if my Default GDB is set to protected and my versioned of the Default GDB is set to public, my outside entity would not be able to sync unless my Default GDB is reset to public.

Your experience and outcome were what we originally anticipated but never realized in our environment.

[mpboyle]

William,

It's been awhile since I created the replica...but if I remember correctly, what I did was use the 'Create Replica' (not Create Replica from Server) tool in the Distributed Geodatabase toolbox.  Input parameters would have been along the lines of:

a. Replica Datasets = 'county' gdb SDE database connection

b. Replica Type = TWO_WAY_REPLICA

c. Geodatabase to replicate to = 'city' geodata service

d. Replica Name = whatever you want

e. Advanced Settings = all defaults

So...the 'county' gdb acted as the parent and pushed data to the 'city' gdb using the 'city' geodata service as the connection.

I did not publish a map service since I did it all through ArcCatalog...and for what we're testing, I don't think it would be necessary or helpful.

I'm not sure what you mean by 'Default GDB is set to protected vs. public'.

I originally struggled with the concept of a geodata service and how to replicate the data using a geodata service.  I believe all SDE gdbs need to be in place as well as all geodata services.  You should be able to then use the 'Create Replica' tool in the Distributed Geodatabase toolbox using the 'child' geodata service as geodatabase to replicate data to parameter...that way you don't have to create a XML or some other static snapshot of your data and load into the 'child' / replica gdb.

[WilliamWang]

Matthew,

Completely agree with you on xml or FGDB matters.

We are testing our replication over the net between 2 networks hence we have to use Create Replica from Server GP tool or else the outside party will need to provide the "parent" DB to me to restore in my SQL then we have to do the final hand shake using Register Existing Dataset.

Thanks again for your info.

[JakeSkinner]

I worked with the State of Maryland where counties are replicating their street centerlines and address point data to the State.  Attached are PDFs for steps for the State to perform to create a geodata service, and another PDF for the county with steps how to replicate and synchronize their data.  For this scenario, it is just a one-way replication from the county (Parent) to the state (Child).  Not sure if this is helpful or not.

[WilliamWang]

Jake,

Your document helps. Likewise with Matthew's notes on Create Replica GP tool.

Interestingly, when using Create Replica tool yesterday, received a general replicating failed error message 00582 but when examining the "child" GDB, the replicating data appears. Any input on this?

Was able to reproduce the error and outcome twice.

[JakeSkinner]

I've seen this when trying to replicate a large amount of data using a geodata service.  One thing to try is to increase the 'The maximum time a client can use the a service'.  You can do this by right-clicking on the geodata service > Service Properties > Pooling.

If this does not work, it's recommended to create an XML file and then manually import this file to create the replica.

ArcGIS Help (10.2, 10.2.1, and 10.2.2)

If the network is available but not reliable, you may still want to consider using disconnected replication. A replica creation process running over a slow network may be time consuming and unreliable. With disconnected replication, you can export to a file and continue working without having to wait for the information to be sent over the network. In this case, however, you will want to create backups of these files in case they are lost before being imported on the target.

[WilliamWang]

Jake,

Your suggestion on increasing the pool time worked for my internal test - within the same server.

Prior to increasing the max time number, regardless of using Create Replica Wizard or GP tool, the replication would failed. In fact, via the Wizard, I did not get any error number except an error window that said replication failed, a syntax error.

Once I increase the max time from default of 600 to 1800, both methods worked. Will try on different and larger dataset that we are having trouble with.

Will you say the definition of "large" dataset that you mentioned earlier is fairly arbitrary? It seems, for city of our size, our data is small in comparison to other major cities or companies.

Also, we are testing and running full model replication, and thus far we are theorizing that a GDB cannot be replicated if it has a FDS that has both Geometric Network and M-N relationship. Any experience on this?

Lastly, on your suggestion of using xml. Is there a check list or some sort to find out if the network isn't at fault?

Thank you.

[mpboyle]

Jake Skinner​,

I'm re-visiting this thread in order to find out what you think is the best workflow for 2-way replication in terms of versioning.  What I mean by this is, in your opinion, if we were to setup a 2-way replication process using a geodata service as one of the gdbs, would you recommend syncing the replicas to a common version, or different versions?

Let me try to provide an example:

Scenario 1

GdbA and GdbB both have an 'Editing' version.  The replicas are based on the Editing versions, so when syncing is done each replica is syncing to the Editing version.

Scenario 2

GdbA and GdbB both have a 'CityEdit' and 'CountyEdit' version.  The County would edit within the 'CountyEdit' version and the City would edit within the 'CityEdit' version.  The County would have a geodata service reading the 'CityEdit' version and the City would have a geodata service reading the 'CountyEdit' version.  This way when edits are synced the would be put within separate versions and then reconciled-and-posted.

I'm wondering if you think it's worth having 2 separate editing versions?...or if it's just as well to have 1 common version that edits would get synced between before being reconciled and posted to the default version?

Perhaps there's a scenario I'm not thinking of that would be a better workflow, and if so, I'd appreciate any insight.

[JakeSkinner]

Hi Matthew,

Either scenario would work, here are some things to consider on how to make your decision.

Scenario 1

-Only one geodata service is required

-Only one reconcile/post would be necessary

Scenario 2

-Two geodata services are required.  One for the CityEdit and another for the CountyEdit.  Geodata services are CPU intensive.  It's usually recommended to dedicate a single core for a geodata service.

-You can reconcile/post the versions at different times, given you a little finer grain control when the parent version is updated