Working with distributed data using ArcGIS Enterprise and ArcGIS Pro

[DavidWatkins]

We’ve had many users ask about plans to support the ArcMap geodatabase replication workflows in ArcGIS Pro and we want to clarify a few things regarding replication workflows moving forward.  The need to access authoritative GIS data from anywhere at any time is more important than ever. This is causing a shift in how we access and interact with data. Web GIS patterns provide the means to share, access, and work with data in a variety of ways extending the ArcGIS Platform.  Because of this shift our general direction has been moving from the client/server model (directly accessing the geodatabase via a database connection) to a web GIS services model. We believe that there are inherent advantages in a services architecture.

In ArcGIS Pro and ArcGIS Enterprise, we have already been actively developing functionality that supports the new feature service sync technology. With ArcGIS Pro 2.1 we introduced offline editing workflows to allow maps to be taken offline when disconnected from the network. This takes the feature service datasets offline to a local geodatabase. Users can perform edits locally, and then sync those edits with the server. The bi-directional sync process allows the offline geodatabase to share changes made and receive updates others have made to the web feature layer. See the ArcGIS Pro documentation for more information: http://pro.arcgis.com/en/pro-app/help/projects/take-a-map-offline.htm

We have a mid-term project planned to further incorporate geodatabase replication workflows into ArcGIS Pro. We are still early in the planning phase of this project, but one aspect of the project involves leveraging the feature service sync technology. We want to leverage sync as it is available across the platform in both our online and enterprise products. The existing geodatabase replication tools will continue to work with data that is compatible with ArcMap and we encourage you to continue to use them until an ArcGIS Pro solution is available.

We are interested in hearing your workflows with distributed data. Please feel free to comment with your business requirements and how you are currently working with distributed data.

[JoshuaBixby]

David, one of the common geodatabase replication workflows in my organization involves replicating from parent geodatabases in a centralized data center to child geodatabases at remote sites in rural areas with low bandwidth or high latency networks.  Typically, the child geodatabases are workgroup geodatabases that others in the office connect to so that the same parent geodatabase data isn't being replicated over and over again on an already challenged last-mile network.  How does this workflow fit into Web GIS?

[ThomasColson]

There is a very fundamental problem with this idea: feature services are the backbone of this framework. I don't see any mention of being able to create a disconnected offline copy of a SDE geodatabase to a file geodatabase. It appears that ArcGIS Server will be required to implement offline editing in the Pro environment, which is the exact opposite of customer requirements and what is being repeated in many GN threads. I have no doubt there are advantages to web services architecture, but there are costs and barriers to implementing web and GIS servers that many of your customers will never be able to overcome. There's a lot of daylight between what you believe are inherent advantages in a services architecture, and what customer business requirements are. There is no dispute that Data as a Service via web architecture is the greatest thing in the world and the sync capabilities in ArcGIS Server are smoother, more reliable, and offer more capability, but the interpretation of your blog post here is, once Arc Map is put to bed, if you want to edit SDE data offline, you need to put an ArcGIS Server in front of it. I really hope ESRI hears this: Not everyone can stand up an ArcGIS Server in order to continue their offline enterprise GDB workflows. Nor do most folks have the capacity to publish every SDE feature class as a service. My ratio is out of 10 SDE feature classes only one of those is allocated the incredibly expensive access to a GIS server. 

So, many of continue to seek an answer to this question: With Pro, and Arc Map out of the support picture (2022? 3?): Will Pro users and DBA's be able to, using ArcGIS Pro, connect to a SDE database and create a local copy for editing as they can with ArcMap without ArcGIS Server in the picture? 

[mpboyle]

The vast majority of our replication processes are one-way replicas, where we are using geodatabase replication as a means of moving production data to replicated databases used for read-only users and web services.

We like the idea of having a production editing environment with a small number of users.  This allows us to easily make schema changes when necessary, and have it only affect a small number of users.  We don't have to notify our read-only users, and we don't have to drop connections to our replicated web services.  We can then propagate those schema changes to our replica databases using the Export Replica Changes, Compare Replica Changes, and  Import Replica Changes tool within the Distributed Geodatabase toolbox...all of which are not included in Pro.

If geodatabase replication tools are no longer available to us, this means we have to publish feature services for all our datasets we want to replicate???...the vast majority of which we have no real reason to ever want to have available as a service.  As Thomas Colson‌ mentions above, this would require a lot of server resources just to have these services running, not to mention, it would likely require us to stand up more than one, perhaps several Server machines, or allocate funds to increasing server resources (CPU, RAM, etc..).

The offline tools that are built into Pro seem nice for disconnected editing ... however, for us personally, we never use geodatabase replication for offline editing ... as I mentioned, we are using it for moving production data to replicated databases.  Instead of moving entire datasets, we are only moving edits between our production and replicated databases, and this workflow has worked very well for us in creating a production environment, an internal read-only environment for non-editors, and a web services environment.

Without the currently available one-way replication tools, the only way I see to maintain our current environment with available Pro tools is:

1. Truncate and Append  OR
2. Delete and Copy

Neither of these options are desirable.  In both cases we'd be moving entire datasets, instead of only moving edits.

[KevinDunlop]

Will the feature service sync be a real time process where edits automatically sent to both databases or will it be a process that must be kicked off (like current replica sync)?  If it is real time then it could cause problems.

For example if one of editors deleted all the records out of a synced feature class, then all our web users would be screaming at us.  By being able to control when a sync happens, we can roll back data if needed when something like this happens.  And it has happened before.

Secondly, will the sync feature service affect performance on both databases and what happens if one is lagging?  We use replication to separate our desktop production users from our web traffic which is mostly read only.  We replicate the production database to an internal web database and an external web database.  This has saved us several times where one system would get hammered while the others were not.  We have had scripts kick off in production database that brought it to its knees but our web services were unaffected since their database was separate.  Like wise we have had massive spikes in our web traffic (10,000 times normal) when an external users pounds us. When that happened our production databases wasn't affected since it was separated.  Have a feature service that connects them risks them having impacts from each other.  

[ThomasColson]

David Watkins user responses bring up many concerns about the future of managing data using SDE. Will any clarification be provided?

[JonHall]

RE: "...our general direction has been moving from the client/server model (directly accessing the geodatabase via a database connection) to a web GIS services model".

The architect of this "general direction" is out-of-touch with your user base.  ESRI needs a reality check.

Currently, web services only make sense for publishing GIS data on the front end, consuming that published data over the web, and in limited editing scenarios.

The plan that feature service editing will replace database replication is not realistic at this time.  A web services architecture might work 10 years from now.  For years, we've discussed similar web services architecture for provisioning distributed NG9-1-1 spatial databases, but no vendor has built even a working prototype yet.

Everyone commenting on this thread has business use cases that author data on one or many local area network clients connected to database servers that replicate to other database servers, in distributed multi-user versioned database editing environments.

ESRI has clearly jumped-the-gun on this. 

Do not experiment on your user base. 

Commit to supporting ArcSDE for another 5 years, while you work out the bugs in a next-generation web services model.

[KariRandall-Secrest]

Agreed. We are in a virtually identical situation to Matthew B. above "The vast majority of our replication processes are one-way replicas, where we are using geodatabase replication as a means of moving production data to replicated databases used for read-only users and web services" and would rather not stand up a portal, just to be able to continue replication processes. Schema changes are fairly painless, which occur more often than one might think when data gets designed. We too, have many feature classes in our SQL server databases that we don't see a need to create or maintain feature services from.

[JonathanFarmer_oldaccount]

Hi Jon,

While I can't speak on the specifics on the next generation data distribution plan, I did want you to be aware that we are certainly taking care of this line:

"Commit to supporting ArcSDE for another 5 years, while you work out the bugs in a next-generation web services model."

ArcMap 10.6.1 is supported until January 2024 and ArcMap 10.7 should be coming out Q1 of this year which will extend that time frame. So the current distributed data model will be alive and well (read supported) for a good bit.

Jonathan

[CarlosKrefft]

David Watkins :  What's the status of the mentioned project planned to further incorporate geodatabase replication workflows into ArcGIS Pro? Can you give us any sort of update?

[NickHetrick]

We use the distributed database tool set as way to separate environments and also as way to integrate GIS data with platforms such as asset management solutions that integrate with ESRI Geodatabases. It has been an effective toolset for us we also integrate checks and QC into the replication script we run nightly to clean or stop bad data from getting moved to certain environments. 

[DavidColey]

I like the idea of moving to branch versioning and leaving current distributed data transfer model behind especially after a decade of battling failed data replication syncs.  We just need the branch versioning model to handle more complex features objects like topology.  Plus we really need the new utility geometric data model to be fully updated to hand water and sanitary.  But once that happens I feel like we will be able to move out of maintaining all of our user SDE databases and work from a single SDE user db.  

[ErikLash1]

Just getting into planning to set up distributed workflows for our municipality. Web GIS does not work for us due to a number of infrastructure related problems and fiber limitations. We tried for years to make the single GIS server hosting up data for editing to work but alas, throwing in the towel on that and moving to distributed and even mobile data servers to accommodate operations that stand up at different locations.

Makes no sense for users to suffer through bandwidth limitations and bottlenecks throttling data flow between offices via a central repository and overworked servers when distributing data directly to a building/office networks directly works so much better and speeds up data editing and manipulation capabilities by factors that make the end user happy.

Somewhat disappointed to see Pro moving away from supporting distributed and enterprise needs in support of workflows that really only work in urbanized areas where data is always open, everyone is always connected, and fiber practically runs to each machine. That's only a small fraction of GIS users realities. While I understand the "dream", it's not practical for many, and many more than ESRI is willing to admit will never be able to capitalize on their new business model.

We can't even get a "view" of one of our contractors DBs hosted on the mainland to work with any consistency or reliability inside our network. How in the world would we be able to support all of our various business functions through "web GIS"? Data throughput would be through the roof to even try.

We are moving from hosting much of our data in GIS server to hosting very little of it in GIS Server. Distributing data to local sites is the future of GIS, not the past, as data needs grow and infrastructure doesn't keep up. 

Plus, with ISPs now legally able to throttle bandwidth however they wish, moving to Web GIS is simply a very risky prospect when you require guaranteed up time. Not to mention the increased frequency and complexity of attacks on the national and international backbone infrastructure by malicious actors. 

My .02 is that Web GIS has already had its heyday … another reason we'll be on ArcGIS Desktop until ESRI finally retires the final version of it. We have to be able to function and Pro doesn't contain the tools that will allow our business units to continue to operate.

Some of my users already switching to QGIS and what we are seeing is a clear differentiation between workflow components - data maintenance and daily workflows on desktop using DBs and "published" final data sets being set up on the GIS server.  

Offline editing of Database data directly is absolutely a root level business necessity for us. We are under grant requirement to maintain local data stores …. another use case that ESRI completely misses the ball with in Pro.

+1 to the idea of branch versioning with complex requirements such as topologies. 

[ErikLash1]

Just adding a comment to my post that while I would posit in a theoretical discussion that web GIS is past its heyday, I see web databasing as just beginning to grow into its own legs...hence the need for better DB management tools in Pro.

Pro is ideally situated to be a leader in web db technology with it's Conda environment leading the way. Spark clustering capabilities, server-side processing in SQL Server via Python and R integration on the Microsoft side, all truly powerful aspects of the implementation.

Problem as I see it is a disconnect between the data analytics capabilities that are so awesome in Pro and what is actually being implemented by the product team for the front end.

Maps are just 'views' of data, the end product of GIS Data, not the starting point. Can't make a map if you don't have data. What Pro lacks are the tools interfaces to actually work with 'data' and data storage containers.

For instance, I'd love to just create some data matrices and then work with them in Pro, much the same way 'R' or any other data analytics tool does. Data is easy. It's some spatial descriptors attached to attributes and stored in a container that allows it to be parsed. So why would I need to make a map to work with my spatial data? Never understood that myself. Generally maps are made so that the data can be understood visually. Totally get that. But the data itself requires a certain integrity before it can be understood in a map and that's where it all begins...working with data storage formats and the attributes contained therein. 

Web GIS without direct access to the data storage mechanism is kind of a broken system imho. It makes Pro just a client side software. If it isn't going to work for data management, they need to dump the FGDB and other proprietary formats and just say "we will work with all yer DBs" and open up their APIs.

I miss the Coverage model.

[JeffClough]

And still no meaningful comment from Esri on the concerns raised.  The silence is deafening...

[CarlosKrefft]

Our Business case: For every one of our construction projects we have contractors from about a dozen separate companies contributing data. These contractors need to be able to extract a Geodatabase (FGDB) to work with (disconnected) using their tools in THEIR environment. They may even import it into their own SDE. Many of them are permitting specialists, surveyors, engineers, etc. They use tools like CAD to complete their work then re-sync the FGDB back into a version of our SDE. 

Offline Sync in ArcPro doesn't cut it. The offline Geodatabase is a black box and can't be exported or imported in any way. We need data management tools in Pro and it would be nice if those tools could be used with offline geodatabases.

[ThomasColson]

https://community.esri.com/ideas/14400-pro-bring-back-the-manage-replicas-tool doesn't show "In Product Plan" yet. Hope that changes soon....

[DavidWatkins]

Thank you all for your comments and feedback. We really appreciate all of your responses. I apologize for not providing an update sooner.  We have been in an information gathering phase, and since my original post we have been following the activity on this thread, the ideas site, and technical support.  We have also met with many users at both our Federal GIS Conference and the Developer Summit, and we have been actively working on a strategy for addressing your needs.  If you have additional items or information from your workflows that you haven’t seen already mentioned in this thread, please continue to add them.  We are currently making some decisions about how we will support your workflows in Pro.  As soon as we have finalized our plan I will share it. 

Thanks again,

David

[DavidWatkins]

Thank you for all your feedback. It has been very helpful as we have been evaluating how we will support your workflows with ArcGIS Pro.  We have started a project that we believe will meet your requirements.  We have planned a phased roll out.  Initially, we want to provide the tools to allow you to create and sync replicas in the next release of ArcGIS Pro.   See the following UC Q&A for more information: https://www.esri.com/en-us/about/events/uc/get-involved/q-a#id=3925

David

[KoryKramer]

And the ArcGIS Pro Roadmap has been updated for July 2019!

https://community.esri.com/docs/DOC-13641-arcgis-pro-roadmap-july-2019

[ThomasColson]

David Watkins‌ https://www.esri.com/en-us/about/events/uc/get-involved/q-a#id=3925 seems to exclusively mention replication in the feature service framework "The long-term plan is to use feature service sync capability as the method for keeping in sync multiple datasets that are being independently edited.". Many of the comments in this thread make extensive reference to the fact that many enterprise customers are not "publishing" each and every feature class as a feature service. Ignoring these comments and blindly proceeding down the "feature services or nothing path" would be a very unfortunate, and disappointing, step by ESRI. If future plans include being able to perform replication and disconnected editing without the need of a Portal or GIS Server in the mix, I recommend clarifying those plans. If opposite, also suggest clarification so we can start porting our enterprise databases to another platform in preparation for ArcMap retirement. I see https://community.esri.com/ideas/14400-pro-bring-back-the-manage-replicas-tool is in "Product Plan". I'd like to point out that the idea does not request that replication be dependent on the services framework. 

[DavidWatkins]

Just to clarify, we are implementing geodatabase replication starting with the create and sync replicas geoprocessing tools along with an interface to manage replicas. This is traditional geodatabase replication. 

At the same time, we will be continuing with the development of services-based sync workflows. You will be able to choose which one best suits your needs.

Thanks,
David

[PaulHoefflerGISS]

David Watkins, what is the target release for the sync replicas geoprocessing tool? We're migrating users to Pro and have an NGDA, production of which is dependent upon geodatabase replicas, thereby pushing them back to ArcMap.

edit: I see "near-term" / "next release or two" in Kory Kramer‌'s post at https://community.esri.com/docs/DOC-13641-arcgis-pro-roadmap-july-2019, but it will be helpful to know whether or not 2.5 is the target release

[NanaDei]

Paul Hoeffler‌, we are targeting the ArcGIS Pro 2.5 release.

[DarylHochhalter]

I would just like to second Mathew Boyle's comment, even though it seems we are already set to get the utilities we need.

There is no good reason for ESRI to limit the users ability to use replication which is really all they would be doing by following the course they seem to have chosen. Basically, it would only be available through online services and now called sync. From my perspective, we need the ability to replicate outside the feature service workflow.

We are using the same workflow Mathew Boyle mentioned. We have some data we make available to the public and some data we collect from the public, the data requires different permissions and is exposed in different ways to the public. it is an extra layer of security to publish the services from replicas of our enterprise database. In fact it works quite well to publish services from replica FGDB's for webmaps that will only be viewed by users having no need or ability to edit.

[PaulHoefflerGISS]

Nana Dei‌ is this still targeted for the ArcGIS Pro 2.5 release? In "January" as we understand it?

Thanks!

[DarylHochhalter]

https://community.esri.com/docs/DOC-13641-arcgis-pro-roadmap-july-2019 

[PaulHoefflerGISS]

Thanks for the link - I'm hoping for word from Esri that's more current than July and more solid than "Next release or two" as we've seen features and dates slip many times.

[KoryKramer]

The gp tools and a manage replicas pane are installed in Pro 2.5.

Unless something crazy happens, we should see this in 2.5.

I hope this helps.

[PaulHoefflerGISS]

Very much, thanks Kory Kramer‌!

[ThomasColson]

Way cool. I just got my EAP link today, I'll check it out. 

[MPC_KineticGIS_Department]

It's promising that we may finally be getting some of these tools again. I do question if the issue in relation to definition queries and replications has been solved yet? Its an issue for all disconnected environments across the platform, Collector for ArcGIS, ArcEnterprise Collaberations, ArcDesktop. FIlters and queries have been the backbone of GIS data visualisation for years but in an offline workflow they are incredibly buggy.

In our use case scenario we have design to push out to the field after it clears quality checks. Simple definition query to only show approved values in the data. Works perfect when a value changes to approved. Where it goes bad is if the value changes back the other way. The features will remain in the child, WITH THE INCORRECT VALUE.

A replication sync needs to complete and then for a filtered replica/service/branched version needs to rebuild the included GlobalID list AFTER the sync has occurred and clean up the child data copy.

[CarlosKrefft]

Is there any way we can try 2.5 now before it's released?

[ShaunWalbridge]

Likely its too late for 2.5 now, but you can find out about beta testing opportunities through this page:

  Early Adopter Community | Esri Early Adopter Program 

[PaulHoefflerGISS]

I downloaded ArcGIS Pro 2.5 beta yesterday - we were told that the Pro Early Adopter (beta) Program is by invitation only and we had to provide specific information to our Esri customer service representative to gain access. I'd contact Esri if you're interested in any of their beta programs - some are more widely available through a site linked from Shaun Walbridge‌'s link above.

[KoryKramer]

Since this sub-thread is blowing up, I already DM'd Carlos Krefft‌ yesterday to contact me offline so that we can get him invited into the EAC.