ArcGIS Data Store Under the Hood

27689
29
06-11-2015 09:02 AM
JimO_Leary3
New Contributor II

What is ArcGIS Data Store under the hood? Sql Server Express? Sqlite? Something proprietary?

Thanks

Tags (1)
29 Replies
DanielUrbach
Occasional Contributor II

Hi Jim!  To answer your question, the Datastore uses PostgreSQL under the hood.

stevecater1
Occasional Contributor

Have you seen or recommend any detailed documentation on the guts of ArGIS Data Store and how it integrates with Portal? Seems like a black box.

0 Kudos
DanielUrbach
Occasional Contributor II

It seems like Datastore is meant to be a simple solution for organizations who want to use Portal to host feature layers without necessarily having to deal with the inner workings of the underlying database.  So in a sense it was intentionally designed to be a black box.  As such I would not expect to see any detailed information about the framework of Datastore made public.

JohnDye
Occasional Contributor III

Yes, PostGreSQL - without the ArcObjects / SDE middleware. That gives the DataStore the advantage of being able to support a lot more Feature Services but on the other hand, don't expect to make a direct connect using pgAdmin. It's possible to direct connect but you can't do anything with the data because there is no ArcObjects / SDE interface to convert the ArcGIS Commands to a DB operation. It's all REST API with the datastore.

AlistairFox
Occasional Contributor

So once you've loaded data into it via a service, can you access it via catalog to export, or do you have to access it completely from the feature service?

0 Kudos
JohnDye
Occasional Contributor III

In short; Yes. You must access it from the Feature Service exclusively. You can add the Feature Service to ArcMap and then right click and export it to a local FC. However, I don't understand why you would. You'd essentially be creating a standalone copy of that data. A better idea might be to right-click on the service, choose editing and create local copy for editing. That way, you can synchronize your edits back to the Service, which will update the data in the datastore.

The problem with thinking about Hosted Services in the same way that you might think about traditional services is that Hosted Services do not have the SDE middleware sitting on top of the Database to expose it to ArcObjects. Without that there to act as the liason between ArcCatalog and the Databas, converting those ArcObjects commands to a valid DB operation, any GP tool you try to run, even an export which relies on ArcObjects, can't be performed.

PaulDavidson1
Occasional Contributor III

John:

What's the SOP for keeping data up to date?

We script most everything via python.

Is that an option with Data Store?

Or do you just republish the updated data via the original method?

Thanks

0 Kudos
JohnDye
Occasional Contributor III

Well, I dont work for Esri but I suppose it depends on the intent of what mean when you say keeping things up to date.

The Datastore, in my humble opinon, is meant to be a wild-wild-west of sorts. That is, since you don't want users creating, editing and deleting authoritative data but you also want them to be able to do some self-servicing of their basic GIS needs, you need a place for them to host their own data. That's the point of the datastore. It's a database that you can set up and pretty much not have to worry about. It's a decentralized database to host all the crazy permutations of end user data. The data within the datastore is not authoritative - it's user data. All their little files and personal layers.

When it comes to authoritative content - which where I work, we tend to define as content that a team of GIS Professionals creates, validates and publishes for end users to leverage within the Portal - that's where setting up an Enterprise Geodatabase and registering it with your federated server(s) comes into play. To do this, you set up a normal Enterprise Geodatabase on SQL Server/Oracle/PostgreSQL/etc. - whatever your DB technology of choice is and then register that enterprise geodatabase with each of your federated servers.

Once that's done, you load your authoritative content into that enterprise geodatabase and work with it the same way you always have. Direct-Connect from Catalog and run GP tools against it or if you're brave, work with it at the DB level. The key to making this work seamlessly with Portal is federation. You must federate your Portal with ArcGIS for Server. Doing so means that anything you publish to ArcGIS for Server will be made available in Portal. It also means that your Portal Groups and Roles will determine who has access to the services being sourced from your Enterprise Geodatabase.

So once you're federated and you've got your data loaded into your enterprise geodatabase and you've added it to an ArcMap session, go ahead and get it all nice and pretty and then publish it directly to your ArcGIS Server Site (Not Portal!). At this point, since ArcGIS Server is Federated with the Portal, it automagically gets picked up by the Portal as well and that resulting feature service will be added to your My Content directory in Portal. Depending on how your database is set up and registered with your Server, you may need to ensure that your Portal Users have database accounts in order to access the service. 

With this approach, what you're essentially doing is offering a decentralized data management system (ArcGIS Datastore) for end-user data so they can go nuts and do whatever they want with their personal data while simultaneously offering a centralized data system (your Enterprise Geodatabase) for authoritative/enterprise data.

In terms of managing data hosted in ArcGIS Datastore via Python, you don't. With the above approach, it's not your data to manage, it's the user's. You manage the Enterprise Geodatabase, user's manage their own data within Portal and the Datastore will take care of itself. If you're really edit data hosted in the Datastore through programmatic means, you're going to need to get familiar with the REST API and manage the data through the Service's REST Endpoints.

ThomasColson
MVP Frequent Contributor

The Portal Data Store is a PostGRE/GIS SQL Server. It essentially is a replica of the AGOL data store, but without "some" of the cool JSON API stuff you can do against the AGOL API. "Content" such as shapefiles, FGDB, etc... are in a sense stored as BLOB's, so you won't be able to interact with them outside of the Portal -> Arc Catalog Service Connection, like you would a bona-fide SDE, or CartoDB SQL database that would sit on top of PostGRE.

If you want to do things with SQL/SDE database content, such as automate data QA using SQL code yet interact with the Geography in Portal, than as others posted, you would have to federate a server to Portal to be a Hosted Feature Server. Hypothetically you could stand up another instance of PostGre within the DataStore and throw SDE on top of it to do this, but I suspect Esri tech support would hang up on you when they see this (and you will be on the phone weekly with TS even with an out-of-the-box install of Portal).