Data management stratigies for small organizations: FGDB, Portal Data Store, AGOL

[LyonMNGIS]

Hello,

I will be setting up an ArcGIS Enterprise Server sometime in the near future for a small organization of about a dozen users.  I would like recommendations as to if I should store data in a File Geodatabase, a Portal Datastore or a combination of the two.

Questions:

• Is there any use case for storing data in a FGDB or should I only use a portal data-store?
• How do users host land ownership parcels that are updated nightly?
• What format do you recommend to author "traditional" ArcGIS Server maps?
• Can we use a FGDB to publish any data to Portal or are we required to use a data-store?
• When do you recommend using ArcGIS Online, vs Portal?
• Do you have any other recommendations for storing data that will be consumed within our organization, a "traditional" ArcGIS Server map, and ArcGIS Portal?

Thank You

I would like to know is more

how I should store our GIS data.  My objective is to maximize performance on our server while keeping our data simple and easy to maintain.

Please let me know when you think it is more appropriate to use a File Geodatabase, a Portal Datastore or a combination of the two.

Are File Geodatabases relevant anymore or should I use a data store for everything?  I would like to maximize performance while maintaining a simple and easy to maintain data setup.

1. Would you FGDB or a data-store or larger dataset such as elevation contours?
2. Can you use a FGDB to share data to Portal or are we required to use data-stores?
3. How do people share datasets such as parcels that are updated nightly?
4. What use cases

[jcarlson]

File Geodatabases aren't a good option for Server, because you won't have an easy way to update the data once it's uploaded to the server machine. An Enterprise Geodatabase is good for publishing complex datasets that you need editing access to.

For simple data layers, utilize the Data Store as much as you can. If you have to publish registered services, consider using shared instances for them. Both options allow your server to more efficiently allocate resources as needed.

For frequently updated layers like parcels, you might consider having a Python script or something similar to push updates to your portal layer.

If you anticipate large volumes of data, Enterprise may be better than AGOL for storage costs. But there are a lot of bigger differences between the two. If you plan to use the Parcel Fabric in a multi-editor manner, you pretty much need Portal.

[LyonMNGIS]

Josh,

I forgot to ask about SQL Enterprise Databases.  I will not have access the full version of Microsoft SQL Server.  Do you think it is worthwhile to use SQL Express? Alternatively can I use a PostGRES Enterprise Geodatabase along side a Portal Data Store that also uses PostGRES?

Thanks!

[jcarlson]

Oh, definitely. PostgreSQL is all we use over here, and it works great. I highly recommend it. And if you create your databases using the built-in st_geometry types instead of the Esri-specific ones, you'll be able to query your features and visualize them right in tools like pgAdmin and the like.

[ChadKopplin]

Using an Enterprise Geodatabase allows your organization to have multiple users editing your data, plus you would be able to serve all of your data to AGOL and Portal in a series of services.  Your larger base layers could be in one of multiple services, and you could have separate services for your organization data.  You would not have to store data on AGOL.  This is the model that we have incorporated here at Wyoming DEQ.   As an example we currently are working to remove 2 of our projects that are consuming 3,000+ AGOL credits every 30 days, because they use large datasets that are currently stored in AGOL.  Use portal primarily if you are creating Web Map's and Web Map Applications that will only be used internally.  Just for your information, if you use portal and you federate the server and the portal, most of the administration of the server moves to your portal.  For the larger datasets I would definantly use the Enterprise Geodatabase, and therefore create a datastore.  I hope this helps you.  Thank you.

[Brian_Wilson]

Some things that were not clear to me (and some that are still hazy)

I think of the "ArcGIS Data Store" component only as storage for our online services.  Things only go into it as hosted data and then are read-only.  They get refreshed as things change. Close to having something automated, currently it's a bunch of scripts.

We keep the working GIS data in SQL Server and since our IT group manages the server it's not my problem to do the boring parts (upgrades, capacity planning, backups, uptime...) Personally I'd use PostgreSQL if I was forced to do that part. That is just a matter of comfort for you, they both work fine.

My understanding of Parcel Fabric is you can use File Geodatabases if you are not using versioning, that means no multi-user access. If you use versioning then you have to use the new versioning which means you have to run all access through Portal. Personally I hate this idea but no one asked me 🙂 Our cartographers will probably not transition to Pro until well after I retire so it's mostly a moot point for me.

We keep our public web services on our Portal, it's not a problem for our small county. We keep a kind of emergency backup on AGOL that uses almost no storage and gets no traffic ordinarily so low credit use.

So,,, direct answers to your questions........

Questions:

• Is there any use case for storing data in a FGDB or should I only use a portal data-store?
• How do users host land ownership parcels that are updated nightly?
• What format do you recommend to author "traditional" ArcGIS Server maps?
• Can we use a FGDB to publish any data to Portal or are we required to use a data-store?
• When do you recommend using ArcGIS Online, vs Portal?
• Do you have any other recommendations for storing data that will be consumed within our organization, a "traditional" ArcGIS Server map, and ArcGIS Portal?

Thank You

I would like to know is more

how I should store our GIS data.  My objective is to maximize performance on our server while keeping our data simple and easy to maintain.

Please let me know when you think it is more appropriate to use a File Geodatabase, a Portal Datastore or a combination of the two.

Are File Geodatabases relevant anymore or should I use a data store for everything?  I would like to maximize performance while maintaining a simple and easy to maintain data setup.

1. Would you FGDB or a data-store for larger dataset such as elevation contours?

I would keep contours as vector data (polylines) in the enterprise GDB and then publish them as vector tiles and store them in the Datastore, because they change very slowly and vector tiles render fastest.

1. Can you use a FGDB to share data to Portal or are we required to use data-stores?

You can register as many FGDBs in your Portal as you want, but they will be basically read-only since they cannot be versioned. If you have one person own them or you know only one person needs to edit at a time that works but why bother? It's work initially to set up a database engine (Postgres or SQL Server) but easier in the long run than keeping track of a bunch of shared FGDBs.

1. How do people share datasets such as parcels that are updated nightly

We keep them in the SQL Server currently and publish them out as a Feature Layer in Portal. They use traditional versioning because the maintainers use ArcMap. I want to have a workflow to publish them out as a vector tile layer for speed in my web maps but I don't have that going yet

We moved directly from shapefiles and FGDBs to SQL Server about 5 years ago, then have been slowly sorting out which services should be "hosted" and vector tiles and what changes slowly vs what needs to be live data, your examples of parcels and contours being great examples of the extremes.

I generate a new "base map" about once a month with roads, parks, water features, etc and the process reprojects everything into Web Mercator into a local FGDB as a workspace and then produces output as Vector Tiles and a Map Image Layer for queries (pop ups) into the data store as "hosted" data.

I have most of a workflow in place to do this with parcels and taxmaps but not quite. Currently the taxlots paint too slowly on the web site but I am the only one who notices so fixing it is not a priority for anyone else.

(It can be used an actual "basemap" but gets put on top of aerial photos in practice.)

You are welcome to look at the public map at https://delta.co.clatsop.or.us/apps/ClatsopCounty/

I can't remember exactly which services are vector tiles (contours?) other than the base layers; I am all about speed and aesthetics and my supervisor is all about getting things done easily and she is comfortable with those slow registered feature services that just work. We are very complementary, she is all about clicking buttons and I write code. Not sure what happens next year when I retire. They will need someone willing to code but not greedy. Why would anyone choose that? Other than me? 🙂