Skip navigation
All People > WSimonazziesriaustralia-com-au-esridist > Walter Simonazzi's Blog > 2016 > March > 02

Walter Simonazzi's Blog

March 2, 2016 Previous day Next day

The term ArcSDE/SDE is being gradually replaced with the term “Multiuser Geodatabase” and this is probably creating some confusion among the users of the ArcGIS platform. In order to understand the difference between ArcSDE and multiuser geodatabase, let’s start clarifying what is and what is not ArcSDE.

What is not ArcSDE:

ArcSDE is not a product. It’s a technology. In the same way ArcGIS is not a product, is a platform. Only prior to ArcGIS 9.2, ArcSDE was a standalone software product. At the ArcGIS 9.2 release, ArcSDE was integrated into both ArcGIS for Desktop and ArcGIS for Server.

ArcSDE is not only the application sever connection, a piece of software that packs the giomgr.exe and a gsrv.exe processes. This is a common mistake that I have mentioned in my previous post but from a more business perspective. I’ll try to do the same exercise now but from a technical perspective.

So, what is ArcSDE  – also called “the database enhanced with the ArcSDE technology” or simply the “ArcSDE geodatabase” or the “Multiuser geodatabase”?

Let’s answer this question step by step.

First of all, ArcSDE is a hybrid technology built up from different components. These components are implemented in two places, the client (ArcGIS for Desktop/Pro or ArcGIS Server, mobile, etc.) and in the relational database management system (any of the supported ones).

The components that make the ArcSDE technology are those shown in the slide number 3 and 4 below.

cmgd

Note that, the above are the slides of the official geodatabase training course version 10.2, taught all around the world therefore, at least until version 10.2, we have been using, explaining and discussing what ArcSDE is.

The slides below are from the same course at version 10.3. Note that the content is the same although the term ArcSDE has been replaced by “Multiuser geodatabase” (except that the translator component is only in the client side reflecting the deprecation of the application server connection).

CMGD_103

What are the components of the ArcSDE/Multiuser geodatabase technology? – Note that the lack of any of the components described below would mean that the multiuser geodatabase wouldn’t work.

  • The client (ArcGIS for Desktop (32 bits) , ArcGIS Pro and ArcGIS Server (64bits), etc.)
  • The repository of ArcSDE/multiuser geodatabase, the geodatabase system tables, stored procedures and functions.
  • The management tools:
    • Geoprocessing tools
    • Database tools – for example, the functionality to backup and restore is considered part of the ArcSDE technology.
    • And until version 10.2.x, the command line tools. Deprecated in version 10.3.
  • And finally, the translator, this is the component that translates every map interaction(clicks on the map performed by the user) into SQL statements – Since the database doesn’t understand “clicks on a map”, it only understands the SQL language, we need something that acts as an “interpreter” between the user and the database.

Notice that in the slide above (version 10.2) the translator component is floating in the middle of the client and the server. There is a reason why this is so… the reason is related with how the client connects to the multiuser geodatabase. Up to version 10.2, there were two different connection types, application server connection and direct connections – first released with version 9.2 -. Let’s review some of the particularities of each connection type:

  • Direct connections (two tier connection): If the client connects using direct connections, the translator component is located in the client side and requires to have the database client installed in the machine where the ArcGIS client is installed -. This is the default connection method.

 

transl

 

  • Application Service connections (3-tiers connection). The application server connection has been the traditional way to connect to an enterprise geodatabase. It requires a service (commonly called “sde” or “arcsde”) which is normally installed and configured in the same machine where the database server is installed.

transl22

 

The sde service packs two processes– the giomgr and the gsvr – the giomgr.exe is in charge of listening for incoming connections (clients creating a connection to the enterprise geodatabase using the sde service), when the connection is accepted, the giomgr fires up the “gsvr.exe” process, one per client connection. Each gsvr.exe process will then translates the clicks on the map performed by the user into SQL statements that are then passed to the database, the database interprets the SQL, generates the answer which is finally sent to the client through the gsvr.exe process, as it is shown below.With this connection type all the translation load is placed on the server side.

 

sde_service

 

As I mentioned before, the application server connection method has been deprecated in version 10.3 and with it, the command line tools.I’m sure there are many reasons why the application server connection has been deprecated. From my point of view there is one reason that deserves to be discussed more in detail. Performance.When the application server connection was released the processing capacity of the servers was much higher than the processing capacity of the workstations. The application server connection was a good approach to put all the translator processing load into the server machine and so to use the server processing capacity to perform the translation.Nowadays however, there are reasons why we wouldn’t want to use application server connections:

  • Today, we can use client’s machines processing capacity to perform the translation and in this way, to share the translation load among all the clients connecting to the multiuser geodatabase. We just need to install the database client and to connect to the multiuser geodatabase using direct connections. In this way, the translation will happen in the client machine, leaving the server to use the processing capacity for other purposes.
  • Application server connections also create single points of failure, if the sde service fails, no one can connect to the geodatabase whereas with the direct connection this risk doesn’t exist.
  • Application server connections put much more load on the server if we compare it with direct connections.

With regards to the last point, the capacity planning tool is an excellent tool to model each connection type and to see how much server processing capacity is required. Below I have done a quick exercise that shows the amount of server processing power required to perform the same workflow using application server connections versus direct connections.

I have modelled a simple workflow, ArcGIS for Desktop 10.2 connecting to a multiuser geodatabase using direct connection [1] and application server connections [2]. Both workflows generate 50.000 transactions per hour (140 concurrent users approx. with a productivity of 6 displays per minutes as average per user (a standard user).

DC

[1] Workflow: ArcGIS Desktop connecting to a multiuser geodatabase using direct connection (DC).

ASC

[2] Workflow: ArcGIS Desktop connecting to a multiuser geodatabase using application server connection (ASC).

Notice that using direct connection the workflow requires 14,7% of the server processing capacity whereas using application service connections it requires 29,3% of the server processing capacity, almost double. This is another reason  to reconsider the use of application server connections, if you are still using them.

I hope I was able to bring some light into this topic. If you are interested in expanding your understanding about enterprise geodatabases, I highly recommend to attend the following training courses (in particular the second one):

Deploying and Maintaining a Multiuser Geodatabase

Implementing Versioned Workflows in a Multiuser Geodatabase

There are many definition of GIS, however the most widely used is the one depicted below. I like this definition due to its simplicity but also because it allows to quickly explain what are the main components of a GIS and to point out the implications that each component has in the success of a GIS deployment.

 

 

 

The image define GIS by its five principal components:  data, software, procedures, hardware and the most important one, the user, the group of people that have the knowledge and expertise to operate the system to support the organization's operations and to adapt it when changes in the business needs occur.

In the next lines, I'll focus in the human factor, and more specifically, in the importance of having a well trained GIS team supported by a well designed training plan and how this fact contributes to the success of the business.

The GIS industry, a highly dynamic environment 

As industries evolve, organizations need to evolve to adapt to industry changes, and software tools need to evolve to adapt to the organization's changing needs.   In response to changes in demand and technology, the ArcGIS platform gets new tools while existing tools evolve adding more capabilities, some decline, and others exit. In consequence, user's knowledge needs to evolve in parallel so they can keep operating the system in the most proper way, taking advantage of the new capabilities offered by the software and reducing to the minimum a possible negative impact a technology change can have in the business's operations.

A well trained GIS team is key for the success of the business

We all know that the decision to implement a GIS is not a trivial one. Generally speaking, we can say that it is not an easy enterprise and that it requires the investment of a considerable amount of time and resources, therefore it's in our best interest to reduce to the minimum the risk of a failing GIS project. Considering training as one of the strategic lines of your GIS project plan will raise exponentially the chance of success of the GIS in the organization.

Benefits of training

A while ago I came across a great blog post from Suzanne B.  from the training services team in the Esri HQ in Redlands that I'd like to share here. In her post she listed some of the most relevant benefits training has in the organization and how they contribute positively to the success of a GIS:

  • Training increases staff efficiency—it’s not uncommon to hear students who have learned a software task in an Esri instructor-led class excitedly claim they will now save hours on the job.  
  • Training increases staff productivity—a natural result of increased efficiency, productivity increases when tasks are completed more quickly and in a more confident way. 
  • Training increases staff knowledge—Knowledge is more than the sum of the topics covered in a class. Knowledge is the synthesis of different concepts and skills learned over time, which enables a person to recognize and act to prevent errors and reduce liabilities.
  • Training leads to new business opportunities— When users are well trained, they are freed to be creative and see possibilities for information products and workflows that may not have been apparent before the training. Interacting with peers in class by exchanging ideas and experiences certainly helps realize this benefit.

 

To complete the list, I would add some of the typical benefits of GIS training:

 

  • Training significantly reduced potential for mistakes (both GIS software user mistakes and GIS deployment mistakes),
  • Training allows to design enterprise architectures based on best practices.
  • Training also allows to implement better workflows and improved data QA & QC which ultimately yield a better return on your investment in GIS.

 

In the same way training is crucial for the success of a GIS, the way we implement training within the organization will determine the success of the training plan. Formal or self training There are many resources freely available that someone can use to increase his understanding on a particular topic; to mention a few: videos, blogs, participation in forums, mailing lists, etc. All these are great resources to increase our understanding about a particular topic , however, very often, they offer a partial view of the topic and many times they are just opinions that might not be correct. This is where formal training comes into play. Benefits of formal training All the resources that I mentioned above are great as a complement to a formal, well structured training session. There are many advantages of formal training compared to self learning. Since I want to make this post short and concise, I have highlighted the most relevant below: Formal training gives:

 

  • the opportunity to dedicate time just for training. Students can be completely focused on learning about a particular topic and so to get the most out of the training session. 
  • the opportunity to do it with a certified Esri trainer using official Esri materials. This gives the student the confidence of having the latest, best structured way to learn about a particular topic, but also to be supported by a trainer that has been certified by Esri and by CompTIA,  therefore a trainer that has the knowledge and skills required to deliver the course and manage the class according to the standards of Esri and the industry.
  • the opportunity to interact with other students in a classroom environment (physical or virtual). This helps students to discuss and compare different ways of approaching similar challenges.

 

The Training Plan

Ideally, A GIS implementation plan should include a training plan that reflects the training requirements of the organisation during the whole life cycle of the GIS deployment. There are several techniques to do that although they all share the same principle, it requires a close collaboration between a training consultant and the stakeholders involved in the project.

A training plan has to reflect the short term training needs of the organization segmented by user group and workflow, and the mid term training needs to ensure the organization can support their enterprise GIS progression and the professional development progression of the staff. This balance between enterprise GIS progression vs staff professional development needs contributes to the stability of the team, another  factor that has a direct impact on the success of the GIS.

Benefits of training in the stability of the team 

There are other less obvious benefits of training that worth to mention here. According to a research done by IBM, employees who do not feel they can achieve their career goals at their current organization are 12 times more likely to consider leaving than employees who do feel they can achieve their career goals. This number increases to about 30 times for new employees. Considering the efforts companies invest in the recruitment process and the time lost to filling the same role again, the impact to performance and margin can be significant.

Training and the investment in developing a "skills building culture" dramatically impacts employee retention.

Only 21% of new hires intend to stay at companies that do not offer training for their current jobs. However, the study reveals that 62% of new hires intend to stay when training is provided. This nearly three-fold increase is a powerful example of the positive impact of training on the new hire retention.

<>

To finish off this post, I leave below a quote from Henry Ford that left me thinking for a while when I first read it.....hoping it has the same effect on you.

"The only thing worse than training your employees and having them leave is not training them and having them stay." Henry Ford