Inside the Esri Technical Certification Exam Development Process
We've said before that the process to create a certification exam is rigorous and time-consuming. In fact, we get a lot of questions about how exam questions are developed. Who writes them? Who validates them? Why does it take so long?
To answer these questions, here's a high-level overview of our exam development process. Like many IT certification programs, we use a third-party consultant for test development. For each certification, we hold a series of workshops, one of which is the question development workshop.
To this workshop, we invite a group of subject matter experts (SMEs) from across Esri, psychometrician facilitators from the consultant, and facilitators from our certification team who monitor consistency. The group gets together in a room to write and review questions, collaborate on appropriate wording and scenarios, and debate each question's merits for measuring specific knowledge. For each item, a committee of 5-7 SMEs has to agree on both its technical accuracy and its congruency to the skill being measured (to make sure it tests what it's supposed to test).
After the workshop, the collection of questions undergoes a psychometric review, a copyedit, a style edit, and is then assembled into the beta exam question pool. When the beta period is over, the psychometricians analyze the results and select questions that are proven to reliably measure the skills and knowledge a qualified candidate for the certification has. When you take an exam at a testing center, you are presented with 95 or so questions from a larger pool of the final questions.
So that's the overview of how certification exam questions are created. Now here's an inside view from one of the many smart people who's participated in the exam development process. Nana Dei, geodata development technical lead with Esri Support Services, acted as a subject matter expert for the Enterprise Geodatabase Management Associate exam. In an intensive four-day workshop, Nana wrote, researched, collaborated, debated, and discovered that creating exam questions is easier said than done.
What knowledge and experience did you contribute to the exam development process?
Nana: I’ve worked on a number of databases since early 2000 including MySQL and Oracle, which led me into SQL Server and ArcSDE. After joining Esri in 2008, I started working on SDE geodatabases, which helped me build a stronger foundation and in-depth knowledge into the workings of a geodatabase. This knowledge helped me effectively communicate and assess questions and answers in the exam development process.
Describe your experience with question writing. What was it like to have your questions debated during the process?
Nana: Placing myself in the shoes of the test taker was one of the things I thought about while writing questions. A question can be easier or difficult depending on how it is phrased. When writing a question, I thought about a test taker taking the exam under time constraints and feeling pressure. How would they react when reading this question and answering it? It was great to have the other SMEs think out loud during the question writing stage. Debating the questions was fun. It was important to gather different perspectives on particular concepts. It helped me modify the question or come up with stronger answers.
What were your expectations going into the certification development process and how did they change as the workshop progressed?
Nana: Initially, I thought it would be easy to write questions. Finding the correct answer to a question was pretty easy, but coming up with incorrect answers for each question was difficult for me. I think I spent a lot of time coming up with [psychometrically acceptable] wrong answers.
Has the experience of participating in the workshop influenced how you do your work today?
Nana: It taught me to be more conscious about not having any ambiguity when explaining a concept or scenario, to be clear and not leave things open to interpretation. In the geodatabase world, there are multiple ways to reach a destination; each method may have slightly different modes of operation even though they will all lead to the same destination. It is always important, I feel, to be on the same page with the person I am discussing a situation with. If we are not on the same page, the entire knowledge transfer will not be effective.
Knowing what you know about the exam development process, what would you say to those who want to achieve certification?
Nana: Study for the exam. Leave no stone unturned. Ensure that you know and understand the concepts being tested.
What was the most valuable or rewarding part of being a contributing subject matter expert?
Nana: During the exam development process, it was rewarding to have other SMEs value my input. I really enjoyed the collaboration with different SMEs. The input by all members of the team made it feel as if we were unified in our approach to reach the goal. The fact that I went through the exam development process and demonstrated my knowledge in the technology—you have to have a level of knowledge to come up with possible scenarios, questions, and answers—was very rewarding. After the exam development process, I feel other employees see me as a go-to person for that specific technology.
What value you do you see in Esri technical certification?
Nana: I think it’s a very good indicator that one is familiar with that technology and that a person can effectively communicate in that subject area when called upon to explain a concept or give help in a specific scenario. It would look good on a resume when applying for jobs because it shows the person really cares about personal growth in that area. The technology changes over the years, and we need to keep up.
