The Science Behind the ArcGIS Desktop Entry Certification
Updated November 8, 2021
The ArcGIS Desktop Entry certification is one of our most popular exams. The Certification team frequently fields questions about the entry-level certification—for example, what types of ArcGIS Desktop skills does the exam measure, how is it scored, and who is this certification designed for?
To shed some light on the ArcGIS Desktop Entry certification (and certification exams in general), we talked with one of the people closely involved in the ArcGIS Desktop Entry exam development process. Lisa S. O’Leary, PhD, is a psychometrician with Alpine Testing Solutions who works closely with the Certification team on exam development, security, and maintenance. Developing the entry-level exam involved many people and many levels of review and analysis. It takes a lot of hard work to ensure we create a valid and reliable measure of a certification candidate’s skills.
We caught up with O’Leary just as she was about to leave her home base of Chicago to get married. Her credentials are impressive: she holds a PhD in Educational Research, Measurement, and Evaluation from Boston College. She worked on higher education assessment and evaluation at MIT, and has consulted for several well known IT certification programs, including those of Microsoft, CompTIA, and Cisco.
Based on your experience in the industry, what value does certification offer to professionals?
O’Leary: “It depends on the value associated with a particular certification. The big three benefits are job advancement (and some certifications are a qualification for a particular job), industry recognition, and peer recognition.
Lisa S. O'Leary
“Peer recognition—particularly in the IT certification world—is a main benefit to candidates. There are also some programs where certification is used for continuing education. So the benefit there is that certification can fulfill other job-related requirements.”
How common are entry-level certification exams?
O’Leary: “Entry-level exams, we sometimes call them gateway exams, are common in IT. They are a gateway for people who want to go up the exam level chain. What varies now in the IT industry is the approach to the entry level.
“Some have a foundation exam that cuts across all domains, and there are other organizations that have tracks (an entry-level exam for each domain).”
- Note: The ArcGIS Desktop Entry-level certification is part of the ArcGIS Desktop track, which includes Associate- and Professional-level exams.
Are there special considerations when developing an entry-level exam?
O’Leary: “With any level of certification, we always need to keep in mind the domain of interest and the person of interest within that domain. I’m reviewing to make sure we’re developing items at the appropriate level of cognitive complexity from Bloom’s Taxonomy for the way the domain and the entry-level candidate have been defined.
“We’re expecting entry-level candidates to be able to recall and perhaps understand and apply knowledge, but not necessarily analyze, evaluate, or create information.”
O’Leary: “My role is to be a sounding board and a technical resource to tap into for any structure or planning that needs to go into the exam development process we’re going to employ and any measurement issues we run into.
"Basically, I’m a go-to for the science and technology behind the development of a particular exam. It’s my role to make sure—in every piece of the process, for every deliverable—that we’re always collecting evidence of exam validity.”
How do you collect evidence of an exam’s validity?
O’Leary: “I review to make sure we’re constantly keeping in mind the purpose and intended use of the test scores, which is the crux of validity. For example, we conduct in-depth statistical analysis using classical test theory and item response theory (Rasch analysis) to gauge the exam performance against its intended purpose and defined content domain.
“There are two levels of analysis that we’re constantly working on: form-level and item-level.”
- Note: Form-level refers to the exam as a whole. A certification has multiple forms in use at all times. Item-level refers to each question.
Describe some of the item-level analysis you perform.
O’Leary: “For item-level analysis, we’re looking into item difficulty through the p-value (the proportion of candidates who correctly answer the item) and other measurement indicators.
“We’re also looking at the correlation between a particular item and total score on the exam. A well performing item is going to have a high positive correlation (if candidates answer the item correctly, they’re likely to do well on the exam). Additionally, we’re looking to see if there are there items that negatively impact a candidate’s positive performance. If so, those items are flagged for review because it’s indicative that there’s likely something wrong with them.”
Is the analysis based on data collected from the beta exam?
O’Leary: “Yes, beta exams provide a lot of useful information. We analyze candidate performance on every item as well as overall performance on the beta forms to determine viable items for the operational (released) exam.
“The time limit for a released exam like the ArcGIS Desktop Entry has taken into account the time every beta tester spent on each item. The time limit ensures that 95% of all candidates would be able to complete the operational exam within that time, looking globally across the entire spectrum of candidates that have taken the exam.
“Also, beta exam takers have the option to comment on every single item. Their comments help with decision making on individual items.”
Describe some of the form-level analysis you perform.
O’Leary: “We look at the reliability of each form: so that if a person were to take Form A, then retake Form A, what is the likelihood of receiving the same score, within measurement error? We’re looking for overall form-level reliability to be 85% or higher, with 1% or less variance between each form.
- Note: To support an exam’s validity argument, the forms must be demonstrably equivalent. In other words, it does not matter which form a candidate receives as each measures the same knowledge, skills, and abilities in the same proportion, at the same level of difficulty, in the same amount of time, with the same reliability.
“We look at the mean performance of the forms. What is the average score that candidates are achieving on each form? We want those mean averages to be within 0.2 decimal places of each other for us to consider the forms equivalent.
“We’re looking at the average time for candidates to take each of the forms. Our standard parameter for median time is the forms should be within 1 minute of each other. We analyze the pass rates for all possible scores (0-95), and we analyze candidate time versus exam score.
“We’ve actually taken form-level analysis a step further. For Esri certification exams, all analyses typically conducted at the form level are now being done at the subdomain level.”
- Note: “Subdomain” refers to one of the categories shown on the Skills Measured tab of the exam’s web page.
Interesting, please elaborate on that.
O’Leary: “Every section on the ArcGIS Desktop Entry exam is balanced, and that allows us to give really great diagnostic feedback to candidates. We can provide failing candidates with empirical evidence of areas in which they performed better and worse. On their score reports, failing candidates will get overall pass-fail and also pass-fail equivalents for each section.”
This is useful information for candidates planning an exam retake. Do all certification programs provide section-level reporting?
O’Leary: “In IT certification, requests for section-level reporting are very common. It’s less common for certification programs to actually conduct the due diligence and the exam development in a way that supports that level of reporting.
“Esri has really moved forward strongly with it. The way the exam development process is being employed absolutely supports the level of information that we’re now providing.”
Thanks, Lisa.
- Interested in the ArcGIS Desktop Entry certification? View the detailed qualifications, skills measured, and preparation resources.
- To learn about other Esri technical certifications, visit esri.com/certification.