REPORT DIGEST

CMS MULTIPLE CHOICE EXAMS

PERFORMANCE AUDIT

Release Date:  December 18, 2019

Audit performed in accordance with
House Resolution 816

State of Illinois, Office of the Auditor
General
FRANK J. MAUTINO, AUDITOR GENERAL

To obtain a copy of the Report contact:
Office of the Auditor General, Iles Park
Plaza, 740 E. Ash Street, Springfield, IL
62703
(217) 782-6046 or TTY (888) 261-2887

This Report Digest and Full Report are also
available on the worldwide web at
www.auditor.illinois.gov

On May 10, 2018, the Illinois House of
Representatives adopted House Resolution
Number 816 which directed the Auditor
General to conduct a performance audit of
the Department of Central Management
Services (CMS) to review and assess the
Department’s automated multiple choice exams
for specific position titles listed in the
audit resolution.  All of the 75 position
titles listed in the audit resolution fall
under automated testing, Group A titles, or
titles that are continuously tested.  The
resolution specified that the audit include
a review of whether those automated multiple
choice exams are biased against minority
exam takers, either in their content or in
their results.

Key findings of the audit include the
following:

• Only a small percentage of applicants who
apply are hired.  During calendar years 2015
through 2018, 226,229 applicants took exams.
Of these exams, only 2,585 (1.1%) resulted
in an applicant being hired (during the same
time period).

• To determine if the automated multiple
choice exams are biased against minority
exam takers, we examined the process CMS
uses to establish and monitor the exams.
Bias as it relates to exams and testing is
difficult to define and quantify.  CMS
generally follows the Uniform Guidelines on
Employee Selection Procedures (29 CFR 1607),
which are designed to prevent discriminatory
practices, in conducting content validity
studies on exams.  CMS also conducts
thorough statistical analyses to ensure an
exam is testing the items it is intended to
test.  The statistical analyses include
reliability testing, correlation analysis,
and discrimination index analysis to
identify potentially ineffective questions.

• Despite these efforts to ensure tests are
fair and afford all applicants an equal
opportunity to compete for State jobs, there
is a disparity in test grades among
different races and ethnic groups for
reasons unknown.

• Compared to their ratio of applications
and passing grades, White applicants had a
high ratio of A grades while Black or
African American applicants had a low ratio
of A grades.  State agencies are required to
hire from the group of candidates that
received the highest passing grade.  If a
certain ethnic group is receiving less A
grades than other ethnic groups, that ethnic
group will not be represented as well in the
hiring pool.

• CMS officials could not easily identify
which position titles had received a content
validity study and could not provide the
content validity study for one of ten titles
tested.  Content validity studies are one of
the three types of validity studies outlined
in the Uniform Guidelines which employers
can use to validate employment tests.

• The CMS Test Development Section does not
have written policies or procedures for
developing exams or for conducting validity
studies and statistical analyses of these
exams.

The audit contains a total of four
recommendations to CMS.

AUDIT SUMMARY AND RESULTS

On May 10, 2018, the Illinois House of
Representatives adopted House Resolution
Number 816 which directed the Auditor
General to conduct a performance audit of
the Department of Central Management
Services (CMS) to review and assess the
Department’s automated multiple choice exams
for specific position titles listed in the
audit resolution.  All of the 75 position
titles listed in the audit resolution fall
under automated testing, Group A titles.
Group A titles are continuously tested
titles where anyone can walk in anytime to
be tested.  The resolution specified that
the audit include a review of whether those
automated multiple choice exams are biased
against minority exam takers, either in
their content or in their results. (See
Appendix A.)  Only a small percentage of
applicants who apply are hired.  During
calendar years 2015 through 2018, 226,229
applicants took exams.  Of the 226,229
examinations taken, only 2,585 (1.1%) of
these resulted in an applicant being hired
during calendar years 2015 through 2018.

To determine if the automated multiple
choice exams are biased against minority
exam takers, we examined the process CMS
uses to establish and monitor the exams.
Bias as it relates to exams and testing is
difficult to define and quantify.  CMS
generally follows the Uniform Guidelines on
Employee Selection Procedures (29 CFR 1607),
which are designed to prevent discriminatory
practices, in conducting content validity
studies on exams.  CMS also conducts
thorough statistical analyses to ensure an
exam is testing the items it is intended to
test.  The statistical analyses include
reliability testing, correlation analysis,
and discrimination index analysis to
identify potentially ineffective questions.
Despite these efforts to ensure tests are
fair and afford all applicants an equal
opportunity to compete for State jobs, there
is a disparity in test grades among
different races and ethnic groups for
reasons unknown.  According to a CMS
official, this disparity is not new and,
many years ago, Test Development staff
attempted to find the reason for the
disparity in test grades among different
race/ethnic groups.  Test Development staff
found that the groups that attained lower
overall test grades typically had
significantly less formal education.  The
official added that this finding, coupled
with socioeconomic factors and less
opportunity, may contribute to the
disparity.  (pages 14-21, 30-33)

Auditors identified the following issues:

• CMS officials could not easily identify
which position titles had received a content
validity study.  Content validity studies
are one of the three types of validity
studies outlined in the Uniform Guidelines
which employers can use to validate
employment tests.

• The validity studies conducted by CMS
generally followed the Uniform Guidelines.
However, CMS could not provide the content
validity study for one of ten titles tested.
CMS officials could not provide the exact
date the original exam was implemented, but
said it was implemented at some point prior
to 1989.  (pages 16-18)

• The CMS Test Development Section does not
have written policies or procedures for
developing exams or for conducting validity
studies and statistical analyses of these
exams. (pages 21-22)

• Six post-exam survey responses (out of
6,300 survey responses over four calendar
years) that are applicable to exam content
were not provided to the Test Development
Section.  (pages 23-24)

• Sixty-four applicants consisting of 127
entries (out of 226,229 examinations over
four calendar years) were allowed to retake
exams within 30 days, which is a violation
of the Illinois Personnel Rules. (pages
24-25)

Applicant Demographics

CMS provided data on applicants that sat for
exams during calendar years 2015 through
2018 for the 75 position titles in the audit
resolution.  Over the four calendar years,
the proportion of applicants in each
ethnicity category remained approximately
the same.  The overall pass rate ranged from
a high of 84.0 percent in 2015 to a low of
81.8 percent in 2018, with an average of
82.9 percent.  The pass rate for White test
takers was consistently above the overall
pass rate, with an average of 87.6 percent,
while the remaining ethnicity categories
were generally below the overall pass rate.
However, for 2018, the pass rates for both
Asian applicants (83.1%) and American Indian
or Alaska Native applicants (84.5%) also
rose above the overall pass rate.  The 2018
pass rates ranged from 86.4 percent (White
applicants) to 76.0 percent (Black or
African American applicants).  (Please see
pdf version of this digest for exhibit.)
White applicants had a high ratio of A
grades compared to their ratio of
applications and passing grades.
Conversely, Black or African American
applicants had a low percentage of A grades
compared to their ratio of applications and
passing grades.  State agencies are required
to hire from the group of candidates that
received the highest passing grade.
Therefore, if a certain ethnic group is
receiving less A grades than other ethnic
groups, that ethnic group will not be
represented as well in the hiring pool.
(pages 28-33) (Please see pdf version of
this digest for exhibit.)

RECOMMENDATIONS

The audit report contains four
recommendations to the Department of Central
Management Services.  The Department agreed
with the recommendations.  Appendix E to the
audit report contains the agency responses.

This performance audit was conducted by
staff of the Office of the Auditor General.

JOE BUTCHER
Division Director

This report is transmitted in accordance
with Section 3-14 and 3-15 of the Illinois
State Auditing Act.

FRANK J. MAUTINO
Auditor General

FJM:TEW