THE STATE EDUCATION DEPARTMENT / THE UNIVERSITY OF THE STATE OF NEW YORK / ALBANY, NY 12234

 

TO:

The Honorable the Members of the Board of Regents

FROM:

James A. Kadamus

COMMITTEE:

Full Board

TITLE OF ITEM:

Regents Test Development Process

DATE OF SUBMISSION:

May 12, 2004

PROPOSED HANDLING:

Information

RATIONALE FOR ITEM:

Implementation of Regents Policy

STRATEGIC GOAL:

Goals 1 and 2

AUTHORIZATION(S):

 

 

SUMMARY:

 

Attached is an update for the Board on the implementation of the new procedures in the Regents test development process relating to use of literary and non-fiction passages on exams.  The report emphasizes changes that have been made and recent actions taken to ensure that passages are not eliminated for consideration because they address controversial topics.

 

 

 

Attachment

Update on Implementation of New Procedures in the Regents Test Development Process

 

Since June 2002 a number of changes have been made in the test development process to ensure that we maintain the quality of the tests, while developing tests in accordance with the guidelines issued by Commissioner Mills in June 2002: 

 

…The tests now being developed will use literary passages without changing the author’s words. Literature will continue to be excerpted for length because entire works cannot appear on a single test.  All passages will cite the author and title of the work….

 

·        Words or phrases will not be deleted or changed in passages.

·        Complete paragraphs or sections of a piece of literature will be used.

·        Deletions of paragraphs for length will be indicated.

·        The author and title will always be cited.

·        The former sensitivity guidelines will not be used. Instead, the criteria will be the inclusion of high quality, meaningful literature, fairness, accessibility, and common sense.

·        Management accountability has been changed to ensure these directives are carried out….

 

In response to these directives, the Office of State Assessment stopped using the extensive sensitivity review guidelines that had been in place. Instead, passages and test questions are now reviewed in accordance with the Commissioner’s guidelines and Section 7.4 of the Standards for Educational and Psychological Testing, described in detail below, which were distributed to all appropriate staff. In addition to the long-standing practice of having committees of teachers participate in the development of tests at many stages, a review by senior Department managers of all exams containing passage- or document-based questions was instituted beginning with the August 2002 examinations. Beginning with the August 2003 examinations, committees of teachers not previously involved in test development review all tests in what is called the “final eyes review.”  A team of Office of State Assessment managers reviews the recommendations made by these final eyes committees and appropriate adjustments, if any, are made to test questions.

 

During a recent periodic review of the various aspects of the test development process, senior staff found that some staff removed, during early stages of the process, 18 literary and nonfiction passages (out of a potential 600 passages) from consideration for possible use in English Regents Exams at least in part because they were viewed as controversial. This was discovered during a review of material requested through a freedom of information request. Although the staff involved believed they were acting in test-takers’ best interests, this did not follow the new guidelines issued in June 2002.

 

The elimination of controversial passages from further consideration resulted from a misunderstanding of the Standards for Educational and Psychological Testing, published by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. The Department is guided by the Standards because they constitute the definitive guidelines for test development nationally. The Standards define fairness in chapter 7, entitled “Fairness in Testing and Test Use.” Standard 7.4 says, in part, “Test developers should strive to identify and eliminate language, symbols, words, phrases, and content that are generally regarded as offensive by members of racial, ethnic, gender, or other groups, except when judged to be necessary for adequate representation of the domain.” (emphasis added)

 

The domain means, essentially, what is being tested. The Standards do not intend the elimination of potential passages just because they include controversial themes. The Standards also says that such material may be necessary and cites the example of social studies tests, which should include controversial material. “For example,” Standard 7.4 says, “history tests may appropriately include material on slavery or Nazis.” Using common sense as a guide, it is clear that the English Regents Exam would not include a passage of Nazi propaganda that advocated the extermination of people the Nazis considered inferior. However, on the other hand, quality literature would not be excluded from English tests just because it deals with controversial topics.

 

The English Regents Exams being produced have not been compromised by the removal of the 18 passages (cited above) from consideration. Passages that are used in the exams have not been edited or words changed for reasons of sensitivity. Passages on the exams have dealt with such potentially sensitive or controversial topics as poverty, disabilities, race and death.

 

The passage selection guidelines and passage review process have recently been reviewed with staff and their proper use has been reinforced.  Staff also instituted changes in the passage selection process to make sure that passages are not eliminated from consideration unless:

 

1)     They have been reviewed by senior managers, or

2)     Data from field testing indicates that the passages or test items based on the passages are unfair to certain groups of students.

 

As the Commissioner wrote in June 2002, the overriding criteria for the selection of passages must be the inclusion of high quality, meaningful literature, fairness, accessibility, and common sense.