Meeting of the Board of Regents | September 2003

SUMMARY:

The attached report describes administrative changes to improve the assessment system. Some of the changes were in place for the August 2003 Regents exams. Others will be phased in for January and June 2004 exams. The aims are to: (1) ensure consistency and fairness; (2) increase our capacity to develop and administer exams; and (3) better inform educators and the public about the exams.

Attachment

Status Report on Operational Changes Affecting the New York State Assessment System

Actions to Ensure High Quality of the Exams

• Beginning with the August 2003 Regents exam administration, for each exam a panel of three-five New York State teachers who were not involved in the preparation of the exams conducted a "whole test" quality control check prior to the test administration. They clarified instructions, corrected diagrams, and eliminated other errors. There were no post-exam errata notices issued for the August 2003 exams. The "whole test" quality control check will be done from now on. We will also continue the practice of having staff from outside the Office of State Assessment review literary passages and document-based questions for accuracy and fidelity to editing rules established by the Commissioner.

• We will create a score validation process for the January 2004 Math A exam, and possibly for one other exam. Operational data (student results) will be collected from a sample of districts immediately after the administration of the exam in order to check on how students performed on the test, thus giving us a virtual field test. The exams slated for this validation process will be scheduled as near as possible to the beginning of the examination period.

• Beginning with the June 2004 field-test period, we will revise our field-test procedures for Regents examinations. We will announce the schedule and process for implementing these changes by November 2003 after consultation with the field. We will use embedded field tests and/or mini-books, similar to the procedure used for elementary/intermediate assessments. This new approach will place field testing much closer in time to actual operational testing in either January or June, thus providing us with better information to predict how students will perform on actual operational tests. This approach will also have the benefit of reducing field-testing burden on schools and result in a much larger pool of quality test items.

Actions to Increase Capacity to Develop and Administer Assessments

We will take the following actions:

• Issue Requests for Proposals (RFPs) to contract for the preparation of some exams required by NCLB. An RFP for the grade-by-grade reading and math tests (grades 3-8) will be issued in September 2003. The RFP is based on advice provided by a practitioner's group that included representatives of school administrators, teachers, parents, and school board members. Use of RFPs to acquire the services of additional contractors will help increase the Department's capacity to accelerate the timeline for preparation of tests and do this in advance of the required implementation dates (2005-06).

• Develop an agreement with an institution of higher education to provide additional subject area, test development and psychometric experts to strengthen our test development process. We will have these new resources in place by early 2004.

• Reassign six additional staff to the Office of State Assessment to carry out research and administrative activities. In addition, we have appointed Howard Goldsmith as Coordinator of Assessment Operations. Howard will assume this role on September 22 and will oversee the day-to-day work of the Office of State Assessment, under the direction of Gerald DeMauro, Director of State Assessment. These new assignments will free time for the Director of State Assessment to address testing and policy issues and will improve the overall efficiency of the assessment process.

• Use computer software, beginning fall 2003, to track the development of every exam. Then we will be able to intervene if the process is not moving according to schedule or if there is any problem with the quality of the exam.

Actions to Improve Public Information on Assessment

• We restructured administrative and communications activities within the Office of State Assessment. Staff has been re-assigned to handle contracts and budget activities. We expect the volume of contract work to increase as we use consultants and contractors to carry out much of the additional work. In addition, staff have been re-assigned to the office to handle telephone and e-mail inquiries. This will free assessment staff to focus on exam development and test administration activities. It will also improve turnaround time for all inquiries.

• We are in the process of changing our methods of providing information to the field and the general public. We will use our web site to provide clear information about assessments to more people much faster than we have before. We also hope that this will reduce the number of telephone calls that staff has to respond to. The emphasis will be on user-friendly information that stresses how the tests are developed and used and how parents can help their children achieve the State learning standards.