Meeting of the Board of Regents | September 2003
|
THE STATE EDUCATION DEPARTMENT / THE UNIVERSITY OF THE STATE OF NEW YORK / ALBANY, NY 12234 |
TO: |
The Honorable the Members of the Board of Regents |
FROM: |
James A. Kadamus |
COMMITTEE: |
Full Board |
TITLE OF ITEM: |
Status Report on Operational Changes Affecting the New York State Assessment System |
DATE OF SUBMISSION: |
September 2, 2003 |
PROPOSED HANDLING: |
Discussion |
RATIONALE FOR ITEM: |
Update on key initiative |
STRATEGIC GOAL: |
Goals 1 and 2 |
AUTHORIZATION(S): |
SUMMARY:
The attached report describes administrative changes to improve the assessment system. Some of the changes were in place for the August 2003 Regents exams. Others will be phased in for January and June 2004 exams. The aims are to: (1) ensure consistency and fairness; (2) increase our capacity to develop and administer exams; and (3) better inform educators and the public about the exams.
Attachment
Status Report on Operational Changes Affecting the New York State Assessment System
Actions to Ensure High Quality of the Exams
- Beginning with the August 2003 Regents exam administration, for each exam a panel of three-five New York State teachers who were not involved in the preparation of the exams conducted a "whole test" quality control check prior to the test administration. They clarified instructions, corrected diagrams, and eliminated other errors. There were no post-exam errata notices issued for the August 2003 exams. The "whole test" quality control check will be done from now on. We will also continue the practice of having staff from outside the Office of State Assessment review literary passages and document-based questions for accuracy and fidelity to editing rules established by the Commissioner.
- We will create a score validation process for the January 2004 Math A exam, and possibly for one other exam. Operational data (student results) will be collected from a sample of districts immediately after the administration of the exam in order to check on how students performed on the test, thus giving us a virtual field test. The exams slated for this validation process will be scheduled as near as possible to the beginning of the examination period.
In the case of Math A, draft score conversion charts will be sent to schools in the usual manner, with exam materials. These charts tell schools how many questions a student has to answer correctly (raw score) on a Regents exam to receive a scale score on the test from 0-100. These conversions will either be confirmed or changed based on the analysis of actual operational test data. For other assessments, the schools would receive final conversion charts with their exam materials, with changes occurring only if the analysis indicated that a change was needed. For Math A, independent advisors will work with the Department to review the data before we adopt the final conversion charts. The independent advisors will produce a public report to validate any adjustment.
This process is similar to that used in other states and by major testing companies. Depending on the results of the pilot in January 2004, this may become a standard part of the exam administration process, beginning with exams administered in June 2004. The independent validation will enable us to know how well the test worked and make any adjustments within one week of the actual operational test administration.
- Beginning with the June 2004 field-test period, we will revise our field-test procedures for Regents examinations. We will announce the schedule and process for implementing these changes by November 2003 after consultation with the field. We will use embedded field tests and/or mini-books, similar to the procedure used for elementary/intermediate assessments. This new approach will place field testing much closer in time to actual operational testing in either January or June, thus providing us with better information to predict how students will perform on actual operational tests. This approach will also have the benefit of reducing field-testing burden on schools and result in a much larger pool of quality test items.
Actions to Increase Capacity to Develop and Administer Assessments
We will take the following actions:
- Issue Requests for Proposals (RFPs) to contract for the preparation of some exams required by NCLB. An RFP for the grade-by-grade reading and math tests (grades 3-8) will be issued in September 2003. The RFP is based on advice provided by a practitioner's group that included representatives of school administrators, teachers, parents, and school board members. Use of RFPs to acquire the services of additional contractors will help increase the Department's capacity to accelerate the timeline for preparation of tests and do this in advance of the required implementation dates (2005-06).
- Develop an agreement with an institution of higher education to provide additional subject area, test development and psychometric experts to strengthen our test development process. We will have these new resources in place by early 2004.
- Reassign six additional staff to the Office of State Assessment to carry out research and administrative activities. In addition, we have appointed Howard Goldsmith as Coordinator of Assessment Operations. Howard will assume this role on September 22 and will oversee the day-to-day work of the Office of State Assessment, under the direction of Gerald DeMauro, Director of State Assessment. These new assignments will free time for the Director of State Assessment to address testing and policy issues and will improve the overall efficiency of the assessment process.
- Use computer software, beginning fall 2003, to track the development of every exam. Then we will be able to intervene if the process is not moving according to schedule or if there is any problem with the quality of the exam.
Actions to Improve Public Information on Assessment
- We restructured administrative and communications activities within the Office of State Assessment. Staff has been re-assigned to handle contracts and budget activities. We expect the volume of contract work to increase as we use consultants and contractors to carry out much of the additional work. In addition, staff have been re-assigned to the office to handle telephone and e-mail inquiries. This will free assessment staff to focus on exam development and test administration activities. It will also improve turnaround time for all inquiries.
- We are in the process of changing our methods of providing information to the field and the general public. We will use our web site to provide clear information about assessments to more people much faster than we have before. We also hope that this will reduce the number of telephone calls that staff has to respond to. The emphasis will be on user-friendly information that stresses how the tests are developed and used and how parents can help their children achieve the State learning standards.