Meeting of the Board of Regents | December 2009

SUMMARY

Issue for Discussion

The Office of the Comptroller conducted an oversight audit of scoring practices on Regents Examinations. The Audit has been completed and the final report was issued on November 20.

Reason for Consideration

For Information

Primary Findings from the Audit Report

The State Comptroller found the following:

• SED reviews of the scoring of selective Regents Examinations have identified inaccuracies by local school districts. These inaccuracies have tended to inflate the academic performance of students and schools. The Comptroller said, while SED has detected this problem, its oversight has not been adequate to ensure that local school districts will correct the problem so that future exams are more accurately scored.
• The Comptroller further found that when local school districts fail to comply with SED requests to submit scored exams for further review, SED was not following up to obtain these examinations. The Comptroller recommended follow up take place because there is considerable risk that the failure to submit scored exams may be a willful attempt to avoid scrutiny of scoring accuracy.
• The Comptroller also concluded that SED had limited assurance that exam raters actually attended annual training for scoring exams. They recommended that SED strengthen its formal guidance for administering the exams to better ensure that raters attend such training.

SED Actions in Response to the Audit:

When the audit was released, Commissioner Steiner issued the following statement, “The integrity of our testing system is of the highest importance to our students, their families, every teacher, and all of us focused on education in our state. The Regents and I are committed to revising New York's assessment system so that our tests more fairly and accurately measure student achievement. We are undertaking a thorough review of all aspects of our accountability system and are taking action appropriately. We are implementing the recommendations in the OSC audit and have moved to put additional improvements in place for the oversight of the scoring process for Regents examinations.”

The Comptroller’s office recommended 11 actions to improve local scoring. We are implementing them as well as other actions to improve test scoring. These include:

• Conducting spike/cluster analysis of Regents Examination scores to identify schools with potentially questionable results to be conducted annually. Schools with high percentages of students receiving exam scores at, or just above, 65 will be required to submit both scoring and oversight plans to the Department and a District Superintendent of schools as designated by the Commissioner. If the potentially questionable results continue, schools will be required to have their exams scored at an alternate site approved by the Department and the District Superintendent;
• Conducting monitoring visits by NYSED, BOCES administrative, and large-city school district administrative staff to approximately 40 schools during the administration of Regents Examinations;
• Establishing and maintaining a computerized database for recording and follow up for all allegations of improper conduct during the administration and scoring of all State examinations, including Regents Examinations;
• Taking action in confirmed and/or suspicious cases including requiring schools to conduct the scoring of Regents Examinations at an alternate site such as a BOCES and/or prohibiting specific individuals from participation in the administration and/or scoring of Regents Examinations;
• Conducting Regents Examination Item-Level Score Collections at both the representative sampling and full test taking census levels which will include erasure and response tampering analysis (this will require a contract with an external vendor);
• Conducting erasure and response tampering analysis of Regents Examination answer papers at the full test taking census, representative sampling, and selected schools levels;
• Directing schools to keep formal documentation of teacher attendance at Examination Training Sessions and revising the Department’s formal Examination certification to include an affirmation that the rater attended training for the exam.
• Requiring Districts to use a standardized answer document that can be scanned.
• Using combinations of the processes listed above to effectively identify concerns with local scoring of Regents Examinations and/or to prevent reoccurrences of improper conduct.

Background Information

• New York State is unique among the 50 states in having the longest standing history of administering end of course high school exams that are scored locally and are tied to diploma conferral. 
• Local scoring of Regents Examinations poses monitoring challenges for the Department. The Department administers over 2 million Regents Exams annually in 17 different titles in approximately 1, 600 school buildings.
• OSC last conducted a scoring audit of Regents Examinations in 1991.
• The Office of State Assessment currently has four professional staff in the Test Administration Unit who focus on the oversight of the local scoring for Regents Examinations. Each staff member is able to expend approximately 10% of their annual work time on this area of responsibility due to other critical functions associated with test administration.

Next Steps: Policy Consideration

The local scoring of Regents Examinations will always be a subject of concern. The Board of Regents will need to determine if they want to continue local scoring or research alternatives which require using an external, independent vendor who has the capacity to ensure a timely turnaround.