SUMMARY

Issue for Decision

In response to the audit of Regents examinations scoring practices conducted by the Office of the State Comptroller, does the EMSC Committee approve the two-step recommendation for scanning Regents examination answer sheets? 

Background Information

The Office of the Comptroller conducted an oversight audit of scoring practices on Regents Examinations. The final Audit report was issued on November 20, 2009.  In December 2009, the Regents Subcommittee on Audits discussed the Audit findings and recommendations and future actions to be taken by the Department to respond to the audit recommendations.   In April 2010, the Regents Subcommittee on Audits discussed the following proposal to scan Regents Exams to aid the Department in their monitoring of scoring processes. They supported the plan and recommended that it be forwarded to EMSC for further review and action.

Points to Consider:

• New York State is unique among the 50 states in having the longest standing history of administering end of course high school exams that are scored locally and are tied to diploma conferral. 
• Local scoring of Regents Examinations poses monitoring challenges for the Department. The Department administers over 2 million Regents Exams annually in 17 different titles in approximately 1, 600 school buildings.
• OSC last conducted a scoring audit of Regents Examinations in 1991.

• The Office of State Assessment currently has four professional staff in the Test Administration Unit who focus on the oversight of the local scoring for Regents Examinations. Each staff member is able to expend approximately 10% of their annual work time on this area of responsibility due to other critical functions associated with test administration.

Update:  Scanning Regents Exams:

The Department responded to the Office of the State Comptroller with a series of actions that we will undertake. One set of significant actions requires conducting Regents Examination Item Level Score Collections at both the representative sampling and full test taking census levels which will include erasure and response tampering analysis (this will require a contract with an external vendor) and conducting annual spike/cluster analysis of scores to identify schools with suspicious results.

              Regents Exams are scored immediately after their administration; schools use a variety of means to score and record the data. The Department has an interest in ensuring that Regents exams are scored accurately since these scores are used on local school report cards, student transcripts, and for accountability reporting.  There is no standardized process in place that allows the State to collect item level and test-taker demographic data that it needs for: USED Title I Peer Review; USED Race To the Top; IDEA & Title III Accommodation Analyses; Office of State Comptroller Test Scoring Monitoring (data forensics); and Technical Manual Development. Without scanning these answer sheets for the information that that they can provide, the Department does not have a mechanism to provide high quality data analysis for its own needs and for providing school systems with actionable data that can used to inform their instructional programs.

              Department staff has discussed requiring statewide scanning of Regents Exam Answer sheets, starting in June 2011, with District Superintendents, Regional Information Center (RIC) Directors, the NYSCOSS Curriculum & Assessment Committee, Big 5 School Districts, NYSSBA, and NYSUT. Currently, there is some scanning of Regents exams around the State, but it is not being done at scale.  We have a combination of RICs providing the service and a limited number of school districts that have purchased equipment, ranging from high-capacity scanners to single-sheet desktop scanners, to do the work.  Department staff has met with RIC Directors to determine answer sheet development and scanning costs (approximately $1.80-2.25 per student, per test), answer sheet design and production, scanning timelines, and reporting. Precise cost implications for districts would vary depending on the number of students tested per year per test and current scanning practices (e.g., some districts pay for scanning now and some have the equipment to do it).   

Recommendation

              The EMSC Committee approve the recommendation below:

              The Department realizes that there will be costs associated with scanning Regents Answer sheets. To that end, we propose the following two step approach:

1. Starting with the June Regents Administration in the 2010-2011 school year, require that all Regents Exams in the following titles be scanned for Department analysis: Comprehensive English, Integrated Algebra, Global Studies and Geography, U.S. History and Government, Living Environment, and Earth Science. These exams are all used for the awarding of a Regents Diploma.
2. Commencing with the 2011-2012 school year (June Regents Administration), require that all remaining Regents Exams be scanned.