THE STATE EDUCATION DEPARTMENT / THE UNIVERSITY OF THE STATE OF NEW YORK / ALBANY, NY 12234

TO:

The Honorable the Members of the Board of Regents

FROM:

James A. Kadamus

COMMITTEE:

EMSC-VESID

TITLE OF ITEM:

Status Report on Operational Changes Affecting the New York State Assessment System

DATE OF SUBMISSION:

August 23, 2004

PROPOSED HANDLING:

Discussion

RATIONALE FOR ITEM:

Monitoring Implementation of Regents Policy

STRATEGIC GOAL:

Goals 1 and 2

AUTHORIZATION(S):

EMSC Commitment To The Board of Regents*

Description of Action and Current Status

Actions to Ensure High Quality Exams

Actions to Increase Capacity to Develop and Administer Exams

Actions to Improve Public Communication

Implemented Final Eyes Review:  Teachers from the field are invited to take and then evaluate upcoming Regents exams.  The process exists for each testing cycle.  (Status: implemented and continuing)

Implemented Senior Management Review:  Kathy Ahearn, Alan Ray and David Abrams review all literary passages and multiple-choice questions from the Comprehensive English Regents Exam and all documents and other previously published materials from the Global History and Geography and U.S. History and Government Regents exams for each testing cycle to determine accuracy, sourcing, and content alignment to State standards.  (Status: implemented and continuing)

Formed State Assessment Cabinet:  Comprised of management representatives from the Office of Elementary, Middle, Secondary and Continuing Education (EMSC), the Office of Vocational and Educational Services for Individuals with Disabilities (VESID), the Office of Management Services (OMS), the Office of Information Technology Services (ITS), the Communications Office, and the Office of Counsel. The Cabinet meets regularly to discuss and resolve major testing issues, fostering efficient inter-departmental communication. (Status: implemented and continuing)

Increased Staffing in the Office of State Assessment (OSA):  Five content specialists, one editor, and four support staff have been added, and six additional waivers have been received.  (Status:  implemented and continuing)

Revised Structure for Technical Assistance Group (TAG): The TAG is comprised of nationally known psychometricians and educational psychologists who consult with SED on issues pertaining to the design, psychometrics, and certification of the New York State Testing Program. To bring stronger structure and purpose to the group, OSA managers crafted new bylaws detailing member responsibilities and obligations.  At the last TAG meeting (May 2004), OSA managers discussed a more active working group with multiple subgroups to aid on specific projects. Also discussed were suggestions for new members, as the TAG will expand to 13 members. TAG meetings will now be used as a form of professional development for members of both OSA and the Office of Curriculum, Instruction and Instructional Technology. (Status: implemented and continuing)

Designed and Implemented Exam Quality Indicators Review:  The Exam Quality Indicators is a set of activities that are designed to help govern, organize and certify the test development process at key checkpoints. The indicators were used for the first time for the June 2004 testing cycle. The indicators are currently under review by the State Assessment Cabinet, the TAG, and other members of OSA for continual analysis and revision. (Status: implemented and continuing)

Implementing Standard-Setting Revisitation Methodology: OSA managers are working with psychometric consultants to develop a new Standard-Setting Revisitation process to verify that exams remain relevant and that there is no shift in the psychometric properties of the exam. After each June testing cycle, at the direction of the Commissioner and the Deputy Commissioner, OSA managers will evaluate selected exams. All Regents exams will undergo this process over time. (Status: in development)

Updated Psychometric Validation Procedures:  To ensure validity and reliability, all psychometric work is checked by multiple groups.  External contractors are required to submit their work to OSA staff researchers for internal review and to a second external contractor for replication and methodology certification.  Certain products (e.g., Regents Examination Score Conversion Charts, Vertical Scaling Methodology) are submitted to the TAG for its certification. (Status: implemented and continuing)

Examination Development Process:  The Assistant Commissioner and OSA managers will work on a comparative analysis with psychometricians from Pearson Educational Measurement to evaluate the State’s examination development process for its validity, psychometric defensibility, and policy implications. (Status: in development)

Implemented Score Validation and Score Collection Process: OSA collects a stratified sample of operational test results immediately after a Regents exam administration; for Math A, SED ran a validation study and, for other exams, we collect data to analyze operational data against field testing data to certify the scaling and equating of the exams (June 2004: Math B and Chemistry collected). This process forms the cornerstone for the development of the Standard Revisitation methodology. (Status: implemented and continuing)

Implemented Revised Field Testing Procedures: OSA researchers designed and implemented a six-year Field Testing Matrix that informs high schools of the tests that they need to administer in a specific academic year. This system is designed to increase the size and reliability of the sample and to have equity in the field testing program.  Also, OSA researchers used the same process for a five-year Field Testing Matrix for the upcoming NCLB grades 3-8 testing program for the same reasons listed above. (Status: implemented and continuing)

RFPs/Contracts to Increase Capacity: OSA has awarded contracts to CTB/McGraw Hill for development of the NCLB grades 3-8 assessments in English Language Arts and Math, and has issued or will be issuing RFPs for foreign language translations of State examinations, the NYS English as a Second Language Achievement Test, scaling and equating, and NCLB grades 3-8 scoring and reporting. (Status: implemented and continuing)

Design and Implementation of the Electronic State Testing Tracking System: This electronic system will allow OSA staff to track the test development process from item creation to the final shipping process. The system is close to completion and is expected to be on-line September 2004. (Status: in development)

Implemented Electronic Consultant Inventory Database (ECI): This database provides contact information, tracking, and reporting of payments to teachers who participate in test development activities. (Status: implemented and continuing)

Implementation of Electronic Test Ordering System: This system will allow schools to order State tests on-line. The system will be pilot tested this fall to prepare for pilot implementation for January 2005 Regents exams. (Status: in development)

Implemented Regents Testing Cycle Review Meetings:  During Regents exam periods, OSA managers meet twice daily to review all phone calls to SED. All calls are logged in and evaluated. If a correction or guidance to the field is necessary, the information is communicated to the field immediately. At the end of the testing cycle, OSA senior managers meet to review calls and discuss their findings with testing associates in OSA. (Status: implemented and continuing)

Implemented Automated Phone System and Dedicated E-mail Box:  The automated phone system gives general answers to the most commonly asked questions from the public and it provides the web address for additional information. The dedicated e-mail box is designed to provide quick responses from OSA to teachers and administrators regarding all aspects of our testing program. (Status: implemented and continuing)

Implemented Regents Examination Website: During Regents exam periods, OSA maintains a special link on the SED website providing updated information regarding the Regents exams to aid schools as they score the tests and to post the score conversion chart for each exam. (Status: implemented and continuing)

District Superintendents’ Curriculum and Assessment Committee Testing Workshop: OSA and Curriculum, Instruction and Instructional Technology managers, District Superintendents, and psychometricians from Pearson Educational Measurement designed and presented a workshop for invited guests from around the State. Topics included: SED testing program information and test development processes; utilization of large-scale assessment programs; valid data analysis techniques; scaling and equating; and a panel discussion to address audience concerns and issues. (Status: completed)

Assistant Commissioner Outreach to Educational Organizations:  The Assistant Commissioner for Standards, Assessment and Reporting routinely contacts executive personnel in educational organizations (e.g., the New York State Council of School Superintendents, the New York State United Teachers, and the New York State School Boards Association) and State Networks/Associations (e.g., Staff and Curriculum Development Network, Big 4 Test Directors Work Group, New York City Department of Education Assessment Cabinet, New York State Math and English Councils) to discuss pertinent issues facing the OSA.  (Status: continuing)

Physics Work Group

Implemented Physics Work Group Suggestions: Reset standards and implemented the revised Physics exam in accordance with Regents and Commissioner’s directives. OSA analyzed January 2004 and June 2004 exams to certify that they were psychometrically comparable, indicating that SED has stabilized the exam. OSA managers will continue to monitor test development and administration of Physics to ensure comparability, validity, and reliability. (Status: implemented and continuing)

Math A Panel Recommendations

Implemented Math A Panel Recommendations:  OSA restructured the Regents exam in accordance with the Math A Panel recommendations, conducted a formal score validation study after the January 2004 administration to certify that field test data and operational test data were aligned, issued a conversion chart for the Math A January exam, and implemented a second form of the revised Math A Regents exam in June 2004 that was comparable to the January form. SED has stabilized the exam as per the Regents and Commissioner’s directive. OSA managers will continue to monitor test development and administration of Math A to ensure comparability, validity, and reliability. (Status: implemented and continuing)

Math Standards Committee:  The Committee has been formed and received its charge from the Commissioner. OSA managers stay abreast of the Committee’s work and attend meetings when appropriate to discuss the development of the new standards. This work will inform the design of the NCLB grades 3-8 testing program and the new Math A exam. The Committee recommendations will be final in the fall, but its preliminary recommendations have already begun to be used by the Test Specification and Framework Committee working on the grades 3-8 math assessments (Status: implemented and continuing)

Grade-by-Grade Testing Under NCLB

As required by the No Child Left Behind Act, New York will be administering English Language Arts (ELA) and mathematics assessments in grades 3–8 beginning in 2006 (January and March, respectively). The contract for this work was awarded to CTB/McGraw-Hill and activities are under way. In June and August respectively, the test specifications and frameworks committees for ELA and math met to develop specifications for the tests in each grade.  These committees are comprised of certified teachers from across the State.  Additional information on these assessments will be provided to the public this fall via direct mailings, a statewide teleconference and regional forums.  We will also provide the Regents with a full briefing on the updated math standards and the grade-by-grade test specifications in November.  (Status:  implemented and continuing)