SUMMARY

Issue for Discussion

Continuation of the policy discussion related to the Grades 3-8 Testing Program. Last month, the Board discussed:

• Test administration dates for English Language Arts and mathematics: advantages and challenges;
• Possible costs associated with moving the tests at the outset of the new testing contract in spring of 2011;
• The testing program’s ability to fulfill the Board’s Growth for All approach to the next generation of accountability; and
• The advantages and challenges associated with the format of test questions; all multiple-choice versus a combination of multiple-choice and open-ended questions.

              Following the February discussion, the Board agreed that it was important to secure additional information from the field to inform the policy discussion.  A survey was developed to gauge the field’s preferences on issues surrounding test scheduling, scoring, and format.   The survey was posted on-line for three weeks and we received 22,569 responses. Over 69% of responses were from teachers, 16% were from school administrators, and 10% were from parents. Respondents were generally very positive about the opportunity to have input on these issues. This month, the Regents will discuss the extensive feedback received from the field.

Reason(s) for Consideration

Review of Policy

Procedural History

During the March 2008 meeting of the Regents Committee on Policy Integration and Innovation, the Commissioner presented information about the administration, scanning, scoring, and score reports for the Grades 3-8 Mathematics and English Language Arts examinations.  We also indicated that we would continue to examine possible short-term and long-term solutions to the issues.

At the May 2008 meeting of the Board of Regents, the Senior Deputy described a plan to significantly expedite the release the Grades 3-8 ELA and Math test scores. The goal described was to return scores within 10 weeks of the last make-up exam administered.   The creation of a system to achieve this has been accomplished.

At the December 2008 meeting of the Regents EMSC Committee, staff presented updated information regarding the administration, scanning, scoring, and score returns for the Grades 3-8 English Language Arts and Mathematics examinations.  The Board then requested a fuller discussion regarding primary policy issues regarding this testing program for the January 2009 EMSC Committee meeting.

During the January 2009 meeting of the Regents EMSC Committee, staff presented updated information on administration, scanning, scoring, and score returns for the Grades 3-8 English Language Arts and Mathematics examinations. Members of the Board requested that the Department staff survey the field for input on test scheduling and that staff evaluate the cost/benefit of possibly moving the tests  from their current administration times to later in the academic year.

              During the February 2009 meeting of the Regents EMSC Committee, staff presented updated information regarding costs and benefits tied to administering the tests at different times in the spring as well as the advantages and challenges associated with multiple-choice versus open-ended test formats. The Board also reviewed how the revised testing program can support the Regents Growth for All accountability initiative.  In order to help inform the Regents policy decision, staff developed a field survey which included questions on test scheduling, test question format and scoring options.  The Department received 22,569 responses to the survey which was posted on-line for three weeks. 

Background Information

Current Purpose of the New York State Grades 3-8 Testing Program

The New York State Testing Program (NYSTP) is designed to evaluate the implementation of the State Learning Standards at the student, school, district, and statewide levels. These tests present the opportunity to: annually assess the implementation of the State’s learning standards; measure individual student and cohort progress; and gather data on student readiness for study at the next level. New York State is required to administer tests in English Language Arts (ELA) and mathematics under the No Child Left Behind Act (NCLB).  These tests must: address the depth and breadth of the State’s Learning Standards; be valid, reliable, and of high technical quality; and be designed to provide a coherent system across grades and subjects.  The Grades 3-8 Testing Program is administered to approximately 300,000 students per grade in both public and nonpublic schools.  With the exception of the small population of severely disabled students who qualify to take the New York State Alternate Assessment (NYSAA), all students, general education, students with disabilities (SWDs), and English Language Learners (ELLs) in all public school districts are required to take these tests. 

The ELA tests target student progress toward three of the four content standards (the Department does not test speaking).  The mathematics tests target student progress toward the five content bands.  The established cut scores classify student proficiency into one of four Performance Levels based on their test performance:

Level 1:  Not Meeting Learning Standards

              Student performance does not demonstrate an understanding of the ELA/mathematics knowledge and skills expected at this grade level.

              Level 2:  Partially Meeting Learning Standards

              Student performance demonstrates a partial understanding of the ELA/mathematics knowledge and skills expected at this grade level.

              Level 3:  Meeting Learning Standards

              Student performance demonstrates an understanding of the ELA/mathematics knowledge and skills expected at this grade level.

              Level 4:  Meeting Learning Standards with Distinction

              Student performance demonstrates a thorough understanding of the ELA/mathematics knowledge and skills expected at this grade level.

              All students who score below the State-designated Performance Level 3 must be provided Academic Intervention Services.

              The attached document provides the data from the on-line field survey conducted between the February and March 2009 meetings of the Board of Regents.  During the three weeks the survey was available on-line we received 22,569 responses.  The attached document also includes advantages and challenges associated with some of the scheduling options presented to the survey respondents. 

Highlights of the Survey Data

• Respondents had no clear-cut preference as to which month the ELA tests should be given. Responses were as follows:               January – 22%, March – 21%, April  – 18%, May  – 25%, and June  – 14%.

• More respondents preferred that the Math tests be given in May (40%), but no month received a majority.

• 70% of all respondents believed that the exams should be separated by one or more months.

• Respondents were almost evenly split on the issue of whether the testing period for Grades 3-8 English Language Arts and Mathematics should be concentrated into one or two weeks during which each test would be administered to all students statewide on the same day.

• 76% of all respondents favored local or regional scoring, while 24% favored State-contracted external vendor scoring.

• Approximately 85% of all respondents preferred that the tests be a combination of both multiple-choice and open-ended questions.

Recommendation

That the Regents discuss the survey results and the advantages and challenges associated with the scheduling options and decide on desired next steps in the policy discussion related to the Grades 3-8 Tests. 

In theory, there are several possible options for dates for the ELA and Math tests:

• Move the ELA test to March and continue to give the Math test in March.

• Give one or both tests in May or even in June.

• Keep the testing administration dates as they are now: January for ELA and March for Math.

• Other

However, there are other considerations:

• The survey indicates that a very large majority want to continue to have tests that include both multiple choice and one-ended questions. Scoring open-ended questions adds considerably to the time needed for scoring.

• The survey also indicates that a large majority want to continue to have local or regional scoring, not scoring by an external vendor.  In addition, a new federal regulation requires that schools must be informed of their accountability status (which in part is based on the ELA and Math scores) before the beginning of the next school year; in order to meet that deadline, the test scores need to be returned before the end of the previous school year.

Staff have carefully analyzed all of the options and considerations described above.  If we continue to have open-ended questions on the tests, continue to score the tests locally,  and if the results are returned before the end of the school year, the options become more limited.  Adding together the time periods needed for test administration (including make-up testing), local scoring, test scanning, and psychometric evaluation yields a minimum of 11 weeks.  It takes an additional two weeks to print Individual Student Reports and distribute them to parents. Therefore, in order to return test scores before the end of the school year, the current structure would require that the tests need to be given no later than March.

Timetable for Implementation

N/A

Attachment

Grades 3-8 Testing Survey Results and Advantages and Challenges Associated with the Options

                  Department staff developed an on-line survey designed to collect input from the field on questions concerning the scheduling, scoring and format of the Grades 3-8 Tests.  We received 22, 569 responses. We heard from all major stakeholder groups and from all regions of the State.  The survey questions and results are detailed below.  For the questions related to scheduling we have also included the advantages and challenges associated with the various options presented in the survey.  The advantages and challenges were not provided in the survey but are included in this item to help to inform the Regents policy discussion. 

A.  Demographics of Respondents 

Respondent’s Role

Respondent’s School Location

B.  Scheduling of Tests

Question 1:  Do you think we should consider rescheduling the Grades 3-8 tests from the current format?  (English Language Arts is administered in January
and Mathematics in March.)

Response:   Approximately 81% of all respondents favored some type of rescheduling for the new Grades 3-8 testing program.   Percentages of school superintendents, administrators and teachers favoring the move ranged from 82-84%; 78% of Board of Education members and 61% of parents favored moving the exams.

Question 2:  During which month do you believe the Grades 3-8 English Language Arts Tests should be administered?

Response:   Respondents had no clear-cut preference as to which month the ELA tests should be given. Responses were as follows: January – 22%, March – 21%, April – 18%, May – 25%, and June – 14%.  The tests are currently administered in March.

• 62% of all respondents preferred the ELA exams to be administered prior to the end of April while 38% preferred the exams to be administered in May/June.

• 48% of superintendents preferred the ELA exams to be administered prior to the end of April while 53% preferred the exams to be administered in May/June.

• 61% of all other administrators preferred the ELA exams to be administered prior to the end of April while 42% preferred the exams to be administered in May/June.

• 61% of teachers preferred the ELA exams to be administered prior to the end of April while 40% preferred the exams to be administered in May/June.

• 53% of BOE members preferred the ELA exams to be administered prior to the end of April while 47% preferred the exams to be administered in May/June.

• 75% of parents preferred the ELA exams to be administered prior to the end of April while 25% preferred the exams to be administered in May/June.

Advantages of Administering ELA Exams Prior to the End of April:

• Current scoring and scanning timelines can still be used.
• Test results for students will be available before the end of the school year.
• Test results will be available for NYCDOE for their promotion/retention policy.
• Test results will be back in time for school accountability status determinations to be made prior to the start of the next school year.
• It will be possible for administrators to start data verification processes for accountability determinations prior to the start of summer.
• Test results will be available for summer curriculum and data analysis work.

Challenges to Administering ELA Exams Prior to the End of April:

• Less time available to teach grade-level curriculum before the test.
• Possible problems with winter weather in selected areas of the state when testing in early spring.
• School vacation periods which are linked with religious holidays differ every year.  Upstate and downstate school districts have different vacation periods. 

Advantages of Administering ELA Exams in May/June:

• More time available to teach grade-level curriculum before the test.
• Satisfies demand that the exams should be administered near the end of the course like Regents Examinations.

Challenges to Administering ELA Exams in May/June:

• Test results for students will not be available before the end of the school year.
• It will not be possible to make school accountability status determinations by the beginning of the next school year as now required by NCLB.
• Test results will not be available for NYCDOE for their promotion/retention policy.
• Test results will not be available until very late into the summer and possibly early fall which hampers schools’ ability to appropriately assign students for Academic Intervention Services (AIS).
• Less time will be available for teachers to score the tests, particularly if the mathematics tests are also administered in May.
• Less time will be available for Regional Information Centers to scan and prepare tests for submission to vendor, especially if the mathematics tests are also administered in May.
• It will not be possible for administrators to start data verification processes for accountability determinations prior to the start of summer.
• Testing schedule will overlap with Grade 4 & 8 Science tests, Social Studies 8 test, and NYSESLAT (administered in May/June), unless those testing dates are changed.
• Test results will not be available for summer curriculum and data analysis work.

Question 3:  During which month do you believe the Grades 3-8 Mathematics Tests should be administered? 

Response:   More respondents preferred that the Math tests be given in May (40%), but no month received a majority. 35% of all respondents preferred the math exams to be administered prior to the end of April while 65% preferred the exams to be administered in May/June. The exams are currently administered in March.

• 32% of superintendents preferred the math exams to be administered prior to the end of April while 69% preferred the exams to be administered in May/June.

• 26% of administrators preferred the math exams to be administered prior to the end of April while 44% preferred the exams to be administered in May/June.

• 31% of teachers preferred the math exams to be administered prior to the end of April while 68% preferred the exams to be administered in May/June.

• 54% of parents preferred the math exams to be administered prior to the end of April while 46% preferred the exams to be administered in May/June.

• 45% of Board of Education members preferred the math exams to be administered prior to the end of April while 54% preferred the exams to be administered in May/June.

• 41% of all others preferred the math exams to be administered prior to the end of April while 60% preferred the exams to be administered in May/June.

Advantages of Administering Math Prior to the End of March

• Test results for students will be available prior to the end of the school year.
• Test results will be available for NYCDOE for promotion/retention policy.
• More time available for teachers to score exams.
• School accountability status determinations can be made prior to start of the next school year.
• Will be possible for administrators to start data verification processes for accountability determinations prior to the start of summer
• Test results will be available for summer curriculum and data analysis work.

Challenges to Administering Math Prior to the End of March

• Less time available to teach grade-level curriculum before the test.
• Possible problems with winter weather in selected areas of the state when testing in early spring.

Advantages of Administering Math Prior to the End of May/June

• More time available to teach grade-level curriculum.
• Satisfies demand that the exams should be administered near the end of the course like Regents Examinations.

Challenges to Administering Math Prior to the End of May/June

• It will not be possible to make school accountability status determinations by the beginning of the next school year as now required by NCLB.
• Test results for students not available prior to the end of the school year.
• Test results will not be available for NYCDOE for promotion/retention policy.
• Less time available for teachers to score the tests, particularly if the ELA tests are administered in a similar time slot.
• Less time for Regional Information Centers to scan and prepare tests for submission to the vendor, especially if the ELA tests are also administered in a similar time slot.
• It will not be possible for administrators to start data verification processes for accountability determinations prior to the start of summer.
• Testing schedule overlaps with Grade 4 & 8 Science tests, Social Studies 8 test, and NYSESLAT (administered in May/June).
• Test results not available for summer curriculum and data analysis work.

Question 4:  Should the English Language Arts and Mathematics Tests be administered during the same month or separated by one or more months? 

Response:  70% of all respondents agreed that the exams should be separated by one or more months.

• 58% of superintendents preferred that the exams to be administered during the same month while 42% preferred that the exams be separated by one or more months.

• 48% of administrators preferred that the exams to be administered during the same month while 52% preferred that the exams be separated by one or more months.

• 48% of teachers preferred that the exams to be administered during the same month while 52% preferred that the exams be separated by one or more months.

• 50% of parents preferred that the exams to be administered during the same month while 48% preferred that the exams be separated by one or more months.

• 61% of Board of Education Members preferred that the exams to be administered during the same month while 49% preferred that the exams be separated by one or more months.

Question 5:  Should the Department concentrate the testing period for Grades 3-8 English Language Arts and Mathematics Tests into one or two weeks during which each test would be administered to all students statewide on the same day? 

Response:  47% of all respondents said yes, while 53% said no.

Advantages of Administering Exams during a Concentrated Period

• Testing period is condensed
• Minimizes impact on day to day school activities

Challenges to Administering Exams during a Concentrated Period

• Possible that students will miss tests and make-up periods due to absence
• Creates logistics challenges for scoring of both ELA and math exams due to number of exams: approximately 1.3 million exams per subject
• Creates logistics challenges for Regional Information Centers to meet aggressive scanning timelines
• Creates possible logistics challenges for schools to meet the needs of Special Education students and English Language Learners

C.  Scoring Process

Question 6:  Which scoring model should be used for the Grades 3-8 English Language Arts and Mathematics Tests?

Response:  34% of all respondents favored Local/District Scoring Only, while 30% favored the ability to choose either local or regional scoring; 12% of respondents favored State mandated regional scoring while 24% favored State-Contracted Scoring only.

• 22% of superintendents favored Local/District Scoring Only, while 36% favored the ability to choose either Local or Regional Scoring; 11% of superintendents favored State mandated Regional Scoring while 30% favored State Contracted Scoring only.

• 27% of administrators favored Local/District Scoring Only, while 30% favored the ability to choose either Local or Regional Scoring; 10% of administrators favored State mandated Regional Scoring while 32% favored State Contracted Scoring only.

• 36% of teachers favored Local/District Scoring Only, while 31% favored the ability to choose either Local or Regional Scoring; 13% of teachers favored State mandated Regional Scoring while 20% favored State Contracted Scoring only.

• 32% of parents favored Local/District Scoring Only, while 27% favored the ability to choose either Local or Regional Scoring; 12% of parents favored State mandated Regional Scoring while 27% favored State Contracted Scoring only.

• 31% of Board of Education Members favored Local/District Scoring Only, while 25% favored the ability to choose either Local or Regional Scoring; 4% of Board of Education Members favored State mandated Regional Scoring while 40% favored State Contracted Scoring only.

Advantages of State-Contracted Scoring

• Districts will not lose instructional time by pulling teachers from classes to score exams
• Regional Information Centers will not have to scan exams
• May result in a quicker return of scores, but this depends upon length of contract that State negotiates with a vendor

Challenges of State-Contracted Scoring

• Costs borne by State
• Will require issuing an RFP for extended number of years to maximize cost savings and to ensure that the State’s exams are not waiting in a queue for processing with other states

Advantages of NYS Teacher Scoring

• Teachers become more familiar with test scoring processes and State rubrics for performance

Challenges of NYS Teacher Scoring

• Costs borne by Districts
• Lost instructional time
• Regional Information Centers must scan exams

D.  Test Format

Questions 7 & 8:   Should the format for the Grades 3-8 ELA and Mathematics tests be all multiple-choice or a combination of multiple-choice and open-ended questions?

Response:  Approximately 85% of all respondents preferred that both tests include a combination of multiple-choice and open-ended questions. Percentages of school superintendents, administrators and teachers favoring this format ranged from 81-93%; Board of Education members averaged 79% and parents averaged 61%.