SUMMARY

Issue for Discussion

Continuation of the policy discussion related to the Grades 3-8 Testing Program. Last month, the Board of Regents discussed:

• The creation of a system that will allow test scores to be returned within 10 weeks of the last make-up exam administered. 
• The possibility of revising the current testing calendar and administering assessments later in the school year starting with the 2010-11 school year, and
• Additional steps that would be needed if the tests are administered later in the year in order to be able to return the test scores before the end of the school year. 

               Following that discussion, the Board of Regents agreed that it was important to secure additional information from the field to inform this policy discussion, and to review a cost/benefit analysis associated with changing the testing calendar.  This month the Board of Regents will discuss those follow-up activities and two additional policy questions that are critical as the Regents consider the next round of contracts for testing.  Those policy questions are:

• How should the grades 3-8 tests be used?
• What types of questions should be on the tests? Should there be multiple choice and open-ended questions or only multiple choice questions?

Reason(s) for Consideration

              Review of Policy

Procedural History

During the March 2008 meeting of the Regents Committee on Policy Integration and Innovation, the Commissioner presented information about the administration, scanning, scoring, and score returns for the Grades 3-8 Mathematics and English Language Arts examinations.  The Board requested that staff continue to examine possible short-term and long-term solutions to the issues.

During the May 2008 meeting of the Board of Regents, the Senior Deputy described a plan to significantly expedite the release the Grades 3-8 ELA and Math test scores. The goal described was to return scores within 10 weeks of the last make-up exam administered.   The creation of a system to achieve this has been accomplished.

During the December 2008 meeting of the Regents EMSC Committee, staff presented updated information regarding the administration, scanning, scoring, and score returns for the Grades 3-8 English Language Arts and Mathematics examinations.  The Board then requested a fuller discussion regarding primary policy issues regarding this testing program for the January 2009 EMSC Committee meeting.

During the January 2009 meeting of the Regents EMSC Committee, staff presented updated information on administration, scanning, scoring, and score returns for the Grades 3-8 English Language Arts and Mathematics examinations. Members of the Board requested that the Department staff survey the field for input on test scheduling and that staff evaluate the cost/benefit of possibly moving the tests  from their current administration times to later in the academic year.

Background Information

Current Purpose of the New York State Grades 3-8 Testing Program

The New York State Testing Program (NYSTP) is designed to evaluate the implementation of the State Learning Standards at the student, school, district, and statewide levels. These tests present the opportunity to: annually assess the implementation of the State’s learning standards; measure individual student and cohort progress; and gather data on student readiness for study at the next level. New York State is required to administer tests in English Language Arts (ELA) and mathematics under the No Child Left Behind Act (NCLB).  These tests must: address the depth and breadth of the State’s Learning Standards; be valid, reliable, and of high technical quality; and be designed to provide a coherent system across grades and subjects.  The Grades 3-8 Testing Program is administered to approximately 300,000 students per grade in both public and nonpublic schools.  With the exception of the small population of severely disabled students who qualify to take the New York State Alternate Assessment (NYSAA), all students, general education, students with disabilities (SWDs), and English Language Learners (ELLs) in all public school districts are required to take these tests. 

The ELA tests target student progress toward three of the four content standards (the Department does not test speaking).  The mathematics tests target student progress toward the five content bands.  The established cut scores classify student proficiency into one of four Performance Levels based on their test performance:

Level 1:  Not Meeting Learning Standards

              Student performance does not demonstrate an understanding of the ELA/mathematics knowledge and skills expected at this grade level.

              Level 2:  Partially Meeting Learning Standards

              Student performance demonstrates a partial understanding of the ELA/mathematics knowledge and skills expected at this grade level.

              Level 3:  Meeting Learning Standards

              Student performance demonstrates an understanding of the ELA/mathematics knowledge and skills expected at this grade level.

              Level 4:  Meeting Learning Standards with Distinction

              Student performance demonstrates a thorough understanding of the ELA/mathematics knowledge and skills expected at this grade level.

All students who score below the State-designated Performance Level 3 must be provided Academic Intervention Services.

Assessment Policy Questions

When should the grades 3-8 ELA and Math tests be given?

              Currently, tests are administered in January and March in a staggered format, Grades 3-5 and Grades 6-8.  There is also a make-up period for the exams to ensure that the school systems have time to test: students who were absent; students with disabilities; and English Language Learners.  The current testing windows respond to input from multiple-stakeholders (e.g., New York State Council of School Superintendents, New York State United Teachers, District Superintendents, The Big 5 School Districts, New York State School Boards Association, and School Administrators Association of New York State).

              Field Survey

In response to the Regents request, and to better understand the needs of the field and the costs associated with a change in the test schedule, Department staff developed a brief online survey designed to get input from the field about testing and scheduling issues tied to the Grades 3-8 Testing Program.   The survey asks questions about preferences for which month the Grades 3-8 mathematics and English Language Arts examinations should be administered as well as preferences for scoring models (school, district, regional or vendor scoring) and for test format (multiple-choice vs. open-ended questions). It also asks respondents to identify other relevant issues such as availability of sufficient room and staffing for the exams.   The survey has been posted on-line and in March the survey results will be provided to the Regents to help inform their policy decision.    

The Regents also asked about the costs and benefits related to moving administration of ELA and Math tests to the spring. Here are possible dates with costs and benefits:

Benefits and Costs of Administering ELA and Math in March and early April

Benefits:

• Permits more time to teach the next year’s curriculum prior to the administration of the next test.
• Allows the current scoring and scanning methods to be used and still return the test results before the end of the school year. This requires a 10-week period of time, so tests must be administered no later than early April.
• Schools will have information they need to place students and plan to help students improve achievement during the next school year.
• Since test results would be returned in the same school year, the Department would also be better able to make school accountability decisions before the start of the next school year. New federal regulations just issued require that states announce school accountability status prior to the school year.

Costs:

• If the change in test administration dates is made in 2010-2011, when the new testing contract is in place, there would be no additional cost to the Department for the scheduling change.
• If the change in test administration dates is made after the new 3-8 contact is in place, this will result in significant costs to the Department and will possibly require: the negotiation of a contract amendment; revision of test specifications and curriculum coverage; and the possibility that the exams will have to be standard set a second time. These costs can easily exceed $2 million.
• Moving test dates will have a fiscal impact on school districts in regard to math. Moving the current math testing dates will require additional curriculum analysis, revision of pacing guides, and subsequent professional development for staff to communicate changes and expectations. It is difficult to estimate these costs statewide. However, since the ELA standards are being revised, school districts will have to make those changes anyway; therefore, moving the ELA test dates will not have an additional fiscal impact.

Benefits and Costs of Administering ELA and Math in May and June

Benefits

• Permits even more time to teach the next year’s curriculum prior to the administration of the next test.
• Responds to educators in the field who have suggested that these tests should be administered at the same time as other end-of-course tests including Regents Exams. The 3-8 tests would be given during the same time period as other State tests.
• A change in the timeframe of the assessments will require a revision of the math curriculum to ensure appropriate content coverage and associated professional development for instructional staff. 

Costs

• Test results could not be returned to districts prior to the end of the school year using the current process of scanning and scoring. We would need to explore options to the current system of teachers scoring the open-ended questions, possibly hiring a vendor to handle centralized scanning and scoring. A single vendor would still require several weeks to scan and score; it is difficult to determine the exact amount of time required without releasing an RFP.
• Based on industry research and costs incurred in other states, vendor costs could range from $15 - $25 million per year.  
• It would be more difficult to be able to return the test scores and make accountability decisions by the beginning of the next school year, as required by new federal regulations.

How should the grades 3-8 tests be used?

“Growth for All”

The Board of Regents wants to ensure that the tests and the accountability system measure and recognize “Growth for All” students. The current grades 3-8 tests do not have enough questions to do that at every level of the scale from easy to hard; this is especially true at Level 1 and Level 4.

In order to accomplish the Regents goal, it will be necessary to have more questions on the new tests. The current ELA Testing Program has as few as 24-28 multiple choice items per test depending on the grade level; we propose increasing this to approximately 50 per test. The current Mathematics Testing Program has as few as 25-30 multiple choice items per test depending on the grade level; we propose increasing this to 60-70 per test.

Increasing the number of multiple choice questions will make the measurement of student achievement much more precise. For example, it will be possible to determine whether a student has scored a 3.2 or a 2.7 or a 3.9. It will also mean increasing the ELA testing time to 150 minutes spread over three days (45 minutes the first two days and 60 minutes the last day). The Math testing time would increase to 150 minutes in 3rd grade and 180 minutes in grades 4-8; that would also be spread over three days. The addition of more questions, possibly doubling the number in certain grades, will have other implications that should be discussed. 

Another way to measure “Growth for All”  is to determine by scale scores how much student achievement has improved from one grade to the next. To do this, the new tests must have a “vertical scale” that continues from 3rd through 8th grade. In other words, a 650 may be the beginning of Level 3 in grade 3, while 670 may be the beginning of Level 3 in grade 4, and 690 is the beginning of Level 3 in grade 6, and so forth. If a third grader has a scale score of 690, it will be possible to say that the student has achieved all the standards for fifth grade.

        What types of questions should be on the tests? Should there be multiple choice and open-ended questions or only multiple choice questions? (For example, in ELA, should there be multiple choice and open-ended questions that measure reading, writing, and listening, or should there be only multiple choice questions that measure only reading and listening?)

Multiple-choice Items

Advantages

• Multiple-choice items provide versatility in assessing a wide range of learning objectives.
• Well constructed multiple-choice items can target factual knowledge, comprehension, analysis, synthesis, or evaluation.
• The objective nature of multiple-choice items ensures reliable, efficient scoring in less time.
• More content can be tested in a given time period through multiple-choice items.
• It is less expensive and takes less time to administer and score multiple-choice items.
• Multiple-choice items allow for more adequate sampling of content.
• Multiple-choice items can be constructed to address various levels of difficulty.
• Multiple-choice items provide stability for scaling and equating.

Challenges

• Multiple-choice items are not effective for measuring the ability to organize or express ideas, formulate arguments, or demonstrate original thought.
• Multiple-choice items are difficult to create because the answer must be right or wrong with no shades or nuances.
• Multiple-choice items cannot be used to test some of the skills in
  New York State’s Core Curricula (e.g., construct various types of reasoning, arguments, justifications, and methods of proof for problems).

Open-ended Items

Advantages

• Open-ended items focus on students' understanding, their ability to reason/problem solve, and their ability to apply knowledge in less traditional contexts.
• Open-ended items can address the essential concepts, processes, and skills within each subject area and can positively impact instruction in problem solving, higher thinking skills, and writing.
• Open-ended items require complex thinking, measure content knowledge in greater depth, and yield multiple solutions that go beyond simply memorizing facts.
• Businesses have noted an increasing need for people who can manage information, see patterns, identify needs, solve problems, and think critically.   
• Open-ended items more directly test the skills portion of core curricula (e.g., construct various types of reasoning, arguments, justifications, and methods of proof for problems).
• Open-ended items improve student writing in the content areas.
• A student may develop responses to open-ended items from his/her own background and experience.  This allows students to relate to the information being asked and to demonstrate creative expression and divergent thinking.
• Open-ended items allow students to show partial grasp of concepts or skills.

Challenges

• Open-ended items require the development of extensive rating materials and training protocols that control for reliability.
• Open-ended items require teachers or evaluators to interpret and use multiple criteria in evaluating responses and therefore necessitate a more extensive scoring process that takes more time and resources.
• Open-ended items require more time for field testing.

Combining Multiple Choice and Open-ended Items on Tests

• Allows for wider coverage of skills in New York State’s Core Curricula.
• Allows for more flexibility in types of skills and content assessed.
• Allows students, especially those with different learning styles, a greater opportunity to demonstrate knowledge and to allow for different response styles.