|
|
THE STATE
EDUCATION DEPARTMENT / THE UNIVERSITY
OF THE STATE OF NEW YORK / ALBANY, NY 12234 |
|
TO: |
The Honorable the Members of the Board of Regents |
|
FROM: |
James A. Kadamus |
|
COMMITTEE: |
EMSC-VESID |
|
TITLE OF ITEM: |
Overview of the Grades 3-8 Testing Program |
|
DATE OF
SUBMISSION: |
October 19, 2004 |
|
PROPOSED
HANDLING: |
Discussion |
|
RATIONALE FOR
ITEM: |
Implement Regents Policy |
|
STRATEGIC GOAL: |
Goals 1 and 2 |
|
AUTHORIZATION(S): |
|
SUMMARY:
This report provides an update on New York State’s grade-by-grade testing program that is being implemented to comply with the No Child Left Behind (NCLB) Act. It is one in a series on assessment issues that have been scheduled on the Committee's upcoming agendas. It updates the report provided to the Board in September 2004.
Introduction
The New York State Testing Program (NYSTP) is designed to evaluate the implementation of the State Learning Standards at the student, school, district, and statewide levels. The Department currently tests students in English Language Arts (ELA) and mathematics in grades 4 and 8 and at the commencement level with the Regents examinations. To meet the new mandates under NCLB, tests in ELA and mathematics are to be administered annually at every grade level from grade 3 to grade 8.
NCLB Testing
Requirements
NCLB specifies that statewide tests must:
· address the depth and breadth of the state content standards;
· be valid, reliable, and of high technical quality; and
· be designed to provide a coherent system across grades and subjects.
With the exception of the small population of severely disabled students who qualify to take the State’s Alternate Assessment, all students, both general education and special populations, in all public school districts, will be required to take the exams. While the decision to test in grades 3-8 was initially mandated by the federal government, these new tests also present the opportunity to:
· annually evaluate the implementation of the New York State Learning Standards;
· measure individual student and cohort progress; and
· gather data on student readiness for study at the next level.
Philosophical Basis for
Tests
A number of test designs were considered to respond to the federal
mandate for testing in grades 3-8.
A Committee of Practitioners was formed to guide the Department's
thinking on the test design. The
Committee rejected the idea of developing multiple choice tests for grades 3, 5,
6 and 7 that could be used only to supplement the current Regents accountability
system. Instead, the Committee
stressed that tests in all grades must test the New York State learning
standards and provide local administrators and teachers with data to help
struggling students and help schools improve instruction. They sought tests that clearly tied
assessment, curriculum and instruction most closely. However, these new tests must be
administered and scored more efficiently than ever before.
The State Education Department has adopted this philosophy in developing the new grades 3-8 tests.
Test Development Process
CTB/McGraw Hill (CTB) was awarded the contract to design, develop, field-test, standard-set and scale the new grades 3-8 testing program in ELA and mathematics. CTB is required to adhere to the New York State Education Department's Examination Development Process, which requires extensive involvement of New York State teachers and administrators. Teachers and administrators have served or will serve on a number of committees and will be an integral part of the test development process.
Test Specifications
The first step in test development was to
create new specifications for test questions. Committees composed of New York
State teachers, State Education Department staff, and staff from CTB met to
ensure that the test questions would be aligned with State standards and
classroom instruction. Information from these committees also provided
guidelines for writing the test questions by selecting standards and performance
indicators most appropriate for statewide assessment, and by identifying formats
best suited for assessing those standards and performance indicators. The
English Language Arts committee also reviewed the reading passages to be used in
the tests.
Item
Development
State-developed specifications were used, along with the State Learning Standards, to train and guide experienced writers at CTB in developing appropriate test questions. Test questions were subjected to a rigorous editing and review cycle by CTB, and were then reviewed by select committees of teachers and administrators from throughout New York State and by Department staff. Questions were reviewed from several perspectives, including:
· alignment to State Learning Standards;
· contexts that were appropriate in both grade level and content;
· a range of the depth of knowledge needed to answer questions;
· clear and concise language;
· content-specific issues, e.g., calculator use; and
· appropriateness of genre for literary texts.
Based on the recommendations of the review committees, test questions were accepted, revised, or rejected. A final review with Department staff will determine the final questions that will be selected for field-testing. After field-testing, only questions shown by data analysis to be valid and reliable measurements of the State Learning Standards are selected for inclusion on the final operational test forms.
Test Structure and
Content
The tests reflect New York State
content/process standards in each grade and subject area. They are designed to
signal priority content and to be instructionally sensitive. That is, they
should be able to be used by teachers and administrators to inform curriculum,
instruction, and locally developed assessment.
Tests in both subjects will be similar in
format to the existing grades 4 and 8 assessments; all tests (ELA and
mathematics) will contain multiple choice, constructed response (short answer),
and extended response (essay) questions. The largest change is that the
Independent Writing Prompt, an open-ended essay, on the ELA grades 4 and 8 exams
has been removed. Additional detail on the test
designs for each subject and grade level can be found in the accompanying
slides.
Administration
Time
All efforts have been made in the test
development process to limit the length of the tests while ensuring the validity
and reliability of the assessment program. Based on current tests formats, the
grades 4, 6 and 8 tests will be somewhat shorter than the current grades 4 and 8
tests, with the grades 3, 5 and 7 tests being about one-half the length of the
current grades 4 and 8 tests. We
have analyzed the testing times of some typical off-grade tests currently used
by schools, and we believe that the total testing time for all tests, grades
3-8, will be the same or even less than the time schools currently devote to
State and off-grade testing.
Participation
of Special Populations
Federal law requires all students to
participate in State assessments regardless of any special needs. For students
with disabilities, the school district’s Committee on Special Education
determines how a student with a disability will be assessed and the majority of
special education students participate in the regular assessments. For severely disabled students, the
State provides the Alternate Assessment. To guarantee appropriate representation
for special population students, Office of State
Assessment
staff have worked collaboratively with staff
from VESID and Bilingual Education in the design of the new testing system.
Educators working with special populations
(students with disabilities and English language learners) have participated in
the test specifications/framework process, item selection, and item review
process. VESID staff have provided assistance on the principles of Universal
Design for Learning, which we are working to ensure are used in the grades 3-8
test development process.
The principles of Universal Design for
Learning include, but are not limited to:
·
development of
instructional and assessment materials that are varied and diverse;
·
development of
tests that do more than accommodate physical, sensory, or cognitive
disabilities;
·
development of
tests that are flexible and promote alternatives; and
·
development of
tests that are inclusive.
Field-Testing
Field-testing is a fundamental component of
a high quality statewide testing program and the Department depends upon school
participation. The results of student performance from a stratified sample of
schools across the State will help the Office of State Assessment to develop the
best test items for the resulting operational tests administered to all
students. While the new NCLB testing requirements in grades 3-8 do not go into
effect until 2006, field-testing at grades 3-8 needs to be conducted in 2005 so
that the results and data collected from these items can be used as the basis
for the development of the operational tests to be administered in
2006.
The Department is aware that some schools
see field-testing as intrusive. The Department and CTB have taken many steps to
limit schools’ field-testing responsibilities, while still collecting robust
stratified samples from the field. The Office of State Assessment has designed a
new Five-Year Field-Test Sampling Matrix for the Elementary and Middle School
State Tests, similar to the system that has been successfully implemented for
Regents examinations.
During the 2004-2005 school year, in
addition to the operational English Language Arts and mathematics tests
administered in grades 4 and 8, field-tests in both content areas will also be
administered in grades 3, 5, 6 and 7. The table below lists the 2005 test
administration dates for ELA and mathematics at each grade
level.
|
Content
Area |
Grade
Levels |
Test Administration
Dates |
|
English Language Arts
Field-Test |
3 through 8 |
February 7 – 11,
2005 |
|
Mathematics
Field-Test |
3 through 8 |
March 14 – 18,
2005 |
|
4 and 8 |
January 31 – February 4,
2005 | |
|
Mathematics Operational
Test |
4 and 8 |
May 10 -11,
2005 |
First Administration of Operational
Tests
The grades 3-8 testing program is scheduled
to begin during the 2005-2006 academic year. ELA will be administered in late
January-early February 2006 and mathematics will be administered in March 2006.
Testing on these dates will allow the Department the time necessary for the
scoring and certification of the testing system, while still getting test
results back to school districts in a timely manner.
Scoring
We are still
studying the best ways to score these tests and get results to schools as
quickly as possible. There are many
challenges associated with scoring these tests in a way that preserves the
reliability of the scoring while not unduly affecting instructional time. Over the summer we issued a Request for
Information related to printing, scanning and scoring these tests. Based on the information we
received, we are considering a number of models, including distributive scoring.
Such a scoring model could allow the use of technology to alleviate some of the
scoring burden on schools, while assuring that teacher expertise is not
lost. Under a distributive scoring
model, test results would be scanned into a computerized database and teachers
would be able to score essays or math problems by accessing them from an online
database. A number of factors,
including cost, will be considered in deciding the most reliable and efficient
process of scoring to use.
Currently, CTB staff, Department staff, and
members of the Department’s Technical Advisory Group are studying the design of
the scaling system to be used for this new family of tests. The tests will still
produce a scaled score that places a student into the 4 current performance
levels: Level 1, Serious Academic Deficiencies; Level 2, Basic; Level 3,
Proficient, and Level 4, Advanced. Furthermore, the Department is studying the
possibility of creating scales that will measure individual student growth. Such a scaling system, combined
with the new individualized student information system, will give schools more
precise information to track the progress of both cohort and individual student
growth over time.
Reporting of
Scores
The Department will contract with a vendor to produce uniform score
reports by school, classroom and individual students. There are a number of such systems
currently in existence in other states.
These basic score reports will be put in a computerized database to be
accessed by authorized school personnel.
The Department is currently studying ways to decrease the turnaround time
from testing to reporting of scores.
A key factor to shorten the length of time from testing to reporting will
be the type of scoring process used.
In addition, the implementation of the individual student information
system should make recording of individual student scores
easier.
Turnaround time will also depend on better training of local school staff
who enter data. In our regional
meetings on grades 3-8 testing, we will be emphasizing ways to make this
reporting faster.
Accountability
Implications
We do not believe that using the results of tests in the additional
grades will adversely affect schools, since they will continue to receive one
accountability “score” per school.
However, there may be a slight performance “dip” at the beginning,
especially in mathematics, where the development of the tests and the
introduction of revised standards are happening simultaneously. Therefore, we have begun discussions,
including with USDOE, to make adjustments in the accountability system as needed
to ensure that all schools are fairly treated.
As we move forward with the planning and implementation of this testing program, we will continue to keep the Board of Regents fully apprised.