|
|
THE STATE
EDUCATION DEPARTMENT / THE UNIVERSITY
OF THE STATE OF NEW YORK / ALBANY, NY 12234 |
|
TO: |
The Honorable the Members of the Board of Regents |
|
FROM: |
Richard P. Mills |
|
COMMITTEE: |
Full Board |
|
TITLE OF
ITEM: |
Independent Panel Report on Math A |
|
DATE OF
SUBMISSION: |
October 1, 2003 |
|
PROPOSED
HANDLING: |
Discussion |
|
RATIONALE FOR
ITEM: |
To provide the Panel’s findings and recommendations |
|
STRATEGIC
GOAL: |
Goals 1, 2 and 3 |
SUMMARY:
The Independent Panel on Math A has completed its final report. In my opinion, the Panel Report is wise, it is right and we must implement most of the major recommendations immediately, just as we acted quickly on their interim report in August.
I thank the Panel members, and their Chair, Dr. William Brosnan, for their careful work. They completed their task with distinction. We thank them best, however, by acting on their good advice. Their writing reveals a panel motivated by a concern for children and a belief that mathematics at a high standard is essential to their education. In creating the Panel, we gave them nine difficult questions. They answered every one of them in a manner that defines the problems with Math A and presents practical solutions.
The Panel is independent in fact as in name. They had unfettered access to documents, data, and people connected with the Regents Math A exam. They alone decided what to recommend. We promised to publish their report without revision and we will do so on October 8 at the meeting of the Board of Regents.
The report includes 22 recommendations to improve the mathematics standards, the creation and scoring of the Math A exam, and the preparation of schools to educate every child to reach the standards. Their recommendations follow from careful analysis.
In the October meeting of the Board of Regents, you will have an
opportunity to discuss this report with the Panel members. At that meeting, there will be an item
for Regents action to put most of the major recommendations into effect at once,
and assign others for committee discussion.
FINAL REPORT
to
the
New York State Board of
Regents
and
the
New York State Commissioner
of Education
Independent Panel on Math
A
Dr. William Brosnan,
Chairperson
Superintendent of
Schools
Northport-East Northport
School District
Former Teacher and
Chairperson of Mathematics
|
Stanley
Chapman Teacher of
Mathematics Clara Barton High
School (NYC) |
Daniel Jaye Assistant Principal -
Mathematics Stuyvesant High
School (NYC) Former Teacher of
Mathematics |
|
Dr. Gregory
Cizek Professor of
Educational Measurement and Evaluation University of North
Carolina (Chapel Hill) |
Dr. Sophia
Maggelakis Professor and
Department Head, Mathematics and Statistics Rochester Institute
of Technology |
|
Franco
DiPasqua Mathematics and
Science Supervisor Frontier School
District |
Theresa
McSweeney Teacher of
Mathematics Marcellus School
District |
|
Andrew
Giordano Professional
Engineer Christa Construction
|
Dr. Alfred
Posamentier Dean, School of
Education, City College Professor of
Mathematics Education |
|
Lidia
Gonzalez Teacher of
Mathematics Washington Irving
High School (NYC) |
Dr. Katherine
Staltare Chairperson,
Mathematics Department Yonkers Public
Schools |
|
Dr. Robert
Gyles Department of
Curriculum and Teaching Hunter College,
CUNY Former Teacher of
Mathematics |
Dr. Alan
Tucker Professor of Applied
Mathematics SUNY Stony
Brook |
October 8,
2003
Acknowledgments
The Panel would like to begin by expressing
its appreciation to the Board of Regents and the Commissioner of Education for
the establishment of this Panel. In
the group are some who have raised questions in the past about the mathematics
graduation requirements, and we are impressed that we have been invited to study
the issue and present a formal report.
The Panel is especially indebted to Associate Commissioner Tom Sheldon
for his help with all aspects of our work; indeed, at the end of its first
meeting, the Panel engaged in a spontaneous round of applause for Mr. Sheldon's
efforts. The Panel would also like
to express appreciation to the members of the State Education Department staff
who were involved in responding to our extensive requests for data and
information. In particular, we would like to thank Jim Kadamus, Jerry DeMauro,
Jackie Marcano, Anne Schiano, Gretchen Maresco, Terry Calabrese-Gray and Deb
Hogan for all of their help.
The Panel would like to express special
appreciation to Mary Ann Jansen, Secretary to Associate Commissioner Sheldon,
for her efficiency and her patience as we worked through several drafts, and to
Ruth Billings, Secretary to the Superintendent for the Northport-East Northport
School District who, as secretary to the Panel's Chairperson, found herself with
some new responsibilities, and who handled all with her usual
aplomb.
The Panel members are deeply appreciative to
our school districts, universities, and employers for their willingness to
permit us to engage in this important work.
Finally, the Panel members express
appreciation to our families for their patience, as we devoted much personal
time to this effort rather than engaging in other family activities. You truly are the foundation for all of
our work.
Table of Contents
I.
Introduction
IV.
The Development of the Math A
Exams
V. Findings and Recommendations
C.
Infrastructure
Issues Related to the Attainment of Math A Standards
D.
Additional Issues
1
Scoring Rubrics, and Communications to the Field
Regarding
2
Calculator Use on
the Math A Exam
E.
The January 2004 Exam, and All Math A Exams Until a New One Is
Designed 43
Appendix A.
Panel's Response to the Nine
Elements of Its Charge
Appendix B. Members of Regents Review Panel for June 2003 Math A
Examination
Appendix C. Documents Reviewed by Math A
Panel
I.
Introduction
Each member of the Math A Panel is very
passionate about the importance of learning mathematics. Each member of the Panel either has
taught, is teaching, or is using mathematics in his/her professional position on
a regular basis. We are all lovers of mathematics, and we want our children
(which we define to be all of New York State's children) to be proficient
in mathematics. Each member of this
Panel applauds the efforts of the Board of Regents and the Commissioner of
Education to provide all children with access to high level
mathematics curricula. We have seen very positive outcomes of these efforts, and
we want to go on record as supporting the continuation of efforts to provide
high quality programs to every child.
In this report, the Panel has been guided by
the nine elements of our charge.
(See Appendix A.) We viewed
the charge as focusing on these broad areas:
*
The Math A standards.
What do we expect our students to know in Math A? Are there improvements the Panel can
recommend?
*
The Math A assessment.
How is the Math A Regents exam created and scored? Are there improvements the Panel can
recommend?
*
The infrastructure.
How prepared are New York State's schools to help every child reach the
Math A standards? Are there
improvements the Panel can recommend?
In our discussions, additional issues were
raised that do not "neatly" fit into the above categories. These are also addressed in this
report.
The Panel felt a tremendous weight of
responsibility. All of its members
are keenly aware of the Math A graduation requirement. This report is not about Math 4 or Math
8, tests to see if children need assistance. This report is not about Math B, the
advanced math test that will be taken by most students heading to college (and
certainly all students contemplating a future in a field requiring a strong
mathematics background). This
report is about Math A, an exam that must be passed before a student can receive
a high school diploma. The estimates with which we were working for our Interim
Report were that approximately 2/3 of the students failed the June 2003 Math A
exam. (These estimates did not
include data from New York City, which were not yet available.) With the adjustment recommended by the
Panel, a scale score of 47 became a passing grade of 65. This adjustment holds this year's
students to the same standard as their counterparts were held in June 2002 yet,
even after this substantial adjustment, the early estimates were that 1/3 of the
students still failed. At our
September 19 meeting, we were provided with the final statewide results. They show that, after the rescaling we
recommended:
- 45% of the State's children failed at the
65 level (33% at the 55 level);
- 59% of New York City's children failed at
the 65 level (47% at the 55 level).
Unless these students pass a Math A exam in
the future, they will not earn a high school diploma, which will render them
ineligible for a wide range of jobs in our society, many of which do not require
high level mathematics skills.
The Panel is also concerned that, even after
the adjustment we recommended, an estimated 16% of the 9th graders
who took the exam failed at the 65 level (11% at the 55 level). Ninth graders who take this exam are
typically considered as strong math students. For one out of six of these students to
fail a Regents examination required for graduation points to a problem that must
be solved.
The weight of responsibility this Panel
feels is about finding balance. On
the one hand, our schools must ensure that our high school graduates have strong
math skills; on the other hand, failing rates such as the ones we have seen with
Math A are, we believe, unacceptable.
The Panel has placed its primary focus on the standards, the assessment,
and the infrastructure. For the
sake of our children, we adults need to get this right.
The Panel supports the Commissioner and the
Board of Regents in the quest to raise standards for all children, and we write
this report in the hope of recommending solutions to the problems the Panel has
identified, so that our State may continue on its path of providing a top
quality education for every child.
Based on the Panel's perusal of math graduation exams from other states,
it is the Panel's opinion that New York State has the highest math standards in
the nation; our recommendations are intended to keep it that
way.
Before Math A, there were two ways students
could meet the math graduation requirement, either by passing the Course I
Regents exam, or by passing the simpler Regents Competency Test (RCT). Math A is a much more challenging
requirement than Course I; it tests more content and it has more problem
solving. It is a challenge to move
students from Course I to Math A. The challenge of moving students from the RCT
to Math A is much greater. Early
on, concerns were raised about the extent of this challenge. An SED report issued in 1998 entitled
"Mathematics Standards and Assessment Review Committee Report" contains the
following caution:
Until (1) the Standards are clearly stated
and communicated to teachers, students, parents and other interested parties;
(2) proper support systems are put in place to give ALL students a fair chance
to meet the Standards; and (3) proper support systems are put in place to
provide ALL teachers with opportunities to develop mathematical content
knowledge and pedagogical strategies, it may be unfair and unrealistic to expect
the passing of the Mathematics A exam to be a requirement for a high school
diploma. (p.
4.)
This Panel has concluded that the standards
are not clear, and that the necessary support systems for students and teachers
are not in place. It is not within
this Panel's charge to discuss graduation requirements; the Panel's work has
focused on changes to Math A to make it more successful.
As noted above, even after a substantial
adjustment recommended by this Panel, 45% of the students failed the June 2003
Math A exam at the 65 level. Such a
failure rate on an adjusted exam points not to a single problem or a few simple
problems; it points to a systemic problem.
This Panel focused on identifying the various facets of this systemic
problem, and on making broad recommendations to the Board of Regents, so these
problems can be addressed, and so we can continue moving forward to raise all of
our students to high levels of math competency.
The Panel spent hours examining pages and
pages of information, graciously provided by SED staff. The Panel spent hours speaking with SED
staff, who patiently put up with our questions day after day. The Panel spent hours discussing what we
were seeing, and then trying to reach consensus on our recommendations. The Panel believes it has identified a
number of areas in which changes need to be made, so that we can continue moving
forward on the path to higher standards for all children.
The Panel has identified 16 findings, and
has developed a total of 22 recommendations, many with several parts, for a
total of 41 recommendations. These
are summarized below.
The Math A Standards
Finding 1:
The Math A standards lack clarity and
specificity (p. 15).
Recommendation 1A: Educationally useful standards must be
developed in mathematics for each grade, K-8, and for Math A and Math B, that
consist of a clear, well-defined set of skills, the mastery of which is
demonstrable (p. 19).
Recommendation 1B: SED should establish a mathematics standards
committee to rewrite the standards into functional form, and to meet regularly
in the future to analyze test results, thus ensuring continuous
relevance (p. 19).
Recommendation 1C: SED should develop and disseminate suggested
curricula for mathematics instruction for each grade K-8, and for Math A and
Math B (p.
19).
Recommendation 1D: To
benefit from the extensive research and deliberation of the current Math A
Panel, some of the current Panel members should be included in both new
committees recommended in this report, i.e., the standards committee, and the
curriculum development committee (p. 20).
Finding 2: The design
concept that the Math A exam should be taken by the typical
student after three semesters of instruction has not been successful
(p.
20).
Recommendation 2: The standards and curricula should be
structured so that the typical student will take the Math A exam after one year
of high school mathematics
(p. 20).
The Math A
Exam
Finding 3: The June 2003 Regents Math A
exam was harder than past Math A exams (p. 25).
Recommendation
3A:
Establish
and maintain narrow statistical targets for difficulty of Parts I, II, III, and
IV of the Math A exam forms (p. 28).
Recommendation 3B:
Review the Math A item
pool (p. 28).
Recommendation 3C:
The
difficulty of problems in the anchor item set, in the
guidance documents provided to teachers, and on the actual tests should be
aligned (p. 28).
Recommendation 3D:
The weighting of the
open-ended items, number of scale points possible on the open-ended item
rubrics, and other aspects of the scoring of open-ended items should be
reconsidered (p. 29).
Recommendation 3E:
The Math A test should focus on a more limited, more
clearly-specified set of content standards and indicators (p. 29).
Finding 4: The Math A tests have not been able to maintain a consistent performance standard over time (p. 29).
Recommendation 4A: Alternative equating designs should be
considered (p. 32).
Recommendation
4B:
Sampling procedures for
estimating item performance must be improved (p. 32).
Recommendation 4C:
Replace the anchor item
set (p. 33).
Recommendation 4D:
Revisit performance standards
(cut scores) (p. 33).
Finding 5: The New York State Education
Department cannot accurately predict performance on Math A test (p. 33).
Recommendation 5A:
SED should implement
procedures for predicting the performance of test forms and groups of students
on future Math A exams (p. 34).
Recommendation 5B:
Policies for field testing and
data collection should be revised (p. 34).
Finding 6: Support
and oversight for the Math A exam program should be improved (p. 34).
Recommendation 6A:
SED should immediately
increase in-house content and technical expertise resources by a minimum of one
psychometrician and two math content specialists (p. 35).
Recommendation 6B:
SED should clarify the responsibilities assigned to
its technical advisory committee, and should request this group to provide regular
reports, including technical analyses, reactions to proposed changes
in test programs, and suggestions for improving State testing programs
(p. 35).
Recommendation 6C:
SED should increase demands
placed on contractors
(p. 36).
Recommendation 6D:
Internal coordination and
documentation should be improved (p. 36).
Infrastructure Issues Related to the
Attainment of Math A Standards
Finding 7: Passing rate data for the State as a whole
were not available until three months after the exam; no data are collected
regarding student performance on individual items, nor even regarding student
performance on the four parts of the exam (p. 37).
Recommendation 7:
SED should increase its data
collection capacity to include item level data, and should accelerate its data
collection timetable (p. 37).
Finding 8: While the
most important use of student performance data is to inform instruction,
statewide data mining models that would enable local schools and teachers to use
these data effectively are not generally available (p. 37).
Recommendation 8:
SED should substantially broaden
its efforts to assist districts in data collection, and the use of data to
inform instruction (p. 37).
Finding 9:
The mathematical background of
teachers delivering math instruction varies widely; yet, raising almost three
million children to higher levels of math achievement will be impossible without
highly skilled teachers (p. 37).
Recommendation 9A:
SED and higher education need to
continue and to strengthen their partnerships to ensure strong teacher education
programs, both pre-service and in-service (p. 37).
Recommendation 9B:
The
certification requirements for elementary teachers and special education
teachers should include a minimum of nine credits of college level mathematics
(see Recommendation 9C), and three credits of teaching techniques in
mathematics (p.
37).
Recommendation 9C: Mathematics courses required for
certification, both for mathematics teachers and elementary and special
education teachers, should be specific not only in terms of number of credits
required to be taken, but also in terms of coursework required to be taken,
e.g., calculus, number theory, algebraic structures, probability and statistics,
etc. (p. 38).
Recommendation 9D: The
Panel believes that, for any teacher responsible for teaching mathematics at any
level, the 175-hour professional development requirement should include specific
mathematics requirements. The
Panel's thinking is that:
·
teachers who teach mathematics exclusively
should be required to take 100 of the 175 hours in the area of
mathematics;
·
secondary teachers who are certified in, and
who teach in, more than one subject area, should be required to take 50 of the
175 hours in the area of mathematics;
·
teachers who teach mathematics as part of a
broad set of teaching responsibilities, e.g., elementary teachers and special
education teachers, should be required to take 30 of the 175 hours in the area
of mathematics.
Additionally, the range of possible courses that would satisfy these requirements should be clearly specified (p. 38).
Finding 10.
The public has very little awareness of Math
A, and may have misunderstandings about the goals of Math A (p. 38).
Recommendation 10: Make greater use of SED communications
capacity to engage the public in conversations about the importance of strong
mathematics skills (p. 38).
Finding 11:
There is often a "disconnect" between
K-12 and higher education (p. 38).
Recommendation 11: SED should encourage conversations at the
local and regional levels of K-12 teachers of mathematics and higher education
professors of mathematics, for the purpose of sharing curriculum, and exploring
professional development opportunities and other possible collaborations, to
bridge the gap between K-12 and higher education (p.
38).
Finding 12:
Raising the level of mathematics achievement
of all students to high levels must start when children are very young, and must
go beyond the school day for school aged children (p. 39).
Recommendation 12: SED should encourage through grants and
other means the expansion of mathematics education initiatives beyond K-12, such
as the creation of partnerships between schools and libraries, and the greater
use of public television and museums (p. 39).
Additional Issues --
Scoring Rubrics, and Communication to the
Field Regarding Grading
Finding 13: The scoring rubrics do not give credit for a
variety of mathematically correct approaches (p. 40).
Recommendation 13A:
Develop more generally worded, holistic
scoring rubrics which permit credit to be granted for atypical, but
mathematically correct, student responses (p. 40).
Recommendation 13B:
Rubrics should be designed so students do not lose 33% or 50% credit for
a minor arithmetic error (p. 40).
Finding 14:
There is a serious "disconnect"
between the perception of the SED content specialists and the perception of
field classroom teachers regarding the application of the scoring rubrics
(p. 40).
Recommendation 14:
On each set of directions for the Math A
exam, a statement should be added confirming that the scoring rubrics are a
guide and should be applied using professional judgment (p.
40).
Finding 15: There needs to be better communication of
SED grading interpretations during the grading process for the Math A exams
(p. 41).
Recommendation 15A:
SED should continue on its path of
setting up a website during Math A Regents exam grading to provide up-to-date
clarifications to teachers grading the exam (p. 41).
Recommendation 15B:
SED should explore ways of sending
up-to-date grading clarifications to the school districts during the grading
period following the administration of the exam, as a backup to the website, to
ensure the greatest possible consistency of grading across the State (p.
41).
Additional Issues --
Calculator Use on the Math A
Exam
Finding 16:
Allowing the option of using a
graphing calculator on the Math A exam provides some students with an advantage
on the exam, thus creating an inequitable situation (p.
41).
Recommendation 16:
The use of calculators on the Math A Regents
exam should be standardized (p. 42).
The January 2004 Exam, and All Math A Exams until
A New One Is Designed
Recommendation 17:
Until the standards are rewritten, new
curricula are developed, the new course is delivered, and a new Math A Regents
is designed and field tested, the Math A Regents exam should be restructured so
the exam includes: 30 Part I items,
5 Part II items, 2 Part III items, and 2 Part IV items (p. 43).
Recommendation 18: The exam should be reviewed by a group of
practitioners, including math teachers, university mathematicians and
mathematics educators, with representatives from this Panel, prior to the
administration of the exam (p. 43).
Recommendation 19: Until new items are developed and properly
field tested, the exam items should be scaled in accord with the procedures used
for the August rescaling of the June 2003 exam
(p. 43).
Recommendation 20: The scaling should not be finalized until
after the exam has been administered and after a post equating procedure has
been implemented to ensure the fairness of the test (p. 43).
Recommendation 21: The 55 passing option on the Math A Regents
Exam for a local diploma should be continued until after the standards have been
clarified, after new curriculum has been developed and disseminated, and after a
new exam has been developed and administered for at least one school year (to
ensure that it is performing in accord with its design)
(p. 44).
Recommendation 22: The math RCT safety net for special
education children should be continued until after the standards have been
clarified, after new curriculum has been developed and disseminated, and after a
new exam has been developed and administered for at least one school year (to
ensure that it is performing in accord with its design) (p. 44).
The Panel believes our recommendations, taken together, will successfully address the problems we have identified in our independent investigation. A suggested timeline for implementation has been developed and is included in the report. (p. 45).
During the 1990s, discussion ensued about raising the standards for mathematics education in New York State. Ultimately, a decision was made to phase out Course I, Course II, Course III and to replace this three-year sequence with Math A and Math B. Conceptually, Math A was to include topics from about a year and a half of the Course I, II, III sequence; and Math B the remainder. A major shift in emphasis was toward more contextual problems and with a greater emphasis on genuine problem solving, i.e., mathematics within a context, where problem-solving strategies can be used. While there was to be a Math A exam and a Math B exam, there was not a curriculum developed. Rather, schools were informed of the math standards, expressed in seven "Key Ideas" which, in turn, were subdivided into 103 "Performance Indicators." Schools were told that they could reach these standards in whatever way they wished but were advised that students would be assessed on these 103 Performance Indicators. Over time, schools worked to develop courses to meet the new standards.
The first Math A exam was administered in
June 1999. For several years, SED
produced both the old and new exams, and schools could offer either one. The last Course I Regents was
administered in January, 2002. It is no longer an option.
During this same time period, the Board of
Regents made a series of policy decisions that resulted in high school
graduation becoming contingent upon the passing of five Regents exams, with a
math exam being one of those exams.
Now, with Course I no longer available, the exam required for graduation
has become Math A. Prior to this
policy change, students could graduate with different types of diplomas. Some
students met the requirement by passing the Math Regents Competency Test (RCT),
a fairly basic test of skills, whereas others met the requirement by passing the
Course I Regents exam.
From the beginning, all knew Math A was a
substantial change, more for some students than for others, but a change for
all. In 1998, a group of math
experts expressed concerns about the difficulty level of Math A. As the exams were phased in, concerns
from the field grew about the difficulty level and the wording of problems. When
the June 2003 Math A exam was administered, the concerns became an outcry. Teachers saw that the test was very
difficult. Early anecdotal evidence
from the schools pointed to a very high failure rate. SED responded by
requesting data from schools. When
the data confirmed a high failure rate, the Commissioner made the decision to
set aside the test for current 11th and 12th graders, and to permit them to
substitute their course grade for the purpose of the graduation
requirement.
Shortly thereafter, this Math A Panel was
created by the Board of Regents and the Commissioner, and asked to respond to a
nine-element charge. (See Appendix
A.)
The Panel dedicated three full days (and
held extensive conversations between meetings) to the first part of its
investigation, which was whether the June 2003 exam was more difficult than
previous exams and, if so, what to recommend as a rescaling to the
Commissioner. The determination was
made that the exam was, in fact, more difficult. In an Interim Report, the Panel
recommended rescaling the June 2003 exam based on the June 2002 results, using
9th grade students as the basis, as the 9th grade groups in both
years were similar. The Panel's
estimate was that this adjustment would raise scores in the middle of the
distribution about ten points. The
Commissioner accepted the Panel's recommendation, and directed SED staff to
implement the adjustment. Within
days, SED generated a new scale for the June 2003 exam; it converted an old 47
to a new 65. According to an SED
press release at the time, the estimates of the impact on passing rates were as
follows:
9th graders:
from
61% to 80% passing
10th graders: from
32% to 64% passing
11th graders: from 28% to 60%
passing
12th graders: from 28% to 55%
passing
The Panel then continued with its work on
the remainder of the elements of the charge. (See Appendix A.) This document is the Panel's final
report to the Commissioner and Board of Regents.
IV.
The
Development of the Math A Exams
Each Math A Regents exam is the result of a multi-year cycle of test development, which results in four actual tests being created each year. Thre