SUMMARY

Issue for Discussion

Should the Regents endorse the proposed growth model for submission to the United States Education Department as required by Chapter 57 of the Laws of 2007?

Reason(s) for Consideration

              Required by State Statute

Proposed Handling

The question will come before the Regents EMSC Committee in July 2008.  Dr. Brian Gong of the Center for Assessment in Dover, New Hampshire will present information to the Committee and respond to questions.

Procedural History

In April 2007, the Legislature passed and the Governor signed into law Chapter 57 of the Laws of 2007, which contains provisions regarding development of growth models. Specifically, Chapter 57 states that: 

“By the start of the 2008-2009 school year, the Regents shall establish, using existing state assessments, an interim, modified accountability system for schools and districts that is based on a growth model, subject to approval of the United States Department of Education where required by federal law.”

Since the enactment of Chapter 57, the Regents have discussed growth models several times:

• At the February 2008 Board of Regents meeting, Dr. Brian Gong presented information to the Regents on growth models.
• On February 28, 2008, another Regents discussion on growth models took place in Syracuse, New York.
• At the March 2008 Regents meeting, an oral report on the Syracuse meeting was provided to the EMSC Committee.
• During the April 2008 Board of Regents meeting, staff presented a discussion item outlining the primary policy questions that the growth model should address; the Board of Regents approved these questions and directed staff to develop the model for the Board of Regents’ review.
• At the June 2008 Board of Regents meeting the EMSC Committee discussed a proposed growth model presented by staff.

Background Information

In November 2005, United States Education Department (USED) Secretary Margaret Spellings announced a pilot growth model project that would permit up to ten states to participate in the pilot.  In December 2007, after nine states had had growth models approved, Secretary Spellings announced that all states would now be permitted to submit growth models for approval.  States that wished to use a growth model based on 2007-08 school year data were required to submit a proposal by February 15, 2008.  (On February 15, 2008, Commissioner Mills wrote to Secretary Spellings informing her of New York State’s intent to submit a Growth Model Proposal to the USED for implementation in the 2008-09 school year.)

New York is committed to having a strong and valid accountability system as a central element of its school improvement strategy.  Recent developments in state statute, federal policy, psychometric theory, and the state assessment and data systems make it now possible to consider inclusion of a growth model to enhance the state’s current accountability system. 

Growth Model and Accountability

The current state accountability system measures school performance primarily in terms of how many students score proficient (level 3 on the state assessment), but does not track individual students’ progress over time.  The proposed growth model would give credit for students who initially scored low but made sufficient progress.  Accounting for growth would provide a more complete portrayal of school performance because it holds schools accountable for helping lower-performing students learn, and gives credit to schools for a student who has made significant progress even if the student has not yet reached the proficient level.

There are many versions of growth models.  The specific growth model proposed for grades 3-8 to the Regents is simple, transparent, and feasible.  It can be easily integrated within the current New York State accountability system.  It shares many features with other states’ growth model proposals that have been approved by the USED for use in their No Child Left Behind accountability systems.  It provides a good foundation on which to build a more elaborate growth- or value-added model.

The proposed growth model for New York State gives a school credit if a student learns enough from one year to the next that the student has grown enough to be “on track” to become proficient within four years or less.  If the student has grown enough, then the student’s performance is worth 2 points in the New York performance index system, which is the same credit as if the student scored at the proficient level.  No points are taken away if the student did not grow enough.  The proposed growth system only adds a second look at students who did not score proficient, and gives them credit if they made significant progress.

Student growth is measured over two years, from grade 3 to grade 4, grade 4 to grade 5, and so on.  Whether the student is on-track to become proficient is determined by projecting the student’s growth into the future and calculating whether at that rate of growth the student would be proficient at the future grade level (e.g., by grade 7).  Because of the different assessment system in high school, student progress is determined by measuring student performance from grade 8 to a Regents examination in mathematics or English language arts.  (See Figure 1.)

Figure 1: Example of Determination of One Student’s Growth from Grade 3 to Grade 4, Projected to Grade 8

Figure 1: This illustrates a student who scores below proficient (red bars) in grades 3 and 4.  Based on the student’s growth from grade 3 to grade 4, the student’s performance is projected to grade 8.  Because the student is projected to become proficient by grade 8, the student is viewed as being on-track to becoming proficient, and the school is given credit for the student’s growth in grade 4.  A similar process would be followed for each student, each year, to calculate growth and make accountability determinations. As explained in Attachment A, the model also includes a provision for adjusting the growth trajectory when a student moves from elementary to middle school.

Context of Proposed 3-8 Growth Model

The proposed growth model is an interim step.  It conforms to what is likely to be approved by the USED under current regulations.  It can be implemented this year (2008-09) if approved.  It can be easily understood by educators, the public, and policymakers as a simple extension of the current system.  It lays credible groundwork for further development of a more sophisticated value-added system called for in the statute.

This growth model features a common ultimate standard—that credit is given only for students scoring proficient or making significant progress enough that they are on track to become proficient within a set amount of time.  By contrast, the New York City progress report system measures performance primarily in relation to whether the percentage of students in a school who are proficient or show what is considered a year’s worth of growth is higher or lower than that of comparable schools and the citywide average performance.

It is likely that adding this specific growth model to the New York school accountability system will have very modest impact on the numbers and types of schools identified.  This is largely because the growth model complies with the federal requirements that sufficient growth is defined as students moving to proficient within four years, which requires a rate of growth that many low-performing students, particularly students with disabilities, do not at present demonstrate.

This interim growth model will allow the field to become familiar with the terminology, data requirements, and additional information provided by a growth model.  It will also allow the Department to extend its data and operational systems to interim growth model calculations and reporting.  The model will allow everyone to gain more experience with the technical qualities and possible uses of this new type of measurement data.  This interim growth model provides a stepping stone toward the next generation of accountability systems, which will include enhanced support and intervention models as well as incorporate the state assessments specified in the upcoming Request for Proposals.

There are two attachments to this item.  The first attachment provides responses to questions that members of the Board of Regents asked at the June 2008 meeting of the EMSC Committee and includes further information regarding high school growth as well as the use of growth in making determinations about high performing and rapidly improving schools.  The second attachment is a general Question & Answer document on Growth Models. 

Recommendation

The Department recommends endorsement of the proposed growth model.  Endorsement by the Regents will allow the Department to apply for peer review and federal approval under the current federal growth model rules consistent with the requirements of Chapter 57 of the Laws of 2007.

Timetable for Implementation

If the Regents endorse the proposed Growth Model, the Department will submit it to the USED so that New York State does not lose the opportunity to have its prospective Growth Model peer reviewed and approved for implementation as required by Chapter 57 of the Laws of 2007.   We will continue discussions with the field about the proposed Growth Model in September and October 2008, seeking more extensive feedback, and will make revisions to the model as appropriate. The Department will also have to modify existing technology to absorb this new accountability structure.  Resources must be made available for that to occur.  If approved by USED, and if all necessary resources are in place, use of the model to make Adequate Yearly Progress decisions based on 2008-09 school year data could begin in September 2009.

Attachments

Attachment A

• Questions from Regents on Growth Models
• How will criteria for “enough growth” be set? 
•  The interim growth model adheres to the requirement established by the United States Department of Education (USED) that schools are held accountable for students becoming proficient within a set amount of time.  Under New York State’s (NYS’s) proposal, a target is established each year so that if a student continues to demonstrate that rate of progress, the student will achieve proficiency within the specified period of time. NYS’s interim growth model does not attempt to predict whether a student who achieves a specified amount of growth will become proficient but rather establishes a trajectory that if met each year will result in a student achieving proficiency within the required timeframe. The NYS Education Department is anxious to extend its research on growth to inform future model development, such as what growth is possible, what growth is typical, and how growth can be measured as precisely as possible.
• How will growth be measured when students enter middle school?
• When a student who is not proficient enrolls in a new school, a new timeline and growth trajectory for that student will be established based upon the student’s performance the prior school year.  Students who enter a middle school in Grades 6, 7, or 8 would need to demonstrate that they are making sufficient growth on the grades 6-8 ELA or mathematics exams so that if this rate of growth continued, the student would be proficient on the Regents Examination in Integrated Algebra and the Regents Comprehensive Examination in English.  A middle school will receive full credit for students who are not proficient but demonstrate such growth. Each year, the student’s target performance is adjusted to reflect the level of growth necessary for the student to remain on track for proficiency. Under the interim growth model, these new trajectories will only apply to schools (and their subgroups), not to district Adequate Yearly Progress (AYP) decisions in instances where students transfer among schools within a district.  This methodology provides greater comparability in terms of the rate of growth that elementary and middle schools must demonstrate in order for their students to be considered on track towards proficiency.  Absent this adjustment, almost all students would be required to be proficient by the end of grade 7, regardless of their grade or performance level upon enrollment in middle school. With this adjustment, middle schools can receive credit for making good progress with students who enter their school as low performers until such time as the students reach proficiency.
• How will growth be measured when students enter high school?
• The interim growth model proposes that:
• A student who enters high school having scored at Level 1 or low-Level 2 on the Grade 8 English language arts (ELA) or Mathematics examinations will be considered on track towards proficiency if the student is able to score between 55-64 on the Regents Examination in Integrated Algebra and Regents Comprehensive Examination in English prior to senior year. For purposes of the Performance Index, schools would receive full credit for the performance of these students.
• For certain limited-English-proficient students, certain students with disabilities, and students who enter high school far below standards, schools will have five years to demonstrate proficiency in ELA and mathematics.
• These two modifications to NYS’s current accountability system will recognize high schools that are successful with newly enrolled low-performing students and will ensure the success of certain students who require an additional year of high school.
•               This proposal uses an existing performance level that is meaningful within the NYS context.  Students must score 65 or above on a set of designated Regents examinations to qualify for a Regents diploma.  Students who score 55-64 may receive a local diploma.  The model gives schools credit for students achieving a score of 55-64, since that represents significant growth from the students’ performance in Grade 8, and represents a rate of growth that would likely result in the student achieving a 65 or higher on the Regents examination if taken at the end of high school.  In proposed rulemaking, USED has recognized that certain students may need more than the standard number of years to meet graduation requirements and schools should receive credit for students who are successful within a specified extended timeframe.
• How does the growth model provide incentives for the continued progress of students who are proficient?
• Staff have proposed tracking the performance of students who score proficient or above to determine whether they continue to be on track to remain proficient or are regressing and on a downward trajectory toward becoming non-proficient over time.  In the latter case, the performance of these students will be recomputed as Level 2 in the Performance Index for the purpose of the Department determining whether a school or district is high performance/rapidly improving.  Schools that wish to maintain a designation as high performing or rapidly improving will need to ensure that their proficient students are not regressing over time.  The Department is also considering adding a criterion for designation as a high performing school or district that would require schools and districts to demonstrate growth above proficiency for a specified percentage of students each year.
• How does the growth model address Students with Disabilities (SWDs) and English Language Learners (ELLs)?
• The performance of SWDs and ELLs is handled the same way as for other students with three exceptions:
• We do not compute a growth trajectory for SWDs who take the New York State Alternate Assessment (NYSAA).
• We do not compute a growth trajectory for ELA for newly arrived ELLs who take the New York State English as Second Language Achievement Test (NYSESLAT) but not the regular ELA state assessment.
• We have proposed giving a fifth year for certain ELLs and SWDs to meet Regents ELA and mathematics requirements.
• Staff do not recommend computing NYSAA growth trajectories because these data portfolios have not been designed to measure growth over time. Staff do not recommend seeking to use the NYSESLAT as the basis for measuring growth on the subsequent year’s ELA exam because the USED has previously prohibited NYS from using the NYSESLAT as a student performance measure for ELA.
• As described above, only a very small percentage of SWDs and ELLs will not be included in the growth model.  In fact, since the SWD and ELL subgroups are most likely to score below proficiency and are the groups that most frequently cause a school to fail to make AYP, the growth model is of most benefit to schools and districts that serve high percentages of these students.
• Could the State simply adopt the New York City growth model rather than develop a different one?
• The New York City progress report system was designed to support the New York City Department of Education’s Children First initiative. The progress report system was not designed to be No Child Left Behind (NCLB) compliant or to serve as the basis for making AYP decisions under NCLB. While the New York City system incorporates value added components that provide useful contextual information about school performance, NCLB currently explicitly prohibits the use of these components in making AYP decisions.
• How accurate are these growth trajectories in terms of their ability to be used to make determinations about individual student growth and teacher and school performance?
• At the individual student and teacher level, growth trajectories have a large standard error of measurement and need to be used with extreme caution. The power of the model comes from the aggregation of data regarding the performance of students. The larger the number of students measured, the more confident we can be in our judgments regarding the growth trends of the particular group of students.
• How can schools and districts use growth data?
• The interim growth model represents a first step in a process that could, over time, have significant implications for instruction, technical support and interventions, and accountability.  Initially, the benefits of a growth model are likely to be modest. The primary benefit is that the growth model will allow the Department to make better judgments regarding which schools make AYP and which schools are designated as high performing/rapidly improving. The model will also assist the Department and districts to redirect resources to focus more on those schools that are both low status and low growth, and to study as possible best practice models of those schools that are high growth. Districts can also use the data to provide differentiated supports for student subgroups that are either showing or not showing patterns of growth. In future years as we accumulate more data, growth models should become ever more useful. For example, a growth model may allow districts to better predict which of two students with the same scale score on the Grade 8 ELA examination need more intensive support based on how the students performed in prior years on State assessments.
• How does the growth model address the Regents desire to simplify the current New York State accountability system?
•  
• These issues are to some degree separate.  The interim growth model is primarily a methodology to make better AYP decisions. However, once the AYP determinations are made, revised categories of school and district accountability status will make the current accountability system more transparent and allow the Department to refine the supports and interventions provided to schools and districts. 
• Attachment B 
• Frequently Asked Questions About the
• NYSED’s “Growth Model”
•  
• Is the NYSED required to develop a “growth model” for school accountability? NYSED is complying with State law, Chapter 57 of the Laws of 2007, that requires the State to implement an interim growth model—if that model is approved by the USED—by 2008-09 and a more sophisticated “value-added model” by 2010-11, if such model is approved by USED and a panel of external experts in testing and measurement.
• Is a growth model being implemented only because State law requires it? No.  A “growth model” would provide schools credit for students who are making progress towards becoming proficient.  By basing accountability on the percentage of students who are proficient plus the percentage of students who are demonstrating that they are on track to become proficient, NYSED can make better, more nuanced judgments about school performance.
• What is “growth” and how is it different from what schools currently are being held accountable for? Currently, schools and districts are held accountable for the percentage of students who are proficient or partially proficient at a given point in time. The current “status” model makes schools and districts responsible for demonstrating that an increasing percentage of students are proficient over time. Under a growth model, when students are not proficient, schools and districts are expected to show that a student is making sufficient annual progress to become proficient within a specified period of time. A status model typically measures the performance of the same grades, but different students, over time, while a growth model measures the performance of the same students over time.
• What will the growth model look like? The growth model will follow the guidelines established by the USED for its Growth Model Pilot and subsequent proposed regulations.  In essence, schools will get credit for having students who are proficient and for students who have grown enough that they are on-track to become proficient within a set time.  NYSED is committed to using a transparent system that is accessible enough that parents, schools, and districts should be able to perform the accountability calculations themselves.
• What will students, schools, and districts need to do to participate in the State’s growth model?  The addition of growth will not cause any additional work for students, schools, or districts, if they are complying with current state policies regarding assessment and data integrity.  The NYSED will calculate growth using existing longitudinal data.
• When will the growth model take effect?  The growth model will take effect as soon as it receives necessary approvals from the NYS Board of Regents and the USED and will require additional resources to modify existing information reporting systems.  The statutory intent is to have the interim growth model take effect for school accountability determinations that incorporate student assessment data from the 2008-09 school year, if NYSED is successful in securing the approval of USED and if the necessary resources to implement the growth model are provided.
• What will be the effect of adding the interim growth model on school accountability?  The NYSED interim growth model is designed to give schools and districts credit for students who have not yet scored proficient on the State’s assessments, if they have made sufficient progress that they are on-track to becoming proficient within a set amount of time.  Thus, the interim growth model will not cause any school or district that would have made AYP under the current system to fail to make AYP because of the inclusion of the growth component.  The number and types of districts and schools that will get credit for making AYP because of the inclusion of the growth model in the AYP determination process will be determined by students’ performances in 2008-09.  Experience from other states with these types of growth models and preliminary empirical modeling of available data indicate that the impact on schools and districts initially will be quite modest.  However, it is likely that the ability to include students who are on track to proficiency in AYP calculations will be of increasing benefit to schools and districts as the Annual Measurable Objective increases each year.
• What will be the benefits for instruction and program evaluation of adding the interim growth model?  Parents, students, schools, districts, and others will be able to see individual and/or aggregated student growth on a yearly basis from the state’s assessments.  Like other large-scale assessments, the growth data provide useful information from credible assessments at a point in time.  As with any large-scale assessment, growth data probably will need to be supplemented with more diagnostic and immediate assessment results gathered at the local level.
• How will NYSED support schools and districts using the growth data in a valid way for professional development and instruction?  NYSED will provide thorough documentation of the growth model to schools and districts, along with regional training regarding the growth model and data.  Districts and schools will need to customize the interpretation and application of the growth results to their individual programs and circumstances.
• How is the State’s growth model similar to or different from other growth models?  There are many possible growth models that are designed to serve different purposes.  The NYSED interim growth model is designed to support the Board of Regents policy to close the achievement gap and improve student performance.  It gives schools and districts credit for students making sufficient progress towards becoming proficient and meeting the NYS learning standards.   The NYS Grades 3-8 growth model methodology has many similarities to the North Carolina model that has been approved by USED.  The high school growth model employs a value tables approach that has been used by states such as Delaware and also been approved by USED. However, at both the elementary-middle and high school levels, the models contain unique features specifically designed to make the model the most appropriate one for use by NYS. Other prominent growth models are designed for other purposes.  For example, the New York City growth model compares schools primarily against like schools within New York City, and the Sander’s value-added model (Education Value Added Assessment System) used by some districts in the State is designed primarily to attribute growth to particular teachers and schools.
• What will the State’s more sophisticated “value-added” model look like?  The current growth model under development is intentionally an interim model.  The state’s goal is to develop the best available growth model with additional features, as required by the State law, by 2010-11.  The more advanced model will draw on advances in data infrastructure, statistical models, and be linked to appropriate support and intervention models.
• How will NYSED ensure there will be a smooth transition from a status model to an interim growth model and from a growth model to a value-added model?  NYSED is working to ensure a smooth transition by making the interim growth model as clear and straight-forward as possible, with as close an integration as possible with the existing assessment and accountability systems.  NYSED is aware of the desirability for a smooth transition to the longer-term “value-added” model, and is keeping that consideration salient in all of its considerations.
• How will the growth data be reported to schools and districts?  NYSED will report growth results as it does for other assessment results.  Schools and districts will receive growth results in combination with the current year data at the individual student, classroom, school, district, and state levels.  Schools and districts may choose to disclose individual or disaggregated results to parents.  The report forms that will be used are currently being designed.
• How will criteria for “enough growth” be set?   The interim growth model follows the requirement established by the USED that schools are held accountable for students becoming proficient within a set amount of time.  The NYSED interim growth model complies with that requirement while advancing an accountability design that is more meaningful for middle and high schools.  NYSED is anxious to extend its research on growth to inform future model development, such as what growth is possible, what growth is typical, and how growth can be measured as precisely as possible.
• Are the state assessments technically adequate for this purpose?  Yes.  NYSED has undertaken a series of research analyses, both in-house and with consultants, to review the technical adequacy of the current assessment system in regards to issues such as scale sensitivity and uniformity, the basis for cross-grade comparisons of student performance, the comparability of grade-level achievement levels, the adequacy of assessments for students with severe cognitive disabilities and students with limited-English-proficiency, and so on.  Based on this analysis, the current State assessment system can be validly and reliably used to support the growth models that have been proposed.
• How will any revisions to the content standards be coordinated with creating interim growth and the longer-term value-added accountability measures?  As the content standards are revised, State assessments, upon which the interim and enhanced growth models are based, will also be revised as needed to measure student achievement of the new content standards.
•