You are currently browsing the category archive for the ‘Evaluation’ category.

Dont-Rock-The-Boat-600x400

In a previous post, I mentioned that I had the opportunity to visit with the HB 1202 committee to discuss assessment.  I followed Grant Guyer, Denver’s Executive Director of Assessment, as well as representatives from Harrison District 2.

So I could get a feel for the HB 1202 group, I arrived early to listen in on the conversation.  I was impressed with the learning and reflective stance I heard the committee members take.  Rather than asserting or defending positions, the committee members were (for the most part) asking really good questions and thinking together.

The contrast of the thoughtful and open approach that the committee had in comparison with advocacy oriented approach Denver took was jarring, at least to me.  DPS came in with a clear agenda: influence the committee to (basically) preserve the status quo when it came to state accountability testing.

Because DPS chose to take such a forceful position, I feel it is appropriate that position be critiqued and vetted in public format so that their thinking can be considered and fully vetted.  Clearly, DPS’s intent was to influence public policy in a strong way.  As this policy impacts every public and charter school in Colorado, examining their claims and thinking is important.

The overarching DPS position is that they (the administration at least) do not support “specific aspects of the shift to minimum federal (assessment) requirements, primarily due to the impact on high schools.”

I’ve attached the report that Grant gave here (DPS Assessment) so readers can review it for themselves (apologies for my scribbles on the scan).  However, here are some of their claims and my critique:

Claim #1 – “Standards implementation could be jeopardized as there would not be a consistent, well-constructed assessment to measure of (sic) student performance at the end of a given grade/course.”

What evidence exists to support the claim that standards implementation would be jeopardized if there were no standardized, summative assessments at the end of each grade?  Some of the best performing education systems in the world do not test core subjects at the end of each grade, yet they seem to be able to consistently teach to high standards.  Further, what evidence or assurances do we have that a machine scored, large-scale, summative assessment is necessary in order for a classroom teacher to teach to high standards?  If we are to subject literally hundreds of thousands of Colorado students to an assessment (spending millions in taxpayer dollars to do so), should not the purpose and impact of that assessment be well understood and proven?

Claim #2 – “This would reduce the amount of formal data available to accurately identify where shifts in instruction are needed.” 

Large scale, machine scored summative tests are woefully inadequate for the purpose of “shifting” instruction.  Primarily, these tests are for accountability purposes and not for guiding formative instructional practices.  This is not to criticize the tests themselves – but they were not primarily designed for this purpose.  The thinking that summative TCAP, CMAS or PARCC test results will result in effective and responsive classroom level shifts in instruction is hopeful theory with a vacuous evidence base.

Claims #3 & #4 – “Less information available to track student progress toward college and career readiness,” & “Less information available for families to make informed decisions about which high schools are the best options for their children.”

The DPS position assumes (wrongly) that an assessment system at federal minimums (or even fewer assessments) would be devoid of student assessment information in those areas where there is no mandated accountability exam.  Clearly, DPS’s approach to improvement is founded on test-based accountability and school choice.  In theory, for those two approaches to work you need assessments to shame and punish and big data to create a more perfect school choice “market.”  Nothing would preclude DPS from heaping all the assessments they want on students to feed their theory of change.  However, if we did not mandate such measures we would not be forcing every other school district in the state to follow DPS’s logic model.

Claim #5 – “Eliminating these data points at the high school level could shift the accountability system to focus too much on status.  This distinctly disadvantages urban districts that have students with low levels of preparedness.”

DPS assumes (wrongly) that whatever growth, accountability, and accreditation system we currently have in place would just continue but without some high school assessments. The current accountability framework was designed with one set of assumptions about available test data.  In a world with fewer accountability tests, a different model would need to be designed.  This different model could conceptualize growth in a number of different ways and could also recognize student poverty demographics and “preparedness” in different ways and it should.  Here, DPS just wrongly assumes we would continue the same system we’ve been operating.  Further, the report states that “DPS strongly values growth data.”  That’s great!  But, if this is indeed true, there is no basis to believe DPS could not continue to assess and measure growth without having a mandated state test in place.  In fact, dollars currently used for large scale assessments could be provided directly to districts for the very purpose of locally determined measures and analysis.

Claim #6 – “Less external data available to assess student growth for teacher evaluation.”

Besides there being no credible, peer-reviewed evidence that using student testing data to evaluate teachers actually improves instruction and the fact that no high performing system on earth uses this approach, the DPS claim is also flawed. As has been previously discussed, if DPS wishes to have machine scored, large scale assessment data to evaluate its teachers there is no prohibition from them doing just that.  The DPS claim seems to infer that without this standardized testing data, our state-wide effort to evaluate teachers using assessment data is in peril – but we already have some 70% of teachers in untested subjects and grades.  It is not clear (at least to me) that the presence or absence of summative statewide assessment data does much in helping us solve the significant technical questions related to using testing data to evaluate teachers.

Claim #7 – “…districts would have to take on the additional burden of creating/purchasing products to ensure that schools are meeting student learning expectations (and) the development of local growth measures to assess the performance of schools and teachers.”

As has been previously discussed, dollars currently appropriated for state level accountability assessments could, at some level, be re-purposed to districts for locally determined and more formative measures so its not clear that there would be an additional burden.  Further, there are a number of growth measures available for districts to use (student growth percentiles, value-added measures, catch-up/keep-up systems) so it also not clear that a district would need to “develop” these measures.

Conclusions

Again, DPS is following a theory of change for improving their organization built on test-based accountability and school choice.  While refraining from a critique of these two approaches to school improvement, I will just say that these are not the only two methods by which a system might build great schools.  In fact, the best performing school systems (based on PISA results or equating studies) were not built using these models.

Regardless, it is up to the community of Denver to decide which model is most appropriate for their community and then hold their school leaders accountable for the results.

The larger problem with DPS’s jarring advocacy stance with the HB 1202 committee is that it effectively forces that theory of change on every other school organization in the state – whether we want it, or if there is any evidence to support it, or not.

Of note, in the course of these discussions I have heard no one arguing for the complete abolition of testing and accountability.  The better question is how we can have an accountability system that is as efficient and balanced as possible, without over-burdening students and schools with testing.  A review the testing approaches in high performing global systems reveals that such a system can be effectively implemented with far fewer tests than we currently use in Colorado.

I encourage further dialogue and discussion on this issue and welcome a response from Grant Guyer (a very nice person, based on my brief interaction with him) or others from DPS. For convenience, I have also posted my presentation materials to the HB 1202 committee for a similar critique, if anyone feels so inclined.

I was recently honored with the brief opportunity to speak to Colorado’s HB1202 Task Force, which is studying the state’s assessment system and responsible for suggesting changes to the Colorado Legislature for consideration in the upcoming legislative session.

I focused my remarks on the importance, process, and evidence on formative measures.  I also spoke to the differences between accountability assessments in the United States (and Colorado) versus other high performing nations or municipalities.

The memo I prepared for the group can be accessed here: HB1202 ECS Flyover

The entire text is also provided below.  I welcome observations, comments, questions, or critique.

Memorandum

From:  Jason E. Glass, Superintendent & Chief Learner

To:      HB 14-1202 Task Force

Re:      Formative Assessment & a Flyover of Assessment in Eagle County

Date:   9.15.2014

Purpose

The purpose of this memorandum is to briefly orient the members of the HB14-1202 Task Force to the large-scale theory of change, an instructionally focused approach to assessment, and some of the formative measures employed in Eagle County Schools.  For clarity, this memo will focus on measures whose chief purpose is for improving instruction, as opposed to measures whose chief purpose is accountability.

The Instructional Core

Eagle County Schools uses an “international benchmarking” approach to school improvement.  That is, practices are drawn from comparative studies of high performing education systems, both within the United States and abroad.  In addition, the organization focuses on practices which have the support of a peer-reviewed body of evidence.

As such, the “in-school” theory of change rests on three major and interrelated tenets which feature prominently in every high-performing educational system.  Liz City and Richard Elmore (2009) capture these three elements in their discussions of the “instructional core,” or the relationship between the teacher and student in the presence of content.

Instructional Core

Important to City and Elmore’s framework, there is an emphasis on the relationship between the three components.  One element cannot change without impacting the other two.  For example, we cannot effectively raise the quality or “rigor” of the content (or standards) without also adapting the instructional approach of the teacher and the engagement level of the student.

Assessment through the Lens of Instruction

Formative measurement is an essential part of bringing the instructional core to life.  For the teacher to effectively reach and engage every student in learning, that teacher must understand the level of current content performance or knowledge of their students.  The teacher must deliver high quality instruction and then determine if that instruction had the desired impact on students (i.e. improved content knowledge or skills).  Almost invariably, some students will require additional supports or a differentiated approach to reach the content or skill standard.  So, the teacher must apply some intervention, customized to the student, and then check again to see if that intervention had the effect of raising the student to the performance standard.

The “response to intervention” or “response to instruction” (RtI) model provides a useful framework for understanding this process.

RtI

Well designed and employed formative assessments are ‘part and parcel’ to the RtI process.  All students should receive a universal screen or benchmark assessment as part of the general education curriculum.  As there may be some time (days, weeks, or months) between the administrations of these assessments, they can be referred to as long cycle.

These long cycle results will reveal some students who struggle to meet the standard in the general education environment, who should then receive some intervention customized to that student’s needs.  Determining the appropriate intervention often requires the use of a diagnostic test to determine the precise area where the student is struggling (ex. phonics vs. phonemic awareness).  Then, once an intervention is applied, the determination as to if the intervention is working should be made through a progress monitoring assessment.  As the time between these assessments is less than at the universal level, they are sometimes called medium cycle assessments and may be administered every few learning sessions or weeks (or longer, as the team of practitioners determine).

Even after a targeted intervention, some students will require an intensive support.  These students will receive diagnostic and progress monitoring even more frequently – perhaps multiple times over the course of the lesson as the teacher iterates to determine what is the barrier to learning and if it is being mitigated through supports or other interventions.

The RtI approach is based on the principles of a “high reliability system” (see Eck et al., 2011), meaning generally that as the probability of failure increases then supports/interventions and monitoring also increases.  The goal is to determine which students are struggling and why as quickly as possible and to intervene so that the student meets the performance standard.

Notably, formative assessments may be more standardized and formal or they may be individualized and informal.  A powerful mode of formative assessment is a teacher walking through a room as students work, asking questions and checking for understanding.  Alternatively, formative assessment may involve sophisticated and computer-based standardized measures.  Variations in formative assessments may stem from variations in the elements of the instructional core (different teachers, different students, and different content) or from constraints related to things like time and technology.  This entire process may happen in a very structured and mechanical way, or it may happen much more naturally and intuitively.  What is most important is that it is, in fact, happening.

It should also be noted that the formative assessment process is not exclusive to the teacher.  Perhaps the most powerful mode of formative assessment is for the student to self-monitor and assess their own progress.

Evidence and Formative Assessments

The body of both comparative and peer-reviewed scientific evidence for the effectiveness of formative assessment is (in my professional opinion) strong.

Black and William (1998), in a meta-analysis, found that student achievement gains associated with formative instructional practices were “among the largest ever reported for educational interventions.”

Similarly, Hattie (2011), also in a meta-analysis of over 50,000 studies, identified strategies related to formative assessment and RtI among the largest effect sizes calculated.

From a comparative system perspective, formative assessment and responsive teaching form the instructional basis of practically every high performing education system.  Finland, a system perhaps more averse to summative accountability testing than any other in the world, uses formative assessment extensively.  In Schwartz & Mehta’s chapter on Finland in Tucker’s comparative study Surpassing Shanghai, it is noted that “While the Finns do not assess for accountability purposes, they do an enormous amount of diagnostic or formative assessment at the classroom level.”

Notably, when a Finnish principal was asked (in Schwartz & Mehta) how well she knew students were performing, she answered that there was so much formative assessment data at her disposal it was impossible not to know.

Formative Assessments in Eagle County Schools

Eagle County Schools relies on a number of formative measures to guide instruction.  Choice over the appropriate use of these formative measures is left to the building practitioners, including the building principal, teacher leaders, and classroom teachers.

Depending on grade/developmental level, student characteristics, staff preferences, content area, or specific purpose – the following is an incomplete list of formative assessments used in Eagle County.

  • Early Childhood & Elementary
    • GOLD Assessment
    • mCLASS (DIBELS Next/IDEL)
    • AIMS Web
    • Core Knowledge Language Arts
    • Engage New York, Literacy & Math (Achieve)
    • District Formative Measures (ECS Teacher Developed)
    • Classroom grades (standards based)
  • Middle School
    • mCLASS (DIBELS Next/IDEL)
    • Renaissance STAR
    • NWEA MAPS
    • Engage New York, Literacy & Math (Achieve)
    • District Formative Measures (ECS Teacher Developed)
    • Classroom grades
  • High School
    • NWEA MAPS
    • District Formative Measures (ECS Teacher Developed)
    • Classroom grades

Conclusion

Eagle County Schools is, admittedly, not yet a globally high performing system.  But, we are in our first year of building an instructionally focused assessment system patterned after global high performers.  As such, formative assessment is central part of that effort.

References

Black, P., & William, D. (1998).  Inside the Black Box: Raising Standards Through Classroom Assessment.  Phi Delta Kappan, 80, 139-148.

City, E., Elmore, R., Fierman, S., & Teitel, L. (2009).  Instructional Rounds in Education: A Network Approach to Improving Teaching and Learning. Cambridge, MA: Harvard Education Press.

Eck, J., Bellamy, G., Schaffer, E., Stringfield, S., Reynolds, D. (2011).  High Reliability Organizations in Education.  Noteworthy Perspectives, 1-48.

Hattie, J. (2011).  Visible Learning for Teachers: Maximizing Impact on Learning.  New York, NY: Routledge.

Tucker, M. (2011).  Surpassing Shanghai:  An Agenda for American Education Built on the World’s Leading School Systems.  Cambridge, MA: Harvard Education Press.

 NCEE Infographic

 Info-graphic from the National Center on Education & the Economy

Hot Air Balloon Festival Buena Vista from ColoradoGuy.com

Colorado’s recently released TCAP results landed across the state with a soft thud.  Overall, scores were flat or down in most subjects and grades.  Even among charter schools, the ballyhooed darlings of the reform movement, results leaned toward the disappointing accented by wild fluctuation.

Reactions from pundits, state education leaders and the state’s largest newspaper, the Denver Post, ranged from somber to puzzled, but ideas about next steps quickly emerged: stay the course or even accelerate the reforms Colorado has been aggressively pursuing.  Namely, that the state should continue with the hyper-accountability (more tests and consequences, even considering extending some form of accountability to the children) or market-based approaches (more charter schools or even expanding to private school voucher schemes).

What is most troubling about the reactions of our state leaders and resident non-profit policy wonks is how completely disconnected their reactions and proposed solutions are from what is really happening in schools across our state. 

How quickly we have forgotten that Colorado has cut education funding by over a billion dollars annually for the past four years.  In many schools, resources went in reverse nearly 20%, resulting in massive layoffs, pay freezes, and the loss of essential school resources like curricular materials and instructional supports for the state’s neediest kids.

All across the Centennial state, our teachers and principals were and are working to achieve more with less.  If any of the so-called or self-proclaimed experts had thought to descend from on high and ask a classroom teacher, then the answers to flat TCAP scores would have been plainly clear.

In spite of this historic gutting of public education in Colorado, our educators – for the most part – held the line on statewide student achievement results.  But instead of standing up for those who stood in the breach for our kids, Colorado’s educators received more blame and shame, more disruption and disparagement.

As our schools struggle to piece together and implement the blizzard of disconnected, often unfunded, and frequently nonsensical state reforms, we should ask:  is it rational to expect any endeavor to become more complex and to produce better outcomes while the means of production are financially devastated?

Yet our state’s “no-excuses” leaders turn on their reality distortion fields and wonder why statewide scores are flat.  Why aren’t our testing, evaluation, and market reforms – that brought such national attention and recognition to Colorado – working as planned?

The answer, quite simply, is that they’ve never worked anywhere at scale and the body of evidence to support these approaches is scientifically anemic and ideologically biased.

There are no high performing education systems in the United States, or anywhere in the world for that matter, that have achieved systemic and sustained greatness through the means Colorado now aggressively pursues.

Instead of working to de-professionalize education by cutting teacher wages, vilifying unions, and allowing practically anyone who isn’t a felon to become a teacher – the high performing systems have worked to make education a high status and very selective profession.  There are no stories of mass shaming, firing, and disenfranchisement among those systems that have actually achieved sustainable greatness.

The best performing education systems on earth aren’t having discussions about opening more charter schools because they don’t have any.  This is not to say we should eliminate Colorado’s charter schools -many of them do a fine job.  It is to say that the work of genuine greatness requires extraordinary effort and execution put behind proven practices.  Handing over the management of public education to some non-profit entity and calling it a charter school does not, by this action alone, make the education better and does not further the goal of system-wide genuine quality.

The best education systems on earth also aren’t discussing the privatization of their schools through voucher schemes.  This is because they are focused on supporting and continuing to make their public schools even greater – instead of intentionally dismantling and disrupting them.

The best education systems are also judicious in their use of assessments.  They test only at key transition points, relying on practitioner developed assessments that measure high level skills and concepts.  Here in Colorado, our kids must take literally dozens of standardized tests over the course of their academic careers. Yet we can’t seem to let go of a single test because the theory of test-rank-punish as a means of improvement is far too ingrained.

Parents ask, “Why are we testing my child from February to May instead of teaching them?” Assessments are important; especially those that help educators tailor instruction to help kids learn.  But the parents and the kids know – standardized testing is not the same thing as learning. 

The problem with years of TCAP staleness starts and ends with the foisting of disconnected state-level reforms that have no basis in evidence.  State-level policies that ignore and supersede the intricate art and science of instruction are too broad and generic to work, resulting in the unintended consequences of overloading schools with rules and regulations handed down without any funding to offset their administrative costs.

The Denver Post’s editorial about Colorado’s TCAP scores ended with a plea to continue the path our state is already on in terms of accountability and market-based approaches.  According to the Post, we need to get these reforms fully implemented and give them time to work.

In the end, I expect the editorial board at the Post will get their wish.  Colorado probably has too much ego, political capital, and careerism invested in these policies to change course now.  But we should also expect many years of future editorials – all with an eerily familiar lament – wondering why, systemically, things just aren’t working out as planned.

bellwether

 

Yesterday, Bellwether Education Partners, “a national non-profit dedicated to helping educational organizations,” released a new report entitled “Genuine Progress, Greater Challenges: A Decade of Teacher Effectiveness Reforms” by Andrew J. Rotherham and Ashley LiBetti Mitchel.  The report can be accessed here.

This report is national in scope, but popped up on my radar when Colorado’s Donnell Kay Foundation tweeted out a link to the report under the Colorado education policy hashtag “#edcolo,” which I review for state news on a regular basis.

After reviewing the report, I added some (admittedly cheeky) commentary on the report via Twitter.  Specifically, I criticized it as pseudo research parading as empirical evidence.  I also noted that no high performing education system has achieved greatness pursuing the strategies recommended in the report.

One of the authors of the report, Andy Rotherham (a known national education policy wonk), replied to my tweet, stating “When U actually read report & engage w/ what’s in it (rather than playing to crowd) we’ll be here @COJasonGlass @bellwethered.”  Almost immediately, Donnell Kay (or whoever handles their account) favorited the tweet and Andy Smarick (another national policy wonk on education reform and a partner at Bellwether) retweeted it. *Profuse apologies for those unfamiliar with “twitter-speak!”*

Given that the report itself is a re-cycle and re-hashing of the same usual suspects and policy positions when it comes to educator quality and all these individuals/groups have an extreme propensity for citing one another’s writings and hyping each other up, I find the accusation that I’m the one “playing to the crowd” downright amusing!

But,  I digress.

Andy did have the courtesy to send me a very respectfully worded email, asking if we could talk about the report and the issues therein and suggested that there was room for common ground.  I sincerely appreciate the civility and spirit of that message and I do think Andy is a quality writer and good thinker.  My critique of the report is in no way personal toward him or his co-author.

I do think that Andy deserves a more full explanation of my concerns with his report and I do apologize for the abruptness and lack of depth in my tweets on this matter – thus is the inherent drawback of using Twitter for complex conversation!

Rather than respond privately to Andy via email, I am choosing to critique his report via this public forum.  The reason for this is that Andy and the Bellwether Foundation have put forth this document in the public realm, ostensibly with the goal of influencing public policy when it comes to educator quality.  As such, a critique of the report also belongs in the public realm.

So, in the spirit of respectful public dialogue and a commitment to a free-market of ideas (which I am sure Andy equally supports), below is my critique of the Bellwether report.

One last thing before I begin – I apologize for the free flowing form of my thoughts in the writing below.  I am a working Superintendent and father and my time is precious.  Forgive me if this lacks the flow and organization of a more professional piece.

AREAS of SUBSTANTIAL AGREEMENT

Instructional quality is of great importance – the Bellwether report makes this statement early and prominently and I could not agree more.

Teachers matter a great deal to student outcomes – with the qualification of “within school factors,” I strongly agree with this statement.  The Bellwether report does acknowledge this qualification.  To be more clear, outside of school factors actually matter more when it comes to student outcomes.  This is not noted as an “excuse” for why our system of education cannot and should not be better, it is noted to say that one cannot reasonably expect to systemically and at-scale improve student outcomes if one ignores the out of school factors.

The industrial union model has been, to a degree, a detriment to the teaching “profession” – While teachers’ unions adopted an industrial and confrontational approach to bargaining for good reason (low wages, discriminatory practices, inhuman working conditions) and have historically gained in these areas as a result, holding on to this model in today’s era is a detriment.  Unions must evolve to be guardians of of quality and of the profession.  In my professional opinion and to the credit of unions, this transformation is underway in the United States – but it has been and continues to be a process.

Educator quality has a long and interesting history – The report notes that efforts to improve educator quality through mechanisms such as licensure and efforts to define “highly qualified”  have been underway for several years.  I would also add educator preparation program accreditation and prospective teacher testing as other levers, which are touched on in the report – if only briefly.

Pension reform is necessary – To which I would add two qualifications.  First, this is not true in all states.  Some states have over-promised and mis-managed their pension systems and created massive unfunded liabilities.  However, other states have been conservative and pragmatic with their systems and they are quite sustainable.  Second, we must be cautious about the motives and plans of those wishing to reform pension systems.  While there are some who genuinely wish to shift the funds to public employees in the form of defined contribution plans and increase direct compensation, there are others who wish to “reform” pensions as a back-door way of de-funding public education and intentionally harming public servants.  Similarly, we must also be suspicious of the motives of Wall Street firms who wish to destroy and privatize pensions so as to create opportunities for profiteering.

Personalize professional development – While I take a bit of exception that this must be in some way hitched to evaluation, to the degree that we empower and provide autonomy to our front line educators to determine and customize professional learning to their context and needs, we are in agreement.

Focus on recruitment – The best performing education systems in the world are damned selective about whom they allow to enter the teaching profession. Generally, this is accomplished through a combination of raising the prestige of the profession, raising the initial compensation levels, and treating the profession with reverence and respect.  If the United States approached the teaching profession in the same way many high performing global systems do (and the way the best performing systems in the U.S. historically have), the thinking that we need to rank and fire people would diminish tremendously.

AREAS of SUBSTANTIAL DISAGREEMENT

A one sided historical narrative – The report attempts to tell the story of educator quality in the United States.  While this is indeed a worthwhile and interesting topic (at least in my judgment!), the report relies on a tired narrative of unions and comatose school administrators as the villians and education reform groups and their “get tough” leaders as heroes.   How can anyone expect a historical review of educator quality to be taken seriously as a scholarly piece without even a mention of John Dewey?

Unions are the problem – As previously mentioned, this story needs a villain and teachers’ unions serve that role in this report.  However, the highest performing education systems on Earth are (for the most part) highly unionized.  In these systems, unions serve as professional guilds and important partners for educator quality.  Using this report as yet another frontal attack on unions does not help us make the transition to that professional and collegial model.  Using the lens of international benchmarking to best systems, dismantling and disenfranchising the union does not seem to be in the playbook.

Evaluation is a mechanism for improving educator quality – This report repeatedly leaps to the conclusion that improving evaluation systems will improve teaching and improve student outcomes.  This causal link has no empirical basis and giants from the field of business management (notably Deming and Herzberg) have been telling us for decades the practice is an ineffective means of improvement.  Yet, the education reform movement has swallowed whole this approach of evaluate/rank/punish as a mechanism for improvement and now we have national education policy build on this unproven and potentially detrimental assumption.

Achievement gaps exist; and teachers are the answer – As discussed previously, teachers are really, really important and on this point we agree.  Yet, by this report’s citations teacher effects account for 7% or 8% of the variance when it comes to student outcomes.  Much of this variance, we know, comes from societal issues relating to student poverty.  Any systemic effort aimed at closing the achievement gap must include a commensurate systemic effort at mitigating the effects of poverty on learning.

“The last few years have produced real progress on teacher effectiveness and more generally in American schools…” – This statement comes directly out of the report and makes the classic logical fallacy of “post hoc, ergo proptor hoc.” More simply, Y followed X, so Y must have been caused by X.  In spite of the constant attacks and shaming of the American education system in an effort to beat the drum of reform, American schools are better performing now than ever and achievement gaps are narrower than ever.  To make any sort of claim that this improvement (which has been underway since the 1960’s) is the result of relatively recent “educator effectiveness” reforms is bogus.  While often maligned as  unresponsive and overly bureaucratic, the American education system has actually been very adaptive to the shifting demands our society has placed on public education.  Rather than a system which has been resistant to change, the American education system has been very successful at meeting change.  See Clayton Christensen’s Disrupting Class for a lengthy discussion on this point.

Removing ineffective educators is the key to large scale improvement – I am unaware of any organization or system, public or private, which achieved systemic and sustained greatness via the creation of large scale, complex and Rube Goldberg-ish attempts to rank and fire employees. Even in the so-called cut-throat world of American business, firing people is a relatively rare occurrence.  Focusing on firing people is more likely to create alienation and fear in an organization than large scale improved performance.  This is not to say that individual accountability isn’t important – some people need a lot of it!  Rather, it is to say that we have other higher leverage strategies more likely to produce the outcome we want, such as more effective recruiting and empowerment of our professionals.

Performance-based compensation is a key element for improving educator quality – The report does acknowledge that the research is “mixed” on this point, but I’d more characterize the evidence to indicate that performance-based compensation has no impact on student outcomes.  I’d urge Rotherham and Mitchel to more closely read the Vanderbilt POINT study, which they do reference.  The “no effect” finding should come as no surprise.  Researchers like Frederick Herzberg and Deci & Ryan have clearly told us that the most important aspects of a compensation system is that it is adequate and fair and that money is not a strong “motivator” for quality.  The simple behavioristic approach of offering merit pay to educators so they will work harder for kids has no basis in evidence and is professionally insulting.

Transparency and choice will lead to improved teacher preparation – This statement comes right out of the report as the authors recommend creating more of a free market for teacher preparation, allowing more groups to prepare teachers, and removing barriers to entering the profession.  Rather than a recipe for quality, this is a recipe for increased variability.  Higher performing education systems actually restrict educator preparation institutions and demand higher quality to get a systemic impact.  No high performing system has used a Teach for America or “let a thousand flowers bloom” approach to educator preparation.

Traditional education “interest groups” have too much power and are the problem – While it is clear that the authors did put some considered thought into this report and their writing, this claim borders on laughable and is, at a minimum, self serving.  The traditional interest groups (of which I would include those groups which represent teachers, school boards, and school administrators) are the only groups representing the large scale voices of practitioners in the field.  It is groups like Bellwether (and Donnell Kay here in Colorado, for that matter) who have worked to shove out these traditional groups and the voices of practitioners and replace them with a parade of of ideologically-minded nonprofits who are all advocating for some vision of an American education system built on test and punishments, the deconstruction of public schools, and the destruction of community-based decision making.  The traditional interest groups are not the problem; the hijacking of education policy by big money philanthropists and their nonprofit fronts are precisely the problem.

IN CLOSING

I’d like to again thank Andy Rotherham for calling me out on my Twitter criticisms of his report.  It is a lengthy piece that deserved more attention than 140 characters could provide.  I hope this blog posting makes my concerns with the report more clear and I look forward to engaging with Andy (or others) in the spirit of open and respectful discussion.

Profession

Colorado has assembled a blue-ribbon council of sorts to review and suggest possible changes to the teaching profession in the state.  Some of the proposed changes would involve removing all barriers to educator licensure except 1) having a Bachelor’s degree 2) not being a felon 3) passing a content test (currently a multiple-choice exam).  The group is also considering using evaluation results to strip educators of licensure if they are rated “ineffective” using the state’s evaluation system, which is hitched to student outcome results.

This group, called the LEAD Compact, is tasked with coming to some level of consensus and recommending legislation for the state General Assembly to consider in the next session.

I am not part of the membership of the LEAD Compact, but as a professional educator and a Superintendent, the decisions that could emerge from that group certainly might have an effect on the schools I am responsible for.  So, I created some guidance for the group to consider in making its determinations.

To the credit of those facilitating the LEAD Compact, my (unsolicited) guidance was accepted and provided to the members of the group.  I present the document here, as I think a larger discussion on the teaching profession is in order, beyond just the membership of this esteemed group.

From:  Jason E. Glass, Superintendent & Chief Learner

To:       Membership of the LEAD Compact

Re:      Redesigning Educator Quality for Genuine and Sustained Greatness

Date:   November 18, 2013

Overview

“The quality of an education system cannot exceed the quality of its teachers.” – McKinsey & Co., 2007

 It is, without question, of paramount importance that we focus on raising educator quality in the state of Colorado if we are to become a high performing education system.  The purpose of this memorandum is to brief the members of the LEAD compact some design elements for raising educator quality that are 1) systemic in nature 2) benchmarked against the highest performing education systems in the world and that 3) views teaching as a high status and highly skilled profession, on par with medicine or law.

Internationally Benchmarked Strategies

The recruitment, development, and retention of a highly skilled and talented educator workforce is a bedrock element in every great education system.  Comparative research on high performing global education systems (see McKinsey & Co., 2007 & 2010; Tucker, 2011; Hargreaves & Fullan 2012) point to systemic and focused efforts to raise the quality of educators through a combination of the following strategies:

  • Raising the status and respect of the teaching profession in society.
  • Increasing beginning pay levels to be competitive with other professional options.
  • Using rigorous and selective preparation programs that balance high levels of content knowledge, empirically-driven pedagogical training, and extensive clinical experience.
  • Being highly selective at the point of entry into the profession via licensure and hiring practices.
  • Providing intensive and quality mentoring support for beginning teachers.
  • Building wisdom and effectiveness in the teacher workforce by retaining experienced educators.
  • Providing career options and pathways through the profession which keep educators connected to teaching.
  • Creating intentional structures where educators work together in teams focused on instruction, using formative measures to guide the delivery of instruction.
  • Using (but not hyper or over-relying) on respectful performance-management systems to provide individualized and timely feedback on instruction.

Using international and national benchmarking strategies, where we observe the practices of the best performing systems, the above strategies emerge as the de facto recipe for improvement of educator quality as a system.  Building real educator quality requires an extraordinary expenditure of focused, intentional, and thoughtful effort what may be years.

There simply are no shortcuts to genuine improvement and the establishment and ongoing perpetuation of a high quality education profession.

Design Principles for Educator Quality

The following questions are intended as design principles, which may be helpful to the members of the LEAD Compact in framing their work.  Considering again those touchstones of 1) systemic in nature 2) internationally benchmarked and 3) based on a professional model of teaching – it is hoped that whatever system the LEAD Compact recommends stands up well when considered against these design elements.

  1. Does the new system of licensure serve to raise the status, respect, and credibility of the teaching profession in our society?
  2. Does the new system of licensure serve to set a high bar for entry into the teaching profession; signaling high and quality levels of content preparation, pedagogical knowledge, and clinical experience?
  3. Does the new system of licensure empower the teaching profession to set its own high professional standards through a governance board of practitioners (as is the case with the Colorado Medical Board or the Colorado Bar Association)?
  4. Does the new system of licensure operate as a high reliability system, ensuring that every child in Colorado has a high quality educator in every subject, grade, and specialty area?
  5. Does the new system of licensure recognize genuine professional growth, as demonstrated through comprehensive and peer-based determinations?
  6. Does the new system of licensure consider a teaching license a rigorous and hard-earned professional property right, which is afforded due process in order to be removed?
  7. Does the new system of licensure aggressively safeguard the quality of the profession by removing the property right of licensure (through a fair mechanism with due process) from those who violate professional ethos, standards of professional quality, or who do harm to children?
  8. Does the new system of licensure recognize the presence of error and bias all measures as well as the presence of Campbell’s law[1]?
  9. Does the new system of licensure consider teaching sacred and honorable?  A profession which has important dimensions of moral purpose and (most importantly) calling?

Conclusion

I commend the work of the LEAD Compact for its interest in supporting efforts to systemically raise educator quality across our great state.  It is my sincere hope that the evidence from other high prestige professions and high performing systems informs your work going forward.

Thank you for your time and consideration of these ideas and I look forward to your recommendations on how educator licensure might be re-imagined for genuine and sustained greatness.

With respect & admiration,

Jason E. Glass

Superintendent & Chief Learner


[1] “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” – Campbell, 1979.

The full document can be found here:

LEAD Memorandum

It should go without saying that the leading American reform du jour is to construct educator (particularly teacher) evaluation systems that use student achievement as a significant component of the evaluation.

The exact components in creating such systems remain a work in progress across the country and always involve significant trade-offs (e.g. efficiency vs. authenticity; complexity vs. understandability; generalizability vs. specificity).  As if designing such systems in schools across the entire country weren’t complex enough, public policy decisions are also already driving how these (still largely experimental) systems are going to be used for things like teacher accountability, dismissal, licensure, and compensation.

Here in our district, Eagle County Schools, this organization has been working on these systems for nearly a decade.  While I think our system is stronger than many being proposed nationally, it is still very much a work in progress and a journey.

This post is not intended to argue the prudence of this work nor is it intended to prognosticate about the probability of this sort of reform resulting in dramatic improvements in educator quality.  It is intended to recognize that, like it or not, much of the country is currently engaged in designing these sort of systems and we can benefit from practical lessons learned from experience.

As our educators wade (yet again) into the work of redesigning an evaluation system, there are some key design elements that we might keep in mind so that the “right drivers” (to steal Michael Fullan’s words) are at the center of these efforts.

As it may be helpful to other educators and school leaders engaged in this work, I present the design principles I’ve asked our educators to hold close as they go about the construction (or re-construction in our case) of a compliant evaluation system.

Please see the link below for the design principles we are using in Eagle County.  As always, I appreciate and look forward to any feedback you might have.

Design Principles for Evaluation Systems

I’ve been working with the Eagle County Schools Board of Education to design an evaluation system for myself (as Superintendent) that is clear, fair, professionally rigorous, and that provides the foundation of many conversations about improvement and growth.

I know others might be looking for resources in their own work in this area.  So, I’m posting the system we designed so that it may inform other similar efforts elsewhere.

A couple of notes.  First, this is intended to be a work in progress.  The Board and I have regularly scheduled opportunities to revise and improve this system through a collaborative approach.  Second, Eagle County Schools and my contract may be a little unique in that the compensation uses a performance-based framework.  So, if you see elements related to performance-bonuses or annual raises based on performance – that’s where it is coming from.  In any case, the standards and rubric should be applicable to any Superintendent.

Please consider this a resource in your own personal and professional growth and feel free to provide me any feedback or comments you may have.

Superintendent Evaluation Rubric Performance Criteria

New York Mountain in Eagle County, CO

New York Mountain in Eagle County, CO

I spent the past 100 (or so) days in Eagle County building a possible new direction for the organization.  This new direction was forged from countless interviews with education professionals and community members in Eagle County as well as from my own personal and professional journey in learning how to build a great education system.

I post it here for your review and consideration and welcome any thoughtful discussion it might bring about.  The full document can be accessed here and it can also be downloaded as a pdf from the same site:

http://issuu.com/eagleschools/docs/altitudereport3

The critical question, at least for me, is clear: How do we raise an entire state to be one of the highest performing school systems in the world?  This question takes up nearly every moment of my being, to the point of near fixation.  I consume volumes of books, journal articles, news stories, reports, editorials, opinions, conversations, charts, tables, and diagrams.  I visit schools and talk to educators, looking and listening for parts of the  answer to the question.  I spend hours and days in airports and airplanes to attend meetings where educational strategies and tactics are espoused and debated, all in pursuit of bettering our schools.

I believe we can take it as granted that everyone (or at least most everyone) wants our schools to be better, much better, than they are now.  Where we come unraveled is in getting agreement on the specific actions we will undertake, as a system, to improve. In looking to the lessons of the world’s highest performing education systems, getting to some level of agreement on the tactics we will collectively take clearly matters.  It matters in that whatever approach we undertake we will need to sustain it through the swings of the political pendulum and we will need to adequately resource the effort to give it the chance to succeed.  A fractured approach does not lead us to that end and is also unlikely to lead us toward having one of the world’s best education systems.

So what tactics and strategies should we undertake?  Where should we place our efforts?  In my studies on how one might raise an entire education system (not a few schools or districts, but the entire system), I am increasingly convinced that both a continuation of past reform efforts (lower class size, incremental annual spending increases, and accountability) or the relatively new breed of American reform strategies (elimination of job protections, individual level evaluations linked to test scores, and school choice) are unlikely to work if our goal really is building an American school system that stands alongside the world’s highest performers.

So we face some choices.  One is to continue the (often) politically motivated infighting and factionalism that dominates the current debate and see who ultimately bludgeons the other side into (temporary) submission.  Another is to do nothing; paralyzed by the fear of making a mistake.  Perhaps the right path is to reject these two options and converge on a set of strategies that is most likely to deliver us at that goal of a world-class education system.

I’d like to propose four lenses to frame that debate.  If the strategy or approach passes through all four lenses, then it fits in the discussion.  If it doesn’t, then it’s out.  Note that being “in” shouldn’t mean it’s in forever – just that the approach makes sense in the current context.  Similarly, being “out” doesn’t mean it’s out forever – it just means that either the timing isn’t right or we need more testing and empirical validation of the approach before we take it to scale across the entire education system.  So, “what are these four lenses that SHOULD frame our education reform agenda,” you ask?

1.  Is it related to the instructional core?  Harvard professor Richard Elmore rightly points out that if you aren’t doing things that have an impact on the relationship between the teacher and the student in the presence of content, you aren’t doing anything that’s going to positively change performance.  Using this first question as a lens is incredibly constructive in helping us sort the wheat from the chaff in where we should place our efforts.  The danger in using this lens in isolation is that there are lots of things that affect this relationship between teachers and students in the presence of content; especially if you allow yourself to birdwalk out on a few limbs.  We can’t just rely on this lens alone.

2.  Is it strongly supported by the evidence?  This lens can be a bit tricky as one can find some evidence to support just about anything.  But we stand a much better chance of being “right” with whatever approach we take if are aligned with evidence that reaches the caliber of being peer-reviewed, journal quality work.  Further, we should pursue approaches that have a preponderance of evidence that supports it.  This helps prevent us from chasing the latest thing or being led astray by a singular research finding that contradicts the larger body of evidence on any particular strategy.  The danger of using this lens in isolation is being paralyzed by analysis, wanting more and more empirical validation before actually doing anything.  Good implementation begins with using evidence to calibrate your shot, but ultimately taking action.

3.  Is it scale-able?  If our goal is really to get a whole education system to improve, we must reject efforts that do not scale as the primary drivers for improvement.  Efforts that do not scale show up dressed in one of two outfits. One is in the form of small-scale pilots and projects, where we have a few schools or districts undertake some effort.  Pilots and projects are incredibly important for experimentation and empirical validation but let’s not fool ourselves into thinking these are going to do anything that will make the whole system move; especially if, at the end of the pilot or program, we never do anything to grow the validated approach.  The second form of efforts that do not scale comes in the guise of attempts at small scale excellence.  Suspects here include many school choice efforts and alternative educator licensure pathways.  Don’t get me wrong; I’m a fan of charter schools as a mechanism for innovation and a fan of approaches like Teach for America in their efforts to bring top talent into education.  But we are badly fooling ourselves if we think either of these efforts has the capacity to raise the quality of our entire education system.  Don’t believe me?  Refer to question #2 above.  The danger of using this lens in isolation is that there are lots of things we could take to scale.  But if it’s not related to the instructional core or if it isn’t supported by evidence we run the risk of creating big, expensive, and ineffective distractions that don’t result in a world-class education system.

4.  Is it supported by international benchmarking?  A great place to start for information on how we might grow our education system into one of the world’s best is by asking questions of what the world’s best education systems actually do.  A comparative analysis of these systems, looking for common approaches and strategies in their rise to greatness, is perhaps our best evidence of what’s going to work to raise our education system to top performing status.  As a contrast, the discussions about pure local control, or the even more rabid version of this which advocates the complete elimination of state authority and state departments of education, is completely absent as a strategy of improvement in studies on the rise of the world’s best education systems.  More directly, there are no examples of world-class education systems that have used this approach and achieved greatness.  The key here is balance, a topic I’ve explored before.  So, using the lens of international benchmarking, seeing what approaches the best performing school systems actually use, can be an incredibly constructive lens in helping us decide which approaches to take.  The danger in using this lens in isolation is that you can fail to take into account that each school system has history, culture, and context – and all of these must strongly be taken into account in choosing a strategy that makes sense.

It’s never too late for us to change tracks and choose approaches and efforts that are much more likely to actually work in pursuit of a better education reform agenda.  In fact, I’d argue it’s too late not to make this change.  No one of these four lenses gets us there completely, but I’m arguing that using all four together gives us a powerful framework from which to make decisions about where we should put our efforts and which approaches to avoid.

Nearly everyone involved in the work of improving or reforming education acknowledges the importance of the people working in our schools as perhaps the most critically important lever to improve learning. Even with technological advances, education remains primarily an endeavor driven by teachers, administrators and those in supporting roles in schools. As an extension, the success, mediocrity, or failure of our schools also rides on the qualities, capacities and talents of the people working with our students.

Hard-charging education reformers put a tremendous amount of faith in the ability of “human capital” systems to deliver educators with the abilities we need to dramatically improve our schools. Using strategies related to human resource processes like recruitment, selectivity, performance management (including evaluation), compensation, and dismissal – this “human capital” frame holds that if schools would use theses human resource processes more effectively the result would be a more capable educator workforce. This viewpoint primarily puts the individual educator as the central point where we should focus our attention and work for improvement.

Juxtaposed against the human capital frame is another viewpoint that great educators emerge from collaborative and collegial environments where educators are given the opportunity to learn from each other, plan together, build relationships among staff and students, be involved in key decisions, and work in an environment where they have the tools and resources to succeed. This “social capital” frame holds that it’s not the people that are the problem, it’s the system (or lack thereof) in which educators are working that is the problem.

It’s difficult (at least for me) to take a hard line against either of these views. It is disingenuous to argue that talent and ability doesn’t matter. Further, it is also disingenuous to say our human capital systems in education are anywhere near as effective as they could or should be. Differences do exist in educator quality that can be attributed to the capabilities and talents of individual educators. Failure to acknowledge or address human capital concerns (or the more common tactic of trying to advance some excuse or “red herring” to distract the argument) does little to advance us toward the common goal of a better education for all of our students.

It is equally insincere to argue that the systems and supports in which people work don’t matter. Anyone who has had the experience of working as part of a high quality, high functioning team or organization knows that the whole is much greater than the sum of its parts and that collective capacity trumps individual heroism when it comes to delivering quality on a consistent basis. On the flip side, anyone who has had to work in a dysfunctional environment or under a tyrannical boss knows that bad culture kills productivity and creativity.

Somehow, the debate on improving education in this country has got to reconcile these two ideas of human and social capital. Too often we place them in contrast to one another when we should be considering how they can (and should) be used to compliment each other.

We aren’t going to achieve greatness through a pure reliance on draconian, Kafkaesque systems of individual accountability. We also can’t achieve greatness through the liberal use of some professional Kumbayah circle.

Talent, intelligence, and ability matter. So do connections, belonging, and love.

For the sake of the American education system (and more importantly our children), we’d better figure it out sooner than later.

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 13,691 other followers

Jason’s Tweets!