You are currently browsing the category archive for the ‘Evaluation’ category.
The critical question, at least for me, is clear: How do we raise an entire state to be one of the highest performing school systems in the world? This question takes up nearly every moment of my being, to the point of near fixation. I consume volumes of books, journal articles, news stories, reports, editorials, opinions, conversations, charts, tables, and diagrams. I visit schools and talk to educators, looking and listening for parts of the answer to the question. I spend hours and days in airports and airplanes to attend meetings where educational strategies and tactics are espoused and debated, all in pursuit of bettering our schools.
I believe we can take it as granted that everyone (or at least most everyone) wants our schools to be better, much better, than they are now. Where we come unraveled is in getting agreement on the specific actions we will undertake, as a system, to improve. In looking to the lessons of the world’s highest performing education systems, getting to some level of agreement on the tactics we will collectively take clearly matters. It matters in that whatever approach we undertake we will need to sustain it through the swings of the political pendulum and we will need to adequately resource the effort to give it the chance to succeed. A fractured approach does not lead us to that end and is also unlikely to lead us toward having one of the world’s best education systems.
So what tactics and strategies should we undertake? Where should we place our efforts? In my studies on how one might raise an entire education system (not a few schools or districts, but the entire system), I am increasingly convinced that both a continuation of past reform efforts (lower class size, incremental annual spending increases, and accountability) or the relatively new breed of American reform strategies (elimination of job protections, individual level evaluations linked to test scores, and school choice) are unlikely to work if our goal really is building an American school system that stands alongside the world’s highest performers.
So we face some choices. One is to continue the (often) politically motivated infighting and factionalism that dominates the current debate and see who ultimately bludgeons the other side into (temporary) submission. Another is to do nothing; paralyzed by the fear of making a mistake. Perhaps the right path is to reject these two options and converge on a set of strategies that is most likely to deliver us at that goal of a world-class education system.
I’d like to propose four lenses to frame that debate. If the strategy or approach passes through all four lenses, then it fits in the discussion. If it doesn’t, then it’s out. Note that being “in” shouldn’t mean it’s in forever – just that the approach makes sense in the current context. Similarly, being “out” doesn’t mean it’s out forever – it just means that either the timing isn’t right or we need more testing and empirical validation of the approach before we take it to scale across the entire education system. So, “what are these four lenses that SHOULD frame our education reform agenda,” you ask?
1. Is it related to the instructional core? Harvard professor Richard Elmore rightly points out that if you aren’t doing things that have an impact on the relationship between the teacher and the student in the presence of content, you aren’t doing anything that’s going to positively change performance. Using this first question as a lens is incredibly constructive in helping us sort the wheat from the chaff in where we should place our efforts. The danger in using this lens in isolation is that there are lots of things that affect this relationship between teachers and students in the presence of content; especially if you allow yourself to birdwalk out on a few limbs. We can’t just rely on this lens alone.
2. Is it strongly supported by the evidence? This lens can be a bit tricky as one can find some evidence to support just about anything. But we stand a much better chance of being “right” with whatever approach we take if are aligned with evidence that reaches the caliber of being peer-reviewed, journal quality work. Further, we should pursue approaches that have a preponderance of evidence that supports it. This helps prevent us from chasing the latest thing or being led astray by a singular research finding that contradicts the larger body of evidence on any particular strategy. The danger of using this lens in isolation is being paralyzed by analysis, wanting more and more empirical validation before actually doing anything. Good implementation begins with using evidence to calibrate your shot, but ultimately taking action.
3. Is it scale-able? If our goal is really to get a whole education system to improve, we must reject efforts that do not scale as the primary drivers for improvement. Efforts that do not scale show up dressed in one of two outfits. One is in the form of small-scale pilots and projects, where we have a few schools or districts undertake some effort. Pilots and projects are incredibly important for experimentation and empirical validation but let’s not fool ourselves into thinking these are going to do anything that will make the whole system move; especially if, at the end of the pilot or program, we never do anything to grow the validated approach. The second form of efforts that do not scale comes in the guise of attempts at small scale excellence. Suspects here include many school choice efforts and alternative educator licensure pathways. Don’t get me wrong; I’m a fan of charter schools as a mechanism for innovation and a fan of approaches like Teach for America in their efforts to bring top talent into education. But we are badly fooling ourselves if we think either of these efforts has the capacity to raise the quality of our entire education system. Don’t believe me? Refer to question #2 above. The danger of using this lens in isolation is that there are lots of things we could take to scale. But if it’s not related to the instructional core or if it isn’t supported by evidence we run the risk of creating big, expensive, and ineffective distractions that don’t result in a world-class education system.
4. Is it supported by international benchmarking? A great place to start for information on how we might grow our education system into one of the world’s best is by asking questions of what the world’s best education systems actually do. A comparative analysis of these systems, looking for common approaches and strategies in their rise to greatness, is perhaps our best evidence of what’s going to work to raise our education system to top performing status. As a contrast, the discussions about pure local control, or the even more rabid version of this which advocates the complete elimination of state authority and state departments of education, is completely absent as a strategy of improvement in studies on the rise of the world’s best education systems. More directly, there are no examples of world-class education systems that have used this approach and achieved greatness. The key here is balance, a topic I’ve explored before. So, using the lens of international benchmarking, seeing what approaches the best performing school systems actually use, can be an incredibly constructive lens in helping us decide which approaches to take. The danger in using this lens in isolation is that you can fail to take into account that each school system has history, culture, and context – and all of these must strongly be taken into account in choosing a strategy that makes sense.
It’s never too late for us to change tracks and choose approaches and efforts that are much more likely to actually work in pursuit of a better education reform agenda. In fact, I’d argue it’s too late not to make this change. No one of these four lenses gets us there completely, but I’m arguing that using all four together gives us a powerful framework from which to make decisions about where we should put our efforts and which approaches to avoid.
Nearly everyone involved in the work of improving or reforming education acknowledges the importance of the people working in our schools as perhaps the most critically important lever to improve learning. Even with technological advances, education remains primarily an endeavor driven by teachers, administrators and those in supporting roles in schools. As an extension, the success, mediocrity, or failure of our schools also rides on the qualities, capacities and talents of the people working with our students.
Hard-charging education reformers put a tremendous amount of faith in the ability of “human capital” systems to deliver educators with the abilities we need to dramatically improve our schools. Using strategies related to human resource processes like recruitment, selectivity, performance management (including evaluation), compensation, and dismissal – this “human capital” frame holds that if schools would use theses human resource processes more effectively the result would be a more capable educator workforce. This viewpoint primarily puts the individual educator as the central point where we should focus our attention and work for improvement.
Juxtaposed against the human capital frame is another viewpoint that great educators emerge from collaborative and collegial environments where educators are given the opportunity to learn from each other, plan together, build relationships among staff and students, be involved in key decisions, and work in an environment where they have the tools and resources to succeed. This “social capital” frame holds that it’s not the people that are the problem, it’s the system (or lack thereof) in which educators are working that is the problem.
It’s difficult (at least for me) to take a hard line against either of these views. It is disingenuous to argue that talent and ability doesn’t matter. Further, it is also disingenuous to say our human capital systems in education are anywhere near as effective as they could or should be. Differences do exist in educator quality that can be attributed to the capabilities and talents of individual educators. Failure to acknowledge or address human capital concerns (or the more common tactic of trying to advance some excuse or “red herring” to distract the argument) does little to advance us toward the common goal of a better education for all of our students.
It is equally insincere to argue that the systems and supports in which people work don’t matter. Anyone who has had the experience of working as part of a high quality, high functioning team or organization knows that the whole is much greater than the sum of its parts and that collective capacity trumps individual heroism when it comes to delivering quality on a consistent basis. On the flip side, anyone who has had to work in a dysfunctional environment or under a tyrannical boss knows that bad culture kills productivity and creativity.
Somehow, the debate on improving education in this country has got to reconcile these two ideas of human and social capital. Too often we place them in contrast to one another when we should be considering how they can (and should) be used to compliment each other.
We aren’t going to achieve greatness through a pure reliance on draconian, Kafkaesque systems of individual accountability. We also can’t achieve greatness through the liberal use of some professional Kumbayah circle.
Talent, intelligence, and ability matter. So do connections, belonging, and love.
For the sake of the American education system (and more importantly our children), we’d better figure it out sooner than later.
Photo courtesy of Chris Palmer via Flickr
Perhaps some of this stems from the fact our field is much more about “support and celebrate” than “punish and shame.” But we aren’t even doing the “support and celebrate” part very well when it comes to recognizing great teaching. Even holding the “fire bad teachers” debate at bay for the moment, are we identifying the most effective and talented teachers we have, celebrating them, and then finding ways to use them in supportive roles to spread their effectiveness and expertise? I’d argue that, for the most part, the answer is no.
Several issues relating to educator evaluation have emerged recently, calling for us to redesign and improve our evaluation systems. First, waivers for increased accountability flexibility from No Child Left Behind require that states improve their evaluation systems to provide regular and more meaningful feedback to educators. Second, the InTASC Standards present a national framework for what good teaching is and how it could be measured from preservice through inservice. Finally, within a few weeks of each other, two studies have recently emerged concerning measures of educator effectiveness and its impact on students’ lives. The Gates’ Foundation work around Measures of Effective Teaching released its Gathering Feedback for Teaching report which positively and significantly correlates a number of teacher evaluation systems with student results and Harvard economists demonstrate that “teachers’ impacts on students are substantial” in terms of increased lifetime earnings for student who have more effective teachers.
So let’s sum up what we know:
1. There is variation in educator effectiveness (teachers aren’t all the same).
2. The components of effective teaching are known.
3. Effective teaching can be validly and reliably measured.
4. Effective teaching has an impact on students’ lives.
5. We, for the most part, ignore all of the above.
Most evaluation systems are “drive-by” and once-every-three-years measures that fail to accurately measure anything and provide little meaningful information. We go through the motions where the forms come out, an observation happens, everyone signs the papers, and then into the file drawer it goes. If this is all we are going to do, it would be a more productive use of everyone’s time to just drop the evaluation process altogether.
Improving evaluation systems means creating measures that both discern effective teaching and provide meaningful, individualized and “at the shoulder” supports for how to improve. So let’s not push against efforts to define, measure, and spread effective teaching. Given the evidence at hand, this is exactly the work we need to be engaged in.
One of the more common and vexing problems of human behavior is that we tend to point the finger of blame toward an individual person or group when something goes wrong, an outcome isn’t achieved, or behavior isn’t exactly what we’d like it to be.
It’s their “fault,” or “someone didn’t do their job,” or “someone has to be held accountable.” More often than not, the issue isn’t with an individual person or group of people. More frequently, the real issue is with the systems or conditions in which people are working or living.
Social psychologists call this effect “fundamental attribution error,” or more simply the tendency we have to blame people for systems issues.
Teachers get this all the time, and both the “blame teachers” movement and the counter-reaction against it are real life examples of fundamental attribution error gone off the deep end. Sure, there are ineffective educators – everyone knows this and probably has even had a few. But more often than not, it’s not the teacher that is failing, it’s the system the teacher is in.
We do the same thing with administrators. I can’t even begin to count the number of times I’ve heard “if principals would just do their jobs.” This comes up a lot in matters related to evaluation. The logic is that if principals would just do their jobs related to evaluation, we wouldn’t have an ineffective educator problem and would be able to remove those that aren’t effective.
But the principals, more often than not, don’t do their jobs when it comes to evaluation. Frequently when they do, it’s a drive-by assessment with little meaningful feedback or improvement. But is it really the principal, or is it the system we’ve put this person in?
And we do have ineffective educators – many of whom have the potential to get better and maybe even become great teachers. But they work in substandard conditions and have no real support systems about what “better” even looks like or how they might get there.
Our answer lies less in individual accountability, though that is important, and more in better systems.
The next time you hear someone make the fundamental attribution error, call it out. Our thinking has to change.
Jason Glass
Des Moines, IA
“Democracy is finding proximate solutions to insoluble problems.”
Reinhold Niebuhr
In education, we often place high value on those things which we can quantify, measure, and analyze using statistical methods. Certainly the most prevalent example of this is the use of the standardized assessments of student achievement that are a ubiquitous part of the American educational system.
While these quantitative measures are important and useful, there are also certainly many things in education (and in life for that matter) for which there are no standardized and quantifiable measures. These things which defy quantification are also things we frequently recognize are incredibly important. Things like the introspective and interpretive value of passionate art and music – like the Edvard Munch painting, or so many of the enduring songs by U2. Also above quantification is the value of the inspiring and uplifting visions of people working together toward a common and important goal – like so many of the images and stories we now hear of the democratic uprisings happening in the Middle East. Perhaps the greatest example is the power of human emotion, relationship, and connection – the apex of these being the complex emotion of love, in all its forms and nuances.
All of these things defy easy quantification, but are certainly of the greatest value.
In education, many of the things that are of the greatest value also defy easy quantification. Take for example the act of teaching. Teaching is perhaps more art than science, and happens in a complex and dynamic environment. While we can certainly identify those things research tells us are ‘best practices’ when it comes to teaching, and even design ‘rubrics’ and scales to evaluate those practices, deciding what is (or what is not) good teaching relies fundamentally on good and old fashioned … human judgement.
Human judgment is fundamentally subjective, contextual, and flawed. It is also incredibly important. Arguably, more important than any quantifiable measure. Rather than trying to mitigate the subjective effects of human judgment by chasing quantitative measures, in many cases we should be embracing the inherently unquantifiable power of human judgment.
A fundamental principle we can apply to using human judgment is balancing it with a democratic process. That is, whenever we need to rely on human judgment to make a decision, we should have more humans doing the judging. Validity and reliability remain critical concepts, but so does creating a democratic process by which we can harness the power of human judgment and mitigate some of the significant power and subjectivity issues.
Consider teacher evaluation. Rather than relying on one person (usually an administrator) to make a subjective human judgment on the quality of teaching, we should involve more people in the act of evaluating and judgment.
As we struggle with how to handle the difficult questions that arise when we really try to evaluate and determine educator effectiveness, capturing the value of human judgment – but balancing it with democracy – is an important principle for us to consider.
Jason Glass
Des Moines, IA
An unfortunate aspect of the current debate on education in our country is the polarizing nature of the discussion. It would seem that everyone must be in the “Education Reform Camp” (teachers and unions are the problem, charter schools and firing people are the answer) or the “Status Quo Defender” camp (all our problems could be solved if we were just left alone, had a better curriculum, and were given more money). Of course, I over-generalize here, just for fun – but I hope you get my drift.
Education leaders on both sides of this debate are heavily influenced by the extremists in their respective camps. Pick a fight and raise hell and you’ll be ballyhoo’ed as a champion. Compromise and you’ll be vilified as a wimp and sell-out to the cause. Unfortunate, but true.
What we should agree on is that our education policies should be all about improvement.
A recent study noted that in the U.S. we recruit teachers from roughly the bottom quartile of college graduates. What are we doing to get higher caliber candidates into the teaching pipeline and are we preparing them for the tough job they have ahead? A recent study from Tennessee suggests, at least in that state – but I bet the results pan out across the country, there are significant differences. We can do better in preparing teachers.
We also do a pretty terrible job of recruiting and screening teacher candidates. Some research even suggests that being really bright, having a high GPA, and coming from a top school actually hurts your chances of getting hired. I did a recent webinar for AASPA on this issue. We can do better.
Teaching in many schools is a lonely experience. Most schools allow for little real and meaningful professional learning, collaboration, or interaction opportunities for educators. The traditional model of professional development is for someone on high to devise an “initiative,” get some funding for it, and then cram it down through various organizations to the classroom teacher. By the time it gets to them, it frequently has little meaning or relevance to what they face on a daily basis. We need to turn this whole system upside down and empower teachers in schools to tell the system what they need, and then configure the system to deliver. Again, we can do better.
We also don’t do a very good job evaluating teachers. Dan Weisberg’s “The Widget Effect” shone the light on this national issue. I would venture that just by paying attention to the fundamental elements of validity (does the evaluation measure good teaching) and reliability (is the evaluation administered consistently) we could dramatically improve the quality teacher evaluations, and actually make them something that can lead to improvement. We can do better!
Then there is the issue of teacher dismissal, which garners all the headlines … and may be all that many of you remember about reading this piece.
Let me be clear on this next point, because we have a really hard time talking about it and I want to be very clearly understood: By far, the vast majority of teachers are amazing, kind, and selfless people who would do anything they could to help kids and help their community. These are the teachers who have inspired us, who saved our lives in one way or another, and who deserve our gratitude for their service and for sharing their gifts with us and our children.
There is also a small minority of teachers who are failing our kids and our schools. Sometimes, a teacher is failing our kids because of the system they have been put into and they lack the necessary supports and leadership to be successful. However, there are also teachers who are failing our kids because they either don’t have what it takes (talent or knowledge) or they don’t have an interest in getting better.
Great teaching is really hard work and (contrary to popular opinion) not everyone can do this well. But the structures we have now treat all these kinds of teachers the same regardless of their quality. We ignore the real differences in teachers because it’s a difficult conversation and the determination of quality involves a degree of human judgment. Because of this, we have created systems and legal protections beyond those in any other field that make it nearly impossible to remove an ineffective teacher.
We have all been complicit in creating and perpetuating this system – teachers, unions, administrators, school boards, legislators, governors, and the voters. All of us are to blame for where we are now … and all of us must share in the responsibility of making it better.
It’s about improvement – we can do better.
Jason Glass
New York, NY
Jason Glass
