Evaluating Results
While student progress should be assessed throughout the implementation process, more formal evaluation on a school-wide basis is essential to determine how progress made can be sustained and enhanced. Success often stimulates awareness that even more can be done if programs are modified and professional expertise is strengthened. Continuous improvement is the goal.
Step eight includes two tasks:
formal evaluation
action research
Commentary
Aggregate data on student learning by itself provides no evidence of ways to improve. It is important to look at differences among students and data on school processes and programs (see Step 2) need to be collected and analyzed.
Conditions that support effective teaching and student engagement are identified by the KEYS survey. Administering the survey before the initiative was implemented and later when it is being evaluated may inform an understanding of the effects of the initiative on student learning.
In interpreting the evidence from an assessment of the effects of a new initiative on student learning, here are some things the school community (and the district) should keep in mind:
When undertaking a new strategy that involves substantially different teacher behaviors and school conditions, it is not uncommon for schools to experience what Michael Fullan calls an “implementation dip”. This dip can be caused when people learn new skills but have not mastered them or when school cultures and processes have not changed sufficiently. This reality may lead to a premature abandonment of a promising program and is one important reason why evidence of student outcomes needs to be examined in the context of information about school conditions that could have influenced student learning.
While an implementation dip is not uncommon, sometimes substantial enthusiasm for a new idea, especially when the school is piloting a program much watched by others, can lead to positive effects that are not sustained because exceptional effort, rather than changes in professional capabilities and school conditions, explain short run success. This phenomenon is called the “Hawthorne effect” and is another reason to study both processes and outcomes carefully when assessing the effects of a new initiative.
Ideally, the introduction of a new initiative is part of a comprehensive and on- going school improvement effort. It can be difficult to sort out the effects of a given initiative on student learning, especially when standardized tests, which are typically not very sensitive to changes in instruction or curriculum (unless the initiative is aimed at aligning curriculum and instruction with the standardized tests) are the measure of program effectiveness.
For a brief introduction to assessing the effects of an intervention, CLICK HERE. [link 8a]