Recently I attended a Cambridge workshop on educational leadership and now, am in the process of planning one for teachers. The one big learning that I have gained in the past few years, which has been further reinforced by this workshop, is the importance and need for measuring impact.
Impact of what we teach in classrooms, of processes we put in place to increase our effectiveness and efficiency, professional development activities that we plan etc. that we undertake every year.
Very often we believe that we are measuring impact of the specific interventions that is deployed. But what in fact most often we end up measuring is monitoring and measuring progress.
Theory of change explains,
‘how activities are understood to produce a series of results that contribute to achieving the final intended impacts. It can be developed for any level of intervention – an event, a project, a programme, a policy, a strategy or an organization.’
This works on the causal model linking inputs and activities to outputs and desired outcomes and impact. The key point here being, that we not only need to look out for output but outcomes. For example, conducting a workshop for teachers would count towards output, i.e. a workshop was conducted for teachers. However, how do we measure the desirable outcomes that were to be achieved and more importantly, how do we measure the impact of those outcomes? One source of data to measure the effectiveness of the workshop is to collect feedback from the participants.
Most often, this is the only data that we collect.
Different sources of data can reduce biases. Therefore, a mix of direct and in-direct measures to collect data would lead to better study of the impact of the strategy being deployed.
Direct measures would include,
- pre and post assessment data
- studying conceptual tasks assigned to students like concept maps
- observing students as they perform a task
- analysis of the body of student work eg. exam scripts, essays, internal assessment,
- anecdotal records
- think aloud protocols allow the students to verbalize their thinking as they solve their problem. etc
Indirect measures could include,
- exit slips
- ‘muddiest point’
- reflections etc.
Why? Because multiple data points help us in getting a better and clearer picture of student learning. While most teachers do collect multiple data points, it is at times not planned during designing the strategy that is to be deployed and therefore the analysis drawn from it is more serendipitous. Moreover, depending on one set of data could be limiting as each measure would have its own limitations and biases. Therefore, a mix of direct and indirect measures would support a more accurate assessment of learning in a classroom or for individual students. You can read more on it here.
Triangulation of data is the process of using at least three sets of data to interpret assessment results or outcomes. When all data points lead to similar conclusions, we can plan interventions to address students needs more confidently.
Which data measures to be used will depend on the impact that is being studied. So, if I am wanting to study the impact of distributed leadership, my question would be something like this:
What impact does distributed leadership have on (school/section name) teacher engagement in their collaborative groups this year.
The evaluative question must specify:
- what is to be evaluated
- intended outcome and how is to be measured
- group of people and their context.
What you need to keep in mind as you plan to study impact:
- Formulate a specific research question.
- Evaluate the strength and limitation of each assessment instrument that you intend to use.
- Decide if you would have a control group.
- Conduct a pre-test, then implement strategy and then conduct a post-test to study impact.
- Calculate effect size as part of analysis
- Report your findings.
This year I plan to use impact study in science classrooms. I will co-create the impact study with the science teacher for one grade and use another grade as a control group:
What effect does an assessment for learning strategy (muddiest point) have on student achievement in grade VII science on the topic human physiology.
The result of the impact study will be shared soon!