Warning for my non-development-worker readers: this post is a bit technical. But I hope I’ve explained everything clearly, and please ask questions if I haven’t! I would love to hear some new perspectives on this topic.
Also, I started writing this post a few months ago, when two big themes in the aid blogosphere were complexity and impact evaluation. People seem to be writing about this a bit less these days, but they’re still important topics (and we don’t have any clear answers) so it’s my turn to weigh in!
Where does all the money go?
So what is impact evaluation? This is the attempt to use rigorous methods to understand what actually works, or has impact, in aid and development. Of course there is no “silver bullet” solution to global poverty, but which interventions are more effective than others? Which things have no impact at all, or a negative impact? Ultimately, rigorous impact evaluation should lead funders and policy-makers to direct funds towards interventions which are most effective and reliable in improving people’s lives.
There is a huge push right now to “show what works” in foreign aid. Citizens are seeing that the Western world has spent trillions of dollars on foreign aid in the past 50 years, yet global inequality is worse than ever before. Of course progress has been made, but have the results justified the spending? Citizens want accountability from their governments, and proof that their hard-earned tax dollars are actually having a positive impact on the lives of those they are targeting in the developing world.
Many agree that the most rigorous form of impact evaluation today is the Randomized Control Trial (RCT). This technique takes a statistically significant sample size and randomizes selection into two groups: control and treatment. The treatment group receives a development intervention, such as crop insurance, or microcredit, or whatever you are trying to test, while the control group receives nothing. Both groups are evaluated over the course of the study (usually 1-5 years) and the results come out somewhere along the spectrum of “yes this works” (how?) to “this has no effect”. Sometimes it’s inconclusive, or the results are not easily generalizable, or there is further research to be done, but RCTs are generally considered the gold standard in evaluating development interventions.
There is a lot of controversy around RCTs because of the high cost and time involved. These studies are not appropriate in all cases. They shouldn’t be used by organizations looking to evaluate past programs, or smallscale projects looking for continued funding. Instead, RCTs should be used to inform development and foreign policy on a large scale. Citizens giving foreign aid want to know, for a fact, which development interventions are the best bang for their buck – and RCTs should, in theory, be able to tell them.
Inside the black box
The second trend these days in the blogosphere is complexity, or complex adaptive systems. Aid on the Edge of Chaos has a good round-up on complexity posts. The bottom line here is that in a complex system, results are unpredictable. The system is not static, or linear, or deterministic; it evolves over time, adapting and growing based on both internal and external influences. When dealing with a complex system, you need to take a “systems approach” by monitoring the whole rather than the individual pieces.
What does this mean for development? Well, people and communities and indeed the world are all complex systems. It is hard to predict when something will change, as we’ve seen with the recent wave of revolutions across the Middle East. From this perspective, it is hard to ever know which development interventions will achieve the results we want.
Most development interventions are designed around something like an impact chain – what will you do, and what results will it produce? However, complexity theory tells us that we should monitor a system generally for results, not just for our predicted or desired results. There are often unintended results of our actions, sometimes negative and sometimes positive. In addition, it can be hard to attribute positive changes to our particular intervention in a complex system – so much is happening at once and there are so many stimuli to the system, you never really know where something originated. This also poses problems when we’re discussing replicability of development interventions – just because something worked once in a particular set of circumstances, doesn’t mean it will work again in a different setting.
So what does it all add up to?
Impact evaluation and complexity. Now it’s time to bring these two concepts together. The big question here is this: Can we understand “what works” in enough detail to be able to predict future results of our interventions?
My answer is yes, but maybe not in the way you’re thinking. In the past, evaluation has usually focused on the question “what intervention worked?” where the answer is “fertilizer subsidies” or “school feeding programs”. I think we need to start looking more at HOW things work. Instead of looking for programs that we can replicate across entire regions, we should be asking, “what worked?”, “under what conditions?” and “with what approach?” which give answers more like “foster innovation”, “promote local ownership” and “give people a choice”. These conditions may be found across many different areas, but may have more of an effect on the success of an intiative than the WHAT of the initiative itself.
I generally support rigorous impact evaluation for 2 reasons:
- fostering a culture of accountability to donors and stakeholders (and taxpayers);
- foster learning so that we understand conditions for success and can set projects up for success in the future.
I think the aid industry has learned (and can learn even more) something about what works, or probably more about what doesn’t work. I also think aid can’t be prescriptive since human beings are complex and our behaviour is irrational and unpredictable. But we can set conditions for success when designing our interventions. And while the results may not be wholly predictable, at least the intervention will be more likely to succeed.
What does this look like in practice?
We are currently in the process of team transition and strategy re-development. Here are a few principles I’m looking to follow with our team strategy as we go forward:
- always have a portfolio of initiatives on the go (don’t put all your eggs in one basket)
- make sure these initiatives all contribute toward the bigger change we’re trying to make in the agric sector
- range of timescales: short-, medium- and long-term changes, informing and building on each other
- constant learning and iteration: testing, getting feedback, adapting, and testing again
- focus on articulation of our observations and learning to external audiences
- high awareness of the system as a whole: what does it look like? where are the strongest influences? the most volatile players? who exerts the most force on the system?
What principles would you add to my list?
Hey readers, Happy Friday!
This post has a bit of a different focus: Canada’s aid effectiveness. Sometimes it’s easy for me to forget that there are thousands (whoa!) of EWBers working hard in Canada to tackle the “other end” of the development problem: foreign aid. This is hugely important and has a MASSIVE effect on the work we’re able to do here in Ghana. I’m seeing more and more every day about how CIDA affects the development environment in Ghana, especially for MoFA – they’re one of the major funders and could be doing a lot of things better (oh hey CIDA!).
That’s where you come in. As part of the new ACT campaign (which you can check out here), EWB has a goal of sending 1000 letters to our local MPs across the country, calling on our elected officials to ACT in making Canada’s aid more effective. ACT stands for Accountable, Creative and Transparent – three qualities that can dramatically improve the effectiveness of our foreign aid. From the website:
“We are asking Canada to ACT on aid effectiveness – making our aid more accountable, creative and transparent. In short, it’s not what we fund, but how we fund it. Good aid and development are stifled by skewed incentives and weak systems. The ACT agenda tackles several of the core issues that undermine successful aid programs, focusing on increasing the impact of aid dollars through a series of evidence-based, thoughtful reforms. It outlines three specific areas of action that we believe all parties can agree on, and that Canadians can get behind, in the leadup to the G8 meeting in Canada.
- Role model our good track record of meeting international promises by creating a standard for tracking and reporting on major international aid commitments, committing to use it, and inviting other G8 countries to follow our lead. Such a standard should monitor progress through the life-cycle of all promises, publicizing this information along with interim progress reports to enable citizen groups and media to hold governments accountable to their promises.
- Establish an expert, independent, arms-length development evaluation agency to assess the results from Canadian aid. These results should be shared with a) Canada’s Parliament; b) the Canadian public; c) our partner country’s government; and d) local civil society in the country’s where we provide aid.
- Set up a $150million venture-focused Implementation Innovation Fund to fund the expansion of promising initiatives. This would complement the research-focused Development Innovation Fund and provide later stage financing to take ideas to scale. The evaluation criteria of Implementation Innovation Fund should be the adoption/seeding of their innovations into regular poverty reduction projects (or the expansion of for-profit pro-poor ventures) worth 1 billion dollars a year within 5 years.
- Join many of the world’s leading donor organizations by signing on to the International Aid Transparency Initiative, a new initiative that aims to make information about aid spending easier to access, use and understand.
- Publish comprehensive, openly available data about all CIDA projects and programs – including the results of all evaluations – on its website. This information should allow the user to track any project from the initial proposal through all stages to project implementation.”
EWB members across the country are working hard to realize this goal. In addition to encouraging you to sign the petition, which can be found on the website, I’d like to challenge everyone to get involved! Here’s the challenge: convince 5 of your closest friends/family to write a letter about the ACT campaign to their local MP. And of course, write one yourself!
Here are some of the essential elements to include in your letter:
- Your name and address, so the MP knows you’re writing from his/her constituency
- Why you’re writing (to call on the MP to support EWB’s campaign to make Canada’s aid more effective)
- A short summary of what ACT is all about (again, website here)
- Pick one action item from the website that you really like and describe it in more detail
- Ask for them to follow up if they have any questions or response to your letter
- Tell them that they’ll be joining the majority of their peers, who EWB has already met with, in supporting this initiative!
And a template to help you out:
A few key things to remember about letters to MPs:
- Handwritten letters are most effective – if you can, write out a personal letter and mail it in!
- Emailing your letter is still fine, but not as good as a handwritten letter!
- Write from your perspective – make a personal connection to the issue and, where possible, to the person you are writing to as well
- Try to keep your letter to one page, two at maximum
- You can find your MP’s address and email at: http://webinfo.parl.gc.ca/
- Be sure to include all of your contact information so the person you are writing can reply.
Finally, if you take the leap and write a letter, or get anyone else to write a letter, please let me know! We’re trying to track the number of letters that get sent to MPs.
Alright, that’s it for this sunny Friday in Tamale! I hope you all have a wonderful relaxing weekend, with a bit of political advocacy thrown in, just for kicks. After all, it’s your government – you have the right to tell them what to do with your money, and to hold them accountable!