Dear Journalists: I know I’ve been hard on you in the past (and yes, you kinda did deserve it). But I’m turning over a new leaf. For the second time (not than anyone’s counting), I’m going to try to be helpful. Here is the first in a series that I’ll add more posts to over time about how you can get the most of your visit to an aid program or project:Understand that you cannot evaluate a project, program or organization during one-day visit. I’m going to say it again: relief and development work are complicated. Just as it takes us concerted effort over time to understand a context, so it also takes an outsider (e.g. you) time and effort to understand what is going on in a program in the field. You can certainly gain impressions, and below I’ll share a few tips on what to look for that will help you gain a more informed opinion. But in general I’ll say that to spend one day or even one whole week with an organization’s relief team in the field and then print declarative statements about whether they’re doing well or doing poorly, or whether the overall relief effort is succeeding or flagging – and whether your statements are positive or negative – is plain ludicrous. Things are very often not what they seem to outsiders. Things that look chaotic might simply be complex. And in the context of a short visit, both success and failure can, to an untrained eye, appear as the other.
It’s fair to ask for copies of evaluation documents, but be aware that many organizations will demure from providing this outright. Ask whether the evaluation was internal or external. Also be aware of where you are in the chronology of a program or disaster response: there may not be an evaluation available.Ask about the learning. Good aid learns from experience. A good relief manager or program director or media spokesperson should be able to articulate how the current program or relief effort is based on learning gained from previous experience. Don’t be surprised if the learning expressed is incremental – development and relief evolve slowly over time through a series of tweaks and fine-tuning from one program or relief effort to the next. An NGO that can’t articulate specifically what’s been learned in the past that’s being applied now should raise a red flag in your mind, as should one that claims to have it all the way right, now.
Ask whether learnings have been published or if there are plans to publish them. Ask whether the organization has already or has plans to participate in a multi-agency learning event or interagency evaluation (fairly common following large disaster responses).
Ask about the process used to design the program, project, emergency response, etc.: Here you are specifically looking for evidence a couple of things: First, you want to hear that local people were involved in articulating the need as well as the design of the project or relief program. Second, you want to hear that there was, in fact, a process: there was an actual assessment (not just a sort of willy-nilly mix of observation and the odd interview), there was an actual program design exercise that involved the analysis of assessment data.
Larger organizations may have their own in-house organization-specific process(es) and an accompanying set of tools (available online in many cases). They’ll most probably be happy to provide this, but be aware that your media colleagues won’t have the documents at their fingertips – they’ll have to track down a programs person for this. Smaller organizations may not have their own model/tools, but should still be able to talk to you in specific terms about the processes that they follow for assessment, program design, monitoring and evaluations.
Ask about outcomes. Someone should be able to tell you what the expected outcomes of a relief or development program are. Or, in hindsight, what they were. In development programs this is frequently expressed as a percentage of change in something (infant mortality decreasing, literacy rates increasing, etc.), while in relief programs this is typically expressed in what we call outputs – numbers of something (number of transitional shelters put up, the number of families with access to clean water, etc.).
It’s fair to ask why proposed outcomes or outputs may have been underachieved (if your conversation is after-the-fact), but don’t assume that underachievement equals failure or incompetence. Hear the explanation. Prices change, overall context evolves, and aid worker’s understanding of situations deepens – all of which potentially affect the outcomes of an aid project or program.
It is becoming increasingly common for organizations to make their evaluation documents available externally. It’s fair to ask. If they won’t share a full evaluation document, ask for an executive summary.
Use logic. Understand Pythagorean logic. Understand that correlation does not equalcausation. Know the difference between issues that humanitarian aid providers can fairly be expected to address or be held responsible for, and those that they cannot (e.g. cholera in Haiti = not the fault of NGOs). Don’t confuse anecdotes with data, or data with evidence, or evidence with proof. Understand the difference between simpleand simplistic. Beware of magik bullets and one-size-fits all solutions: Anyone who’s selling their widget or approach as the thing that will solve the world’s problems deserves closer questioning around learning, and process. Watch out for solutions in search of problems.
Ask the “why this, not that?” question. It’s fair for an NGO to define it’s focus or niche in a particular sector or place. I tend to be skeptical, though, of an organization that only does one thing (only one tool in it’s tool box). There should be a logical explanation for why they’ve chosen to do what they’re doing in the place where they’re doing it.
Understand ambiguity. With few exceptions, neither successes nor failures are total. Even the most stellar, most widely acclaimed aid organization or approach or program has areas that didn’t work well, aspects for improvement. See also “ask about learnings”: Any organization or project that presents a 100% rosy picture probably need deeper scrutiny. On the other hand, despite bold headlines, few aid programs are unmitigated failures. Thing are never cut-and-dried.
Ask for descriptions of the context. Where programs seemed very successful, it’s fair to ask about challengs and learnings. Where programs seemed to have failed or been extremely marginal, it’s fair to ask why. Also ask about learnings. Also ask about areas where there may have been unexpected successes (although these might not have substantially affected the overall outcome).
Understand that things are almost never the way they seem at first blush. Recipients of aid, local authorities, and local partners can all have many reasons for telling you that they loved project X when in fact they hated it, or vice versa, none of which are related to the reality of what project X is or does or did. Understand that there is a ubiquitous dynamic called “screw the outsider”, and understand – further – that you may be variously treated an outsider or an insider due to factors over which you have precisely zero influence. In very simple language, just because a local person goes on about how much they love project Z, isn’t proof that project Z works or is “good” or even that that person really loves it.
Frustrating? Yes, welcome to our world. And also, see the very first point: it takes time to triangulate information; it takes experience and expertise and also time to be able to figure out what’s going on.
Any questions?
14112010Good aid has five characteristics. Let’s talk about those:
1) Starts and ends with the needs of those affected by poverty, disaster, and conflict (a.k.a. “the poor”, “aid recipients”, “program participants”, “beneficiaries”…). Some might want to articulate this point as aid needs to be demand driven, rather than supply driven. How we think about aid – how we rationalize it, how emotionally and intellectually honest we are about why we do it, and why we do it the way we do it – matters. But if we’re to do it right, if we’re to plan and implement good aid, our starting point needs to be those whom we seek to serve. If that starting point is anything else (for example, the needs of a particular donor, surplus of something…) then a recipe for bad aid has already been started.
2) Follows good process. To put it very simply, you start with what the need is, define the most logical good solution, implement that solution, and evaluate your program or project against what you defined as the need. Some call this Project Cycle Management (PCM) (See also) Of course in actual practice there is a lot that goes into each of those steps. Knowing what the need is requires commitment and follow-through on assessments.
Defining the most logical good solution can require an amazing amount of organizational honesty and discipline – especially when an organization has defined it’s focus, capacity or “niche” very specifically. One of the most common mistakes which leads to bad aid is when an oranization, project or individual defines the solution in terms of what they have to offer, rather than in terms of what the most logical solution is. (I call this the “solutions in search of problems” approach.)
Evaluation, like assessment, is very often glossed over. Amateurs typically focus on implementation (and of course good implementation is critical), but implementation outside of the context of overall good process is meaningless.
3) Is evidence-based: Digging a little deeper into the assessment, planning, and evaluation steps of PCM, it is absolutely critical that assessments be done properly (seeTexas In Africa’s outstanding series on how social scientists think – assessments are more than just asking a few villagers what they want). If you don’t understand clearly both the issue (problem) that you’re trying to address, you can’t design a workable response, and at the end you can’t know if you’ve been successful.
Yet in my experience, this is the single most common downfall of small startup NGOs/projects and amateurs: skimping on assessments and evaluations, or simply not doing them at all. This is partially because those things require specialized skills, resources and organizational bandwidth. Sometimes where an organization or project defines it’s “product” very specifically (volunteer teams, 1,000,000 T-shirts, shoes, etc.), there is little point in doing either assessments, planning, or evaluation because the implementors already know what they’re going to do.
Good aid will be disciplined enough to go through the process of collecting and analyzing evidence, and then basing action on need, not on the surplus of a particular resource.
4) Tool-box approach: Expanding the above point with respect to action/programming, good aid will approach available resources as “tools” inside a “tool-box.” Which is to say that good aid will select the right tool for the job. Bad aid, by contrast, typically uses backward logic by selecting the tool (action, program…) in advance of having evidence.
Where an organization or project has only a few tools in it’s box, it can take real organizational discipline to say “no” to programming. Again, this is a very common mistake of small startups and amateurs: the desperate desire to act (or the real, survival need for resources) drive many to try to operate outside of their actual capacity or expertise – sort of like using a screwdriver to pound nails.
5) Learns lessons-learned: While humanitarian aid and development are not as old as other fields (accounting or urban-planning), there is already a substantial body of experience and lessons-learned. Good aid does not repeat the mistakes of the past. While this sounds simple, in my experience this also can require an amazing level of organizational discipline (organizational discipline is a common theme…), particularly (again) where an organization is founded on the premise of a specific kind of activity, particularly where that activity contravenes known best-practices. The existence still of foreign-run “orphanages” across the Third World are but one outstanding example in real-life.
Learning the lessons-learned also requires that an individual, project or startup NGO be aware of the history, be looped into the overall aid conversation, be current with industry thinking. This, also, requires organizational bandwidth and (wait for it…)discipline. It requires that people prioritize thinking and learning along with doing. It requires that organizations dedicate resources towards participating in learning events (sometimes feel like HRI-style “life-saving meetings”). It requires participation in coordination specifically, and generally being part of the larger community of practice overall. While few people would argue against learning the lessons learned in principle, in actual practice the realities of tight budgets, scarce resources, and staff already working 18-hour days frequently mean that the lessons learned do not get learned. And the result is frequently that while as an industry we know what to do and how to do it, many individual entities within the industry don’t.
Note: But what about “Local”? I know that some of you are already itching to fill my comments thread with hate-mail because I didn’t include something about “local knowledge” or “local NGOs” as one of the five characteristics of good aid. There are two main reasons why I did not.
- “Local” is a cross-cutting issue: In my direct experience in almost two decades of practice, getting the above five things right invariably means involving local… people, organizations, partnerships, knowledge, etc. When the above five things are done properly (really done properly), “local” happens organically
- “Local” is not a magik bullet: The above five apply equally to “local.” And in my experience, local NGOs are just as prone as INGOs, local staff just as likely as expats, to get the above wrong. (see also: here) The concepts of needs-based logic, good process, evidence-based action, picking the right tool for the job, and learning from past experience are as important (and as easy to get wrong) for local NGOs as for INGOs.
