I’ve been drafting project documents. It’s one of those jobs that has to be done, to give us a better work specification than “we want to build something to help analysts fill in the humanitarian information gap”. I’ve also been thinking for a while about how to best compare hackathon projects to determine which to support further (if the team’s willing to be supported of course) after each hackathon.
Part of project management is asking whether something should be done, how it fits into existing ecosystems and how much gain it brings over alternative methods. In business, this is covered by return on investment – the percentage cash gain from the project relative to the money invested in it every year. It’s more complicated of course – project accounting needs to think about cash flows over time via things like net present value and IFO (income from operations) curves – but the bottom line is usually ‘how big is the financial return’, before asking questions like ‘how does this fit our company values’.
How does this apply to humanitarian systems? There are two parts to this question: 1) how do we compare the social gains of one decision over another, and 2) how can we monetize these social gains to explain why finance-sensitive agencies (governments, big business etc) should invest in humanitarian work.
I’m starting with three sources for this (I have Patrick McNamara to thank for pointing me at the first two):
- Benetech’s Return on Humanity (ROH) metrics.
- REDF’s Social Return on Investment (SROI), OASIS and RISE metrics.
- DfID’s metrics in the 2011 Multilateral Aid Review.
DfID compared aid agencies on two main value-for-money scales: 1) how much each agency contributed to UK development objectives (it also listed results in terms of the numbers of people helped, items (food, mosquito nets etc) delivered and deaths prevented) and 2) organizational behavior & values (including transparency, value consciousness, drive and partnering). DfID looked intelligently (with help from Alison Evans and Lawrence Haddad) at the value chain from costs and inputs to outputs and (qualitative & quantitative) outcomes – in their own words, “The further we go up the chain, from costs, to inputs, to outputs and then to outcomes, the more difficult it becomes to measure the things that we are interested in. But it is still important to try to do this, because this is what matters: not the number of teachers trained, but the difference this makes to poor children’s life chances. Our approach has therefore been to measure what we can, and look at proxy measures for the rest.”
Most of DfID’s metrics are more applicable to an organization rather than the results that it outputs, but the thinking used to generate these metrics, and the methods used to obtain and weigh evidence could be useful at the results level. One thought is about alignment with goals – for the UN, the high-level question here is “how well do these results align with the Millennium Development Goals”, but there are interesting lower-level questions about goals to be had here too. There’s also a metaquestion – DfID used two main metrics because they were difficult to combine, and our organisation outputs are likely to be comparably complex. Perhaps in development, the answer isn’t a bottom-line single score (e.g. ROI) but a spider graph of three or more (perhaps including ROI to businesses, who are also part of the development ecosystem).
REDF “…blends social and financial returns, includes performance measurement, and creates an “efforts-to-outputs” ratio that can be looked at across enterprises.” Again, this looks like an institutional metric.
ROH is described as “analyzing four factors: social return on investment (financial and social benefits to society), benchmarking (how a program or product compares with other options to solve the same problem), financial sustainability (sufficient customer support to result in financial break even), and sales (dollar volume and number of customers served).” Now this could be promising.
- Multilateral Aid Review, DfID, 2011
- Paris declaration on aid effectiveness, 2005
- Multilateral Organisation Performance Assessment Network, http://www.mopanonline.org/
One thought on “Return on Humanity and other metrics”
Organisations often resort to metrics that measure how much they are what they want to be rather than how well they do their jobs because the latter is, as you suggest, very difficult. Ultimately it is easier to substantiate we are good guys, doing things in good ways than we are doing lots of good things so that’s what people often do. Also, of course, the metric will pretty soon come to alter the behaviour, so a metric that doesn’t cause bad behaviours might be better than one that encourages some good ones but discourages others. Just my two cents.
Comments are closed.