Failure is an integral part of learning: the adage “If at first you don’t succeed, try, try, try again” was originally coined to encourage children to persevere with their school work (Palmer, 1840). In the same way that we expect people to fail during the learning process as they gain new skills, theoretically we accept that failure is part of the scientific process – the quickest way to disprove a hypothesis is for an experiment, which tests that hypothesis, to fail to give the expected result.
In principle then, academia should embrace failure: it is a community built on the twin pillars of education and research using the scientific method, with failure playing a vital role in both. In reality , though, those who pursue research careers are often people who are used to being the “smart” ones, those who did not experience much failure at school. In a competitive environment where most people are trying to prove their credentials to progress in (or keep!) their job, admitting to less than perfect outcomes is hard. And the less we talk about failure, the more stigmatized it becomes.
In international development, refusal to talk about failure hides a multitude of sins. We regularly see news stories, flashy videos, and research papers about the early development of technologies that promise to solve a problem in developing countries. Far too many of these quietly disappear when tests start to show that results are not as positive as first thought. This biased dissemination of evidence relating to a technology results in a one-sided view of its performance, which can result in organizations wasting time and money by trying to scale up technologies that are not fit for purpose. Take the PeePoo bag as an example. This “self-sanitising single use biodegradable toilet” looked technically promising at small scales (Vinnerås et al., 2009; Patel et al., 2011) and early results were well-publicized (Bhanoo, 2011; Ali, 2013). However, this success did not translate into large scale trials or bring significant change to sanitation in urban slums. Clearly, something went wrong, though it is not obvious what. After all, the official website (www.peepoople.com) is still functional (though it has not been updated since 2016), suggesting the PeePoo bag may still be a viable development strategy for sanitation in urban slums.
In the field of water, sanitation, and hygiene (WASH) research and practice, the most common mode of failure is when an intervention, designed to improve lives in some way, does not provide the intended benefits. According to the scientific method, this is simply a null result. As long as the research has been performed rigorously, this is actually a successful research trial, because we learned something, even though we produced a result other than the one we were expecting. However, the researcher who wants to report that result faces pressures from multiple angles. First, interventions that do not achieve their aim are not necessarily zero harm and the researcher may be unwilling to publicly acknowledge the detrimental impacts of their work on the lives of purported ‘beneficiaries’ (Barrington et al., 2017). For those who are willing to make that step, there may be additional pressures from funders and project partners. Those who funded, developed, or implemented the intervention may have a vested interest in it succeeding, and disincentives for admitting it did not work. This could be the technology developer who stands to lose funding for the roll-out of a product, local political leaders who had staked their re-election on the success of a trial, or a funder who does not want to admit that funds were spent on the ‘wrong solutions’. Even if the researcher manages to convince all parties of the benefits of sharing their findings, they may still face barriers to publishing. Despite having done everything ‘right’, journals (particularly those with the highest impact) often prefer not to publish null results because those are not generally considered sufficiently novel. With all of this in mind, it is no surprise that for many, the effort put into writing up failures and dealing with the fallout is not worth the limited rewards.
This reluctance to discuss ‘failed’ research causes problems for research and problems for researchers. If we do not openly discuss research failures, we increase the likelihood that other research will repeat the same study and discover the same null results. Repeating studies costs time, money, and in some disciplines, runs the risk of costing lives. For researchers, particularly those early in their research careers, not discussing failures gives a false view of how science works, namely that every piece of research will be accompanied by a ‘eureka!’ moment. This in turn heightens the feeling of ‘imposter syndrome’ among researchers – the fear that they are not good enough to be an academic or a researcher, and that it is only a matter of time before others discover that they are a fraud because their hypotheses are often proven wrong.
With that in mind, it takes a degree of bravery to talk about failures in WASH. There are countless examples of interventions that turned out to be inappropriate, such as the PlayPump, a roundabout that was connected to a water pump so that children’s play could be harnessed to deliver clean water to communities across Africa (http://www.playpumps.co.za/). The shortcomings of this solution are well-documented (Borland, 2011). The joy of roundabouts is that they spin freely, so the PlayPump was not as much fun to use as you might think, requiring constant effort to keep it moving. This meant that in many places, children did not play on the roundabout for as long as was needed to ensure an adequate water supply for the community. Instead, the task of pushing the roundabout fell to women collecting water, when to them a hand pump would have been much preferred and required considerably less physical effort. Often (and in a surprisingly common theme for the sector) communities were not consulted on the water problems they faced before the PlayPump was selected as a solution, so even when sufficient children played on the roundabout to pump water, the PlayPump did not address issues of water scarcity or water quality. The PlayPumps also raised ethical questions about the blurring of the line between work and play. Many of the installed PlayPumps have since been abandoned or retrofitted with hand pumps, and the original backers of the technology have admitted that their extensive plans for roll-out were overly ambitious. Although exact numbers are hard to find, it is generally accepted that ‘millions of dollars’ were spent on PlayPumps, each of which cost as much as four conventional wells with hand pumps (Chambers, 2009).
In some respects, wasting money is the least of concerns in the international development sector. In WASH, there are documented cases of sanitation behavior change interventions causing harm, with those who defecate in the open being punished by others throwing stones at them (Devine, 2009) or dumping feces in their kitchens (Chatterjee, 2011). It is of little surprise that some of these interventions result in negative impacts on physical health and social cohesion (Barrington et al., 2017). In these scenarios, the people that we are trying to help end up in a significantly worse situation than before we stepped in.
One of the challenges with WASH interventions is that water and sanitation form part of complex systems that include people, technology, and the environment. Too frequently, interventions focus on only one aspect of that system, leading to unintended consequences elsewhere. There are recorded cases where individuals have died or been injured from falling into pit latrines, because the vital role of maintenance has been neglected or overlooked. In 2014, a five-year-old boy drowned in a school pit latrine in South Africa that collapsed under him during his first week of school. In 2018, the incident was repeated with a five-year-old girl drowning in another school pit latrine. This led to the government insisting on the eradication of pit latrine toilets from South African schools, but with little detail of what would replace them in water scarce areas where waterborne sanitation was not an option (Fihlani, 2018). Though the government understood that this was a catastrophic failure of the system, they were uncertain how to quickly reshape the system to ensure a safer service in the future.
Clearly, talking about failure has its advantages. In WASH research and practice, repeating failed interventions can lead not only to money spent on inappropriate implementation, but can also harm the intended beneficiaries, in the very worst cases resulting in death. How can we morally justify not discussing our failed ideas if sharing them can save money and lives?
Sharing negative outcomes of research helps to ensure that mistakes are not repeated. Anyone who has undertaken health and safety training knows that for every death that happens, there are numerous injuries and many more ‘near miss’ events. The reason that health and safety officers are so insistent on ‘near miss’ reporting is that this provides valuable data to understand how to prevent more serious incidents from occurring. Research, particularly in fields where our work impacts on people’s everyday lives, needs to take more heed of that approach. Talking about failure should be a vital aspect of research transparency.
There are global campaigns to push for all medical trials to be registered in advance so that negative results are captured (Goldacre & Gray, 2016). Proponents point out that this allows researchers to look at all the evidence relating to a drug, rather than only the pieces of information that support the pharmaceutical industry’s desired results. Perhaps international development needs to take a similar approach.
Convincing researchers to share failures is only the first hurdle, however. Funders are powerful players in any research partnership, and whilst some funders do share projects that do not turn out as planned, it is not common. One of the advocates for funders to share negative outcomes as well as positive outcomes, so that lessons can be learned and carried forward, is Elhra. In 2017, they funded Action Against Hunger Spain and the London School for Hygiene and Tropical Medicine to test if Moringa leaves could be used as an alternative to soap in Ghana, following successful laboratory trials. These new tests showed that, rather than acting as a disinfectant, when mixed with water the leaves provided the perfect breeding ground for E. coli. These results are easily accessible on the Elhra website (Action Against Hunger Spain, 2018). Whilst Elhra offers one positive example of how funders can share negative outcomes, there are many more organizations with much larger research budgets that informally prefer negative results to be quietly left out of reporting. As such, there is still more to be done in identifying convincing arguments for funders to openly share failures to promote learning.
Yet, as stated above, the bias associated with which results are published and which are not does not lie solely with the researchers and project partners. Journals want to publish novel research and failure is too often not seen as novel. This culture of success-ism is damaging to research and is highly unscientific given that the scientific method explicitly makes room for failure. The drive towards discussing failures needs to come from journals as well, not just from researchers. Open Science pushes for all publicly funded research to be made publicly available, and that should hold true even when research gives null results or leads to failure. If this were the case, researchers would not be pushed to dredge data for “significant results”; publishing null results would be accepted practice. Data dredging is seen as one of the main causes of non-reproducible results and so opening up academic publishing to failures and null results would support the move towards more reproducible and more transparent science.
If we know what fails, we have a better chance of working out why so that we can succeed in future. Failure, particularly in complex systems such as those encountered in international development, is a vital part of the puzzle for achieving impactful research; the consequences of attempting to solve the puzzle without all the pieces can be extraordinarily high. With this understanding of the importance of publishing failures, we hope that this will one day be the norm. In the meantime, we encourage all researchers to find ways to talk more openly about failure, within projects and to a wider audience. As the founders of ‘WASH Failures’, we have used game show style events at conferences to give people space to talk about failures in their work and are carrying out research to understand how organizational culture can impact how failure is communicated in water and sanitation organizations working in sub-Saharan Africa. We need more structured ways to report on failures – at conferences, in academic writing, in reports to funders, and with the general public. We were pleased to see that the Journal of Trial and Error is one of those nascent outlets for failure. We certainly hope that it will not be the last.
Action Against Hunger Spain. (2018). Moringa oleifera efficiency as handwashing product in Ghana. Elhra. Action Against Hunger Spain. https://www.elrha.org/project-blog/moringa-oleifera-efficacy-handwashing-product-ghana/
Ali, L. (2013, January 22). Tackling the ‘flying toilets’ of Kibera. Aljazeera. Aljazeera. https://www.aljazeera.com/features/2013/1/22/tackling-the-flying-toilets-of-kibera
Barrington, D. J., Sridharan, S., Shields, K. F., Saunders, S. G., Souter, R. T., & Bartram, J. (2017). Sanitation marketing: A systematic review and theoretical critique using the capability approach. Social Science & Medicine, 194, 128–134. https://doi.org/10.1016/j.socscimed.2017.10.021
Bhanoo, S. N. (2011). The PeePoo, a biodegradable toilet for the developing world. https://www.nytimes.com/2011/09/27/health/27toilet.html
Borland, R. (2011). Radical plumbers and PlayPumps – objects in development [PhD Thesis, Trinity College]. Trinity’s access to research archive. http://hdl.handle.net/2262/89797
Chambers, A. (2009, November 24). Africa’s not-so-magic roundabout. The Guardian. https://www.theguardian.com/commentisfree/2009/nov/24/africa-charity-water-pumps-roundabouts
Chatterjee, L. (2011, June 9). Time to acknowledge the dirty truth behind community-led sanitation. The Guardian. https://www.theguardian.com/global-development/poverty-matters/2011/jun/09/dirty-truth-behind-community-sanitation
Devine, J. (2009). Introducing SaniFOAM: A framework to analyze sanitation behaviors to design effective sanitation programs. Water and Sanitation Program. https://www.wsp.org/sites/wsp/files/publications/GSP_sanifoam.pdf
Fihlani, P. (2018, June 3). South Africa’s school pit latrine scandal: Why children are drowning. BBC News. https://www.bbc.com/news/world-africa-44329712
Goldacre, B., & Gray, J. (2016). OpenTrials: Towards a collaborative open database of all available information on all clinical trials. Trials, 17(164), 1–12.
Palmer, T. H. (1840). The teacher’s manual: Being an exposition of an efficient and economical system of education, suited to the wants of a free people. Lyon and Webb.
Patel, D., Brooks, N., & Bastable, A. (2011). Excreta disposal in emergencies: Bags and Peepoo trials with internally displaced people in Port-au-Prince. Waterlines, 30(1), 61–77. https://doi.org/10.3362/1756-3488.2011.006
Vinnerås, B., Hedenkvist, M., Nordin, A., & Wilhelmson, A. (2009). Peepoo bag: Self-sanitising single use biodegradable toilet. Water Science and Technology, 59(9), 1743–1749. https://doi.org/10.2166/wst.2009.184