Evidence-Based Coaching, or Coaching That Only Feels Evidence-Based

In Dandora, northeast of Nairobi, researchers paired young women running microenterprises with experienced local businesswomen, and over the following seventeen months the mentees reported weekly profits about 20 percent higher than a comparison group, while a business class run alongside left profits flat. As the mentoring matches dissolved, the gain faded with them. Evidence-based coaching for founders starts from both findings. Each one is known because the researchers kept a comparison group and came back to measure it.

Key Takeaways

Randomized trials of small-business mentoring and follow-up support have been run in Kenya and Uganda. In the Nairobi trial, mentoring by an experienced local businesswoman raised weekly profits by about 20 percent, and the effect faded as the mentoring relationships ended.
Allison Piper Kimball lists mentor hours, participant counts and workshops delivered as volume metrics that do not necessarily equate to effectiveness.
The Three Receipts turn a coaching claim into evidence: an action someone other than the founder can check, a comparison group that went without the coaching, and a measurement taken after the coaching stops.
Realistic effects are modest. David McKenzie’s 2021 meta-analysis puts the average effect of business training on profits and sales at 5 to 10 percent, which most single programs are too small to detect.
Program managers can start before the next cohort opens, by rewriting the mentor session log and booking the six-month follow-up now.

What the Nairobi mentoring trial found, and what it lost

Wyatt Brooks, Kevin Donovan and Terence Johnson published the Dandora study in 2018 in the American Economic Journal: Applied Economics. One group of inexperienced women business owners received a formal business course. A second group was matched with an experienced entrepreneur from the same area. A third group received neither, and served as the comparison.

The course changed some business practices among the women who took it. Their profits stayed where they were. The mentored women gained, and the authors traced the gain to localized, specific knowledge from someone in the same trade. Mentees were 30 percent more likely than the comparison group to switch suppliers, and J-PAL’s summary of the trial ties their higher profits to lower supply and input costs.

The second finding came later. As mentor and mentee stopped meeting, the profit advantage shrank, the way a stock of advice runs down once the inflow stops. Among the pairs still meeting a year after the program began, the advantage held. Those pairs chose to keep meeting, though, so their result may reflect the mentees who stayed as much as the meetings.

A second East African trial points the same way from a different angle. Christopher Blattman and colleagues studied a program for war-affected women in northern Uganda that combined $150 in cash, five days of business training and ongoing supervision visits. Incomes doubled after sixteen months. When the researchers later varied the supervision, which accounted for half of the program’s cost, the extra visits kept more businesses alive a year on and left consumption where it was. Follow-up support bought survival at a real price, and measuring both arms let the researchers see both.

Both trials studied microenterprise owners in petty trade, tailoring and food sales, a step away from an incubator founder in Kampala building a product with staff, so their results reach her as the nearest local evidence and a hypothesis to test.

Why mentor hours and satisfaction scores feel like evidence

Incubators often report their coaching in units that are easy to count. A quarterly funder report might say 340 mentor hours, 96 percent of sessions rated useful, and 28 founders completing the track (an illustrative report, invented for this example). Every number is true. Each one counts the program’s activity or the founders’ feelings, and the 28 businesses themselves stay off the page.

Allison Piper Kimball, writing in Data-Driven Decision Making in Entrepreneurship (CRC Press, 2024), names the first kind of number plainly. Programs track volume, such as “the number of mentor hours or workshops provided”, and she notes that volume does not necessarily equate to effectiveness. She adds that UBI Global’s world rankings score incubators and accelerators on factors about the programs themselves, such as jobs created, services offered and mentoring hours offered, so an hour of input counts toward the rank.

Jennifer Wisdom’s chapter in the same book supplies a ladder for the second kind. Following Tian and colleagues, she separates five levels of outcome: the participant’s satisfaction, changes in knowledge and attitude, changes in behavior and practice, improved work results, and organizational results such as retention, performance and survival. A session rating sits on the first rung. A switched supplier sits on the third. A shop’s profit or survival sits on the fifth, where the business itself is the unit, and most coaching reports stop near the bottom of that climb.

Wisdom also flags a confound that every cohort program should read twice. Offering development may raise interest and retention “because of the startup’s commitment to its workers and growth more than the specific content of the training.” Translated to a cohort, founders may stay engaged because someone is paying attention to them. That attention has real value. A claim that the coaching content worked needs evidence that can tell the two apart.

The Three Receipts: a test for any coaching claim

The Three Receipts is a test a program manager can run on a mentor log, a funder report or her own website. A coaching claim counts as evidence when it carries all three.

The behavior receipt. Each session ends with one action a person other than the founder can check within fourteen days: a supplier switched and an invoice to prove it, a price changed on the shelf, a ledger opened and shown to the mentor. Wisdom’s third rung is behavior, and this receipt is how a small program reaches it cheaply.
The comparison receipt. Name the founders who went without the coaching, and track the same numbers for them. Waitlisted applicants and the next round’s applicants are the usual candidates. The Global Accelerator Learning Initiative compares accelerated ventures with rejected applicants for this reason: accelerated ventures grew revenue by about 50 percent in their acceleration year against about 30 percent for those turned away. Funding growth varied widely by program, and in some programs the participants raised less than the rejected pool. The comparison can also be an alternative intervention. Stephen Anderson and David McKenzie found in a Nigerian trial that linking firms to finance and marketing skills, by insourcing a worker or outsourcing tasks to a professional, did better than business training at improving business practices and at least as well as consulting at half the cost. A coaching track can fairly be compared with paying for a bookkeeper.
The persistence receipt. Measure again six months after the mentoring ends. The Dandora gain faded with the relationship, and the Ugandan supervision visits bought survival. A program that measures only at graduation reports the peak and misses the decay.

The behavior receipt tells advice given from advice taken. The comparison receipt matters most where selection is strong: a program that picks founders for promise will see growth whether or not its coaching helped, and only the comparison can sort the two. The persistence receipt catches the lift that lasted only as long as the mentor kept visiting.

Where the American evaluation model stops working in Kampala

Both chapters picture a startup with employees inside an American support system. Wisdom names U.S. federal agencies such as the National Science Foundation among the providers of cohort programs, and the exit ranking Kimball reports is topped by Techstars Boulder and Techstars Seattle. Two of their assumptions break in an East African cohort.

The first is infrastructure. Wisdom’s measurement plan uses pre-post tests, staff performance reviews, retention metrics and, where possible, external evaluation consultants. A Kampala founder running a small shop is usually the whole human resources department. Informal trading is common too. In the drop-out split we published from 99 alumni, in Who Actually Finishes, a quarter told us they were running an informal business, a figure for those alumni alone. The upper rungs therefore have to be read from the shop’s own cash book, its sales and whether it is still trading, which is what the Nairobi and northern Uganda trials measured.

The second is the outcome. One comparison Kimball reports, Beta Boom’s 2021 ranking, scores accelerators by exits drawn from Crunchbase, with the top five reaching exit rates of 19.3 to 24.7 percent. Exits are rare events in a Ugandan or Kenyan cohort of micro and small firms, so an exit rate would read as zero for almost every program. Profit, sales and survival are the honest East African outcomes.

There is a third gap that runs in our favor. Wisdom calls the research linking startup leaders’ program engagement to outcomes “mixed, primarily because of heterogeneity in program aims, small sample sizes of studies, and lack of uniform definitions.” The studies she cites for professional development working at all come from teachers, human services professionals, physicians and nurses, and her survey of startup training needs comes from Russia. Our own region holds randomized evidence on small-business training and mentoring, including a Kampala study by Glaub, Frese and colleagues in which owners given a personal-initiative training of under a month ended with higher yearly sales than a randomized control group. An East African program manager has more local evidence to work from than the handbook suggests.

The strongest case against measuring coaching this way

Angus Deaton and Nancy Cartwright have argued that the trust placed in randomized trials outruns what they deliver. A trial’s estimate applies to its own sample, often a convenience sample, and extending it to another group needs its own justification. Their sharpest line is that “at best, an RCT yields an unbiased estimate, but this property is of limited practical value.” On this view, a Dandora result from a particular year carries over to a Gulu program only with a separate argument for why Gulu resembles Dandora.

The concession is real and should be made in full. A 30-founder cohort has too few businesses to detect the 5 to 10 percent average effect McKenzie reports for business training, and its comparison group of rejected applicants differs from the coached founders in ways the program chose. Deaton and Cartwright are right that a borrowed trial result is a hypothesis about your cohort, and that your cohort is too small to settle it.

What survives the critique is the direction of travel. A local comparison, read honestly as a rough signal across several cohorts, beats an activity count read as proof. The Three Receipts supply a modest share of the careful reasoning about context the critique asks for, because the comparison comes from the same city and applicant pool as the coached founders.

What a coach is for after the program ends

The persistence receipt exposes a gap the measurement frame can name and cannot close. Every trial above wants gains that outlast the coach, and its instruments are built to record the fade. Christian discipleship carries a teaching pattern built to design against it. Paul writes to Timothy, “the things you have heard me say in the presence of many witnesses entrust to reliable people who will also be qualified to teach others” (2 Timothy 2:2). That is four generations of learners in one sentence, and it designs for the teacher’s absence from the first day. A mentoring track built on that pattern would ask each mentee, before graduation, to teach one peer the specific thing her mentor taught her, such as the cheaper supplier or the new pricing rule. That act is also a behavior receipt, and a cohort can test whether it slows the fade with the same three receipts. Stewardship of a funder’s money and a founder’s time asks for that kind of testing.

Our own public figure, 600+ founders trained, is a volume count of the kind Kimball describes, and it sits below the first rung of Wisdom’s ladder. We report it as reach. The next step for any program, ours included, is to put the three receipts beside it. The accelerator that learns from its own cohorts is the program-level version of the same discipline.

Frequently Asked Questions

Does business mentoring actually increase profits?

In a randomized trial in Dandora, Nairobi, mentoring by an experienced local entrepreneur raised women business owners’ weekly profits by about 20 percent over seventeen months, while a business class left profits flat. The gain faded as mentoring relationships ended, so the evidence favors mentoring that continues. Whether mentees passing lessons to peers slows the fade is still untested.

How can a small incubator measure coaching outcomes without a randomized trial?

Use the Three Receipts. End each session with an action someone else can check within fourteen days, track waitlisted or rejected applicants on the same numbers, and measure again six months after mentoring ends. Read the results across several cohorts as a direction, since one small cohort cannot settle the question alone.

Why did the Kenya mentoring effect fade?

The researchers linked the gain to local, specific advice from an ongoing relationship, such as which supplier to buy from. As meetings stopped, the profit advantage shrank, while pairs who chose to keep meeting a year on kept it. In northern Uganda, similarly, extra supervision visits kept more businesses alive a year later.

Is classroom business training a waste of money?

Classroom training changes some practices, and a 2021 meta-analysis by David McKenzie estimates average gains of 5 to 10 percent in profits and sales. The issue is size: most programs are too small to detect effects that modest, so claims about any single course should be made carefully and checked against a comparison group.

What effect size should a coaching program expect?

Expect modest effects. The nearest benchmark is business training, where McKenzie’s 2021 meta-analysis finds average gains of 5 to 10 percent in profits and sales; one Nairobi mentoring trial found about 20 percent while matches lasted. A program reporting far larger gains without a comparison group is probably measuring the founders it selected.

Next Step

If you run a cohort program or fund one, book a strategy call with Joshua before your next intake opens. In one hour we will rewrite your mentor session log around the Three Receipts and name your comparison group, so your next funder report carries checked actions beside the hours, and the report after the six-month follow-up carries outcomes.

Sources and Evidence

Brooks, Donovan and Johnson, “Mentors or Teachers? Microenterprise Training in Kenya”, American Economic Journal: Applied Economics 10(4), 2018: the Dandora trial, the 20 percent profit gain, the fade and the flat result for the business class.
J-PAL evaluation summary of the Dandora mentoring trial: profits over seventeen months, supplier switching and input costs, the fade as matches dissolved, and the pairs still meeting after 12 months.
Blattman, Green, Jamison, Lehmann and Annan, “The Returns to Microenterprise Support Among the Ultra-Poor: A Field Experiment in Post-War Uganda”, NBER Working Paper 21310, published in American Economic Journal: Applied Economics 8(2), 2016: the package for war-affected women in northern Uganda, doubled incomes, supervision at half of program cost raising survival.
McKenzie, “Small business training to improve management practices in developing countries: re-assessing the evidence for ‘training doesn’t work'”, Oxford Review of Economic Policy 37(2), 2021: 5 to 10 percent average effect, the power problem, and localization and mentoring as ways to improve training.
Global Accelerator Learning Initiative, “Initial Insights from GALI”: accelerated versus rejected applicants, 50 versus 30 percent revenue growth.
GALI press release, “Startup Accelerator Programs Work, New Report Finds”: wide variation in funding growth across programs.
Anderson and McKenzie, “Improving Business Practices and the Boundary of the Entrepreneur: A Randomized Experiment Comparing Training, Consulting, Insourcing, and Outsourcing”, Journal of Political Economy 130(1), 2022: the Nigeria comparison of training, consulting, insourcing and outsourcing.
Glaub, Frese, Fischer and Hoppe, Academy of Management Learning & Education 13(3), 2014, IDB Skills Bank entry, doi:10.5465/amle.2013.0234: the randomized personal-initiative training with Kampala business owners, under one month long, and yearly sales against the control group.
Deaton and Cartwright, “Understanding and misunderstanding randomized controlled trials”, Social Science & Medicine 210, 2018: the hostile witness.
Blacksmith and McCusker (eds.), Data-Driven Decision Making in Entrepreneurship: Tools for Maximizing Human Capital, CRC Press, 2024, doi:10.1201/9781003196853: chapters by Jennifer Wisdom (ch. 4) and Allison Piper Kimball (ch. 11).
2 Timothy 2:2, New International Version: the faith paragraph.