He is not wrong, but there is also a certain danger in this approach that goes unremarked upon. It is often difficult to measure the outcome you actually desire, and a lot easier to measure some proxy for it. But attaching incentives to optimising some metric - making an intermediate goal the goal in itself - tends to make it an extremely unreliable proxy for the original worthy cause.
We know that Gates is aware of this, because he is credited with having once said: "Measuring programming progress by lines of code is like measuring aircraft building progress by weight." We can only hope that he is keeping this in mind; he certainly makes no reference to the problem in this piece.
The Gates Foundation has in the past spent billions driving ruthlessly and with careful measurement toward what turned out to be entirely counterproductive goals. Simply measuring more stuff will not be sufficient to prevent it from doing so again.
Dietrich Dörner, The Logic of Failure (1996) covers this ground very well, including with case studies of people trying to save a fictional third-world country in a computer simulation of an aid program... with uniformly disastrous results.
In general social policy is hard. Really hard. In part due to some of the things you talk about.
While the Gates Foundation did waste time and money on the small schools initiative, but in some ways I view it as a model for how social research should work.
They had a theory and some supporting data. They tried to implement a program to optimize it and measured the results. And while they may have been ruthless in driving this, they were equally ruthless with their measurement. They didn't hide their results. They came out and said, "we are disappointed with our results". http://www.gatesfoundation.org/speeches-commentary/Pages/bil...
If more social research was done in this way and was as transparent I suspect we'd make some better progress in education. But unfortunately, most people have taken sides -- the issue has been politicized. And so outside of groups like Gates, the data no longer matters.
Well, to answer your specific question, see the link in my reply to tptacek.
But I want to clarify that I am not condemning them (and I wasn't trying to use the term "ruthless" in a pejorative sense, so it was probably a poor choice). I am simply pointing out that if you want to use the engineering method to effect change on a global scale - and I think that is absolutely the right approach - then you first need to make sure that it is good engineering. Gates, who by all accounts is an excellent engineer, is (if the quote is genuine) aware of the problems with software metrics. It is less clear whether he is carrying that scepticism over to the use of other metrics, and that is my concern.
> We know that Gates is aware of this, because he is credited with having once said: "Measuring programming progress by lines of code is like measuring aircraft building progress by weight." We can only hope that he is keeping this in mind; he certainly makes no reference to the problem in this piece.
From my experience, it seems the solution the software industry used was very much like putting rocket engines on planes made of lead.
We know that Gates is aware of this, because he is credited with having once said: "Measuring programming progress by lines of code is like measuring aircraft building progress by weight." We can only hope that he is keeping this in mind; he certainly makes no reference to the problem in this piece.
The Gates Foundation has in the past spent billions driving ruthlessly and with careful measurement toward what turned out to be entirely counterproductive goals. Simply measuring more stuff will not be sufficient to prevent it from doing so again.
Some recommended reading material on the subject:
Ritter & Webber, "Dilemma's in a General Theory of Planning" (1973) introduced the concept of "wicked problems" - http://www.uctc.net/mwebber/Rittel+Webber+Dilemmas+General_T...
Dietrich Dörner, The Logic of Failure (1996) covers this ground very well, including with case studies of people trying to save a fictional third-world country in a computer simulation of an aid program... with uniformly disastrous results.