ANOTHER RESEARCH REPORT crosses our desk. This one is about policy experiments, something China undertakes on a scale that exceeds any other country. Some scholars have long argued that China’s policy experiments — and learning from failures — have been instrumental in its economic development over the past three decades.
Shaoda Wang of the University of Chicago’s Harris School of Public Policy and David Y. Yang of Harvard University have reviewed 633 policy experiments at regional and lower levels of government initiated by 98 central ministries and commissions since the 1980s. They conclude that most were unsuccessful. Nonetheless, they found that 45.5% of the tested policies were adopted nationally.
To this Bystander’s untutored eye, that did not look to be such a shabby success rate. However, informed by reading deeper into the paper, all may not be as it seems.
The core findings of Wang and Yang’s research suggest some inherent bias in the set up of policy experiments that may skew the decision about whether to proceed with a policy nationally based on the results of the experiment:
- four times out of five, policy experiments tend to be sited where there are the resources available to help them succeed;
- local politicians can favour allocating those resources to experiments, thus exaggerating their effectiveness, which cannot be repeated nationally because the overallocation of resources is not replicable; and.
- central government accounts for neither of those two factors fully when assessing outcomes.
In short, the experiments may be telling central government officials less than they think. Something to bear in mind, for example, when evaluating the extended property tax experiments now promised.
One intriguing finding is how experiments tend to go to places where local politicians are sufficiently far from retirement and have ample room for upward mobility so that promotion incentives will encourage them to make policy experiments work. Local officials close to retirement will not be so bothered. Political patronage also shapes how ministers choose experimentation sites.
The outstanding question is what quantitative difference does the biases that Wang and Yang have uncovered make to the quality of national policy adoption. Their paper makes it clear that central government should discount the reported success of most policy experiments. In some borderline cases, that would imply more policies should not be adopted nationally than in fact are.
It may be that even skewed experiments bring better outcomes than implementing policies untested. This is not something Wang and Yang address (as they note) but would be a valuable avenue of future research.