Toby Young and the power of education

Toby Young is in the news again. The Tories’ favourite born again educationalist has been given a seat on the board of Jo Johnson’s new universities “regulator”, the Office for Students. This gives me an excuse to publish something I failed to put out last time he was in the news, for claiming that schools can’t do much to reduce inequalities in attainment.

Toby Young has been at the centre of some controversy about the ability of schools to help disadvantaged children. He wrote an article for Teach First arguing that schools can’t really achieve much, which the charity subsequently took down because they disagreed with it. The article is now published on Toby’s blog, and he has been anointed by some as a free speech martyr (although he very modestly says that “martyr is putting it a bit strongly”).

But what about that article? Is it right?

There are a few different threads to it, including Toby’s usual futurology about IQ-enhancing drugs, but the central claim about the efficacy of schools is based on research that attributes variation in GCSE results to different causes. According to Toby, this research finds that IQ accounts for 60-70% of the observed variation in results, differences between schools (such as funding, class size and quality of teachers) account for 10% and the other 20-30% is accounted for by other environmental factors.

I don’t know this research so I’ll leave it for others to debate whether it’s any good and whether Toby is describing it correctly. The results are presumably from a multiple regression of observational data, so the usual caveats about causation versus correlation and unobserved variables will apply. But let’s set that to one side and take the results at face value: what do they mean for schools policy?

The conclusion Toby draws is a tempting one: that schools can’t do much to ameliorate the effects of inequalities. I think that’s the wrong conclusion to draw from these numbers for three reasons, which I’ll address in order of increasing complexity.

The first is trivial: reducing inequalities in attainment by 10% sounds like a major achievement to me. We should do this! (In fact it may be slightly unfair to suggest Toby is arguing otherwise.)

The second is more subtle and requires us to think about what those numbers actually mean. 10% of the observed variation in GCSE results is accounted for by the observed variation in school characteristics. So if we were to equalise all schools on these characteristics (things like funding, class size and quality of teachers) then variation in results would reduce by 10%. If the only intervention we could possibly make in schooling was to equalise these things across schools, then we could only eliminate 10% of current variation in attainment. But this isn’t the only thing we can do. What if we made schools in deprived areas better than those in more affluent areas? What if we gave additional help to the children who face the greatest disadvantages at home? The 10% figure tells us nothing about the efficacy of these things.

The third also relates to the way that 10% figure is constructed. It’s the variation attributed to differences between schools divided by total observed variation, so it’s a function of three things: how big the differences are between schools, how strong an effect school differences have on attainment, and how much variation there is from other sources (like home environment and IQ). So we can’t just look at the 10% figure and say that’s a small number so schools can’t have a strong effect on children’s attainment. If our schools were much more unequal in funding and class size then this number would go up, while if they were identical on these measures it would go down to zero – but these changes would tell us nothing about the power of education to drive attainment. If we were able to reduce variation due to other environmental factors (say, by reducing income inequality between the families of schoolchildren) then the 10% schools figure would increase. This would not mean that schools had become more effective at driving attainment.

So taking all of this into account, what can these figures tell us about schools policy? What would we do differently if this figure were 50% instead of 10%? It seems to me that the answer to both these questions is “very little”. Either way we should make sure that already-disadvantaged children don’t end up in schools that have fewer resources, larger class sizes and worse teaching, since these factors do compound their disadvantage. Either way we should consider helping disadvantaged children with targeted policies, about which these numbers tell us nothing. Where interventions cost money, we will want to know if the effect size is large enough to justify that expenditure, but these numbers tell us nothing about the absolute effect size of school interventions.

In fact, it’s hard to conclude that these numbers are much use at all for policy. The nature versus nurture debate has become an ideological battleground, but its relevance to education policy seems very limited. We can’t control nature, but we can decide how to nurture our children. We can do that by designing, testing and implementing good education policies. Whether in the end these policies explain more or less of the population-level variation in attainment than genetics is rather beside the point, if they are effective and cost-effective at improving attainment and reducing inequalities. It’s easy to see this if you consider an example from another policy area: if you have a demonstrably cost-effective behavioural intervention that reduces the chances of high-risk people developing cancer, but I tell you that only 10% of the observed variation in cancer risk is related to behaviours, does that have any bearing on whether you implement your policy?

What the shared parental leave survey actually tells us

A recent survey, conducted by My Family Care and the Women’s Business Council, asked employers about take-up of shared parental leave in the UK. It has led to much hand-wringing. The hands of the mainstream press were the first to be wrung: why have only one percent of men taken up shared parental leave? What could have led to such a catastrophic policy failure?

For most men, the answer is of course that they haven’t had a baby in the past year. The majority of employers responding to the survey weren’t able to identify how many of their male employees had become fathers, so the take-up rates were presented as a proportion of all male employees – and widely misinterpreted by the press.

And so the hand-wringing quickly spread to more numerate commentators, who rightly bemoaned this as one of the worst cases of statistical misreporting in living memory. Factual corrections were made to some of the offending articles, but the headlines still pronounce the policy a failure – the proportion of men opting for shared parental leave is “tiny”.

*           *           *           *           *

This begs a question: tiny compared to what? Sure, 1% of men is tiny compared to all men, but that isn’t the relevant comparison. If 1% of men were to die tomorrow from a mysterious plague, tiny would be a rather inappropriate description of events.

A more useful comparison is with the proportion of male employees who have had a baby in the past year. Although this information isn’t readily available, we can make a reasonable estimate. On BBC Radio 4’s More or Less, Tim Harford gives us a back-of-the-envelope figure of around 5% – which seems to be the total number of babies born divided by the total number of male employees. With a larger envelope, we can do a bit better.

Tim’s estimate is probably higher than the real figure, since not all men who have babies are in employment. What’s more, both the likelihood of having a baby and the likelihood of being in employment are related to age. The ONS has data on both [i].

If we mash these numbers together, it looks like around 3.5% of male employees had a baby in 2014. So how does that compare to the results of the survey? Well, although the survey was widely reported as saying that 1% of men took up shared parental leave, these are the only numbers I could find in the report:

A quarter of employers didn’t know what proportion of men had taken shared parental leave – which seems strange in itself. But let’s ignore this group and look in a bit more detail at those employers that did respond. A histogram seems like a more informative way to look at the data.

Of the employers that could provide a figure, more than half said that not a single male employee had taken shared parental leave. This seems a worrying statistic. But we need to remember that there will be random variation between employers, particularly when we are looking at small companies. 15% of the employers surveyed had fewer than 50 employees. If around half of their employees are male, that’s fewer than 25 men per employer. A rough calculation [ii] suggests that, in any given year, more than half of these employers wouldn’t have a single man eligible for shared parental leave. Larger organisations are more likely to have new fathers in their workforce, but there will be a few that don’t.

Even if take-up were 100% among new fathers, we would expect around 10% of the employers surveyed to report that no male employees took shared parental leave. If the take-up rate were 10%, we’d expect over 45% of employers to return a zero.

*           *           *           *           *

So what should we make of these results? Overall, it is hard to defend the negative tone of the coverage of this survey. Clearly there are a lot of fathers not taking shared parental leave, but a decent number are. For a newly introduced policy, this could be considered a modest success.

But even accounting for random variation, the number of employers reporting no men taking shared parental leave seems high. On the other hand, a non-negligible number report quite high rates. This suggests systematic differences between organisations – perhaps some companies or professions are more accepting of fathers taking time off than others.

That said, what we should really conclude from this exercise is that we need better data. The published analysis of the survey is inadequate, but I suspect there is richer data lying behind it. My Family Care and the Women’s Business Council should release this data publicly so that others can analyse the impact of the policy in more detail. And if the government is serious about implementing policies to improve gender equality in the workplace – and the home – then it should monitor the impact of these policies properly, rather than leaving it to third parties to conduct flawed surveys.

 

---

[i] The data I’ve used on employment rates is for both sexes. I’m sure the ONS has the data for men only, but I simply cannot spend any more time searching for data on their atrocious website. By using data for both sexes I’ve implicitly assumed that the age profiles of the male and female workforces are the same. In reality, I guess the female workforce might be younger (with some women in their 30s and 40s dropping out of work to raise a family). If so, my estimate for the proportion of men in the workforce having babies will be a bit too high.

[ii] In case you’re interested, I’ve modelled this as the sum of a set of binomial distributions with p=0.035 and n=the size of the employers who responded to the survey.

 

---

UPDATE

Via @wonkypolicywonk, it seems that the survey *did* actually collect some data on the proportion of new fathers who took up shared parental leave. As well as asking employers, they also spoke to “over 1000” employees and found out that, of male employees that had a baby in the past year, around a third took up shared parental leave.

This is hidden away at the back of the report, which instead takes its headline figures from a survey of employers. There is some justification for this. The survey of employees was very small: 1000 employees would probably equate to around 18 new fathers, of whom six have taken up shared parental leave. This is not a big enough sample to base any conclusions on, but you would think that these results might have led the researchers to consider whether it was misleading to say that “the overall take-up of SPL is still very low, i.e. less than 1% of men have engaged”.

Meanwhile, the sample of 200 employers contained some large organisations, with a total workforce (by my estimate) of over 200,000. We would expect this to include more than 3,500 new fathers, so this sample could the basis of some better analysis. It is quite plausible that analysis might draw a similar conclusion to the small sample of parents: a take-up rate of around a third among new fathers is entirely consistent with 1% take-up among all male employees, provided our estimate that 3.5% of male employees become fathers each year is about right.

But this analysis hasn’t been done, and it seems unlikely that it will be unless My Family Care and the Women’s Business Council release the survey data for others to analyse.

 

Three golden rules for discussing progressivity

What does it mean for a policy to be progressive? The way this question is addressed by the media (and often by government) can be infuriating. It’s got to the point where I am tempted to say the term should be banned, but instead I am going to make one last attempt to clarify it by proposing three golden rules.

I was reminded of this issue when reading Jo Maugham’s analysis* of the impact of the new “social care precept”. This is essentially a £2bn rise in council tax, a significant proportion of which will be paid by poorer households. Here are the figures that Jo gives:

So is this tax progressive or not? Well, rich people pay more, and for some that’s good enough. Fraser Nelson, for example, likes to point out that “the top 3,000 taxpayers in Britain stump up more income tax than the lowest-paid 9 million”. It is more common to look at how paying tax affects the living standards of different groups by comparing tax paid as a proportion of income. By this measure, council tax is regressive.

This is as far as the discussion usually goes. But both of these comparisons have no basis in reality – unless that reality involves collecting these taxes and throwing the money in the sea. The fact is that this money will show up somewhere else, either as increased spending or lower taxes. Which brings me to my first golden rule: the distributional effect of a policy change can’t be assessed without looking at both sides of the equation.

Since this is called the “social care precept”, we might think that it will lead to increased spending on social care. It’s not easy to find numbers on how social care spending is split between income groups, but modelling done in 2011 for the Dilnot Commission (see figure 11 here) made some estimates. Reading the numbers off the chart, I get something like this.

Social care spending is more heavily weighted towards poor people than council tax collection, so lower income groups make a net gain from this policy. That is, the introduction of this policy increases the total amount of redistribution that the government does, which is the only sensible definition I can think of for the word “progressive”, with reference to a change in policy.

But is higher spending on social care really the effect of this policy? That is, if it weren’t for the social care precept, would we see lower social care spending? You could argue that social care spending is going to rise either way, since we’ve got more old people than ever. If it’s not paid for by council tax rises it will be paid for by higher taxes elsewhere, higher borrowing, or cuts to other services.

So this is my second golden rule: identify a realistic counterfactual. We need to know whether the policy leads to more redistribution than what would otherwise have happened. That “what” can have a huge impact on how we view the distributional consequences.

Let’s say we believe that without the social care precept higher social care spending would have to be funded through an increase in income tax – or perhaps higher borrowing now, funded by future increases in council tax. As Jo points out, income tax is much more targeted on rich people than council tax. Here’s the net effect of raising £2bn through council tax instead of income tax.

This policy change would give money to the richest 20% at the expense of everyone else. I think we can all agree that’s regressive. So depending on the counterfactual, the social care precept is either highly progressive or highly regressive. Take your pick. We need to decide which counterfactual is more realistic. In this case, the first one is probably closer to the truth (raising income tax and borrowing more are not top of this government’s agenda) so I’d argue that the policy is probably progressive.

But there’s another more fundamental question here: how much redistribution do we want? Requiring all policy changes to be “progressive” implies that we think we don’t currently have enough. But at some point, if we were to go on increasing redistribution, we’d have too much. Views will differ wildly as to what the optimal level is, but in principle there must be one. I imagine few people think that there should be no redistribution and few think that redistribution should fully equalise living standards.

And even if we want more redistribution, we might not care if a policy is regressive if its effect on the overall level of redistribution is small and it achieves some other aims. The state does not exist solely for the purpose of moving money from the rich to the poor. My third golden rule is therefore this: the overall level of government redistribution is what matters. We need to know whether we want to increase or decrease it, and how much we care about small changes relative to other policy aims.

These rules really are essential. It is impossible to say anything about whether a policy change is progressive without considering both sides of the equation and being clear about what you think the alternative scenario is. It is impossible to know whether you actually want the policy to be progressive, or whether you care, without considering the overall level of redistribution.

These are not new insights and many organisations (the IFS, the OBR, sometimes even the Treasury) are quite diligent about doing distributional analysis properly. But much of the public discussion of “progressivity” fails to follow a single one of these rules, giving us a rather poor standard of debate on the state’s role in reducing inequalities.

---

* I’m not picking on Jo’s analysis because it is a particularly egregious example. On the contrary, I am picking on it because is one of the better examples of someone tackling this question, so Jo has done most of my work for me.

The UK does not spend a disproportionate amount on benefits

OK, I know I’m a bit late to this party, but even by the standards of government publications, this paragraph from George Osborne’s recent Summer Budget (section 3.4) is a stinker:

However, despite progress during the last Parliament there is still more to do. Taxpayers are still being asked to pay for welfare expenditure that remains disproportionately high. 7% of global expenditure on social protection is spent in the UK, despite the fact that the UK produces 4% of global GDP and has only 1% of the world’s population. As chart 1.14 shows, spending on working-age welfare has increased significantly in real terms over the last few decades. Too many families continue to be trapped on benefits. The Budget sets out the next stage of welfare reform, delivering on the government’s commitment to save £12 billion from the working age welfare bill.

I’m not sure where these figures came from, but there are very plausible. It would in fact be surprising if the UK did not spend a “disproportionate” amount on social protection (broadly speaking, public pensions, social care and benefits) when compared to the whole world. Many people living in developing countries are in desperate need of social protection, either because they are too ill to work, there are no jobs available, or for any number of other reasons. But they don’t have access to it. The governments of these countries may be corrupt and not care about the needs of large proportions of the population, or they may simply lack the infrastructure needed to collect enough taxes to provide meaningful social protection. Western countries are much richer than the world as a whole and have much stronger institutions, so they are able to spend more than the global average on social protection. This is a good thing.

So it is a nonsense to compare the UK to the global average. This is the wrong comparator group and sets a laughably low ambition for what a government can do to enhance the welfare of its citizens. It makes more sense to compare the UK to other OECD countries.

The figures look rather as you would expect. The UK spends a smaller share of its wealth on social protection than almost all other rich European countries. If we are spending a disproportionate amount on this, spare a thought for the poor French! The Summer Budget figures imply that they account for 11% of global social protection spending and only 4% of global GDP.

There is one country that clearly bucks this trend. The US is a very rich country (GDP per capita is nearly 50% higher than in the UK) but it spends relatively little on social protection. However, while the US is a great country that UK would do well to emulate in many areas, it is not a leading light in the field of social protection. To take a random example, women in the US get precisely zero paid maternity leave.

To summarise: the UK does not spend a disproportionate amount on social protection. In fact, we spend less than most comparable countries in Europe. Nonetheless, the US shows that it’s possible to spend a smaller proportion of GDP on these things, but we might have to become 50% richer and cancel maternity leave to achieve it.

Reasonable arguments can be made for reforming parts of our benefits system, but this is one of the weakest and most disingenuous I have seen. While it is careful to stop short of telling an outright, falsifiable lie, its purpose is clear: it is seeking to mislead the reader. Although it removes the risk of that gotcha moment when a full-blown lie is exposed, seeking to mislead is morally equivalent to lying and politicians should be called out for it more often.