A pioneering new study published in the journal Science has analysed the effectiveness of 1,500 climate policies documented in a new OECD global climate policy database.
Led by Climate Econometricians at the Potsdam Institute for Climate Impact Research (PIK), the Mercator Research Institute on Global Commons and Climate Change (MCC) and the University of Oxford, it is the first time a global dataset of climate policies has been compared and ranked in this way.
Go deeper with GlobalData
The results, however, are sobering. Across four sectors, 41 countries, two decades and 1,500 policies, only 63 successful policy interventions with large effects were identified, which reduced total emissions by between 0.6Gt and 1.8Gt of CO₂.
Policymakers need to learn from the effective cases where climate policies led to meaningful reductions to get back on track, the authors have said.
Using a methodology developed by Climate Econometrics at the Institute for New Economic Thinking at the Oxford Martin School (INET Oxford), the researchers measured ‘emission breaks’ that followed policy interventions. The break detection methodology, called indicator saturation estimation, developed at Climate Econometrics allows break indicators for all possible dates to be examined objectively using a variant of ML.
Energy Monitor spoke to report co-author Ebba Mark, a researcher at the Calleva Project at the Institute for New Economic Thinking at the Oxford Martin School and Smith School of Enterprise and Environment, to discuss the study’s findings.
How well do you really know your competitors?
Access the most comprehensive Company Profiles on the market, powered by GlobalData. Save hours of research. Gain competitive edge.
Thank you!
Your download email will arrive shortly
Not ready to buy yet? Download a free sample
We are confident about the unique quality of our Company Profiles. However, we want you to make the most beneficial decision for your business, so we offer a free sample that you can download by submitting the below form
By GlobalDataCan you briefly explain the methodology you used to assess the effectiveness of these policies and how and why you chose the policies you did?
Ebba Mark: Normally when you evaluate the effectiveness of a particular climate policy, you start by identifying the particular policy with some hypothesis that it had an effect or it didn’t. So, you can only ask a question about that specific policy.
At Climate Econometrics, the research group where I work, an econometric method has been developed over time in which we use a machine learning (ML) algorithm that essentially does a model selection process for us. We start from a standard baseline emissions model that incorporates something like population GDP and climate variables, and then we do an indicator saturation estimation, where basically we populate this model with a potential reduction in emissions at every time step in every country and every sector.
The algorithm itself then searches over all these possible models to see which of these emission reductions actually reflect the real-world data. Essentially, we look at the emissions data, and then, using this algorithm, detect the effect of the policy or policies. We know then that in these 63 cases, something happened in those countries that we can’t explain with our normal model to cause significant reductions in emissions. We were able to do this because we had a database from the OECD that covers all these different types of climate mitigation measures, so we were able to then link those policies to these effects.
There are challenges with that approach as well, but one of the nice things is it allows us to be more objective about our hypothesis at the start. We don’t make any kind of assumption about what policies would have had an effect or not, it allows the data to speak for itself in a way that is quite unique and novel.
Why were there such few successes, and what was the most surprising finding from the study?
In the climate policy space, it does feel like an uphill battle most of the time, so we weren’t necessarily surprised, but it is certainly concerning. It’s not a great track record globally. However, there are important caveats due to our methodology.
We were looking to detect large emission reductions so we set a threshold of detection, which was a 5% emission reduction, which is actually quite large, so we’re not detecting effect sizes that are smaller. It’s very possible that some of these policies had more minor effects that we weren’t able to detect in our methodology. The other thing is that a lot of climate policy builds on momentum. So, it might be that some of the policies in the database weren’t detected according to our criteria, but were really important in terms of laying the groundwork or increasing the political feasibility of more ambitious policies that we then later detected.
But, even so, it’s not the best track record as a globe. One of the things that we’ve been stressing though is it shows us very clearly that just implementing climate policies doesn’t necessarily translate into an emissions reduction. It’s more a matter of how strategic you are with these policy mixes. So, are you complimenting pricing measures with subsidy-based measures? These are the carrots and sticks. Bringing together policy measures that tackle emissions from multiple angles seems to be the strategy that worked in these 63 success cases. We need to try to focus a little bit more on these success cases going forward – to learn from what has worked and try to be more strategic in our approach to climate policy.
Can you elaborate on this need for tax and price incentives in well-designed policy mixes?
The two things that are clearest are that, in these 63 success cases, we see mixes as being the most important element, but then the second result is that these price-based instruments – mixes that incorporate price-based instruments like taxes – serve as an amplifier. The combination of mixes, as well as understanding that a mix with a pricing element, tends to be most effective, which is probably one of the least surprising results of our research.
For example, policies like labelling in the building and transport sectors, as well as fossil fuel subsidy reforms, were only ever associated with a reduction in emissions when they were combined with other policies – the evidence suggests that they’re insufficient on their own but tend to be powerful in a mix. The same goes for things like bans and building codes and energy efficiency mandates and subsidies, which are very common policies – they are also only ever really effective in mixes and have a smaller effect by themselves.
In the case of the building sector, when policies were part of a mix, their effect tended to be a 32% emission reduction on average, but it was 13% if implemented alone. We were also able to detect that taxation tends to be more effective than subsidies alone.
Across the world, which policies proved to be the most effective at reducing emissions, and which were the least effective?
You see that in the case of electricity in developed economies, for instance, the most success tends to come from a combination of subsidies, regulations and pricing-based instruments. In the sectors in which company activity tends to dictate emissions, like electricity and industry, you tend to see that the pricing element is more effective, whereas in the case of buildings and transport, where you have a lot of consumer behaviour dictating the emissions pattern, then there’s potential for this kind of complementary policy mix in which you’re both punishing bad behaviour but also incentivising things like adoption of green technology or helping to make it less expensive or to make your home more energy efficient. It really depends on the sector you’re looking at. However, because of our methodology, we weren’t really able to prove that any climate policies were definitively ineffective.
From your research, you say you’ve been able to derive best practices for the building, electricity, industry and transport sectors. Can you provide examples of these?
In the UK, for instance, we detected a significant reduction in electricity-related emissions. So we found that in around 2015-16, there were two emissions reductions one right after the other. And the broader literature had attributed that to the carbon price floor that was set in the context of the EU ETS in 2013 but hadn’t really picked up on the fact that there were other policies that were implemented that had subsequently amplified the effect. There were both some command and control measures, like some renewable portfolio standards and also the announcement of a phase-out of coal power plants and then renewable feed-in tariffs and auctions at the same time. Those really amplified the emissions reductions.
In the building sector, there’s the Swedish example. There was a significant reduction in around 2004, and it tends to have been attributed to increases in the national carbon tax from €40 to €100, but they also combined some subsidy schemes, both for more energy-efficient windows and also for heating-system conversions. That’s one of these things where you see this combination of a pricing mechanism and a subsidy mechanism ushering in that game-changing moment for Sweden’s emissions.
For transport, there’s the US. In the US, we found a combination of fuel standards, put in place as part of the Renewable Fuel Standard in around 2007, as well as subsidies for green-vehicle purchases, led to a large reduction in emissions. Again, the combination of both restrictive and positive mechanisms proved most effective.
The study also found that best practices differ when it comes to industrialised and developing countries. Which policies work best for industrialised nations, and which are most effective for developing ones?
As I said, one of the headline results was that pricing was really effective across sectors in general, but this tends to be less true in the developing-country context. Pricing was quite effective in the electricity sector in developed economies, but even though 13% of policy implementations in the electricity sector in developing economies were pricing instruments, none of them were actually associated with any kind of meaningful emissions reductions in electricity in those countries.
We had a lot of conversations trying to figure out why that might be the case and one of the theories is that in the beginning stage of creating a climate policy landscape in a country, you tend to focus on more regulation-based instruments before you can advance to the effectiveness of pricing instruments. Also developing countries have less liberalised markets, so it’s harder to see the direct effect of pricing instruments in those contexts.
Based on your research, what should policymakers focus on to design more effective climate policies in the future? And how optimistic are you that current policy trends will help meet global net zero targets?
Across the sectors, the average effect size of the policies that we detected was 19%, and we did a calculation to work out that if every country across every sector basically reduced emissions by this average effect size, how close would we be to closing this emissions gap? For the policies we detected in the study itself, we counted that these had reduced global emissions by between 0.6Gt and 1.8Gt of CO2, whereas the current emission gap for 2030 to achieve 1.5°C, is 23Gt. So, it’s already a very small proportion.
But if we extended that, in the hypothetical exercise, to look at applying the average effect size per sector across all these countries, we’d be able to close that emission gap by 26%. So, even if every country in our study replicated the success effect across each sector within their country, we’d still have a really big portion of this emissions gap to cover.
What this research shows to policymakers is that just implementing a one-off policy without thinking about how it interacts with other policies, is not enough. It’s more about thinking about what countries have done in the past that has worked – these kinds of complementary policy measures across sectors – and trying to replicate that going forward.
It’s basically a plea to be not just ambitious, but also more strategic. The calculation that we’ve done is quite sobering, but it’s important we know what we’ve currently achieved and also what we need to do. We’re hoping that this research helps narrow down the possibilities for what countries should be doing, and identifies the options they have available to them.
