The Gateway to Algorithmic and Automated Trading

Weather veins

Published in Automated Trader Magazine Issue 30 Q3 2013

Some companies will scour the earth for the right weather data to power trading strategies. Adam Cox reports on the business of weather information, a type of data that can play an enormous role for a host of markets.

A Guatemalan mountaintop might seem an odd place to find your edge. But if you're hoping to get ahead in some markets, you may need to be prepared to travel far for your data.

"Understandably, ag traders want the data exactly where the crops grow. That's not always possible. You have a lot of basis risk between where the crops grow and where the weather is observed," said David Whitehead, director of US operations at Speedwell Weather. "But there's always a push. Everyone wants it exactly where the crops grow."

Speedwell sells information from government meteorological offices around the world, cleaning or recalibrating the data to add value. In some cases, it has its own weather stations or it may work with local met offices to supply historical records that would otherwise not be available.

Energy and agricultural trading firms have long recognised the value of weather data, and they're not the only ones. "We've got a number of clients that are hedge funds," Whitehead said "It's one group feeling like they've got a better forecast or better data than the other. It's a very information-driven market."

Weather is very big business. From a fundamental perspective, it directly affects one third of businesses worldwide, according to the Chicago Mercantile Exchange, which has a large stable of weather derivative contracts.

Weather is a central factor in both supply and demand considerations for energy and commodities sectors.

It can affect supply chains for swathes of industry due to its logistics implications. And there have been numerous studies looking at how weather affects equity market trading.

What's more, extreme weather is occurring more frequently, while meteorologists' ability to forecast weather has improved significantly over the years. There is also a latency play to be had, for those firms that are ready to spend the money.

Taken together, all of this would suggest that weather data would be a rich vein for firms hoping to find alpha via proprietary models. To be sure, there is plenty of anecdotal evidence that some companies are engaged in this area. And yet, the focus on weather and weather data has not panned out exactly as some expected about a decade ago.

Should adventurous traders view this as an opportunity to do something different from the crowd? Conversations with meteorology offices, data vendors, analytics firms and academics suggest that weather offers possibilities that are not being exploited widely.

"A world of difference"

As it happens, information from that Guatemalan mountain is currently in high demand, and it's due to a disease called coffee rust.

"The propagation of that disease is weather-dependent. So you start looking at countries like El Salvador and Guatemala, and where the coffee crops are grown at elevation, people are trying to obtain data that's very localised," Whitehead said.

Weather data in some areas can be much more varied than in others. "If you're working in a mountainous region, or a tropical region, it makes a world of difference if you just go a couple miles in each direction or change the elevation 1,000 metres," he added.

Weather stations. A map of approximately 10,000 weather stations used by Speedwell to monitor commodities. These do not represent the total universe of weather stations.
Weather stations
A map of approximately 10,000 weather stations used by Speedwell to monitor commodities. These do not represent the total universe of weather stations.

Two other factors complicate the picture for data-hungry firms.

The first is that some data is simply difficult to get your hands on.

"For Guatemala, you know, it's a developing country. They do a great job with the resources they have, but it isn't the same type of budget that NOAA has here in the US," Whitehead said, referring to the National Oceanic Atmospheric Administration.

"So we've been in discussions with them to actually visit the met office, go down into their archives and take paper copies of all their records, make copies and then digitise them ourselves, such that clients can have access to some of these unique data sets."

The second complicating issue is the data quality, and that can mean the quality of the raw data inputs or of the models derived from those inputs.

There are a handful of major services that supply weather data and run models that produce forecast data. In addition to the data from the NOAA, there is a European model that is supplied by the Met Office in the UK, as well as data from other large countries such as China or Canada. There are also many smaller national met offices. NOAA doesn't charge for its data, though some met offices do.

Major met offices run global models, predicting the weather for the entire world. But global models are at a low resolution. Meteorologists divide the world into grids, typically anywhere from 25x25 kilometres down to 2.5x2.5 kilometres. The smaller the grid, the more accurate the weather forecast is likely to be. So-called mesoscale models, which focus on smaller geographical areas, have smaller grids so the data may be more precise.

Meteorological models start with a snapshot of the status of the atmosphere, based on surface observations at airports, balloon launches, data from satellites, radar measurements and other sources. That's called an initialisation point. Then, using physics, a model will try to predict weather movements.

"From a physics perspective the atmosphere is like a liquid. It moves like water does," said Robert Bunge, the software branch chief for the telecommunications operations centre at NOAA's National Weather Service.

Bunge helps companies that want to set up direct feeds for weather data.

"We know our data is heavily used by the energy sector, we know our data is heavily used by trading firms," Bunge said, adding that trading firms were some of the biggest customers. "I know from talking to them that they're downloading these files, they're running computer algorithms against them and they're generating a package of some sort that they're giving to their traders."

Anyone can get terabytes of data for the whole world's weather system from NOAA for free via the internet. But that means that when each batch of weather data comes out - NOAA uploads new data every six hours - users are fighting for bandwidth.

As a result, many of the trading firms that want to ensure they get weather data reliably will pay for dedicated lines. NOAA doesn't collect information on what types of firms are downloading the data.

It has a policy of offering a level playing field, so firms that pay for direct lines to the agency's servers get the data at the same time as anyone on the internet. But those who pay still have an edge.

"If they have a dedicated circuit they might be getting it a little bit faster, only because they won't be competing against everyone else," Bunge said. "Somebody using just the internet to download the data, the risk that they take is that our bandwidth is overwhelmed by some event."

Leigh Henson, global head of energy markets at Thomson Reuters, said latency is becoming more of an issue for the commodities market, though firms trading on weather data are not operating at the same low latency levels seen in other asset classes.

"If you go back five years, then it would have been a good eight to 10 minutes before you'd start to see the effects of the new weather forecasts in the price development of, say, natural gas in the US," Henson said.

"Now it's probably closer to 30 to 60 seconds. If you think about it, 30 to 60 seconds isn't really low latency. But for the weather industry it shows development."

David Whitehead, Speedwell Weather

David Whitehead, Speedwell Weather

"All weather data is not equal."

Cost and quality

But even though competition in the financial sector has never been more intense, Bunge reports that demand for the direct lines has basically been steady.

For a start, although the data may be free, a direct line can end up costing tens of thousands of dollars a year. In the wake of the global crisis-induced recession, some firms that had used direct lines decided to rely on internet delivery, he said.

Also, taking weather data from one source could present other issues. The model that NOAA produces is different from the models that come from other services such as the UK Met Office.

"They use different algorithms, they're run with different sets of data, they're initialised differently. They run at different times of the day," Bunge said.

Those differences mean that in some situations, one model will invariably work better than another. On-the-ground operational meteorologists will often know which model is performing better for different issues. "I think it's famously known that the European model was better at predicting the path of Sandy than the US model was," Bunge said, referring to Hurricane Sandy, which swept over the east coast of the United States in 2012 "Next year it might be different," he said.

Trading firms can also take the raw data that feeds into a model and run their own forecasts with it.

"The cost of high performance computing is dropping so what you will find is organisations - especially organisations that are interested in this specific area -will download the initialisation data, and in some cases you can go out and download the algorithms that run the model," Bunge said.

Whitehead of Speedwell put it bluntly: "All weather data is not equal."

For Speedwell, a lot of the value is in data cleaning or recalibrating datasets to correct for historical discontinuities.

"Let's say you have your thermometer out on an airfield, and they build a new runway right by the thermometer. Historically that thermometer will most likely record warmer every single day from here until the next change," he said. So Speedwell will make an adjustment to account for that in its historical database.

Thomson Reuters, which has a large client base in the commodities and energy sectors, presents customers with weather data and commodities supply chain details in a visual map-based format, and also allows users to take the raw data and run proprietary models.

"Typically our larger clients will have been in the business of building and running their own models for many, many years. And for the smaller clients, they don't necessarily have the same resources or horsepower to do that. In the case of the smaller clients they welcome the fact that we're bringing a significant resource to bear on this," Henson said.

Thomson Reuters in recent years has beefed up its analysis capabilities by acquiring specialist companies such as Point Carbon, which provides information for the energy sector, and US crop forecaster Lanworth.

Deutsche Bank is one financial group that employs its own meteorologists. Michael Lewis, head of commodities research at Deutsche, said there are two who work directly in the commodities business unit.

"We've seen over the last three to four years successive years of extreme weather conditions in the US, which led to declining yields, particularly on corn but also on soybeans as well," Lewis said.

"And getting a clue as to whether we will see upward or downward revisions to yield projections made by the USDA have been important for the underlying physical fundamentals," Lewis said, referring to the US Department of Agriculture.

The period between June and September is typically the most volatile period for projections for US agriculture and that's primarily driven by weather, he noted.

Superstorm Sandy Superstorm Sandy, as displayed in the Thomson Reuters interactive weather map in its Eikon product. The orange diamonds represent oil refineries.
Superstorm Sandy
Superstorm Sandy, as displayed in the Thomson Reuters interactive weather map in its Eikon product. The orange diamonds represent oil refineries.

Weather derivatives

Weather analysis plays a key role for many markets, but in some cases weather is the market. Exchange-traded weather derivatives such as temperature- or rainfall-based contracts listed on the CME offer a hedging tool for end users and also attract some speculative interest. The market got its start in 1997 with the first weather derivative product.

Still, trading has not taken off in the way that some had predicted.

Steve Wilcockson, industry manager for financial services at Mathworks, said that seemed surprising given the apparent increase of weather-related crises of floods, hurricanes and drought.

"If I look back to the 1999-2001 era after the introduction of NETA (New Electricity Trading Arrangements) in the European Union, weather derivative modelling was all the rage with banks and Enron-type consultancies eager to sell 'exotic' option-based hedges to energy firms apparently wanting to manage their trading exposure and competitive electricity delivery provision," Wilcockson said. But he said he did not believe MathWorks had been asked to support a weather derivatives evaluation or run a weather derivatives event in years.

Whitehead of Speedwell also noted the market had not developed as expected. "It's not what people envisioned 10 years ago, but with that said it's a very healthy market," he said.

"After the financial crisis, that centre of the market took quite a hit. A lot of companies pulled out, resulting in less liquidity. It's kind of a Catch-22. If you don't have liquidity, a lot of shops simply aren't able to trade it, but with that said, as the exchange-traded market decreased in size, the OTC market has expanded very, very rapidly."

That has made the business of trading weather risk much more idiosyncratic.

"It's a very different market now," Whitehead said. "Whereas in the past, there were a handful of stations that were traded kind of consistently, now everything's game. Everything from weather stations in the mountains of El Salvador to the outback of Australia to Russia, China, you name it."

Raphael Markellos, University of East Anglia

Raphael Markellos, University of East Anglia

"The vast majority of academic studies, including ours, concentrate on getting statistically significant results. Now, it's not always the case that these can be transferred into profits."

The Sunshine Effect

The availability of so much weather data for free also makes it easier for the academic community to examine weather and its relationship to financial markets. In fact, there are numerous behavioural economics studies that look at how weather is correlated to equities. For instance, the so-called sunshine effect posits that sunny weather increases the optimism of some investors and makes them more likely to take long positions.

Raphael Markellos, chair in finance for the Norwich Business School at the University of East Anglia, said there was a plethora of work in top journals that focused on the sunshine effect. "But there was little evidence of the effect that weather variables have on volatility."

So he co-authored a study of weather and volatility in 26 cities, from 1982 to 1997. It suggested that cloudiness and length of night-time were inversely related to historical, implied and realised measures of volatility, and it found that the strength of the relationship varied depending on how close a given exchange was to the equator.

But despite the empirical evidence on offer, Markellos said weather did not appear to be used extensively in the real world for equities trading.

"The vast majority of academic studies, including ours, concentrate on getting statistically significant results. Now, it's not always the case that these can be transferred into profits," Markellos said. "There's a difference between statistical significance and economic significance."

He said it was possible to have a regression that was statistically significant and which could improve forecasting, but that once used in a trading system it would fail to make enough money to make up for transaction costs.

Willcockson of Mathworks offered an additional reason why weather did not feature more prominently in mainstream financial analysis.

"Quant finance has typically been populated by engineers, physicists and mathematicians. Look at any job ad for a quant hedge fund or HFT desk, which will routinely seek physics, mathematics or computer science PhDs. It is doubtful that they would have encountered significant 'weather'-based studies in their university careers and perhaps too this is culturally deemed 'softer' science so perhaps not engrained within the technical world-view of typical quant hires," he said.

Leigh Henson, Thomson Reuters

Leigh Henson, Thomson Reuters

"Do we think the consideration of weather data in trading strategies will grow? The answer is yes,"

"Few firms, insurance and energy firms excepted, go out of their way to seek people with this particular expertise. If weather is identified as a factor relevant to trading, particularly during extreme event situations, then maybe this should change," he added.

Much of the use of his firm's MATLAB product for weather analysis is done by companies that want to optimise sales based on weather. For instance, one supermarket runs a five-year weather/sales model, utilising regression testing methods that output coefficients and deploying the forecasts in stories. A 10 degree increase in temperature raises demand for coleslaw by 50% but reduces demand for Brussels sprouts by 15%.

Wilcockson saw no technical excuse for not undertaking weather analysis and incorporating it as a factor in trading analytics, adding that technological advances made analysis of weather data much more feasible. He cited a University of Illinois example where parallel computing helped reduce the analysis time from a month to overnight on 10 years' worth of data.
"Where analytics can be parallelised, then parallel and grid computing, on multi-core, clusters and GPUs adds significant value," he said.

"In my experience though, the analytics applied in these and other packages are relatively similar to those in classic financial time-series and statistical analysis," he said, mentioning for instance regression techniques, machine-learning, Monte Carlo methods, optimisers for parameter estimation, clustering, factor analysis and correlation analysis.

He added: "I do see a step change in reduced cost of computing power, providing greater bang for buck in realistic, granular simulations."

In the same way that hardware improvements and grid computing have helped meteorologists improve their accuracy by using bigger simulations, better parameterisation and accommodation of stronger spatial coverage and variability, Wilcockson said it's possible that such models could be more easily implemented in finance at more reasonable cost.

Thomson Reuters, perhaps expectedly, argues weather data will gain in stature.

"Do we think the consideration of weather data in trading strategies will grow? The answer is yes," Henson said.

"There are people in the traditional fundamental trading firms who been trying to do this kind of modelling of the impact of weather for a long time. The area of the market that hasn't really done this with any great degree is the big hedge funds and algorithmic traders."

He argues that that will change over the next few years. "Weather will become an increasingly interesting input to some of those trading strategies and models because of its impact not only on the commodities … but on the financial health of some of the companies that operate in that sector."