A Simple Proposal to Re-Level the Playing Field after the Release of USDA Crop Reports

By Scott Irwin
Department of Agricultural and Consumer Economics
University of Illinois

The purpose of USDA crop reports is to facilitate efficient functioning of agricultural markets and reduce information asymmetries (Lusk, 2013). The latter objective is often referred to as “leveling the playing field.”  What this means in practice was nicely summarized by Surls and Gajewski, (1990, p.5) three decades ago, “If all forecasting were left to private firms, small players could be left in the cold.  USDA’s forecasts make objective information available to all market participants at the same time.”  The ability of the USDA to meet this objective has been challenged like never before with the change to “real-time” release of major crop reports, such as Prospective Plantings, Grain Stocks, and Crop Production reports.  The purpose of this article is to present a simple proposal to re-level the playing field after the release of these important and market-moving USDA reports.

The Problem

Before May 2012, important USDA crop reports were released outside of the regular trading hours of commodity futures exchanges.  Some were released before the opening of trading and some after the close of trading.  In either case, futures markets were closed immediately before or after USDA crop reports were released.  Everything changed in May 2012 when USDA crop reports began to be released during regular trading hours of futures exchanges.  Initially, “real-time” release occurred because futures exchanges expanded trading hours to overlap with USDA report releases.  The newfound flexibility of futures exchanges with regard to trading hours was the result of the move from open outcry “pit” trading to electronic trading.  With electronic trading, limits on futures trading hours became more of a choice rather than a necessity.  It did not take long for the USDA to realize that it did not really matter what release time they chose because futures exchanges had the flexibility to trade around the clock if so desired.  In light of this situation, the USDA  moved the release time for important crop reports to 11am CST in January 2013, roughly midway through the traditional daytime trading session for grain futures markets.  This was obviously more convenient for USDA employees who no longer had to stay up all night for the “lock up” on report release days.

The move to real-time release of USDA crop reports (livestock reports are still released after futures markets are closed) has been controversial from the start.  The key to understanding the heart of the controversy is the radically different incentives faced by market participants with a trading halt compared to real-time release.  With release under a trading halt it is relatively easy to assure a level playing field in terms of access to the USDA information.  Since futures markets are closed during release, the value of getting the information one second after release versus 30 minutes after release simply does not matter much in economic terms.  Regardless when an individual gains access to the information within the trading halt period, the futures market is still closed and it is difficult to act upon the information.  Economic incentives change completely when USDA reports are released into live futures marketing trading.  In this case, there is a huge economic incentive to gain access to the report before other market participants because one can trade upon the information virtually at the same time.  The incentives for what economists call “run games” after real-time USDA report releases should be obvious, particularly with electronic trading and the ability of some market participants to trade literally at the speed of light.

From the beginning of real-time release of USDA crop reports there have been complaints of unequal access to the reports:

1. Some complaints have been related to congestion at USDA servers that slowed access for some to a crawl.  I have personally experienced this numerous times, even accessing this through the relatively fast network at the University of Illinois.  Wait times in the first few years of real-time release could be as much as 15-20 minutes to download the reports.  This has improved over time as the USDA has increased server capacity, but in my experience it can still easily be a five-minute delay.
2.  The USDA long had a policy of allowing select news organizations to have access to crop reports up to 90 minutes ahead of release from within the secure lock-up where the reports are prepared.  This allowed the news organizations to prepare articles that could be released to the public simultaneously with release.  This undoubtedly assisted with the dissemination of the reports during the halt-era of release.  However, the USDA recently discovered that the networks of the news organizations were faster than their own network and this provided approximately a two-second speed advantage to subscribers to the news organizations over the general public when accessing the report through USDA servers.  In light of this disparity, the USDA stopped allowing news organizations to have early access to crop reports starting in July 2018.
3. Release of the November 2018 Crop Production report was delayed by around 10 minutes due to USDA server maintenance that was not co-ordinated with release of the report (Walljasper, 2019).  This created considerable chaos and confusion as it was unclear in those 10 minutes why no one could get access to the report.  In addition, it appears that some market participants gained access to the report before the end of the 10 minutes through USDA emails.

This litany of issues clearly indicates there is a problem with equal and timely access to USDA crop reports at the present time, which is very concerning because this is one of the two core objectives of the USDA reporting system in the first place.  It turns out there is a simple and extremely low cost mechanism to address this problem.

The Proposal

The main problem with the current USDA system for releasing crop reports is access to the pdf documents placed on the USDA servers.  A bottleneck is created because this is the only way to access the reports.  Furthermore, if problems occur for some reason with the USDA’s servers, as they did in November 2019, there is no redundancy in the system.  This is even more problematic since media organizations were banned from getting pre-release access to crop reports.

My proposal is to create an alternative announcement channel using social media.  The basic idea is to send out a tweet on Twitter with the headline numbers in an attached graphic for a particular report or group of reports.  This would never be intended to replace release of the full report on USDA servers, but instead, provide a quick and virtually costless method of circumventing congestion problems on the USDA network in the critical first few seconds and minutes after release.  Recent estimates indicate that Twitter has around 275 million users on a monthly basis and a massive server infrastructure to deal with the traffic on the social media giant.  Even the most important USDA crop reports would not put a dent in the capacity of Twitter.  While there is no hard evidence on the number of grain traders, market analysts, journalists, and farmers on Twitter, my experience suggests it is a very large number.  It is important to remember that the objective is not to replace the current USDA server system for releasing crop reports but instead to complement it by taking advantage of the massive infrastructure in place for Twitter to quickly propagate key headline information to market participants.

Some may argue that managing this alternative release mechanism would put an undue burden on the USDA to choose which headline numbers to feature in the tweet.  In reality, this is not as hard as it might appear.  Market journalists already send out emails in a format to record key USDA information in a particular report so it can be compared to market expectations.  An example from Reuters reporter Julie Ingwersen for the January 10th USDA reports is found in the Appendix to this article.  The key USDA information in this email table can be used as a guide to what is considered the most important information to the market.  This information could easily be placed in one spreadsheet table, captured as an image, attached to a tweet, and then sent out.  A committee of market journalists and traders could be assembled by USDA if needed to help determine the numbers to release in the headline tweet.

This plan would require co-ordinated release of the full report in pdf format on USDA servers and the headline tweet.  One way to do this would be to schedule release of the tweet to go out at a predetermined time using tools like Tweetdeck and Hootsuite.  The USDA could delay release of the headline tweet by a second or two if needed to make sure that the pdf versions were technically released first.  Finally, the USDA could consider releasing the headline information simultaneously on several different social media platforms, such as Facebook and LinkedIn.  This would create redundancy in social media streams and speed the propagation of key crop report information even further.

Summary

A key objective of the USDA crop reporting system is to provide equal access to important supply and demand information across the spectrum of market participants.  Meeting this objective has become much more complicated with “real-time” release of USDA crop reports since May 2012.  With real-time release during the active hours of futures trading, what previously would have been seen as trivial differences in access times have been magnified many, many fold.  Seconds and even milliseconds now matter in terms of access to USDA crop reports.  The fundamental difficulty is congestion and reliability of USDA servers when crop reports are released.  A simple proposal is made here to create an alternative channel for disseminating key information in crop reports.  Specifically, the proposal is to have the USDA determine “headline” numbers before important crop reports and then distribute them on Twitter (and perhaps other social media platforms) at the same time that the full reports are made available on USDA servers.  This proposal is not only simple but extremely low cost.  If implemented, the proposal would go a long ways towards re-leveling the playing field after the release of USDA crop reports.