{"id":32465,"date":"2018-11-13T15:07:32","date_gmt":"2018-11-13T20:07:32","guid":{"rendered":"https:\/\/digital.hbs.edu\/platform-rctom\/submission\/will-ibms-deep-thunder-finally-be-the-key-to-accurate-weather-forecasting\/"},"modified":"2018-11-13T15:07:32","modified_gmt":"2018-11-13T20:07:32","slug":"will-ibms-deep-thunder-finally-be-the-key-to-accurate-weather-forecasting","status":"publish","type":"hck-submission","link":"https:\/\/d3.harvard.edu\/platform-rctom\/submission\/will-ibms-deep-thunder-finally-be-the-key-to-accurate-weather-forecasting\/","title":{"rendered":"Will IBM\u2019s Deep Thunder Finally be the Key to Accurate Weather Forecasting?"},"content":{"rendered":"

Limitations of Forecasting Capabilities \u2013 Data is NOT the Problem: <\/em><\/strong>Bill Belichick, head coach of the New England Patriots, summed up the national sentiment towards weather forecasters well when he stated \u201cif I did my job the way they did theirs, I\u2019d be here about a week.\u201d[1] <\/sup>While the historical error rate has been somewhat overstated (2013 study showed the top three weather providers were ~82% accurate in predicting next day precipitation)[1]<\/sup>, the remaining gap doesn\u2019t come from a lack of data. Currently there are ~1,000 weather satellites in space and \u201chundreds of thousands of government and private weather stations\u201d on the ground, all capturing data on temperatures, wind speeds, cloud patterns, etc.[2] <\/sup>\u00a0The National Oceanic and Atmospheric Administration (NOAA) collects this data and feeds it into its own public models (Global Forecast System and Numerical Weather Prediction) and makes it available for use of private companies such as The Weather Company (TWC).3\u00a0 <\/sup>If all this data is available, why are forecasts still inaccurate?<\/u><\/em><\/p>\n

One reason is that the NOAA and TWC haven\u2019t been able to build models to the level of specificity needed for short-term weather prediction. In the case of the NOAA, efforts have been broadly focused on high-impact events such as thunderstorms, tornados, and hurricanes. While the NOAA has begun to use machine learning techniques to fuse model output with greater observation capabilities (storm duration, wind speed, precipitation and aviation turbulence),[4] \u00a0<\/sup>these models still fail to deliver accurate information at the localized level<\/u><\/strong>.[5]<\/sup><\/p>\n

\"\"<\/a><\/p>\n

Figure 1: Global Forecast System (GFS) and Numerical Weather Prediction (NWP) Output, www.ncdc.noaa.gov<\/p>\n

\"\"<\/a><\/p>\n

Figure 2: The Weather Company Interactive Radar Map, www.weather.com<\/a><\/p>\n

 <\/p>\n

Local Forecasting via IBM\u2019s Deep Thunder:<\/em><\/strong> In 1996, IBM attempted to tackle weather forecasting at the \u2018hyper-local\u2019 level, aiming to solve for variation within given regions.[6] <\/sup>The theory behind the project, dubbed \u201cDeep Thunder,\u201d was that local surface features (i.e. hilly terrain, bodies of water, urban design, etc.) were not accounted for in NOAA\u2019s data measurement\/analysis, but lead to significant variance in measurable weather metrics within small geographic areas.[6]<\/sup> In many cases, this level of granularity is required: for example, Belichick would not care about weather across his county, only the radius around Foxboro Stadium. Similarly, utilities companies don\u2019t care about precipitation across an entire city, just the localized wind speeds (and maximum gusts) that would result in power outages. IBM felt uniquely positioned to create this model given public access to NOAA data and its existing AI platform.<\/p>\n

However, Deep Thunder didn\u2019t gain momentum until recently when IBM purchased TWC and gained access to its existing models\/private data centers, which in tandem with NOAA account for ~100 terabytes per day.[2] <\/sup>While in the short-term Deep Thunder adapts new daily data to refine its local models (currently as narrow as ~1\/5th<\/sup> mile resolution)[7]<\/sup>, the company is also rapidly expanding its database to build for the medium-to-long term. Through Weather Underground, a subsidiary of TWC, Deep Thunder now has access to a network of +250,000 personal weather stations (units on consumer homes, commercial buildings, etc.) that feed back into the centralized database.[8] <\/sup>This data is presented to consumers via the weather apps, including the Deep Thunder app, which is pay-for-service to city\/state governments and utilities companies (limited monetization to-date).<\/p>\n

\"\"<\/a><\/p>\n

Figure 3: Cross Sections of Deep Thunder Model, public.dhe.ibm.com<\/p>\n

\"\"<\/a><\/p>\n

Figure 4: Personal Weather System Heat Map & Illustrative Example, www.weatherunderground.com<\/p>\n

 <\/p>\n

Where Should Deep Thunder Go Next?<\/em><\/strong> In terms of short-term goals, Deep Thunder should continue to focus on expanding its network of personal weather stations by i) incentivizing individuals in remote areas to install units, potentially by subsidizing costs in exchange for ongoing Weather Underground access, and ii) exploring new ways of increasing touchpoints. For example, barometric pressure sensors have been built into smartphones since 2011 for location services; as of 2016 Deep Thunder was tapping into ~75k reports per hour, but estimated this could increase to \u201cover a billion per day.\u201d[9] <\/sup>GE has also begun implementing \u201csmart streetlights\u201d in MSAs that measure humidity, light and air quality, In terms of the medium-to-long term, IBM should more formally partner with NOAA to facilitate better transfer of data\/existing models and AI methodologies. As NOAA tackles the impact of severe weather systems from the top-down and IBM tackles daily weather from the bottom-up, the ideal outcome would be a system that can predict severe systems at a hyper-local level.<\/p>\n

As Deep Thunder is incorporating new data, it\u2019s imperative to always consider new observable characteristics that might impact weather patterns. Some upgrades on the horizon include i) updated snow coverage, ii) internal\/external solar radiation, and iii) real-time updated vegetation database.<\/p>\n

\"\"<\/a><\/p>\n

Figure 5: Initial Deep Thunder Hurricane Mapping, ibm.com\/waston<\/p>\n

Further Impact?<\/em><\/strong><\/p>\n