High-Resolution Weather Data with R

August 04, 2017 by Lee Robb
Topics:   SkyWise | Developers |

One of the pain points we run into at WDT is customers that need bulk weather data but have no experience with scientific data formats.

Since the release of our high-resolution analysis data (in NetCDF format), this issue has been coming up more frequently. Although we’ve offered CSV files in the past, there are a number of limitations with this approach.

Australian High Resolution Hourly Temperature

The biggest issue is that CSV files will often be more than 20–30 times the size of the NetCDF. So, looking at a years worth of our hourly analysis data for 10 variables, the NetCDF distribution might weigh in at 750GB compared to 20TB for the equivalent CSV data.

Additional benefits of using NetCDF data is best described by its FAQ:

NetCDF data is:
Self-Describing. A netCDF file includes information about the data it contains.
Portable. A netCDF file can be accessed by computers with different ways of storing integers, characters, and floating-point numbers.
Scalable. A small subset of a large dataset may be accessed efficiently.
Appendable. Data may be appended to a properly structured netCDF file without copying the dataset or redefining its structure.
Sharable. One writer and multiple readers may simultaneously access the same netCDF file.
Archivable. Access to all earlier forms of netCDF data will be supported by current and future versions of the software.

While the benefits are clear, that still doesn’t make the format necessarily easy to work with. In order to fix some of these accessibility issues, I’ve written 2 more R packages. The first is a thin API wrapper that allows you to download the data in a convenient manner: SkyWiseDataTransferR

Usage is simple:

> Authorize('app_id', 'app_key')
> DataTransfer('skywise-conus-surface-analysis', directory = '.')

The second is a library for working with NetCDF files: SkyWiseNetCDFR

# extract the temperature grid
> grid <- ExtractGrid(fileName, "temperature")

# find the data value at a certain lat / lon
> val <- GetValueAtPoint(35, -97, grid)
# Extract a subset of the grid
> vals <- mapply(GetValueAtPoint, lats, lons, MoreArgs = list(grid))
CONUS High Resolution Temperature
Check out additional SkyWise products we offer and take a look at more blogs on this topic here or on our Medium page. Whether it’s a mobile or web application, ESRI platform, or your own Common Operating Picture, SkyWise APIs' weather data is ready to work for you.
Check out SkyWise

Lee Robb

Lee is responsible for the strategy, roadmap, and feature definition of the SkyWise product line. As part of this role he leads the teams responsible for all of WDT’s SkyWise components. With vast experience building complex distributed systems and delivering solutions that delight users, Lee is considered a pivotal part of the WDT management team. He also leads WDT in establishing and executing software engineering best practices. Over the last 5 years at WDT, Lee has modernized WDT’s software development process, increasing both development velocity and quality. He has been a software engineering leader for 20 years, including shipping commercial products with millions of users.