Data served here

From its inception, NCAR provided a wide range of research-oriented computing and observing facilities. The initial focus was on centralized capabilities: networking was virtually nonexistent, and the huge cost of computers constrained university use. These factors were changing rapidly by the late 1970s, when forward-thinking leaders at NSF and universities began to envisage the transformative effect of networks and small computers on science education as well as research.

A far-reaching consequence was UCAR’s University Data Program (Unidata). Its founding coincided with National Weather Service plans to drop teletype and facsimile services that had fed weather data directly to universities on leased phone lines in those pre-Internet days. The NWS decision was a “bombshell” to professors and students, said John Dutton (Pennsylvania State University). Concern culminated in 1982, when university department chairs called on UCAR to offer prompt, affordable access to weather data.

The next summer, an 80-person workshop in Madison, Wisconsin, considered data access alongside ambitious views of a future where networking and computing were nearly ubiquitous. The meeting endorsed a plan for creating Unidata, which began in 1984 under two principal investigators: Dutton (who coined the moniker) and the late Verner Suomi, a satellite meteorology pioneer at the University of Wisconsin–Madison (UW). UCAR would be the host institution, with NSF as funder. “When the need for Unidata arose, both organizations were prepared to act,” says Dutton.

Photo of computer data cartridges
Within the StorageTek silo installed in 1989, robotic arms could choose and mount any of thousands of data cartridges.

Unidata’s distributed architecture departed from UCAR’s earlier centralized patterns. Free software and real-time data streams allow faculty and students to acquire, manage and analyze a vast array of data on their own computers without a central repository. “If there was a Unidata mantra in my time,” says founding director David Fulker, “it might have been, don’t do anything centrally that universities themselves can do well.” In this spirit, Unidata adopted a university governance model, and to this day the program remains community-driven.

Software distributed by Unidata also reflects community generosity: every package has deep roots in other organizations, including UW, Purdue University, NASA, and the University of Rhode Island. A recent example is the Unidata-developed Integrated Data Viewer (IDV), built on a visualization library from UW’s William Hibbard.

More than 1,500 academic institutions and 7,000 organizations—extending well beyond meteorology departments—employ Unidata software tools or participate in its data-exchange network. Together, these have radically altered how data are transported, managed, and studied in contexts as diverse as the NWS and the Intergovernmental Panel on Climate Change, as well as at universities. With data and analysis tools close at hand, “the entry barrier has been lowered for undergraduate students who want to get involved with research,” says Gary Lackmann (North Carolina State University).

Meanwhile, for the comprehensive global modeling and other large-scale tasks handled by NCAR supercomputers, a central data archive remains critical. The center’s mass storage system, launched in the 1970s, went through a major upgrade in the late 1980s, with data transferred from open reels to cartridges that provided quicker access. Still, NCAR staff had to manually mount and dismount each tape so that scientists could access past findings and store new results.

Automation came to the rescue in 1989 as NCAR installed its first robotic storage system, built by StorageTek. The device retrieved and replaced data on request from a library of more than 67,000 cartridges holding a total of roughly one terabyte (one trillion bytes) of data. The workings of the machine’s mechanical arms, hidden within a cabinet-like silo, could be watched by visitors through an internal camera. The latest incarnation of this approach, AMSTAR (Augmentation of the Mass Storage Tape Archive Resources), arrived in 2008. Designed by Sun Microsystems, it will bolster the maximum storage capacity to 30 petabytes—or about 30,000 times the system’s 1989 capacity—over four years.


Today — Observations on demand

Photo of Illiot Atlas

"Real-time data access is a major help to solve instrument issues."

—Elliot Atlas, University of Miami

The field catalogs maintained by NCAR’s Earth Observing Laboratory are a data lover’s dream come true. The online archive includes observations, forecasts, reports, flight tracks, and other details from more than 50 observing campaigns held across every continent. Launched within UCAR’s Joint Office for Science Support in 1995, the field catalog system migrated to NCAR in 2005.

Participants in field studies don’t even have to wait for a project to conclude before accessing the catalog. Data are shuttled through the system and made available within minutes, which enables field coordinators to shape a given day’s activities far more effectively—especially on airborne missions.

“We’re now getting data from the aircraft to the ground through a satellite link in real time,” notes NCAR’s Mike Daniels. “People can communicate with the aircraft and direct a project from their university via instant messaging and chatrooms.” In the past, Daniels notes, scientists in the field would have to wait for a research flight to touch down before they could upload and scrutinize data. “Now everybody’s looking at the data from wherever they are. People on the ground can chat with people on the plane and say, ‘Hey, this measurement looks suspicious. Can you check out the instrument’s status?’ It really helps to improve data quality during the crucial sampling phase.”

Photo of a woman sitting and working at scientific equipment
NCAR’s Teresa Campos tracks data while flying off the coast of Chile in 2008 for VOCALS-REX, the VAMOS (Variability of the American Monsoon Systems) Ocean Cloud Atmosphere Land Study–Regional Experiment.

For the START08 experiment (Stratosphere-Troposphere Analyses of Regional Transport), NCAR’s Laura Pan and colleagues used chatrooms and Google Earth to direct flights studying air circulation and chemistry across the thin, critical region known as the tropopause, which lies between the troposphere and stratosphere. “For dynamics-oriented experiments such as START08, the ability to modify the flight plans in real time helps optimize success,” says Pan. “In some cases, it’s necessary for success.”

The space needed for instrumentation often limits the number of scientists who can fly on airborne chemistry experiments. However, NCAR’s real-time communication capability allows “virtual operators” to steer the data collection strategy. “This maximizes the chance that instruments will be able to obtain more and better measurements during any particular flight,” notes Elliot Atlas, a START08 investigator from the University of Miami.