Register for the 2018 Unidata Users Workshop

The Unidata Users Committee invites you to join Unidata staff, community members, and distinguished speakers this June in Boulder, Colorado. The goal of this year's workshop is to raise awareness of the current trends in the academic geoscience community with an emphasis on Data Proximate Analysis, Machine Learning and Data Analytics, Using GOES-16 Data, and Working with Ensemble NWP Output.

Double the data: Putting weather observations in the cloud increases access

May 1, 2018 | Meteorologists and other users accessed more than twice as much U.S. weather radar data after the National Oceanic and Atmospheric Administration (NOAA) partnered with Amazon and the University Corporation for Atmospheric Research (UCAR) to make the data available in the cloud. The collaboration is part of the NOAA Big Data Project. Launched in 2015, the project aims to make NOAA's vast storehouse of environmental data easier to access with the hope that both the public and industry will find ways to capitalize on it and spur new economic growth. Partners in the project include Amazon Web Services, Google Cloud Platform, IBM, Microsoft, and the Open Commons Consortium. The first dataset chosen for the project was from the Next Generation Weather Radar (NEXRAD) Weather Surveillance Radar system, and the first collaborator to put that dataset in the cloud was Amazon. The entire archive of NEXRAD data, which stretches back to 1991, is now available via NEXRAD on AWS, and Amazon has partnered with UCAR's Unidata program to update the database in real time and to provide the tools users need to make sense of the data. "As a leading provider of geoscience data to universities across the country, it made perfect sense to partner with Amazon to explore how cloud computing can expand our reach and provide new capabilities to our users," said Unidata Director Mohan Ramamurthy. "The success of this project has given us a chance to be a part of what the future of data delivery will look like." In a new paper published recently in the Bulletin of the American Meteorological Society, collaborators describe the project's early accomplishments. Chief among them: users are now accessing 2.3 times more NEXRAD Level II data. This data, collected from 160 sites and updated approximately every 5 minutes, characterizes precipitation and winds from across the U.S. and is an important input for weather forecasts. Prior to the NOAA Big Data project, archived Level II NEXRAD data was stored only at NOAA's National Centers for Environmental Information (NCEI). While this archive represents a critical dataset for researchers interested in large-scale analysis of how weather patterns may have changed over time, in practice it was very difficult to use. In part that's because of its size — about 270 terabytes — and in part because of the time and cost to obtain the files. The authors of the new study estimate that it would have cost $203,310 and taken 540 days for NCEI to fulfill one researcher's request to obtain the entire NEXRAD Level II data archive. Now, scientists can access the data on the Amazon cloud at no charge. As a result, about 80 percent of the NEXRAD data accessed by users now comes from the cloud, and only about 20 percent comes from NCEI. Jeff Weber, who leads Unidata's part of the project, said that this kind of easy, open access to geoscience data "removes the friction" of doing the science. Moving forward, Weber envisions having satellite data and weather model output available alongside the radar data."Once we're able to bring all these components together in the cloud, I think we're going to see a huge leap in the science," he said. "When you make it easier for scientists to just focus on the science — and not worry about accessing and storing huge amounts of data — breakthroughs are bound to happen." The collaboration with Amazon has allowed Unidata to test how scientists will respond to data stored in the cloud. Part of Unidata's long-term vision is to take data that it now pushes out to users across the country and move it instead to a cloud platform. Aside from the benefit of scientists being able to access the data from anywhere, moving data to the cloud also addresses the reality that increasingly complex and high-resolution models and observational instruments are straining the physical capacity to deliver data to individual research institutions. "The data volumes are growing exponentially — they are growing so fast that we can't just keep pushing all of the data out to our users," Ramamurthy said. "But just putting data out there in the cloud isn't enough either. We need to train our community on how to access that data and provide them the tools they need in the cloud to make it as easy as possible to use the data that's stored there."About the articleTitle: Unlocking the Potential of NEXRAD Data through NOAA’s Big Data PartnershipAuthors: Steve Ansari, Stephen Del Greco, Edward Kearns, Otis Brown, Scott Wilkins, Mohan Ramamurthy, Jeff Weber, Ryan May, Jed Sundwall, Jeff Layton, Ariel Gold, Adam Pasch, and Valliappa LakshmananJournal: Bulletin of the American Meteorological Society, DOI: 10.1175/BAMS-D-16-0021.1Writer/contact:Laura Snider, Senior Science Writer

NCAR|UCAR hurricane experts available to explain storm behavior, potential impacts

BOULDER, Colo. — As Hurricane Harvey takes aim at Texas, scientists at the National Center for Atmospheric Research (NCAR) and its managing organization, the University Corporation for Atmospheric Research (UCAR), are closely watching the storm and testing high-resolution computer models.Hurricane experts are available to explain issues such as:How we can better predict the possible impacts of hurricanes, including wind damage, flooding, and subsequent spread of disease-bearing mosquitoes;How people respond to hurricane forecast and warning messages and how risk communication can be improvedWhether climate change is affecting hurricanes and what we can expect in the future;The importance of improving weather models to safeguard life and property.Antonio Busalacchi, UCAR president (please contact David Hosansky for interview requests)An expert on ocean-atmosphere interactions, Busalacchi has testified before Congress on the importance of improving the nation's weather forecasting capabilities to better protect life and property, bolster the economy, and strengthen national security. He has firsthand experience with storms along the Gulf Coast as a part-time New Orleans resident, and he is a member of the Gulf Research Program Advisory Board of the National Academy of Sciences.Christopher Davis, director, NCAR Mesoscale and Microscale Meteorology Laboratory, cdavis@ucar.edu, 303-497-8990Davis studies the weather systems that lead to hurricanes and other heavy rainfall events. His expertise includes hurricane prediction and how computer models can be improved to better forecast storms. His NCAR weather lab is running experimental computer simulations of Hurricane Harvey.James Done, NCAR scientist, done@ucar.edu, 303-497-8209Done led development of the innovative Cyclone Damage Potential (CDP) index, which quantifies a hurricane's ability to cause destruction, using a scale of 1 to 10. It can also be used to examine the damage potential for cyclones in the future as the climate warms.David Gochis, NCAR scientist, gochis@ucar.edu, 303-497-2809An expert in hydrometeorology, Gochis studies the causes of floods and how to better predict them. He helped develop pioneering software that is at the core of the National Water Model. The National Oceanic and Atmospheric Administration Office of Water Prediction uses this model to provide a continuous picture of all the waterways in the contiguous United States and alert officials to potentially dangerous floods.Matthew Kelsch, UCAR hydrometeorologist, kelsch@ucar.edu, 303-497-8309Kelsch has studied some of the biggest U.S. flood events connected to hurricanes and tropical storms. He trains scientists and forecasters from around the world on emerging hydrology and weather topics.Rebecca Morse, NCAR scientist, morss@ucar.edu, 303-497-8172Morss studies the predictability of hurricane-related hazards, including storm surge and inland flooding, and hurricane and flood risk communication and evauation decision making.Kevin Trenberth, NCAR senior scientist, trenbert@ucar.edu, 303-497-1318Trenberth is an expert on the global climate system. He has been in the forefront of scientists examining the potential influence of climate change on the intensity of tropical storms and hurricanes and the increased widespread flooding that they cause.Jeff Weber, UCAR meteorologist, jweber@ucar.edu, 303-497-8676As an expert on hurricanes and severe weather in general, Weber closely monitors the behavior of individual storms and the larger atmospheric and oceanic conditions that influence them.

From GOES-16 to the world

March 6, 2017 | As atmospheric scientists around the world look forward to seeing extraordinarily detailed images from the new GOES-16 satellite, the University Corporation for Atmospheric Research (UCAR) and National Center for Atmospheric Research (NCAR) are preparing for central roles in disseminating the satellite's data.The first of a series of next-generation National Oceanic and Atmospheric Administration (NOAA) satellites, GOES-16 was launched in November and is expected to become fully operational late this year. It will immediately improve weather forecasts with its rapid, high-resolution views of hurricanes, thunderstorms, and other severe events, as well as provide a breakthrough lightning mapping system and more detailed monitoring of geomagnetic disturbances caused by the Sun."Scientists are rightfully excited because this is a revolutionary system," said Mohan Ramamurthy, director of UCAR's Unidata Program. "It's going to truly transform weather forecasting and research."GOES-16 captured this view of the mid-Atlantic and New England states on Jan. 15. (Image by National Oceanic and Atmospheric Administration.) Data from GOES-16 will be transmitted to a new downlink facility at the NCAR Mesa Lab. Unidata, which provides data, software tools, and support to enhance Earth system science education and research, will then make that data widely available.  As the only open-access and free source of GOES data in real time, Unidata's services have become indispensable to scientists as well as to operational forecasters in regions that lack their own downlink facilities, such as parts of Latin America.In addition, NCAR's Earth Observing Laboratory (EOL) will produce customized data products from GOES-16 to support field campaigns. EOL currently uses observations from GOES satellites and other sources to help scientists make critical decisions as they're taking measurements in the field.More data than everFor years, NCAR and UCAR have provided real-time data from a series of NOAA satellites known as GOES (Geostationary Operational Environmental Satellite). These satellites, which provide views of the Americas and adjoining ocean regions, are part of a global network of satellites whose observations are shared by forecasters and researchers worldwide.But the advantages of GOES-16 also create new challenges. The satellite has three times as many spectral channels as its predecessors, each with four times more resolution. It can scan the entire Western Hemisphere every 15 minutes and simultaneously generate images of severe weather every 30-60 seconds. All this data will amount to about 1 terabyte per day, more than 100 times the amount of data produced by an existing GOES satellite. And even more data can be expected when NOAA launches additional advanced GOES satellites in coming years.Thanks to a NOAA grant, UCAR and NCAR have installed a direct broadcast receiving station to receive the data, as well as the computers and electronics needed to process and transmit it. In addition to Unidata and EOL, NCAR's Research Applications Laboratory helps operate the downlink facilities for existing GOES satellites and relies on satellite data for the development of specialized forecasting products.The volume of information means that Unidata will continue to move toward making data available in the cloud. It will store GOES-16 data for about 10 days and is in discussions with Amazon over long-term storage options.EOL will customize GOES-16 observations for worldwide field projects, which advance understanding of Earth system science, including weather, climate, and air quality. Such projects deploy teams of scientists with aircraft, ships, ground-based instruments, and other tools. They rely on detailed forecasts and real-time updates about evolving atmospheric conditions."The data from GOES 16 will provide invaluable information for flight planning and decision making during field projects," said EOL director Vanda Grubišić. "This will enable scientists to gather additional observations, further advancing our understanding of the atmosphere and related aspects of the Earth system."EOL will also include the GOES data in their field catalog, along with measurements from field campaigns and other observations. This catalog is widely used by scientists when analyzing results from past campaigns or planning new ones.Other scientists say they are looking forward to the new capabilities that GOES-16 offers."The observations collected by the Geostationary Lightning Mapper on GOES-16 have the potential to help advance our understanding of hurricanes and their intensity changes," said Kristen Corboseiero, a professor in the Department of Atmospheric and Environmental Sciences at the University of Albany-SUNY. "Being able to access this data through Unidata will streamline and expedite our research."In Costa Rica, agencies are planning to use the GOES-16 data from Unidata for weather forecasting and research. In addition, the data will help with monitoring water levels for hydropower to avoid possible power cuts during the dry season, as well as for observing volcanic ash that can affect aviation and farming near San Jose."Several institutions will be using the new GOES-16 data in ways that will help safeguard society from potential natural disasters as well as avoiding energy shortages," said Marcial Garbanzo Salas, an atmospheric sciences professor at the Universidad de Costa Rica (University of Costa Rica). "This is extremely important to us, and we're very pleased that Unidata will be making it available."Writer/contact:David Hosansky, Media Relations ManagerFunder:National Oceanic and Atmospheric Administration

EarthCube Webinar: Survey Analysis Via Visual Exploration (SuAVE)

The November EarthCube webinar features Ilya Zaslavsky from the San Diego Supercomputer Center on SuAVE (Survey Analysis via Visual Exploration), a new online tool for sharing and visually exploring surveys and image collections.

Free EarthCube Webinar: Doing Geoscience with EarthCube Tools

The EarthCube Tools Webinar series provides practical demonstrations of how NSF-funded EarthCube projects can help you to collect, access, share, and visualize geoscience data.  Each one-hour webinar showcases an EarthCube project followed by ample time for questions and conversation. For instructions on how to call in to this free event, click here.

Our next webinar is Friday, Oct. 7, at noon (2 pm EDT)

Forecast for big data: Mostly cloudy

May 31, 2016 | The rise of big data has big implications for the advancement of science. It also has big implications for the clogging of bandwidth. The growing deluge of geoscience data is in danger of maxing out the existing capacity to deliver that information to researchers. In response, scientific institutions are experimenting with storing data in the cloud, where researchers can readily get the relatively small portion of the data they actually need. Helping blaze the way is Unidata, which partnered with Amazon Web Services last year to make Next Generation Weather Radar (NEXRAD) data from the National Oceanic and Atmospheric Administration (NOAA) available in the cloud in near real time. The project is one of the ways Unidata, a community program of the University Corporation for Atmospheric Research (UCAR), is exploring what the future of data access may look like. "One of the roles we play at Unidata is to see where the information technology world is going and monitor the new technologies that can advance science," said Unidata Director Mohan Ramamurthy. "In the last 10 years, we've watched the cloud computing environment mature. It's become robust and reliable enough that it now makes sense for the scientific community to begin to adopt it." Inside an Amazon Web Services data center. (Photo courtesy Amazon.) The data deluge Since 1984, Unidata has been delivering geoscience data in near real time to researchers who want it. Today, Unidata also offers those scientists tools they can use to analyze and visualize the data. In 2008, Unidata's servers delivered 2.7 terabytes of data a day to 170 institutions. Just five years later, the program was providing 13 terabytes—or the equivalent of about 4.5 million digital photos—a day to 263 institutions. Today, Unidata is delivering about 33 terabytes of data a day. And the volume is only expected to grow.  For example, NOAA's new weather satellite, GOES-R (Geostationary Operational Environmental Satellite R-Series), is scheduled to launch in October. When GOES-R is up and running, it alone will produce a whopping 3.5 terabytes of data a day. "We've been pushing out data for 30-plus years here at Unidata," said Jeff Weber, who is heading up Unidata's collaboration with Amazon. "What we're finding now is that the volume of available data is just getting to be too large," We can't keep putting more and more data into the pipe and pushing it out—there are physical constraints." The physical constraints are not just on Unidata's side. Many universities and other institutions that rely on Unidata do not have the local bandwidth to handle a huge increase in the incoming stream of data. To address the problem, Unidata decided a few years ago to begin transitioning its services to the cloud—a network of servers hosted on the Internet that allow you to access and process data from anywhere. The vision is to create a future where scientists could go to the cloud, access the data they need, and then use cloud-based tools to process and analyze that data. At the end of their projects, scientists would download only their finished products: a map or graph, perhaps, or the results from a statistical analysis. "With cloud computing, you can bring all your science and the analytic tools you use to the data, rather than the old paradigm of bringing the data to your tools," Ramamurthy said.  'Navigating the waters' These advantages were part of the motivation behind the U.S. Department of Commerce's announcement last spring that NOAA would collaborate with Amazon, Google, IBM, Microsoft, and the Open Commons Consortium with the goal of "unleashing its vast resources of environmental data" using cloud computing. A NEXRAD data product available to researchers through Unidata. (Image courtesy Unidata.) Amazon Web Services was one of the first out of the gate on the NOAA Big Data Project, uploading the full archive of NEXRAD data to the cloud last summer. But to figure out how to continue to feed the archive with near real time observations and to help make sense of the data — how people might want to use it and what kinds of tools they would need — Amazon turned to Unidata. "It made a lot of sense for Unidata to partner with Amazon and vice versa," Ramamurthy said. "They wanted expertise in atmospheric science data. We wanted an opportunity to introduce cloud-based data services to our community and raise awareness about what it can do." The scientific community is perhaps more hesitant to rely on the cloud than other user groups. Datasets are the lifeblood of many research projects, and knowing that the data are stored locally offers a sense of security for many scientists, Ramamurthy said. Losing access to some data could nullify years of work. But the truth is that the data are likely more secure in the cloud than on a local hard drive, Ramamurthy said. "Mirroring" by multiple cloud servers means that data are always backed up. If the Amazon project, and the NOAA Big Data Project in general, are successful in winning scientists over, it could go a long way toward helping Unidata make its own transition to the cloud. Unidata will be studying and learning from the project – including how to make a business model that will work -- with an eye toward its own future. "We're navigating the waters to find out what works and what doesn't so we can report back to the National Science Foundation," Weber said. "We want to see how this paradigm shift might play out — if it makes sense, if it doesn't, or if it makes sense in a few ways but not others." Writer/contactLaura Snider, Senior Science Writer and Public Information Officer

UCAR to support EarthCube: Cyberinfrastructure will advance science

BOULDER – EarthCube, a landmark initiative to develop new technological and computational capabilities for geosciences research, will be supported by the University Corporation for Atmospheric Research (UCAR) under a new agreement with the National Science Foundation (NSF). Created by NSF in 2011, EarthCube aims to help researchers across the geosciences from meteorology to seismology better understand our planet in ways that can strengthen societal resilience to natural events. More than 2,500 EarthCube contributors – including scientists, educators, and information professionals – work together on the creation of a common cyberinfrastructure for researchers to collect, access, analyze, share, and visualize all forms of data and related resources. "EarthCube offers the promise to advance geoscience research by creating and delivering critical new capabilities,” said UCAR scientist Mohan Ramamurthy, principal investigator and project director of the new EarthCube office at UCAR. "This is a great opportunity for UCAR to leverage its successful track record in managing large scientific projects that advance our understanding of the planet," said Michael Thompson, interim UCAR president. "The EarthCube project offers the potential to significantly benefit society by helping scientists use the power of diverse big datasets to better understand and predict the natural events, from severe storms to solar disturbances, that affect all of us." EarthCube is designed to foster collaborations across the geosciences. The technology helps scientists in different disciplines better understand the far-reaching influences of natural events, such as how major storms like Sandy (above) affect coastal and inland flooding. This unique view of Sandy was generated with NCAR's VAPOR visualization software, based on detailed computer modeling. (©UCAR. Visualization by Alan Norton, NCAR, based on research by NCAR scientists Mel Shapiro and Thomas Galarneau. This image is freely available for media & nonprofit use. Click here for higher resolution.) UCAR will administer the day-to-day operations of EarthCube under the three-year, $2.8 million agreement with NSF. The EarthCube science support office, currently funded through an NSF grant to the Arizona Geological Survey in Tucson, Arizona, will move to UCAR's Boulder offices starting this month. EarthCube is designed to help researchers across the geosciences address the challenges of understanding and predicting the complexity of the Earth system, from the geology and topography to the water cycle, atmosphere, and space environment of the planet. This approach is critical for improved understanding of the environment and better safeguarding society. In order to better predict the potential effects of a landfalling hurricane on inland mudslides, for example, scientists from multiple disciplines, including meteorology, hydrology, geography, and geology, need a common platform to work together to collect observations, ingest them into advanced computer models of the Earth system, and analyze and interpret the resulting data. "The EarthCube Science Support Office will help us find and share the data geoscientists collect and use to answer critical science questions about the Earth," said Eva Zanzerkia, program director in NSF’s Division of Earth Sciences. Ramamurthy said UCAR is well positioned to help EarthCube meet its goals, since UCAR provides technological support to the geosciences community, including its 109 member universities. UCAR has been involved with EarthCube since NSF launched the initiative. "Currently researchers are spending an enormous amount of time on routine tasks because there is no data system, database, or data infrastructure where they can get all the information they need in some kind of a uniform way from a single interface," Ramamurthy said. "If EarthCube can facilitate the integration of data from multiple domains in a way that is easier and faster, and if there is interoperability in terms of standards for data to be input into a common environment, then integration becomes more easily possible." UCAR is a nonprofit consortium of more than 100 member colleges and universities focused on research and training in the atmospheric and related Earth system sciences. UCAR’s primary activity is managing the National Center for Atmospheric Research (NCAR) on behalf of NSF, NCAR’s sponsor. UCAR also oversees a variety of education and scientific support activities under the umbrella of the UCAR Community Programs, which will administer EarthCube.

Patricia: A global view

October 23, 2015 | As the most intense hurricane on record in the Western Hemisphere, Hurricane Patricia is taking direct aim at Mexico’s Pacific coast and expected to spawn impacts as far away as Texas and the central United States. This image, created with a specialized visualization system built at the University Corporation for Atmospheric Research, shows Patricia's strength as it approaches Mexico as well as the connections the storm is making with other atmospheric disturbances further away. It uses global satellite infrared measurements that produce thermal images of clouds. The reds and blues show colder, higher cloud tops that are associated with towering cumulus clouds and more powerful storms. With Patricia and its 200 mile-per-hour winds bearing down on them, residents across a broad region of southwestern Mexico are being warned to prepare for potentially catastrophic impacts. The storm also is beginning to interact with low pressure over the central United States. The image shows how Patricia's outer reaches flow into a band of clouds that stretches across the country's mid-section. This connection is expected to feed the low-pressure system, potentially bringing more rain to parts of the country. After Patricia moves inland tonight, the storm’s central core will be torn apart by Mexico’s mountains but its incredible moisture will continue on to Texas, where forecasters fear a major flooding event in the next two to three days. Patricia also appears to be interacting with Hurricane Olaf, a less-intense system clearly visible in this image in the central Pacific, far to the west of Patricia. "When we look at a hemispheric view like this, we can see teleconnections across thousands of miles," said Jeff Weber, a meteorologist with UCAR’s Unidata program who created this image. "Olaf and Patricia are in phase with each other, and their movements appear to be related. Patricia may be drawing in some of the moisture that Olaf is bringing up into the atmosphere." Although not directly linked to the hurricanes, the V-shaped cloud structure in the northern Pacific also signals stormy weather ahead. These are a pair of extratropical cyclones creating an amplified wave train. This will likely affect the U.S. west coast over the next seven to ten days, creating stormy and windy conditions, with cold air to the south and warm air to the north. "We’ve got a lot of activity in this part of the world right now," Weber said. "It's all related to El Niño and the unusually warm waters in the Pacific." Weber made the image using the Integrated Data Viewer. This specialized software, created by Unidata, helps scientists visualize and analyze atmospheric conditions. Writer/contactDavid Hosansky

Unidata Users Workshop

The Unidata Users Committee invites you to join Unidata staff, community members, and distinguished speakers this June for a workshop is to raise awareness of important new trends in geoscience technology, including cloud computing, data management, and the place of the Python language in geoscience computing infrastructure. The workshop is a chance for the academic community to share hands-on activities, course materials, and ideas for improving research and education.


Subscribe to Unidata