Yellowstone supercomputer delivers early results

April 3, 2013 | Can something as gentle as sea spray influence a hurricane’s might? That’s an important question for atmospheric scientists as they work toward improving forecasts of hurricanes and other powerful storms over the water. But it’s no easy matter to figure out how millions of saltwater droplets moving chaotically between an enormous expanse of ocean and a rotating storm that reaches to the stratosphere are affecting the course of nature. At a special March 14 seminar, David Richter laid out the challenges of this type of research. A postdoctoral fellow at NCAR, Richter is creating simulations of the droplets by drawing on the vast power of the new Yellowstone supercomputer—the pioneering petaflop system that promises to usher in a new era of discovery in the geosciences. He believes that such simulations will shed light on the influence of spray on transferring additional heat and moisture from the ocean surface to the atmosphere, thereby increasing the strength of tropical cyclones. Sea spray's role in hurricanes. Using the Yellowstone supercomputer, scientists can simulate sea spray's complex movements between the ocean and atmosphere. Such work can help researchers better understand the potential impacts of spray on the strength of hurricanes. View the animation. (@UCAR. Image by John Clyne, Yannick Polius, and David Richter, NCAR. This image is freely available for media & nonprofit use.) The research can help researchers learn about on other problems related to the movement of particles, such as landslides of sand beneath the ocean that can rupture underwater fiber optic cables or damage offshore drilling platforms. The March 14 seminar illustrated the diversity of science that is already being performed on Yellowstone by scientists at NCAR and across the university community. Supported by the National Science Foundation and housed at the new NCAR-Wyoming Supercomputing Center in Cheyenne, Yellowstone began operations at the end of September 2012. The seminar featured 3 of the 11 projects whose computing was carried out at the NWSC through the Accelerated Scientific Discovery initiative. ASD allowed a select group of computing-intensive projects to be carried out during the first several months of Yellowstone’s operations. Scientists are now beginning to analyze those results. (See our roundup of ASD projects.) The hunt for weather events While Richter is focusing on the motion of tiny droplets that can be measured in millimeters, NCAR scientist Justin Small is looking at atmospheric processes that span tens to hundreds of kilometers. He is using Yellowstone to run the powerful Community Earth System Model (CESM), simulating atmospheric events in more detail than is typical for global climate studies. Weather fronts and storms in a global model. When run on the Yellowstone supercomputer, the Community Earth System Model simulates global climate in considerable detail. This animation captures such small-scale events as weather fronts moving across North America and tropical cyclones forming over the tropical oceans. (@UCAR. Visualization by Tim Scheitlin and Justin Small, NCAR. This video is freely available for media & nonprofit use.) Thanks to the computational power of Yellowstone, Small was able to simulate the world’s climate at a resolution as fine as roughly 15 miles (25 kilometers) between points on a three-dimensional grid. By depicting global climate at this level of detail, scientists can begin to capture smaller-scale atmospheric events such as tropical cyclones, strong midlatitude fronts, and intense storm systems. These storms, moreover, interact with a more realistic ocean environment. Since ocean conditions influence storm intensity, this is an important feature that is not possible with coarser-scale simulations. Small demonstrated a new CESM visualization of atmospheric processes, including fronts and tropical cyclones, moving across the globe—a possible preview of the next generation of climate simulations that will begin to look more like weather models. A portrait of tomorrow’s pollution The future of pollution. Initial studies using the Yellowstone supercomputer indicate that a warming climate will aggravate ozone pollution over the United States in the middle of the century. However, if emissions of pollutants continue to decline, U.S. ozone levels should improve even as temperatures rise. (@UCAR. Image by Gabriele Pfister, NCAR. This image is freely available for media & nonprofit use.) NCAR’s Gabriele Pfister is drawing on Yellowstone’s power to answer a very different question: what will U.S. air quality be like in mid-century, based on current projections in pollution emissions and climate? Some of Pfister’s early computer runs indicate the extent to which warming temperatures are likely to worsen incidents of ground-level ozone and other types of pollutants. On the other hand, her initial runs also indicate that the nation may be able to reduce the number of times that air pollution levels exceed public health standards if progress continues in reducing some emissions. This research also has climate implications. Some of the initial computer runs indicate that local pollutants such as certain gases or aerosols (tiny airborne particles or droplets) can nudge temperatures up or down by about 2-3 degrees Fahrenheit over some regions. Speed matters Pfister noted that her research, which involved creating realistic simulations of atmospheric chemistry that interacted with a detailed climate model, would have taken far too long with Bluefire. “I would have been planning to analyze the data by the time I retired,” she said. Other ASD projects seek new understandings of earthquakes, solar storms, and weather forecasting. While six of the projects were led by NCAR principal investigators, the other five were led by researchers at the University of Colorado Boulder, University of Southern California, Cornell University, University of Delaware, and the Center for Ocean-Land-Atmosphere Studies. Researchers hope to begin formally writing up their results and submitting them for peer review as soon as the end of this year. More about Yellowstone and the NWSC NCAR-Wyoming Supercomputing Center opens (October 15, 2012) Multimedia Gallery Fact Sheet In Depth: Big Data

First Place: NCAR-Wyoming Supercomputing Center Recognized for Outstanding Design Implementation

News Release Multimedia Gallery Fact Sheet More NWSC News   BOULDER—The National Center for Atmospheric Research (NCAR) has taken top honors in the prestigious 2013 Green Enterprise IT (GEIT) Awards. The center’s NCAR-Wyoming Supercomputing Center won first place in the “Facility Design Implementation” category for its sustainable approach in designing and building the new NCAR-Wyoming Supercomputing Center (NWSC). The NCAR-Wyoming Supercomputing Center in Cheyenne, Wyoming, which opened last year, has received international recognition for its sustainable approach. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.) The closely watched GEIT Awards, bestowed by the Uptime Institute, showcase organizations for pioneering projects and innovations that significantly improve energy productivity and resource use in information technology. The Facility Design Implementation Award recognizes cutting-edge data center projects that demonstrate energy and resource efficiency in a new, operational data center. “We are honored and pleased to receive this recognition for the NWSC,” says Aaron Andersen, Deputy Director of Operations & Services at NCAR’s Computational and Information Systems Laboratory. “Nearly 10 years of planning and hard work went into designing this facility to be as sustainable as possible, and it is gratifying to have the facility in production use and be able to share what we’ve done. We hope this facility advances the entire industry.” “Our goal is to meet the highest standards possible for sustainability in supercomputing while advancing scientific knowledge,” says Thomas Bogdan, president of the University Corporation for Atmospheric Research (UCAR), which manages NCAR on behalf of the National Science Foundation (NSF). “The GEIT Award suggests we’re on the right track, and we are deeply gratified by this international recognition.” H+L Architecture and the engineering firm RMH Group shared the award with NCAR for their role in the design of the facility and its systems. Construction of the NWSC was made possible by the sponsorship of NSF and a unique collaborative partnership between local, state, and federal government entities and private industry to provide project funding and governance. The NWSC is operated by NCAR on behalf of NSF and UCAR. “The State of Wyoming is proud to be a partner in this supercomputing facility,” says Wyoming Gov. Matt Mead. “The designers did an excellent job putting to work Wyoming’s natural advantages for data centers, and I join in congratulating them on this award.” The NWSC's complex mechanical systems were designed with performance, flexibility, and energy efficiency in mind.  (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.) “My congratulations to the NCAR-Wyoming Supercomputing Center for its international recognition as a world-class facility,” says Wyoming Congresswoman Cynthia M. Lummis, a long-time supporter of the center. “NWSC's design capitalizes on Wyoming's climate and availability of renewable resources while also incorporating other best practices to achieve an energy efficient and sustainable facility. Considering the leading computational research conducted at the NWSC, it seems only fitting that the facility housing this research be recognized for its key role in enabling advancements in the field of supercomputing.” The NWSC, which is home to one of the most powerful supercomputers dedicated to Earth system science, also achieved a Leadership in Energy and Environmental Design Gold certification from the US Green Building Council last year. LEED certification depends on a number of sustainable criteria, such as energy efficiency, water conservation, and the use of recycled or locally sourced construction materials. “Our organization set out to build a world-class data center for scientific computing that would raise the bar in sustainability, longevity, and manageability,” says NCAR interim director Maura Hagan. “Today we are excited to be advancing science through supercomputing from a facility that is living up to those goals.” The 2013 GEIT Awards are sponsored by Sabey Data Centers. Entries were thoroughly reviewed by an international committee of independent judges following a double-blind process. NCAR will receive the award at the eighth annual Uptime Institute Symposium in Santa Clara, California, in May. As part of the symposium’s agenda, NCAR and a portion of the design team will present a case study about the NWSC to the event audience. NWSC Design Highlights Sustainable materials: During construction, more than 70% of construction waste was diverted from landfills and used for recycling. The building itself is made with over 510 tons of recycled concrete, 60 tons of recycled wood, and 26 tons of recycled metal. Water: The ultra-efficient cooling tower configuration as well as use of native species for landscaping enables water savings of up to 6 million gallons per year. Infrastructure and Space Use: The center's super-efficient use of energy means that the mechanical systems, electrical systems, and office space expend less than 10% of the total power consumption. Heating: Waste heat from the supercomputer is captured and reused to heat the building and melt snow and ice on exterior walkways and loading docks. Cooling: During design, project planners estimated that Wyoming’s cool, dry climate would allow natural cooling of the facility for 96% of the year. Early experience with the facility indicates that 98% to 99% is achievable. Power: Renewable wind energy provides direct power to the facility, starting at 10% of supply with the ability to raise that percentage as conditions permit. Flexibility and Longevity: The design of the NWSC includes “future proofing” to anticipate adaptation to evolving technologies and deployment of future supercomputing systems yet to be developed. The design is also highly modular, allowing critical power and cooling components to be provisioned only when needed. This enhanced flexibility helps minimize capital expenditures by providing only what is needed when it is needed. Combining all of these factors, the NWSC not only minimizes the environmental footprint but also directs operating funds toward productive scientific work while reducing overhead expenses. About Uptime Institute Uptime Institute provides independent thought leadership, certification, education, and professional services for the global digital infrastructure industry. It serves all industry stakeholders, including enterprise and third-party data center owners and operators, manufacturers, service providers, and engineers. Through Uptime Institute Professional Services, Uptime Institute delivers due diligence assessments and certifications of site infrastructure and site management in accordance with the Tier and Operational Sustainability Standards. Uptime Institute is a division of The 451 Group.  Headquartered in New York, The 451 Group also owns 451 Research, a leading technology-industry syndicated research and data service focused on the business of enterprise IT innovation, and Yankee Group, the preeminent research and advisory firm equipping companies to profit in a mobile world.

NCAR-Wyoming Supercomputing Center opens

News Release Multimedia Gallery Fact Sheet FAQ More NWSC News   CHEYENNE—The NCAR-Wyoming Supercomputing Center (NWSC), which houses one of the world’s most powerful supercomputers dedicated to the geosciences, officially opens today. Scientists at the National Center for Atmospheric Research (NCAR) and universities across the country are launching a series of initial scientific projects on the center’s flagship, a 1.5-petaflop IBM supercomputer known as Yellowstone. These first projects focus on a wide range of Earth science topics, from atmospheric disturbances to subterranean faults, that will eventually help to improve predictions of tornadoes, hurricanes, earthquakes, droughts, and other natural hazards. A handful of the Yellowstone supercomputer's 100 racks. An iconic scene from its namesake national park is featured mosaic-style on the ends of each rack. The image by Michael Medford, licensed to National Geographic, centers on Fountain Geyser. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.) View more images in the Multimedia Gallery. “This center will help transform our understanding of the natural world in ways that offer enormous benefits to society,” says Thomas Bogdan, president of the University Corporation for Atmospheric Research (UCAR), which manages NCAR on behalf of the National Science Foundation (NSF). “Whether it’s better understanding tornadoes and hurricanes, or deciphering the forces that lead to geomagnetic storms, the Yellowstone supercomputer and the NWSC will lead to improved forecasts and better protection for the public and our economy.” Bogdan took part this morning in a formal opening ceremony with Wyoming Gov. Matt Mead, NSF director Subra Suresh, University of Wyoming (UW) vice president of research relations William Gern, NCAR director Roger Wakimoto, and other political and scientific leaders. “This is a great day for scientific research, for the University of Wyoming, and for Wyoming,” says Mead. “Wyoming is proud to be part of the collaboration that has brought one of the world’s fastest computers to the state. The center will have a positive impact on our future, through the research done here and by sending the message that Wyoming is honored and equipped to be the home of this amazing facility.” “The NCAR-Wyoming Supercomputing Center will offer researchers the opportunity to develop, access, and share complex models and data at incredibly powerful speeds,” says Suresh. “This is the latest example of NSF's unique ability to identify challenges early and make sure that the best tools are in place to support the science and engineering research communities.” Public-private partnership Located on the western fringe of Cheyenne, Wyoming, the NCAR-Wyoming Supercomputing Center officially opened on October 15, 2012. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.) View more images in the Multimedia Gallery. The NWSC is the result of a broad public-private partnership among NCAR, NSF, UW, the state of Wyoming, Cheyenne LEADS, the Wyoming Business Council, and Cheyenne Light, Fuel & Power. NCAR’s Computational and Information Systems Laboratory (CISL) will operate the NWSC on behalf of NSF and UCAR. “We are delighted that this successful public-private partnership has delivered a major supercomputing center on time and on budget,” says NCAR director Roger Wakimoto. Through the NWSC partnership, which will also seek to advance education and outreach, UW will have research use of 20 percent of NWSC’s main computing resource. In turn, UW will provide $1 million each year for 20 years in support of the program. The state of Wyoming also contributed $20 million toward the construction of the center. “Our access to Yellowstone will allow the university to reach new heights in our educational and research endeavors in engineering; atmospheric, hydrological, and computational sciences; Earth system sciences; and mathematics,” says UW President Tom Buchanan. “The supercomputer is a huge draw for students and faculty. It opens the door to scientific innovation and discovery that will benefit our state, the nation, and the world.” Located in Cheyenne’s North Range Business Park, near the intersection of I-80 and I-25, the 153,000-square-foot supercomputing center will provide advanced computing services to scientists across the United States. Most researchers will interact with the center remotely, via a laptop or desktop computer and the Internet. Relative to the most recent ranking of the world’s fastest supercomputers, issued in June, the 1.5 petaflops peak system ranks in the top 20. The rankings constantly change as new and increasingly powerful supercomputers come online. The main components consist of a massive central file and data storage system, a high performance computational cluster, and a system for visualizing the data. Scientists will use the new center’s advanced computing resources to understand complex processes in the atmosphere and throughout the Earth system, and to accelerate research into severe weather, geomagnetic storms, climate change, carbon sequestration, aviation safety, wildfires, and other critical geoscience topics. Future-proof design Some of the NWSC's complex mechanical systems, which were designed with performance, flexibility, and energy efficiency in mind.  (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.) View more images in the Multimedia Gallery. CISL has operated supercomputers at NCAR’s Mesa Laboratory in Boulder since the 1960s, even though the building was not designed with supercomputing in mind. In recent years, new research questions have required more powerful computers to run increasingly complex computer simulations. The Mesa Lab has now reached the limits of its ability to provide the necessary energy and cooling capacity essential for the next generation of supercomputers. The NWSC is expected to advance scientific discovery for the next several decades. Its design and construction have been "future proofed" by providing the scope to expand as supercomputing technology that does not exist today becomes available in the future. Raised floors are key to the facility’s flexible design, allowing the computing systems, electrical supply, and cooling to be positioned and controlled for optimal energy use and ease of maintenance. The raised floor is also vented, so air can be circulated as needed to computing systems and servers. The NWSC was awarded LEED Gold certification for its sustainable design. The center takes full advantage of Cheyenne’s elevation and cool, dry climate by employing ambient air to cool the facility nearly year round. This will significantly reduce the facility's energy use. A minimum of 10 percent of the power provided to the facility will be wind energy from the nearby Happy Jack Wind Farm. NCAR and UCAR will continue to explore options to increase the percentage of renewable energy provided to the facility in future years.

Supercomputing with Yellowstone - Multimedia Gallery

Multimedia Gallery Fact Sheet FAQ NWSC News   On this page Scientific visualizations:  Video  |  Images  Photos:  Supercomputer & facility   Scientific Visualizations Using Supercomputing - Video      Coupled Weather-Fire Simulation of the Esperanza Wildfire. This fire-behavior simulation reproduces the October 2006 Esperanza Fire near Cabazon, California. For description and credits please see the YouTube description. (©UCAR. This video is freely available for media & nonprofit use.)      Intense Storm Observed during the ERICA Field Campaign. The low-pressure center modeled here, called an extratropical cyclone, was observed over the central North Atlantic Ocean in early January 1989. For description and credits please see the YouTube description. (©UCAR. This video is freely available for media & nonprofit use.)     Solar Magnetic "Tornado." Computer modeling based on a discovery made in 2012. A virtual camera travels around, above, and into a funnel of rotating solar magnetism.  For description and credits please see the YouTube description. (©UCAR. This video is freely available for media & nonprofit use.) Scientific Visualizations Using Supercomputing - Images Sunspot visualization. The interface between a sunspot's umbra (dark center) and penumbra (lighter outer region) shows a complex structure with narrow, almost horizontal (lighter to white) filaments embedded in a background having a more vertical (darker to black) magnetic field. Farther out, extended patches of horizontal field dominate. In a first, NCAR scientists and colleagues modeled this complex structure in a comprehensive 3D computer simulation, giving scientists an unprecedented glimpse below the visible surface to understand a sunspot's underlying physical processes. more about this study >   (©UCAR, image courtesy Matthias Rempel, NCAR.)   Earth’s climate system. This image depicts a single month from a simulation of the 20th century by the NCAR-based Community Climate System Model (version 4). The CCSM4 is one of the world’s most powerful computer models for simulating the complex interactions of Earth’s climate system, including the atmosphere, oceans, sea ice, and land surface. This image captures wind directions, ocean surface temperatures, and sea ice concentrations. (©UCAR, image courtesy Gary Strand, NCAR.) Photos of the NWSC: Supercomputer and facility A fish-eye view of some of the Yellowstone supercomputer's 100 racks. An iconic scene from Yellowstone National Park is featured mosaic-style on the ends of each rack. The image by Michael Medford, licensed to National Geographic, shows Fountain Geyser. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.)   Side view of some of the Yellowstone supercomputer's 100 racks. An iconic scene from Yellowstone National Park is featured mosaic-style on the ends of each rack. The image by Michael Medford, licensed to National Geographic, centers on Fountain Geyser. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.) The building housing the NCAR-Wyoming Supercomputing Center in Cheyenne, Wyoming, officially opened on October 15, 2012. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.)   The NWSC visitor center features views of the Yellowstone system and five stations scattered throughout the lobby that explore the many facets of supercomputing and related science. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.)   Housed in a set of 100 interconnected cabinets, the Yellowstone system includes more than 70,000 processors, as well as high-performance bandwidth, memory, and visualization functions to transmit, store, and view the results. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.)   Over a dozen miles of cable permit Yellowstone's tens of thousands of processors and other key components to interact with each other. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.) Beneath the floor of the supercomputing rooms lies a vast, 10-foot high utility space, the key to the facility’s flexible, energy efficient design. The electrical supply and cooling systems, including the fans at right in this photo, can be positioned and controlled for optimal energy use, and air can be circulated as needed to computing systems and servers. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.)     Some of the NWSC's complex mechanical systems, which were designed with performance, flexibility, and energy efficiency in mind.  (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.)

Key moments in NCAR supercomputing

 October 5, 2012 State-of-the-art computing and big-data processing have been part of NCAR from the center’s earliest years. Here's a snapshot of selected hardware advances. Micro-processorpeakteraflops  Memory(terabytes) Processors  Years inservice 1963 First supercomputer   CDC 3600 (Control Data)NCAR staff wrote the operating system  0.000001   0.000000032    1    2.5   1977 First vector-based system   C1 (CRAY-1A, Cray Computer)NCAR becomes Cray Computer's first customer for this vector-processing system  0.000160    0.000008    1    12   1988 First parallel processing system Capitol (Connection Machine 2,Thinking Machines)With over 8,000 processors, thismachine enabled the NCAR-University of Colorado Center for AppliedParallel Processing  0.00717    0.001    8,192    4.5   1999 Transition to clusteredshared-memory processors Blackforest (SP RS/6000, IBM)NCAR makes a major investment inconversion from vector-based toparallel processing systems   1.96     0.7    1,308   5.5   2008 Supercharged speed and efficiency Bluefire (Power 575 Hydro-Cluster, IBM)The first in a highly energy-efficientclass of water-cooled machines to beshipped anywherein the world arrives,with speed that more than triplesNCAR's computing capacity  77    12.2    4,096    4.5   2012 The next great leap   Yellowstone (iDataPlex, IBM)NCAR enters petascale computing withthis massively parallel machine  1,600    149.2    74,592    new     More supercomputing history at NCAR NCAR's Computational and Information Systems Lab provides a detailed timeline of our supercomputers.

First up: Accelerated scientific discovery

October 8, 2012 | First in the queue for Yellowstone is a set of 11 computing-intensive projects approved as part of the two-month-long Accelerated Scientific Discovery initiative. Chosen from applicants at NCAR and in the university community, the ASD projects will serve to inaugurate Yellowstone, carrying out large amounts of computing over a short period, while tackling major problems in Earth and atmospheric science. For more detail, see the formal descriptions on the ASD site for university-led projects and NCAR-led projects. TopicsOcean fronts and turbulence3-D mapping of Earth's upper layersInside cumulus cloudsBetter long-range forecastsHigh-resolution climate: local to globalTurbulent clouds in detail               Air pollution projections through 2055When turbulence and rotation meetExtending predictabilitySolar magnetism and electricity in spaceSea spray and the atmosphere Ocean fronts and turbulence        Baylor Fox-Kemper, University of Colorado Boulder, project leadArrest of frontogenesis in oceanic submesoscale turbulence Goal: Resolve the nature of the process that impedes sharpening of oceanic fronts (roughly 1–10 kilometers or 0.6–6 miles wide) in regions of turbulence. Such fronts are common over the globe’s ocean surface; they separate areas of differing temperature and salinity and influence circulation patterns and other key aspects of oceans and air-sea exchanges. 3-D mapping of Earth's upper layers        Thomas Jordan, University of Southern California, project leadCommunity computational platforms for developing three-dimensional models of Earth structure Goal: Image Earth’s upper layers by full 3-D tomography using two different approaches. The resulting improvements in regional and global models will better characterize the flow within Earth’s mantle, the evolution of plate-tectonic forces, and the potential seismic effects of earthquakes and nuclear explosions. Inside cumulus clouds      Lance Collins, Cornell University, project leadDirect numerical simulation of cumulus cloud core processes over larger volumes and for longer times Goal: Simulate particle-turbulence interactions in conditions that mimic cumulus cloud cores, over distances ranging from millimeters up to a few meters, for periods of about 20 minutes. These simulations will help improve models of cloud dynamics and could also benefit climate modeling. Better long-range weather forecasts      William Skamarock, NCAR, project leadGlobal cloud-permitting atmospheric simulations using MPASGoal: Test the performance of the Model for Prediction across Scales (MPAS) by producing 10-day forecasts for two periods. By varying the spacing between the centers of each hexagonal grid cell from 60 kilometers down to 3 km in five steps, the experiment will help determine where and when higher resolutions are most critical to forecast quality. High-resolution climate: local to global      R. Justin Small, NCAR, project leadMeso- to planetary-scale processes in a global ultra-high-resolution climate model Goal: Conduct and analyze simulations using the Community Earth System Model with roughly 100 times more grid points than commonly used. Among other outcomes, the study will shed light on how climate responds to the coupling of ocean and atmospheric fronts within a model and how features such as polynyas (holes within Arctic sea ice) affect climate. Turbulent clouds in detail      Andrzej Wyszogrodzki, NCAR
, project leadPetascale simulation of physics and dynamics of turbulent clouds Goal: This project will close the spatial gap between two numerical approaches to model cloud dynamics and cloud microphysics at scales ranging from the size of small cumulus clouds (about 1.0 mile) down to cloud droplets (about 0.0005 inches). The results will benefit weather and climate models at regional and global scales. Air pollution projections through 2055      Gabriele Pfister, NCAR
, project leadPrediction of North American air quality Goal: Perform high-resolution simulations with a nested regional climate model that includes interactions between chemistry and meteorology, in order to study possible changes in weather and air quality over North America between present and future time periods (2020–2030 and 2045–2055). The analysis will focus on summertime U.S. air pollution events. When turbulence and rotation meet      Annick Pouquet, NCAR, project leadRotation and stratification at unit Burger number in turbulent flows Goal: Examine the role of helicity (corkscrew-like motion) in turbulent fluids that are both stratified and rotating. Because this type of fluid behavior resembles key aspects of Earth’s atmosphere, the findings could help illuminate some of the complexities of flow in and around tornadoes, hurricanes, and other cyclones. Extending predictability      James Kinter (pictured) and Ben Cash, Center for Ocean-Land-Atmosphere Studies, project leadsTowards seamless high-resolution prediction at intraseasonal and longer timescales Goal: Explore the impact of increased atmospheric resolution on model fidelity and prediction skill in the operational ECMWF (European Centre for Medium-Range Weather Forecasts) coupled climate model. The results will help understand and quantify predictability in weather and climate for periods ranging from days to years. Solar magnetism and electricity in space      Project lead: Michael Shay, University of DelawareTurbulence in the heliosphere: The role of current sheets and magnetic reconnection Goal: Perform the first systematic study of how thin sheets of electric current generated by the Sun reconnect in the presence of turbulence. Understanding the process through which intense concentrations of energy dissipate during these reconnections is among the major challenges of solar physics. Sea spray and the atmosphere      David Richter, NCAR, project leadTurbulence modification in the spray-laden atmospheric marine boundary layer Goal: Examine the effect of sea spray suspended by turbulence above the ocean on the transfer of heat and momentum to the ocean surface. The results will help improve understanding of how effects related to sea spray could influence the development of hurricanes.  

Power meets efficiency

October 5, 2012 | Though it’s become a trendy concept, “big data” is a realm that NCAR scientists and their collaborators pioneered to help shape the nation’s infrastructure for weather prediction and related research. Largely because the atmosphere is so vast, and because tiny changes can have a major impact over time, atmospheric research involves enormous amounts of data. Over the last 40 years, hundreds of scientists at dozens of institutions have relied on NCAR’s computational muscle and massive data storage systems in order to make progress in forecasting weather, projecting future climate, and other key tasks. (See Key moments in NCAR supercomputing.) Located on the western fringe of Cheyenne, Wyoming, the NCAR-Wyoming Supercomputing Center officially opens in October 2012. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.) The latest chapter in the story of NCAR computing begins in the fall of 2012, as the NCAR-Wyoming Supercomputing Center opens for business. The center’s flagship hardware—an IBM system dubbed Yellowstone, in honor of Wyoming’s iconic national park—will rank among the speediest supercomputers in the world when it debuts. Yellowstone will feature 74,592 parallel processors, each carrying out calculations simultaneously when the supercomputer is running at its peak. “We are delighted that Yellowstone is here,” says UCAR president Tom Bogdan. “This supercomputer and the NWSC as a whole will provide a long-sought and much-needed boost to the capabilities of researchers in the atmospheric and Earth sciences.” "We've tried to design a balanced system to support data-intensive supercomputing." —Anke Kamrath, director of operations and services for NCAR's Computational and Information Systems Laboratory Years ago, it became clear that the computing center at NCAR’s Mesa Laboratory was reaching the limits of its ability to provide space, power, and cooling for ever-larger supercomputers. The center’s search for an alternative led to Cheyenne, where the NWSC now stands as the latest example of an expanding research corridor of tech-oriented development along the Front Range of the Rocky Mountains. The NWSC will advance a broad range of geoscience research, with potential benefits to society in everything from storm prediction to air quality monitoring and the assessment of energy and water resources. More results, more detail, more efficiency Yellowstone will provide almost 30 times the computing power of its NCAR predecessor, Bluefire. All else being equal, that might allow an experimental short-term weather forecast to be carried out in nine minutes versus three hours, or a climate projection to be produced in a week as opposed to half a year. The speed boost can also support more and better science in other ways, producing a greater number of simulations or increasing their complexity, depending on the goals of a given study. The NWSC offers more than sheer power, though. Both the building and the supercomputing inside have been designed with maximum efficiency in mind. Cheyenne’s windy High Plains climate provides a great deal of natural cooling; the NWSC takes advantage of it through a louvered air lock system. With the help of this and other energy-saving features, more than 90% of the power entering the NWSC goes directly to its computational mission—a notable improvement on the rates of 70% or lower found at many data centers. And even before its doors opened, the NWSC garnered a LEED Gold rating for Leadership in Energy and Environmental Design from the U.S. Green Building Council. More than 90% of the power entering the NWSC goes directly to its computational mission (IT LOAD), compared to about 66% for most data centers operating today. (Image courtesy NCAR/CISL.) The Yellowstone supercomputer is highly efficient in its own right. Compared to Bluefire, the amount of power required by Yellowstone per unit of computing will be reduced by more than 85%, thanks to a variety of improvements in computing hardware. The machine promises to serve as a powerful proving ground for computational scientists—at NCAR and at research universities and labs around the country and the world—working to maximize the usefulness and nimbleness of weather and climate models. Built with the future in mind While the NWSC’s grand opening is the climax to more than five years of planning and construction, it’s also just the beginning: the center is well equipped to serve scientists far into the future. Yellowstone occupies only part of the NWSC’s available computing spaces, or modules. Moreover, the facility is designed so that next-generation capacity can be added without disrupting current operations. Housed in a set of 100 interconnected cabinets, the Yellowstone system includes more than 70,000 processors, as well as high-performance bandwidth, memory, and visualization functions to transmit, store, and view the results. (©UCAR. Photos by Carlye Calvin.) Expanded scientific horizons will be part of the picture from the outset. The roster of initial projects encompasses a wide range of our environment’s fluid behavior, from the effects of sea spray on hurricanes to the motion of tectonic plates that leads to earthquakes. The NWSC is setting the stage for major progress in such critical areas as geophysical turbulence, which not only rattles aircraft but influences the behavior of countless processes all around us. With an ever-broadening range of disciplines represented, and new scientific partnerships already taking shape, the NWSC could go a long way toward helping us steward our planet’s resources, preserve its health, and avoid the natural hazards that so profoundly affect people and the economy. “The vision for Yellowstone parallels the principles that have guided the design of the NWSC,” says NCAR director Roger Wakimoto. “In both instances, we have taken an approach that maximizes the science we can do and the benefit of that science to society.” The NCAR-Wyoming Supercomputing Center in Cheyenne, Wyoming, takes advantage of naturally cool air and many other energy-saving features to maximize efficiency of both the building and its supercomputers. The building has also achieved LEED Gold certificaton for Leadership in Energy and Environmental Design from the U.S. Green Building Council.   Related Links NWSC Fact Sheet

A public-private success story

September 17, 2012 | A great deal of work had already taken place before workers turned over the first mound of soil on the prairie that now houses the NWSC. The public-private partnership that led to the NWSC’s creation emerged through a fortuitous mix of geographical, technological, institutional, and human variables, plus the ability to recognize opportunity and act on it. We asked two of the principals involved to share the story as they saw it first hand: Randy Bruns, head of the nonprofit economic development group Cheyenne LEADS, and Jeff Reaves, associate UCAR vice president for business services. A roster of contributing partners is here. Jeff Reaves Randy Bruns How did Cheyenne emerge as the location for the NWSC? Reaves: We looked at a number of options for meeting our need for a new data center: expanding on the NCAR mesa, leasing or remodeling an existing data center, retrofitting some other building. Then we decided to look at co-hosting it with a university along the Front Range, somewhere between Golden and Fort Collins, where there would be adequate transmission-level power, high-speed bandwidth fiber, and room for future growth. We were pretty far down the road in selecting a Front Range site when we got the call from Randy asking if we would entertain a proposal from Cheyenne. Bruns: We’d been trying to recruit companies involved in computer-centric technology. Our university and our community colleges are turning out tech-capable people, but for the most part, they leave the state for jobs. We don’t have a huge workforce in Wyoming. But we do have abundant power. And here in Cheyenne, we’re sitting on a lot of fiber that runs coast to coast but is largely untapped. Cheyenne exists because of the transcontinental railroad, and we also became an interstate corridor. And both rail and highway corridors tend to serve as corridors for data. The NWSC (turquoise pin) is located several miles west-northwest of downtown Cheyenne. (Map © 2012 Google.) Reaves: There were several proposed sites in Wyoming. The ones in Cheyenne had access to transmission-level power, which was important for us. The site we ended up choosing was about double the size of our minimum requirement for land, which offered a number of advantages in terms of mechanical and electrical infrastructure and potential expansion. Also, they were willing to deed the land [24 acres] to us, whereas some of the universities wanted to retain ownership, even with a no-cost lease. Then there were financial incentives: the land at no cost, plus infrastructure to the site. The state legislature offered $20 million in cash toward construction of the facility, and the university offered $1 million per year for the next 20 years for the ongoing operation of the facility. Cheyenne Light, Fuel & Power agreed to provide both primary and secondary power at no extra cost, so that if one substation goes down, they can switch us to another. They were also willing to provide 10% of our power needs with renewable energy from the nearby Happy Jack wind farm. Also, NSF encouraged us to consider Wyoming because it was an EPSCoR state. So there was this added benefit, which opened up opportunities to serve a broader community. [EPSCoR, the Experimental Program to Stimulate Competitive Research, helps direct funding to 31 U.S. states and territories that receive a disproportionately small fraction of NSF research funding.] Bruns: We involved the Wyoming Business Council [the state’s economic development arm], the governor’s office, and the University of Wyoming, all at the highest levels. The fortunate thing is that, in a state this size, we all know one another—the academic world, the private sector, and the government side, both state and local—so it’s fairly easy to reach across organizational boundaries. Essentially the entire legislature understood the significance of this. In a matter of hours and days, they were all on board. It wasn’t the largest construction project by any means that we’ve attracted here. It certainly wasn’t the largest employer. And yet in a very subtle but profound way, everybody recognized the potential and the power of this relationship. UW’s stake in NWSC computing Seven major University of Wyoming projects on the NWSC's Yellowstone supercomputer will address topics ranging from hydrology of the Colorado River Basin to planet formation from star debris to fluid dynamics of wind turbines. In addition, through EPSCoR (see above), the university recently received its largest-ever research grant, a $20 million, five-year award for a variety of water-related studies involving four UW colleges and 11 departments.  “UW’s computational researchers are working on projects that are of great importance to Wyoming, the U.S. and, in fact, the entire world,” says Bryan Shader, UW’s special assistant to the vice president of research and economic development. “All of this research is big science.” Have you already seen benefits in Wyoming from the NWSC? Bruns: With the prospect of this center being here, the university immediately began to make changes, with undergraduate and graduate offerings, and began expanding the relationship with NCAR in terms of personnel and shared staff. It also helped certify the case that Cheyenne in particular, and Wyoming in general, had substantial capabilities in the tech sector. It highlighted those not only for outsiders but in some respects even for people right here in Wyoming. Echostar Communications has expanded—they’re now up to more than 400 employees, all tech workers. Then in April 2012, Microsoft announced that they’re putting a huge data center complex here in Cheyenne. So the momentum is building. What about having a data center located nearly 100 miles from the rest of NCAR? Reaves: We recognized that technology had evolved to the point where you really didn’t have to have the machines collocated with the scientists. We’d already been serving the university community all over the world. Then it was a matter of how far away we could go from Boulder, which depended on broadband accessibility. One of the things that made Cheyenne doable was that BiSON already existed. [BiSON, the Bi-State Optical Network, connects the University of Wyoming to the Denver-based Front Range GigaPop, which in turn links research and higher education institutions across northern Colorado, including NCAR.] Looking to the future of big data and supercomputing, what is next for Wyoming, and what role might the NWSC play? Bruns: One of the things we’ve learned is that there’s a lot of commercial fiber here. We believe we’re in an excellent position for Microsoft and companies like them. We’re also very confident of our long-range ability to provide power to these centers. And NCAR has been willing to share much of what they learned with us, so we can talk with some authority about the unique advantages that our high, dry, cool environment provides for efficient operation of data centers. I think the real power of this center being here is an expansion of the Colorado technology corridor into Wyoming in a very tangible way. We’ve also opened the doors for expanded science at the University of Wyoming and elsewhere. At the end of the day, it’s about creating the capability to do better science, more science, faster science, in an area that’s critically important to the United States and arguably the world: understanding the environment that we live in. What are the keys to success for a public-private partnership like this one? Reaves: It’s crucial to know what your needs and requirements are and whether or not you can really satisfy them. You might be willing to make some tradeoffs, but at least you know the impacts of those tradeoffs. We knew what our then-current computing capacity was and where we needed to go in the next 10 to 20 years. We had a pretty good idea of what was required in terms of power, cooling, and computing to meet that demand. So our specs and requirements were very clearly spelled out. Another extremely important aspect is the partners themselves. I can’t emphasize enough what a great partnership we’ve developed with people in Wyoming—the legislature, the business community, the academic community. We listened and did what we could to meet their needs while satisfying our own. We developed a true partnership, up and down the line. Bruns: First of all, you have to get the right people and the right organizations to the table. In our case, there were key people at very significant points who grasped a vision for this project. [Former NCAR director] Tim Killeen was extraordinarily open to shifting directions at a time when NCAR’s selection process was starting to narrow. Governor [Dave] Freudenthal grasped the potential for this to expand the university, expand Wyoming’s presence in the tech sector, and tie our state even more solidly to science and research. The university president, Tom Buchanan, very quickly saw the big-picture, long-term implications of this center as the basis for an expanding relationship and for making significant positive changes at his institution. So there were a handful of people who intuitively understood the potential of this and committed their organizations to exploring how to make it happen. I think having people of vision like that was an amazing alignment of stars. This photo of the NWSC, looking northeast, was taken during final stages of building construction. It shows meeting areas (left) and the entrance foyer (right). (©UCAR. Photos by Carlye Calvin.)  

10 ways Yellowstone will make a difference

September 11, 2012 “The Yellowstone supercomputer will dramatically advance our understanding of Earth," says Al Kellie, director of NCAR’s Computational and Information Systems Laboratory (CISL). “Its computing capacity and speed will allow us to investigate a wide range of phenomena that affect our lives, with more detail than ever before.” Here are just 10 examples of how the new system coming online at the NCAR-Wyoming Supercomputing Center—and systems to come—are poised to tackle major challenges confronting our society.  Thunderstorms and tornadoes Scientists will be able to simulate these small but dangerous systems in remarkable detail, zooming in on the movement of winds, raindrops, and other features at different points and times within an individual storm. By learning more about the structure and evolution of severe weather, researchers will be able to help forecasters deliver more accurate and specific predictions, such as which locations within a county are most likely to experience a tornado within the next hour. Water availability Yellowstone will support global and regional climate modeling in much greater detail, helping to answer such questions as which regions will receive more or less precipitation in a warming world and which may face especially withering drought. Yellowstone’s high-resolution capabilities will also enable scientists to represent the heights of mountains more precisely within climate models, which will help project future snowpack in regions that supply water to vast agricultural tracts and millions of residents, such as California’s Sierra Nevada and the Colorado Rockies. Subsurface energy and carbon storage As the nation’s portfolio of energy options continues to grow, oil and gas remain key elements of the mix. Yellowstone can be used to model large oil and gas deposits in difficult-to-access regions several miles below Earth’s surface in greater detail. Subsurface models can also help identify areas that could be used as reservoirs to store carbon, keeping it out of the atmosphere and helping to reduce the effects of burning fossil fuels. Long-term forecasting Farmers, shipping companies, utilities, and other planners would benefit enormously from forecasts that accurately predict weather conditions a month in advance. Because large-scale oceanic and atmospheric patterns play such a major role at this time scale, scientists will rely on supercomputers such as Yellowstone to provide needed detail on the effects of these big patterns on future local weather events. Yellowstone’s size also allows for more ensembles—multiple runs of the same simulation, each with a small change in the initial conditions—that can shed important light on the skill of longer-term forecasts. Wildfires Wildfires create their own weather, interacting with nearby atmospheric conditions to drive the blazes through terrain in ways that are highly complex and difficult to simulate. By enabling more realistic modeling, Yellowstone can lead to improved predictions of fire patterns that may help both residents in threatened areas and firefighters trying to control the blaze. Arctic sea ice As the extent and thickness of sea ice diminishes, industries in the United States and elsewhere are eying the Arctic for potential shipping lanes and increased extraction of natural resources. Using supercomputers such as Yellowstone, scientists can work toward the development of seasonal forecasts of sea ice to help decision makers anticipate sea ice patterns months in advance. Climate modeling on Yellowstone also will help determine how soon the Arctic might experience ice-free conditions in summer, which could have major effects on local ecosystems and on regional weather and climate. Hurricanes While forecasters have made great strides in predicting the track of a hurricane several days in advance, it remains difficult to predict major changes in intensity or to identify which clusters of tropical thunderstorms will develop into hurricanes. The supercomputers at the NWSC will help answer those questions by enabling researchers to decipher the impacts of the complex processes involved, including sea surface temperatures, upper-level winds, regions of dry and moist air in the larger environment, and small-scale changes of temperature and humidity within a tropical cyclone. Air quality Pollutants such as particulate matter and ozone-filled smog threaten both human health and the environment, including crops. Atmospheric chemists plan to run simulations on Yellowstone that will help us better understand the regional and global evolution and movement of pollutants, potentially leading to forecasts of local air quality several days in advance. Renewable energy sources To harness more wind and solar energy, utilities must better anticipate shifts in local wind and cloud conditions. Yellowstone will help that effort by enabling scientists to simulate very detailed wind flows over various types of terrain, as well as cloud development and change for several types of clouds that have differing effects on solar radiation. Earthquakes As scientists gain new insights into the geometry of tectonic plates and their dynamic interactions, they need more powerful supercomputers to simulate faults worldwide and better understand various types of earthquakes. Such knowledge can ultimately help lead to earlier warnings of potentially deadline climate events.   Accelerating science Eleven initial experiments scheduled for fall 2012 will help put Yellowstone through its paces. See Accelerated scientific discovery for the details. All photos ©UCAR, except as indicated.

NCAR selects IBM for key components of new supercomputing center

  News Release Multimedia Gallery Fact Sheet FAQ More NWSC News   BOULDER—The National Center for Atmospheric Research (NCAR) today announced that IBM will install critical components of a petascale supercomputing system at the new NCAR-Wyoming Supercomputing Center (NWSC). The company was selected following a competitive open procurement process. The building housing the NCAR-Wyoming Supercomputing Center in Cheyenne, Wyoming, was completed in the summer of 2011. (©UCAR. Photo by Carlye Calvin. This image is freely available for media & nonprofit use.*) The IBM components consist of a massive central resource for file and data storage, a high performance computational cluster, and a resource for visualizing the data. The new system, named Yellowstone, runs on an IBM iDataPlex and is expected to be delivered to the NWSC early next year. It will be the new facility’s inaugural system. Once installed, the system will go through a testing period before being made fully available for scientific research in the summer of 2012. Yellowstone is expected to deliver 1.6 petaflops performance, or nearly 30 times the capacity of the system currently in use at NCAR’s Mesa Laboratory in Boulder, known as bluefire. Petaflops refers to a machine’s ability to perform one quadrillion calculations, called floating point operations (FLOPS), per second. Scientists will use these advanced computing resources to understand complex processes in the atmosphere and throughout the Earth system, and to accelerate research into climate change, severe weather, geomagnetic storms, carbon sequestration, aviation safety, wildfires, and other critical geoscience topics. “Yellowstone will provide needed computing resources to greatly improve our understanding of Earth and produce significant benefits to society,” says Anke Kamrath, director of operations and services for NCAR’s Computational and Information Systems Laboratory (CISL). “We are very pleased to have such a high-performance system inaugurate the new supercomputing center.” “The vision for Yellowstone parallels the principles that have guided the design of the NWSC,” says NCAR director Roger Wakimoto. “In both instances, we have taken an approach that maximizes the science we can do and the benefit of that science to society.” The NWSC is the result of a partnership between NCAR; the University of Wyoming; the State of Wyoming; Cheyenne LEADS; the Wyoming Business Council; Cheyenne Light, Fuel and Power; and the University Corporation for Atmospheric Research. NCAR is sponsored by the National Science Foundation. Inside the NCAR-Wyoming Supercomputing Center, a 12,000-square-foot computer room stands ready to receive its first components, which will occupy about one-third of the space. The small, temporary "loadbanks" shown here have been used to test the room's readiness to handle electrical and heat loads and will be removed when the new IBM system arrives. (©UCAR. Photo by Carlye Calvin. This image is freely available for media use. For more information, see Media & nonprofit use.*) The facility, located in the North Range Business Park in Cheyenne, Wyoming, was designed with sustainability and flexibility in mind so it can be easily adapted to future technologies and changing requirements in scientific computing. Construction and commissioning was completed last month. "We join our partners at NCAR and in Cheyenne in great anticipation as the supercomputer project nears completion," says University of Wyoming president Tom Buchanan. "Our faculty are poised to take full advantage of all that the center will bring to our educational and research endeavors in Earth system sciences; atmospheric, hydrological and computational sciences; mathematics; and engineering. The outcomes will enhance science and technology throughout Wyoming and the nation.”  Details of the new system Yellowstone is an IBM iDataPlex supercomputer system, consisting of Intel Sandy Bridge EP processors and a Mellanox FDR InfiniBand full fat tree. It will have 149.2 terabytes of memory, 74,592 processor cores, and a peak computational rate of 1.6 petaflops. The central file and data storage resource will consist of file system servers and storage devices that are linked to the data analysis and visualization (DAV) resources and to the supercomputer systems. These centralized file systems will allow scientists to generate model output on the supercomputer and then analyze or visualize it on the DAV resource, without the usual bottleneck when moving such large quantities of data between separate systems. Yellowstone’s central file system will have nearly 17 petabytes of usable disk space, 12 times what is available to NCAR’s scientific research community today. The DAV resource is made up of two systems, one designed to facilitate large-scale data analysis, and the other for parallel processing and visualization activities. Taken together, these components will dramatically improve capabilities central to NCAR’s mission, such as climate modeling, forecasting, and preservation of critical research data. The NWSC will serve researchers across the United States and around the world who will interact with its systems remotely. “NCAR has a longstanding tradition for excellence in deploying resources to address unique challenges requiring high-performance computing capabilities,” says David Jursik, vice president of high-performance computing at IBM. “Yellowstone will substantially expand NCAR's ability to investigate climate change, severe weather events, and other important subjects. We are pleased to be selected as IBM continues to pursue high-performance computing for the purpose of helping scientists and leading decision makers address critical issues for a smarter planet.” CISL director Al Kellie emphasizes the importance of the integrated computing resource, explaining that what makes this system exceptional for geoscience research is the linking of a very large centralized file and data storage system to a high-performance computational cluster and visualization resource. “While we wanted to make sure we had adequate computing capacity, we knew that it would be of limited use if we didn’t ensure easy access to the data and appropriate resources for storing and analyzing it,” Kellie says. “In addition to high-performance machines, researchers need quick access to their data and a way to analyze it, to see what it means. This system addresses those needs elegantly.” Quick facts One petaflop is one quadrillion (1,000,000,000,000,000) floating point operations per second, which is approximately 143,000 calculations per second for every man, woman, and child on Earth. At 1.6 petaflops, Yellowstone will be capable of more than 221,000 calculations per second for every person on Earth. The Yellowstone system features 9.7 million times the computational rate, 3.4 million times the disk capacity, and 19 million times the central memory size of one of the world’s first supercomputers, the Cray 1-A, which supported NCAR’s computational science between 1977 and 1989.


Subscribe to NWSC