SGIUG

Remote Imageryand Supercomputing:A Match Made “OnEarth”: 

Remote Imagery and Supercomputing: A Match Made “OnEarth” Lucian Plesea Jet Propulsion Laboratory

Introduction: 

Introduction OnEarth is the name of the public JPL Web Map Service (WMS) server, hosting a half arc-second, global earth image mosaic. Has 3600 times more pixels than previous global earth images, at about 15m per pixel This mosaic contains about three trillion pixels, more than five Terabytes of data. Creating such a large mosaic was only possible by using large parallel computers.

WMS Global Mosaic: 

WMS Global Mosaic

Slide4: 

Atacama Desert, Chile, South America

Slide5: 

Atacama Desert, South America, IR View

Slide6: 

Atacama Desert, South America, Default View (IR and Visual)

Slide7: 

Atacama Desert, South America, Visual View

Slide8: 

Coral Atolls, Maldives, Indian Ocean

Slide9: 

Belcher Islands, Hudson Bay, Canada, North America

Slide10: 

Brandberg Formation and Sand Storm, Namibia, Africa

Slide11: 

Bosphorus Strait, Istanbul, Eurasia

Slide12: 

Polders, Netherlands, Europe

Slide13: 

Etna Volcano, Italy, Europe

Slide14: 

Fuji-San, Japan, Asia

Slide15: 

Grand Canyon Fire of 2000, USA, North America

Slide16: 

Glacier, Canada, North America

Slide17: 

Fog, Iceland, Atlantic Ocean

Slide18: 

Shrimp Farms, Manila Bay, Philippines, Asia

Slide19: 

Namib Desert, Namibia, Africa

Slide20: 

Rock and Sand, Namib Desert, Namibia, Africa

Slide21: 

Algae and Mud, Queens Channel, North Australia

Slide22: 

Paris, France, Europe

Slide23: 

Everglades Park, Florida, USA, North America

Slide24: 

Zion Park, Utah, USA, North America

The Project: 

The Project NASA is leading an effort to increase the accessibility of remote imagery. The availability of a recent, high resolution, global coverage map of the earth was seen as an important component of this effort. The project started in earnest in Jan 2003, with an expected completion date of Dec 2003 Release One on-line since April 2004

WMS Access: Custom Images: 

WMS Access: Custom Images

Project Components: 

Project Components Dataset: GeoCover, orthorectified Landsat 7 Storage: RASCHAL, a 40TB storage cluster Data Format: Journaling Image File Format Application: Parallel Image Mosaic Engine Data Access: Network Image Protocol Computation Resource: IPG Mosaic: Release 1 Mosaic availability: WMS

Dataset: 

Dataset NASA purchased global orthorectified Landsat 7 as a part of the Scientific Data Purchase Phase II. Expected 8000 scenes, with a final delivery date of July 2003. Currently 10700 scenes have been received at JPL Each scene has 9 spectral bands, in UTM coordinates, 500-550 Megabytes.

Slide29: 

For this project, and a few others, a very large storage system was necessary. Raid Again Storage using Commodity Hardware And Linux (Raschal), is a 40TB NAS storage cluster, built in-house. RASCHAL became operational April 2003, and has been in continuous use since then. OnEarth uses about 15TBytes Storage

Storage: 

Storage

Data Format: 

Data Format A Journaling Image File Format is used extensively, for both the input scenes and the output mosaic. It is a tiled, multispectral and multi-resolution file format that supports lossless and lossy compression at the tile level. A level of indirection in data access, adding journaling features which ensures file consistency

Data Format: 

Data Format Input images have been converted to this format, using bz2 (block arithmetic) lossless compression. Lower resolution input images are pre-built, for lower resolution test runs. The output mosaic itself is stored using libz compression, less efficient but faster. Files larger than 1Tbyte have been generated during this project

Components: 

Components Dataset: GeoCover, orthorectified Landsat 7 Storage: RASCHAL, a 40TB storage cluster Data Format: Journaling Image File Format Application: Parallel image mosaic engine Data Access: Network Image Protocol Computation Resource: IPG Mosaic: Release 1 Mosaic access: WMS

Application: 

Application An image mosaic builder for very large geographic datasets, applying in a single pass the coordinate transformation, color correction and scene blending. Unique capabilities include UTM to geographical reprojection , blend mask driven data selection and feathering, and per band first order polynomial color correction. Implemented as a chain of custom SGI Image Library Operators.

Application: 

Application The input scenes are first analyzed, data location and the statistical distribution of each spectral band are determined. This data can saved for later use. The data saved in the analysis stage can be used to generate a global spectral matching solution. Blend masks are generated, derived from the location information.

Input Scenes: 

Input Scenes Each band stored in a separate file, as large as 17,000x17,000 Images are in UTM projection, in 60 different zones. Image swath has different orientations, depending on altitude

Blend Mask: 

Blend Mask

Application: 

Application Many algorithms for the spectral matching are possible, a few have been tried. The current approach uses the MODIS derived, “Blue Marble” 1 Km image brightness to guide the Landsat 7 panchromatic band brightness. The other 8 bands are using the coefficients from the panchromatic band, followed by a recursive filtering to reduce neighbor scene differences.

Grayscale Blue Marble: 

Grayscale Blue Marble

Slide40: 

Panchromatic Band

Application: 

Application The mosaic engine works on a small area of the output at a time, using only a few scenes. The matching correction is applied if available, then a geometrical transformation matches the scene projection and resolution to the output. The images are now in a common projection and can be combined, using a blend mask weighted sum.

Application: 

Application Since the mosaic engine only operates on a small area of output at a time, checkpoint and restart capabilities are built in. Use of the Journaling File Format allows for multiple processing cluster to generate output for the same file at the same time. The mosaic builder code is using a shared memory architecture, and uses the SGI Image Vision Library.

Data Access: 

Data Access Computation resources are located remote from the storage resources, direct access to the data is not possible. An image specific data access subsystem allows small regions of the input and output images to be transferred independently. The Network Image Protocol is used, separating the location and specific file format from the application.

Data Access: 

Data Access Data tiles are transferred in either raw or compressed format, under user control. A single plain text setup file provides access to all input files, in this case, more than 70,000. The Network Image Protocol was implemented as an Image Format Library module, no significant changes were required to the mosaic engine

Computation Resource: 

Computation Resource The mosaic engine is a shared memory application, it benefits from faster CPUs and a very large memory footprint. Chapman, a 1024 CPU Origin 3000, with 600MHz CPUs and 256MB per CPU was used. This machine is located at AMES, and connected to JPL via a 15MB/sec link, the NREN network component of the IPG

Computation Resource: 

Computation Resource Four groups of 32 CPUs were running simultaneously to produce the mosaic, using about 50000 CPU-hours total. This architecture provided a balance of data access and computational loads, achieving a CPU load between 30 and 60 percent. Peak data transfer rates of 12MB/sec were seen, with an average of 2.5MB/sec data read and 1.25MB/sec data write

Components: 

Components Dataset: GeoCover, orthorectified Landsat 7 Storage: RASCHAL, a 40TB storage cluster Data Format: Journaling Image File Format Application: Parallel image mosaic engine Data Access: Network Image Protocol Computation Resource: IPG Mosaic: Release 1 Mosaic access: WMS

WMS Global Mosaic: 

WMS Global Mosaic There are nine separate images, at three different resolutions. Coverage area is 180W to 180E and S85 to N85. 600 GB, panchromatic band, 0.5 arc-second (15m), 2,592,000x1,224,000, or 3 trillion pixel. 50 GB, thermal band, high and low gain, 2 arc-second per pixel (60m). 900GB, three visual and three near IR bands, stored at 1 arc-second (30m).

Access: WMS: 

Access: WMS Access to the mosaic is best done via the WMS server. A prototype client is available on http://OnEarth.jpl.nasa.gov The server is implemented as a CGI application, and uses the same technologies as the mosaic application. Server provides color selection, pan-sharpening, multiple projections. Image control using Styled Layer Descriptor.

To Be Continued: 

To Be Continued Pending: Web Site Refresh Packaging for delivery to interested parties Possible Further Development Improve and maintain the mosaic SRTM Elevation Web Terrain Service

Credits: 

Credits Dave Curkendall, shares the blame for the whole thing Kacie Shelton, the tape and data shepherd Jimi Patel, built most of RASCHAL Richard Schreyer, summer student, wrote the current WMS parser engine. George Percivall, manages the WMS Global Mosaic project for NASA

Sponsors: 

Sponsors WMS Global Mosaic Geospatial Interoperability Office, YO Image Access Technologies: ESTO-CT Computing Resources: AMES CNIS Program