740

UK eScience AHM 2006Visualization of a Hyperspectral Remote Sensing Model for Coastal Water Quality Applications : 

UK eScience AHM 2006 Visualization of a Hyperspectral Remote Sensing Model for Coastal Water Quality Applications Gen-Tao Chiang, Martin Dove, Stuart Ballard, Ian Frame National Institute for Environmental eScience NERC and University of Cambridge September 18, 2006

Contents: 

Contents Monte Carlo Hyperspectral Remote Sensing Model (MCHRSM) NIEeS Grid Environment GRID-MCHRSM Visualization tools Conclusion

Monte Carlo Hyperspectral Remote Sensing Model (MCHRSM): 

Developed by Florida Tech and NASA Kennedy Space Center used for Submerged Aquatic Vegetation (SAV) monitoring project. In water quality applications, remote sensing techniques have been used to assess water quality constituents. For instance, chlorophyll, temperature, suspended sediment…. etc. Monte Carlo Hyperspectral Remote Sensing Model (MCHRSM)

What is the purpose of MCHRSM?: 

What is the purpose of MCHRSM? Using MCHRSM to simulate a images for water quality algorithm development and testing.

What is the problem now?: 

What is the problem now? A Monte Carlo model needs to process on the order of one hundred thousand to one million photons or more for each pixel. The MCHRSM and associated image processing techniques need a large amount of computation time.

For example:: 

For example: 1.6 GHz CPU 1,000,000 photons We need 30 minutes to run per pixel Today, we want to simulate a 1000x1000 pixel image: 30 min X (1000 X 1000)=30,000,000 min =500,000 hours =20833 days=57 years 57 years X 3(Bands, RGB)=171 years

Hyperspectral Cube generated by Analytical Model with 60 bands (400nm to 700nm): 

Still not able to generate a MC based hyperspectral cube with limited computing resource Hyperspectral Cube generated by Analytical Model with 60 bands (400nm to 700nm)

How to solve this problem?: 

How to solve this problem? from Parallel Processing to Grid Computing

Grid Computing (eScience): 

Grid Computing (eScience) MCHRSM was written by Fortran-90 and MPI. However, due to the limitation of computing ability within single cluster, one can only generate three bands and a 1000*1000 pixels image in a reasonable time (about a week on 20 processors). We are moving to eScience environment for accessing more computing resource and better data management.

Methods: 

Methods Building an e-Infrastructure is the first step in order to achieve the scope of eScience. Then we run the MCHRSM on this testbed to test the whole procedure.

Modification MCHRSM for eScience Environment: 

Modification MCHRSM for eScience Environment The MCHRSM is using FORTRAN-90 and MPI. and has been developed in order to produce an image matrix. Each matrix containing the water surface reflectance value. Value of each pixel in this matrix is calculated by the Monte Carlo model. Each band using different coefficient values from each wavelengths. One can then choose any three bands to display an RGB image using visualization tools such as GIS software or Google Earth.

Parameters: 

Parameters specific absorption coefficient for pure water, the specific absorption coefficient for chlorophyll, the specific absorption coefficient for suspend sediment, the specific absorption coefficient for dissolved organic matter (DOM), the specific backscattering coefficient for pure water, the specific backscattering coefficient for chlorophyll, the specific backscattering coefficient for suspend sediment, the concentration of chlorophyll, the concentration of suspend sediment, the concentration of dissolved organic matter (DOM), the depth of the water column at that pixel location.

GRID-MCHRSM : 

GRID-MCHRSM A Grid enable MCHRSM can be produced by two different modifications of the code. The first modification approach is to take out the MPI code and rewrite the code or use a script to submit each pixel and input data to a Grid environment. However, in this case, one has to know which condor process is running which pixel, location, or band. This process would therefore be quite challenging. The second modification approach is simply keep the original MPI version but using Grid tools for accessing more resources. However, the challenge will be on system side.

First Approach: 

First Approach

Second Approach: 

Second Approach This approach presented the challenge of configuring the grid environment to support MPI. Unfortunately, most departments within the CamGrid VO are currently not supporting MPI, nor the file sharing system. Coordinating different departments to support MPI and overcome firewall issues between each site is the hardest part. To overcome these issues, a MPI testbed was built within NIEeS grid environment, to test the whole workflow.

GRID-MCHRSM: 

GRID-MCHRSM We have setup a small testbed to support MPI and testing the workflow using My_Condor_Submit (MCS), Condor-G, Globus and SRB.

GRID-MCHRSM: 

GRID-MCHRSM MCS Job Description File ****************************************************************************** Executable = syntheticp2 Notification = NEVER GlobusRSL = (stdout=sig-test.out)(job_type=mpi)(count=2) Globusscheduler = iguana.niees.group.cam.ac.uk/jobmanager-pbs #Transfer_input_files = one, two_three, four # Force overwriting when uploading / downloading files SForce = true SRBHome = /usr/local/srb/SRB3_4_0/utilities/bin Sdir = /NIEeS/home/gtniees.NIEeS-1/SIGtest/ Sget = * Sput = * queue ********************************************************************************

GRID-MCHSIM: 

GRID-MCHSIM Outputs will be put in Data Grid (SRB) [gtniees@iguana SIG_onelayer]$ Sls C-/NIEeS/home/gtniees.NIEeS-1/SIGtest/480 C-/NIEeS/home/gtniees.NIEeS-1/SIGtest/520 C-/NIEeS/home/gtniees.NIEeS-1/SIGtest/650 [gtniees@iguana SIG_onelayer]$ Scd 520 [gtniees@iguana SIG_onelayer]$ Sls /NIEeS/home/gtniees.NIEeS-1/SIGtest/520: matrixo-pure-water-520.txt matrixo-sav-520.txt

Visualization for Model Results: 

Visualization for Model Results Then, the outputs can be imported by ENVI and create a RGB image. Finally visualized using GoogleEarth or other tools.

KML : 

KML <?xml version="1.0" encoding="UTF-8"?> <kml xmlns="http://earth.google.com/kml/2.0"> <Folder> <name>NIEeS Google Earth Test</name> <GroundOverlay> <Icon><href>http://cete.niees.group.cam.ac.uk/googlemaptest/img/tmp1.jpg</href></Icon> <visibility>1</visibility> <name>tmp1.jpg</name> <color>ffffffff</color> <refreshInterval>60</refreshInterval> <LatLonBox> <north>27.874026246046025</north> <south>27.783348822698574</south> <east>-80.39687630293932</east> <west>-80.49796036022442</west> </LatLonBox> </GroundOverlay> <Placemark><name>NIEeS</name> <visibility>1</visibility> <description> Created by GT for SIG testing</description> </Placemark> </Folder> </kml>

Pure water: 

Pure water

Submerged Aquatic Vegetation (SAV): 

Submerged Aquatic Vegetation (SAV)

Suspended Sediments: 

Suspended Sediments

Conclusions: 

Conclusions We are submitting the GRID-MCHRSM to UK National Grid Services (NGS) to make sure it can be executed on a production GRID. Facing communication protocol issues. we are evaluating OpenGIS software such as GRASS for image enhancement and GoogleEarth for visualization. Next step, using FOX library. FOX is a set of Fortran libraries developed by eMinerals allowing one to deal with XML within Fortran program.

Acknowledgement: 

Acknowledgement Thanks for Richard Bruin helping on MCS, Andrew Walker, Mark Calleja on system issues, and all eMinerals, Materials Grid, and NIEeS people.

Thank You Very MuchPlease contact us if you think any potential applications can be benefit from eScience and willing to run a pilot demo project.http://www.niees.ac.uk: 

Thank You Very Much Please contact us if you think any potential applications can be benefit from eScience and willing to run a pilot demo project. http://www.niees.ac.uk NIEeS