Zhigang Zhu

PRISM: Planning a Center for Perceptual Robotics, Intelligent Sensors & Machinesat the City College of New York: 

PRISM: Planning a Center for Perceptual Robotics, Intelligent Sensors & Machines at the City College of New York Zhigang Zhu Visual Computing Laboratory Department of Computer Science City College of the City University New York http://www-cs. ccny.cuny.edu/~zhu/ John (Jizhong) Xiao Robotics and Intelligent Systems Lab Department of Electrical Engineering City College of the City University New York Website: http://www-ee.ccny.cuny.edu/www/web/jxiao/jxiao.html

Outline: 

Outline Who Are We & Project Goal Problems and Promises Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan Summary

Outline: 

Outline Who Are We & Project Goal Problems and Promises Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan Summary

CUNY System w/ Vision and Robotics: 

CUNY System w/ Vision and Robotics CUNY City College Hunter Brooklyn Queens Graduate Center … SOE CS EE ME (PhD granting institution * ) PRISM Center Planned Center for Perceptual Robotics, Intelligent Sensors and Machines under NSF MII planning project Grant No. CNS-0424593 Flagship campus HSI / MI * Site of PhD programs in Engineering

Outline: 

Outline Who Are We & Project Goal Problems and Promises Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan Summary

Research Theme and Threads: 

Research Theme and Threads One Research Theme: RISE-NET Robotized Intelligent SEnsor NETworks in a 3D space w/ air, ground, wall-climbing robots Four Research Threads Perceptual Robots – navigate, network & plan For surveillance & inspection Intelligent Sensors – multimodal, multi-level For surveillance, inspection & education Smart Machines – HCI - M.I and H.I. For human interfaces in surveillance & advanced e-learning Reliable Communication Ad hoc networks, Multimedia networks More Detailed Technical Experiences Pervasive Computing Workshop

Thread 1 Perceptual Robotics: Mobile Sensor Network in 3Dfor surveillance and security: 

Thread 1 Perceptual Robotics: Mobile Sensor Network in 3D for surveillance and security Joint Work with George Wolberg, John Xiao @ CCNY Bob Haralick @ CUNY Graduate Center Ed Riseman, Al Hanson, Rod Grupen @ UMass Tom Huang @ UIUC Supported by NSF, DARPA,ARO, NYSIA, ACTI, etc PRISM Center

Goal: 

Goal Dream: transform the present 2-D world of mobile rovers into a new 3-D universe – air, ground, under-object, and wall-climbing move on ground, climb walls, walk on ceilings, transit between surfaces. Applications: Urban warfare applications: surveillance and reconnaissance, weapon delivery, guiding perimeter around a building, etc Security and counter-terrorist applications: intelligence gathering about a hostile situation within a building, etc. Inspection and maintenance applications: routine inspection of buildings, nuclear containment domes, and other hard-to-reach places, inspection of aircraft, sand blasting of ship hulls, etc. Other Civilian applications: assistance in firefighting, search and rescue operations, etc.

Some Real Examples: 

Some Real Examples City/Urban modeling (NSF/AFRL/NYSIA) - airborne video: far-view (1000 ft) Robot Navigation (DARPA, ARO)- ground video: medium range (~100ft) Under-vehicle inspection (ACTI) - car drives over camera: near-view (< 8 in) Facade inspection (ARO) – climbing robots: near-view (inches?)

Objectives: 

Objectives Rapidly create large FOV image mosaics that are geo-referenced, with 3D (stereo) viewing and with all dynamic targets identified As a light UAV flights over an area

Region-Based Stereo Matching: 

Region-Based Stereo Matching Left mosaic – Full Depth Map

Major Robots Acquired/Fabricated: 

Major Robots Acquired/Fabricated Ground Robots at CCNY Wall-Climbing Robots at CCNY

Robot Localization and View Planning: 

Robot Localization and View Planning Mutual awareness Heterogeneous sensors Active Sensing View Planning for tracking and recognition

Thread 2 Intelligent Sensors:Multimodal Human Signatures for Surveillance & Inspection: 

Thread 2 Intelligent Sensors: Multimodal Human Signatures for Surveillance & Inspection Joint Work with Weihong Li, George Wolberg (CCNY) Ning Xiang (RPI), Tom Huang (UIUC) Lt. Jonathan Lee, Mr. Robert Lee (AFRL) Supported by AFRL, CUNY RF Publications : OTCBVS05 PRISM Center

Multimodal Human Signatures: 

Multimodal Human Signatures Collaboration with Prof. Tom Huang at UIUC Prof. George Wolberg Goal Remote hearing by human and machines Challenges Noisy LDV signals Low-res, non-front faces Target detection??

Major Sensor Acquisition: 

Major Sensor Acquisition Polytec Laser Doppler Vibrometer (LDV) OFV-505 Sensor Head, OFV-5000 Controller Tip-Tilt Stage VIB-A-P05 with Telescope FLIR ThermoVision Camera ThermoVision® A40M Infrared Camera Canon Color/IR PTZ Camera Canon VC-C50i Low Light IR PTZ Camera Other video cameras Omnidirectional cameras, stereo head Camcorders, webcam, CMOS sensors

Thread 3 Smart Machines: Multimedia Integration in a Virtual Classroom: 

Thread 3 Smart Machines: Multimedia Integration in a Virtual Classroom Collaborators Rick Adrion, Ed Riseman, Al Hanson, Jim Kurose, (UMass) Parviz Kermani, IBM / PINY Student: Weihong Li, Hao Tang, Chad McKittrick Supported by CUNY CISDD PRISM Center

Multimedia Virtualized Classroom: 

Multimedia Virtualized Classroom Goal: automating e-learning in MVC Automatic capture (A, V, WB, BB) Intelligent media integration User-customized presentation Problem and Promise Large proposal with UMass declined by NSF CRI Keep Trying… NSDL Collaborative Research with UMass who has a successful VIP on campus MII -> CRI, more competitive

VC: Automatic MM capture: 

VC: Automatic MM capture

VC: Media Integration: 

VC: Media Integration Question Window, ToC, etc Toward an immersive VC interface

VC: Media Integration: 

VC: Media Integration Instructor Extraction…and integration video image shadow contour

Outline: 

Outline Who Are We & Project Goal Problems and Promises Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan Summary

PRISM Team (15+ people in CS, EE &ME): 

PRISM Team (15+ people in CS, EE &ME) Prof. Myung Lee (EE) Wireless sensor networks, ad hoc networks, Multimedia Networks Prof. Esther Levin (CS) Spoken language HCI, machine learning Prof. Ali Sadegh (ME) MEMS, micro device, biomechanics body/machine interaction Prof. George Wolberg (CS) Vision, graphics Prof. Jizhong (John) Xiao (EE)* Robotics, control, sensor network Prof. Zhigang Zhu (CS)* Vision, multimodal sensors, HCI * Co-PI of the MII project

Faculty Involvement: 

Faculty Involvement Problem MII/CRI limited funding for faculty support Heavy teaching load at CCNY Lack interest in student mentoring work Center allocates funding for small focused projects Call for proposals from mentoring faculty Funding for faculty and student research* Obtain course release time (commitment from CCNY) Seek External Funding (NSF, DoD, etc)

Current Funded Projects: 

Current Funded Projects Zhu, Wolberg, Haralick, Dynamic Pushbroom Stereo Mosaics for 3D and Moving Target Extraction. 02/15/05- 02/14/09, $267,437 (AFRL) Xiao, Zhu, Cooperative Wall-climbing Robots in 3-D Environment for Surveillance and Target Tracking, $375,237, 11/01/04-10/31/07 (ARO) Uyar, Xiao, Smart Reconfigurable Miniature Robot System Based on System-On-Programmable-Chip Technology”, $202,844, 09/04 –09/06 (NSF MRI) Zhu, Integration of Laser Vibrometry with Infrared Video for Multimedia Surveillance Displays, $94,076, 08/24/03 – 10/24/04 (AFRL)

Current Funded Projects: 

Current Funded Projects Wolberg, Log-Polar Transforms for Optical Image Processing and Target Recognition, $600,000, 3/1/03 -10/1/06 (ONR) Lee (Co-PI), CTA ( Collaborative Technology Alliance), Wireless ad hoc networks, routing, MAC, multicasting, reliable transport, $2.5 M, 07/01-05/06 (ARL) Lee (Director), Samung-CCNY Joint Laboratory on wireless sensor networks , 2003-07 $1.2M (Industry) Sadegh, Northrop Gruman, $10K, Crown Corp., $52K, ALCOA Inc., $14K, Con Edison $8.5K. (Industry)

Current Funded Projects: 

Current Funded Projects Zhu, Tracking Moving Objects via a PTZ or Moving Camera, $23,333, 08/01/04- 07/31/05 (NYSIA) Zhu, Wolberg, Integration of Laser Vibrometry, Infrared and Video for Multimodal Human Detection, $50,000, 02/19/04-02/18/05 (CUNY Research Equipment Grant) Xiao, Lee, Acquisition of equipment for multi-robot and mobile sensor networks research, $22,250, 7/05~7/06 (CUNY Research Equipment Grant)

Pending Proposals to NSF…: 

Pending Proposals to NSF… Zhu, Wolberg, Huang (UIUC), Multimodal Remote Hearing for Humans and Machines, , $731, 776.00, 01/0/1/06-12/31/08 (NSF IIS) Zhu, Levin, Adrion & Riseman (UMass), Transforming Digital Video Libraries for Intelligent Access and Navigation, $240,499.00 (NSF NSDL) Levin, 2 NSF IIS proposals Grossberg, NSF CAREER proposal … … NSF CRI proposal in progress MII planning award ->CRI project

Summary: Funding for Faulty Research: 

Summary: Funding for Faulty Research Overall > $5 M in the last 5 years Vision + Robotics > $1.6 M in the last 3 years New faculty involvement Active small-scale collaborations SOE at CCNY investment Physical infrastructure, new science center, release time, lecture series, very attractive startups => A Center for multidisciplinary, large-scale collaborations in research and education

Outline: 

Outline Who Are We & Project Goal Problems and Promises Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan Summary

Infrastructure: 

Infrastructure Existing Equipment As described with the research threads Wish List for CRI 2006 Airborne and Outdoor Mobile Platforms UAVs, outdoor robots Heterogeneous Sensor Packages Miniature and high-end cameras, acoustic GPS, INS , ladars, onboard processors Various Levels of Vision Processors Compact, high-end, low-cost, low-power Wireless Sensor Networks Practical for imaging sensors and robots

Outline: 

Outline Who Are We & Project Goal Problems and Promises Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan Summary

Education and Outreach: 

Education and Outreach CCNY is both MI and HSI Student retention New courses and academic programs Undergrad, MS and PhD Outreach to K-12 students Complete the pipeline

Education and Outreach (I): 

Education and Outreach (I) CCNY is MI and HSI 1/3 African Americans, 1/3 Hispanic-Americans Urban institution without dorms Long distance commuters Student faces financial hurdles Majority of students hold part/full-time jobs MAJOR challenge at CCNY Students retention Solution ? attractive research and education programs

Education and Outreach (II): 

Education and Outreach (II) Introduce new courses video computing, mobile sensor network Reshape capstone design courses Combine robotics, vision and mechatronics Create “Pipeline” to attract high school students to CISE disciplines Retain and timely graduate under-rep undergrads Support talented minority students for graduate ed. All team members are on the newly established CpE program

Education and Outreach (III): 

Education and Outreach (III) NYC-LSAMP Louis Stokes Alliance for Minority Participation New York Academy of Sciences (NYAS) Summer Research Training Program (SRTP) New York/New Jersey FIRST For Inspiration and Recognition of Science and Technology Mentor high-school kids participating FIRST robotics competitions Organize workshops CCNY Robotics Club (organize outreach activities) Climbing Robot Competition 2005 in UK Affinity Mentoring

Outline: 

Outline Who Are We & Project Goal Problems and Promises Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan Summary

Academic Outreach / Partnership: 

Academic Outreach / Partnership Advisory Committee of experts in robotics, computer vision and sensor net Lecture Series on computer vision, robotics and HCI Visits Institutions with successful PRISM programs MI/HSI institutions

Advisory Committee: 

Advisory Committee Roderic A. Grupen Director, Laboratory for Perceptual Robotics, UMass-Amherst Thomas. S. Huang Chair of HCII, Head of IFP, Beckman Institute, UIUC Allen R. Hanson and Edward Riseman Co-Directors, Computer Vision Laboratory, UMass-Amherst Richard Volyes Director, Collaborative Systems Laboratory, UMinn Ning Xi Director, The Laboratory of Robotics and Automation, MSU USC, UCSD, UCF, CU…

Lecture Series: 

Lecture Series Web page http://www-cs.ccny.cuny.edu/ ~zhu/CcvcL/html/ccny_lecture_series.html Objectives excite interests, foster collaboration & increase visibility Lectures given Huang at UIUC, Hanson/Riseman & Grupen at UMass, Shah at UCF Wittenburg at MERL, Kumar at Sarnoff ( CUNY GC events) Lectures in plan Ning at MSU, Trivedi at UCSD , Medioni/Nevatia at USC and more

Visits to other institutions: 

Visits to other institutions Visits Made USC IRIS and IMSC (vision, robotics and graphics labs) Presentation by Z. Zhu UCSD Robotics and Computer Vision Lab UMass Computer Vision Lab and Lab for Perceptual Robotics Presentations by Z. Zhu and J. Xiao In Plan MERL (August for given a talk) CMU Robotics Institute (Fall) UPenn GRPASP Lab (Fall) U. Texas at El Paso (Fall) MSU, Uminn…

Outcomes of Academic Outreach : 

Outcomes of Academic Outreach All the three objectives – pleasant surprise excite interests, foster collaboration & increase visibility Faculty and students involvement Recognition by outside world Multiple Joint Proposals UIUC, UMass, RPI USC, UT, etc in plan

Outline: 

Outline Who Are We & Project Goal Problems and Promises Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan Summary

Evaluation Plan: 

Evaluation Plan Research achievement metrics and Educational effectiveness metrics based on the corresponding NSF guidelines Possible metrics immediately tangible research results seeded by the funding (e.g., student participation, publications) improvement of minority students’ retention rates number of students entering graduate schools, etc Under development…

Outline: 

Outline Who Are We & Project Goal Problems and Promises Research Theme/Threads Team and Faculty Involvement (*) Infrastructure Education and Outreach (*) Academic Outreach / Partnership Evaluation Plan Summary

Summary: 

Summary Conclusions Bases internal examination and feedback from external committee members Recognition in Research and Education Programs confident in excelling our research and innovating the minority educational programs under the theme of the PRISM center: RISE-NET Hardware infrastructure is critical for both research and ed., and this is well recognized The most challenging issue to find mechanisms (funding!) to motivate talented researchers and educators to devote into PRISM center Goal - to make CCNY a center of excellence in cutting-edge research and a national urban model for minority education in the fields of robotics, computer vision and sensor network

Slide47: 

Thank you! http://www-cs. ccny.cuny.edu/~zhu/ http://www-ee.ccny.cuny.edu/www/web/jxiao/jxiao.html