CB3M: Content-Based 3D Mosaics with Dynamic Pushbroom Stereo
Project Status: Active (Started: 2005)
Research Description
We propose a content-based three-dimensional (3D) mosaic representation for long video sequences of 3D and dynamic scenes captured by a camera on a mobile platform. The motion of the camera has a dominant direction of motion (as on an airplane or ground vehicle), but 6 degrees-of-freedom (DOF) motion is allowed. In the first step, a pair of generalized parallel-perspective (pushbroom) stereo mosaics is generated that captured both the 3D and dynamic aspects of the scene under the camera coverage. In the second step, a segmentation-based stereo matching algorithm is applied to extract parametric representation of the color, structure and motion of the dynamic and/or 3D objects in urban scenes where a lot of planar surfaces exist. Based on these results, the content-based 3D mosaic (CB3M) representation is created, which is a highly compressed visual representation for very long video sequences of dynamic 3D scenes. Our goal is to acquire geo-referenced 3D mosaic maps with motion tracking information for 3D (moving) targets using a light aerial vehicle flying over or a ground vehicle move across an unknown 3D environment.
Sponsors
NSF I-Corps (Award #1243737): Transforming 2D Video into an Interactive 3D Viewing Experience, Zhigang Zhu (PI), Edgardo Molina (EL), Peng Chang and John Blaho (IMs), Duration: July 1, 2012 – August 31, 2013
Northrop Grumman Corporation (NGC), Three-Dimensional Rendering of EO and IR Images. Zhigang Zhu (PI),$35,000, 07/01/2011-06/30/2013
AFOSR 2011 Summer Faculty Fellowship Program, Multimodal Layered Sensing for Modeling and Detection, Zhigang Zhu (PI), Edgardo Molina (PhD Student), T. Wang (PhD Student), 05/23/2011- 07/30/2011
DARPA SBIR (Award No. W31P4Q-09-C-0459) “A Semantic Analysis Based Automatic Object Classification and Activity Perception System for Large-view Urban Environments”, PI: Xiaokun Li, Genshe Chen/DCM; Co-PI:Z. Zhu, 07/03/2009 – 07/31/2010
AFRL RASER Program (Award No. FA8650-05-1-1853), Dynamic Pushbroom Stereo Mosaics for 3D and Moving Target Extraction – A New Geometric Approach, 03/07/2005 – 09/06/2009, PI – Z. Zhu, Co-PIs, G. Wolberg, R. T. Haralick
New York Software Industrial Association (NYSIA) Research Project, Tracking Moving Objects via a PTZ or Moving Camera, August 2004- July 2005, PI- Z. Zhu
PSC/CUNY 36 Research Award, Dynamic Pushbroom Stereo Mosaics for Video Surveillance over Large-Scale Scenes, 07/01/2005-06/30/2006, PI- Z. Zhu