Slide1: Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon University


Motivation: Motivation UPC Barcode QR Code Datamatrix


HCCB: HCCB Microsoft’s High Capacity Color Barcode


Application: Application Uniquely identifying commercial audiovisual works such as motion pictures, video games, broadcasts, digital video recordings and other media


Goal: Goal Locate and Segment the barcode from consumer images


Overview: Overview Design specifications of Microsoft’s HCCB Approach Localization Segmentation Progressive Strategy Results Conclusions


Microsoft’s HCCB: Microsoft’s HCCB 4 or 8 colors Triangles String of colors palette


Microsoft’s HCCB: Microsoft’s HCCB


Microsoft’s HCCB: Microsoft’s HCCB


Microsoft’s HCCB: Microsoft’s HCCB


Microsoft’s HCCB: Microsoft’s HCCB R rows S symbols per row S = (r+1)*R Aspect ratio: r


Approach: Approach Thresholding Orientation prediction Corner localization Row localization Symbol localization Color assignments Barcode localization Barcode segmentation point inside the barcode is known


Localization: Thresholding: Localization: Thresholding Identify thick white band and row separators Normalization Adaptive


Localization: Orientation: Localization: Orientation orientation orientation distance -90 90 0 summation


Localization: Corners: Localization: Corners Rough estimates whiteness mask non-texture mask combined mask


Localization: Corners: Localization: Corners Gradient based refinement


Localization: Corners: Localization: Corners Line based refinement


Segmentation: Rows: Segmentation: Rows Summation Flip?


Segmentation: Symbols: Segmentation: Symbols Local search Number of symbols per row q(S,E) = Sq(samples|S,E)


Segmentation: Colors: Segmentation: Colors Palette


Observations: Segmentation results given accurate localization Satisfactory Corner localization Unsatisfactory No one strategy works well on all images However (1) Errors of different strategies are complementary (2) Results are verifiable with decoder in the loop! Observations


Progressive strategy: Progressive strategy Hence – progressive strategy! Similar to ensemble of weak classifiers Or hypothesize-and-test Multiple strategies: Rough + gradient + line, or rough + line, or rough + gradient, or rough alone Different values of threshold during rough corner detection Total 12 Order of strategies


Results: Results Dataset of 500 images Performance metric: % barcodes successfully decoded Decoder model: Barcode successfully decoded if 80% of symbols are correctly identified


Results: Results Allows for explicit trade-off between accuracy and computational time


Results: Results


Results: Results


Results: Results


Results: Results


Results: Results


Results: Results


Results: Results


Results: Results


Results: Results


Results: Results


Conclusions: Conclusions 2D High Capacity Color Barcode (HCCB) Successful localization and segmentation of HCCB from consumer images Varying densities, aspect ratios, lighting, color balance, image quality, etc. Simple computer vision and image processing techniques Progressive strategy


Acknowledgements: Acknowledgements Microsoft Research Larry Zitnick Andy Wilson Zhengyou Zhang Carnegie Mellon University Advisor: Tsuhan Chen


Slide37: Thank you!