Swimming Style Classification from Color Video Sequences: Swimming Style Classification from Color Video Sequences Speaker：Ming-Je Liu g9131@cs.nccu.edu.tw Visual Information Processing Laboratory, Department of CS, National Cheng-Chi University


Outline: Outline Introduction System Framework Color-based Segmentation Connected Component Detection Regression Analysis Motion Classification Demo and Experimental Results Conclusions


Objective: Objective To classify swimming motion into four styles, namely, Back-stroke (仰式) Breast-stroke (蛙式) Butterfly (蝶式) Free style (自由式) using video taken with an above-water camera looking down as the swimmer is approaching.


Motivation: Motivation Two distinct purposes for analyzing sports video: To organize and segment the raw footage into semantic units for easy indexing and retrieval. To perform qualitative and quantitative analysis of human motion to identify weak points and offer suggestions for improvement.


Key Assumptions: Key Assumptions Constraints for source video clips: The swimmer is sited in a FINA Olympic standard pool with distinct lane ropes. The video is taken with an above-water camera looking down as the swimmer is approaching. There is only one swimmer within the camera’s field of view.


System Framework: System Framework


Color-based Segmentation: Color-based Segmentation Original color image The hue component RGB  HSV conversion. Label the pixels with a hue value between 0.3 and 1.5 0 / 2π 0.3 1.5 Segmented regions


Connected Component Detection: Connected Component Detection Locate connected components. Morphological filtering to remove scattered blobs. Exclude components that: Have centroids close to the boundaries Have size smaller than 500 pixels Only the two largest blobs are retained.


Regression Analysis: Regression Analysis To estimate the relative position of the limbs efficiently. Other possible approaches for carrying out the calculation: Thinning algorithms to reduce the blob to a single line and approximate its slope Hough transform


Regression Analysis (Cont.): Regression Analysis (Cont.) Divide X-axis and Y-axis of the points of a blob into two array: X = [ p1, p2,…, pN ], Y = [ q1, q2,…, qN ] Treating X as independent variable and Y as dependent variable, we can fit these points with a line , where


Regression Analysis (Cont.): Regression Analysis (Cont.) The correlation coefficient: where R can be used to remove lane rope blob.


Regression Analysis (Cont.): Regression Analysis (Cont.) Detected regions and corresponding regression lines：


Motion Classification: Motion Classification Decision tree: the rules are mainly derived from the characteristics of the upper body movements. Example: in certain subsequence of the backstroke style, the arms will be held in the vertical direction. Slope a will be large and aspect ratio (AR) will be small. A score of 10 is assigned to the frame’s backstroke score if |a|>3 , AR <0.4.


Scoring system: Scoring system Score assignment is based on the measure of proximity to the prominent features of a specific style. Four types of backstroke:


Ambiguities: Ambiguities


Scoring tree: Scoring tree # of regions a1 ∙a2 a, AR 2 1 Breaststroke(3) Breaststroke(1) Butterfly(5) Butterfly(8) Free style(5) Free style(6) Free style(8) Backstroke(5) Backstroke(6) Backstroke(8) Backstroke(10) |a|0.9, AR1.5, AR2, AR3, AR<0.4 …


Scoring tree (Cont.): Scoring tree (Cont.) # of regions a1 ∙a2 |a1+a2| Butterfly (8) Negative slope found on Butterfly (10) Butterfly (5) Breaststroke(5) a, AR … … … 2 1 >0 <0 <0.2 right left |a1|<0.5 and |a2|<0.5


Linear scoring system : Linear scoring system A continuous scoring system is adopted to simplify the tree structure as well as increase the precision of the recognition results. A weighting factor is added to take into account the influence from the previous frames.


Linear scoring system (Cont.): Linear scoring system (Cont.)


Weighting factor: Weighting factor Complete cycle length：(NTSC Standard, 29.97frames/sec) Butterfly：28 frames Back-stroke：25 frames Breast-stroke：30 frames Freestyle：20 frames Since it only makes sense to relate the motion patterns within one cycle, k should be less than 20. In practice, k is usually set between 5 and 10 to account for dropped frames.


Weighting factor (cont.): Weighting factor (cont.) if (count[fly] > count [breast]) weight[breast] *= (1-weight[fly]); else if(count [back] > count [breast]) weight[breast] *= (1-weight[back]); else if(count [free] > count [breast]) weight[breast] *= (1-weight[free]); else if(count [breast] > count [fly]) weight[fly] *= (1-weight[breast]); else if(count [breast] > count [back]) weight[back] *= (1-weight[breast]); else if(count [breast] > count [free]) weight[free] *= (1-weight[breast]); Fine-tuning the weight：


Implementation: Implementation Borland® C++ builder 6.0 Analyze existing video clips or live DV input through IEEE 1394. Processing the input data at a rate over 20 frames per second using a computer with a Pentium 4® 1.7 GHz CPU. Software available for download at: http://vision.cs.nccu.edu.tw/SSC


Demo: Demo


Experimental Results: Experimental Results We have tested our system with a collection of about 50 video clips obtained from http://swim.ee We obtain robust classification results for all the video clips that satisfy the constraints discussed previously.


Experimental Results (Cont.): Experimental Results (Cont.)


Experimental Results (Cont.): Experimental Results (Cont.)


Experimental Results (Cont.): Experimental Results (Cont.)


Experimental Results (Cont.): Experimental Results (Cont.)


Experimental Results (Cont.): Experimental Results (Cont.)


Conclusions: Conclusions The features extracted from the upper body parts are sufficient for classifying swimming style. The proposed decision tree and scoring system prove to work well for video clips satisfying proper constraints.


Future Work: Future Work Detection of the turn movement to enable analysis of IM (individual medley). The ability to identify the beginning and end of a motion cycle. Combine above-water and under-water video to analyze whole body movements.


Backward problem: Backward problem The scoring system is no longer valid due to the design of swimming suit the limbs are sheltered from water splash Possible solution: find Textured Motion of water splash use water splash as another feature to build a new scoring system


Q & A: Q & A