mmsp 5

MULTIMEDIA SIGNAL PROCESSING ALGORITHMS – BASIC PROBLEMS: 

MULTIMEDIA SIGNAL PROCESSING ALGORITHMS – BASIC PROBLEMS

Slide2: 

FROM PREVIOUS LECTURES WE KNOW THAT MULTIMEDIA INFORMATION PROCESSING IS EXCELLENTLY DONE BY THE HUMAN INFORMATION PROCESSING SYSTEM

Slide3: 

OUR PROBLEM IS: Biological systems perform processing of audiovisual information using special ”hardware” (which could be called ’wetware’) and ’software’ that is algorithms. The question is: Can we make processing of audiovisual information using different hardware and software? Maybe algorithms could be similar?

Slide4: 

IN HUMAN VISUAL SYSTEM PROCESSING STARTS IMMEDIATELY IN THE RETINA AND THERE ARE COLOR PROCESSING AND BLACK AND WHITE LIGHT ACQUISITION AND PROCESSING SYSTEMS Let us take visual processing as example

Slide5: 

FROM COLOR AND BLACK & WHITE RECEPTORS SIGNALS GO TO INITIAL PROCESSING ELEMENTS IT IS IMPORTANT TO NOTICE THAT THE NUMBER OF COLOR PROCESSING ELEMENTS IS MUCH LOWER THAN BLACK AND WHITE OUTPUT LINKS

Slide6: 

WHAT THESE PROCESSING ELEMENTS DO? I MOST RECENT MEASUREMENTS OF RETINAL NEURAL CELLS SHOW THAT THEIR RECEPTIVE FIELDS ARE QUITE IRREGULAR IN THE FOLLOWING PAGES SOME INFORMATION ABOUT WHAT THESE CELLS ARE DOING IS GIVEN

Slide7: 

BAR OF LIGHT IS MOVED OVER PHOTORCEPTORS IN DIFFERENT DIRECTIONS OUTPUT OF THE PHOTORECPTORS IS SUMMED WITH POSITIVE SIGN (EXICITATION) OR NEGATIVE SIGN (INHIBITION)

Slide8: 

DEPENDING ON THE DIRECTION OF MOTION SIGNALS SUM UP STRONGLY OR NOT

Slide9: 

HERE THE MEASURED SIGNALS ARE SHOWN FOR CELLS WHICH REACT STRONGLY TO WHITE BAR ON BLACK BACKGROUND AND OPPOSITE (off)

Slide10: 

HERE WE SEE THE RESPONSE MEASURED IN TIME

Slide11: 

WE CAN SEE THAT INITIAL PROCESSING IN THE EYE INCLUDES DETECTION OF DIRECTIONAL CHANGES IN LIGHT INTENSITY THIS MIGHT BE DONE FOR DIFFERENT COLORS TOO

Slide12: 

WE CAN NOW ASK FOLLOWING QUESTIONS: WHY THE PROCESSING IS ORGANISED IN THIS WAY? FOR THE ANSWER WE CAN THINK THAT THE PROCESSING IS OPTIMISED IN SOME WAY. WHAT MIGHT BE OPTIMISATION CRITERIA? WHAT ARE THE GENERAL PRINCIPLES OF HUMAN/BIOLOGICAL INFORMATION PROCESSING?

Slide13: 

OVERLAPPING SQUARES OR NOT???

Slide14: 

WHY WE SEE HERE THREE SQUARES AND NOT CUT OUT SQUARES? NOTE THAT ONLY ONE SQUARE IS FULLY VISIBLE, OTHERS ARE COVERED, IN FACT THEY MAY NOT BE SQUARES THIS IS BECAUSE THE VISUAL SYSTEM PRODUCES INTERPRETATION WHICH IS MOST PLAUSIBLE (GENERIC) BUT IT MAY BE WRONG TOO, ALTHOUGH WE WOULD BE SURPRISED IT WOULD REALLY BE!!!

Slide15: 

THE INTERPRETATION PRODUCED IS FOR DETECTING MOST PROBABLE OBJECTS THE UPPER FIGURE IS DETECTED AS ARCH OVERLAID ON THE SAWTOOH THIS IS THE MOST PROBABLE INTERPRETATION THE BOTTOM FIGURE INTERPRETATION IS SURPRISING, BUT IT COULD ALSO BE PRODUCED IF THERE WILL BE MORE EVIDENCE

Slide16: 

VISUAL SYSTEM ASSUMES THAT LIGHT IS COMING FROM TOP LIGHT DIRECTION SAME PICTURE UPSIDE DOWN

Slide17: 

The statistics-based system works normally in almost perfect way. As we could see it fails sometimes when input signals are highly improbable and/or if most probable interpretation is not correct. This can be seen in visual illusions. We will look at them closer since recent statistical approach is explaining them. This provides for us a hint what kind of processing is done.

Slide18: 

WE CAN NOW ASK FOLLOWING QUESTIONS: WHY THE PROCESSING IS ORGANISED IN THIS WAY? FOR THE ANSWER WE CAN THINK THAT THE PROCESSING IS OPTIMISED IN SOME WAY. WHAT MIGHT BE OPTIMISATION CRITERIA? WHAT ARE THE GENERAL PRINCIPLES OF HUMAN/BIOLOGICAL INFORMATION PROCESSING?

Principles we can identify now:: 

Principles we can identify now: Statistical processing matched to the real world signal statistics – provides responses to most probable signals. This is very natural principle Minimization of information processed, as much information as possible is eliminated, minimum information needed to provide response is used. This principle allows to minimize energy and processing effort.

Slide20: 

A book which appeared in 2005 based on earlier research:

Slide21: 

The authors are visual psychologists, they consider vision as a system interpreting world from images projected onto the eye: Light from external source bounces of objects and is projected. This projection is not unique (e.g. objects of different size will have the same projection depending on their distance

Slide22: 

In visual illusions projection gives rise to improper interpretation Natural scene, illusion persists Stimuli changes, illusion persists,

Slide23: 

This picture gives strong of depth because of combination of many mutually consistent cues: perspective texture gradient Shading and shadow

Slide24: 

Geometry of natural scenes Geometrical illusions represent wrong interpretation od real world. To find out why researchers took pictures with depth map Laser range scanner for Measuring distance Real pictures with corresponding distances marked by colors

Slide25: 

If large number of such pictures is taken a database can be created in which real world objects are matched with distances and statistics is calculated. Example: subjective metrics Let’s think about lines of different lengths which are seen in real world. If all length would have the same probability there would be linear relation between the stimulation for every length. But if this is not the case, some length will be stimulated more often. This can lead to distortions in perception.

Slide26: 

Example: Line length illusion Variation of apparent length as function of orientation In experiments people report changing length depending on angle

Slide27: 

Why it is so? Let’s sample lines in pictures from database Grid of templates to overlay on picture with straight lines White – accepted lines, Black – rejected lines The points in the picture were compared with measured by laser range to see if they correspond to lines in real world. Total of 1.2x10^7 line segments were collected Probability distribution of of lines vs. length for different orientations Cumulative distribution (lines shorter than x) This shows how many lines at certain orientation corresponded to real lines of length shorter or equal to x

Slide28: 

Prediction of apparent length based on probability Take e.g lines of length 7 at orientation 20 deg, their cumulative probability is 0.15 which means that 15% lines is shorter than 7 pixels and 85% is longer. For all orientations we get this plot This is very smilar to the one measured in experiments with people!!!

Slide29: 

Why such biases exist? In nature lines do not appear often, horizontal lines are typically generated from horizontal flat surfaces Vertical lines are limited by gravity and by this rare and lines at 20-30deg even more Rare, and they are mostly projected from perspective

Slide30: 

Visual illusions: Angles All angles in this picture have 90 deg but when they are projected on the eye, projections may differ up to 60 deg Bias in angle estimation between two lines B,C,D) Angle illusions

Slide31: 

To explain this a database of angles is made, as before Extraction of angles Probability distributions for different Types of angles (bottom line) in natural scenes and scenes with human created objects We can see bias: angles close to 90 deg are less likely to occur

Slide32: 

Bias and illusions Angles close to 90 deg are more likely To come from planar surface, which is typically larger than surface from lines interesecting at smaller angles. Thus 90 deg angles are less likely Probability distribution of angles is not linear, cumulative probability is biased Thus predicted perceived angle is different from actual one, for 90 deg it is the same The magnitude of angle misperception (lines) vs. experimentally measured values

Slide33: 

Explanation of angle illusions Why vertical line is tilted? We take reference line at 60 deg (black) and check probability of occurence of physical sources of a second line oriented at different angles. Since the angle between the lines is 30 deg we look at the probability for 30 deg and then into cumulative probability (previous page) which gives value 0.184 which multiplied by 180 gives angle 33,2 deg in agreement with measurements

Slide34: 

Size illusion According to the previous explanations the reason for this illusion is: Probability distributions of the possible sources of the targets, given their different contexts, are different To check this hypothesis database was searched for circular objects and probabilities of the sources of targets in the context were calculated: Various size illusions of center and surrounding

Slide35: 

Experimental conditions a) The inner circle is surrounded by the 4 circles with changing diameters b) Probability of occurence of center circle with specific size for outer circles with different diameters. Dashed line shows probability for circle with 14 pixels diameter. (Bigger surrounding circles are much less likely to appear) c) Cumulative probability for 14 pixel circle d) Examples of scenes with large circles and small circles Why there are statistical differences? Circles originate from planar projections, larger circles are less likely. Why the presence of surrounding circles changes the occurence of target central circles differently? Larger circles arise from larger planes in the world, they are flat areas – then it is more probable that the central circle will be larger. In other words, the presence of larger surrounding circles increases the probability of of occurence of physical sources of larger central circles. In result probability Distribution of central circles is changing according to the size of surrounding circles.

Slide36: 

Changing the interval between center and surrounding circles Probabilities when the distance is changing Dashed line is for circle of size 14 Cumulative Probability for the 14 pixel circle

Slide37: 

Comparison of inner circle with single circle Probability distribution of singel circle vs. diameter Probability for single circle superimposed with probability of central circle surrounded by outer circle, dashed line is for 24 pixel circle, probability curve is for outer circle 32 pixel diameter, cumulative probability is much higher – there is bias When the outer circle is much bigger the cumulative probability is smaller The changing cumulative probability ratios and dependence on the central and outer circle sizes is well seen – and illusion depends on these parameters in exactly the same way

Slide38: 

Distance illusions When objects are close perceived distance is overestimated to physical one Objects which are close to each other are perceived as being at the same distance The distance to close objects is overestimated, the distance to far objects is underestimated Objects on the ground when they are about 7m distance appear closer and with increasing distance they appear more elevated

Slide39: 

According to the methodology probability distribution of distances is measured but there are several variables here: Probability of all distances from scanner Probability of the differences in distances between objects for three different horizontal angles Probability of horizontal distances different heights with respect to eye level

Slide40: 

Interpretation of these probabilities This curve for all distances has strong peak for distance of 3m . This is in agreement with experiments in which people seeing single objects hanging in completely dark scene report them as being in the distance of 2-4 m When the angular separation between the objects is small they tend to be seen at equal distance but this tendency decreases when the angle is increasing The dependency of probability of distance vs. eye level has peak at distance of 4 m. Thus for objects at distance less than 4 m will be overestimated and those at distance more than 4 m will be underestimated. This agrees with experiments

Slide41: 

The size illusion The size illusion does not depend nn particular type of endings It can be induced even without line and even (but less strongly) with dots Why this happens? Again, for explanation database is searched for such patterns and probabilities are calculated. Here we consider case when both gigures ar inline, on the left/right Templates used Templates overlaid on pictures

Slide42: 

Results of probability calculations Probability of lines with specific length and arrows pointing inwards and outwards Cumulative probabilities Superimposed cumulative probabilities showing differences Example of two lines of length 50 pixels. One can see that cumulative probability for outward arrows is higher which corresponds to the bigger length. Figures are in-line extending to the left or to the right

Slide43: 

Angle illusion The line is interrupted by vertical occluder It is then perceived as two segments shifted Why this happens? Again statistics of such patterns is calculated from the database od pictures

Slide44: 

Templates for calculation Shows the templates, for each red line there is one template corresponding to the shift The templates are matched in the pictures and statistics can be calculated Other templates can be used for different configurations of this illusion Definition of the difference in location of the line segments

Slide45: 

Probability distributions measured We can see peaks which are at nonzero shift So the most probable interpretation from this statistics is that that there is nonzero shift

Slide46: 

One can also study what is the effect of angle of the line and the width of the distractor Change of line orientation Change of width of the distractor As can be seen whent the are larger, The peak moves towards greater shifts which implies that the illusion will be stronger – and it is really so

Slide47: 

The processing of information in biological systems is statistical – it aims for producing MOST PROBABLE response to the signals coming from real world. This type of processing must be based on statistics of signals and models from real world. Result of processing is most plausible answer for ”normal conditions” and assumptions. This we have seen in the examples before and they are repeated next.

CONCLUSION: 

CONCLUSION Statistics based processing seems to be very strong in explaining visual illusions (many of them in the same way) The principle of statistical processing is powerful: The system collects information about most likely distribution of signals and provides most probable interpretations for them. This will work in most cases. Only when signals are very nontypical it will fail but this is rare.

BUT….: 

BUT…. We have to remember that biological systems are able to deal with extreme variations of signals and still extract right information from them. This will be illustrated now by the example of face recognition Faces can be distorted in many ways and still recognized. We can guess something about PRINCIPLES OF FACE PROCESSING

Slide50: 

We can recognize FAMILIAR faces from extremely low resolution pictures. How this is done? – We do not have clear idea – but it points to the minimization of processed information

Slide51: 

Contour information is not enough

Slide52: 

Face is processed somehow as a ”whole” and not as composed by parts. From the combined picture on the left we see new face, when we split it we recognize other faces

Slide53: 

Eyebrows are very important for the identification of faces

Slide54: 

Faces can be recognized despite extreme distortions

Slide55: 

Faces seem to be encoded in memory in exaggerated. caricature way: Average face (averaged from a number of persons Some typical face Face created by taking bid deviation from average Such faces are recognized even better than typical ones

Slide56: 

Newborn babies turn more attention to more face-like objects (upper row) than not face-like

Slide57: 

Faces and antifaces: If face within green circle is observed for some time the center one will not be correctly recognized but as one in the red circle (more distance from the center means more differences) This means that there is some kind of prototype encoding and tuning to it

Slide58: 

Impact of skin pigmentation Row 1: Faces differ only in shape Row 2: Faces differ only in skin pigmentation but not shape Row 3: Faces differ in shape and pigmentation We see that pigmentation has significant impact (row 2)

Slide59: 

Color helps: Left original Middle black and white Right color only, eyes can be located more precisely

Slide60: 

From negative picture it is impossible to identify faces

Slide61: 

Face recognition is strongly compensated for the direction of ilumination, pictures above are easily recognized as same person

Slide62: 

Resonse of neural cell of monkey in the face processing area of the brain. Response to something like face is much more stronger than for hand. (But remember that milions and milions of cells are processing at the same time Measurement from human brain: signal from face-like pictures is much stronger than from other objects

Slide63: 

The examples shown for faces indicate how sophisticated is information processing in biological systems. What is very amazing is getting correct results despite extreme distortions. For the most part, we do not know how this is done and we have difficulty in thinking how To develop algorithms which would have similar capabilities. This is the topic for studies in the future