Higher-level interpretation and recognition of images or videos, which includes pattern recognition, pattern learning and semantic interpretation as fundamental aspects. These aspects involve the detection, categorisation, identification, authentication of image or video patterns. For this purpose, image or video data are acquired and preprocessed. In the next step, distinctive features are extracted. Based on these features or representations derived from them, matching, clustering or classification is performed which may lead to one or several decisions, related confidence values (e.g. probabilities), classification or clustering labels. The aim is to find an explanation or to derive a specific meaning.

Pattern recognition or pattern learning in a specific, image or video-related context that includes:

Further details are given in the Definition statement of group G06V 10/00. Image or video recognition can be carried out by using electronic means (G06V 10/70) or by using optical means (G06V 10/88).

Typically, a pattern recognition system involves one or more of the following techniques:

Pattern recognition techniques in general are classified in group G06F 18/00.

Some techniques of image or video understanding performed in the preprocessing step — which start with a bitmap image as an input and derive a non-bitmap representation from it — can also be encountered in general image analysis. If these techniques do not involve one of the functions of image or video pattern authentication, identification, categorisation or clustering, classification should be made only in the appropriate subgroups of subclass G06T.

Some examples of these techniques are: general methods for image segmentation, e.g. obtaining contiguous image regions with similar pixels, for position and size determination of an object without establishing its identity, for calculating the motion of an image region corresponding to an object irrespective as to the identity of the object, for camera calibration, etc.

Techniques based on coding, decoding, compressing or decompressing digital video signals using video object coding are classified in group H04N 19/20.

Velocity or trajectory determination systems or sense-of-movement determination systems using radar, sonar or lidar are classified in groups G01S 13/58, G01S 15/58, G01S 17/58, respectively. Radar, sonar or lidar systems specially adapted for mapping or imaging are classified in groups G01S 13/89, G01S 15/89, G01S 17/89.

General purpose image data processing, in particular image watermarking, is classified in group G06T 1/00, while selective content distribution, such as generation or processing of protective or descriptive data associated with content involving watermarking is covered by group H04N 21/8358. General purpose image data acquisition and related pre-processing using digital cameras, and processing used to control digital cameras is classified in group H04N 5/00. Play-back, editing or synchronising of a music score, including interpretation therefor, as well as transmission of a music score between systems of musical instruments for play-back, editing or synchronising is classified in subclass G10H.

Pattern recognition or pattern learning techniques for image or video understanding involving feature extraction or matching, clustering or classification should be classified in groups G06V 10/40 or G06V 10/70 irrespective whether an application-related context provided by the groups G06V 20/00 - G06V 40/00 exists.

The functions performed at each step in the operation of an image or video recognition or understanding system.

These steps include:

Processing steps involved in a pattern recognition or understanding system

Classification of each of these steps may be made in groups as follows:

The process of acquiring still images or video sequences for the purpose of subsequently recognising patterns in the acquired images.

Image capturing arrangements which visually emphasise those features of the objects that are relevant to the pattern recognition process.

Optimising the image capturing conditions, such as correctly placing the object with respect to a camera, choosing the right moment for triggering the image sensor, or suitably setting the parameters of the image sensor.

Devices for image acquisition including sensors that generate a conventional two-dimensional image irrespective of its nature (e.g. grey level image, colour image, infrared image, etc.), a three-dimensional point cloud, a sequence of temporally-related images or a video.

Constructional details of the image acquisition arrangements are covered by a hierarchy of subgroups branching from group G06V 10/12:

Illustrative examples of subject matter classified in this place:

1.

2.

Illumination by casting infrared (IR) light onto a person to highlight regions of a hand, to assist in gesture recognition.

CCTV and image transmission systems are classified in group H04N 7/00.

Any kind of processing of acquired image or video data before the steps of feature extraction and recognition; devices configured to perform this processing.

Processing to prepare an image for feature extraction.

Processing to enhance image quality with the intent to emphasise structures in the image, which inform the automated recognition of objects or categories of objects.

Processing to attenuate or discard elements of the image, which are unlikely to be useful for the pattern recognition process.

Processing converts image to a standard format suitable for feature extraction and pattern recognition routines.

Specific aspects of pre-processing are covered by the subgroups of group G06V 10/20; they particularly relate to aspects such as:

Illustrative example of subject matter classified in this place:

Alignment of the image of a face by affine transformations to obtain a pose-invariant image.

Different image pre-processing in general is covered in groups as follows:

Guiding a pattern recognition process or device to a specific region of an image where the pattern recognition algorithm is to be applied, e.g. using fiducial markers.

The use of reference points in images, e.g. patterns having unique combinations of colours or other image properties, which make them useful for guiding a pattern recognition process.

Illustrative examples of subject matter classified in this place:

1.

A fiducial marker placed in the centre of an object is used for its object detection and recognition.

2.

3.

A pattern present on a marker gives additional information about the scene to be recognised.

Determination of position or orientation of image objects using fiducial markers is covered by group G06T 7/70.

Methods or arrangements for aligning or centring the image pattern so that it meets the requirements for successfully recognising it; for example, adjusting the camera's field of view such that a face, a person or another object of interest is located at the centre of the image.

Adjusting the field of view such that the object is entirely visible without any parts of the object extending beyond the boundaries of the image

Correcting the image alignment by changing from landscape to portrait mode.

Detecting or correcting images that were flipped upside-down or left-right.

Compensating for image skew.

Illustrative examples of subject matter classified in this place:

1A.

1B.

Compensation for the tilt angle of a face captured by a mobile phone by aligning the image.

Methods or arrangements for identifying regions in two-dimensional images, or volumes in three-dimensional point cloud data sets, which contain information relevant for recognition.

Identifying regions or volumes of interest in an image, point cloud or distance map which are likely to lead to successful object recognition.

These notes provide more information about the technical subject matter that is classified in this place:

A region or volume of interest [RoI or VoI] could include, for example, a human face (in case of a CCTV system), a vehicle or a pedestrian (in case of a camera-based traffic monitoring system), an obstacle on the road (in case of an advanced driver assistance system) or an item on a conveyor belt (in case of an industrial automation system).

The determination of a region or volume of interest is in essence a task of object detection, that is to say detecting the presence of a particular kind of object in images and localising the object(s).

It is the necessity of localising an object and, in particular, of describing the position and the spatial extent of the object (e.g. by outputting a bounding box around it) that distinguishes "object detection" algorithms from "object recognition" algorithms. This is because an "object detection" algorithm will merely assess whether a given visual object exists at a given image location. It may automatically generate a bounding box (e.g. around weeds in a field of vegetables) without solving the problem of "object classification" (e.g. analysing an image of a weed to determine its species and to output its botanical name).

Algorithms for detecting ROIs or VOIs in video sequences typically use frame differencing or more advanced optical flow methods for detecting moving objects.Algorithms that determine a region or volume of interest [ROI or VOI] may use visual cues to establish the location of a boundary box, e.g. by evaluating features such as colour distributions or local textures.

The determination of a region or volume of interest may be facilitated by using special illumination, such as casting light in a specific direction where an object is to be expected in autonomous driving, or by treating the images of specimens with special staining, as is the case in classification of objects in microscopic imagery.

More recently developed algorithms use neural networks [NN] which integrate object detection and recognition. An example is the region-based convolutional neural network [R-CNN] which uses segmentation algorithms for splitting the image into individual segments to find candidate ROIs, followed by inputting each ROI to a classifier for subsequent object recognition.

Other solutions, such as the you only look once [YOLO], region-proposal networks [RPN] or single shot detector [SSD] networks integrate the ROI detection into the actual object recognition step.

Illustrative example of subject matter classified in this place:

Using a mixed architecture based on region-proposal convolutional networks [R-CNN or RPN] to define a region of interest [ROI] and classifying it by another mixed convolutional neural network [CNN] using 2D and 3D information.

Determination of a ROI for character recognition is classified in group G06V 30/146.

Methods and arrangements for segmenting patterns in images or video frames, e.g. segmentation algorithms. Note: segmentation algorithms divide images or video frames into distinct regions, so that boundaries between neighbouring regions coincide with changes of some image properties.

Segmentation algorithms which operate directly on the image by considering the pixel values and their neighbourhood relationships, e.g. mathematical-morphology based algorithms, such as region growing, watershed methods and level-set methods.

Segmentation algorithms which generate a hierarchy of segmentations by starting with a coarse segmentation, which includes only few segments, and successively refine this coarse segmentation by splitting (possibly recursively) the coarse image segments into finer segments (coarse-to-fine approaches).

Graph-cut algorithms such as normalised cuts or min-cut which use graph-based clustering algorithms for image segmentation.

Region growing algorithms which start with few seed points and iteratively expand these into larger regions until some optimality criterion is fulfilled.

The use of classifiers for foreground-background separation. Note: classifiers calculate a score function which expresses a probability (or belief) that a given region of the image is a foreground object or part of the background. The image is then segmented based on these score values.

Deep learning models, in particular different encoder-decoder architectures based on convolutional neural networks [CNN's], applied to semantic image segmentation (a task which requires not only splitting the image into regions, but also consistently assigning labels to image object categories, e.g. "sky", "trees", "road").

Detection of occlusion. Note: Sometimes an object (e.g. a trunk of a tree) partly occludes another object, e.g. a dog behind the tree, which may cause the other object to be split into multiple disjoint segments; occlusion detection algorithms deal with such situations so as to join semantically linked segments into a single segment.

Other algorithms (e.g. some active contour models) which start from an initial image region, which is large enough to surely enclose an object in the image, and they iteratively shrink this region until its boundary is tightly aligned with the contour of the object.

These notes provide more information about the technical subject matter that is classified in this place:

Segmentation algorithms divide images or video frames into distinct regions, so that boundaries between neighbouring regions coincide with changes of some image properties.

Segmentation algorithms may determine regions of homogeneous texture, regions having characteristic colours, regions enclosing individual objects, etc.

Some segmentation algorithms are in essence clustering algorithms. They disregard the spatial arrangement of pixels in the image and compute clusters in a feature space (e.g. by running the k-means algorithm on all colour values in an image). They then group spatially-connected pixels belonging to the same cluster into a region (a "segment").

Illustrative examples of subject matter classified in this place:

1.

Colour segmentation of a skin region of a face using the clustering in a colour space.

2.

Example of a scene frequently encountered in autonomous driving and its semantic segmentation map with regions such as "road", "sky", "trees", etc.

Variational methods used for object recognition, such as active contour models [ACM, or "snakes"], active shape models [ASM] or active appearance models [AMM] are classified in group G06V 10/74.

Methods and arrangements for quantising the image with the effect that the number of possible pixel values does not exceed a predetermined maximum number.

Note: In the limit, this quantisation generates a binary two-tone (black-and-white) image, e.g. an image in which foreground objects appear white and any objects in the background appear black. Quantisation to other numbers of pixel values is also possible.

Quantisation algorithms which calculate a histogram of the grey value distribution and then use one or more thresholds to divide the grey values into different ranges, and then map grey values in the same range to the same target value.

These notes provide more information about the technical subject matter that is classified in this place:

Image quantisation refers to a technique in which the number of possible pixel values is set so that it does not exceed a predetermined maximum number. The source image may be an analogue image, which is quantised for being storable in digital form, or a digital image, which is quantised to a smaller bit depth.

Quantisation may be uniform or non-uniform. The subsequent encoding might adapt the number of bits to encode the representation, using more bits to represent those ranges of grey values which are considered to be particularly relevant for the subsequent image recognition step.

Colour or grey value quantisation can cause artefacts in the resulting image, such as apparent edges or quantisation boundaries which did not exist in the original image (e.g. colour banding). These artefacts can be mitigated by dithering techniques (e.g. by using the Floyd-Steinberg algorithm).

Quantisation may be performed globally (using the histogram of the whole image) or locally (using statistics of local image patches).

Illustrative examples of subject matter classified in this place:

1.

2.

Detection of faces in colour images by creating a single-channel image, e.g. a greyscale image, and subsequent binarisation by thresholding.

Image enhancement or restoration by the use of histograms is covered by group G06T 5/40.

Techniques for noise removal or filtering such as thresholding in the frequency domain (e.g. after a Fourier or wavelet transform), edge-preserving smoothing techniques such as anisotropic diffusion (also called Perona-Malik diffusion) or deep learning approaches to image denoising, e.g. using convolutional neural networks [CNN's].

Linear smoothing filters (e.g. for convolving the original image with a low-pass filter such as a Gaussian kernel matrix or applying a Wiener filter) and non-linear filtering such as median filtering or bilateral filtering (see also group G06V 10/36), when applied for the purpose of noise removal.

Noise estimation techniques based on a reference image, wherein the reference image may be:

Estimation of noise parameters based on different noise models, e.g. additive white Gaussian noise, speckle noise, etc.

Detection of blur or defocusing of the image pattern.

Illustrative examples of subject matter classified in this place:

1.

Face image denoising

2.

Face denoising using an autoencoder convolutional neural network architecture (above), followed by face recognition using a discriminator architecture (below).

3.

Processes and devices for bringing image or video data to a standard format, so that it may be compared with reference data (e.g. with images in an image database or gallery images serving as reference templates).

Normalisation or standardisation of the size of images, e.g. by cropping, by reducing the image size via downscaling or sub-sampling, or by enlarging images via up-scaling and interpolation.

These notes provide more information about the technical subject matter that is classified in this place:

Normalisation can involve adjustments to guarantee that all objects to be recognised have similar size or appearance (e.g. by rescaling facial images so that they are centred and that the area of the face covers a predetermined fraction of the image, or by only selecting frontal images).

Illustrative example of subject matter classified in this place:

Correction of the region detected for the face image by cropping around the face region.

Techniques for binary and grayscale morphological analysis of image patterns. These techniques include:

Illustrative example of subject matter classified in this place:

Extraction of the skeleton representation of a human body by applying morphological operations.

Image or video pre-processing techniques which examine a local neighbourhood around a pixel and assign a value to the pixel, which is a function of the values (e.g. colour values or luminance values) of the pixels in this local neighbourhood.

The application of local operators in the spatial domain (e.g. by convolving the image with a predefined kernel matrix) or in the frequency domain (e.g. by calculating the Fourier transform and performing a point-wise multiplication in the frequency domain).

These notes provide more information about the technical subject matter that is classified in this place:

1. Usually the local neighbourhood is defined as a rectangular region of pixels with the pixel of interest placed at its centre, e.g. a 3*3 pixels neighbourhood or a 5*5 pixels neighbourhood; neighbourhoods having other shapes are also possible, but they are less common.

2. Local operators include:

Use of low-pass filter matrices for noise removal – group G06V 10/30.

Use of median filters for noise removal – group G06V 10/30.

Use of the Sobel operator and the Marr Hildreth operator for edge detection – group G06V 10/44.

Illustrative example of subject matter classified in this place:

Analysis of local image patches of a face image using a local operator and encoding the representation for subsequent face recognition.

Methods and arrangements for extracting visual features which are subsequently input to an object recognition algorithm.

These notes provide more information about the technical subject matter that is classified in this place:

Formerly, the choice of suitable feature extraction algorithms was a crucial design choice in the art of pattern recognition algorithms. It had a strong influence of the overall performance. With the advent of deep learning, particularly in convolutional neural networks, the need for the hand-picked design of dedicated feature extraction algorithms has decreased to some extent, because the inner layers of the neural networks are trained to automatically find suitable features from the training data.

Subgroups of group G06V 10/40 focus on specific kinds of feature extraction techniques. These include:

Note: Global feature extraction techniques often involve domain transformations, such as frequency domain transformation. The global descriptors contain numerical data, such as vectors or matrices, but they can also represent the image or object in an abstract form as a string of symbols from a predetermined alphabet, which are integrated using a grammar (covered by group G06V 10/424).

Illustrative examples of subject matter classified in this place:

1.

2.

Quantifying local image properties, in particular the local gradient, using a local probe.

3.

Different types of features used for object recognition, e.g. contours, line segments, continuous lines.

Feature extraction techniques in which additional (invariant) information is calculated from certain image regions or patches or at certain points, which are visually more relevant in the process of comparison or matching.

Feature extraction techniques in which information from multiple local image patches can be combined into a joint descriptor by using an approach called "bag of features" (from its origin in text document matching), "bag of visual features" or "bag of visual words".

These notes provide more information about the technical subject matter that is classified in this place:

1. The image regions referred to in this place are called "salient regions", and the points are called "keypoints", "interest points" or "salient points". The information assigned to these regions or points is referred to as a local descriptor due to the inherent aspect of locality in the image analysis.

A local descriptor aims to be invariant to transformations of the depicted image object (e.g., invariant to affine transforms, object deformations or changes in image capturing conditions such as contrast or scene illumination, etc.).

A local descriptor may capture image characteristics across different scales for reliably detecting objects at different sizes, distances or resolutions. Typical descriptors of this kind include:

At a salient point, the pixels in its immediate neighbourhood have visual characteristics, which are different from those of the vast majority of the other pixels. The visual appearance of patches around a salient point is, therefore, somewhat unique; this uniqueness increases the chance of finding a similar patch in other images showing the same object.

Generally, salient points can be expected to be located at boundaries of objects and at other image regions having a strong contrast.

2. A "bag of visual words" is a histogram, which indicates the frequencies of patches with particular visual properties; these visual properties are expressed by a codebook, which is commonly obtained by clustering a collection of typical feature descriptors (e.g. SIFT features) in the feature space; each bin of the histogram corresponds to one specific cluster in the codebook.

The process of generating a bag of features typically involves:

A training phase comprising:

And an operating phase comprising:

Illustrative example of subject matter classified in this place:

Defining key-patches for different object classes from a training set, computing features from them and using a set of support vector machine [SVM] classifiers to detect those objects in new images.

The present group does not cover biologically inspired approaches of feature extraction based on modelling the receptive fields of visual neurons, such as Gabor filters, and convolutional neural networks [CNN].

The use of neural networks for image or video pattern recognition or understanding is classified in group G06V 10/82.

When a document presents details on a sampling technique and a clustering technique (bagging), then it should also be classified in group G06V 10/774.

Classical "bag of words" techniques remove most image localisation information (geometry).

When local features are matched directly from one image to another without involving a bagging technique (and thereby retaining geometric information), e.g. when triplets of features are matched using a geometric transformation with a RANSAC algorithm, then the document should also be classified in group G06V 10/75.

Techniques that map the image space into a parameter space using a transformation, such as the Hough transform.

These notes provide more information about the technical subject matter that is classified in this place:

The object of the transformations classified in this place is to allow better interpretation and increase the separability between the pattern classes. Each dimension of the parameter space may be linked to a specific feature parameter of an object, e.g. its distance from the origin of the image coordinate system and its orientation. The function which performs the mapping to the parameter space may be invertible, i.e. the original representation could be recovered from the representation in the parameter space.

In case of the Hough transform, the parameter space is partitioned into individual bins, which form a so-called accumulator array (a two-dimensional histogram). A voting process maps features in images to individual bins of the accumulator array to finally determine the most probably parameter configuration by retrieving the bin, which has received the maximum bin count.

The generalised Hough transform can be applied for recognising arbitrary shapes, e.g. analytic curves such as lines and circles, or binary or grey-value pattern templates.

Other examples are the generalised Radon transform, the Trace transform, etc.

Illustrative examples of subject matter classified in this place:

1.

A line in the plane is described by the parameters "d" and "ϕ" (distance to the origin and angle).

2A.

2B.

The two lines in the input image (fig. 2A) are mapped by the Hough transform in the parameter space (d,ϕ), and the representation leads to two distinct corresponding bright spots (fig. 2B).

3.

Detection of the visible edges of a cube as points in the Hough parameter space.

Global feature extraction for image or video recognition or understanding is classified in group G06V 10/42.

Fourier-transform based representations, scale-space representations or wavelet-based representations have a different aim than improving the discriminability in the representation space. The Fourier transform is usually chosen for its geometric invariance properties in the Fourier space (e.g. translation invariance), while the scale-space and wavelet-based representations aim at capturing the variability of the pattern at multiple representation scales. For this reason, the latter two representations are classified as global feature extraction (group G06V 10/42) and, respectively, local feature extraction by scale-space analysis (group G06V 10/52).

Feature extraction techniques that perform operations within image blocks or by using histograms.

Summation of image intensity values and projection along an axis, e.g. by binning the values into a histogram, to arrive at a more compact feature representation.

These notes provide more information about the technical subject matter that is classified in this place:

The processing classified in this group might involve:

Block-based arithmetic or logical operations (including non-linear operators such as "max", "min", etc.);

Histograms of various measurements computed on a block-basis, e.g. histogram of oriented gradients [HOG];

Quantification of local geometric arrangements of features by block-based analysis, e.g. local binary patterns [LBP].

The blocks need not necessarily be arranged in a form of a grid. They can overlap or can be arranged in different geometrical patterns.

Frequently used local feature descriptors which are classified in this group include:

- Histogram of oriented gradients [HOG];

- Edge oriented histogram [EOH];

- Local binary pattern [LBP] and its refinements:

- Shape context;

- Gradient location and orientation histogram [GLOH];

- Local energy-based shape histogram [LESH];

- Oriented histogram of flows [OHF];

- Binary robust independent elementary features [BRIEF];

- Spin image.

Illustrative examples of subject matter classified in this place:

1.

The local oriented histograms of the gradients or HOG descriptor.

2.

The "shape context", a representation which performs binning of the contours of the shape in a circular-like pattern.

Scale-space representations which allow analysis of the image or video at multiple scales.

These notes provide more information about the technical subject matter that is classified in this place:

One primary goal of scale space methods is to achieve scale-invariance, i.e. being able to detect and recognise objects regardless of their size in the image. The scale is usually selected by convolving the image with a parametric "size function", also called a kernel. After the convolution, which typically blurs the fine-scale structures to a certain degree and which is often followed by a suitable sub-sampling of the blurred image, the actual feature extraction can take place at the selected scale.

A very common example of a kernel is the Gaussian kernel:

Given the input image f, the scale-space representation is obtained by convolving it with the Gaussian kernel: where t is the scale of analysis.

Scale space approaches can also use Gaussian derivatives, Laplacians of Gaussians, difference of Gaussians (DoG's), Gabor functions, wavelets (in continuous or discrete form, e.g. Haar, Daubechies).

Other alternatives to constructing a scale space which do not use a kernel exist, for instance, by applying to the image a diffusion equation starting with the initial condition . In more general terms, these techniques analyse the differential intrinsic structure of the image in order to construct scale-space representations.

Techniques based on morphological scale-space construct representations at different scales using mathematical morphology methods, e.g. erosion, dilation, opening, closing.

Some other techniques construct multi-scale temporal representations based on the analysis of optical flow for feature extraction in video.

Multi-resolution methods implicitly provide representations at multiple scales; such methods are also classified in the present group in as far as they concern image or video feature extraction.

Illustrative examples of subject matter classified in this place:

1.

2.

Wavelets applied at different scales for the extraction of facial features.

Texture feature extraction for image or video recognition or understanding, either by identifying the boundaries of texture regions, or by analysing the content of the regions themselves.

These notes provide more information about the technical subject matter that is classified in this place:

Examples of algorithms used for feature extraction include:

Illustrative example of subject matter classified in this place:

Texture feature extraction allows identification of an animal (zebra) in natural images.

Colour feature extraction for image or video recognition or understanding.

Colour feature extraction based on colour invariance.

Colour feature extraction based on colour descriptors.

These notes provide more information about the technical subject matter that is classified in this place:

1. Colour invariance or, conversely, compensation of colour variations, is important for increasing the robustness in image matching or object recognition. Colour variations are often caused by changing lighting conditions (e.g. the colour of an object typically looks different under ambient light or when the object is being illuminated by an incandescent light bulb). They can also be caused by other factors (e.g. sun-tanned skin has a different colour than pale skin).

2. Colour descriptors associate colour information with various image structures such as points, contours or blobs/regions. Colour descriptors (e.g. colour histograms, average colour values etc.) are frequently used in image recognition or image understanding. Typical applications include feature detection based on a model of skin colour, traffic sign detection based on colour information, colour image object detection or video analysis for finding objects with a special colour (e.g. nudity detection).

Illustrative examples of subject matter classified in this place:

1.

Colour histograms used to detect vegetation in natural scenes.

2.

Detection of a person based on his/her skin colour.

3.

Discrimination between skin image regions and nail image regions by clustering in a three-dimensional colour space.

Techniques for feature extraction in hyperspectral image data.

These notes provide more information about the technical subject matter that is classified in this place:

The goal of feature extraction in hyperspectral imaging is to obtain a representation of the relevant features captured by the spectral content of a scene, with the purpose of finding relevant objects and identifying materials. The data can be visualised as a 3D cube, also called a hyperspectral cube, where 2D images corresponding to different spectral wavelengths are superposed. Typical examples of applications are in astronomy, microscopy and satellite image analysis.

Depending on the number of spectral bands, one often distinguishes between multispectral imaging (e.g. 3 to 15 bands) and hyperspectral imaging (often several hundred spectral bands). Group G06V 10/58 encompasses both alternatives.

Illustrative example of subject matter classified in this place:

Example of a 3D representation containing hyperspectral features or representations derived from them.

Feature extraction in the visible spectrum using colour representations is not regarded as pertaining to this group and it is provided under colour feature extraction – group G06V 10/56.

Techniques in which a model of illumination or reflectance of the image object is relevant for performing feature extraction or for object detection/recognition.

These notes provide more information about the technical subject matter that is classified in this place:

Information relating to the object in terms of its surface or geometry and other information relating to the scene (e.g. camera and illumination sources) may be used to compensate for, or to eliminate, the effect of changes in illumination.

When involved in the process of image or video recognition or understanding, the techniques covered include:

Illustrative examples of subject matter classified in this place:

1.

Modelling the reflection of human skin by considering its light scattering properties.

2.

Light source direction determination by modelling the albedo and the shape of an object.

3.

Person identification in different illumination (lighting) conditions by grouping the images pertaining to a certain illumination condition.

Techniques involving time-related feature extraction and pattern tracking for image or video recognition or understanding. Such techniques include:

These notes provide more information about the technical subject matter that is classified in this place:

1. Tracking may be implemented using a single camera or a system with multiple cameras, with possibly overlapping field of views [FOV].

2. In time-related feature extraction and pattern tracking, the features extracted from the video can be low-level (e.g. pixel colours, gradient, motion cues), mid-level (e.g. edges, corners, interest points, regions, etc.) or high-level (e.g. geometrical arrangements of parts of an object). The tracking often involves the foreground-background segmentation or background modelling in order to focus only on the objects of interest and reduce the overall complexity. Target representations are models of the objects of interest which rely on visual cues such as shape, texture, colour. There are rigid models (e.g. regions or volumes of interest), articulated models (e.g. kinematic chains) or deformable models (e.g. fluid models, point-distributions, appearance models).

An inherent problem during tracking is that of localisation, which is usually solved:

Models employed during tracking include graphical models (e.g. Markov models), graph-matching based tracking, camera-link model [CLM] or statistical models such as maximum a-posteriori estimation (MAP).

Problems frequently occurring are that of context modelling (e.g. changes in background, clutter, duration of the tracking events), or in the case of a multiple camera system, that of re-identification, i.e. detection of the same object in the field of view of these cameras.

Neural networks have been more recently applied to the problem of tracking, examples of architectures include: generic object tracking using regression networks [GOTURN], multi-domain network [MDNet], long short-term memory [LSTM] networks, recurrent you only look once [ROLO] networks.

Illustrative example of subject matter classified in this place:

Tracking, person re-identification in a multiple camera system.

Methods and arrangements for pattern recognition or machine learning in image or video data.

These notes provide more information about the technical subject matter that is classified in this place:

Pattern recognition algorithms try to identify or discover regularities in data (such as a collection of representative features derived from images) through the use of computer algorithms. These regularities are used to take actions such as classifying the data into different categories. Modern approaches include the use of techniques from machine learning for this purpose.

Pattern recognition and machine learning algorithms can operate in a supervised fashion, an unsupervised fashion or in hybrid forms (e.g. semi-supervised). Supervised methods require not only exemplary feature patterns for training the model, but also a-priori knowledge in the form of associated class labels that indicate a respective category or class. Using labelled inputs and outputs, the accuracy can be measured and the method can adapt/learn over time. In contrast, unsupervised learning may discover hidden patterns in data without the need for human intervention, with the goal of, e.g. clustering unlabelled data sets.

Specific aspects of the pattern recognition or machine learning in the recognition or understanding of images or video are classified in subgroups as follows:

Techniques that handle data quality issues such as data accuracy (obtaining the correct data entries), data completeness and data consistency (data written to a database must be valid according to all defined rules) in the context of image or video recognition or understanding.

Examples of techniques classified here include:

Illustrative examples of subject matter classified in this place:

1.

2.

Selection of vectors in a multi-dimensional space by considering the median of their subsets and discarding those above a certain distance range from the median.

Matching, which involves comparison of pixels values, combinations thereof or features derived from them, in which one entity is considered as a template pattern and the other is the input pattern (template matching). The matching process may involve shifting, deforming or transforming patterns to accommodate distortions or positional errors.

Histogram-based matching, wherein a histogram can be regarded as a quantised representation of the grey-level probability distribution function of pixels into intervals, called bins. Other statistical measures may be used for matching include probabilities, confidence intervals, etc.

Variational techniques such as active contour models [ACM, or "snakes"], active shapes models [ASM] or active appearance models [AAM] in which a contour or a shape of the object is obtained by iterative matching.

These notes provide more information about the technical subject matter that is classified in this place:

Aside pixels, other types of entities that may be involved in matching processes include lines, edges, object contours, object shapes, corners, key-points and statistical measures computed in a defined image neighbourhood.

The matching may be performed in a different representation space than the image space, e.g. using an eigenspace representation of the image object, using shape manifolds, using a Hough transform, using a Fourier transform etc., which implies applying a transformation from the image to this representation space prior to matching. The transformation is usually chosen due to the invariant properties sought by the matching process (e.g. Fourier transformation offers invariance to translation of the pattern in the image).

The proximity measures used during matching may include classical distances, such as Euclidian distances, or more involved distances, divergences or other measures between probability distribution functions or other statistical representations (e.g. mean, standard deviation, moments, kurtosis, Chi-square distance), for instance:

Group G06V 10/75 covers more detailed aspects of the matching process, such as:

Illustrative examples of subject matter classified in this place:

1.

2.

Eye detection by matching a circle/ellipse to the iris using a 2D projection onto a 3D representation of the eye.

3.

Fitting an active appearance model [AAM] to the face using key points detected for prominent facial features.

Techniques of grouping patterns together in order to reveal a certain structure or a meaning in images or video.

These notes provide more information about the technical subject matter that is classified in this place:

The object of techniques classified here is to identify groups of similar entities and to assign entities to a group (cluster) according to a measure of their similarity.

Separability is determined by measuring the similarity or dissimilarity. Such techniques are usually performed in a high-dimensional feature space constructed by extracting features from the image or video, but can also be performed in the original domain, e.g. in the image domain in the case of image segmentation.

Regarding the grouping of patterns, any pattern may belong exclusively to a single cluster (hard clustering) or it may belong simultaneously to more than one cluster up to a certain degree (fuzzy clustering) according to a similarity (or proximity) measure. In addition, depending on the clustering method used, proximity may be defined (a) between vectors, (b) between a vector and a set of vectors (or a cluster), and (c) between sets of vectors (or different clusters).

Examples of proximity measures are: dissimilarity measures (based on l₁, l_2, and l_∞ norms), similarity measures (inner product, cosine, Pearson's correlation coefficient, Tanimoto distance, etc.).

Clustering algorithms include:

a) clustering based on statistical measures (which mainly employ numerical data) which adopt a cost function J related to possible groupings which is subject to a global or local optimisation criterion, and return a clustering that optimises J. Examples of such algorithms are:

Hierarchical clustering is a popular technique in the class of graph-based clustering, with its agglomerative or divisive variants. Various criteria can be used for determining the groupings, such as those based on matrix theory involving dissimilarity matrices.

Algorithms included in this scheme are:

Illustrative examples of subject matter classified in this place:

1.

2.

Clustering face images to detect affinity between persons using a graph-based clustering algorithm.

Classification of images or videos to identify the category or set of categories (classes) to which a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known.

Novelty detection (e.g. classification of "unseen" observations), anomaly detection or outlier detection.

These notes provide more information about the technical subject matter that is classified in this place:

Individual observations may be analysed into a set of quantifiable properties, known as explanatory variables or features. These properties may be categorical, ordinal, integer-valued, real-valued, etc. Other classifiers perform a class assignment by comparing current observations to previous observations by means of a similarity or distance function.

A classifier can be parametric or non-parametric depending on the type of model adopted for the observations.

Classification algorithms include those:

The decision surface of the classifier may be a linear classifier or a non-linear classifier. Linear classifiers model the boundaries between different classes in the feature space as hyperplanes. Non-linear classifiers use, e.g. quadratic, polynomial or hyperbolic functions instead.

Illustrative examples of subject matter classified in this place:

1.

2.

A linear support vector machine classifier which attempts to define a linear boundary between two classes (205, 210) of feature vectors originating from images containing "persons" and "non-persons", such as to separate them into two different classes.

3.

Decision tree classifying objects in the image data using an efficient hardware implementation with FIFO buffers.

Techniques for image or video recognition or understanding using regression.

These notes provide more information about the technical subject matter that is classified in this place:

The term "regression" refers to statistical techniques for estimating the relationships between a dependent variable (often called the "outcome" or "response" variable) and one or more independent variables (often called "predictors", "covariates" or "explanatory variables"), where the variables model the underlying image or video data.

Common forms of regression are:

Illustrative example of subject matter classified in this place:

Example of adaptive regression analysis for classification.

Techniques which deal with the problem of reducing the dimensionality of a representation of features in high-dimensional feature spaces.

These notes provide more information about the technical subject matter that is classified in this place:

The problem of reducing the dimensionality of a representation of features in high-dimensional feature spaces is sometimes referred to as "the curse of dimensionality". Generally, having to consider too many features increases the requirements regarding processing power and memory capacity; moreover, available data samples may be too sparsely distributed in a high-dimensional feature space for reliably recognising patterns and the number of training samples, which are necessary to obtain a good estimate of the actual data distribution, increases exponentially. The distances between randomly chosen pairs of training samples can be expected to exhibit little differences, causing a nearest neighbour search to become unreliable the more dimensions the feature space has.

Different types of analysis can be considered:

Another way of dealing with the problem is to integrate or reduce data by deriving representatives through clustering.

A further concept covered by this group is the blind source separation [BSS] which involves estimating individual source components from mixtures of multiple sources, e.g. blended images that are obtained by superimposing one image onto another, or an image that is deteriorated by a noise process.

The reduction of the representation by principal component analysis [PCA] has been extensively applied in various application-related contexts, one example being face recognition ("Eigenfaces").

Various subgroups cover further aspects relating to processing features in high-dimensional feature spaces.

In particular, group G06V 10/771 covers techniques relating to feature selection, e.g. selecting those features which are the most representative from a multi-dimensional feature space. Well-known ways to carry out feature selection are:

Group G06V 10/772 covers techniques for determining representative reference patterns, e.g. by averaging or distorting patterns, or for generating dictionaries, i.e. sets of templates which are usually organised efficiently for different purposes, such as matching.

Group G06V 10/776 covers techniques for validation and performance evaluation. They usually involve considerations of partitioning the available data into a training set to be used for training a classification model, and a validation set used to assess the validity of the classification and to evaluate its performance.

Group G06V 10/778 covers techniques for active pattern learning, e.g. online learning.

Group G06V 10/80 covers techniques for fusion, i.e. combining data from various sources at the sensor level, pre-processing, feature extraction or classification level, mainly to improve the performance of a pattern recognition system for images or video.

Techniques for validation and performance evaluation of algorithms for image or video recognition or understanding.

These notes provide more information about the technical subject matter that is classified in this place:

Validation and performance evaluation of algorithms for image or video recognition or understanding normally involve:

Common classification metrics to evaluate the models are true positive rate [TPR] or sensitivity, false positives rate [FPR] or fall-out, true negatives rate [TNR] or specificity, false negative rate [FNR] or miss rate, receiver operating characteristic [ROC] curves (TP rate divided by the FP rate), z-score, accuracy, precision (or positive predictive value), recall, negative predictive value, intersection over union [IoU], the Jaccard index (also referred to as Tanimoto index), etc. Other metrics are also possible, for instance regression metrics, explained variance, validation curves, detection error trade-off etc. In case of decision-tree learning, the compactness of a cluster, the purity of a cluster in terms of class labels, the minimum distance of samples from the class boundary, a calculated likelihood score, etc.

In order to get more stable results and use all valuable data for training, a data set can be repeatedly split into several training and validation data sets. This strategy is known as cross-validation.

The performance can be measured automatically, e.g. by a stochastic process such as when using bootstrapping, or by a human operator in the case of relevance feedback.

Illustrative examples of subject matter classified in this place:

1.

2.

Example of an iterative "loss function" calculation for four different recognition models trained with different subsets of images, which is indicative of the performance of the classification of each model.

Techniques for active pattern learning for image or video recognition or understanding which dynamically adapt a learning algorithm (e.g. a neural network), either by interactively querying a supervisor (user) or from some other information source, like a teacher module, to classify or learn new data. Examples of techniques classified here include:

Illustrative example of subject matter classified in this place:

Active learning of weights and biases at different stages of a convolutional neural network for image classification.

Combining the information from several sources in order to form a unified representation for image or video recognition or understanding.

These notes provide more information about the technical subject matter that is classified in this place:

A simple fusion process combines raw data from several sensors or several sensor modalities (e.g. fusing spatial and temporal data). Besides fusing the raw sensor data, it is also possible to first process the sensor data to extract features and then combine the extracted features into a joint feature vector. Alternatively, it is possible to fuse classification results, e.g. inputting the features from different sensor modalities to separate classifiers, receiving respective classification scores from each classifier, and combining the individual scores into a final classification result.

Examples are probabilistic fusion, statistic fusion, fuzzy reasoning fusion, fusion based on evidence and belief theory, e.g. Dempster-Shafer, fusion by voting.

Fusion can also be applied at different stages of a recognition system for different purposes, e.g. for dimensionality reduction, computing robustness, improving precision and certainty in the classification decisions, etc.

Illustrative examples of subject matter classified in this place:

1.

Sensor-level fusion followed by classification.

2.

Feature-level fusion by combining colour, shape and texture representations.

Neural networks [NN] specially adapted for image or video recognition or understanding, in particular specific architectures and specific learning tasks for this purpose.

These notes provide more information about the technical subject matter that is classified in this place:

Examples of architectures are:

Examples of learning tasks are:

Illustrative examples of subject matter classified in this place:

1A.

1B.

Siamese network showing not similar (left) and similar (right) input pairs.

2.

Recurrent neural network for action recognition.

3.

Region proposal neural network for region of interest (ROI) detection.

4.

Adversarial learning with a generative adversarial neural network for object recognition on different backgrounds.

Graphical models for image or video recognition or understanding, with states modelled as nodes in a graph and transitions between states as graph edges, and where it is assumed that the future state of a system depends on the present state.

These notes provide more information about the technical subject matter that is classified in this place:

Examples of graphical models include:

Applications include learning spatial context for object detection, learning spatio-temporal events for activity recognition, gesture recognition, video segmentation and understanding, etc.

Illustrative examples of subject matter classified in this place:

1.

2.

Human activity recognition using a hidden Markov model [HMM].

Methods and arrangements which use syntactic or structural representations of the image or video patterns for recognition or understanding where objects can be represented by a variable-cardinality set of symbolic, nominal features.

Syntactic pattern recognition which represents structures by means of strings of symbols and formal language analysis algorithms, such as parsing with grammars.Recognition based on graph matching to find relations between patterns.

These notes provide more information about the technical subject matter that is classified in this place:

These methods allow the representation of structures by taking into account interrelationships between patterns or their attributes.

Illustrative examples of subject matter classified in this place:

1.

2.

Face recognition by elastic bunch graph matching.

Optical devices which are specially adapted for recognising patterns; methods making use of these devices.

These notes provide more information about the technical subject matter that is classified in this place:

Detection and recognition of an object usually involves optical correlation between an input (optical) image and either a reference optical mask containing the object of interest, or an optical image of the reference object. The convolution is efficiently performed as superposition of these images (dot products) in the Fourier domain, the latter representation being obtained by a lens or a system of lenses.

At least one element of a processing chain for recognising patterns in image and video data may be an optical hardware component, e.g. a ring-wedge detector, or an optical correlator to determine the similarity between two patterns. The effect of the optical element may be present in both spatial domain and in the frequency domain.

Typically, the optical elements used in these approaches are: specially designed filter masks, spatial light modulators [SLM], holographic masks, phase-only filters, acousto-optic cells, waveguides, polarisers, etc.

Illustrative example of subject matter classified in this place:

Example of an optical correlator.

Hardware solutions (e.g. individual electronic circuits or networks of interacting electronic devices) or software architectures (e.g. data structures or software libraries), which are specially adapted for pattern recognition or image or video understanding.

These notes provide more information about the technical subject matter that is classified in this place:

An example of special adaptation of software might include software arranged to perform a sequence of mathematical operations, which can run particularly efficiently on a particular graphical processing unit [GPU].

An example of special adaptation of hardware might be a processor which is designed to perform operations, that are particularly relevant for pattern recognition (e.g. convolutions), in a power-efficient manner. Another example might be a hardware interface, which makes it possible to communicate an extracted visual pattern very efficiently (in terms of speed or bandwidth) to a server for further processing.

Illustrative example of subject matter classified in this place:

Distributed pattern recognition system in which different resources are placed at different geographical locations.

Processes and devices which control the execution of pattern recognition algorithms.

These notes provide more information about the technical subject matter that is classified in this place:

The control may cause the result of a pattern recognition algorithm to be available within a predetermined time frame, e.g. by prioritising pattern recognition tasks over other tasks, by delegating tasks to processors which are currently idle, by lowering the frame rate (e.g. discarding every other image frame), or by altering the resolution of the image.

The control may also take the urgency of pattern recognition tasks into account; for example, an autonomous vehicle could prioritise tracking a detected pedestrian over tracking other more distant objects.

The control may also be adaptive to the available processing power or to the available bandwidth, e.g. by selecting simpler and less accurate algorithms when being executed on a mobile device, by dynamically altering between batch processing and real-time processing depending on predetermined criteria, or by skipping dispensable pre-processing steps.

Illustrative example of subject matter classified in this place:

Task scheduling to accommodate different steps, e.g. image recording, exposure control, recognition and video output.

Methods and arrangements for detecting or correcting errors in an acquired pattern.

These notes provide more information about the technical subject matter that is classified in this place:

Regarding error correction in acquired images, errors may be detected or corrected automatically, or with the help of an operator. The process may also be semi-automatic; for example, it may involve the displaying of an image on a graphical user interface in order to prompt human intervention, if the quality of the image is insufficient for successful recognition.

Detecting errors, in particular, may comprise evaluating the quality of given image or video data in order to assess its suitability for analysis by an automated pattern recognition process. Typical quality criteria are sharpness/blurriness, resolution, contrast and brightness.

More advanced quality assessment algorithms check the image for objects that are only partly visible (e.g. due to occlusions or because parts of the object have moved outside the field of view). These algorithms may also detect the presence of clutter or shadows, and check whether the position and orientation of the object are as expected, or they determine whether the image complies with quality standards of a particular technical application (e.g. the visibility of the eyes in case of biometric authentication).

If the quality of the image or video data is considered insufficient, the process or device may attempt to improve the quality by capturing a further image, potentially by changing parameter settings of the image capturing process (e.g. by switching on active infrared illumination when the captured image is found to be too dark) or by providing the user with instructions on how to re-capture the image.

Illustrative examples of subject matter classified in this place:

1A.

1B.

The quality of the acquisition of a fingerprint image is influenced by a hair present on the sensor. A stand-alone image of the hair is either removed (left) from the acquired image or the regions containing the hair are discarded in the subsequent analysis (right).

2.

Flowchart according to which face recognition is performed only when the acquired image fulfils a predetermined quality standard.

Scene-specific image or video recognition or understanding according to the category of scene that is perceived by the observer or the scene-specific processing performed.

Examples of different categories of scenes are underwater scenes, terrestrial scenes, augmented reality scenes, albums, collections, shared content such as social networks photos or video, videos such as a film of a TV broadcasting. The context of an image or video includes scenes under surveillance, traffic scenes, scenes exterior to a vehicle, scenes on the interior of a vehicle. Various types of objects can be analysed, such as three-dimensional objects, microscopic objects, food, trinkets, scene text, etc. Examples of scene-specific processing includes semantic and syntactic analysis and classifying the scene content.

Detection, identification and recognition of objects specifically adapted to underwater scenes.

Categorising underwater objects.

Detection, identification and recognition of underwater structures, such as oil or gas pipes.

Detection, identification and recognition of objects or animals located on the sea floor.

Adapting the recognition according to the underwater conditions, e.g. light scattering or absorption, artifacts, blurring, non-uniform lighting, etc.

Recognising underwater objects in the context of simultaneous localisation and mapping [SLAM].

Illustrative example of subject matter classified in this place:

Example of a system for analysing underwater scenes.

Arrangements and methods specifically adapted to recognise terrestrial scenes:

Illustrative example of subject matter classified in this place:

Perspective view of the region imaged by consecutive acquisitions of a hyperspectral sensor.

Recognising patterns corresponding to different image structures (e.g., objects) in remotely sensed satellite images or video, e.g. optical data (images or video), GPS, radar, LIDAR measurement data or in combination.

Object detection, deriving hyperspectral signatures from objects within satellite images. Categorisation of man-made objects / image targets within satellite images.

Vegetation detection or monitoring canopy growth within satellite images.

Cloud detection and cloud mask segmentation within satellite images.

3D measurement of man-made objects, such as building roofs, within satellite images.

Change detection, e.g. assessing influence of natural disasters, presence of new objects (anomalies) against a known background within satellite images.

Weather condition monitoring by image or video analysis of satellite images.

Illustrative example of subject matter classified in this place:

Automatic classification of objects in satellite images.

This group covers techniques specifically adapted for remotely sensed satellite images or video. Recognising patterns in aerial images or video acquired from aircrafts, helicopters, unmanned aerial vehicles (UAVs), balloons, etc., is classified in group G06V 20/17. The difference between these two groups consists of how the images are acquired. Image or video techniques classified in group G06V 20/13 lack perspective (depth) information, while in the group G06V 20/17, images acquired from aircrafts, helicopters, UAVs, etc. contain the perspective (depth) information.

Recognising patterns corresponding to different image structures (e.g. objects) in aerial images or video acquired from aircraft, helicopters, unmanned aerial vehicles (UAVs) or drones, balloons, etc.

Categorisation of man-made objects/image targets in aerial images or video.

Vegetation detection or monitoring canopy growth in aerial images or video.

3D measurement of man-made objects such as building roofs wherein the scene is taken from planes or by drones.

Inspection of buildings or other man-made objects, e.g. damage classification, wherein the scene is taken from planes or by drones.

Recognising flying entities, such as insects or birds, from images or video captured by drones.

Recognising or monitoring the activity of military targets in aerial images or video acquired from aircrafts, helicopters, UAVs or drones, balloons.

Illustrative example of subject matter classified in this place:

1.

Recognising and assessing the damage to a building using a drone.

2.

Determining the surface of roofs using UAVs.

This group covers techniques specifically adapted for aerial images or video acquired from aircraft, helicopters, unmanned aerial vehicles (UAVs), balloons, etc.

Recognising patterns in remotely sensed satellite images or video are covered by group G06V 20/13. The difference between these two groups lies in the manner of acquisition of the images. Image or video techniques covered by group G06V 20/17 relate to perspective (depth) information, while techniques covered by group G06V 20/13, relate to images acquired from satellites, which lack perspective (depth) information.

Object recognition operating in an augmented reality environment and adapted to provide additional information about a scene to a user. The underlying processing may involve one or more of the following steps:

1. acquiring an image of a real scene by an image capture device;

2. detecting and recognising objects in the depicted scene;

3. acquiring additional information which is related to these objects (e.g. from a database);

4. presenting this information on the original image in an overlaid / superimposed manner.

The object detection and recognition processes need to be fast due to real-time constrains. For this reason, additional information provided by other sensors (e.g. accelerometers, gyroscopes, GPS, solid state compasses or RFID) can be used to define or limit the analysis based on the information they provide.

Examples of adaptations include:

These notes provide more information about the technical subject matter that is classified in this place:

Object recognition for augmenting scene information and adapted for head mounted displays and portable devices needs to be fast due to real-time constrains. For this reason, additional information provided by other sensors (e.g. accelerometers, gyroscopes, GPS, solid state compasses, RFID) may be used to define or limit the space subject to analysis.

Different approaches to camera pose estimation and registration may be essential to successful object recognition.

Illustrative example of subject matter classified in this place:

Recognition of a real-world object by a head-mounted computing device.

Determining position or orientation of objects or cameras is covered by group G06T 7/70. Input arrangements or combined input and output arrangements for interaction between user and computer is covered by group G06F 3/01.

Generating groups or clusters from images or video based on their similarity, based on events, backgrounds, identified individuals, etc.

Comparing and forming connections between image collections using matching, classification and clustering.

Detecting or recognising events in an image collection and ordering these events in an event timeline, based on image content.

Construction of a social network by analysis of an image collection.

Illustrative example of subject matter classified in this place:

Steps for constructing a dynamic social network from raw video data of observations of people.

Detection of events in video surveillance, in particular suspicious activities or objects, is classified in group G06V 20/52. Labelling of scene content, e.g. by semantic segmentation, is classified in group G06V 20/70.

Video summarisation/abstraction, e.g. key-frame extraction, extracting of video features or fingerprints, extraction of representative shots, detecting important frames by analysing the reactions of the viewers or by monitoring parts of the video, such as the TV logo.

High-level semantic clustering, classification and understanding of video scenes, e.g. detection, labelling or Markovian modelling. Examples of video content subject to such analysis are sport broadcast events or TV news.

Low-level semantic clustering or determination of sections in videos such as scenes and shots; classification of shots, e.g. as close-up shot, medium shot or long shot.

Extraction of features, e.g. histogram similarity measures, manifolds, by use of video fingerprints, etc. Examples of low-level features are colour or texture-based features, local interest points (key-points), filter responses, edge features, local descriptors (SIFT, SURF, etc.) or combinations of them (see also group G06V 10/40). Examples of high-level features are features related to camera motion (tracking visual features), the presence of skin, the number of faces present, the size of faces or other human features visible, text or other objects that are identifiable in each frame.

Matching video sequences, e.g. by frame or temporal analysis.

Segmenting video sequences, e.g. parsing or cutting the sequence.

Video categorisation, e.g. classify video content into sport/music/news or recognise commercials in media content for substitution.

Sport games analysis, e.g. tactic analysis in sport videos for assistance of coaches and players; final pitching shot indexing for baseball game; indexing the important parts, such as shots, score points, etc; video monitoring the score table.

Generation of compact representations of the video sequence as a result of pattern recognition or image understanding, e.g. creating thumbnails or representative icons.

Detection and recognition of harmful/sexual/violent content.

Discovery of relationships between objects or persons in videos.

Detecting a key/anchor person from a video; characterising the main characters.

Association of a video with semantic information (e.g. keywords) to describe the content (using e.g. Markov random fields).

Generation of semantic labels using a graph which describes the video content, where the nodes are objects or activities and edges are the relationships between them.

Illustrative examples of subject matter classified in this place:

1.

Clustering of the representative frames containing a given face and creation of face thumbnails of a video sequence containing faces.

2.

Recognising football players in a football match and displaying the representative shots in which a certain player was active.

Recognising and understanding scenes according to the context or the environment of the scene, e.g. the type of scene or the situation in which it is acquired.

Recognising different types of objects is classified in group G06V 20/60. Classification in groups G06V 20/60 and G06V 20/50 or subgroups is applied when a certain type of object is recognised in a specific scene context. For example, recognition of license plates, covered by group G06V 20/62, is classified also in group G06V 20/52 when the recognition is performed in the context of a scene under surveillance, such as a parking lot.

Detection and recognition of objects or events in scenes under surveillance, for example by:

The subgroup G06V 20/54 concerns recognising and understanding of traffic scenes, by detection, identification, classification and recognition of traffic patterns, e.g. cars on the roads, traffic junctions, traffic jams, estimating the travel time.

Illustrative examples of subject matter classified in this place:

1.

Monitoring activities for a scene under surveillance.

2.

Monitoring the occupancy of a parking lot for a scene under surveillance.

Detection, identification and recognition of road lanes, lane markings and borders, free road ahead.

Lane and road marking categorisation, e.g. solid lines, dashed lines, markings at pedestrian crossings, direction indicating arrows, etc.

Estimation of road geometry characteristics, such as curvature, slope or elevation, using e.g. disparity maps of road surfaces or the relative motion of surrounding objects using a clothoidal lane model.

Detection of physical entities located at the side of the road, such as structural barriers (e.g. wall, guardrail), delineators and markers.

Recognition of the drivers' driving pattern in relation to the road lanes perceived from the vehicle.

Recognising the trajectory of a car relative to the road.

Detecting the drivable area ahead of the host vehicle, or of the clear path.

Detection or recognition of road surface characteristics, e.g. cracks, holes.

Detection, classification and recognition of road signs, indicators, etc.

Detection or recognition of potential obstacles, e.g. vehicles ahead, pedestrians.

Recognising surrounding objects by the analysis of their relative position or velocity, possibly with the aid of additional sensors.

Recognition of surrounding objects with the aid of a map of the environment. Categorising vehicles, e.g. car, lorry, bicycle.

Detection or recognition of available parking places; parking assistance by recognising surrounding objects and producing an image of the environment during the parking process with an overview of the host vehicle surroundings, such as a bird-eye view.

Detection of foreign matter on the windshield, e.g. water, dirt, snow.

Adapting the recognition according to the weather conditions, e.g. rain, fog, snow.

Recognition of illumination non-uniformities, e.g. discriminating between objects and shadows.

Recognition of scene objects using special illumination, e.g. infrared light for night vision.

Recognition and compensation for the effects of non-uniformities in illumination, e.g. shadows.

Recognition of light-casting objects, such as traffic lights, lights of the cars ahead, etc.

Illustrative examples of subject matter classified in this place:

1.

Recognition of lane markers for autonomous driving.

2.

3.

Recognition of the road geometry (e.g. its slope) by image analysis.

Recognising and understanding of scenes for autonomous driving makes extensive use of a mix of sensors or modalities which are classified in different places in IPC (see the informative references indicated below).

The functions of acquisition, pre-processing, feature extraction, pattern recognition and machine learning classified in group G06V 10/00 are also classified in group G06V 20/56 according to the function. For instance, special illumination (e.g. infrared) used for night vision, is classified in group G06V 10/143 and in group G06V 20/56. Other examples are techniques for determination of a region of interest (ROI) defining the obstacles ahead, classified in group G06V 10/25 and in group G06V 20/56.

Recognising seat occupancy, e.g. forward or rearward facing child seat.

Recognising driver or occupant position, e.g. for automatic seat adjustment, adjustment of the driving wheel or mirrors.

Recognising the drivers' state, behaviour, emotions, e.g. attention, drowsiness, hands on the wheel, drivers' gaze ("eyes-off-road"), potential alcohol consumption, etc.

Recognising the state of vehicle controls, e.g. dashboard indicators such as speedometers, fuel meters, etc.

The recognition may be performed on images taken from an on-board camera located within the vehicle or from images taken from cameras located outside of the vehicle.

Illustrative examples of subject matter classified in this place:

1.

Recognising indicators on the dashboard.

2.

Detecting faces within a vehicle, when the camera is located outside of the vehicle.

Detection and recognition of text or logo regions in scene imagery, e.g. detection and recognition of street names, business logos or names, license plate numbers, or numbers on the clothing of players in a sporting activity.

Localising and recognising text regions on postal items, parcels or containers.

Detection and recognition of overlay text in broadcast video, including embedded captions in TV videos or images.

Illustrative examples of subject matter classified in this place:

1.

Image wherein the overlay text (detection and recognition of the TV station) is recognised.

2.

Image wherein the overlay text (the logo License plate) is recognised.

Recognition of objects based on their three-dimensional geometric structure ("3D shape"), potentially also exploiting other visual cues such as surface texture, grey-level image values or colours.

Note:

The analysed data is three-dimensional in nature, or the reference/template is three-dimensional. The three-dimensional representation can be very varied: depth/range images, also called 2.5D-images (potentially including texture information), point cloud representations, meshes/tessellations/wire frames or finite element representations, voxel representations, representations as manifolds (continuous, smooth or Riemannian manifolds; using local charts; as null sets of a certain set of functions, etc.). The majority of the techniques involved recognise the 3D-surface, or part of the 3D-surface ("front side relative to a camera") of the three-dimensional object rather than the interior or its volume.

Illustrative examples of subject matter classified in this place:

1A.

1B.

1C.

3D object recognition for guiding a robot gripper.