Patent Application
20200178902
The present invention relates to the extraction of information from video sequences and, in particular to the extraction of physiologically-related information (often referred to as vital signs).
It is possible to analyze video sequences of a living subject and detect small changes in the images which are the result of physiological processes of that subject. Amongst these physiological process are such things as blood flow, breathing and sweating.
Certain physiological processes can be observed via skin reflectance variations. The human skin can be modelled as an object with at least two layers, one of those being the epidermis (a thin surface layer) and the other the dermis (a thicker layer underneath the epidermis). A certain percentage 5% of an incoming ray of light is reflected at the skin surface. The remaining light is scattered and absorbed within the two skin layers in a phenomenon known as body reflectance (described in the Dichromatic Reflection Model). The melanin, typically present at the boundary of epidermis and dermis, behaves like an optical filter, mainly absorbing light. In the dermis, light is both scattered and absorbed. The absorption is dependent on the blood composition, so that the absorption is sensitive to blood flow variations. The dermis contains a dense network of blood vessels, about 10% of an adult's total vessel network. These vessels contract and expand according of the blood flow in the body. They consequently change the structures of the dermis, which influences the reflectance of the skin layers.
Other physiological processes such as breathing cause movement in the surface of patient.
Other physiological processes such as variations in blood oxygenation level can manifest themselves as small colour changes.
It is possible to detect and extract signals which have some periodic content in these changes and from that obtain a result such as a frequency in the case of periodic processes. For example, a subject may be illuminated with ambient light and filmed using a video camera. By analyzing changes in the values of corresponding pixels between frames of the sequence of images, a time-variant signal can be extracted. This signal may be transformed into frequency-like domain using something like a Fast Fourier Transform and from the frequency-domain spectra, a value for the subject's heart-rate may be arrived at as a physiological measurement. These physiological measurements are often called vital signs.
The changes in the pixel values are often small and often more pronounced in 1 colour channel than the others. Thus the signal that is being looked for is correspondingly small.
There may be other changes in the pixel values such as those due to changes in the general image and these can be large in comparison to the signal. There are also sources of random change in the pixel values such as movement of the subject, changes in the illumination (such as flicker) and noise in the image sensor and variations in the illumination. All of these are, to all intents and purposes, uncorrelated with the signal being sought. Therefore the signal to noise ratio is small and it may be difficult to obtain a meaningful physiological measurement.
There is provided a method of extracting a physiological information comprising, for a defined time-period, extracting a plurality of signal segments in a time-domain form, selecting a subset of the plurality of signal segments wherein selecting comprises rejecting segments having a high maximum signal level, converting at least said subset to produce a plurality of transformed signal segments, and combining the at least said subset of the plurality of transformed signal segments to produce a combined signal segment.
This has the advantage of reducing the influence of motion and or illumination-induced changes on the final result by removing parts of the raw signal segments which are worst affected by these.
According to an embodiment, the method comprises rejecting segments having a maximum signal level greater than a threshold. This exploits the fact that the signal of interest which contains the physiological information is small and so large signals can be considered as being due to motion or illumination changes.
According to an embodiment, the selecting comprises weighting segments with weights having an inverse relationship to their maximum signal level. This has the advantage of not requiring a hard threshold which may be difficult to set satisfactorily.
According to an embodiment, the inverse relationship is of the form of a power law or an inverse exponential which has the advantage of providing a strong selectivity for small signal levels as compared to large signal levels.
According to an embodiment, the conversion is performed using one of a power spectral density function, an auto-correlation function or a Fourier transform which provide the signal in a frequency-like domain from which the physiological information, which is in the form of a frequency, may be extracted.
According to an embodiment, the combining of the signal segments comprises averaging the signal segments which affords a signal which had further reduced levels of unwanted components.
According to an embodiment, method further comprisises epeating at least once the method described above, storing the combined signal segment at each repetition to produce a plurality of combined signal segments, and averaging the plurality of combined signal segments to produce a final combined signal. This affords further reduction of unwanted components.
According to an embodiment, the settings for the averaging of the plurality of combined signal segments are derived from a previously stored combined signal segment. This allows for adaptive filtering or averaging which helps reduce unwanted components.
According to an embodiment, the method further comprises performing an interpolation of the combined signal which provides further smoothing an unwanted component reduction.
According to an embodiment, the method further comprises extracting a value representative of the physiological information which may then be provided as an output.
According to an embodiment, the method further comprises acquiring a sequence of video frames and the extracting of the plurality of signal segments is performed using remote photoplethysmography. This has the advantage of affording a non-contact (or non-invasive) method of acquiring the signals for analysis and may be performed using a video-camera.
There is provided a system for extracting a physiological information representative of a vital sign comprising a signal extraction unit configured to, for a defined time-period, extract a plurality of signal segments in a time-domain form, a selection unit configured to select a subset of the plurality of signal segments wherein the selection comprises rejecting segments having a high maximum signal level, a convertion unit configured to convert at least said subset to produce a plurality of transformed signal segments, and a combining unit configured to combine the at least said subset of the plurality of transformed signal segments to produce a combined signal segment.
According to an embodiment, the combining unit of the system further comprises a time-averaging unit configured to store the combined signal segments, to store interpolation settings, to apply those stored settings to interpolate subsequent combined signal segments and to combine the stored combined signal segments. This allows reduction of unwanted components by averaging over a longer period of time.
There is provided a computer program software product, stored on a computer-readable medium, configured, when executed on a processor, to execute method described herein. This allows a general purpose computer, when linked to a video camera to operate the method described herein.
There is provided a physiological measurement equipment comprising a system as described herein.
The above, as well as additional objects, features and advantages of the disclosed systems and methods, will be better understood through the following illustrative and non-limiting detailed description of embodiments of devices and methods, with reference to the appended drawings, in which:
In the following description, same references designate like elements.
Thus there is method of extracting a physiological information over a time-period which comprises extracting a plurality of signal segments in a time-domain form, selecting a subset of the plurality of signal segments according their maximum signal level and then converting at least said subset to produce a plurality of transformed signal segments, and then combining the at least said subset of the plurality of transformed signal segments to produce a combined signal segment.
The processing chain 20 may be implemented in one or more general purpose processors running appropriate software. This has the advantage of being possible with pre-existing hardware and allows for subsequent modification and tuning However it can result in a solution which is slower and/or more expensive than a mode dedicated solution. Alternatively some or all of the individual components may be implemented in microcontrollers running firmware designed to implement the relevant functions. This solution may be less expensive when production volumes are sufficiently high enough. Yet another possibility is to implement the functions in dedicated hardware. In high volumes, this is often cheaper and gives higher processing speed per unit cost.
The patches are selected using one or more of a number of methods. A process which is sometimes called ‘segmentation’ is performed. It is convenient to start by selecting the general area of interest. The face is suitable whenever blood flow is the physiological process of interest so a face-identification algorithm may be used. A suitable algorithm for implementing face detection is described in Viola, P. and Jones, M. J., “Robust real-time object detection”, Proc. of IEEE workshop on statistical and computational theories of vision, 13 Jul. 2001. Alternative algorithms for recognizing shape and colour patterns also exist and these may be used for detecting the facial area. For other processes like breathing, other methods for identifying the thorax may be used.
The selection is performed so as to remove those signal segments which would degrade the overall signal-noise-ratio. The inventors have found that a significant source of problems are those signal segments that come from patches where motion of the subject 1 or variations in the illumination are observable and that these patches can be identified by the amplitude of the time-varying signal extracted therefrom. The inventors have made the surprising observation that amplitudes which are too great are indicative of the presence of problems, particularly motion or illumination-variation artifacts. This is somewhat counter intuitive in that normally in a search for greater signal-to-noise ratios, the skilled person would prefer greater signal amplitudes and reject those with smaller amplitudes. There are various embodiments for the selection process. Firstly, the patches may be assessed in a number of ways. The standard deviation of the pixel values in the patch over the time period concerned may be used because this is related to the amplitude of the time-varying signal extracted therefrom. Another possibility is to assess the total power or energy of the signal which is given by the variance of the pixel values of the patch over time. These have the advantage of being easy to integrate into the signal processing chain since such functions are also used for the motion tracking. A further possibility is to assess the maximum signal amplitude in the actual time-varying signal segments, which is more direct and may be easier to tune but requires extra functions and processing. In the case where the selection is performed by examining the pixel values of the patch, the selection could be made before signal extraction which could save some processing power.
Next the results of the assessment are used to make a selection. This selection can be considered as rejecting signal segments having higher maximum signal levels in favour of those having lower maximum signal levels. This can be selecting a maximum threshold value and rejecting all cases exceeding this. An alternative is to apply a weighting to the individual time-varying signal segments where the weightings have an inverse relationship to the signal amplitude (measured or inferred). It is also possible to use a relationship which is not linear but has a power law form like
f(x)=ax−k [1]
or an inverse exponential form like
f(x)=ae−bx [2]
The linear scaling factor a, the power law exponent and the exponential scaling factor may be adjusted to give the desired results.
where T is the threshold and P, is a feature of the signal segment reflecting the amplitude of the i-th signal, from a group Si of 1 to n signal segments. For amplitudes P, below the threshold T, the weight is substantially equal to 1. For amplitudes above T, the weight shows a strong decrease as a function of the amplitude. The feature Pi may be statistical characteristic of the pixel values over time, such as the standard deviation or the total power of the signal as given by the variance of the pixel values over time.
The threshold may be set by a calibration routine run at manufacturing time. This has the advantage of being simple to implement and lower cost. Another method is to arrange for the system to run a training routine or sequence where the threshold is set by checking the actual vital signs result and adjusting the threshold accordingly. This adjustment could be under the control of a technician or indeed under automatic control of the system itself Where automatic control is used, the adjustments routine could be arranged to run over a long period of use or until the system finds that the threshold value obtained is no longer varying.
Using this weighting technique, particularly with a non-linear function has the advantage that it allows the rejection of the outlying i.e. very large values whilst reducing the difficulty of accurately determining the best parameters (which might vary between situations or equipment characteristics) in that they operate more ‘softly’. Nevertheless, it is also possible to combine selection methods such as using a threshold and then a linear and/or non-linear weighting methods. And advantage of this could be computation speed in that very extreme results could be excluded without the need to calculate the weightings, thereby saving time and computational effort.
The selected time-varying signal segments Sb1−n are then transformed by the conversion unit 25 into transformed signal segments Ta1−n, Tb1−n, of a form from which a dominant frequency or periodicity can be extracted. Such operations could be transforming into a spectrum, i.e. frequency domain, by using something like a Discrete or Fast Fourier Transform (DFT or FFT). From the spectrum the DC component and other components considered out-of-band may be discarded and a peak corresponding to the fundamental frequency of the pertinent physiological process.
A DFT may be expressed as, for a sequence of N complex numbers xn
X
k
Σn=0N−1xn·e−j2πkn/N [5]
Another method could be to use an autocorrelation or ACF (also sometimes known as a cross-autocorrelation or serial correlation) function to arrive at result indicative of a quasi-periodic signal. By way of illustration only, a common formation of an estimation for autocorrelation function for a signal for which n observations have been made and for which there are mean μ and variance σ2:
where k is an integer less than n.
From the inverse of the time lag between the peaks of the autocorrelation, a frequency of the periodic signal can be derived.
A third method could be to use a power spectral density function (PSDF). This function represents the frequency distribution of the power of a signal. It is sometimes defined or expressed as, for a finite time series xn of samples of a signal, the samples being at discrete times xn=x(nΔt) for a total time period T=NΔt:
where n is between 1 and N
It may be useful to vary the above expressions or choose different formulations, possibly with other terms, when implementing them in a system for extracting physiological information.
Another possibility is a Laplace transform which can also be used to obtain a frequency-domain representation of a signal from its time-domain form.
Other possibilities exist such as the multiple signal classification (MUSIC) algorithm, the pitch detection algorithm (PDA), the average magnitude different function (ADMF), the average squared mean difference function (ASDMF). There are also algorithms known as the YIN algorithm and the MPM algorithms respectively.
It may also be possible to use multiple transform methods. This would allow comparison of the results for the purposes of confirmation and judging the quality of the result. In a situation where the quality was too low, a reselection with tighter criteria (for example, a lower threshold, coefficients adjusted to give higher non-linearity) could be performed.
It should be noted that the step of selecting signal segments according to their maximum amplitude could also be performed in the frequency domain i.e. after transformation. This would have the advantage of simplifying the selection a little at the expense of extra computational effort transforming segments that would otherwise have been rejected.
Next the transformed signal segments Ta1−n, Tb1−n are combined by the combining unit 27 into combined signal segments CS1−n. At this point, the combined signal segments CS1−n still correspond to the time periods t1−n from which they came, within the limits that some elimination may have occurred during the selection procedure described previously. It should be understood that the combination, which can be averaging the respective signal segments together, is now possible because the transformation has removed the relative phase information that prevented combination of the time-domain representations. This has the advantage of reducing still further unwanted components such as noise, distortions or artefacts introduced by the conversion process. The combined signal segments CS1−n, which have been collected over time, are further combined into a final combined signal 41 from which the physiological information can be derived by finding a dominant frequency or periodic component. In this combining step, noise may be reduced by (weighted) averaging over combined signal segments CSn, collected over a period of time. This can be considered as executing an identical temporal filter per element of signal segments. This filter may be implemented as an FIR or IIR filter. FIR and IIR filters will give different results and the choice between them is made according to different constraints—for example a requirement on linear phase could lead more toward an FIR. Such a choice is within the reach of the skilled person. It is advantageous to perform interpolation to increase the resolution. Examples of suitable methods of interpolation are linear, polynomial or spline. Both steps can be combined and thus the combination unit acts as a time-averaging unit which is configured to store the combined signal segments, to store interpolation values, to store the filter states, and to combine the combined signal elements. The filter settings may be adaptively controlled. If filters with a short time constant already provide a clear indication of the physiological feature of interest (e.g., the frequency or repetition rate of the blood volume pulse) one may select and use this setup. However, if the feature of interest does not stand out sufficiently in the output signal, a larger time constant (i.e. other filter coefficients) can be chosen. The decision method may be as simple as a comparing values obtained for the feature to a threshold. For example, in case where the feature is a frequency associated with the highest peak in the Fourier transform, one may determine the peak value relative to the background level (environment) and let that serve as a control variable. Thus the filter settings are controlled by a local contrast measure within the results output by the function being used. A similar mechanism can be operated when searching for a delay in an ACF. Using a smaller time constant gives the advantage of obtaining a result more quickly (i.e., less latency) and/or with a lower computational load. Using longer time constants may be more accurate or give more stable results. The threshold may be chosen using a calibration routine when the apparatus is started up which would have the advantage of tending to give higher accuracy but requires more time and complexity in the equipment. Alternatively, the threshold could be set either by design or at manufacturing time—which would be cheaper. This process serves to further reduce unwanted components.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. Furthermore, in many cases, embodiments presented as alternatives may be wholly or in part combined.
In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer or processing unit. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
Aspects of the invention may be implemented in a computer program product, which may be a collection of computer program instructions stored on a computer readable storage device which may be executed by a computer. The instructions of the present invention may be in any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. The instructions can be provided as complete executable programs, partial executable programs, as modifications to existing programs (e.g. updates) or extensions for existing programs (e.g. plugins). Moreover, parts of the processing of the present invention may be distributed over multiple computers or processors.
Storage media suitable for storing computer program instructions include all forms of nonvolatile memory, including but not limited to EPROM, EEPROM and flash memory devices, magnetic disks such as the internal and external hard disk drives, removable disks and CD-ROM disks. The computer program product may be distributed on such a storage medium, or may be offered for download through HTTP, FTP, email or through a server connected to a network such as the Internet.
| Number | Date | Country | Kind |
|---|---|---|---|
| 17169542.2 | May 2017 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2018/060674 | 4/26/2018 | WO | 00 |
