Patent Application
20240071084
In many different types of settings, the state of a person relative to a stationary object is useful information to know. For example, in hospital and other clinical and medical situations, whether a patient or other person is on or off his or her bed can signify whether the person is safe or not. If the person is not on the bed, then he or she may have fallen off, for instance, or may have wandered out of the room.
As noted in the background section, the state of a person relative to a stationary object may be useful information to detect. In hospital and other environments, it may not be uncommon to have a video camera in each room, so that the patients in the rooms can be remotely monitored at a nurses' station or other location. However, to detect whether a person has fallen off or otherwise has exited the bed in his or her room, personnel still have to continuously monitor the video feeds from the cameras. The responsible personnel may have many video feeds to monitor, and also may have other responsibilities, which can mean that a person who has left his or her bed may be overlooked or not quickly detected.
Existing techniques to automatedly detect the state of a person relative to a stationary object, such as whether a person is on or off a bed, may rely on video cameras that provide depth information. Such depth sensor cameras may employ infrared light to capture an image of the person and the stationary object in the infrared spectrum. However, such video cameras are generally more expensive than more ordinary cameras that capture just full-color (or black-and-white) images in the visible light spectrum, rendering such state detection techniques cost prohibitive. Moreover, whereas many existing institutions such as hospitals may already have regular video cameras in patient rooms, relatively few have depth cameras, meaning that existing cameras cannot be leveraged for such techniques.
Described herein are techniques for automatedly determining the state of a person relative to a stationary object that overcomes these issues. A first machine learning model, such as a pretrained off-the-shelf mask region-based convolutional neural network, is applied to an image of a person and a stationary object, which may be captured using a non-depth camera device, such as a full color red-green-blue digital camera device. The machine learning model generates a first intermediate image including a simplified pose representation of the person in the image corresponding to the pose of the person. A simplified representation of the stationary object in the image, including key points of the stationary object, is added to the first intermediate image.
A second machine learning model can then be applied to a second intermediate image corresponding to the first intermediate image. For example, if a first intermediate image is generated for each of a number of consecutive frames of a video feed of the person and the stationary object, the second intermediate image may be a concatenation or other combination of these first intermediate images. The second machine learning model may be a residual neural network trained on a minimum number of training images of people and the same stationary object. The second machine learning model outputs the state of the person relative to the stationary object in the image, as either a first or second state.
A first machine learning model 110 is applied (112) to the frames 108, or images, of the person and the stationary object to generate (114) intermediate images 116 corresponding to the frames 108. That is, for each frame 108, a corresponding intermediate image 116 is generated. Each intermediate image 116 includes a simplified pose representation of the person in a corresponding frame 108. The simplified pose representation corresponds to the pose of the person, and may be a stick figure representation of the person's torso and limbs.
The first machine learning model 110 may be a mask region-based convolutional neural network. An example of such a machine learning model is the PyTorch-based modular object detection library known as Detectron2, which is described on the Internet web page ai.facebook.com/blog/-detectron2-a-pytorch-based-modular-object-detection-library-/, and available on the Internet at github.com/facebookresearch/detectron2. The first machine learning model 110 may thus be a pretrained, off-the-shelf machine learning model leveraged for usage herein. As such, the first machine learning model 110 does not have to be trained on images of people relative to stationary objects, and does not have to be trained on images captured using the video camera 106 that captured the video 102. The first machine learning model 110, in other words, does not have to be specific to state determination of people relative to stationary objects within images.
A simplified representation 118 of the stationary object in the frames 108 is added (120) to each intermediate image 116. The resulting intermediate images 116′ therefore each include the simplified pose representation of the person in a corresponding frame 108 and the simplified representation 118 of the stationary object that is common to all the frames 108. The simplified representation 118 includes key points of the stationary object in the frames 108, such as the corner points of the stationary object. For example, the simplified representation may be a polygon having corner points corresponding to these key points.
The key points can be manually prespecified or otherwise manually identified. For example, the video camera 106 may be stationary, and the stationary object is by definition stationary. Once the video camera 106 and the object have been placed at a given location, such as in a room, both are likely at most to be infrequently moved. Therefore, the key points of the stationary object within the frames 108 may have to be manually identified just once, when the architecture 100 is first set up. A user may, for instance, select the corners of the stationary object within an initial video 102 captured by the camera 106, in order to prespecify the key points of the stationary object.
The intermediate images 116′ corresponding to the frames 108 are combined (122) to generate a combined intermediate image 124, which may also be referred to as a composite image. For instance, the intermediate images 116′ may be a grid (i.e., a two-dimensional array) of pixels in each of three color (e.g., red, green, and blue) channels. The intermediate images 116′ can therefore be concatenated into a combined intermediate image 124 that is a tensor of n such grids in each color channel. For example, if each intermediate image 116′ is a 112×112 grid of pixels in three color channels, and if there are sixteen intermediate images 116′, then the combined intermediate image 124 may be a three-dimensional tensor. That is, the combined intermediate image 124 in this case is a 3×16×112×112 tensor.
A second machine learning model 126 is applied (128) to the combined intermediate image 124 to determine the state 132 of the person relative to the stationary object in the video 102 captured by the camera 106. The second machine learning model 126 can be a residual neural network, an example of which is described in detail later in the detailed description. The second machine learning model 126 is specific to person state relative to stationary object determination. The second machine learning model 126 may be trained using training videos of people and stationary objects of the same type as the stationary object captured in the video 102.
In one implementation, the second machine learning model 126 may be trained for the specific stationary object captured in the video 102 by the camera 106 and that has the prespecified simplified representation 118. For example, once the camera 106 and the stationary object have been placed at a given location and the simplified representation of the stationary object specified, the second machine learning model 126 may be trained using labeled training videos captured using the video camera 106. The person in each training video may not be the same, and may not be the same as the person in the video 102.
Furthermore, the stationary object is the same type of object in each training video, and is the same type of object as the stationary object in the video 102. However, the actual object in each training video may differ, and may differ from the actual object in the video 102. For example, the type of stationary object may be a bed. The actual bed that is used in each training video, therefore, may be the same or different, and may be the same or different as the actual bed in the video 102.
In this implementation, the second machine learning model 126 (but not the first machine learning model 110, which as noted may be an off-the-shelf pretrained model) may be trained using training videos captured when the architecture 100 is first set up. As to the specific example of the second machine learning model 126 described in detail below, it has been found that a relatively small number of training images (e.g., video frames) is sufficient for the second machine learning model 126 to have accuracy greater than 87% in correctly determining the state of a person relative to a stationary object. Specifically, fewer than 100 labeled training videos may be sufficient in this respect, and the running length of the training videos in total may be about 36.5 minutes. Each video frame may be considered a training image, such that at 30 frames per second, there are fewer than 66,000 training images. By comparison, existing machine learning models may require more than 1,000,000, or even 2,000,000, training images to have such accuracy.
The state 132 of the person relative to the stationary object in the video 102 captured by the camera 106 can be a first state of the person relative to the stationary object or a second state of the person relative to the stationary object. For example, in the case of a bed, the first state may be that the person is on the bed, and the second state may be that the person is not on (i.e., is off or has exited) the bed. When the second machine learning model 126 is trained, training images of people in both states relative to the stationary object are therefore used. Each training image is labeled as to the state of the person relative to the stationary object.
In the architecture 100, an action 134 can be performed depending on whether the determined state 132 of the person relative to the stationary object in the video 102 is the first state or the second state. For example, in the case of detecting whether or not a person is on a bed, appropriate personnel may be notified if the second machine learning model 126 detects the person's state as not being on the bed. Such personnel may include those who are responsible for the care of the person in question, or personnel who are detected as being inside the room in which the bed is located or closest to this room.
Similarly, if the video 102 captured by the video camera 106 is displayed at a remote location such as a nurses' station, an audible or visual alarm may sound at the station to draw the attention of any monitoring personnel. As another example, responsive to determining that the person is off the bed, the door to the room in which the bed is located may be automatically locked in such a way to prevent the person from leaving the room, but not prevent other people from entering the room. Therefore, a person who is in a potentially confused or impaired state is unable to wander away from the room and become lost.
The architecture 100 has been described in relation to frames 108 of video 102 captured by the camera 106. In such instance, an intermediate image 116 is generated for each frame 108 using the first machine learning model 110. The resulting intermediate image 116′ for the frames 108, to which the simplified representation 118 of the stationary object has been added, are combined into a combined intermediate image 124, such as a tensor. On the basis of the combined intermediate image 124, the second machine learning model 126 determines the state 132 of the person in the video 102 relative to the stationary object in the video 102.
However, the architecture 100 can also be performed in relation to one image, such as a single video frame, of a person and a stationary object that may be captured the camera 106 or by a different camera. An intermediate image 116 is again generated for the image using the first machine learning model 110, to which the simplified representation 118 of the stationary object is added to yield an intermediate image 116′. In this case, since there is just one intermediate image 116′, the combined intermediate image 124 is in effect the intermediate image 116′ itself. That is, on the basis of the intermediate image 116′, the second machine learning model 126 determines the state 132 of the person in the image relative to the stationary object in the image.
More generally, it can be said that the one or multiple intermediate images 116 (and 116′) are first intermediate images, and the combined intermediate image 124 is a second intermediate image corresponding to the first intermediate image(s). In the case of video 102 having multiple frames 108 of a person and a stationary object, the second intermediate image corresponds to multiple first intermediate images in that it is a combination (e.g., a concatenation) of the first intermediate images. In the case of a single image of a person and a secondary object, the second intermediate image corresponds to the first intermediate image in that it is the first intermediate image.
The machine learning model 300 therefore include serially connected pairs 302A, 302B, 302C, and 302D of residual blocks, which are collectively referred to as the serially connected pairs 302 of residual blocks. Each pair 302 of residual blocks includes first and second residual blocks. Specifically, the pairs 302A, 302B, 302C, and 302D respectively include first residual blocks 304A, 304B, 304C, and 304D, which are collectively referred to as the first residual blocks 304, and second residual blocks 306A, 306B, 306C, and 306D, which are collectively referred to as the second residual blocks 306.
Each residual block 304 and 306 includes multiple convolutional layers. Specifically, the first residual blocks 304A, 304B, 304C, and 304D respectively include convolutional layers 308A, 308B, 308C, and 308D, which are collectively referred to as the convolutional layers 308, and which abstract their respective inputs. Similarly, the second residual blocks 306A, 306B, 306C, and 306D respectively include convolutional layers 310A, 310B, 310C, and 310D, which are collectively referred to as the convolutional layers 310, and which likewise abstract their respective inputs.
Each residual block 304 and 306 also includes a skip connection connecting the input of the residual block 304 or 306 in question to the output of this residual block. Specifically, the first residual blocks 304A, 304B, 304C, and 304D respectively include skip connections 312A, 312B, 312C, and 312D, which are collectively referred to as the skip connections 312. Similarly, the second residual blocks 306A, 306B, 306C, and 306D respectively include skip connections 314A, 314B, 314C, and 314D, which are collectively referred to as the skip connections 314.
The machine learning model 300 also includes an initial convolutional layer 316 connected to the first pair 302A of residual blocks. The machine learning model 300 further includes an average pooling layer 318 connected to the last pair 302D of residual blocks, and a fully connected layer 320 connected to the average pooling layer 318. An intermediate image 322, such as the combined intermediate image 124 in
For instance, the intermediate image 322 may be a 3×16×112×112 tensor that is input to the convolutional layer 316 having a kernel size of 3×7×7 with pooling stride of 1×2, which results in a tensor of size 64×16×56×56. This tensor is input to the residual block 304A having convolutional layers 308A of the same kernel size, such as 3×7×7, and that output a square image of size 56×56. Specifically, the tensor is input to the first convolutional layer 308A of the residual block 304A, and further, via the skip connection 312A, to the first convolutional layer 310A of the residual block 306A. The tensor is therefore added to the output of the second convolutional layer 308A of the residual block 304A, resulting in a tensor of size 64×16×56×56.
This tensor is input to the residual block 306A having convolutional layers 310A of the same kernel size as that of the convolutional layers 308A, or 3×7×7, and that output a square image of the same size 56×56. Specifically, the tensor is input to the first convolutional layer 310A of the residual block 306A, and further, via the skip connection 314A, to the first convolutional layer 310B of the residual block 306B. The tensor is therefore added to the output of the second convolutional layer 310A of the residual block 306A, again resulting in a tensor of size 64×16×56×56.
This tensor is input to the residual block 306B having convolutional layers 310B of the same, smaller kernel size of 3×3×3, and that output a larger square image of size 128×128. Specifically, the tensor is input to the first convolutional layer 310B of the residual block 306B, and further, via the skip connection 314B, to the first convolutional layer 308B of the residual block 304B. However, because the image increases in size in the residual block 306B, the skip connection 314B downsamples the input tensor, such as via a three-dimensional convolutional layer having a kernel size of 1×1 and a stride of two. This downsampling is indicated in
This tensor is input to the residual block 304B having convolutional layers 308B of the same kernel size as that of the convolutional layers 310B, or 3×3×3, and that output a square image of the same size 128×128. Specifically, the tensor is input to the first convolutional layer 308B of the residual block 304B, and further, via the skip connection 312B, to the first convolutional layer 308C of the residual block 304C. The tensor is therefore added to the output of the second convolutional layer 308B of the residual block 304B, again resulting in a tensor of size 128×8×28×28.
This tensor is input to the residual block 304C having convolutional layers 308C of the same kernel size of 3×3×3, and that output a larger square image of size 256×256. Specifically, the tensor is input to the first convolutional layer 308C of the residual block 304C, and further, via the skip connection 312C, to the first convolutional layer 310C of the residual block 306C. Because the image increases in size in the residual block 304C, the skip connection 312C downsamples the input tensor, such as in the same way as the skip connection 314B does, and therefore is dashed in
This tensor is input to the residual block 306C having convolutional layers 310C of the same kernel size of 3×3×3, and that output a square image of the same size 256×256. Specifically, the tensor is input to the first convolutional layer 310C of the residual block 306C, and further, via the skip connection 314C, to the first convolutional layer 310D of the residual block 306D. The tensor is therefore added to the output of the second convolutional layer 310C of the residual block 306C, again resulting in a tensor of size 256×4×14×14.
This tensor is input to the residual block 306D having convolutional layers 310D of the same kernel size 3×3×3, and that output a larger square image of size 512×512. Specifically, the tensor is input to the first convolutional layer 310D of the residual block 306D, and further, via the skip connection 314D, to the first convolutional layer 308D of the residual block 304D. Because the image increases in size in the residual block 306D, the skip connection 314D downsamples the input tensor, such as in the same way as the skip connection 314B does, and thus is dashed in
This tensor is input to the residual block 304D having convolutional layers 308D of the same kernel size 3×3×3, and that output a square image of the same size 512×512. Specifically, the tensor is input to the first convolutional layer 308D of the residual block 304D, and further, via the skip connection 312D, to the average pooling layer 318. The tensor is therefore added to the output of the second convolutional layer 308D of the residual block 304D, again resulting in a tensor of size 512×2×7×7.
The average pooling layer 318 averages this tensor to reduce dimensionality and size without reducing the number of connections, outputting a tensor of size 512×1×1×1. This tensor is passed through the fully connected layer 320 for flattening. The output of the fully connected layer 320 is therefore the state 324 of the person relative to the stationary object in the image or video frames on which basis the intermediate image 322 was generated.
The described machine learning model 300 has been shown to have an accuracy of greater than 87% in correctly determining the state of a person relative to a stationary object within an image or video frames. Moreover, just a relatively small number of training images having to be used to train the machine learning model 300 in this respect, when used in the context of the architecture 100. As noted above, for instance, fewer than 66,000 training images are sufficient for the machine learning model 300 to have state determination accuracy greater than 87%.
Techniques have been described herein for determining the state of a person relative to a stationary object in an automated manner. A pretrained, off-the-shelf machine learning model can be used to generate an intermediate image of a simplified pose representation of the person, to which a simplified representation of the stationary object can then be added. Another machine learning model, an example of which has been delineated herein and which can be trained on a relatively small number of training images while still providing high accuracy, can then be applied to the resultant intermediate image to output the state determination of the person relative to the stationary object.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2021/013693 | 1/15/2021 | WO |
