Patent Application
20230230294
The present invention relates to a picture processing device, a picture processing method and a non-transitory computer-readable recording medium that processes a video imaged by an imager.
As of 2020, COVID-19 (hereinafter, referred to as a new coronavirus) infections are expanding worldwide. As preventive measures against infection with the new coronavirus, masks, hand washing, securing of social distancing, and the like are mentioned. In addition, in order to prevent infection via objects, it is required to frequently disinfect an object touched by a person.
In a conference room or a convention hall, a large number of people may gather, and there is a risk of cluster infection. It is necessary to thoroughly clean and disinfect after the end of use of the conference room or the convention hall.
Patent Literature 1: JP 2011-257849 A
However, when it is necessary to clean and disinfect a room in a short time, it is difficult to thoroughly clean and disinfect the entire room. When cleaning and disinfecting the room, if a place to which a virus may be attached is known, the place can be preferentially disinfected, and a disinfection work can be made efficient.
A picture processing device according to one aspect of the present embodiment includes: a video acquisitor structured to acquire a video of a space in which a person exists, the video being imaged by an imager; a video analyzer structured to analyze the acquired video and detect a place where the person has stayed in the space; and a picture constructor structured to construct display pictures that allow all stay places where the person has stayed to be recognized, from start of imaging by the imager.
Note that transformation of any desired combination of the aforementioned constituting elements or representation of the present embodiment, among devices, methods, systems, recording mediums, computer programs, and the like, may also be effective as an aspect of the present embodiment.
Embodiments will now be described by way of examples only, with reference to the accompanying drawings which are meant to be exemplary, not limiting and wherein like elements are numbered alike in several Figures in which:
The invention will now be described by reference to the preferred embodiments. This does not intend to limit the scope of the present invention, but to exemplify the invention.
A first embodiment of the present invention relates to a person monitoring system that photographs an occupant with a camera installed in a vehicle interior, and monitors and records a place touched by the occupant by his/her hand.
As the imaging device 10, a dedicated camera may be installed, or a camera of a drive recorder or a vehicle interior monitoring system may be used. The vehicle interior monitoring system is a system that monitors the entire vehicle interior including a passenger seat and rear seats, and can detect, in addition to dozing and inattentive driving of the driver, the number of occupants sitting on the passenger seat and the rear seat, whether all occupants including the driver wear seat belts, and the like.
The imaging device 10 includes a lens, a solid-state imaging element, and a signal processing circuit. As the solid-state imaging element, a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor can be used. The solid-state imaging element converts light incident through the lens into an electrical video signal and outputs the video signal to the signal processing circuit. The signal processing circuit performs signal processing such as A/D conversion and noise removal on the video signal input from the solid-state imaging element, and outputs the processed signal to the picture processing device 20.
The imaging device 10 may include a depth sensor for constructing a depth map. As the depth sensor, for example, a three-dimensional light detection and ranging (3D-LiDAR) can be used. The LiDAR emits laser light in a photographing direction, measures a time until the reflected light is received, and detects a distance to an object in the photographing direction. Furthermore, the imaging device 10 may include a binocular stereo camera. In this case, it is possible to construct the depth map based on a parallax between the two pictures without providing the depth sensor.
The picture processing device 20 may be implemented on a dedicated IC substrate, or may be implemented by being integrated with a car navigation system or a drive recorder system. The picture processing device 20 may be structured as one function of a general-purpose information terminal device such as a smartphone and a tablet. The imaging device 10 and the picture processing device 20 are connected to each other in a wired or wireless manner. In a case where the picture processing device 20 is implemented on a dedicated IC substrate, the picture processing device 20 and the imaging device 10 may be installed in the same housing, or may be installed in separate housings.
The video acquisitor 21 acquires a video imaged by the imaging device 10. The video analyzer 22 analyzes the video acquired by the video acquisitor 21. In the first embodiment, the video analyzer 22 detects a place touched by an occupant by his/her hand in the vehicle interior.
The video analyzer 22 includes an object recognizer 22a, an object tracker 22b, and a three-dimensional space recognizer 22c. The object recognizer 22a searches for an object in a frame in the video acquired by the video acquisitor 21. The object recognizer 22a includes, as dictionary data, a discriminator for a specific object constructed by learning a large number of pictures in which the specific object appears. In the first embodiment, a person's hand discriminator is prepared. The person's hand discriminator includes at least a discriminator for a state where the back of a hand is opened. Note that as a hand discriminator, it is preferable to prepare a plurality of discriminators for various states of a hand viewed from various angles.
In addition to the person's hand discriminator, discriminators for various accessories existing in the vehicle interior may be prepared. For example, discriminators for a steering wheel, a seat belt, a shift lever, and the like may be prepared.
The object recognizer 22a searches the frame of the video using a discriminator for each object. For recognition of an object, for example, histograms of oriented gradients (HOG) feature amounts can be used. Note that a Haar-like feature amount, a local binary patterns (LBP) feature amount, or the like may be used. In a case where an object exists in the frame, the object recognizer 22a supplements the object with a rectangular detection frame.
The object tracker 22b tracks the object recognized by the object recognizer 22a in the subsequent frames. For tracking the object, for example, a particle filter or a mean shift method can be used. In the first embodiment, an object to be tracked is a hand of a person recognized by the object recognizer 22a.
The three-dimensional space recognizer 22c maps an object detected by the object recognizer 22a in a three-dimensional space specified by the depth map acquired from the imaging device 10. In the first embodiment, the three-dimensional space recognizer 22c detects a place touched by an occupant's hand in the three-dimensional space. In a case where the accessories present in the vehicle interior are also recognized as an object, an accessory touched by the occupant's hand can also be specified.
When the three-dimensional space recognizer 22c detects a frame picture in which the occupant's hand is in a state of touching the accessories, the detected frame picture is temporarily stored in the picture holder 23. The picture holder 23 is a volatile memory for temporarily storing the frame pictures.
On the basis of the frame pictures in the picture holder 23, the picture constructor 24 constructs pictures for displaying contact places (hereinafter, referred to as contact place pictures) that allow the user to recognize all the contact places touched by the occupant in the vehicle interior, from the start of imaging by the imaging device 10. The picture constructor 24 may construct, as the contact place pictures, heat map pictures in which a place more likely to have been touched by the occupant by his/her hand is displayed in a more conspicuous color. For example, a heat map picture with three-stage classification based on the contact possibility may be constructed. For example, a heat map picture in which a region having been touched by the occupant's hand is displayed in red, a region likely to have been touched by the occupant's hand is displayed in yellow, and a region having not been touched by the occupant's hand is displayed in green may be constructed. Note that a region not touched by the occupant's hand may be colorless.
The picture constructor 24 may construct contact place pictures in which brightness of the region touched by the occupant's hand is displayed brightest, the brightness of the region likely to be touched by the occupant's hand is displayed next brightest, and the brightness of the region not touched by the occupant's hand is displayed darkest. In addition, contact place pictures in which the region touched by the occupant's hand is blinking may be constructed.
The region in which a display mode is made conspicuous may be in units of accessories or in units of regions actually touched. In the case of making conspicuousness in units of accessories, for example, even when only part of the seat belt is touched, the entire seat belt is made conspicuous.
The picture constructor 24 may construct a contact place picture in a display mode in which the contact place is more conspicuous as the number of times of touch by the occupant by his/her hand is larger. For example, saturation may be made higher at a contact place where the number of times of touch is larger.
The picture constructor 24 may connect a plurality of frame pictures in which the occupant's hand is in a state of touching somewhere, being stored in the picture holder 23, and construct digest moving pictures for displaying contact places. In addition, the picture constructor 24 may construct one still picture by superimposing a plurality of frame pictures in which the occupant's hand is in a state of touching somewhere.
Note that the picture constructor 24 may construct pictures in which accessories touched by his/her hand of an occupant in the vehicle interior are simply listed in text information. In this case, it is not necessary to accumulate, in the picture holder 23, the frame pictures in which the occupant's hand is in a state of touching the accessory, and it is sufficient to hold management information for specifying the accessories touched by the occupant's hand.
When a user who intends to clean and disinfect the inside of the vehicle interior performs a display instruction operation of contact place pictures to a console (not illustrated), the picture constructor 24 constructs contact place pictures and causes the display device 30 to display the constructed contact place pictures.
The display device 30 may be a display (for example, a liquid crystal display or an organic EL display) of a car navigation system, a display audio system, or a drive recorder system installed in the vehicle C1. In addition, the display device 30 may be a display of a smartphone or a tablet in cooperation with the car navigation system, the display audio system, or the drive recorder system. Moreover, the display device 30 may be a display in a meter panel installed in the vehicle. Note that the display device 30 may be a display dedicated to the person monitoring system 1 according to the present embodiment.
Note that the picture constructor 24 may construct contact place pictures in real time while the vehicle C1 is traveling and cause the display device 30 to display the contact place pictures. In this case, when the occupant touches something in the vehicle interior, the contact place pictures are updated. The picture constructor 24 may cause the display device 30 to display, as contact place pictures, AR pictures in which an attention marker indicating a contact place is superimposed on a video imaged by the imaging device 10.
The recording medium 40 is a non-volatile recording medium for recording a video imaged by the imaging device 10. For example, a semiconductor memory card (for example, an SD card) or an optical disk can be used. The contact place picture constructed by the picture constructor 24 can also be recorded on the recording medium 40.
In the above description, it is assumed that the video analyzer 22 can acquire the depth map from the imaging device 10. If a depth sensor is provided in the imaging device 10 or the imaging device 10 has a binocular structure, the cost increases. In the following description, a processing example in a case where the video analyzer 22 cannot acquire the depth map from the imaging device 10 will be described. In this processing example, the three-dimensional space recognizer 22c is not provided, and a place touched by an occupant by his/her hand in the vehicle interior is detected only from a two-dimensional video.
In this processing example, for each accessory in the vehicle interior, a plurality of discriminators for recognizing a state where the occupant touches the accessory by learning a large number of pictures in which the occupant touches the accessory in various appearances are prepared as dictionary data. A set of discriminators for recognizing a state where the occupant touches the accessory may be prepared for each vehicle type. In this case, the position of each accessory can be specified with high accuracy in the imaged video, and whether the occupant touches the accessory can be determined with high accuracy.
Furthermore, as the dictionary data, a discriminator for recognizing the face of a person wearing a mask may be prepared. In addition, a discriminator for recognizing a state where the occupant is coughing may be prepared.
When detecting an occupant not wearing a mask in the frame pictures, the object recognizer 22a outputs a non-mask wearing detection signal to a warning controller 25. When detecting a coughing occupant in the frame picture, the object recognizer 22a outputs a cough detection signal to the warning controller 25. The object recognizer 22a outputs a distance approach detection signal to the warning controller 25 when a distance between the two occupants sitting on the back seat is equal to or less than a set value in the frame picture. When receiving the non-mask wearing detection signal, the cough detection signal, or the distance approach detection signal from the object recognizer 22a, the warning controller 25 causes the audio output device 50 to output a warning sound or a warning message.
The audio output device 50 may be a speaker of the car navigation system or the display audio system installed in the vehicle C1. In addition, the audio output device 50 may be a speaker of a smartphone or a tablet in cooperation with the car navigation system, the display audio system, or the drive recorder system. Note that the audio output device 50 may be a speaker dedicated to the person monitoring system 1 according to the present embodiment.
When the non-mask wearing detection signal is input from the video analyzer 22, the warning controller 25 causes the audio output device 50 to output a call attention message such as “Please wear a mask.”. In addition, when the cough detection signal is input, the warning controller 25 causes the audio output device 50 to output call attention messages such as “Please ventilate.”, “Please open the window.”, “Please set the air conditioner to the outside air introduction mode.”, “Run negative ion generator.”. Note that the person monitoring system 1 according to the present embodiment may cooperate with an air conditioner system in the vehicle C1. In this case, when the cough detection signal is input, the warning controller 25 can automatically switch the air conditioner system to the outside air introduction mode.
When the distance approach detection signal is input from the video analyzer 22, the warning controller 25 causes the audio output device 50 to output a call attention message such as “Please make the distance between your seats a little.”. Note that when the video analyzer 22 detects three occupants in the rear seat but does not detect an occupant in the passenger seat, a seating change instruction signal may be output to the warning controller 25. When the seating change instruction signal is input from the video analyzer 22, the warning controller 25 causes the audio output device 50 to output a call attention message such as “Could someone please move to the passenger seat?”.
By adding a picture of a specific person, it is possible to exclude the specific person from the monitoring process object for the place touched by his/her hand. Specifically, the face of a person to be excluded from the monitoring process object is imaged by the imaging device 10 and added as dictionary data of the object recognizer 22a. Note that a still picture in which the face of the person appears may be input from the outside to the picture processing device 20.
When recognizing the face of a person in the frame picture, the object recognizer 22a determines whether the face of the person is the face of the person added as an exclusion target. In the case of the face of the person as an exclusion target, the object recognizer 22a or the three-dimensional space recognizer 22c excludes the hand of the person from a monitoring target. In a case where the face of the person recognized in the frame picture is not the face of the person of the exclusion target, detecting/recording processing of the above-described place touched by the hand is applied.
As a person to be added as exclusion target, the owner of the vehicle C1 and his/her family can be considered. In addition, a person determined to be less likely to be infected according to the results of a PCR test, an antigen test, or an antibody test may be added. For example, the load on the picture processing device 20 can be reduced by adding the owner of the vehicle C1 and his/her family as the exclusion target.
Note that the contact place picture Ft may be obtained by superimposing an attention marker, which indicates a region where a virus may exist, on an actually imaged picture of the vehicle interior, or by superimposing the attention marker on a simplified schematic diagram of the vehicle interior. Note that the attention marker may be superimposed by default on accessories (for example, a steering wheel, a shift lever, a winker lever, a door knob, and the like) that the occupant always touches by his/her hand. Note that the contact place picture Ft may be constructed as a heat map picture.
As described above, with the first embodiment, by constructing and displaying the contact place pictures indicating a region that may have been touched by the occupant, it is possible for the user who intends to clean and disinfect to intuitively grasp where to be preferentially disinfected. In particular, it is effective for a person who was not in the vehicle C1 and needs to efficiently clean and disinfect the inside of the vehicle interior in a short time, such as a cleaning person of a car rental company or a gas station clerk. Naturally, since it is difficult even for a person who was in the vehicle C1 to grasp and memorize places touched by all the occupants, it is useful to clean and disinfect while viewing the contact place pictures.
Next, a person monitoring system 1 according to a second embodiment will be described. In the first embodiment, the person monitoring system 1 is described that photographs the occupant with a camera installed in the vehicle interior, detects the occupant's hand, and monitors and records a place touched by the occupant by his/her hand. By contrast, the second embodiment relates to the person monitoring system 1 that detects a person in a room with a camera installed in the room in a building and monitors and records a place where the person stayed.
The person monitoring system 1 according to the second embodiment is assumed to monitor a room in a wide indoor space such as a conference room or a convention hall. Indoor monitoring cameras are typically installed on the ceiling of a room. In a case where a large room is monitored by one monitoring camera, it is necessary to photograph at a wide angle, which makes a person in a frame picture smaller in the size. In addition, a large number of people may be included in the frame picture. By contrast, in the first embodiment, since a narrow vehicle interior space is photographed, the size of the person in the frame picture is not equal to or smaller than a certain size. In the case of an ordinary vehicle, the maximum number of people included in the frame picture is five.
As described above, it is difficult to detect a hand of each person by picture recognition from frame pictures imaged by the imaging device 10 installed in a conference room, a convention hall, or the like and track the detected hand. Therefore, in the second embodiment, the head or the whole body of a person is detected, and the detected head or the whole body is tracked.
In the following description, an example is assumed in which the imaging device 10 is installed at the center of the ceiling of the room to photograph the entire room in a bird's eye view. In this case, the monitoring target is the head of the person viewed from above. Note that the imaging device 10 may be installed at a corner between the ceiling and a wall, and the entire room may be photographed in a manner of being overlooked obliquely. In this case, the monitoring target is the head or the whole body of the person viewed obliquely.
The picture processing device 20 may be structured as one function of a general-purpose information terminal device such as a PC, a tablet, and a smartphone. The picture processing device 20 and the imaging device 10 are connected in a wired (for example, a LAN cable or a USB cable) or wireless (for example, a wireless LAN) manner, and the video acquisitor 21 acquires a video from the imaging device 10.
In the second embodiment, the video analyzer 22 detects a place where the person has stayed in the room. The object recognizer 22a includes a discriminator for the head or the whole body of the person as dictionary data. In addition to the discriminator for the head or the whole body of the person, discriminators for various fixtures existing in the room may be prepared. For example, discriminators for a desk, a chair, and the like may be prepared.
The object recognizer 22a may set an accompanying region around the detection region of the person in the frame picture. For example, a region in a picture corresponding to a circle having a radius of about 1 m to 2 m around the person may be set as the accompanying region. Hereinafter, a region obtained by combining the detection region of the person and the accompanying region set around the person is referred to as a behavior region. Note that the user can change the conversion ratio between the distance in the real space and the distance in a picture and the setting range of the accompanying region.
The object tracker 22b tracks the head or the whole body of the person recognized by the object recognizer 22a. When a plurality of persons are recognized in the frame picture, tracking is performed for each person.
The three-dimensional space recognizer 22c detects a place where the person has stayed in a three-dimensional space specified by a depth map acquired from the imaging device 10. In a case where the fixtures existing in the room are also recognized as an object, a fixture estimated to have been touched by the person can also be specified. For example, when an overlapping portion occurs between the behavior region of the person and the detection region of the fixture, it is estimated that the person touched the fixture.
When detecting a frame picture in which the person is estimated to be in a state of touching the fixture, the three-dimensional space recognizer 22c stores the detected frame picture temporarily in the picture holder 23. Note that in the second embodiment, all the frame pictures in which the person is detected may be temporarily stored in the picture holder 23.
On the basis of the frame picture in the picture holder 23, the picture constructor 24 constructs pictures for displaying stay places (hereinafter, referred to as stay place pictures) that allow the user to recognize all the stay places where the person has stayed in the room, from the start of imaging by the imaging device 10. The picture constructor 24 may construct stay place pictures that allow a movement trajectory of the person in the room to be recognized. The picture constructor 24 may construct stay place pictures that allow a trajectory of the behavior region associated with the movement of the person in the room to be recognized.
The picture constructor 24 may construct, as the stay place pictures, heat map pictures in which a place closer to a place where the person has stayed is displayed in a more conspicuous color. For example, a heat map picture in which a region where the person has stayed is displayed in red, an accompanying region is displayed in yellow, and other regions are displayed in green (or colorless) may be constructed. In addition, a heat map picture in which a region where the person has stayed is displayed in red, and colors gradually change from red→yellow→green (or colorless) as a distance from the region where the person has stayed increases may be constructed.
In addition, the picture constructor 24 may construct stay place pictures in which the brightness of a region where the person has stayed is maximized and the brightness decreases as a distance from the region where the person has stayed increases. In addition, stay place pictures in which a region where the person has stayed is blinking may be constructed.
In a case where an overlapping portion occurs between the behavior region of the person and the detection region of the fixture, the picture constructor 24 may be in a display mode in which the entire fixture is conspicuous, or a display mode in which only the overlapping portion is conspicuous. In a case where the height of the fixture from the floor can be estimated, the picture constructor 24 may be in a display mode in which the fixture is more conspicuous as the height of the fixture from the floor is closer to the height of a hand of the person in a state of sitting or standing.
The picture constructor 24 may be in a display mode in which a region is more conspicuous where detection regions of a plurality of persons are more overlapping. For example, the saturation may be made higher in a region where the number of overlapping detection regions is larger. Note that the picture constructor 24 may be in a display mode in which a region is more conspicuous where behavior regions of a plurality of persons are more overlapping.
The picture constructor 24 may construct digest moving pictures for displaying stay places by connecting a plurality of frame pictures stored in the picture holder 23 in which an overlapping portion occurs between the behavior region of the person and the detection region of the fixture. In addition, the picture constructor 24 may construct one still picture by superimposing a plurality of frame pictures where an overlapping portion occurs between the behavior region of the person and the detection region of the fixture.
Note that the picture constructor 24 may construct pictures in which fixtures that are estimated to have been touched by the person in the room are simply listed in text information. In this case, it is not necessary to accumulate the frame pictures in the picture holder 23, and it is sufficient to hold management information for specifying the fixtures that are estimated to have been touched by the person.
When a user who intends to clean and disinfect the room performs a display instruction operation of stay place pictures to a console (not illustrated), the picture constructor 24 constructs a stay place picture and causes the display device 30 to display the constructed stay places picture.
The display device 30 may be a display of a general-purpose information terminal device such as a PC, a tablet, and a smartphone. Furthermore, the display device 30 may be a projector connected to the information terminal device. In this case, the projector projects the stay place picture on a screen. Furthermore, the display device 30 may be a head mount display connected to the information terminal device. In that case, the head mount display can display the stay place pictures as VR pictures. Note that, in a case where the display pictures for the contact place are constructed as stereoscopic pictures, the head mount display can display the stereoscopic VR video.
Note that the picture constructor 24 may construct stay place pictures in real time during use of the room (for example, during a meeting) and cause the display device 30 (for example, a projector) to display the stay place pictures. In this case, the stay place pictures are updated when the person enters or leaves the room or when the person moves from place to place in the room. The picture constructor 24 may cause the display device 30 to display, as stay place pictures, AR pictures in which an attention marker indicating a stay place of the person is superimposed on a video imaged by the imaging device 10.
Also in the second embodiment, the object recognizer 22a can output a distance approach detection signal to the warning controller 25 when a distance between the persons is equal to or less than a set value in the frame picture. When receiving the distance approach detection signal from the object recognizer 22a, the warning controller 25 causes the audio output device 50 to output a warning sound or a warning message. Note that, even in a case where a non-mask wearing person or a person coughing is detected, it is possible to cause the audio output device 50 to output the warning sound or the warning message.
Moreover, also in the second embodiment, by adding pictures of a specific person, it is possible to exclude the added person from the monitoring process object for the stay place.
In each frame picture, the object recognizer 22a determines whether the detection region of each of the persons P1, P2, and P3 and the detection region of the first desk D1 overlap, and whether the detection region of each of the persons P1, P2, and P3 and the detection region of the second desk D2 overlap. The object recognizer 22a determines that a desk having an overlap is used (the first desk D1 in
In a stay place picture Fs3b illustrated in
A stay place picture Fs3c illustrated in
The predetermined distance is set to 2 m, for example, in consideration of social distancing. The predetermined distance may be switched depending on whether the mask is worn. In this case, the predetermined distance for a non-mask wearing person is set to be longer than the predetermined distance for the person wearing a mask.
Furthermore, in a case where the object recognizer 22a can detect the movement of the hand of the person, a range within which the hand has moved in the region of the entire desk may be marked in red as an attention region. At this time, the region marked in the entire region of the desk may be a region obtained by adding a predetermined buffer region to the range where the movement of the hand is actually detected. In addition, the region marked in the entire region of the desk may be a region overlapping with the behavior regions of the persons P1 and P2.
Note that an overlapping range R3 of the first range R1 and the second range R2 may be marked in a relatively conspicuous color as a higher-level attention region. For example, in the first range R1 and the second range R2, a range where both ranges do not overlap may be marked in green, and the overlapping range R3 of both ranges may be marked in red. In addition, a range of the predetermined distance for one person may be marked in green, an overlapping region of ranges of the predetermined distance for two persons may be marked in yellow, and an overlapping region of ranges of the predetermined distance for three or more persons may be marked in red.
As described above, with the second embodiment, by constructing and displaying the stay place pictures indicating a region where the person has stayed, it is possible for the user who intends to clean and disinfect to intuitively grasp where to be preferentially disinfected. In particular, it is effective for a person who did not use the room (for example, who did not participate in the meeting) and needs to efficiently clean and disinfect the inside of the room in a short time, such as an external cleaner. Naturally, since it is also difficult for a person who used the room to grasp and memorize the movement of all the persons in the room, it is useful to clean and disinfect while viewing the stay place pictures.
Next, a person monitoring system 1 according to the third embodiment will be described. In the second embodiment, an example is described in which the picture constructor 24 constructs stay place pictures that allow a movement trajectory of a person in the room to be recognized, and causes the display device 30 to display the stay place pictures. In the third embodiment, whether a person in the room is speaking is further considered. If a person not wearing a mask speaks in the room, there is a possibility that droplets are scattered, and a possibility that a virus is present in a neighboring region of the person increases.
The sound collecting device 60 includes a microphone for collecting sound in a space assumed to be a monitoring target of the imaging device 10. The sound collecting device 60 may be built in the imaging device 10. The sound collecting device 60 converts collected aerial vibration into an electrical sound signal, and outputs the converted sound signal to the picture processing device 20.
The audio acquisitor 26 acquires a sound signal from the sound collecting device 60. The audio analyzer 27 analyzes the sound signal acquired by the audio acquisitor 26. The audio analyzer 27 analyzes a spectrum of the sound signal to detect presence or absence of human speech sound. In a case where human speech sound is detected, the audio analyzer 27 detects the number of speakers. Furthermore, the audio analyzer 27 detects and holds at least one of the number of speaking times, cumulative speaking time, and loudness of the voice of each speaker. Note that, in the present embodiment, since it is not necessary to recognize speaking content, analysis using an acoustic model or a language model is not required.
As described above, the video analyzer 22 can recognize whether a person wears the mask. In the third embodiment, the video analyzer 22 further includes a discriminator for various states of the mouth of a person. In a case where a person not wearing a mask is detected, the video analyzer 22 recognizes the shape of the mouth of the person for each frame picture and determines the movement of the mouth in a series of the frame pictures. The video analyzer 22 determines whether the person is speaking from the movement of the mouth of the person. The video analyzer 22 detects the number of speakers. Furthermore, the video analyzer 22 detects mouth opening degree of each speaker.
In a case where the number of speakers detected by the video analyzer 22 is smaller than the number of speakers detected by the audio analyzer 27, a possibility that a person who does not fall within an angle of view of the imaging device 10 is speaking, a possibility that a person who does not face a direction of the imaging device 10 is speaking, or a possibility that a person wearing a mask is speaking is considered. In that case, the video analyzer 22 may regard all of the person who does not face the direction of the imaging device 10 and the person wearing a mask as speakers.
The picture constructor 24 references to at least one of the number of speaking times, the speaking time, the loudness of the voice, and the mouth opening degree of each person among the persons imaged by the imaging device 10, and determines a display mode of an accompanying region associated with a detection region of each person in the stay place picture.
The picture constructor 24 may set the area of the accompanying region associated with the detection region of a person who spoke to be larger than the area of the accompanying region associated with the detection region of a person who did not speak. At that time, the picture constructor 24 may set the area of the accompanying region associated with the detection region of a person to be larger as the number of speaking times of the person is larger, as the cumulative speaking time of the person is longer, as the voice of the person is larger, or as the mouth opening degree of the person is larger. Note that the accompanying region the area of which increases may be only a region in a direction in which the face of the person faces (for example, a 180-degree region).
The picture constructor 24 may set the area of the accompanying region associated with the detection region of the person who did not speak to be smaller than the area of the accompanying region associated with the detection region of the person who spoke.
The picture constructor 24 may mark the accompanying region associated with the detection region of the person who spoke in a more conspicuous color (for example, red or purple) than the color of the accompanying region associated with the detection region of the person who did not speak. At that time, the picture constructor 24 may set the color of the accompanying region associated with the detection region of a person to be deeper as the number of speaking times of the person is larger, as the cumulative speaking time of the person is longer, as the voice of the person is larger, or as the mouth opening degree of the person is larger. Note that the accompanying region the color of which is made deeper may be only a region in a direction in which the face of the person faces.
For example, in the above-described
The picture constructor 24 may make color of a region darker where accompanying regions associated with the detection regions of persons who spoke are more overlapping. The picture constructor 24 may mark the accompanying region associated with the detection region of the person who did not speak in a less conspicuous color (for example, green or colorless) than the color of the accompanying region associated with the detection region of the person who spoke.
In the above-described
The picture constructor 24 may mark the overlapping range R3 of the first range R1 and the second range R2 in red when the first person P1 did not have a conversation with the second person P2, and may mark the overlapping range R3 in purple when the first person P1 had a conversation with the second person P2. Here, the attention level of purple is higher than that of red.
As described above, with the third embodiment, when constructing and displaying stay place pictures indicating a region where the person has stayed, by displaying a region where droplets are likely to have been scattered in a conspicuous color, a region that is to be preferentially cleaned and disinfected can be appropriately presented to the user.
The present invention has been described above based on the embodiments. The embodiments are intended to be illustrative only and it will be understood by those skilled in the art that various modifications to their constituting elements and processes can be made and that such modifications are also within the scope of the present invention.
In the first embodiment, for example, a driver basically holds a steering wheel with his/her hand during driving. On the other hand, pictures in which the driver is in a state of performing other movements (for example, movement of touching a bottle holder to drink water, or the like) are greatly different from pictures in which the driver is in a normal state of gripping the steering wheel. The object recognizer 22a may store, in the picture holder 23, all frame pictures in which the driver does not hold the steering wheel with the hand in the frame pictures. The picture constructor 24 may construct digest moving pictures for displaying contact places by connecting a plurality of frame pictures stored in the picture holder 23. In addition, the picture constructor 24 may construct one still picture by superimposing a plurality of frame pictures stored in the picture holder 23. In this case, although the accuracy is reduced, it is possible to construct a contact place picture simply.
In the second embodiment, the object recognizer 22a may store, in the picture holder 23, all frame pictures in which the person is detected in the frame pictures. The picture constructor 24 may construct digest moving pictures for displaying stay places by connecting a plurality of frame pictures stored in the picture holder 23. In addition, the picture constructor 24 may construct one still picture by superimposing a plurality of frame pictures stored in the picture holder 23. In this case, although the accuracy is reduced, it is possible to construct a stay place picture simply.
In the first embodiment, an example in which a person's hand is a monitoring target and the inside of the vehicle interior is monitored is described. In this regard, the monitoring is not limited to the example of monitoring the inside of the vehicle interior as long as it is monitoring of a narrow space where a hand of a person is easily detected. Even in a room in a building, the person monitoring system 1 according to the first embodiment effectively functions as long as it is monitoring a small room allowing only a few people to enter.
In the second embodiment, an example in which a person itself is a monitoring target and the inside of a room of a building is monitored is described. In this respect, the person monitoring system 1 according to the second embodiment can also be applied to monitoring an indoor space in a moving body such as a train, a bus, an airplane, or a ship. It is also applicable to monitoring of a certain range of outdoor space.
Furthermore, the new coronavirus is killed in about 7 days at the maximum, depending on a substance to which the new coronavirus attaches and environmental conditions. Therefore, when a set time or more has elapsed after the end of use of a vehicle or a room, it is not required to display an attention region in the pictures.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-158426 | Sep 2020 | JP | national |
| 2020-158427 | Sep 2020 | JP | national |
| 2020-189227 | Nov 2020 | JP | national |
This application is a Continuation of International Application No. PCT/JP2021/020804, filed on Jun. 1, 2021, which in turn claims the benefits of Japanese Application No.2020-158427, filed on Sep. 23, 2020, Japanese Application No.2020-158426, filed on Sep. 23, 2020, and Japanese Application No.2020-189227, filed on Nov. 13, 2020, the disclosures of which Application is incorporated by reference herein.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2021/020804 | Jun 2021 | US |
| Child | 18187739 | US |
