Patent Grant
10909384
The present disclosure relates to a monitoring system and a monitoring method that display an image indicating a sound source.
In recent years, in a surveillance system as an example of a monitoring system, monitoring a fixed monitoring target area is performed using voice data collected by a microphone in addition to video data captured by a monitor camera. In this surveillance system, the monitor microphone and the monitor camera are installed in a location separate from a location of a monitor device on which the video data captured by the monitor camera is displayed. By performing the monitoring using the voice data in addition to the video data, in a case of any incident or trouble, a background of the incident or the trouble can be clarified by the voice data in the vicinity of a sound source (that is, origin of the incident or the trouble) appearing in the video data.
In addition, as a head mounted display mounting a microphone and is mounted on a head, a head mounted display is known, in which collected sound information is displayed so as to be superimposed on the visual video data. For example, such technologies are disclosed in Japanese Patent Unexamined Publication No. 2012-133250. The head mounted display displays a sound source position on the visual video data and a level of the sound output from the sound source is displayed by a size of a circle and a frequency of the sound output from the sound source is displayed by a color of the circle.
However, in the literature described above, the disclosed configuration is not assumed to be applied to a surveillance system. In other words, in the surveillance system, the sound information of the voice data collected by the monitor microphone is not displayed on the monitor device which displays the video data captured by the monitor camera. Therefore, in the surveillance system in the related art, even if a user views the video image displayed on the monitor device, for example, it is not possible to easily and visually recognize the sound source generating the abnormal sound.
An object of the present disclosure is to provide a monitoring system and a monitoring method that display a sound source appearing on video data obtained by imaging a monitoring target area as visual image information, and improve the efficiency of a monitoring business.
The present disclosure provides a sound source display system including: an omnidirectional camera that images an imaging area; a microphone array that collects a voice in the imaging area; a display that displays image data of the imaging area imaged by the omnidirectional camera; and a signal processor that derives a sound parameter relating to a sound source appearing in image data of the imaging area using voice data collected by the microphone array. The signal processor causes sound source image information in which the sound parameter relating to the sound source is converted into visual information according to the comparison between the derived sound parameter relating to the sound source and a threshold value, to be displayed on the display so as to be superimposed on the image data of the imaging area.
In addition, the present disclosure provides a sound source display method in a sound source display system including an omnidirectional camera and a microphone array. The method includes: imaging an imaging area using the omnidirectional camera; collecting a voice in the imaging area using the microphone array; deriving a sound parameter relating to the sound source appearing in the image data of the imaging area using the voice data collected by the microphone array; and causing sound source image information in which the sound parameter relating to the sound source is converted into visual information according to the comparison between the derived sound parameter relating to the sound source and a threshold value, to be displayed on the display so as to be superimposed on the image data of the imaging area.
According to the present disclosure, a sound source appearing on video data obtained by imaging a monitoring target area can be displayed as visual image information, and it is possible to improve the efficiency of the user's monitoring business. Therefore, the user can visually recognize the sound source appearing on the video image of the displayed area.
Hereinafter, embodiments (hereafter, referred to as the present embodiments) which specifically disclose a monitoring system and a monitoring method according to the present disclosure will be described in detail with reference to the appropriate drawings. However, in some cases, descriptions more detailed than required will be omitted. For example, in some cases, a detailed description for widely known items or a repeated description of substantially the same configuration will be omitted. This is to prevent the description from becoming unnecessary redundant and to facilitate the easy understanding for those skilled in the art. The drawings attached hereto and the description hereinafter are provided in order for those skilled in the art to sufficiently understand the present disclosure, and there is no intention to limit the aspects described in the Claims attached hereto. Hereinafter, a sound source display system that displays a sound source appearing on video data obtained by imaging a monitoring target area as visual image information will be described as an example of the monitoring system. However, the monitoring system and the monitoring method according to the present disclosure are not limited to the usage of monitoring a state of a target such as a person, but may be applied to each usage of performing an observation and a measurement for checking a state in a predetermined area or examining comments and evaluations of a product or a service.
Omnidirectional camera CA is a camera mounting a fish-eye lens which can image a state of a monitoring target area as an omnidirectional image (that is, an image having directions of 360°). In other words, omnidirectional camera CA functions as a monitor camera that can image the state of the monitoring target area (that is, imaging area) where the device itself is installed. Omnidirectional camera CA transmits the data of the omnidirectional image obtained by imaging (hereafter, simply referred to as omnidirectional image data) to directivity control device 2 via network NW.
Microphone array MA collects voices in the monitoring target area (that is, sound collection area) where the device itself is installed in a non-directional state. In the present embodiment, the imaging area and the sound collection area are described as being the same. Microphone array MA includes donut-shaped housing (housing is not illustrated) 5 in which, for example, an opening is formed at the center thereof. A plurality of microphone units is concentrically arrayed along the circumferential direction around the opening. For example, a high sound quality and small type electret condenser microphone (ECM) is used as the microphone unit (hereafter, simply referred to as a microphone). Microphone array MA transmits the voice data obtained by the sound collection to directivity control device 2 via network NW.
In addition, by omnidirectional camera CA being incorporated in the inner side of the opening formed at the center of housing 5 of microphone array MA, omnidirectional camera CA and microphone array MA are coaxially disposed and accommodated in housing 5. In this way, since an optical axis of omnidirectional camera CA and a central axis of housing 5 of microphone array MA are coincident, the imaging area and the sound collection area become substantially the same in the axial circumferential direction, and thus, an image position (that is, a position of a subject seen from omnidirectional camera CA) and a collected sound position (that is, a sound source position seen from microphone array MA) can be expressed on the same coordinates system. Therefore, a correspondence relationship between the image position and the collected sound position can be clear. Thus, when forming the directivity of the voice collected by microphone array MA described below as the user designation-based directivity direction, directivity control device 2 can use the coordinates indicating the direction obtained by omnidirectional camera CA based on the user's designation as the directivity direction seen from microphone array MA, and thus, it is possible to easily perform a calculation of the directivity direction.
Directivity control device 2 can form the directivity on the voice data collected by microphone array MA and can emphasize the voice from the directivity direction. Directivity control device 2 is configured to have, for example, a personal computer (PC). In addition, directivity control device 2 calculates a sound pressure at the position corresponding to a pixel configuring image data in unit of a pixel based on the image data of the imaging area captured by omnidirectional camera CA and the voice data collected by microphone array MA, and generates a voice heat map described below. In addition, directivity control device 2 is connected to monitoring monitor 3 and displays the voice heat map on monitoring monitor 3. Directivity control device 2 may be configured to include a communication terminal such as a mobile phone, a tablet terminal or a smart phone instead of using the PC.
Monitoring monitor 3 as an example of a display, displays the omnidirectional image data captured by omnidirectional camera CA. In addition, monitoring monitor 3 displays composite image data in which the voice heat map generated by directivity control device 2 is superimposed on the omnidirectional image data. Monitoring monitor 3 may be configured as a device integrated with directivity control device 2.
Sensor 4 detects an event which starts the sound source display operation described below. As sensor 4, for example, an illuminance sensor that detects the brightness around a location where sensor 4 is installed, a temperature sensor that detects the temperature of the location where sensor 4 is installed, and a human sensor that senses infrared reflected from a person in the vicinity of the location where sensor 4 is installed, are included.
Compression processor 6 generates a packet of the voice data based on the digital voice signals output from A/D converters A1 to An. Transmitter 7 transmits the packet of the voice data generated by compression processor 6 to directivity control device 2 via network NW.
As described above, microphone array MA amplifies the output signals from microphones M1 to Mn using amplifiers PA1 to PAn, converts the analog voice data to the digital voice data using A/D converters A1 to An, and then, generates the packet of the voice data using compression processor 6, and transmits the packet of the voice data to directivity control device 2 via network NW.
CPU 11 performs signal processing that totally manages the overall operation control of each unit in omnidirectional camera CA, data input output processing between other units, data operation processing, and data storage processing. Processors such as a micro processing unit (MPU) or a digital signal processor (DSP) may be provided instead of CPU 11.
For example, CPU 11 generates cut-out image data which is an image of a specific range (direction) cut out from the omnidirectional image data according to a designation of the user who operates directivity control device 2, and stores the cut-out image data in memory 15.
Image sensor 14 is configured to include, for example, a complementary metal oxide semiconductor (CMOS) sensor or a charge-coupled device (CCD), and acquires the omnidirectional image data by performing processing for capturing an optical image of the reflection light collected by fish-eye lens (not illustrated) from the imaging area on a light receiving surface.
Memory 15 includes ROM 15z that stores a program or set value data regulating an operation of omnidirectional camera CA, RAM 15y that stores the omnidirectional image data or the cut-out image data which is the image cut out therefrom, or work data, and memory card 15x that is removably connected to omnidirectional camera CA and stores various data items.
Communicator 12 is a network interface (I/F) that controls the data communications with network NW connected to communicator 12 via network connector 16.
Power manager 13 supplies a DC power to each unit of omnidirectional camera CA. In addition, power manager 13 may supply the DC power to the devices connected to network NW via network connector 16.
Network connector 16 is a connecter that transmits the omnidirectional image data or two-dimensional panorama image data to directivity control device 2 via network NW, and that can supply the power via a network cable.
Setting manager 29 includes a coordinates conversion formula that converts the coordinates indicating the position on the omnidirectional image data designated by the user into coordinates indicating the directivity direction from microphone array MA toward the actual sound source position corresponding to the position on the omnidirectional image data designated by the user on monitoring monitor 3 on which the omnidirectional image data captured by omnidirectional camera CA is displayed. Directivity direction calculator 20 described below calculates the coordinates (θMAh, θMAv) indicating the directivity direction from microphone array MA toward the sound source position corresponding to the position on the omnidirectional image data designated by the user, using the coordinates conversion formula held by setting manager 29. Details of the coordinates calculation processing are disclosed in, for example, in Japanese Patent Unexamined Publication No. 2015-029241. Here, θMAh represents a horizontal angle of the directivity direction from microphone array MA toward the voice position, and θMAv represents a vertical angle of the directivity direction from microphone array MA toward the voice position. The sound source position is a position of the actual sound source corresponding to the position designated from operator 22 by a user's finger or a stylus pen on the video data displayed on monitoring monitor 3.
In addition, setting manager 29 holds a first threshold value and a second threshold value which are compared with sound pressure p for every pixel calculated by sound pressure calculator 24. Here, sound pressure p is used as an example of a sound parameter regarding the sound source, represents a magnitude of the sound collected by microphone array MA, and is different from a volume which represents the level of the sound output from loudspeaker device 27. The first threshold value and the second threshold value are values compared with the sound pressure of the voice generated in the imaging area, and are set to arbitrary values determined to be noisy as the sound source by the user. Here, the sound source is a sound source in a broad sense including not only the sound source actually generating the sound but also a place where a sound as a result of propagation from the sound source can be heard. In addition, a plurality of threshold values may be set, and thus, two threshold values of the first threshold value and the second threshold value which is larger than the first threshold value are set according to the present embodiment (the first threshold value<the second threshold value). In addition, as described below, a part of pixel having a sound pressure higher than the second threshold value is drawn, for example, in red on monitoring monitor 3 on which the omnidirectional image data is displayed. In addition, a part of pixel having a sound pressure higher than the first threshold value and equal to or lower than the second threshold value is drawn, for example, in blue on monitoring monitor 3 on which the omnidirectional image data is displayed. In addition, a part of pixel having a sound pressure equal to or lower than the first threshold value is drawn, for example, in colorless on monitoring monitor 3 on which the omnidirectional image data is displayed, that is, the display color of the omnidirectional image data is not changed.
Communicator 21 receives the omnidirectional image data or the cut-out video data transmitted from omnidirectional camera CA and the voice data transmitted from microphone array MA, and outputs the data items to signal processor 23.
Operator 22 is a user interface (UI) for notifying signal processor 23 of the content of the user's input operation, and is configured to include pointing devices such as a mouse and a keyboard. In addition, operator 22 may be configured to include a touch panel or a touch pad which is arranged corresponding to a screen of monitoring monitor 3, and a direct input operation can be input by the user's finger or a stylus pen.
When a red region R1 of voice heat map MP (refer to
Memory 28 is configured to include, for example, a random access memory (RAM), and functions as a program memory, a data memory, and a work memory when directivity control device 2 operates. In addition, memory 28 stores voice heat map MP described below (refer to
Signal processor 23 is configured to include a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP), and performs control processing that totally manages the overall operation control of each unit of directivity control device 2, data input output processing between other units, data operation (calculation) processing, and data storage processing. Signal processor 23 includes at least directivity direction calculator 20, sound pressure calculator 24, and output controller 25.
When an arbitrary position on monitoring monitor 3 on which the omnidirectional image data captured by omnidirectional camera CA is displayed is designated by the user, directivity direction calculator 20 calculates the directivity direction from microphone array MA toward the sound source position corresponding to the designated position using the coordinates conversion formula held in setting manager 29. That is, directivity direction calculator 20 calculates the coordinates (θMAh, OMAN) of the directivity direction from microphone array MA toward the sound source position corresponding to the coordinates (X, Y) of the designated position on the omnidirectional image data.
Sound pressure calculator 24 calculates the sound pressure of each pixel configuring the omnidirectional image data based on the omnidirectional image data captured by omnidirectional camera CA and the voice data collected by microphone array MA. The sound pressure calculation processing is a known technology and the detailed description of the processing will be omitted. In this way, sound pressure calculator 24 generates a sound pressure map in which the calculated value of the sound pressure is assigned to the corresponding position of the pixel for each pixel configuring the omnidirectional image data. The description is made such that sound pressure calculator 24 generates the sound pressure map in which the value of the sound pressure calculated in unit of pixel is assigned to the corresponding position of the pixel. However, sound pressure calculator 24 may generate the sound pressure map by calculating the average value of the sound pressure values in the unit of pixel block formed from a predetermined number of pixels (for example, 4 pixels) without calculating the sound pressure for each pixel, and assigning the corresponding average value of the sound pressure values to the corresponding predetermined number of pixels.
Output controller 25 controls each operation of monitoring monitor 3 and loudspeaker device 27, causes the omnidirectional image data or the cut-out video data transmitted from omnidirectional camera CA to be output and displayed on monitoring monitor 3, and causes the voice data transmitted from microphone array MA to be output to loudspeaker device 27.
In addition, output controller 25 performs directivity forming processing on the voice data collected by microphone array MA using the voice data collected by microphone array MA and the coordinates (θMAh, θMAv) of the directivity direction calculated by directivity direction calculator 20, and then, performs the processing that emphasizes the voice data of the directivity direction. The voice data directivity forming processing is a known technology disclosed in, for example, Japanese Patent Unexamined Publication No. 2015-029241.
In addition, output controller 25 generates voice heat map MP (refer to
As an example of a voice output, loudspeaker device 27 outputs the voice data collected by microphone array MA or the voice data collected by microphone array MA and the directivity of which is formed by signal processor 23. Loudspeaker device 27 may be configured as a device separate from directivity control device 2.
Operations of the above-described sound source display system 1 will be described.
In
When the image transmission request is received from directivity control device 2, omnidirectional camera CA starts an operation (imaging operation) for imaging a situation in the imaging area (T2). Directivity control device 2 performs a voice transmission request for requesting microphone array MA to transmit the voice data of the sound collection area (T3). When the voice transmission request is received from directivity control device 2, microphone array MA starts an operation (sound collection operation) for collecting the voice in the sound collection area (T4). Here, an office, a warehouse, a hall, and the like can be exemplified as the imaging area (sound collection area).
Omnidirectional camera CA transmits the omnidirectional image data of the imaging area obtained from the image to directivity control device 2 (T5). When the omnidirectional image data transmitted from omnidirectional camera CA is received, directivity control device 2 performs processing that converts the image data into the image data suitable for displaying on monitoring monitor 3 (T6), and outputs the conversion resulting image data to monitoring monitor 3 (T7). The processing of STEP T6 may be omitted. When the image data output from directivity control device 2 is received, monitoring monitor 3 displays the image data of the imaging area captured by omnidirectional camera CA (T8).
When the voice data transmitted from microphone array MA is received (T9), directivity control device 2 generates the sound pressure map in which the calculation value of the sound pressure is assigned to the position of a corresponding pixel for each pixel configuring the omnidirectional image data (T10). The sound pressure map is the data in which the calculated sound pressure value of the voice is assigned to the image data obtainable by imaging the imaging area for each pixel configuring the image data.
In
Setting manager 29 sets the origin point (0, 0) of the coordinates (X, Y) representing the position of each pixel configuring the omnidirectional image data captured by omnidirectional camera CA (S2). Hereinafter, in order to make it easy to understand the description, the coordinates (X, Y) representing the position of the pixel are set in a rectangular-shaped range while a left upper corner of the omnidirectional image data is the origin (0, 0) and the right lower corner of the omnidirectional image data is the final position (MAX, MAX). The processing in STEP S2 may be performed by directivity direction calculator 20.
Oriented direction calculator 20 calculates the coordinates (θMAh, θMAv) of the directivity direction from microphone array MA toward the actual position corresponding to the position indicated on the coordinates (X, Y) on the omnidirectional image data. Sound pressure calculator 24 calculates sound pressure P on the coordinates (θMAh, θMAv) of the directivity direction corresponding to the coordinates (X, Y) on the omnidirectional image data (S3).
Then, sound pressure calculator 24 determines whether or not the coordinate X reaches the value MAX (S4). In a case where the coordinate X does not reach the value MAX (No in S4), sound pressure calculator 24 increments the value of the coordinate X (S5). Then, the processing by directivity control device 2 returns to STEP S3.
On the other hand, in a case where the coordinate X reaches the value MAX (Yes in S4), sound pressure calculator 24 determines whether or not the coordinate Y reaches the value MAX (S6). In a case where the coordinate Y does not reach the value MAX (No in S6), sound pressure calculator 24 returns the value of the coordinate X to zero and increments the value of the coordinate Y (S7). Then, the processing by directivity control device 2 returns to STEP S3.
On the other hand, in STEP S6, in a case where the coordinate Y reaches the value MAX (Yes in S6), sound pressure calculator 24 generates the sound pressure map in which sound pressure P (X, Y) corresponding to each pixel is assigned to the position of each pixel of the omnidirectional image data, using the calculation result in STEP S3. Sound pressure calculator 24 stores the data of the generated sound pressure map in memory 28 (S8). Then, the processing by directivity control device 2 returns to STEP 51.
Back to
In this way, output controller 25 compares the calculation value of the sound pressure with a plurality of threshold values including the first threshold value and the second threshold value, and generates voice heat map MP having the sound image information such that the sound image information in the omnidirectional image data changes depending on the level of the sound pressure. In this way, using voice heat map MP, the user can know not only the location of the sound source but also the level of the sound pressure in the omnidirectional image data captured by omnidirectional camera CA. Here, two threshold values such as the first threshold value and the second threshold value are used as the threshold value. However, one threshold value or three or more threshold values may be used. In addition, the threshold value can be set as an arbitrary value in accordance with the sound pressure of the voice that the user wants to monitor.
Directivity control device 2 transmits the data of voice heat map MP generated by the color conversion processing to monitoring monitor 3 (T12). Monitoring monitor 3 generates and displays a composite image (refer to
However, according to the present embodiment, in order not to degrade the visibility of the composite image, among the data of voice heat map MP, a portion of the sound image information generated in the color conversion processing is superimposed on the omnidirectional image data. The sound pressure value in the sound pressure map may not be or may be superimposed on the omnidirectional image data.
STEP T5 to STEP T13 described above are repeated until directivity control device 2 detects a stop event and the stop event is performed on omnidirectional camera CA to stop the image transmission and microphone array MA to stop the voice transmission. The stop event includes the events that the power of directivity control device 2 is turned off, that directivity control device 2 is shutdown, and that processing is performed in order to instruct to stop the voice heat map MP generation processing.
In
In addition, for example, red region R2 drawn in red and blue region B2 drawn in blue surrounding red region R2 are displayed at the left side of the sheet of composite image GZ1. A ring tone of a fixed telephone installed in an office is present in the range surrounded by red region R2 and blue region B2 as the sound source.
In addition, for example, red region R3 drawn in red and blue region B3 drawn in blue surrounding red region R3 are displayed at a slightly lower side of the left side of the sheet of composite image GZ1. A sound of a fan of the PC installed in the office is present in the range surrounded by red region R3 and blue region B3 as the sound source. In another portion of composite image GZ1, since the calculated sound pressure value is equal to or lower than the first threshold value (refer to
In
In addition, red region R5 drawn in red and blue region B5 drawn in blue surrounding red region R5 and extending to the right side are displayed at a slightly right side of the sheet of composite image GZ2. An operation sound of a machine installed in the office is present in the range surrounded by red region R5 and blue region B5 as the sound source. In another portion of composite image GZ2, since the calculated sound pressure value is equal to or lower than the first threshold value (refer to
Here, the omnidirectional image is displayed on monitoring monitor 3 as a circle-shaped image as it is. However, the image may be converted into a panorama image and displayed as a two dimensional rectangular image (panorama image). In this case, the visibility of the composite image for the user is further improved.
Back to
Directivity control device 2 forms the directivity of the voice collected by microphone array MA to the directivity direction from microphone array MA toward the actual sound source position corresponding to the position on the composite image data designated by the user, and emphasizes the voice in the directivity direction (T16). Loudspeaker device 27 outputs the voice which is emphasized in the directivity direction (T17). In this way, the user can clearly hear the voice coming out from the sound source displayed as the sound image information, and can grasp and pursue the causes of the sound.
As described above, in sound source display system 1 according to the present embodiment, omnidirectional camera CA images the image of the imaging area. Microphone array MA collects the voice in the imaging area. Monitoring monitor 3 displays the image of the imaging area captured by omnidirectional camera CA. Sound pressure calculator 24 in signal processor 23 calculates the sound parameter (for example, sound pressure P) relating to the sound source appearing in the image data of the imaging area using the voice data collected by microphone array MA. Output controller 25 in signal processor 33 compares sound pressure P obtained from the calculation with the threshold values (for example, the first threshold value and the second threshold value), and causes the sound source image information (for example, the sound image information) in which the sound pressure is converted into the visual information to be superimposed on the image data of the imaging area according to the result of the comparison, and to be displayed on monitoring monitor 3.
In this way, sound source display system 1 can display the sound source (that is, sound generation source) appearing on the video data obtained by imaging the monitoring target area (for example, the imaging area) as the visual sound image information. Accordingly, the user can visually recognize the sound source appearing in the displayed video image of the imaging area.
In addition, in sound source display system 1, for example, a plurality of threshold values are set, and sound pressure calculator 24 of signal processor 23 generates a plurality of types of sound image information according to the result of comparison between sound pressure P of the sound source and each of the threshold values (for example, the first threshold value and the second threshold value). In this way, directivity control device 2 can generate voice heat map MP that can make the user intuitively grasp not only the location (position) of the sound source appearing on the omnidirectional image data captured by omnidirectional camera CA but also the level of the sound pressure in the sound source.
In addition, in sound source display system 1, in a case where the sound image information (for example, red region R2) of the composite image data (for example, refer to
In addition, in sound source display system 1, sensor 4 that detects a predetermined target is provided. In a case where the predetermined target is detected by sensor 4, omnidirectional camera CA transmits the omnidirectional image data obtained by imaging to directivity control device 2, and microphone array MA transmits the voice data obtained by collecting the sound to directivity control device 2. In this way, in a case where the condition desired by the user is satisfied, directivity control device 2 can start the operation for displaying the sound source on the omnidirectional image data, and thus, it is possible to improve the convenience for the user.
In addition, in sound source display system 1, omnidirectional camera CA and microphone array MA are coaxially disposed. In this way, since an optical axis of omnidirectional camera CA and a central axis of housing 5 of microphone array MA are coincident, the imaging area and the sound collection area become substantially the same in the axial circumferential direction, and thus, an image position (that is, a position of a subject seen from omnidirectional camera CA) and a collected sound position (that is, a sound source position seen from microphone array MA) can be expressed on the same coordinates system. Therefore, a correspondence relationship between the image position and the collected sound position can be clear. Thus, when forming the directivity of the voice collected by microphone array MA as the user designation-based directivity direction, directivity control device 2 can use the coordinates indicating the direction obtained by omnidirectional camera CA based on the user's designation as the directivity direction seen from microphone array MA, and thus, it is possible to easily perform a calculation of the directivity direction.
Next, another embodiment will be described.
In addition, microphone array device MA, camera device CA and lighting device 50 are installed in an imaging target area (for example, monitoring area 9) in a store, warehouse, a factory, or the like. On the other hand, directivity control device 30 and recorder 60 are installed in an office or a waiting room (for example, monitoring room 8) corresponding to a backyard of monitoring area 9.
Camera device CA as an example of a capture is a pan-tilt-zoom (PTZ) camera which can freely move to the panning direction, move to the tilting direction, and perform zooming processing, is installed on a flat surface such as a wall or a ceiling in the room, and functions as a monitor camera that can image in an arbitrary direction of monitoring area 9 where the device itself is installed.
Camera device CA may be an omnidirectional camera which can image a state around monitoring area 9 as an omnidirectional image (that is, an image having directions of 360°). In a case where camera device CA is an omnidirectional camera, as an example of the omnidirectional image, camera device CA may generate circle-shaped omnidirectional image data in which all the direction around monitoring area 9 is projected, or may perform two-dimensional panorama conversion processing on the circle-shaped omnidirectional image data and generate panorama image data.
In addition, camera device CA may be configured not in one camera, but in a plurality of fixed cameras of which the angles of view which are different from each other are fixedly set. In a case where camera device CA is a plurality of fixed camera, any one fixed camera can be arbitrarily selected from the plurality of fixed cameras by the user (for example, a monitoring person) operating directivity control device 30.
Camera device CA stores a time when the image is captured (that is, an imaging time) in association with the image data and transmits the image data including the imaging time data to directivity control device 30 via network NW. Hereinafter, the image obtained by imaging monitoring area 9 using camera device CA (for example, any fixed camera selected from the PTZ camera, the omnidirectional camera, or a plurality of fixed cameras) will be referred to as “capture image”. Therefore, according to the present embodiment, as illustrated in
Microphone array device MA as an example of a sound collector is installed on a flat surface such as a ceiling in the room and non-directionally collects the voices of all the direction around monitoring area 9 where the device itself is installed. Microphone array device MA includes a housing in which an opening is formed at the center thereof, and further includes a plurality of microphone units concentrically arrayed along the circumferential direction around the opening. Each of the microphone units are arrayed at equal intervals. For example, a high sound quality and small type electret condenser microphone (ECM) is used as the microphone unit (hereafter, simply referred to as a microphone).
Microphone array device MA stores the voice data obtained by collecting the sound in association with a time when the sound is collected (that is, a sound collection time), and transmits the voice data including the sound collection time data to directivity control device 30 via network NW.
In addition, in monitoring control system 10, in a case where camera device CA is an omnidirectional camera, camera device CA and microphone array device MA may be coaxially disposed. In this way, since an optical axis of omnidirectional camera CA and a central axis of the housing of microphone array MA are coincident, the coordinates system of an image capturable region of camera device CA and the coordinates system of the sound collectable region of microphone array device MA are substantially the same. In other words, a position of a subject seen from camera device CA and a sound source position seen from microphone array device MA can be expressed on the same coordinates system. Therefore, a correspondence relationship between the position of the subject and the sound source position can be clear. Thus, when forming the directivity of the voice data collected by microphone array device MA as the user designation-based directivity direction (refer to the description below), directivity control device 30 can use the coordinates indicating the direction obtained by camera device CA based on the user's designation as the directivity direction seen from microphone array device MA, and thus, it is possible to easily perform a calculation of the directivity direction. The coordinates indicating the direction obtained by camera device CA based on the user's designation are the coordinates that camera device CA calculates the direction toward the actual position in space corresponding to the coordinates in a case where the position on the screen of monitoring monitor 36 on which the image data captured by camera device CA is designated by the user and directivity control device 30 transmits the coordinate data of the designated position to camera device CA.
Directivity control device 30 is configured to include a stationary personal computer (PC) installed outside (for example, monitoring room 8) of the room (for example, monitoring area 9) where, for example, microphone array device MA and camera device CA are installed. When the position on monitoring monitor 36 where the image data captured by camera device CA is designated by the user (for example, the monitoring person), the directivity control device 30 forms the directivity of the omnidirectional voice non-directionally collected by microphone array device MA to the direction (hereinafter, referred to as directivity direction) from microphone array device MA toward the sound source position corresponding to the position on the designated image data, and then, emphasizes the voice of the directivity direction.
In addition, directivity control device 30 estimates the position of the sound source (hereinafter, referred to as simply “sound source position”) in monitoring area 9 based on the voice data collected by microphone array device MA. Directivity control device 30 may be configured to include a communication terminal such as a mobile phone, a tablet terminal, or a smart phone instead of the PC.
Here, a specific configuration of directivity control device 30 will be described. Directivity control device 30 illustrated in
Setting manager 39 includes coordinates conversion formula that converts the coordinates indicating the position on the image data designated by the user into the coordinates indicating the directivity direction from microphone array device MA toward the actual sound source position corresponding to the position on the image data designated by the user on monitoring monitor 36 on which the image data of monitoring area 9 captured by camera device CA is displayed. In the coordinates conversion processing, setting manager 39 calculates the coordinates (θMAh, θMAv) indicating the directivity direction from microphone array device MA toward the sound source position corresponding to the position on the image data designated by the user, using the coordinates conversion formula described above. Details of the coordinates calculation processing are disclosed in, for example, in Japanese Patent Unexamined Publication No. 2015-029241. Here, θMAh represents a horizontal angle of the directivity direction from microphone array device MA toward the sound source position, and θMAv represents a vertical angle of the directivity direction from microphone array device MA toward the sound source position. The sound source position is a position of the actual sound source corresponding to the position designated from operator 32 by a user's finger or a stylus pen on the video data displayed on monitoring monitor 36. The coordinates conversion processing described above may be performed by signal processor 33, rather than by setting manager 39.
In addition, setting manager 39 stores and holds threshold values (specifically, a sound source detection threshold value (refer to the description below) used by sound source detector 34, and a first threshold value, a second threshold value, and a third threshold value used by output controller 35 when output controller 35 generates the sound source direction image) which are compared with sound pressure p of each pixel calculated by sound source detector 34 described below, in memory 39z. Here, sound pressure p is used as an example of a sound parameter regarding the sound source, and represents a magnitude or an absolute value of the magnitude of the sound collected by microphone array device MA, and is different from a volume which represents the magnitude of the sound output from loudspeaker device 37. The sound source detection threshold value, the first threshold value, the second threshold value and the third threshold value are values compared with the sound pressure of the voice generated in monitoring area 9, and, for example, are set to arbitrary values determined by the user to be suspicious abnormal sound or warning sound (that is, attention needed sound) while the level relationships of the sound pressure being defined. For example, the first threshold value>the second threshold value>the third threshold value the sound source detection threshold value. Here, the sound source is a sound source in a broad sense including not only the sound source actually outputting the sound but also a place where a sound as a result of propagation from the sound source can be heard. In addition, the number of threshold values set in advance is not limited to three, but may be one, two, or equal to or more than four.
Communicator 31 receives the image data transmitted from camera device CA and the voice data transmitted from microphone array device MA and outputs the data items to signal processor 33. In addition, when designation information (refer to description below) by the operation using the user's finger or a stylus pen is received from operator 32 via signal processor 33, communicator 31 transmits the designation information to camera device CA.
Operator 32 is a user interface (UI) for notifying signal processor 33 of the content of the user's input operation, and is configured to include pointing devices such as a mouse and a keyboard. In addition, operator 32 may be configured to include a touch panel or a touch pad which is arranged corresponding to a screen of monitoring monitor 36, and a touch input operation can be input by the user's finger or a stylus pen. When an arbitrary position on the data of voice heat map image MP (refer to
Memory 38 is configured to include, for example, a random access memory (RAM), and functions as a program memory for storing a program regulating the operation of directivity control device 30 and data used at the time of executing the program when directivity control device 30 operates, and as a work memory for temporarily storing and holding the data when directivity control device 30 operates. In addition, memory 38 stores voice heat map image MP.
Signal processor 33 is configured to include a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP), and performs control processing that totally manages the overall operation control of each unit of directivity control device 30, data input output processing between other units, data operation (calculation) processing, and data storage processing.
As described above, signal processor 33 may calculate the direction (that is, the directivity direction) from microphone array device MA toward the sound source position corresponding to the position designated by the user on the capture image data based on the coordinates of the position designated by the user on the capture image data captured by camera device CA using the coordinates conversion formula held in setting manager 39. In other words, signal processor 33 calculates the coordinates (θMAh, θMAN) of the directivity direction from the microphone array device MA toward the sound source position corresponding to the position appeared on the coordinates (X, Y) on the capture image data displayed on monitoring monitor 36.
Sound source detector 34 in signal processor 33 calculates the sound pressure for each pixel configuring the capture image data based on information relating to the size of each pixel configuring the capture image data captured by camera device CA and the voice data collected by microphone array device MA. In this way, sound pressure calculator 34 generates sound pressure map data in which the calculated value of the sound pressure is assigned to the corresponding position of the pixel for each pixel configuring the capture image data. The sound pressure map data is data in which the position of each pixel configuring the capture image data and the calculated value of the sound pressure at the position (that is, the pixel) are in association with each other. The description is made that sound pressure calculator 34 generates the sound pressure map data based on the calculated value of the sound pressure of each pixel. However, sound pressure calculator 34 may generate the sound pressure map data using an average value, a maximum value, or a minimum value of the calculated values of the sound pressure for each pixel block form from a predetermined number of pixels (for example, four pixels), without using the calculated value of the sound pressure for each pixel.
In addition, sound source detector 34 as an example of a sound source detector detects a sound source direction of the voice data based on the voice data collected by microphone array device MA. The sound source detection processing in sound source detector 34 is a known technology and the detailed description thereof will be omitted. For example, sound source detector 34 divides monitoring area 9 into a plurality of blocks, forms directivity of the voice to the direction from microphone array device MA toward the each block, and determines whether or not the voice exceeding the predetermined sound source detection threshold value is present in the voice of the directivity-formed direction, and then, estimates the sound source direction in monitoring area 9.
Sound source detector 34 determines whether or not the voice in the detected sound source direction is an abnormal sound or a warning sound based on a suspicious event in monitoring control system 10. The suspicious event includes a scream of an official in monitoring area 9, a forced opening or destruction of a door or a window provided in monitoring area 9, or the like when, for example, a suspicious person enters monitoring area 9. Specifically, sound source detector 34 determines whether or not the sound pressure of the voice in the sound source direction exceeds the predetermined sound source detection threshold value for determining the abnormal sound or a warning sound.
Output controller 35 in signal processor 33 controls the operations of monitoring monitor 36 and loudspeaker device 37, and outputs the capture image data transmitted from camera device CA to monitoring monitor 36 to be displayed, and outputs the voice data transmitted from microphone array device MA to loudspeaker device 37. In addition, output controller 35 may control the operations of turning on and off the lighting of lighting device 50.
In addition, output controller 35 performs the directivity forming processing on the voice data collected by microphone array device MA using the voice data collected by microphone array device MA and the coordinates (θMAh, θMAv) of the directivity direction calculated by signal processor 33 or setting manager 39, and then, performs the processing that emphasizes the voice data of the directivity direction.
In addition, output controller 35 generates the sound source direction image which is based on the comparison result between the calculated value of sound pressure p and the threshold values (specifically, the first threshold value, the second threshold value, and the third threshold value) using the sound pressure map data generated by sound source detector 34. The sound source direction image is an image indicating the presence of the sound source of the abnormal sound or the warning sound according to the comparison result between the calculated value of sound pressure p calculated by sound source detector 34 and the first threshold value, the second threshold value, and the third threshold value. Details of the sound source direction image based on the comparison result between the calculated value of sound pressure p and the first threshold value, the second threshold value, and the third threshold value will be described below with reference to
Furthermore, output controller 35 generates voice heat map image MP (for example, refer to
As an example of the voice output, loudspeaker device 37 outputs the voice data collected by microphone array device MA or the voice data collected by microphone array device MA and the directivity of which is formed by signal processor 33. Loudspeaker device 37 may be configured as a device separate from directivity control device 30.
Lighting device 50 is a device that can light a part or all of the predetermined space in monitoring area 9. Lighting device 50 has, for example, a swing mechanism, and when receiving the instruction of a lighting direction from microphone array device MA or directivity control device 30, turns the light lamp toward the lighting direction to start lighting. In addition, lighting device 50 may have a plurality of lighting lamps installed on a plurality of locations in monitoring area 9, and may select a lighting lamp that can light the lighting direction instructed from microphone array device MA or directivity control device 30 to start lighting. In addition, lighting device 50 may be a single lighting lamp that can light the space of the entirety of monitoring area 9, and may light the entirety of imaging area when receiving the lighting instruction from microphone array device MA or directivity control device 30. Furthermore, lighting device 50 may have a dimming function that can adjust an amount of illumination of the lighting in step wise, and may start the lighting by an initial value of the amount of illumination according to the lighting instruction from microphone array device MA or directivity control device 30 and then, may perform the increasing or decreasing the amount of illumination in a case where a next instruction is received. The light of the lighting lamp includes the white lights, LED lights, and the halogen lighting.
Recorder 60 is configured to include a controller (not illustrated) for controlling each processing such as data recording and a recorder (not illustrated) for storing the capture image data captured by camera device CA and the voice data collected by microphone array device MA in association with each other. In addition, the starting or the ending of data recording in recorder 60 is performed according to the instruction from microphone array device MA or directivity control device 30. In addition, recorder 60 may store the data of voice heat map image MP described above.
Amplifiers 231, 232, . . . , 23n amplify the voice signal collected by the plurality of microphones MA1, MA2, . . . , MAn. A/D converters 241, 242, 243, . . . , 24n convert the voice signal respectively amplified by amplifiers 231, 232, . . . , 23n into digital voice data.
CPU 251 performs overall control processing that totally manages the overall operation control of each unit in microphone array device MA, data input output processing between other units, data operation (calculation) processing, and data storage processing. In microphone array device MA, processors such as an MPU or a DSP may be provided instead of CPU 251.
CPU 251 inputs the voice data collected by the plurality of microphones MA1, MA2, . . . , MAn and converted by A/D converters 241, 242, 243, . . . , 24n, and performs voice output processing based on the voice data.
Encoder 281 encodes the voice data output from CPU 251 and generates a voice packet that can be transmitted via network NW. Communicator 291 transmits the voice data encoded by encoder 281 to directivity control device 30 via network NW.
In microphone array device MA, when communicator 291 receives designation information including the coordinates of the sound source direction where the abnormal sound or the warning sound is generated from directivity control device 30, CPU 251 transmits a lighting instruction for lighting device 50 to lighting device 50, and further, transmits an imaging start instruction for camera device CA to camera device CA. Furthermore, after receiving the imaging start instruction for camera device CA from communicator 291, CPU 251 of microphone array device MA transmits a recording start instruction for recorder 60 to recorder 60 from communicator 291 after the imaging direction of camera device CA is adjusted.
In the present embodiment, in a case where camera device CA is configured to include a plurality of fixed cameras, information that any one fixed camera is selected by directivity control device 30 according to the operation of the user (for example, monitoring person) is included in the designation information described above, and in this case, microphone array device MA transmits the imaging start instruction to the fixed camera selected by directivity control device 30. In this way, even in a case where camera device CA is configured to include, for example, a plurality of existing fixed cameras not an omnidirectional camera nor a PTZ camera, microphone array device MA can cause the optimal fixed camera selected by the user (for example, monitoring person) to appropriately image the situation in monitoring area 9 after the generation of the abnormal sound or the warning sound.
In the present embodiment, directivity control device 30 determines whether or not the voice collected by microphone array device MA is the abnormal sound or the warning sound, and further, estimates the sound source direction of the abnormal sound or the warning sound. However, microphone array device MA may determine whether or not the abnormal sound is generated and may estimate the sound source position thereof.
In
Sound waves from sound source 80 are incident on each microphone MA1, MA2, MA3 to MA(n−1), and MAn incorporated in microphone array device MA at a certain angle (angle of incidence=(90−θ0) degree). The angle of incidence θ may be a horizontal angle θMAh or may be a vertical angle θMAv to the directivity direction from the microphone array device MA toward the voice position.
Sound source 80 is, for example, a conversation of persons that are the subject of camera device CA present in the sound collection direction of microphone array device MA, and is present in the direction of predetermined angle θ to the surface of housing 21 of microphone array device MA. In addition, interval d between each microphone MA1, MA2, MA3, . . . , MA(n−1), and MAn is constant.
The sound wave output from sound source 80 initially arrives at microphone MA1 and collected, and next, arrives at microphone MA2 and collected, similarly collected by the microphones one after another, and finally arrives at microphone MAn and collected.
Microphone array device MA performs AD conversion on the analog voice data collected by each microphone MA1, MA2, MA3, . . . , MA(n−1), and MAn into the digital voice data using A/D converters 241, 242, 243 to 24(n−1), and 24n.
Furthermore, microphone array device MA gives delay time corresponding to the difference of arrival time to each microphone MA1, MA2, MA3, . . . , MA(n−1), and MAn using delayers 251, 252, 253 to 25(n−1), and 25n and aligns the phases of all the sound waves, and then, adds the delaying processed voice data using adder 261. In this way, microphone array device MA forms the directivity of the voice data of each microphone MA1, MA2, MA3, . . . , MA(n−1), and MAn in the direction of predetermined angle θ.
As described above, microphone array device MA can simply form the directivity of the collected voice data by changing delay times D1, D2, D3 to Dn−1, and Dn set to delayers 251, 252, 253 to 25(n−1), and 25n.
CPU 41 performs control processing that totally manages the overall operation control of each unit of camera device CA, data input output processing between other units, data operation (calculation) processing and data storage processing. In camera device CA, processors such as an MPU or a DSP may be provided instead of CPU 41.
Image sensor 45 is configured to include, for example, a complementary metal oxide semiconductor (CMOS) sensor, and acquires the image data by imaging the situation in monitoring area 9. A charge coupled device (CCD) sensor may be used instead of the CMOS sensor.
Memory 46 includes ROM 46z that stores the operation program and the data of the set value, RAM 46y that stores the image data and work data, and memory card 46x that is removably connected to camera device CA and stores various data items.
Communicator 42 is a network interface (I/F) that controls the data communications with network NW connected via network connector 47.
Power manager 44 supplies a DC power to each unit of camera device CA, or supplies a DC power to the devices connected to network NW via network connector 47. Network connector 47 transmits the communication data such as the image data, or is a connector that can supply the power via a network cable.
The monitoring operation in monitoring control system 10 will be described.
Microphone array device MA starts the sound collection operation when the power is turned on, and when the voice output request is received from directivity control device 30, repeatedly transmits the voice data to directivity control device 30 (T4).
When the voice data is received from microphone array device MA, sound source detector 34 of signal processor 33 in directivity control device 30 calculates the sound pressure of each pixel configuring the capture image data based on the information relating to the size of each pixel configuring the capture image data captured by camera device CA and voice data collected by microphone array device MA. In addition, sound source detector 34 generates sound pressure map data in which the calculated value of the sound pressure is assigned to the corresponding position of the pixel for each pixel configuring the capture image data. Output controller 35 generates the sound source direction image which is based on the comparison result between the calculated value of sound pressure p and the threshold values (specifically, the first threshold value, the second threshold value, and the third threshold value) using the sound pressure map data. Output controller 35 generates voice heat map image MP (for example, refer to
When the data of the voice heat map image is input, monitoring monitor 36 displays the data of voice heat map image MP (T6). In STEPs subsequent to STEP T6, since the data of the voice heat map image output from directivity control device 30 in STEP T5 is displayed at any time, the voice heat map image MP is displayed as the time-series video images.
Here, in a case where there is no sound source in the voice map data, which satisfies a predetermined condition (for example, the sound pressure exceeds the sound source detection threshold value), monitoring monitor 36 displays a video that reflects nothing (for example, an image which makes the screen black as it is) (T7). The sound source which satisfies the predetermined condition includes various sound sources such as a sound source that outputs a voice of high or low frequency range and a sound source that outputs a musical tone or a noise, not limited to a sound source of which the sound pressure exceeds the sound source detection threshold value.
An abnormal sound or a warning sound of which the sound pressure exceeds the sound source detection threshold value is generated (T8), microphone array device MA transmits the voice data including the abnormal sound or the warning sound to directivity control device 30 (T9). Directivity control device 30 obtains the direction of the sound source generating the abnormal sound or the warning sound, and outputs the data of voice heat map image MP including the sound source direction image of the abnormal sound to monitoring monitor 36 (T10). Monitoring monitor 36 displays voice heat map image MP in which the sound source direction image that indicates the sound source direction seen from microphone array device MA and sound pressure value of the sound source generating the abnormal sound or the warning sound is drawn in a display pattern different from another direction seen from microphone array device MA (T11). Voice heat map image MP is generated by output controller 35 of signal processor 33 of directivity control device 30.
In
Directivity control device 30 transmits the result of detecting the abnormal sound or the warning sound (that is, information relating to the sound source direction seen from microphone array device MA where the abnormal sound or the warning sound is generated) by the sound source detector 34 to microphone array device MA (T12). When the result of detecting the abnormal sound or the warning sound by sound source detector 34 transmitted from directivity control device 30 is received, microphone array device MA transmits the lighting instruction to lighting device 50 (T13). In the lighting instruction, the information relating to the direction of the sound source generating the abnormal sound or the warning sound seen from microphone array device MA is included. For example, lighting device 50 holds the information indicating the direction of the sound source seen from microphone array device MA and the information indicating which area lighting device 50 may light in association with each other, and thus, it is possible to turn on the light to the area of the direction of the sound source based on the information relating to the direction of the sound source generating the abnormal sound or the warning sound seen from microphone array device MA included in the lighting instruction described above.
When receiving the lighting instruction, lighting device 50 lights the space in monitoring area 9 including the sound source direction of the abnormal sound or the warning sound seen from lighting device 50 (T14). Here, the detection of the sound source direction where the abnormal sound or the warning sound is generated is performed by directivity control device 30. However, for example, microphone array device MA having the same configuration as sound source detector 34 may perform the detection instead of directivity control device 30. In addition, the lighting instruction to lighting device 50 may be performed by directivity control device 30 instead of microphone array device MA.
After performing the lighting instruction to lighting device 50, microphone array device MA transmits the imaging start instruction to camera device CA (T15). That is, microphone array device MA transmits the imaging start instruction to a direction where the sound source direction detected by sound source detector 34 in directivity control device 30 is replaced by the direction seen from camera device CA. The information relating to what direction seen from camera device CA is the direction of the sound source generating the abnormal sound or the warning sound seen from microphone array device MA is included in this imaging start instruction. The processing that the direction of the sound source generating the abnormal sound or the warning sound is converted into the direction seen from camera device CA may be performed by microphone array device MA or may be performed by setting manager 39 or signal processor 33 of directivity control device 30. Here, in a case where camera device CA is the PTZ camera, microphone array device MA instructs camera device CA to perform the panning, tilting, and zooming operation. In a case where camera device CA is configured to include a plurality of fixed cameras, microphone array device MA designates any one of the fixed camera. In addition, in a case where camera device CA is the omnidirectional camera, microphone array device MA transmits only the imaging start instruction. In this case, camera device CA may perform the panorama conversion on the circle-shaped image (that is, the omnidirectional image including the direction of 360°) captured by the omnidirectional camera according to the cut-out direction included in the imaging start instruction, and then, may generate the panorama image.
Camera device CA transmits the capture image data to monitoring monitor 36 and instructs to start displaying the capture image data (T17). Then, the image data captured by camera device CA is displayed on monitoring monitor 36. At the same time, camera device CA transmits the capture image data to recorder 60 (T18). Monitoring monitor 36 starts to display the image data captured by camera device CA (T19). In this case, the imaging area is lit by lighting device 50 and thus, the image data in the lit imaging area is projected on the screen of monitoring monitor 36. However, in this stage, since the direction of camera device CA is not adjusted (in other words, the optical axis is not facing the direction of the sound source generating the abnormal sound or the warning sound), a capture image of a direction different from the direction of the sound source generating the abnormal sound is projected on the screen of monitoring monitor 36.
After starting imaging, camera device CA adjusts the imaging direction to the sound source direction instructed from the microphone array device MA or directivity control device 30 (T20).
After elapse of the time it is determined that the imaging direction adjustment is finished, microphone array device MA instructs recorder 60 to start recording (T21). When receiving the recording start instruction, recorder 60 starts operating and start recording (T22). By starting recording after the adjustment of the imaging direction, the image data before the adjustment may not be recorded, and thus, it is possible to record only the effective image data.
When the predetermined time has elapsed after the imaging start instruction is given to camera device CA, microphone array device MA determines that the imaging direction adjustment is finished. However, in a case where there is a response signal (Ack) from camera device CA after waiting the response signal, it may be determined that the imaging direction adjustment is finished.
When camera device CA finishes the imaging direction adjustment, monitoring monitor 36 displays the image data of the sound source direction designated by microphone array device MA (T23). In this stage, the sound source direction of the abnormal sound or the warning sound collected by microphone array device MA and the imaging direction (that is, the optical axis direction) by camera device CA are coincident with other. On the screen of monitoring monitor 36, image data in which suspicious person hb is trying to break window wd is projected in the sound source direction where the abnormal sound or the warning sound is generated.
In addition, user hm designates an arbitrary position on monitoring monitor 36 which displays the image data, for example, via operator 32 of directivity control device 30 while viewing the image data displayed on monitoring monitor 36 (T24). In a case where monitoring monitor 36 is configured to have a touch panel, user hm may designate an arbitrary position on the image by directly touching the screen of monitoring monitor 36. Output controller 35 of signal processor 33 in directivity control device 30 forms a voice directivity to a direction (that is, a directivity direction) from microphone array device MA toward the sound source position corresponding to the position designated by user hm (T25), and transmits the voice emphasized in the directivity direction to loudspeaker device 37 (T26). Loudspeaker device 37 outputs the voice which is emphasized by output controller 35 in the directivity direction (T27). In this way, user hm can check the image data at the position where the abnormal sound or the warning sound is generated, and furthermore, can hear the voice in the sound source direction. Therefore, it is possible for user hm to thoroughly perform the monitoring using monitoring control system 10 according to the present embodiment.
In
Sound pressure p of these sound source directions are compared with three threshold values (the first threshold value>the second threshold value>the third threshold value) by output controller 35 described above, and the sound source direction images corresponding to each result of comparison are generated to be displayed in circle shapes. For example, in sound source direction image sd1, output controller 35 generates sound source direction image sd11 in which the calculated value of sound pressure p exceeds the first threshold value while drawing the image in red (display in dark dots). Output controller 35 generates sound source direction image sd12 in which the calculated value of sound pressure p is equal to or lower than the first threshold value and exceeds the second threshold value while drawing the image in yellow (display in normal dots). Output controller 35 generates sound source direction image sd13 in which the calculated value of sound pressure p is equal to or lower than the second threshold value and exceeds the third threshold value while drawing the image in blue (display in thin dots). The sound source direction image in which the calculated value of sound pressure p is equal to or lower than the third threshold value is not particularly drawn remaining transparency as it is, but may be drawn in a specific color. In this case, it can be explicitly understood that the sound pressure is low. In addition, the sound source direction image may be expressed in other colors, patterns, shadings, or the like.
In addition, for example, when any one of sound source direction images sd1 to sd4 determined to be the abnormal sound and displayed on the screen of monitoring monitor 36 is designated by an operation of touching the panel, the voice in that sound source direction is output from loudspeaker device 37. In this way, it is possible for user hm to hear the voice determined to be the abnormal sound or the warning sound in the nervous sound source direction, and to thoroughly perform the monitoring while viewing the image on monitoring area 9. As described above, in monitoring control system 10 according to the present embodiment, camera device CA images the situation in monitoring area 9 as an example of the imaging target area, and microphone array device MA collects the voices in monitoring area 9. Monitoring monitor 36 displays the image data (video data) captured by camera device CA. Sound source detector 34 in directivity control device 30 detects the sound source direction of the voice collected by microphone array device MA. Output controller 35 in directivity control device 30 generates the sound source direction image illustrating the sound source direction of the abnormal sound or the warning sound detected by sound source detector 34. In a case where sound pressure p of the voice collected by microphone array device MA exceeds the threshold value (that is, sound source detection threshold value), lighting device 50 lights the area in the sound source direction where the abnormal sound or the warning sound is generated among monitoring area 9 according to the instruction from microphone array device MA or directivity control device 30. Monitoring monitor 36 displays the sound source direction image generated by output controller 35 so as to be superimposed on the image (capture image) captured by camera device CA.
In this way, even in a case where monitoring area 9 is a dim place or a suspicious abnormal sound or a warning sound is generated at night, if sound pressure p calculated by sound source detector 34 exceeds at least one of the first threshold value to the third threshold value, monitoring control system 10 images the situation in monitoring area 9 using camera device CA after lighting device 50 lights monitoring area 9. Therefore, it is possible to make user hm (for example, the monitoring person) easily recognize and determine what the suspicious abnormal sound or the warning sound generated in monitoring area 9 is. In addition, in voice heat map image MP, sound source direction images sd1, sd2, sd3, and sd4 are displayed in a display pattern different from that in another image portion. Therefore, user hm can visually understand the sound source direction easily.
In addition, in monitoring control system 10 according to the present embodiment, the sound source direction images are displayed in a plurality of display patterns which are different from each other according to the signal level (for example, level of the sound pressure) of the abnormal sound or the warning sound. Here, sound source direction image sd11 in which the sound pressure exceeds the first threshold value is drawn in red. Sound source direction image sd12 in which the sound pressure is equal to or lower than the first threshold value and exceeds the second threshold value is drawn in yellow. Sound source direction image sd13 in which the sound pressure is equal to or lower than the second threshold value and exceeds the third threshold value is drawn in blue. Therefore, the level of the sound pressure generated from the sound source can roughly be understood. In this way, many information items relating to the sound source can be obtained.
In addition, in monitoring control system 10 according to the present embodiment, in a case where it is determined that the abnormal sound or the warning sound is generated, directivity control device 30 records the image data in which the area of the sound source direction of the abnormal image is captured, in recorder 60. In this way, in a case where the abnormal sound or the warning sound is generated, user hm can easily perform the operation of checking the sound source later where the abnormal sound or the warning sound is generated, and thus, it is possible to improve the efficiency of the monitoring business. In addition, in monitoring control system 10 according to the present embodiment, when sound source direction image sd displayed on monitoring monitor 36 is designated by the operation of user hm, directivity control device 30 converts the sound source direction indicated in sound source direction image sd into the coordinates (θMAh, θMAv) indicating the directivity direction from microphone array device MA (that is, the direction from microphone array device MA toward the actual sound source position corresponding to sound source direction image sd), and forms the directivity of the voice using the coordinates after the conversion processing, and then outputs the voice. In this way, user hm can hear the voice generated from the sound source to be focused. In this way, it is possible to thoroughly perform the monitoring.
In addition, in monitoring control system 10 according to the present embodiment, in a case where a plurality of abnormal sounds or the warning sounds is generated among the voices collected by microphone array device MA, output controller 35 of directivity control device 30 generates sound source direction images sd1, sd2, sd3, and sd4 different from each other for each abnormal sounds or the warning sounds. Monitoring monitor 36 displays sound source direction images sd1, sd2, sd3, and sd4 different from each other so as to be superimposed on the capture image captured by camera device CA. In this way, even in a case where a plurality of abnormal sounds or the warning sounds is generated in monitoring area 9, since sound source direction images sd1, sd2, sd3, and sd4 different from each other are superimposed on the capture image captured by camera device CA, user hm can visually and intuitively grasp the presence or absence of the sound source of the individual abnormal sound or the warning sound among the capture images captured by camera device CA, and thus, it is possible to improve the efficiency of the monitoring business of monitoring area 9.
As above, the embodiments are described with reference to the drawings. It is needless to say that the present disclosure is not limited to such examples. It is apparent that those skilled in the art can conceive various change examples or modification examples within the scope of the aspects of the disclosure, and it is understood that those examples can naturally belong to the technical scope of the present disclosure.
For example, in the embodiment described above, the directivity control device detects the sound source direction, calculates the sound pressure, and determines the abnormal sound of which the sound pressure exceeds the threshold value. However, the microphone array device may prepare those functions described above. In this way, it is possible to reduce the load of processing performed by the directivity control device.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2015-140863 | Jul 2015 | JP | national |
| 2015-145857 | Jul 2015 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4264927 | Raymond | Apr 1981 | A |
| 5790181 | Chahl et al. | Aug 1998 | A |
| 6043837 | Driscoll, Jr. et al. | Mar 2000 | A |
| 6157403 | Nagata | Dec 2000 | A |
| 7002617 | Smith | Feb 2006 | B1 |
| 7162043 | Sugiyama | Jan 2007 | B2 |
| 7260241 | Fukuhara et al. | Aug 2007 | B2 |
| 7298930 | Erol et al. | Nov 2007 | B1 |
| 7536029 | Choi et al. | May 2009 | B2 |
| 7893985 | Ahiska et al. | Feb 2011 | B1 |
| 8427538 | Ahiska | Apr 2013 | B2 |
| 8547416 | Ozawa | Oct 2013 | B2 |
| 9036001 | Chuang et al. | May 2015 | B2 |
| 9098910 | Hinkel | Aug 2015 | B2 |
| 9407851 | Kweon | Aug 2016 | B2 |
| 20010010555 | Driscoll, Jr. | Aug 2001 | A1 |
| 20020075295 | Stentz et al. | Jun 2002 | A1 |
| 20030160862 | Charlier et al. | Aug 2003 | A1 |
| 20030160868 | Kakou et al. | Aug 2003 | A1 |
| 20050140810 | Ozawa | Jun 2005 | A1 |
| 20060122841 | Shin et al. | Jun 2006 | A1 |
| 20070070190 | Yin et al. | Mar 2007 | A1 |
| 20080010060 | Asano et al. | Jan 2008 | A1 |
| 20080131092 | Takanezawa | Jun 2008 | A1 |
| 20090041378 | Yamaoka et al. | Feb 2009 | A1 |
| 20090185028 | Ogawa | Jul 2009 | A1 |
| 20100123785 | Chen et al. | May 2010 | A1 |
| 20100185308 | Yoshida et al. | Jul 2010 | A1 |
| 20110013075 | Kim et al. | Jan 2011 | A1 |
| 20110243543 | Pace et al. | Oct 2011 | A1 |
| 20120045149 | Arai et al. | Feb 2012 | A1 |
| 20120093339 | Wu | Apr 2012 | A1 |
| 20120124603 | Amada | May 2012 | A1 |
| 20120162259 | Sakai | Jun 2012 | A1 |
| 20120169842 | Chuang et al. | Jul 2012 | A1 |
| 20120218377 | Oku | Aug 2012 | A1 |
| 20120317594 | Thorn et al. | Dec 2012 | A1 |
| 20130321568 | Suzuki et al. | Dec 2013 | A1 |
| 20140085538 | Kaine et al. | Mar 2014 | A1 |
| 20140340513 | Nagata et al. | Nov 2014 | A1 |
| 20140376740 | Shigenaga et al. | Dec 2014 | A1 |
| 20150016641 | Ugur et al. | Jan 2015 | A1 |
| 20150095818 | Lee | Apr 2015 | A1 |
| 20150350621 | Sawa et al. | Dec 2015 | A1 |
| 20160065820 | Yamamoto et al. | Mar 2016 | A1 |
| 20160142620 | Sawa | May 2016 | A1 |
| 20160277863 | Cahill et al. | Sep 2016 | A1 |
| 20170132474 | Matsumoto | May 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 1968405 | May 2007 | CN |
| 101086741 | Dec 2007 | CN |
| 8-7199 | Jan 1996 | JP |
| 9-101197 | Apr 1997 | JP |
| 10-51889 | Feb 1998 | JP |
| 11-146295 | May 1999 | JP |
| 11-205772 | Jul 1999 | JP |
| 11-331827 | Nov 1999 | JP |
| 2000-196941 | Jul 2000 | JP |
| 2000-209689 | Jul 2000 | JP |
| 2003-008974 | Jan 2003 | JP |
| 2003-18561 | Jan 2003 | JP |
| 2003-189273 | Jul 2003 | JP |
| 2004-77277 | Mar 2004 | JP |
| 2004-180197 | Jun 2004 | JP |
| 2005-274707 | Oct 2005 | JP |
| 2006-003941 | Jan 2006 | JP |
| 2006-168421 | Jun 2006 | JP |
| 2006-174216 | Jun 2006 | JP |
| 2007-295335 | Nov 2007 | JP |
| 2007-300220 | Nov 2007 | JP |
| 2008-141703 | Jun 2008 | JP |
| 2008-193196 | Aug 2008 | JP |
| 2008-271157 | Nov 2008 | JP |
| 2009-118318 | May 2009 | JP |
| 2009-296232 | Dec 2009 | JP |
| 2010-26834 | Feb 2010 | JP |
| 2010-103773 | May 2010 | JP |
| 2010-187363 | Aug 2010 | JP |
| 2010-213091 | Sep 2010 | JP |
| 2010-232888 | Oct 2010 | JP |
| 2011-71686 | Apr 2011 | JP |
| 2011-188016 | Sep 2011 | JP |
| 2011-211675 | Oct 2011 | JP |
| 2012-105199 | May 2012 | JP |
| 2012-129873 | Jul 2012 | JP |
| 2012-133250 | Jul 2012 | JP |
| 2013-015468 | Jan 2013 | JP |
| 2013-106298 | May 2013 | JP |
| 2013-240000 | Nov 2013 | JP |
| 2014-137323 | Jul 2014 | JP |
| 2014-143678 | Aug 2014 | JP |
| 2014-212443 | Nov 2014 | JP |
| 2015-29241 | Feb 2015 | JP |
| 5939341 | Jun 2016 | JP |
| 5979458 | Aug 2016 | JP |
| 2017-92938 | May 2017 | JP |
| 2011114610 | Sep 2011 | WO |
| 2014125835 | Aug 2014 | WO |
| 2015162645 | Oct 2015 | WO |
| Entry |
|---|
| Busset et al., “Detection and tracking of drones using advanced acoustic cameras,” Proc. of SPIE vol. 9647 96470F-1, 2015, 8 pages. |
| Case et al., “Low-Cost Acoustic Array for Small UAV Detection and Tracking,” Wright State University, Dayton, Ohio, IEEE 2008, pp. 110-113. (4 pages). |
| Chinese Office Action and English translation of Chinese Search Report, dated Jun. 1, 2017, for Chinese Application No. 201380068876.3, 7 pages. |
| English translation of Chinese Search Report and Office Action, dated Dec. 31, 2019, for Chinese Application No. 201680062295.2, 26 pages. |
| Extended European Search Report, dated Sep. 23, 2015, for European Application No. 13868107.7-1902, 8 pages. |
| International Search Report and Written Opinion, dated Apr. 1, 2014, for International Application No. PCT/JP2013/007681, 9 pages. |
| International Search Report and Written Opinion, dated May 29, 2017, for International Application No. PCT/JP2017/005872, 16 pages. |
| International Search Report, dated Mar. 16, 2017, for International Application No. PCT/JP2016/004793, 6 pages. |
| Japanese Office Action, dated Jan. 19, 2016, for Japanese Application No. 2015-007242, 6 pages. (With English translation). |
| Japanese Office Action, dated Jul. 5, 2016, for Japanese Application No. 2013-252468, 8 pages. (With English translation). |
| Kloow, “Acoustic eraser used with the acoustic camera on disturbing sources,” Joint Baltic-Nordic Acoustics Meeting, Jun. 18-20, 2012, Odense, Denmark, 7 pages. |
| O'Donovan et al., “Imaging Concert Hall Acoustics Using Visual and Audio Cameras,” Perceptual Interfaces & Reality Lab, Computer Science & UMIACS, University of Maryland, College Park, IEEE 2008, pp. 5284-5287. (4 pages). |
| Number | Date | Country | |
|---|---|---|---|
| 20170019744 A1 | Jan 2017 | US |
