MONITORING MOTION USING SKELETON RECORDING DEVICES

Abstract

A system and a method for monitoring motion skeleton recording devices is described. The method includes detecting, by a processor of a monitoring system, at least one human skeleton in a field of view (FOV) of the first skeleton recording device. Based on the detection, a message is transmitted to rest of the plurality of skeleton recording devices to switch ON and OFF corresponding infrared (IR) sensors in a round robin manner. The method further includes identifying one or more second skeleton recording device based on a direction of traversal of the at least one human skeleton from the FOV of the first skeleton recording device to a FOV of the one or more second skeleton recording devices. Based on the identification, the one or more second skeleton recording devices are notified to activate the corresponding IR sensor.

Description

TECHNICAL FIELD

The present subject matter relates, in general, to motion detection and, in particular, to a system and a method for monitoring motion using a plurality of skeleton recording devices.

BACKGROUND

Skeleton recording devices are generally used for detecting movement of human skeletons, particularly individuals. In recent years, skeleton recording devices are being used increasingly for different purposes, such as in indoor surveillance. Such detection plays a decisive role in scenarios, for example, monitoring elderly people, identifying people in high security areas, remote tracking, and the like. Among other uses, the skeleton recording devices are used for identification of individuals based on biometrics to distinguish individuals based on their physical and/or behavioral characteristics, such as gait. Various skeleton recording devices are employed to recognize an individual based on the individual's style of walking. The skeleton recording devices may detect the motion of human skeletons through infrared (IR) sensors, camera systems, radio frequency, magnetic sensors, and the like.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanying figures. In the figures, the left-most digits of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the figures to reference like features and components. Some embodiments of systems and methods in accordance with embodiments of the present subject matter are now described, by way of example only, and with reference to the accompanying figures, in which:

FIG. 1 illustrates a network environment implementing a monitoring system, according to an embodiment of the present subject matter.

FIG. 2( a) illustrates a flowchart depicting a triggering mechanism employed by the monitoring system, according to an embodiment of the present subject matter.

FIG. 2( b) illustrates a flowchart depicting a compression technique of the monitoring system, according to an embodiment of the present subject matter.

FIG. 2( c) illustrates a bit pattern of a transmitted data block, according to an embodiment of the present subject matter.

FIG. 3( a) illustrates a graph indicating power consumption by two skeleton recording devices employed with the monitoring system, according to an embodiment of the present subject matter.

FIG. 3( b) illustrates a graph indicating change in statistical properties of skeleton recording devices due to compression techniques employed by the monitoring system, according to an embodiment of the present subject matter.

FIG. 3( c) depicts a graph indicating accuracy in identification of individuals due to compression technique employed by the monitoring system, according to an embodiment of the present subject matter.

DETAILED DESCRIPTION

Typically, skeleton recording devices are used for detecting a change in position of a human skeleton relative to its surroundings. Amongst other things, the skeleton recording devices may employ infrared (IR) sensors to determine movement of the human skeleton, such as a human skeleton. A skeleton recording device, such as a Kinect®, are used for recording features of the human skeleton. For example, in case of an individual, the Kinect® may track the individual over time to determine change in position of the individual. Further, the Kinect® is used to record skeleton joints of the individual. The skeleton joints may include head, shoulder centre, shoulder left, shoulder right, spine, and the like. Data recorded by the skeleton recording devices is further processed based on the requirements.

Generally, single skeleton recording device is used for detection and recording of data pertaining to a human skeleton. As the single skeleton recording device has limited viewing angle, natural movement pattern of the human skeleton may not be captured. Referring back to the example of the human skeleton, in case the human skeleton is partially or completely outside a Field Of View (FOV) of the Kinect®, the Kinect® based systems may fail to record data, such as skeleton joint data of the human skeleton. This may result in difficulty in capturing sufficient data of the human skeleton using a fixed Kinect®. In addition, when the surveillance area is huge, for example, when the Kinect® is deployed in a large hall, such as having a dimension of 20×20 sq. ft., the single Kinect® may be unable to cover the entire area and thus may nit be able to capture data pertaining to the human skeleton.

In the past, few attempts have been made in order to have a wider FOV. For example, a panoramic sensor for wide FOV may be used in the skeleton recording devices. However, the panoramic sensor does not facilitate in fine focusing due to their lower pixel density. Moreover, due to the wide FOV, a slight defocus can cause major negative impact in image quality of the panoramic sensor. To overcome the issues raised by the panoramic sensor, multiple skeleton recording devices may be deployed for recording data and tracking the human skeleton. Further, each of the multiple skeleton recording devices is connected to a processing device, such as a Kinect® controller to capture the data. However, the use of multiple skeleton recording devices may demand huge bandwidth while sending raw data from the skeleton recording devices to a server. Also, as surveillance systems are expected to operate 24 hours a day, overall power consumption of the surveillance system may increase by using multiple skeleton recording devices. In addition, installing multiple skeleton recording devices in close range may introduce infra red (IR) interference and may introduce noise in the human skeleton data, this may result in inaccuracies or errors in further processing of the human skeleton data, such as for identification of individuals.

In accordance with the present subject matter, a system and a method for monitoring motion using multiple skeleton recording devices is described. The system as described herein is a monitoring system. In an implementation of the present subject matter, multiple skeleton recording devices, such as Kinects®, are positioned, in an area of surveillance, in such a way that limitation of the FOV of a single Kinect® is compensated by other Kinects®. In other words, the Kinects® are placed in the area of surveillance with non-overlapping FOV such that the complete area of surveillance can be monitored. Further, the Kinects® are time synchronized with respect to a pre-defined time using a known Network Time Protocol (NTP). Therefore, in combination, the Kinects® provide a wider FOV. In an implementation, each Kinect® is connected to a monitoring system (also referred to as a Kinect® controller). Further, the monitoring system of a skeleton recording device may be connected to monitoring systems of other skeleton recording devices in the area of surveillance through wired or wireless network connections. Examples of such connections may include, but is not limited to, Ethernet and Wi-Fi.

In an implementation, each of the monitoring systems may ensure that the IR sensor of one skeleton recording device is turned ON at a time. For example, each skeleton recording device may be in a passive state or an active state. In the passive state, an IR sensor of the skeleton recording device is switched OFF, such as by the monitoring system associated with the skeleton recording device. The passive skeleton recording device thereafter waits for a signal from the monitoring system of an active skeleton recording device to become active. In the active state, the IR sensor of the skeleton recording device is turned ON and the IR sensor searches for an event to track an human skeleton, such as a human skeleton. The skeleton recording device for which the IR sensor is turned ON may be referred to as an active device (ACT_DEVICE). Further, the event may be understood as detection of an human skeleton in the FOV of the ACT_DEVICE. As mentioned above, when one skeleton recording device is active, the monitoring system of the ACT_DEVICE may transmit a message to the remaining monitoring systems to switch ON and switch OFF the IR sensors of their respective skeleton recording devices in a round robin manner. These skeleton recording devices may be referred to as passive devices (PASSV_DEVICE).

Initially, all the skeleton recording devices may be assigned a weight based on their probability of detecting the human skeleton. When no human skeleton is detected, every skeleton recording device is activated for a short but fixed duration that is proportional to the weight assigned to the skeleton recording device, in the round robin manner. Further, the monitoring system may control every skeleton recording device by two Boolean variables ‘Active’ and ‘Event’. The monitoring system may be configured to set values of the variables based on a state of the skeleton recording device. For example, when a skeleton recording device is activated, the monitoring system sets an ‘Active’ value to true and ‘Event’ value to false and may broadcast a message to other monitoring systems to set their ‘Active’ and ‘Event’ values to false, thereby indicating the IR sensors of other skeleton recording devices to remain off (in passive state). When the duration of the ACT_DEVICE is over, the ACT_DEVICE may switch state to passive by setting the ‘Active’ value to false and the monitoring system may activate the next skeleton recording device by sending a message. As the skeleton recording devices turn ON and OFF in a round robin manner, the chance of missing an event by all the skeleton recording devices reduces.

In a scenario, when the ACT_DEVICE detects the human skeleton in its FOV, the monitoring system associated with the ACT_DEVICE sets ‘Event’ value of all the skeleton recording devices to true and starts tracking the human skeleton until out of the FOV of the ACT_DEVICE. In this circumstance, all the PASSV_DEVICES wait for an activation signal from the ACT_DEVICE. Further, the monitoring system associated with the ACT_DEVICE may record the human skeleton data as well as track direction of traversal of the human skeleton by motion modeling techniques, such as Kalman filter. In an implementation, more than one human skeleton may be captured in the FOV of a Kinect®. The Kinect® may track each of the human skeletons in its FOV and extract skeleton data pertaining to the human skeletons. As may be appreciated, the present subject matter may employ other human motion modeling techniques, such as particle filtering techniques, for recording the skeleton data and tracking direction of the movement. When the human skeleton comes to an edge of the FOV of the ACT_DEVICE, the associated monitoring system predicts the next skeleton recording device using Kalman filtering and activates immediate next skeleton recording device (PASSV_DEVICE) according to the predicted direction of movement by sending a message. In case more than one human skeleton is detected in the FOV the ACT_DEVICE, the associated monitoring system broadcasts a message to one or more PASSV_DEVICE in whose FOV the human skeletons are moving. When the human skeleton goes out of view of ACT_DEVICE, the ACT_DEVICE becomes a part of the round robin, i.e., remaining Kinect® turn their IR cameras ON and OFF in the round robin manner. The above-described process goes on until an ACT_DEVICE finds its event value to be false, i.e., no human skeleton is detected by the ACT_DEVICE. The monitoring system of the present subject matter activates/deactivates the skeleton recording devices without any data loss.

In an implementation, each monitoring system may extract the skeleton data of the one or more human skeletons that appears in the FOV of the skeleton recording device associated with the monitoring system. In an example, the monitoring system may extract 3-Dimensional skeleton information of each of the individuals in the form of one or more skeleton frames. The monitoring system may store the skeleton information in a memory thereof for later use. Further, when the skeleton recording devices are in an idle state or the skeleton recording devices' memory is full, their respective monitoring system transfers the skeleton data, captured by the associated skeleton recording devices, to a backend server. In another example, the monitoring systems may transfer the skeleton data, captured by the associated skeleton recording devices, to the backend server at regular intervals. The monitoring system employs a compression technique for compressing the raw skeleton data, before transferring to the backend server, as will be discussed later with reference to the figures. The compression technique facilitates in reducing utilization of bandwidth during transfer of the skeleton data.

In an implementation, the compressed data may be shared with the backend server for various applications. One such application may include identification of individuals based on the compressed skeleton data received from the Kinect® controllers. In this case, the backend server may be trained to extract 3D skeleton joint coordinates, i.e., x, y, and z coordinates, of a plurality of skeleton joints of each individual. In one implementation, the plurality of joints of each individual may include head, shoulder centre, shoulder left, shoulder right, spine, hand left, hand right, elbow right, elbow left, wrist right, wrist left, hip left, hip right, hip centre, knee right, knee left, foot left, foot right, ankle right, and ankle left. The identification server thereafter obtains the skeleton data and then linearly shifts the skeleton data obtained from the human skeleton, captured by one Kinect®, with respect to the skeleton data obtained from the human skeleton captured by another Kinect®, so that the skeleton data can be mapped into a single co-ordinate system. After the mapping is done, the backend server may merge all the controller-inputs in time-interleaved fashion.

Based on the skeleton data, the backend server detects one or more gait cycles of each of the individuals. As described earlier, the backend server maps the skeleton data into a single co-ordinate system, therefore, although each Kinect® captures skeleton of an individual, only one continuous walking pattern is detected for the individual. For each of the one or more gait cycles of each of the known individuals, the backend server may extract a plurality of gait features of the individual, from the skeleton data. Examples of the gait features that are extracted, in accordance with one implementation, include area related features, such as mean of area occupied by an upper body portion and mean of area occupied by a lower body portion of the individual. Further, the gait features may include angle related features, such as mean, standard deviation and maximum of angle of the upper left leg relative to the vertical axis, angle of the lower left leg relative to the upper left leg, and angle of the left ankle relative to horizontal axis, and mean, standard deviation and maximum of angle of the upper right leg relative to the vertical axis, angle of the lower right leg relative to the upper right leg, and angle of the right ankle relative to horizontal axis.

Accordingly, the present subject matter employs multiple skeleton recording devices that work together as an integrated system to broaden the coverage area of surveillance. Further, each skeleton recording device is associated with a monitoring system, such as a Kinect® controller. The monitoring systems may switch the IR sensors of different skeleton recording devices in a round robin manner for optimum resource utilization. The monitoring system may facilitate in reducing an average power consumption of the motion sensing devices and may ensure that there is no data loss caused. The IR sensors of different skeleton recording devices may capture raw skeleton data of an human skeleton when the human skeleton falls within the FOV of the active skeleton recording device. This raw skeleton data may be compressed by using time series compression mechanism for transferring to the backend server. The monitoring system may transfer the compressed data to the backend server at regular intervals to reduce bandwidth utilization.

FIG. 1 illustrates a network environment 100 implementing a monitoring system 102, in accordance with an embodiment of the present subject matter. In one implementation, the network environment 100 can be a public network environment, including thousands of personal computers, laptops, various servers, such as blade servers, and other computing devices. In another implementation, the network environment 100 can be a private network environment with a limited number of computing devices, such as personal computers, servers, and laptops.

The monitoring system 102 may be implemented in a variety of systems, such as a Kinect® controller, an edge processing device, and the like. Further, it will be understood that the monitoring system 102 is connected to a plurality of skeleton recording devices 104-1, 104-2, . . . , and 104-N, collectively referred to as skeleton recording devices 104 and individually referred to as a skeleton recording device 104. In an implementation, the skeleton recording device 104 may be a Kinect®. As shown in FIG. 1, the skeleton recording device 104 may be communicatively coupled to the monitoring system 102 over a network 106 through one or more communication links for facilitating one or more end users to access and operate the monitoring system 102.

In one implementation, the network 106 may be a wireless network, a wired network, or a combination thereof. The network 106 may also be an individual network or a collection of many such individual networks, interconnected with each other and functioning as a single large network, e.g., the Internet or the Intranet. The network 106 may be implemented as one of the different types of networks, such as the Intranet, local area network (LAN), wide area network (WAN), the Internet, and such. The network 106 may either be a dedicated network or a shared network, which represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), etc., to communicate with each other. Further, the network 106 may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, and the like.

It is to be understood that the specific skeleton recording devices 104 shown in the FIG. 1 is only for the purpose of explanation. Any other conventionally known skeleton recording devices can be used for the purpose of recording data associated with the human skeletons without deviating from the scope of the invention. Further, the monitoring system 102 of the present subject matter is explained by considering the human skeletons as a human skeleton for identification of individuals, however, it may be evident to a person skilled in the art that the monitoring system 102 may be implemented in other applications. Further, the monitoring system 102 may be connected to a server 108, such as the backend server 108, through the network 106.

According to an implementation, the monitoring system 102 includes processor(s) 110, interface(s) 112, and memory 114 coupled to the processor(s) 110. The processor(s) 110 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) 110 may be configured to fetch and execute computer-readable instructions stored in the memory 114.

The memory 114 may include any computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM), and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes.

Further, the interface(s) 112 may include a variety of software and hardware interfaces, for example, interfaces for peripheral device(s), such as a product board, a mouse, an external memory, and a printer. Additionally, the interface(s) 112 may enable the monitoring system 102 to communicate with other devices, such as web servers and external repositories. The interface(s) 112 may also facilitate multiple communications within a wide variety of networks and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. For the purpose, the interface(s) 114 may include one or more ports.

The monitoring system 102 also includes module(s) 116 and data 118. The module(s) 116 include, for example, an activation module 120, a tracking module 122, a compression module 124, and other module(s) 126. The other modules 126 may include programs or coded instructions that supplement applications or functions performed by the monitoring system 102. The data 118 may be weights data 128, coordinates data 130, and other data 132. The other data 132, amongst other things, may serve as a repository for storing data that is processed, received, or generated as a result of the execution of one or more modules in the module(s) 116.

In one implementation, the activation module 120 may toggle the skeleton recording devices 104 between an active state and a passive state. Further, the activation module 120 may define state of every skeleton recording device 104 by two Boolean variables ‘Active’ and ‘Event’. The different states corresponding to the Boolean variables is shown in the below table. In an example, the skeleton recording devices 104, such as Kinects® may be placed side by side in a surveillance area with non overlapping FOV.

TABLE 1
Active	Event	Kinect State
False	False	IR sensor off and no event detected
False	True	IR sensor off but event detected by another skeleton
		recording device
True	False	IR sensor on but no event detected
True	True	IR sensor on and an event detected by this skeleton
		recording device

The skeleton recording device 104-1 in an active state may hereinafter be referred to as ACT_DEVICE and the skeleton recording device 104-2 in the passive state may hereinafter be referred to as PASSV_DEVICE. In an implementation, the activation module 120 may detect presence of one or more human skeletons in an area of surveillance. In an implementation, the area of surveillance employs a plurality of skeleton recording devices 104 and the activation module 120 may detect presence of the one or more human skeletons in a field of view (FOV) of a first skeleton recording device 104. Based on the detection, the activation module 120 may turn ON the IR sensor of the ACT_DEVICE and for tracking the one or more human skeletons. Further, the activation module 120 may send a notification to rest of the plurality of skeleton recording devices 104 to operate their respective IR sensors in the round robin manner. In an implementation, the activation module 120 may turn the IR sensor of PASSV_DEVICE to ON and OFF and instruct the PASSV_DEVICE to wait for signal from the ACT_DEVICE or till a human skeleton is detected by any one of the PASSV_DEVICE.

Further, probability of all the skeleton recording devices 104 to detect the human skeleton is not the same. For example, for an indoor surveillance area, the skeleton recording device 104 located near a main entrance door holds the maximum probability to detect the human skeleton, such as the human skeleton, first with respect to the back door. Similarly, the skeleton recording devices 104 located in between front and back door hold probability values in between these two of detecting the human skeleton. Based on the probability of detecting the human skeleton first, the activation module 120 may assign a weight to each of the skeleton recording device 104. Further, the activation module 120 may store the weights as weights data 128. In an implementation, when no event is detected by the skeleton recording devices 104, the activation module 120 may activate each skeleton recording device 104 for a short but fixed duration. For example, the activation module 120 may activate a timer of each of the skeleton recording devices 104 to the duration that is proportional to the weight assigned to each of the skeleton recording devices 104.

In an implementation, upon activation, the activation module 120 may set the ‘Active’ value to TRUE and ‘Event’ value as FALSE. When one skeleton recording device 104 is activated, the monitoring system 102 associated with the ACT_DEVICE 104 may set a timer of the ACT_DEVICE to infinity. In other words, the timer of the skeleton recording device 104 may remain active as long as the human skeleton lies in the FOV of the ACT_DEVICE 104. The activation module 120 may then broadcast a message to other skeleton recording devices 104 to notifying them to modify the states of the remaining skeleton recording devices 104 of ‘Active’ and ‘Event’ as FALSE, thereby indicating the other skeleton recording devices 104 to remain in the passive state. When the fixed duration is over, the activation module 120 may turn the ACT_DEVICE to the passive state by setting its ‘Active’ value to FALSE. Further, the activation module 120 may activate the next skeleton recording device 104 by sending a unicast message. Accordingly, in the absence of any event, as the skeleton recording devices 104, such as Kinects®, turn ON and OFF in a round robin manner, the chance of missing an event by all the skeleton recording devices 104 is very less.

In an implementation, when any of the ACT_DEVICE detects at least one human skeleton, the activation module 120 may set ‘Event’ value of all the skeleton recording devices 104 as TRUE. Further, the tracking module 122 may start tracking the human skeletons until it is visible. The tracking module 122 may record information as well as may track direction of the human skeleton movement. In an example, the skeleton recording device 104 may record skeleton information and track the human skeletons by human motion modeling using Kalman filter. Further, when any of the human skeletons come to one of the edges of the FOV of the ACT_DEVICE, the tracking module 122 may identify one or more second skeleton recording devices 104 based on a direction of traversal of the human skeletons. In an implementation, the tracking module 122 may track the human skeletons and when the human skeletons move from the FOV of the first skeleton recording device 104 to the FOV of the one or more second skeleton recording devices 104, the tracking module 122 may activate the next skeleton recording devices 104, i.e., PASSV_DEVICE according to the predicted direction of movement by sending a message. Further, when the human skeletons goes out of view of ACT_DEVICE, the ACT_DEVICE becomes passive, i.e., the ‘Active’ value is set to FALSE. This process goes on until the ACT_DEVICE finds its event value to be false.

Considering an example when one human skeleton is detected by the skeleton recording device 104. In order to track a position of the human skeleton for the skeleton recording device 104, the tracking module 122 may confirm (x, y, z) coordinates of the human skeleton and its direction of movement. The tracking module 122 may store the coordinates of the human skeleton as coordinates data 130. The tracking module 122 may club this information together to form a state variable vector (x). The state variable vector (x) may indicate the movement of the human skeleton, in this case the human skeleton, in every frame so that the skeleton recording device 104 may be switched to another one when the human skeleton comes to the edge of the FOV. In an example, in case of detection of multiple human skeletons, another state variable may be formed by the tracking module 122 for clubbing information about each human skeleton. The IR sensor of the ACT_DEVICE may also capture how far the human skeleton has travelled from its initial position. As raw data obtained from skeleton recording device 104 is noisy, the state variable vector may include a component of error. Further, the tracking module 122 may use a state estimation technique, such as Kalman filter, that considers the noise component for state estimation by integrating noisy measurements (uncertainty). Accordingly, for each of the human skeletons being tracked, the tracking module 122 may compute centroid of each human skeleton to form a single view world model in 3D world coordinate system. This spatiotemporal model represents x and y position and velocity of body centroid (i.e., a 4D state vector (x,y,{umlaut over (x)},ÿ)) together with uncertainty measurement. The tracking module 122 may model transition relation for any human skeleton using equation (1):

$\begin{array}{cc}x\left(t+\Delta t\right)=F\left(\Delta t\right)·x\left(t\right)+\Delta tv\left(t\right)& \left(1\right)\\ F\left(\Delta t\right)=\left(\begin{array}{cccc}1& 0& \Delta t& 0\\ 0& 1& 0& \Delta t\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right)& \left(2\right)\end{array}$

where Δt is the transitional time from the last updated position. v(t) is Gaussian noise present in the measurement and given state vector estimate at time instance t−Δt, namely {umlaut over (x)}(t−Δt|t−Δt) and measurement uncertainty factor M(t-Δt|t−Δt), the predicted estimation of state vector and associated uncertainty for time instance t is given by:

{umlaut over (x)}(t|t−Δt)=F(Δt)·{umlaut over (x)}(t−Δt|t−Δt)

M(t|t−Δt)=F(Δt)M(t−Δt|t−Δt)F^T(Δt)+|Δt|N_x(t)  (3)

Now given an observation y(t), predicted state vector {umlaut over (x)}(t|t−Δt) and associated uncertainties M(t|t−Δt) and N_y(t), the location of {umlaut over (x)}(t|t−Δt) may be predicted by Kalman filter using equation 4, as

{umlaut over (x)}(t|t)={umlaut over (x)}(t−Δt)+K(t)[y(t)−h(t,{umlaut over (x)}(t|t−Δt))]

M(t|t)=M(t−Δt)−K(t)H(t)M(t|t−Δt)  (4)

Where h(t, x) is non linear. Further, Kalman gain is represented by equation (5):

$\begin{array}{cc}K\left(t\right)=M\left(t-\Delta t\right){H^T\left(t\right)\left[H\left(t\right)M\left(t-\Delta t\right)H^T\left(t\right)+N_x\left(t\right)\right]}^{-1}& \left(5\right)\\ \mathrm{and}& \\ H\left(\Delta t\right)=\left(\begin{array}{cccc}\frac{\delta j}{\delta x}& \frac{\delta j}{\delta y}& 0& 0\\ \frac{\delta i}{\delta x}& \frac{\delta i}{\delta y}& 0& 0\end{array}\right)& \left(6\right)\end{array}$

The tracking module 122 of the monitoring system 102 may receive human skeleton data pertaining to the human skeletons. In an implementation, the human skeleton data pertaining to plurality of joints of the human skeleton is tracked by each of the plurality of skeleton recording devices 104. In case of the human skeleton, a plurality of skeleton joints of the human skeleton may include head, shoulder, centre, shoulder left, shoulder right, spine, and the like. To effectively utilize the bandwidth between the skeleton recording devices 104 and the backend server 108, the compression module 124 may apply lossy compression technique. The lossy compression may be understood as a JPEG compression. The compression module 124 may apply the lossy compression technique to compress each of the x, y, z coordinates of the human skeleton joints. For example, the compression module 124 may apply the lossy compression technique to compress each of the 20 skeleton joints (total of 60 time series data) per skeleton. The compressed data may later be used for people identification. As this compression is lossy, the data length of the compressed data is lesser than a length of actual data. The lossy compression may therefore facilitate in reducing the bandwidth for transmitting the skeleton data to the backend server 108. The compression algorithm ensures that the statistical properties, such as mean and standard deviation, of the skeleton data are preserved after the skeleton data is compressed. The extent of the preservation of the statistical property is governed by the accuracy of the people identification. The different steps involved in the compression algorithm will be explained in conjunction with FIG. 2(b).

In an implementation, the monitoring system 102 may thereafter share the human skeleton data with the backend server 108 for further processing. In an example, the backend server 108 may process the compressed data received from the monitoring system 102 for identification of the individual. In this case, the backend server 108 may be trained for identifying individuals. During training, the backend server 108 may receive an input from a user, say an administrator. The input may include a total number of individuals and their respective unique identifiers. In one example, a unique identifier of an individual may be a contact number, date of birth, and the like. According to an example, the user may provide a list of the individuals and their respective unique identifiers to the backend server 108 in a specific order.

For identifying the individuals, the backend server 108 may receive compressed data from all monitoring systems 102. Thereafter, the backend server 108 may extract 3D skeleton joint coordinates and aggregate the 3D skeleton joint coordinates in accordance with timestamps associated with the 3D skeleton joint coordinates received from each of the monitoring systems 102. In an implementation, the timestamp information contains the time when the human skeleton gets first detected in a skeleton recording device 104. As mentioned earlier, the backend server 108 may aggregate the 3D skeleton joint coordinates in time interleaving mode. The aggregation of the 3D skeleton joint coordinates may facilitate in obtaining a complete gait cycle that may be captured partially by two skeleton recording devices 104.

Accordingly, the monitoring system 102 may interact with multiple skeleton recording devices 104 that work together as an integrated system to broaden the coverage area of surveillance. Further, each skeleton recording device 104 is associated with a monitoring system 102, such as a Kinect® controller. The monitoring system 102 may use Kalman filtering for switching ON and OFF the IR sensors of different skeleton recording devices 104 for optimum resource utilization. The triggering mechanism facilitates in reducing average power consumption of the monitoring systems 102 and ensures that there is no data loss caused. The IR sensors of different skeleton recording devices 104 may capture raw human skeleton data of one or more human skeletons, when the human skeletons fall within the FOV of the ACT_DEVICE. This raw skeleton data may be compressed by using a time series compression mechanism for transferring to the backend server 108. The compression mechanism facilitates in preserving the statistical properties, such as mean and standard deviation, of the skeleton data. The monitoring system 102 may transfer the compressed data to the backend server 108 at regular intervals to reduce bandwidth utilization.

FIG. 2( a) illustrates a flowchart depicting a method 200 employed by the monitoring system 102 for monitoring multiple skeleton recording devices 104, according to an embodiment of the present subject matter. FIG. 2(b) illustrates a flowchart depicting a method 250 employed by the monitoring system 102 for compressing data, in accordance with an embodiment of the present subject matter. The methods 200 and 250 may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, human skeletons, components, data structures, procedures, modules, functions, etc., that perform particular functions or implement particular abstract data types. The methods 200 and 250 may also be practiced in a distributed computing environment where functions are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, computer executable instructions may be located in both local and remote computer storage media, including memory storage devices.

The order in which the methods 200 and 250 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the methods 200 and 250, or alternative methods. Additionally, individual blocks may be deleted from the methods without departing from the spirit and scope of the subject matter described herein. Furthermore, the methods 200 and 250 can be implemented in any suitable hardware, software, firmware, or combination thereof.

Referring to FIG. 2(a), at block 202, the method 200 may include receiving a message from an active skeleton recording device (ACT_DEVICE) 104, such as a Kinect®, to turn on an Infrared (IR) sensor thereof. In an implementation, the activation module 120 of the active skeleton recording device 104 may send the message to a passive skeleton recording device (PASSV_DEVICE). The activation module 120 may switch multiple skeleton recording devices 104 between an active state and a passive state to reduce overall power consumption by the skeleton recording devices 104. As may be understood, the message may be received by the PASSV_DEVICE, to become the ACT_DEVICE.

At block 204, the method 200 may include determining occurrence of an event. The event may be understood as detecting presence of one or more human skeleton in the FOV of a first skeleton recording device 104 (also referred interchangeable as the ACT_DEVICE). In an implementation, the ACT_DEVICE may search in its FOV for the one or more human skeletons. Once a human skeleton is detected within the FOV of the ACT_DEVICE, the ACT_DEVICE may extract information from the human skeleton.

At block 206, the method 200 may include determining whether the human skeleton was detected by the ACT_DEVICE or not. If not, the method 200 moves to block 208. If at least one human skeleton is identified, the method 200 moves to block 212.

At block 208, the method 200 may include identifying whether the IR sensor of the ACT_DEVICE has timed out or not. In an implementation, if no human skeleton is detected, the IR sensors of the ACT_DEVICE may turn OFF after a fixed duration of time as may be defined through the activation module 120. If the IR sensor of the ACT_DEVICE has not timed out, the method 200 goes back to block 204 for detecting the human skeleton.

If the IR sensor of the ACT_DEVICE has timed out without detecting the human skeleton, the method 200 moves to block 210. At block 210, the activation module 120 may turn OFF the IR sensor of the ACT_DEVICE, i.e., the ACT_DEVICE now goes in an idle or passive state and a message may be sent to a PASSV_DEVICE to turn on the IR sensors.

As shown in block 212, the method 200 may include extracting human skeleton data from the one or more human skeletons. The ACT_DEVICE may store the human skeleton data for later reference. In an implementation, when the human skeletons are identified by the ACT_DEVICE, the activation module 120 may set the timer of the IR sensor to infinity.

Thereafter, at block 214, the method 200 may include tracking direction of traversal of the human skeletons. In an implementation, the tracking module 122 may track the direction of motion, such that the tracking module 122 may determine the next skeleton recording device 104 in whose FOV the human skeletons are moving.

At block 216, the method 200 may include determining if any of the human skeleton is out of view of the ACT_DEVICE or not. In an implementation, the tracking module 122 may check whether any of the human skeletons has gone out of the FOV of the ACT_DEVICE or not. If the human skeletons are not out of view, the method 200 may move to block 218.

At block 218, the method 200 may include setting the timer of the IR sensor of the ACT_DEVICE to infinity. In an implementation, the tracking module 122 may set the timer to infinity to ensure that the IR sensor does not turn OFF when the human skeletons are within its FOV.

Further, if all of the human skeletons has moved out of view of the ACT_DEVICE, the method 200 may move to block 220. At block 220, the method 200 may include turning OFF the IR camera of the ACT_DEVICE. In an implementation, the tracking module 122 may turn OFF the IR sensors of the ACT_DEVICE when all of the human skeletons move out of view of the ACT_DEVICE. Further, the tracking module 122 may send a message to one or more next skeleton recording devices 104 for activating corresponding IR camera.

At block 220, the method 200 may include compressing the human skeleton data and sharing with the backend server 108. In an implementation, the compression module 124 may compress the human skeleton data by using time series compression technique.

Referring to FIG. 2(b) a flowchart 250 depicting a compression mechanism employed in the monitoring system 102 is illustrated, according to an embodiment of the present subject matter.

At block 252, the method 250 may include splitting an input data stream into equal length blocks. In an implementation, the compression module 124 may split the input data stream.

Further, at block 254, the method 250 may include shifting each block to zero mean, normalizing between −1 and +1, and multiplying with 127 for obtaining 8 bit signed representation.

As depicted in block 256, the method 250 may include performing Discrete Chebyshev Transform (DCT) on each block. In an implementation, the DCT may result in a set of numbers (c) called as Chebyshev coefficients, as indicated in equation 7, as:

$\begin{array}{cc}C\left(i\right)=\frac{2}{N}\sum _{m=1}^Nf_mT_{\mathrm{im}}& \left(7\right)\end{array}$

Where c(i) is the i^th Chebyshev coefficient of data block f. N is the length of the input data block and T_im is a cosine lookup table. The DCT converts the time series data into spectral information so that the information exists in a quantitative form that can be manipulated for compression by the compression module 124.

At block 258, the method 250 may include omitting the first component of the Chebyshev coefficient. In an implementation, as the data block is set to zero mean, the first component of Chebyshev coefficient is always zero and therefore, the compression module 124 may omit the first component. For the remaining components, the compression module 124 may identify all Chebyshev coefficients whose absolute values are below a threshold (Th). For such components, the compression module 124 may set the coefficient value to zero.

$\begin{array}{cc}\begin{array}{c}{c}^{\prime }\left(i\right)=c\left(i+1\right),c\left(i+1\right)>\mathrm{Th},\forall i=1\dots N-1\\ =0\mathrm{for}\mathrm{the}\mathrm{rest}\end{array}& \left(8\right)\end{array}$

As shown in block 260, the method 250 may include quantizing non-zero coefficients and transmitting in the subsequent images. In an implementation, the compression module 124 may quantize the non-zero coefficients by rounding mantissa and exponent of the binary form of retained coefficients with the sign bit and shifting the radix point to the leading ‘1’. The fraction part is rounded off upto first 3 bits. The quantized number is finally represented by placing four bits of the exponent, followed by the sign bit and the three bits of the fraction (a total of 8 bits).

At block 262, the method 250 may include generating a control word (cw) that represents location of the coefficients which are being transmitted. In an implementation, the compression module 124 may generate the cw for each data block as:

$\begin{array}{cc}\begin{array}{c}\mathrm{cw}\left(k\right)=0,\mathrm{if}c^{\prime \left(k\right)}=0,\forall k=1\dots N-1\\ =1\mathrm{otherwise}\end{array}& \left(9\right)\end{array}$

Considering that the length of original control word be M. The compression module 124 may truncate the control word after its last ‘1’. Further, one more bit is saved from transmitting by removing the trailing ‘1’ of the control word. Thus the number of transmitting coefficient is more than the length of truncated control word. In an implementation, a log₂M bits long length specifier represents the length of the truncated control word. For example, for a certain data block of length 16, let a 15 bit control word is ‘110100111000000’. Truncating after the last ‘1’ control word becomes ‘110100111’. Again after truncating the trailing ‘1’, it becomes ‘11010011’. Accordingly, the length of the final control word is 8 and the length specifier in 4 (log₂16=4) bits is ‘1000’.

At block 264, a data block is encoded. In an implementation, the compression module 124 may encode the data block as depicted in FIG. 2(c). As shown in the figure, first 11 bits represent the mean of the data block, followed by 6 bit length specifier and the control word represented by length specifier. These are followed by the quantized coefficient.

Further, at block 266, the method 250 may include determining whether there are any data blocks left for encoding. If yes, the method 250 may move back to block 252 for repeating the above-mentioned process. If no, the method 250 may move to block 268.

At block 2686, the method 250 may include arranging data blocks in packets for transmission. In an implementation, the compression module 124 may arrange the data blocks together and may also append timestamp information to the data packet in 16 bit format.

Referring to FIGS. 3(a), 3(b), and 3(c), graphs 300-1, 300-2, and 300-3 indicating experimental results of the monitoring system 102 are illustrated, according to an embodiment of the present subject matter. FIG. 3(a) illustrates the graph 300-1 indicating power consumption by two skeleton recording devices, such as skeleton recording device 104-1 and skeleton recording device 104-2 being employed in a surveillance area. The two skeleton recording devices 104 may be placed next to each other. As described with reference to FIG. 1, the monitoring systems 102, such as the Kinect® controllers, associated with the motion sensing devices 104 incorporate the triggering mechanism to reduce power consumption by the skeleton recording devices 104. In an implementation, both skeleton recording devices 104 were set for the equal duration of their active states, i.e., 1 minute. In the present implementation, total duration of the test was set for around 7 minutes. An individual walks in front of the skeleton recording devices 104 during the test.

In an implementation, the monitoring system 102 may measure instantaneous power consumption of the skeleton recording devices 104. As indicated in the graph 300-1, instantaneous power drawn by the two skeleton recording devices 104 during the test. The shaded regions 302 in the graph 300-1 indicate the instances when the skeleton recording devices 104 are active and tracking a skeleton. Further, state of the skeleton recording devices 104 at different time interval during the test is indicated in the below table:

Instance	Skeleton recording device 1	Skeleton recording device 2
(seconds)	status	status
0-5	Delay to start the application	Delay to start the application
6-65	Active, but no human skeleton	Passive
	in view
126-135	Passive as time-out	Active, but no human skeleton
		in view
136-190	Active and tracking an human	Passive
	skeleton
191-220	Passive, as the human	Active, but no human skeleton
	skeleton is out of view	in view yet
221-305	Passive	Human skeleton in view do
		remains active until it is
		visible
306-320	Data compression and posting	Data compression and posting
	to a backend server	to a backend server
321-380	Active again but no human	Passive
	skeleton in view
381-440	Passive as time-out	Active, but no human skeleton
		in view

As indicated in graph 300-1, the power consumption of the skeleton recording devices 104 reach their maximum values, such as upto 7.3 watt, when they are tracking the human skeleton and are very close to zero when the skeleton recording devices 104 are passive (PASSV_DEVICE). The skeleton recording devices 104 consume about 5.5 watt of power in average when the skeleton recording devices 104 are active but tracks nothing, as indicated by unshaded high amplitude regions 304. The maximum power consumption scenario occurs when one skeleton recording devices 104 is active and tracking an event, and the other skeleton recording devices 104 is passive. In such scenarios, the state of art systems, the power consumed by the two skeleton recording devices 104 can reach upto 13.1 Watts. The monitoring system 102 of the present subject matter facilitates the skeleton recording devices 104 to consume power of about 8.0 Watts, which is about 40% less than the conventional skeleton recording devices.

FIG. 3( b) illustrates the graph 300-2 indicating change in statistical properties of the skeleton recording devices 104, such as skeleton recording device 104-1 and skeleton recording device 104-2 due to the compression mechanism employed by the monitoring system 102, according to an embodiment of the present subject matter. The graph 300-2 includes a sample time signal containing all 60 3D coordinate time series human skeleton data appended to each other for analyzing the change in its statistical properties due to compression. The percentage (%) error in mean and standard deviation between reconstructed data and original data for different threshold values (1 to 5) is plotted against compression ratio. The compression ratio may be indicative of the ratio of the bit length of the actual data to the compressed data.

As the monitoring system 102 of the present subject matter applies a lossy compression mechanism, the data length of the compressed data is lesser than the actual data. The lossy compression mechanism ensures that the statistical properties, such as mean and standard deviation, of the skeleton data is preserved after the compression. In an implementation, a threshold value is defined for the applied on the DCT coefficients of the input data block for selecting a part of coefficients for further transmission. The amount of data loss during compression is controlled by the threshold value (Th) as shown in equation (8). For example, a higher threshold value indicates more compression ratio, however higher threshold value degrades performance of the people identification in terms of accuracy. Hence the “Th” is chosen such that the extent of preservation of statistical properties of the compressed data enables in achieving the desired people identification accuracy. Accordingly, the compression ratio increases with increased threshold values. As may be seen from the graph 300-2, although the error in mean values is not affected much, the error in standard deviation increases with the increase in compression. At 6 times of compression, the error in standard deviation is close to 1.5%, accordingly the monitoring system 102 reduces error and brings effectiveness in the compression.

Referring to FIG. 3(c), the graph 300-3 indicating accuracy in identification of individuals due to compression mechanism employed by the monitoring system 102 is depicted in accordance with an embodiment of the present subject matter. The accuracy in detection of an individual is tested on both raw skeleton data as well as on the compressed skeleton data for performance comparison. In an example, skeleton data is recorded for 10 individuals by aggregating two Kinects® in triggering mode. The performance evaluation in terms of recognition accuracy for different compression ratio is done based on the F1 score. The F1 score may be understood as a harmonic mean of precision (P) and the recall (R), and is represented as:

The graph 300-3 indicates average F1 score obtained in person detection against average compression ratio for different threshold values, ranging from 1 to 5, for 5 and 10 subjects respectively. In an implementation, compression ratio 1 indicates the data with no compression. An increase in threshold value subsequently increases the compression ratio. Without compression the recognition accuracy is around 0.7 for 10 persons, whereas due to compression, the accuracy varies between 0.66 to 0.72. As mentioned with reference to FIG. 2(b), the noisy data of Kinects® gets partially cleaned due to the quantization process of the compression mechanism.

Considering that the uncompressed skeleton data is D and the compressed skeleton data is D_comp and P is the threshold value. Then difference in the mean values of the compressed skeleton data and the uncompressed skeleton data is given by: Diff_mean=absolute(mean(D)−mean(D_compressed))˜0 (nearly equals to zero). Further, difference in standard deviation between the two values is represented by: Diff_std=absolute(std(D)−std(D_compressed))˜0 (nearly equals to zero). As there is less variation in the statistical properties, such as mean and standard deviation, in the compressed and the uncompressed skeleton data, the monitoring system 102 of the present subject matter preserves the statistical properties of the skeleton data and at the same time reduces bandwidth usage.

Although embodiments for methods and systems for monitoring motion using skeleton recording devices have been described in a language specific to structural features and/or methods, it is to be understood that the invention is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as exemplary embodiments for monitoring motion using skeleton recording devices.

Claims

1. A method for monitoring motion using a plurality of skeleton recording devices, the method comprising: detecting, by a processor of a monitoring system, at least one human skeleton in a field of view (FOV) of a first skeleton recording device from the plurality of skeleton recording devices, wherein each of the plurality of skeleton recording devices is connected with a separate monitoring device;based on the detection, transmitting, by the processor, a message to rest of the plurality of skeleton recording devices to switch ON and OFF corresponding infrared (IR) sensors in a round robin manner;identifying, by the processor, one or more second skeleton recording devices based on a direction of traversal of the at least one human skeleton from the FOV of the first skeleton recording device to a FOV of the one or more second skeleton recording devices; andbased on the identification, notifying, by the processor, the one or more second skeleton recording device to activate the corresponding IR sensors.

2. The method as claimed in claim 1 further comprising: extracting, by the processor, the skeleton data tracked by the first skeleton recording device; andcompressing, by the processor, the skeleton data of the skeleton recording device, wherein the compressed skeleton data is analyzed for identification of individuals.

3. The method as claimed in claim 2 further comprising transmitting, by the processor, the compressed skeleton data to a backend server, wherein the identification of individuals is performed at the backend server.

4. The method as claimed in claim 2, wherein the identification of individuals is performed at the monitoring system.

5. The method as claimed in claim 2, wherein the skeleton data is compressed by a lossy compression technique, wherein the lossy compression technique preserves statistical properties of the skeleton data for performing people identification with pre-defined accuracy.

6. The method as claimed in claim 5, wherein the lossy compression technique comprises performing Discrete Chebyshev Transform (DCT) on the skeleton data.

7. The method as claimed in claim 2, wherein the identification of individuals comprises: retrieving three dimensional (3D) skeleton joint coordinates from the skeleton data;aggregating the 3D skeleton joint coordinates in accordance with timestamps in time interleaving mode for obtaining a natural walking pattern of an individual;extracting a plurality of gait features of the individual, for each of the one or more gait cycles, from the 3D skeleton joint coordinates on the skeleton data; andidentifying the individual based on the plurality of gait features.

8. The method as claimed in claim 1, wherein each of the plurality of skeleton recording devices is associated with a weight based on a probability of detection of the at least one human skeleton by each of the plurality of skeleton recording devices.

9. The method as claimed in claim 8, wherein each of the plurality of skeleton recording devices remain active for a pre-defined time period in absence of the detection of the at least one human skeleton, and wherein the time period is defined in accordance to the weight assigned to each of the plurality of skeleton recording devices.

10. The method as claimed in claim 1, wherein the tracking of the at least one human skeleton is performed by a Kalman filter technique.

11. A monitoring system comprising: a processor;an activation module, coupled to the processor, to, detect presence of at least one human skeleton in a field of view (FOV) of a first skeleton recording device from a plurality of skeleton recording devices; anda tracking module, coupled to the processor,track the at least one human skeleton within the FOV of the first skeleton recording device, wherein the tracking includes determining a direction of traversal of the at least one human skeleton and extracting skeleton data from the at least one human skeleton; andbased on the determination, notify one or more skeleton recording devices of the rest of the plurality of skeleton recording devices to monitor the at least one human skeleton.

12. The monitoring system as claimed in claim 11 further comprising a compression module, coupled to the processor, to, compress the skeleton data by utilizing a lossy compression technique;transmit the compressed data to a backend server for identification of individuals.

13. The monitoring system as claimed in claim 11, wherein the activation module further transmits a notification to rest of the plurality of skeleton recording devices to switch ON and OFF corresponding infrared (IR) sensors in a round robin manner.

14. The monitoring system as claimed in claim 11, wherein each of the plurality of skeleton recording devices remain active for a pre-defined time period in absence of the detection of the at least one human skeleton, and wherein the time period is defined in accordance to the weight assigned to each of the plurality of skeleton recording devices.

15. A non-transitory computer-readable medium having embodied thereon a computer program for executing a method comprising: detecting, by a sensor of a first skeleton recording device, at least one human skeleton in a field of view (FOV) of the first skeleton recording device, wherein based on the detection, a message is transmitted to rest of the plurality of skeleton recording devices to switch ON and OFF corresponding infrared (IR) sensors in a round robin manner;extracting, by the processor, skeleton data pertaining to the at least one human skeleton tracked by the first skeleton recording device, wherein the skeleton data is extracted by the sensor of the skeleton recording device;identifying, by the processor, one or more second skeleton recording devices based on a direction of traversal of the at least one human skeleton from the FOV of the first skeleton recording device to a FOV of the one or more second skeleton recording devices; andbased on the identification, notifying, by the processor, the one or more second skeleton recording devices for monitoring the at least one human skeleton.

16. The method as claimed in claim 15 further comprising: extracting, by the processor, the skeleton data tracked by the first skeleton recording device; andcompressing, by the processor, the skeleton data of the skeleton recording device, wherein the compressed skeleton data is analyzed for identification of individuals.

17. The method as claimed in claim 16 further comprising transmitting, by the processor, the compressed skeleton data to a backend server, wherein the identification of individuals is performed at the backend server.