Patent Application
20190215183
This disclosure relates to a smart building that includes sensors and computing resources for accurately identifying a project team conducting a meeting in the smart building, and presenting the project team with relevant information related to their project.
Smart buildings include an installed network of sensors, information capturing devices (e.g., video cameras), and computing resources for operating building systems in a more efficient manner by automating certain tasks based on information obtained by the sensors. Smart buildings are also able to provide personalized interactions with users within the smart building based on recognizing individual users. It follows that the computing resources may provide advanced features such as facial recognition, smart learning controls of building systems to promote efficient utilization of the building systems, as well as predictive analysis to promote various automation features within the smart building.
As in any other building utilized for enterprise purposes, project teams of resources may gather in a smart building. Strategies for taking advantage of the installed layout of sensors and other connected devices within a smart building are disclosed for advancing efficiency and productivity of project teams running to accomplish enterprise projects within a smart building.
In order to take advantage of the benefits offered by a smart building, systems and methods are described that enable seamless identification of a project team comprised of occupants gathering within the smart building. The disclosure describes a project team identification tool (PTI tool) running on a computer device for managing, at least a part, of the smart building resources. The PTI tool may be comprised of software, hardware, and/or circuitry for implementing the features described herein. The PTI tool utilizes media components (e.g., video cameras, microphones) and sensors (e.g., motion sensors) installed throughout the smart building environment, to detect individual persons and groups of people gathered together within a specific area (e.g., meeting room, conference room, open space, or other defined area) in the smart building. After detecting the meeting event (that is, a gathering of people appearing to be together for a business meeting), the PTI tool references employee, or other known user, profile information to identify each of the detected people (constituting the gathering). The PTI tool is further configured to predict a project team the group of identified people belong to, as well as one or more projects associated with the predicted project teams. The PTI tool utilizes the advanced technology offered by the smart building to provide a unique solution for seamlessly identifying a project team of people meeting within the smart building, whether the meeting is previously scheduled (e.g., scheduled on a calendar system) or unscheduled (e.g., random ad-hoc meeting). Although reference is made to an employee throughout this disclosure, the employee may be representative of any person known, and/or identifiable, by a smart building. For example, a known user may include a contractor hired to be part of a project team to accomplish a specific project. The known user may also include a client agent to an enterprise organization, where the client agent is included on a project team to supervise and observe the other employee project team members.
An enterprise may include various predetermined groupings of people within its organization structure. For example, employees may be profiled into specific fixed teams according to their work function (e.g., accounting fixed team, human resources fixed team, finance fixed team, legal fixed team). Whereas the fixed team attribute is a long-term team, an employee may further be grouped into shorter term project teams. Project teams are comprised of a specific team of employees tasked with completing a specific project task for the enterprise. An employee may be included on one or more project teams at the same time.
A project team will often gather in groups to hold meetings within the smart building. To enhance efficiency and productivity of these meetings, the PTI tool tracks team members as they travel throughout the smart building, identifies when the team members are together as a group, and seamlessly presents fresh and relevant project materials to the team members and/or presentation devices that are being used. This technological improvement utilizes the smart components within the smart building to more efficiently (e.g., improved speed, data handling, and resource utilization, less or no interaction required) retrieve project materials and present them to team members when the team members are detected to be meeting.
The smart building 100 includes, for example, a plurality of smart building components such as HVAC (heating, ventilation, and air-conditioning) systems that are controlled based on feedback obtained from various smart sensors installed in the smart building 100. The smart building 100 also includes video cameras for capturing (e.g., recording and/or real-time streaming) occurrences within the smart building. Image recognition processes may be applied to the recorded video images to interpret information about the occurrences that are depicted in the recorded video images. The smart building 100 also includes microphones for capturing (e.g., recording and/or real-time streaming) sounds occurring in the smart building 100. Voice recognition processes (e.g., natural language processing) may be applied to the captured audio to recognize spoken language identified on the recorded audio. The smart building 100 also includes motion detectors for detecting movement and controlling systems within the smart building 100 (e.g., HVAC temperature settings) based on detected movement. The smart building 100 also includes various smart sensors for measuring characteristics within the smart building 100. The measured characteristics from the smart sensors are references to control various systems within the smart building to operate the smart building more efficiently. The smart building 100 also includes lighting controls that are controlled based on, for example, detected sensor measurements to be more efficient. The smart building also includes a network of connected, smart displays which can display content served by an integrated content management system (CMS) and according to an identified project team and its state.
Included in the room layout 120 is a motion detector 124, a video camera 125, and a microphone 126. The motion detector 124 may be an optical, microwave, or acoustic based sensor for detection movement within the room layout 120. The video camera 125 captures digital video of the room layout. The video camera 125 may also include depth/distance measuring components for measuring a distance between objects and/or mapping a three-dimensional image. The microphone 126 captures sounds occurring within the room layout. The motion detector 124, the video camera 125, and the microphone 126 may together comprise an identification unit 127. Based on information obtained from the identification unit 127, the PTI tool may detect three distinct groups of people within the room layout 120. A first group 121 includes three people, a second group 122 includes two people, and a third group 123 includes five people standing in front of a smart display 127. The PTI tool may apply the specific strategies described herein to determine which of these groups of people are predicted to be a project group, and select one or more projects associated with the predicted project group.
The GUIs 205 and the I/O interface circuitry 206 may include touch sensitive displays, voice or facial recognition inputs, buttons, switches, speakers and other user interface elements. Additional examples of the I/O interface circuitry 206 includes microphones, video and still image cameras, headset and microphone input/output jacks, Universal Serial Bus (USB) connectors, memory card slots, and other types of inputs. The I/O interface circuitry 206 may further include magnetic or optical media interfaces (e.g., a CDROM or DVD drive), serial and parallel bus interfaces, and keyboard and mouse interfaces.
The communication interfaces 208 may include wireless transmitters and receivers (“transceivers”) 210 and any antennas 212 used by the circuitry of the transceivers 210. The transceivers 210 and antennas 214 may support WiFi network communications, for instance, under any version of IEEE 802.11, e.g., 802.11n or 802.11ac, or other wireless protocols such as Bluetooth, Wi-Fi, WLAN, cellular (4G, LTE/A). The communication interfaces 208 may also include serial interfaces, such as universal serial bus (USB), serial ATA, IEEE 1394, lighting port, I2C, slimBus, or other serial interfaces. The communication interfaces 208 may also include wireline transceivers 214 to support wired communication protocols. The wireline transceivers 214 may provide physical layer interfaces for any of a wide range of communication protocols, such as any type of Ethernet, Gigabit Ethernet, optical networking protocols, data over cable service interface specification (DOCSIS), digital subscriber line (DSL), Synchronous Optical Network (SONET), or other protocol.
The communication interfaces 208 is shown to be in communication with other smart buildings through the network 130. Although three other smart buildings are shown to be in communication with the present smart building 100, the communication interfaces 208 enables the central command computer 110 to communicate with any number of other smart buildings. The communication interfaces 208 further enables the PTI tool running on the central command computer 110 to communicate with enterprise databases 240 through the network 130. The enterprise databases 240 includes a projects database 241 storing profile information on active, and in some embodiments also historical, enterprise projects that have assigned project teams of employees. Each project profile may include project attribute information. The project attribute information may include one or more of the following: project name, project description, critical dates, current status/state, meeting history, project sponsors, project manager(s), project team members, project team synthesis, as well as other project descriptive information. Each project profile may also include a topic signature for the projects, where the topic signature includes a storage of historical communications related to the project (e.g., emails related to the project, attachment documents to electronic correspondences related to the project, documents related to the project). The information included in the project profile (e.g., the topic signature alone or with other information in the project profile) may be utilized as a type of domain or dictionary of terms and documents that may later be referenced when analyzing a current project team meeting in the smart building 100.
The enterprise databases 240 also includes a schedule database 242 that includes the schedules for enterprise employees. The employee scheduling information may include information obtained from employee electronic calendar, as well as scheduling information such as paid time off and sick leave information obtained from human resources. The enterprise databases 240 also includes an employee database 243 that includes employee profiles describing employee attributes. The enterprise databases 240 also includes a historical knowledge database 244 that stores performance information from iterations of the PTI tool that may be referenced during subsequent iterations of the PTI tool to improve a performance of the PTI tool.
The system circuitry 202 may include any combination of hardware, software, firmware, or other circuitry. The system circuitry 202 may be implemented, for example, with one or more systems on a chip (SoC), application specific integrated circuits (ASIC), microprocessors, discrete analog and digital circuits, and other circuitry. The system circuitry 202 may implement any desired functionality of the PTI tool. As just one example, the system circuitry 202 may include one or more instruction processor 216 and memory 220.
The memory 220 stores, for example, control instructions 223 for executing the features of the PTI tool, as well as an operating system 221. In one implementation, the processor 216 executes the control instructions 223 and the operating system 221 to carry out any desired functionality for the PTI tool. For example, the control instructions 223 for the PTI tool includes a facial recognition module, a voice recognition module, an employee recognition module, and a project recognition module. Each module may include the instructional logic for implementing the associated recognition features of the PTI tool. The memory also includes control parameters 222 that provide and specify configuration and operating options for the control instructions 223, operating system 221, and other functionality of the central command computer 110.
Initially, the PTI tool references sensors within the room layout 120 to detect activity that can be attributed to a presence within the room layout 120 (301). For example, the motion detector 124 may be referenced to determine whether a movement is detected in the room layout 120. If the motion detector 124 detects movement, this triggers the PTI tool to further investigate the activity. Other triggering events may be detected to determine an activity has occurred in the room layout 120. For example, the microphone may detect a sound from the room layout 120.
To further investigate the detected activity, the PTI tool compares the detected activity with known, pre-configured, patterns (302). The PTI tool then determines whether the detected activity is relevant for further processing based on the results of the comparison (303). For example, the movement detection may be compared against a pre-configured pattern that describes a relevant activity as a grouping of people (e.g., at least a predetermined number of people positioned within a predetermined distance from a next person in the proposed group), a movement lasting for a predetermined length of time, and/or a movement covering a predetermined distance. When an audio recording is being analyzed, the PTI tool may process the audio recording through a natural language processing (NLP) engine 330 to extract spoken words from the audio recording. The extracted words may then be compared against a pre-configured pattern of words to determine whether the activity is relevant.
When the detected activity is determined to not be a relevant activity, the flow diagram 300 returns to the start of the process. When the detected activity is determined to be a relevant activity, facial recognition components are activated (304). This includes the video camera 125, and a facial recognition engine 340. The PTI tool initiates a scan of the room layout 120 by controlling the video camera 125 to record a video recording of the room layout 120 (305).
The PTI tool runs the facial recognition engine 340 on the video recording to determine a grouping of people in the room layout 120, and a number of people in the grouping of people (306). When the PTI tool determines not enough people are detected in the grouping of people, the flow diagram 300 returns to the start of the process.
When the PTI tool determines enough people are detected in the grouping of people, the flow diagram 300 proceeds to identify the people in the detected group (307). For example, a predetermined number (e.g., four people) may determine whether enough people are included from the detected group. In this case, the third group 123 in the room layout 120 is the only detected group to include more than four people. When identifying the people in the detected group, the PTI tool applies the facial recognition engine 340 to the video images of the detected group to identify each detected person from the detected group (307). The facial recognition engine 340 may compare the faces detected from the video images of the detected group against employee headshots included in employee profiles stored on the employee database 243, and/or against other images of known employees processed from video images of other previously occurring meetings. The PTI tool may also apply the NLP engine 330 to compare voices captured in audio recordings of the room layout 120 against employee voice recordings included in employee profiles stored on the employee database 243, and/or against other audio files of voices of known employees processed from audio captured from other previously occurring meetings. The PTI tool may also reference real-time information captured from other information capturing devices (e.g., video cameras, microphones), to determine whether a person being considered to be the detected person from the detected group has been identified at another location within the smart building 100. If the person being considered is identified at another location within the smart building 100, the probability that they are the person being identified in the room layout 120 is low to none.
By applying the facial recognition engine 340 and the NLP engine 330, an identification, or non-identification, of each person included in the detected group is accomplished by the PTI tool. Each predicted identification of a person may be done with a calculated confidence level, such that the final prediction is a selection of an employee predicted to be the detected person with a highest confidence level considering all of the referenced factors. The identification of the people also confirms an employment status of the corresponding person. Any new information relied upon to identify employees may be stored as new employee identifying attributes in the employee's profile stored on the employee database 243, or stored on historical knowledge database 244 for subsequent reference when identifying employees.
After identifying the people from the detected group, the PTI tool determines whether at least a predetermined number of employees are identified from the detected group (308). By identifying at least the predetermined number of employees (as opposed to non-employees) from the detected group, the PTI tool is predicting with an acceptable level of confidence that the detected group is a project team for the enterprise. When the identified number of employees is less than the predetermined number of employees, the flow diagram 300 returns to the start of the process.
When the identified number of employees is greater than or equal to the predetermined number of employees, the PTI tool proceeds to further analyze the detected group (309). The detected group may be further analyzed to determine a positional arrangement of the people in the detected group, where certain predetermined arrangement of people are assigned a higher probability of being a project team (e.g., people seating around a conference table). The detected group may also be further analyzed to calculate the ratio of employees to non-employees, where a higher ratio of employees is assigned a higher probability of being a project team. The speech captured from the detected group may also be analyzed to determine a relevance to known project teams. For example, the NLP engine 330 may recognize specific terminology from the captured speech (e.g., industry-standard terminology, internal terms, or other key topic terms), and create a localized topic signature for the detected group. The PTI tool may then compare the localized topic signature with the topic signatures included in the project profiles stored on the projects database 241 to determine one or more projects sharing an overlap of information. A larger number of overlapping information may increase a relevance score for a project considered for matching to the detected group.
As continued in
The PTI tool queries the projects database 241 to select active projects described as being assigned to a project team comprised of team members comparable to the members identified in the current project team (311). When such projects are found, the PTI tool may return a shortlist of active projects from the projects database 241 that are assigned to a project team having at least some similarity to the current project team (e.g., minimum number of overlapping team members) (312). For example, projects from the projects database 241 being assigned to project teams sharing at least four out of the five same people as the current project team may be included in the shortlist returned by the PTI tool. The PTI tool may also consider the overlap of information from the comparison of topic signature to localized topic signatures from step (309), where projects having a larger relevance score from the comparison have a higher probability of being selected for the shortlist (e.g., top×predetermined number of projects with the highest relevance scores are included in the shortlist).
When projects that include team members having an acceptable level of similarity to the current project team cannot be found in the projects database 241 as reflected by an empty shortlist, the flow diagram 300 returns the process to the start. However, when at least one project that includes team members having an acceptable level of similarity to the current project team is found in the projects database 241 and included on the shortlist, the overlap of team members between the current project team and the projects selected onto the shortlist are analyzed (313). Missing team members, different team members, as well as extra team members are analyzed and considered for each of the projects on the shortlist. The more similar the project teams are to the current project team, the higher the relevance score for the corresponding project on the shortlist.
To further account for any differences, the PTI tool accesses the schedule database 242 to determine whether any of the projects included in the shortlist have scheduled team meeting times that overlap with the time and/or location the current project team was identified as having the meeting (314). When overlaps are found, the relevance score for the corresponding project on the shortlist is increased (e.g., the overlap in meeting times increases the probability the project being considered from the shortlist matches up to the current project team identified as having the meeting).
To further account for any differences, the PTI tool accesses the employee database 243 to determine whether any of the projects included in the shortlist have team members that are schedule to be out of the office (e.g., paid time off, holiday, sick leave, medical reason, jury duty) on the dates/times the current project team was recorded meeting (315). When the scheduled time offs are found to account for gaps from missing team members, the relevance score for the corresponding project on the shortlist is increased (e.g., the scheduled time off information indicates the missing team member is scheduled for time off, which accounts for why the missing team member was not present when the current project team was identified as having the meeting, so the missing team member should not adversely affect the prediction). The employee database 243 (or the schedule database 242) may further include a scheduling calendar for known users (e.g., employees). When the scheduling calendar accounts for the absence of certain missing team members, the relevance score for the corresponding project on the shortlist is increased (e.g., the scheduling calendar indicates the missing team member is scheduled to be at another meeting, which accounts for why the missing team member was not present when the current project team was identified as having the meeting, so the missing team member should not adversely affect the prediction).
To further account for any differences, the PTI tool accesses the projects database 241 to determine historical meeting times and places for the projects included on the shortlist, to identify overlaps with the time and/or location the current project team was recorded meeting (316). When overlaps are found, the relevance score for the corresponding project on the shortlist is increased.
To further account for any differences, the PTI tool accesses the historical knowledge database 244 to determine team meeting patterns based on the historical meeting times and places for the projects included on the shortlist, to identify overlaps with the time and/or location the current project team was recorded meeting (317). When overlaps are found, the relevance score for the corresponding project on the shortlist is increased.
The PTI tool updates an overall relevance score for each of the projects included on the shortlist in view of the considerations from steps (314-317), to determine their relevance to the current project team (318). The PTI tool sorts the projects included on the shortlist in an order of highest relevance to lowest relevance (319). Each of the projects on the shortlist are then analyzed to determine a confidence level for single selection that would present the project with the highest relevance score to the current project team (i.e., the third group 123) in the room layout 120 (320). For example, the confidence level may be calculated based on whether the highest rated project according to its relevance score exceeds a predetermined threshold. In addition to the relevance score, the confidence level may also consider other factors determined throughout the process described by flow diagram 300, such as the confidence level with which each detected person from the detected group was earlier identified.
When the determined confidence level exceeds the predetermined threshold, the project with the highest relevance score is presented directly to the current project team as a single selection (323). The project presentation may include timely, fresh, and/or relevant content for the project, as obtained from any one of the enterprise databases 240. The project presentation materials may be presented on the smart display 127 known to be in the vicinity of the detected third group 123.
However, when the confidence for single selection does not exceed the predetermined threshold, a selection GUI is first presented to the current project team to select a project from the shortlist (321). According to some embodiments, the selection GUI may be replaced, and/or supplemented by an audio UI that outputs voice requests, and receives voice commands from the current project team. After the selection input is received (322), the PTI tool proceeds with the presentation of the selected project (323). The presentation may further include personalized, contextual, and/or fresh content for the project (324).
In addition to the presentation of project content, the PTI tool may further control operation of one or more smart building systems based on the selected project from the shortlist. For example, the selected project may include a presentation having specific playback guidelines that require the lighting to be set to specific levels. Other systems such as the temperature setting and media controls (e.g., video recording) may also be controlled based on the selected project and/or identified team members from the current project team.
To illustrate the initial process for determining the overlap between the people in the detected group and known project groups at step (311), the following relationship algorithm is provided:
Then the identified team members from the detected group, {E1, E5, E9, E609}, are compared to the active project teams, Project A and Project C, in the first table 610 to determine an overlap of team members. A second table 620 illustrates the resulting overlap from the comparison. The overlap of team members between the detected group, {E1, E5, E9, E609}, and Project A is 4/9 members=44.4%. The overlap of team members between the detected group, {E1, E5, E9, E609}, and Project C is 4/12 members=33.3%.
As described at step (315) above, the PTI tool may access the employee database 243 to determine scheduled time off for employees. So, if the PTI tool determines that employees {E11, E15, E19, E45, E141, E415} are currently out of office due to, for example, paid time off, the PTI tool removes these employees from the list of employees in each project team that are available for consideration. For example, Project A includes employees E11 and E45 that are now known to be out of the office, and therefore these two employees are excused from consideration from Project A, which results in the more accurate overlap calculation of 4/7=57.1%. For Project C, this new information on employees that are known to be out of the office results in six employees being removed from consideration in Project C, and the overlap comparison is updated to be 4/6=66.6%. So by referencing additional information related to known employees that are out of the office, the PTI tool is able to make a more accurate assessment and adjust the relevance scores accordingly. Based on the additional information related to known employees that are out of the office, now Project C becomes the most probable project for the detected group, {E1, E5, E9, E609}.
According to another exemplary scenario, the PTI tool may determine that the detected group in the room layout 120 is attending a monthly leadership meeting based on context determined from NLP applied to captured voice from the detected group, which may be further supplemented by employee profile information identifying the detected group, {E1, E5, E9, E609}, as being employees having an employee status eligible for attending the monthly leadership meeting. By recognizing the monthly leadership is occurring, materials for the monthly leadership meeting may be presented, for example, on the smart display 127 without prompting from any of the group members.
Recognizing the monthly leadership is occurring also allows the PTI tool to remove (i.e., excuse) non-leadership level employees from being present, which may result in a fewer number of eligible team members for consideration. Consider a third table 630 illustrating a scenario where {E1, E5, E9, E609, E45, E1235, E1901} are recognized to be leadership level employees, while the remaining employees are recognized not to be leadership level employees. Then, for Project A the PTI tool still identifies four overlapping team members with the detected group, {E1, E5, E9, E609}, but instead of counting nine total eligible team members in Project A, the PTI tool only counts seven because the PTI tool recognizes that E1235 and E1901 are not leadership level employees and would not be in the meeting anyways. Therefore, the adjusted overlap ratio become 4/7=57.1%. By the same adjustment, the PTI tool adjusts the overlapping ratio for Project FR to 3/7=42.8%.
The overlap ratio may be further adjusted in view of additional relevant information obtained by the PTI tool from the smart building 100 that accounts for additional missing employees from the detected group. For example, a fourth table 640 reflects a scenario where the PTI tool obtains information that employees E1235 and E1901 are detected by the smart building 100 to be sitting in another meeting, and thus could not physically join the current detected group, employees E1235 and E1901 may be removed from consideration, and the overlap ratio would again go up for Project A and Project C to 4/5=80%. The overlap ratio, either directly or a weighted version thereof, may be utilized as a relevance score.
Various implementations have been specifically described. However, other implementations that include a fewer, or greater, number of features for each of the apparatuses, methods, or other embodiments described herein are also possible.
