Patent Application
20240135930
This disclosure relates generally to user interaction (e.g., control) with one or more interactive targets in an enclosure. The interactive targets can comprise an optically switchable device (e.g., tintable window in a building), projected media, environmental appliance, sensor, or any other apparatus that is communicatively coupled to a communication network in an enclosure.
The ability to control and/or customize environmental conditions within at least a portion of the enclosure is gaining increased popularity, as well as deployment and manipulation of related apparatuses such as seniors, emitters, and/or devices that affect the environment. Controlling and/or customizing the environment may be with the aim to increase comfort of occupant(s) and/or to reduce power consumption and improving the efficiency of systems controlling the environment of the enclosure (e.g., heater, cooler, vent, and/or lighting). A user's control and/or customization of the environment, however, traditionally entails manual adjustment of devices using physical user interfaces.
Various aspects disclosed herein alleviate as least part of the shortcomings and/or aspirations related to controlling an environment of a facility. Various embodiments herein relate to methods, systems, software and networks for automatic control of one or more environmental aspects. Various embodiments disclosed herein relate to identifying a user using external behavior recognition. Environmental customizations (e.g., specific to the user) can be implemented based on the user's identity, and voice and/or gesture controls can be provided to the user based on the user's identity.
In another aspect, a method for external behavior recognition for control of an environment of a facility, the method comprises: capturing, with an imaging system of the facility, a plurality of successive images of a user of the facility; obtaining, from the plurality of successive images, external behavior data of the user, wherein the external behavior data is representative of one or more physical actions taken by the user during the capturing of the plurality of successive images; determining an identity of the user based at least in part on the external behavior data of the user; and implementing environment customizations associated with the identity of the user.
In some embodiments, the one or more physical actions taken by the user comprises a gait of the user. In some embodiments, the imaging system comprises a camera, an infrared (IR) camera, a lidar sensor, or an imaging radar system. In some embodiments, the imaging system comprises the camera or the IR camera, and wherein the plurality of successive images comprise video frames. In some embodiments, the imaging system is operatively coupled with a controller. In some embodiments, obtaining the external behavior data of the user comprises extracting a respective pose of the user from each image of the plurality of successive images. In some embodiments, determining the identity of the user based at least in part on the external behavior data of the user comprises determining a unique identifier associated with the user. In some embodiments, implementing environment customizations comprises controlling an environmental aspect using one or more building systems. In some embodiments, the one or more building systems comprises a device ensemble having a housing that encloses the one or more devices that comprise: (i) sensors, (ii) a transceiver, or (iii) a sensor and an emitter. In some embodiments, the device ensemble is disposed in a fixture of the facility, or is attached to a fixture of the facility. In some embodiments, the one or more building systems comprise a tintable window. In some embodiments, the tintable window comprises an electrochromic window. In some embodiments, the device ensemble comprises the imaging system. In some embodiments, implementing the environment customizations comprises adjusting a temperature, window tint, and/or lighting within the facility. In some embodiments, capturing the plurality of successive images of the user is responsive to a triggering event. In some embodiments, the triggering event comprises detection of the user at a location of the facility. In some embodiments, determining the identity of the user is further based on sensor information regarding the user. In some embodiments, the sensor information comprises information indicative of a sound made by the user, dimensions of the user, and/or biometric information regarding the user. In some embodiments, determining the identity of the user is further based on one or more device inputs received from the user. In some embodiments, the one or more device inputs comprise a temperature setting, a window tint setting, and/or a lighting setting. In some embodiments, the lighting setting comprises a brightness and/or color of light. In some embodiments, the method further comprises generating the identity of the user, the identity comprising a unique code or number associated with the user. In some embodiments, determining the identity of the user comprises using machine learning to determine the identity. In some embodiments, the method further comprises training a machine learning model used to perform the machine learning. In some embodiments, training the machine learning model comprises using previously-obtained sets of images of the user taking the one or more physical actions as a positive dataset. In some embodiments, the previously-obtained sets of images are obtained by the imaging system.
In another aspect, an apparatus for external behavior recognition for control of an environment of a facility, the apparatus comprises one or more controllers comprises circuitry, which one or more controllers are configured to: capture, or direct capturing of, a plurality of successive images of a user of the facility, wherein the plurality of successive images are captured with an imaging system of the facility; obtain, or direct obtaining of, external behavior data of the user from the plurality of successive images, wherein the external behavior data is representative of one or more physical actions taken by the user during the capturing of the plurality of successive images; determine, or direct determining of, an identity of the user based at least in part on the external behavior data of the user; and implement, or direct implementing of, environment customizations associated with the identity of the user.
In some embodiments, the one or more physical actions taken by the user comprises a gait of the user. In some embodiments, the imaging system comprises a camera, an infrared (IR) camera, a lidar sensor, or an imaging radar system. In some embodiments, the imaging system comprises the camera or the IR camera, and wherein the plurality of successive images comprise video frames. In some embodiments, to obtain or direct the obtaining of external behavior data of the user, the one or more controllers are configured to extract, or direct extracting of, a respective pose of the user from each image of the plurality of successive images. In some embodiments, to determine or direct the determining of the identity of the user based at least in part on the external behavior data of the user, the one or more controllers are configured to determine, or direct determining of, a unique identifier associated with the user. In some embodiments, to implement or direct the implementing of environment customizations, the one or more controllers are configured to control, or direct controlling of, an environmental aspect using one or more building systems. In some embodiments, the one or more building systems comprises a device ensemble having a housing that encloses the one or more devices that comprise: (i) sensors, (ii) a transceiver, or (iii) a sensor and an emitter. In some embodiments, the device ensemble is disposed in a fixture of the facility, or is attached to a fixture of the facility. In some embodiments, the one or more building systems comprise a tintable window. In some embodiments, the tintable window comprises an electrochromic window. In some embodiments, the device ensemble comprises the imaging system. In some embodiments, to implement or direct the implementing of environment customizations, the one or more controllers are configured to adjust, or direct adjusting of, a temperature, window tint, and/or lighting within the facility. In some embodiments, the one or more controllers are configured to capture or direct the capturing of the plurality of successive images of the user responsive to a triggering event. In some embodiments, the triggering event comprises detection of the user at a location of the facility. In some embodiments, the one or more controllers are configured to determine or direct the determining of the identity of the user further based on sensor information regarding the user. In some embodiments, the sensor information comprises information indicative of a sound made by the user, dimensions of the user, and/or biometric information regarding the user. In some embodiments, the one or more controllers are configured to determine or direct the determining of the identity of the user further based on one or more device inputs received from the user. In some embodiments, the one or more device inputs comprise a temperature setting, a window tint setting, and/or a lighting setting. In some embodiments, the lighting setting comprises a brightness and/or color of light. In some embodiments, the one or more controllers are further configured to generate, or direct the generating of, the identity of the user, the identity comprising a unique code or number associated with the user. In some embodiments, to determine or direct the determining of the identity of the user the one or more controllers are configured to use, or direct the using of, machine learning to determine the identity. In some embodiments, the one or more controllers are further configured to train, or direct training of, a machine learning model used to perform the machine learning. In some embodiments, to train or direct the training of the machine learning model, the one or more controllers are configured to use, or direct the using of, previously-obtained sets of images of the user taking the one or more physical actions as a positive dataset. In some embodiments, the previously-obtained sets of images are obtained by the imaging system.
In another aspect, a non-transitory computer program product for external behavior recognition for control of an environment of a facility, which non-transitory computer program product contains instructions inscribed thereon that, when executed by one or more processors, cause the one or more processors to execute operations comprises: capturing, with an imaging system of the facility, a plurality of successive images of a user of the facility; obtaining, from the plurality of successive images, external behavior data of the user, wherein the external behavior data is representative of one or more physical actions taken by the user during the capturing of the plurality of successive images; determining an identity of the user based at least in part on the external behavior data of the user; and implementing environment customizations associated with the identity of the user.
In some embodiments, the one or more physical actions taken by the user comprises a gait of the user. In some embodiments, the imaging system comprises a camera, an infrared (IR) camera, a lidar sensor, or an imaging radar system. In some embodiments, the imaging system comprises the camera or the IR camera, and wherein the plurality of successive images comprise video frames. In some embodiments, the imaging system is operatively coupled with a controller. In some embodiments, the operations for obtaining the external behavior data of the user comprise operations for extracting a respective pose of the user from each image of the plurality of successive images. In some embodiments, the operations for determining the identity of the user based at least in part on the external behavior data of the user comprise operations for determining a unique identifier associated with the user. In some embodiments, the operations for implementing environment customizations comprise operations for controlling an environmental aspect using one or more building systems. In some embodiments, the one or more building systems comprises a device ensemble having a housing that encloses the one or more devices that comprise: (i) sensors, (ii) a transceiver, or (iii) a sensor and an emitter. In some embodiments, the device ensemble is disposed in a fixture of the facility, or is attached to a fixture of the facility. In some embodiments, the one or more building systems comprise a tintable window. In some embodiments, the tintable window comprises an electrochromic window. In some embodiments, the device ensemble comprises the imaging system. In some embodiments, the operations for implementing environment customizations comprise operations for adjusting a temperature, window tint, and/or lighting within the facility. In some embodiments, the operations for capturing the plurality of successive images of the user are responsive to a triggering event. In some embodiments, the triggering event comprises detection of the user at a location of the facility. In some embodiments, determining the identity of the user is further based on sensor information regarding the user. In some embodiments, the sensor information comprises information indicative of a sound made by the user, dimensions of the user, and/or biometric information regarding the user. In some embodiments, determining the identity of the user is further based on one or more device inputs received from the user. In some embodiments, the one or more device inputs comprise a temperature setting, a window tint setting, and/or a lighting setting. In some embodiments, the lighting setting comprises a brightness and/or color of light. In some embodiments, the operations further comprise generating the identity of the user, the identity comprising a unique code or number associated with the user. In some embodiments, the operations for determining the identity of the user comprise operations for using machine learning to determine the identity. In some embodiments, the operations further comprise training a machine learning model used to perform the machine learning. In some embodiments, the operations for training the machine learning model comprise operations for using previously-obtained sets of images of the user taking the one or more physical actions as a positive dataset. In some embodiments, the previously-obtained sets of images are obtained by the imaging system.
In another aspect, an apparatus for controlling a facility, the apparatus comprising at least one controller having circuitry, which at least one controller is configured to: (a) operatively couple to one or more sensors disposed in the facility, and to one or more devices disposed in the facility; (b) identify, or direct identification of, a user; (c) track, or direct tracking of, location of the user in the facility by using the one or more sensors; (d) receive an input related to the user; and (e) automatically control (e.g., alter), or direct automatic control (e.g., alteration) of, one or more devices in the facility by using the input and location information of the user.
In some embodiments, at least one controller is configured to utilize location of the user that is a present location of the user or a past location of the user. In some embodiments, the at least one controller is configured to identify, or direct identification of, the user at least in part by (I) receiving an identification card reading, or (II) performing image recognition on a captured image of the user in the facility. In some embodiments, the one or more sensors comprise a camera or a geolocation sensor. In some embodiments, the geolocation sensor comprises an ultrawide bandwidth sensor. In some embodiments, the geolocation sensor can locate the user with a resolution of at least twenty (20) centimeters or higher. In some embodiments, the input related to the user comprises a service request made by, on behalf of, or for, the user. In some embodiments, the input related to the user relates to activity of the user in an enclosure of the facility in which the user is located. In some embodiments, the input related to the user comprises an electronic file. In some embodiments, the input related to the user comprises a gesture and/or voice command made by the user. In some embodiments, the input related to the user relates to preference of the user. In some embodiments, the preference of the user is provided by a machine learning module that considers past activities of the user, wherein the at least one controller is operatively coupled to the machine learning module. In some embodiments, the preference of the user is input by the user. In some embodiments, the one or more devices comprises a lighting, a ventilation system, and air conditioning system, a heating system, a sound system, or a smell conditioning system. In some embodiments, the one or more devices is configured to affect an atmosphere of an enclosure of the facility in which the user is disposed. In some embodiments, the one or more devices comprises a service, office and/or factory apparatus. In some embodiments, the one or more devices are disposed out of an enclosure of the facility in which the user is located. In some embodiments, the one or more devices are disposed in an enclosure of the facility in which the user is located. In some embodiments, the one or more devices comprise a media projecting device. In some embodiments, the one or more devices comprise a tintable window. In some embodiments, the one or more devices comprise an electrochromic window. A non-transitory computer readable medium for controlling a facility, the non-transitory computer readable medium, when read by one or more processors, is configured to execute operations comprising operations of any of the above one or more controllers.
In another aspect, the present disclosure provides systems, apparatuses (e.g., controllers), and/or non-transitory computer-readable medium (e.g., software) that implement any of the methods disclosed herein.
In another aspect, the present disclosure provides methods that use any of the systems and/or apparatuses disclosed herein, e.g., for their intended purpose.
In another aspect, an apparatus comprises at least one controller that is programmed to direct a mechanism used to implement (e.g., effectuate) any of the method disclosed herein, wherein the at least one controller is operatively coupled to the mechanism.
In another aspect, an apparatus comprises at least one controller that is configured (e.g., programmed) to implement (e.g., effectuate) the method disclosed herein. The at least one controller may implement any of the methods disclosed herein.
In another aspect, a system comprises at least one controller that is programmed to direct operation of at least one another apparatus (or component thereof), and the apparatus (or component thereof), wherein the at least one controller is operatively coupled to the apparatus (or to the component thereof). The apparatus (or component thereof) may include any apparatus (or component thereof) disclosed herein. The at least one controller may direct any apparatus (or component thereof) disclosed herein.
In another aspect, a computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to direct a mechanism disclosed herein to implement (e.g., effectuate) any of the method disclosed herein, wherein the non-transitory computer-readable medium is operatively coupled to the mechanism. The mechanism can comprise any apparatus (or any component thereof) disclosed herein.
In another aspect, the present disclosure provides a non-transitory computer-readable medium comprising machine-executable code that, upon execution by one or more computer processors, implements any of the methods disclosed herein.
In another aspect, the present disclosure provides a non-transitory computer-readable medium comprising machine-executable code that, upon execution by one or more computer processors, effectuates directions of the controller(s) (e.g., as disclosed herein).
In another aspect, the present disclosure provides a computer system comprising one or more computer processors and a non-transitory computer-readable medium coupled thereto. The non-transitory computer-readable medium comprises machine-executable code that, upon execution by the one or more computer processors, implements any of the methods disclosed herein and/or effectuates directions of the controller(s) disclosed herein.
The content of this summary section is provided as a simplified introduction to the disclosure and is not intended to be used to limit the scope of any invention disclosed herein or the scope of the appended claims.
Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in this art from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.
These and other features and embodiments will be described in more detail with reference to the drawings.
All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference.
The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings or figures (also “FIG.” and “Figs.” herein), of which:
The figures and components therein may not be drawn to scale. Various components of the figures described herein may not be drawn to scale.
While various embodiments of the invention have been shown, and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein might be employed.
Terms such as “a,” “an,” and “the” are not intended to refer to only a singular entity but include the general class of which a specific example may be used for illustration. The terminology herein is used to describe specific embodiments of the invention(s), but their usage does not delimit the invention(s).
When ranges are mentioned, the ranges are meant to be inclusive, unless otherwise specified. For example, a range between value 1 and value 2 is meant to be inclusive and include value 1 and value 2. The inclusive range will span any value from about value 1 to about value 2. The term “adjacent” or “adjacent to,” as used herein, includes “next to,” “adjoining,” “in contact with,” and “in proximity to.”
The term “operatively coupled” or “operatively connected” refers to a first element (e.g., mechanism) that is coupled (e.g., connected) to a second element, to allow the intended operation of the second and/or first element. The coupling may comprise physical or non-physical coupling. The non-physical coupling may comprise signal-induced coupling (e.g., wireless coupling). Coupled can include physical coupling (e.g., physically connected), or non-physical coupling (e.g., via wireless communication). Additionally, in the following description, the phrases “operable to,” “adapted to,” “configured to,” “designed to,” “programmed to,” or “capable of” may be used interchangeably where appropriate.
An element (e.g., mechanism) that is “configured to” perform a function includes a structural feature that causes the element to perform this function. A structural feature may include an electrical feature, such as a circuitry or a circuit element. A structural feature may include a circuitry (e.g., comprising electrical or optical circuitry). Electrical circuitry may comprise one or more wires. Optical circuitry may comprise at least one optical element (e.g., beam splitter, mirror, lens and/or optical fiber). A structural feature may include a mechanical feature. A mechanical feature may comprise a latch, a spring, a closure, a hinge, a chassis, a support, a fastener, or a cantilever, and so forth. Performing the function may comprise utilizing a logical feature. A logical feature may include programming instructions. Programming instructions may be executable by at least one processor. Programming instructions may be stored or encoded on a medium accessible by one or more processors.
The following detailed description is directed to specific example implementations for purposes of disclosing the subject matter. Although the disclosed implementations are described in sufficient detail to enable those of ordinary skill in the art to practice the disclosed subject matter, this disclosure is not limited to particular features of the specific example implementations described herein. On the contrary, the concepts and teachings disclosed herein can be implemented and applied in a multitude of different forms and ways without departing from their spirit and scope. For example, while the disclosed implementations focus on electrochromic windows (also referred to as smart windows), some of the systems, devices and methods disclosed herein can be made, applied or used without undue experimentation to incorporate, or while incorporating, other types of optically switchable devices that are actively switched/controlled, rather than passive coatings such as thermochromic coatings or photochromic coatings that tint passively in response to the sun's rays. Some other types of actively controlled optically switchable devices include liquid crystal devices, suspended particle devices, and micro-blinds, among others. For example, some or all of such other optically switchable devices can be powered, driven or otherwise controlled or integrated with one or more of the disclosed implementations of controllers described herein.
In some embodiments, an enclosure comprises an area defined by at least one structure (e.g., fixture). The at least one structure may comprise at least one wall. An enclosure may comprise and/or enclose one or more sub-enclosures. The at least one wall may comprise metal (e.g., steel), clay, stone, plastic, glass, plaster (e.g., gypsum), polymer (e.g., polyurethane, styrene, or vinyl), asbestos, fiber-glass, concrete (e.g., reinforced concrete), wood, paper, or a ceramic. The at least one wall may comprise wire, bricks, blocks (e.g., cinder blocks), tile, drywall, or frame (e.g., steel frame and/or wooden frame).
In some embodiments, the enclosure comprises one or more openings. The one or more openings may be reversibly closable. The one or more openings may be permanently open. A fundamental length scale of the one or more openings may be smaller relative to the fundamental length scale of the wall(s) that define the enclosure. A fundamental length scale may comprise a diameter of a bounding circle, a length, a width, or a height. The fundamental length scale may be abbreviated herein as “FLS.” A surface of the one or more openings may be smaller relative to the surface the wall(s) that define the enclosure. The opening surface may be a percentage of the total surface of the wall(s). For example, the opening surface can measure about 30%, 20%, 10%, 5%, or 1% of the walls(s). The wall(s) may comprise a floor, a ceiling or a side wall. The closable opening may be closed by at least one window or door. The enclosure may be at least a portion of a facility. The facility may comprise a building. The enclosure may comprise at least a portion of a building. The building may be a private building and/or a commercial building. The building may comprise one or more floors. The building (e.g., floor thereof) may include at least one of: a room, hall, foyer, attic, basement, balcony (e.g., inner or outer balcony), stairwell, corridor, elevator shaft, fagade, mezzanine, penthouse, garage, porch (e.g., enclosed porch), terrace (e.g., enclosed terrace), cafeteria, and/or duct. In some embodiments, an enclosure may be stationary and/or movable (e.g., a train, a plane, a ship, a vehicle, or a rocket).
In some embodiments, the enclosure encloses an atmosphere. The atmosphere may comprise one or more gases. The gases may include inert gases (e.g., argon or nitrogen) and/or non-inert gases (e.g., oxygen or carbon dioxide). The enclosure atmosphere may resemble an atmosphere external to the enclosure (e.g., ambient atmosphere) in at least one external atmosphere characteristic that includes: temperature, relative gas content, gas type (e.g., humidity, and/or oxygen level), debris (e.g., dust and/or pollen), and/or gas velocity. The enclosure atmosphere may be different from the atmosphere external to the enclosure in at least one external atmosphere characteristic that includes: temperature, relative gas content, gas type (e.g., humidity, and/or oxygen level), debris (e.g., dust and/or pollen), and/or gas velocity. For example, the enclosure atmosphere may be less humid (e.g., drier) than the external (e.g., ambient) atmosphere. For example, the enclosure atmosphere may contain the same (e.g., or a substantially similar) oxygen-to-nitrogen ratio as the atmosphere external to the enclosure. The velocity and/or content of the gas in the enclosure may be (e.g., substantially) similar throughout the enclosure. The velocity and/or content of the gas in the enclosure may be different in different portions of the enclosure (e.g., by flowing gas through to a vent that is coupled with the enclosure). The gas content may comprise relative gas ratio.
In some embodiments, a network infrastructure is provided in the enclosure (e.g., a facility such as a building). The network infrastructure is available for various purposes such as for providing communication and/or power services. The communication services may comprise high bandwidth (e.g., wireless and/or wired) communications services. The communication services can be to occupants of a facility and/or users outside the facility (e.g., building). The network infrastructure may work in concert with, or as a partial replacement of, the infrastructure of one or more cellular carriers. The network may comprise one or more levels of encryption. The network may be communicatively coupled to the cloud and/or to one or more servers external to the facility. The network may support at least a fourth generation wireless (4G), or a fifth-generation wireless (5G) communication. The network may support cellular signals external and/or internal to the facility. The downlink communication network speeds may have a peak data rate of at least about 5 Gigabits per second (Gb/s), 10 Gb/s, or 20 Gb/s. The uplink communication network speeds may have a peak data rate of at least about 2 Gb/s, 5 Gb/s, or 10 Gb/s. The network infrastructure can be provided in a facility that includes electrically switchable windows. Examples of components of the network infrastructure include a high speed backhaul. The network infrastructure may include at least one cable, switch, (e.g., physical) antenna, transceivers, sensor, transmitter, receiver, radio, processor and/or controller (that may comprise a processor). The network infrastructure may be operatively coupled to, and/or include, a wireless network. The network infrastructure may comprise wiring (e.g., comprising an optical fiber, twisted cable, or coaxial cable). One or more devices (e.g., sensors and/or emitters) can be deployed (e.g., installed) in an environment, e.g., as part of installing the network infrastructure and/or after installing the network infrastructure. The device(s) may be communicatively coupled to the network. The network may comprise a power and/or communication network. The device can be self-discovered on the network, e.g., once it couples (e.g., on its attempt to couple) to the network. The network structure may comprise peer to peer network structure, or client-server network structure. The network may or may not have a central coordination entity (e.g., server(s) or another stable host).
In some embodiments, a building management system (BMS) is a computer-based control system. The BMS can be installed in a facility to monitor and otherwise control (e.g., regulate, manipulate, restrict, direct, monitor, adjust, modulate, vary, alter, restrain, check, guide, or manage) the facility. For example, the BMS may control one or more devices communicatively coupled to the network. The one or more devices may include mechanical and/or electrical equipment such as ventilation, lighting, power systems, elevators, fire systems, and/or security systems. Controllers (e.g., nodes and/or processors) may be suited for integration with a BMS. A BMS may include hardware. The hardware may include interconnections by communication channels to one or more processors (e.g., and associated software), e.g., for maintaining one or more conditions in the facility. The one or more conditions in the facility may be according to preference(s) set by a user (e.g., an occupant, a facility owner, and/or a facility manager). For example, a BMS may be implemented using a local area network, such as Ethernet. The software can utilize, e.g., internet protocols and/or open standards. One example is software from Tridium, Inc. (of Richmond, Va.). One communication protocol that can be used with a BMS is BACnet (building automation and control networks). A node can be any addressable circuitry. For example, a node can be a circuitry that has an Internet Protocol (IP) address.
In some embodiments, a BMS may be implemented in a facility, e.g., a multi-story building. The BMS may function (e.g., also) to control one or more characteristics of an environment of the facility. The one or more characteristics may comprise: temperature, carbon dioxide levels, gas flow, various volatile organic compounds (VOCs), and/or humidity in a building. There may be mechanical devices that are controlled by a BMS such as one or more heaters, air conditioners, blowers, and/or vents. To control the facility environment, a BMS may turn these various devices on and/or off under defined conditions. A core function of a BMS may be to maintain a comfortable environment for occupants of the environment, e.g., while minimizing heating and cooling costs and/or demand. A BMS can be used to control one or more of the various systems. A BMS may be used to optimize the synergy between various systems. For example, the BMS may be used to conserve energy and lower building operation costs.
In some embodiments, the facility comprises a multi-story building. The multi-story building may have at least 2, 8, 10, 25, 50, 80, 100, 120, 140, or 160 floors, e.g., that are controlled by the control system and/or comprise the network infrastructure. The number of floors controlled by the control system and/or comprising the network infrastructure may be any number between the aforementioned numbers (e.g., from 2 to 50, from 25 to 100, or from 80 to 160). The floor may be of an area of at least about 150 m2, 250 m2, 500 m2, 1000 m2, 1500 m2, or 2000 square meters (m2). The floor may have an area between any of the aforementioned floor area values (e.g., from about 150 m2 to about 2000 m2, from about 150 m2 to about 500 m2 from about 250 m2 to about 1000 m2, or from about 1000 m2 to about 2000 m2).
In some embodiments, a window controller is integrated with a BMS. For example, the window controller can be configured to control one or more tintable windows (e.g., electrochromic windows). In one embodiment, the one or more electrochromic windows include at least one all solid state and inorganic electrochromic device, but may include more than one electrochromic device, e.g. where each lite or pane of an IGU is tintable. In one embodiment, the one or more electrochromic windows include only all solid state and inorganic electrochromic devices. In one embodiment, the electrochromic windows are multistate electrochromic windows. Examples of tintable windows can be found in, in U.S. patent application Ser. No. 12/851,514, filed on Aug. 5, 2010, and titled “Multipane Electrochromic Windows,” which is incorporated herein by reference in its entirety.
In some embodiments, one or more devices such as sensors, emitters, and/or actuators, are operatively coupled to at least one controller and/or processor. Sensor readings may be obtained by one or more processors and/or controllers. A controller may comprise a processing unit (e.g., CPU or GPU). A controller may receive an input (e.g., from at least one device or projected media). The controller may comprise circuitry, electrical wiring, optical wiring, socket, and/or outlet. A controller may receive an input and/or deliver an output. A controller may comprise multiple (e.g., sub-) controllers. An operation (e.g., as disclosed herein) may be performed by a single controller or by a plurality of controllers. At least two operations may be each preconformed by a different controller. At least two operations may be preconformed by the same controller. A device and/or media may be controlled by a single controller or by a plurality of controllers. At least two devices and/or media may be controlled by a different controller. At least two devices and/or media may be controlled by the same controller. The controller may be a part of a control system. The control system may comprise a master controller (MC), floor (e.g., comprising NC) controller, or a local controller. The local controller may be a target controller. For example, the local controller may be a window controller (e.g., controlling an optically switchable window), enclosure controller, or component controller. The controller may be a part of a hierarchal control system. They hierarchal control system may comprise a main controller that directs one or more controllers, e.g., floor controllers, local controllers (e.g., window controllers), enclosure controllers, and/or component controllers. The target may comprise a device or a media. The device may comprise an electrochromic window, a sensor, an emitter, an antenna, a receiver, a transceiver, or an actuator.
In some embodiments, the network infrastructure is operatively coupled to one or more controllers. In some embodiments, a physical location of the controller type in the hierarchal control system changes. A controller may control one or more devices (e.g., be directly coupled to the devices). A controller may be disposed proximal to the one or more devices it is controlling. For example, a controller may control an optically switchable device (e.g., IGU), an antenna, a sensor, and/or an output device (e.g., a light source, sounds source, smell source, gas source, HVAC outlet, or heater). In one embodiment, a floor controller may direct one or more window controllers, one or more enclosure controllers, one or more component controllers, or any combination thereof. The floor controller may comprise a floor controller. For example, the floor (e.g., comprising network) controller may control a plurality of local (e.g., comprising window) controllers. A plurality of local controllers may be disposed in a portion of a facility (e.g., in a portion of a building). The portion of the facility may be a floor of a facility. For example, a floor controller may be assigned to a floor. In some embodiments, a floor may comprise a plurality of floor controllers, e.g., depending on the floor size and/or the number of local controllers coupled to the floor controller. For example, a floor controller may be assigned to a portion of a floor. For example, a floor controller may be assigned to a portion of the local controllers disposed in the facility. For example, a floor controller may be assigned to a portion of the floors of a facility. An MC may be coupled to one or more floor controllers. The floor controller may be disposed in the facility. The MC may be disposed in the facility, or external to the facility. The MC may be disposed in the cloud. A controller may be a part of, or be operatively coupled to, a building management system. A controller may receive one or more inputs. A controller may generate one or more outputs. The controller may be a single input single output controller (SISO) or a multiple input multiple output controller (MIMO). A controller may interpret an input signal received. A controller may acquire data from the one or more components (e.g., sensors). Acquire may comprise receive or extract. The data may comprise measurement, estimation, determination, generation, or any combination thereof. A controller may comprise feedback control. A controller may comprise feed-forward control. Control may comprise on-off control, proportional control, proportional-integral (PI) control, or proportional-integral-derivative (PID) control. Control may comprise open loop control, or closed loop control. A controller may comprise closed loop control. A controller may comprise open loop control. A controller may comprise a user interface. A user interface may comprise (or operatively coupled to) a keyboard, keypad, mouse, touch screen, microphone, speech recognition package, camera, imaging system, or any combination thereof. Outputs may include a display (e.g., screen), speaker, or printer. In some embodiments, a local controller controls one or more devices and/or media (e.g., media projection). For example, a local controller can control one or more IGUs, one or more sensors, one or more output devices (e.g., one or more emitters), one or more media, or any combination thereof.
In some embodiments, a BMS includes a multipurpose controller. By incorporating feedback (e.g., of the controller), a BMS can provide, for example, enhanced: 1) environmental control, 2) energy savings, 3) security, 4) flexibility in control options, 5) improved reliability and usable life of other systems (e.g., due to decreased reliance thereon and/or reduced maintenance thereof), 6) information availability and/or diagnostics, 7) higher productivity from personnel in the building (e.g., staff), and various combinations thereof. These enhancements may derive automatically controlling any of the devices. In some embodiments, a BMS may not be present. In some embodiments, a BMS may be present without communicating with a master network controller. In some embodiments, a BMS may communicate with a portion of the levels in the hierarchy of controllers. For example, the BMS may communicate (e.g., at a high level) with a master network controller. In some embodiments, a BMS may not communicate with a portion of the levels in the hierarchy of controllers of the control system. For example, the BMS may not communicate with the local controller and/or intermediate controller. In certain embodiments, maintenance on the BMS would not interrupt control of the devices communicatively coupled to the control system. In some embodiments, the BMS comprises at least one controller that may or may not be part of the hierarchical control system.
The methods, systems and/or the apparatus described herein may comprise a control system. The control system can be in communication with any of the apparatuses (e.g., sensors) described herein. The sensors may be of the same type or of different types, e.g., as described herein. For example, the control system may be in communication with the first sensor and/or with the second sensor. A plurality of devices (e.g., sensors and/or emitters) may be disposed in a container and may constitute an ensemble (e.g., a digital architectural element). The ensemble may comprise at least two devices of the same type. The ensemble may comprise at least two devices of a different type. The devices in the ensemble may be operatively coupled to the same electrical board. The electrical board may comprise circuitry. The electrical board may comprise, or be operatively coupled to a controller (e.g., a local controller). The control system may control the one or more devices (e.g., sensors). The control system may control one or more components of a building management system (e.g., lightening, security, and/or air conditioning system). The controller may regulate at least one (e.g., environmental) characteristic of the enclosure. The control system may regulate the enclosure environment using any component of the building management system. For example, the control system may regulate the energy supplied by a heating element and/or by a cooling element. For example, the control system may regulate velocity of an air flowing through a vent to and/or from the enclosure. The control system may comprise a processor. The processor may be a processing unit. The controller may comprise a processing unit. The processing unit may be central. The processing unit may comprise a central processing unit (CPU). The processing unit may be a graphic processing unit (GPU). The controller(s) or control mechanisms (e.g., comprising a computer system) may be programmed to implement one or more methods of the disclosure. The processor may be programmed to implement methods of the disclosure. The controller may control at least one component of the forming systems and/or apparatuses disclosed herein. Examples of a digital architectural element can be found in PCT patent application serial number PCT/US20/70123 that is incorporated herein by reference in its entirety.
The computer system can include a processing unit (e.g., 206) (also “processor,” “computer” and “computer processor” used herein). The computer system may include memory or memory location (e.g., 202) (e.g., random-access memory, read-only memory, flash memory), electronic storage unit (e.g., 204) (e.g., hard disk), communication interface (e.g., 203) (e.g., network adapter) for communicating with one or more other systems, and peripheral devices (e.g., 205), such as cache, other memory, data storage and/or electronic display adapters. In the example shown in
The processing unit can execute a sequence of machine-readable instructions, which can be embodied in a program or software. The instructions may be stored in a memory location, such as the memory 202. The instructions can be directed to the processing unit, which can subsequently program or otherwise configure the processing unit to implement methods of the present disclosure. Examples of operations performed by the processing unit can include fetch, decode, execute, and write back. The processing unit may interpret and/or execute instructions. The processor may include a microprocessor, a data processor, a central processing unit (CPU), a graphical processing unit (GPU), a system-on-chip (SOC), a co-processor, a network processor, an application specific integrated circuit (ASIC), an application specific instruction-set processor (ASIPs), a controller, a programmable logic device (PLD), a chipset, a field programmable gate array (FPGA), or any combination thereof. The processing unit can be part of a circuit, such as an integrated circuit. One or more other components of the system 200 can be included in the circuit.
The storage unit can store files, such as drivers, libraries and saved programs. The storage unit can store user data (e.g., user preferences and user programs). In some cases, the computer system can include one or more additional data storage units that are external to the computer system, such as located on a remote server that is in communication with the computer system through an intranet or the Internet.
The computer system can communicate with one or more remote computer systems through a network. For instance, the computer system can communicate with a remote computer system of a user (e.g., operator). Examples of remote computer systems include personal computers (e.g., portable PC), slate or tablet PC's (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones (e.g., Apple® iPhone, Android-enabled device, Blackberry®), or personal digital assistants. A user (e.g., via a client application) can access the computer system via the network.
Methods as described herein can be implemented by way of machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system, such as, for example, on the memory 202 or electronic storage unit 204. The machine executable or machine-readable code can be provided in the form of software. During use, the processing unit 206 can execute the code. In some cases, the code can be retrieved from the storage unit and stored on the memory for ready access by the processor. In some situations, the electronic storage unit can be precluded, and machine-executable instructions are stored on memory.
Multiple processors, sometimes contained on two or more controllers (e.g., MC 108, NC 106, WC 104), may operate in concert to implement the processing requirements of a personal computing unit. In such cases, a distributed processing control mechanism may be necessary to coordinate the operation of the multiple processors. In certain embodiments, distributed processing for a personal computing unit is implemented in a peer-to-peer or a master-slave configuration. In certain embodiments, the distributed processing system employs a block chain technology such as Bitcoin or the open source program Gridcoin that uses the Berkeley Open Infrastructure for Network Computing. To implement distributed processing, software for interfacing with the processors may have a container architecture. In some embodiments, the container architecture is implemented via a container management layer in a network protocol. One example of resource for implementing distributed processing via a container architecture is Docker for the Linux Containers (LXC) format. Docker provides namespaces to isolate an application's view of the operating system, including process trees, network resources, user IDs, and file systems. The distributed processing may allow load balancing by using a product such as, for example, IBM's Cloud Orchestrator™.
The computing power of a computer system (e.g., computer system 200) is a function of (i) the power of individual processors and (ii) the number of such processors working in concert. In some cases, many processors throughout a building or even throughout multiple buildings are used by a computer system or are at least available to the computer system during its operating session. Because the processing power grows with the number of available processors, a computer system may in some cases attain much greater processing power than is conventionally available with a single device.
The processing resources for a computer system can be chosen based on various criteria. The current and/or projected availability of spare processing power when a computer system is established may be one criterion. The physical location of processors may be another criterion. The ability of processors to participate in a multi-processor computing environment may be still another criterion. In one example, multiple processors are selected for inclusion in a computer system and configured in a manner that allows them to conduct fog computing. Fog computing also known as fog networking or fogging, is a decentralized computing infrastructure in which data, computing/processing power, storage, and applications are distributed in the most logical, efficient place between a data source and a user. If an external network (e.g., the internet or a cloud computing resource) are involved, the computing infrastructure is chosen taking into consideration the location of relevant cloud or internet resources.
In certain embodiments, mobile devices and/or other computing devices that are not physically part of the computer system may be temporarily connected to the computer system and thereby become available as a processing resource for a computer system. In a sense, such external devices may serve as part of virtual window resource and provide its untapped processing power for an occupant/computational user of a computer system.
The code can be pre-compiled and configured for use with a machine have a processer adapted to execute the code or can be compiled during runtime. The code can be supplied in a programming language that can be selected to enable the code to execute in a pre-compiled or as-compiled fashion. In some embodiments, the processor comprises a code. The code can be program instructions. The program instructions may cause the at least one processor (e.g., computer) to direct a feed forward and/or feedback control loop. In some embodiments, the program instructions cause the at least one processor to direct a closed loop and/or open loop control scheme. The control may be based at least in part on one or more sensor readings (e.g., sensor data). One controller may direct a plurality of operations. At least two operations may be directed by different controllers. In some embodiments, a different controller may direct at least two of operations (a), (b) and (c). In some embodiments, different controllers may direct at least two of operations (a), (b) and (c). In some embodiments, a non-transitory computer-readable medium cause each a different computer to direct at least two of operations (a), (b) and (c). In some embodiments, different non-transitory computer-readable mediums cause each a different computer to direct at least two of operations (a), (b) and (c). The controller and/or computer readable media may direct any of the apparatuses or components thereof disclosed herein. The controller and/or computer readable media may direct any operations of the methods disclosed herein. The controller may be operatively (communicatively) coupled to control logic (e.g., code embedded in a software) in which its operation(s) are embodied.
In some embodiments, optically switchable windows forms or occupies substantial portions of a building envelope. For example, the optically switchable windows can form substantial portions of the walls, facades and even roofs of a corporate office building, other commercial building or a residential building. A distributed network of controllers can be used to control the optically switchable windows. For example, a network system may be operable to control a plurality of IGUs. One primary function of the network system is controlling the optical states of electrochromic devices (ECDs) (or other optically switchable devices) within the IGUs. In some implementations, one or more windows can be multi-zoned windows, for example, where each window includes two or more independently controllable ECDs or zones. In some embodiments, the network system (e.g., MC 300 of
In some embodiments, another function of the network system is to acquire status information from the IGUs (hereinafter “information” is used interchangeably with “data”). For example, the status information for a given IGU can include an identification of, or information about, a current tint state of the ECD(s) within the IGU. The network system also can be operable to acquire data from various sensors, such as temperature sensors, photosensors (also referred to herein as light sensors), humidity sensors, air flow sensors, or occupancy sensors, whether integrated on or within the IGUs or located at various other positions in, on or around the building.
The network system can include any suitable number of distributed controllers having various capabilities or functions. In some implementations, the functions and arrangements of the various controllers are defined hierarchically. For example, the network system can include a plurality of distributed window controllers (WCs), a plurality of NCs, and an MC. The NCs may be included in the floor controllers. In some implementations, the MC can communicate with and control tens or hundreds of NCs. In various implementations, the MC issues high level instructions to the NCs over one or more wired and/or wireless links. The instructions can include, for example, tint commands for causing transitions in the optical states of the IGUs controlled by the respective NCs. Each NC can, in turn, communicate with and control a number of WCs over one or more wired and/or wireless links. For example, each NC can control tens or hundreds of the WCs. Each WC can, in turn, communicate with, drive or otherwise control one or more respective IGUs over one or more wired and/or wireless links.
In some embodiments, the MC issues communications including tint commands, status request commands, data (for example, sensor data) request commands or other instructions. The MC 308 may issue such communications periodically, at certain predefined times of day (which may change based at least in part on the day of week or year), or based at least in part on the detection of particular events, conditions or combinations of events or conditions (for example, as determined by acquired sensor data or based at least in part on the receipt of a request initiated by a user or by an application or a combination of such sensor data and such a request). In some embodiments, when the MC determines to cause a tint state change in a set of one or more IGUs, the MC generates or selects a tint value corresponding to the desired tint state. In some embodiments, the set of IGUs is associated with a first protocol identifier (ID) (for example, a BACnet ID). The MC then generates and transmits a communication-referred to herein as a “primary tint command”—including the tint value and the first protocol ID over the link via a first communication protocol (for example, a BACnet compatible protocol). The MC may address the primary tint command to the particular NC that controls the particular one or more WCs that, in turn, control the set of IGUs to be transitioned.
In some embodiments, the NC receives the primary tint command including the tint value and the first protocol ID and maps the first protocol ID to one or more second protocol IDs. Each of the second protocol IDs may identify a corresponding one of the WCs. The NC may subsequently transmit a secondary tint command including the tint value to each of the identified WCs over the link via a second communication protocol. For example, each of the WCs that receives the secondary tint command can then select a voltage or current profile from an internal memory based at least in part on the tint value to drive its respectively connected IGUs to a tint state consistent with the tint value. Each of the WCs may then generate and provide voltage or current signals over the link to its respectively connected IGUs to apply the voltage or current profile, for example.
In some embodiments, the various targets (e.g., IGUs) are (e.g., advantageously) grouped into zones of targets (e.g., of EC windows). At least one zone (e.g., each of which zones) can include a subset of the targets (e.g., IGUs). For example, at least one (e.g., each) zone of targets (e.g., IGUs) may be controlled by one or more respective floor controllers (e.g., NCs) and one or more respective local controllers (e.g., WCs) controlled by these floor controllers (e.g., NCs). In some examples, at least one (e.g., each) zone can be controlled by a single floor controller (e.g., NC) and two or more local controllers (e.g., WCs) controlled by the single floor controller (e.g., NC). For example, a zone can represent a logical grouping of the targets (e.g., IGUs). Each zone may correspond to a set of targets (e.g., IGUs) in a specific location or area of the building that are driven together based at least in part on their location. For example, a building may have four faces or sides (a North face, a South face, an East Face and a West Face) and ten floors. In such a didactic example, each zone may correspond to the set of electrochromic windows on a particular floor and on a particular one of the four faces. At least one (e.g., each) zone may correspond to a set of targets (e.g., IGUs) that share one or more physical characteristics (for example, device parameters such as size or age). In some embodiments, a zone of targets (e.g., IGUs) is grouped based at least in part on one or more non-physical characteristics such as, for example, a security designation or a business hierarchy (for example, IGUs bounding managers' offices can be grouped in one or more zones while IGUs bounding non-managers' offices can be grouped in one or more different zones).
In some embodiments, at least one (e.g., each) floor controller (e.g., NC) is able to address all of the targets (e.g., IGUs) in at least one (e.g., each) of one or more respective zones. For example, the MC can issue a primary tint command to the floor controller (e.g., NC) that controls a target zone. The primary tint command can include an (e.g., abstract) identification of the target zone (hereinafter also referred to as a “zone ID”). For example, the zone ID can be a first protocol ID such as that just described in the example above. In such cases, the floor controller (e.g., NC) receives the primary tint command including the tint value and the zone ID and maps the zone ID to the second protocol IDs associated with the local controllers (e.g., WCs) within the zone. In some embodiments, the zone ID is a higher level abstraction than the first protocol IDs. In such cases, the floor controller (e.g., NC) can first map the zone ID to one or more first protocol IDs, and subsequently map the first protocol IDs to the second protocol IDs.
In some embodiments, the MC is coupled to one or more outward-facing networks via one or more wired and/or wireless links. For example, the MC can communicate acquired status information or sensor data to remote computers, mobile devices, servers, databases in or accessible by the outward-facing network. In some embodiments, various applications, including third party applications or cloud-based applications, executing within such remote devices are able to access data from or provide data to the MC. In some embodiments, authorized users or applications communicate requests to modify the tint states of various IGUs to the MC via the network. For example, the MC can first determine whether to grant the request (for example, based at least in part on power considerations or based at least in part on whether the user has the appropriate authorization) prior to issuing a tint command. The MC may then calculate, determine, select or otherwise generate a tint value and transmit the tint value in a primary tint command to cause the tint state transitions in the associated IGUs.
In some embodiments, a user submits such a request from a computing device, such as a desktop computer, laptop computer, tablet computer or mobile device (for example, a smartphone). The user's computing device may execute a client-side application that is capable of communicating with the MC, and in some examples, with a master controller application executing within the MC. In some embodiments, the client-side application may communicate with a separate application, in the same or a different physical device or system as the MC, which then communicates with the master controller application to affect the desired tint state modifications. For example, the master controller application or other separate application can be used to authenticate the user to authorize requests submitted by the user. The user may select a target to be manipulated (e.g., the IGUs to be tinted), and directly or indirectly inform the MC of the selections, e.g., by entering an enclosure ID (e.g., room number) via the client-side application.
In some embodiments, a mobile circuitry of a user (e.g., mobile electronic device or other computing device) can communicate, e.g., wirelessly with various local controllers (e.g., WCs). For example, a client-side application executing within a mobile circuitry of a user (e.g., mobile device) can transmit wireless communications including control signals related to a target to the local controller to control the target, which target is communicatively coupled to the local controller (e.g., via the network). For example, a user may initiate directing a tint state control signals to a WC to control the tint states of the respective IGUs connected to the WC. For example, the user can use the client-side application to control (e.g., maintain or modify) the tint states of the IGUs adjoining a room occupied by the user (or to be occupied by the user or others at a future time). For example, a user may initiate directing a sensor frequency change control signals to a local controller to control the data sampling rate of a sensor communicatively coupled to the local controller. For example, the user can use the client-side application to control (e.g., maintain or modify) the data sampling rate of the sensor adjoining a room occupied by the user (or to be occupied by the user or others at a future time). For example, a user may initiate directing a light intensity change control signals to a local controller to control the light of a lamp communicatively coupled to the local controller. For example, the user can use the client-side application to control (e.g., maintain or modify) the light intensity of the light adjoining a room occupied by the user (or to be occupied by the user or others at a future time). For example, a user may initiate directing a media projection change control signals to a local controller to control the media projected by a projector communicatively coupled to the local controller. For example, the user can use the client-side application to control (e.g., maintain or modify) the media projected by a projector in a room occupied by the user (or to be occupied by the user or others at a future time). The wireless communications can be generated, formatted and/or transmitted using various wireless network topologies and protocols, for example.
In some embodiments, the control signals sent to the local controller (e.g., WC) from a mobile circuitry (e.g., device) of a user (or other computing device) override a previously sent signal (e.g., a tint value previously received by the WC from the respective NC). The previously sent signal may be automatically generated, e.g., by the control system. In other words, the local controller (e.g., WC) may provide the applied voltages to the target (e.g., IGUs) based at least in part on the control signals from the mobile circuitry of the user (e.g., user's computing device), e.g., rather than based at least in part on the predetermined signal (e.g., the tint value). For example, a control algorithm or rule set stored in and executed by the local controller (e.g., WC) may dictate that one or more control signals from a mobile device of a user (e.g., an authorized user's computing device) that will take precedence over a respective signal received from the control system (e.g., a tint value received from the NC). In some embodiments, such as in high demand cases, control signals (such as a tint value from the NC) take precedence over any control signals received by the local controller (e.g., WC) from a mobile circuitry of a user (e.g., a user's computing device). A control algorithm or rule set may dictate that control signal (e.g., relating to tint) overrides from only certain users (or groups or classes of users) may take precedence based at least in part on permissions granted to such users. In some instances, other factors including time of day or the location of the target (e.g., IGUs) may influence the permission to override a predetermined signal of the control system.
In some embodiments, based at least in part on the receipt of a control signal from a mobile circuitry of a user (e.g., an authorized user's computing device), the MC uses information about a combination of known parameters to calculate, determine, select and/or otherwise generate a command signal (e.g., relating to a tint value) that provides (e.g., lighting) conditions requested by a (e.g., typical) user, e.g., while in some instances also using power efficiently. For example, the MC may determine a state of a target based at least in part on preset preferences defined by or for the particular user that requested the target status change via the mobile circuitry (e.g., via the computing device). For example, the MC may determine the tint value based at least in part on preset preferences defined by or for the particular user that requested the tint state change via the computing device. For example, the user may be required to enter a password or otherwise login or obtain authorization to request a change in a state of a target (e.g., tint state change). The MC may determine the identity of the user based at least in part on a password, a security token and/or an identifier of the particular mobile circuitry (e.g., mobile device or other computing device). After determining the identity of the user, the MC may then retrieve preset preferences for the user, and use the preset preferences alone or in combination with other parameters (such as power considerations and/or information from various sensors) to generate and transmit a status change of the target (e.g., tint value for use in tinting the respective IGUs).
In some embodiments, the network system includes wall switches, dimmers, or other (e.g., tint-state) controlling devices. A wall switch generally refers to an electromechanical interface connected to a local controller (e.g., WC). The wall switch can convey a target status change (e.g., tint) command to the local controller (e.g., WC), which can then convey the target status change (e.g., tint) command to an upper level controller such as a local controller (e.g., NC). Such control devices can be collectively referred to as “wall devices,” although such devices need not be limited to wall-mounted implementations (for example, such devices also can be located on a ceiling or floor or integrated on or within a desk or a conference table). For example, some or all of the offices, conference rooms, or other rooms of the building can include such a wall device for use in controlling the state of a target (e.g., tint states of the adjoining IGUs, or light state of a light bulb). For example, the IGUs adjoining a particular room can be grouped into a zone. Each of the wall devices can be operated by an end user (for example, an occupant of the respective room) to control the state of grouped targets (e.g., to control tint state or other functions or parameters of the IGUs that adjoin the room). For example, at certain times of the day, the adjoining IGUs may be tinted to a dark state to reduce the amount of light energy entering the room from the outside (for example, to reduce AC cooling requirements). For example, at certain times of the day, the adjoining heaters may be turned on to a warmer temperature to facilitate occupant comfort. In some embodiments, when a user requests to use a room then the user can operate the wall device to communicate one or more control signals to cause a (e.g., tint state) transition from one state of a target to another state (e.g., from the dark state to a lighter tint state of an IGU).
In some embodiments, each wall device includes one or more switches, buttons, dimmers, dials, or other physical user interface controls enabling the user to select a particular tint state or to increase or decrease a current tinting level of the IGUs adjoining the room. The wall device may include a display having a touchscreen interface enabling the user to select a particular tint state (for example, by selecting a virtual button, selecting from a dropdown menu or by entering a tint level or tinting percentage) or to modify the tint state (for example, by selecting a “darken” virtual button, a “lighten” virtual button, or by turning a virtual dial or sliding a virtual bar). In some embodiments, the wall device includes a docking interface enabling a user to physically and communicatively dock a mobile circuitry (e.g., portable device such as a smartphone, multimedia device, remote controller, virtual reality device, tablet computer, or other portable computing device (for example, an PHONE, IPOD or PAD produced by Apple, Inc. of Cupertino, CA)). The mobile circuitry may be embedded in a vehicle (e.g., car, motorcycle, drone, airplane). The mobile circuitry may be embedded in a robot. A circuitry may be embedded in (e.g., be part of) a virtual assistant AI technology, speaker, (e.g., smart speaker such as Google Nest, or Amazon Echo Dot). Coupling of the mobile circuitry to the network may be initiated by a user's presence in the enclosure, or by a user's coupling (e.g., weather remote or local) to the network. Coupling of the user to the network may be security (e.g., having one or more security layers, and/or require one or more security tokens (e.g., keys)). The presence of the user in the enclosure may be sensed (e.g., automatically) by using the sensor(s) that are coupled to the network. The minimum distance from the sensor at which the user is coupled to the network may be predetermined and/or adjusted. A user may override its coupling to the network. The user may be a manager, executive, owner, lessor, administrator of the network and/or facility. The user may be the user of the mobile circuitry. The ability to couple the mobile circuitry to the network may or may not be overridden by the user. The ability to alter the minimum coupling distance between the mobile circuitry and the network may or may not be overridden by the user. There may be a hierarchy of overriding permissions. The hierarchy may depend on the type of user and/or type of mobile circuitry. For example, a factory employee user may not be allowed to alter coupling of a production machinery to the network. For example, an employee may be allowed to alter the coupling distance of his/her company laptop computer to the network. For example, an employee may be permitted to allow or prevent coupling of her/his personal cellular phone and/or car to the network. For example, a visitor may be prevented from having the visitor's mobile circuitry connected to the network. The coupling to the network may be automatic and seamless (e.g., after the initial preference have been set). Seamless coupling may be without requiring input from the user.
In such an example, the user can control the tinting levels via input to the mobile circuitry (e.g., portable device), which is then received by the wall device through the docking interface and subsequently communicated to the control system (e.g., to the MC, NC, or WC). The mobile circuitry (e.g., portable device) may include an application for communicating with an API presented by the wall device.
In some embodiments, the wall device can transmit a request for a status change of a target (e.g., a tint state change) to the control system (e.g., to the MC). The control system (e.g., MC) might first determine whether to grant the request (for example, based at least in part on power considerations and/or based at least in part on whether the user has the appropriate authorizations or permissions). The control system (e.g., MC) could calculate, determine, select, and/or otherwise generate a status change (e.g., tint) value and transmit the status change (e.g., tint) value in a primary status change (e.g., tint) command to cause the target to change (e.g., cause the tint state transitions in the adjoining IGUs). For example, each wall device may be connected with the control system (e.g., the MC therein) via one or more wired links (for example, over communication lines such as CAN or Ethernet compliant lines and/or over power lines using power line communication techniques). For example, each wall device could be connected with the control system (e.g., the MC therein) via one or more wireless links. The wall device may be connected (via one or more wired and/or wireless connections) with an outward-facing network, which may communicate with the control system (e.g., the MC therein) via the link.
In some embodiments, the control system identifies the target (e.g., target device) associated with the wall device based at least in part on previously programmed or discovered information associating the wall device with the target. For example, the MC identifies the IGUs associated with the wall device based at least in part on previously programmed or discovered information associating the wall device with the IGUs. A control algorithm or rule set can be stored in and executed by the control system (e.g., the MC therein) to dictate that one or more control signals from a wall device take precedence over a tint value previously generated by the control system (e.g., the MC therein), for example. In times of high demand (for example, high power demand), a control algorithm or rule set stored in and executed by the control system (e.g., the MC therein) may be used to dictate that the tint value previously generated by the control system (e.g., the MC therein) takes precedence over any control signals received from a wall device.
In some embodiments, based at least in part on the receipt of a request or control signal to change to a state of a target (e.g., tint-state-change request or control signal) from a wall device, the control system (e.g., the MC therein) uses information about a combination of known parameters to generate a state change (e.g., tint) value that provides lighting conditions desirable for a typical user. Accordingly, the control system (e.g., the MC therein) may use power more efficiently. In some embodiments, the control system (e.g., the MC therein) can generate the state change (e.g., tint) value based at least in part on preset preferences defined by or for the particular user that requested the (e.g., tint) state change of the target via the wall device. For example, the user may be required to enter a password into the wall device or to use a security token or security fob such as the BUTTON or other 1-Wire device to gain access to the wall device. The control system (e.g., the MC therein) may then determine the identity of the user, based at least in part on the password, security token and/or security fob. The control system (e.g., the MC therein) may retrieve preset preferences for the user. The control system (e.g., the MC therein) may use the preset preferences alone or in combination with other parameters (such as power considerations or information from various sensors, historical data, and/or user preference) to calculate, determine, select and/or otherwise generate a tint value for the respective IGUs.
In some embodiments, the wall device transmits a tint state change request to the appropriate control system (e.g., to the NC therein). A lower level of the control system (e.g., to the NC therein) may communicate the request, or a communication based at least in part on the request, to a higher level of the control system (e.g., to the MC). For example, each wall device can be connected with a corresponding NC via one or more wired links. In some embodiments, the wall device transmits a request to the appropriate NC, which then itself determines whether to override a primary tint command previously received from the MC or a primary or secondary tint command previously generated by the NC. As described below, the NC may generate tint commands without first receiving a tint command from an MC. In some embodiments, the wall device communicates requests or control signals directly to the WC that controls the adjoining IGUs. For example, each wall device can be connected with a corresponding WC via one or more wired links such as those just described for the MC or via a wireless link.
In some embodiments, the NC or the MC determines whether the control signals from the wall device should take priority over a tint value previously generated by the NC or the MC. As described above, the wall device is able to communicate directly with the NC. However, in some examples, the wall device can communicate requests directly to the MC or directly to a WC, which then communicates the request to the NC. In some embodiments, the wall device is able to communicate requests to a customer-facing network (such as a network managed by the owners or operators of the building), which then passes the requests (or requests based therefrom) to the NC either directly or indirectly by way of the MC. For example, a control algorithm or rule set stored in and executed by the NC or the MC can dictate that one or more control signals from a wall device take precedence over a tint value previously generated by the NC or the MC. In some embodiments (e.g., such as in times of high demand), a control algorithm or rule set stored in and executed by the NC or the MC dictates that the tint value previously generated by the NC or the MC takes precedence over any control signals received from a wall device.
In some embodiments, based at least in part on the receipt of a tint-state-change request or control signal from a wall device, the NC can use information about a combination of known parameters to generate a tint value that provides lighting conditions desirable for a typical user. In some embodiments, the NC or the MC generates the tint value based at least in part on preset preferences defined by or for the particular user that requested the tint state change via the wall device. For example, the user may be required to enter a password into the wall device or to use a security token or security fob such as the IBUTTON or other 1-Wire device to gain access to the wall device. In this example, the NC can communicate with the MC to determine the identity of the user, or the MC can alone determine the identity of the user, based at least in part on the password, security token or security fob. The MC may then retrieve preset preferences for the user, and use the preset preferences alone or in combination with other parameters (such as power considerations or information from various sensors) to calculate, determine, select, or otherwise generate a tint value for the respective IGUs.
In some embodiments, the control system (e.g., the MC therein) is coupled to an external database (or “data store” or “data warehouse”). The database can be a local database coupled with the control system (e.g., the MC therein) via a wired hardware link, for example. In some embodiments, the database is a remote database or a cloud-based database accessible by the control system (e.g., the MC therein) via an internal private network or over the outward-facing network. Other computing devices, systems, or servers also can have access to read the data stored in the database, for example, over the outward-facing network. One or more control applications or third party applications could also have access to read the data stored in the database via the outward-facing network. In some embodiments, the control system (e.g., the MC therein) stores in the database a record of all tint commands including the corresponding tint values issued by the control system (e.g., the MC therein). The control system (e.g., the MC therein) may also collect status and sensor data and store it in the database (which may constitute historical data). The local controllers (e.g., WCs) may collect the sensor data and/or status data from the enclosure and/or from other devices (e.g., IGUs) or media disposed in the enclosure, and communicate the sensor data and/or status data to the respective higher level controller (e.g., NCs) over the communication link. The data may move up the control chain, e.g., to the MC. For example, the controllers (e.g., NCs or the MC) may themselves be communicatively coupled (e.g., connected) to various sensors (such as light, temperature, or occupancy sensors) within the building, as well as (e.g., light and/or temperature) sensors positioned on, around, or otherwise external to the building (for example, on a roof of the building). In some embodiments, the control system (e.g., the NCs or the WCs) may also transmit status and/or sensor data (e.g., directly) to the database for storage.
In some embodiments, the network system is suited for integration with a smart thermostat service, alert service (for example, fire detection), security service and/or other appliance automation service. On example of a home automation service is NEST®, made by Nest Labs of Palo Alto, California, (NEST® is a registered trademark of Google, Inc. of Mountain View, California). As used herein, references to a BMS can in some implementations also encompass, or be replaced with, such other automation services.
In some embodiments, the e control system (e.g., the MC therein) and a separate automation service, such as a BMS, can communicate via an application programming interface (API). For example, the API can execute in conjunction with a (e.g., master) controller application (or platform) within the controller (e.g., MC), and/or in conjunction with a building management application (or platform) within the BMS. The controller (e.g., MC) and the BMS can communicate over one or more wired links and/or via the outward-facing network. For example, the BMS may communicate instructions for controlling the IGUs to the controller (e.g., MC), which then generate and transmit primary status (e.g., tint) commands of the target to the appropriate lower level controller(s) (e.g., to the NCs). The lower hierarchical level controllers (e.g., the NCs or the WCs) could communicate directly with the BMS (e.g., through a wired/hardware link and/or wirelessly through a wireless data link). In some embodiments, the BMS also receives data, such as sensor data, status data, and associated timestamp data, collected by one or more of the controllers in the control system (e.g., by the MC, the NCs, and/or the WCs). For example, the controller (e.g., MC) can publish such data over the network. In some embodiments in which such data is stored in a database, the BMS can have access to some or all of the data stored in the database.
In some embodiments, the controller (e.g., “the MC”) collectively refers to any suitable combination of hardware, firmware and software for implementing the functions, operations, processes, or capabilities described. For example, the MC can refer to a computer that implements a master controller application (also referred to herein as a “program” or a “task”). For example, the controller (e.g., MC) may include one or more processors. The processor(s) can be or can include a central processing unit (CPU), such as a single core or a multi-core processor. The processor can additionally include a digital signal processor (DSP) or a network processor in some examples. The processor could also include one or more application-specific integrated circuits (ASICs). The processor is coupled with a primary memory, a secondary memory, an inward-facing network interface, and an outward-facing network interface. The primary memory can include one or more high-speed memory devices such as, for example, one or more random-access memory (RAM) devices including dynamic-RAM (DRAM) devices. Such DRAM devices can include, for example, synchronous DRAM (SDRAM) devices and double data rate SDRAM (DDR SDRAM) devices (including DDR2 SDRAM, DDR3 SDRAM, and DDR4 SDRAM), thyristor RAM (T-RAM), and zero-capacitor (Z-RAM®), among other suitable memory devices.
In some embodiments, the secondary memory can include one or more hard disk drives (HDDs) or one or more solid-state drives (SSDs). In some embodiments, the memory can store processor-executable code (or “programming instructions”) for implementing a multi-tasking operating system such as, for example, an operating system based at least in part on a Linux® kernel. The operating system can be a UNIX®- or Unix-like-based operating system, a Microsoft Windows®-based operating system, or another suitable operating system. The memory may also store code executable by the processor to implement the master controller application described above, as well as code for implementing other applications or programs. The memory may also store status information, sensor data, or other data collected from NCs, window controllers and various sensors.
In some embodiments, the controller (e.g., MC) is a “headless” system; that is, a computer that does not include a display monitor or other user input device. For example, an administrator or other authorized user can log in to or otherwise access the controller (e.g., MC) from a remote computer or mobile computing device over a network to access and retrieve information stored in the controller (e.g., MC), to write or otherwise store data in the controller (e.g., MC), and/or to control various: functions, operations, processes and/or parameters implemented or used by the controller (e.g., MC). The controller (e.g., MC) can include a display monitor and a direct user input device (for example, a mouse, a keyboard and/or a touchscreen).
In some embodiments, the inward-facing network interface enables one controller (e.g., MC) of the control system to communicate with various distributed controllers and/or various targets (e.g., sensors). The inward-facing network interface can collectively refer to one or more wired network interfaces and/or one or more wireless network interfaces (including one or more radio transceivers). For example, the inward-facing network interface can enable communication with downstream controllers (e.g., NCs) over the link. Downstream may refer to a lower level of control in the control hierarchy.
In some embodiments, the outward-facing network interface enables the controller (e.g., MC) to communicate with various computers, mobile circuitry (e.g., mobile devices), servers, databases, and/or cloud-based database systems, over one or more networks. The outward-facing network interface can collectively refer to one or more wired network interfaces and/or one or more wireless network interfaces (including one or more radio transceivers). In some embodiments, the various applications, including third party applications and/or cloud-based applications, executing within such remote devices can access data from or provide data to the controller (e.g., MC) or to the database via the controller (e.g., MC). For example, the controller (e.g., MC) may include one or more application programming interfaces (APIs) for facilitating communication between the controller (e.g., MC) and various third party applications. Some examples of APIs that controller(s) (e.g., MC) can enable can be found in PCT Patent Application No. PCT/US15/64555 (Attorney Docket No. VIEWP073WO) filed Dec. 8, 2015, and titled MULTIPLE INTERACTING SYSTEMS AT A SITE, which is incorporated herein by reference in its entirety. For example, third-party applications can include various monitoring services including thermostat services, alert services (e.g., fire detection), security services, and/or other appliance automation services. Additional examples of monitoring services and systems can be found in PCT Patent Application No. PCT/US2015/019031 (Attorney Docket No. VIEWP061WO) filed Mar. 5, 2015 and titled MONITORING SITES CONTAINING SWITCHABLE OPTICAL DEVICES AND CONTROLLERS, which is incorporated herein by reference in its entirety.
In some embodiments, one or both of the inward-facing network interface and the outward-facing network interface can include a Building Automation and Control network (BACnet) compatible interface. BACnet is a communications protocol typically used in building automation and control networks and defined by the ASHRAE/ANSI 135 and ISO 16484-5 standards. The BACnet protocol broadly provides mechanisms for computerized building automation systems and devices to exchange information, e.g., regardless of the particular services they perform. For example, BACnet can be used to enable communication among (i) heating, ventilating, and air-conditioning control (HVAC) systems, (ii) lighting control systems, (iii) access and/or security control systems, (iv) fire detection systems, or (v) any combination thereof, as well as their associated equipment. In some examples, one or both of the inward-facing network interface and the outward-facing network interface can include an oBIX (Open Building Information Exchange) compatible interface or another RESTful Web Services-based interface.
In some embodiments, the controller (e.g., MC) can calculate, determine, select and/or otherwise generate a preferred state for the target (e.g., a tint value for one or more IGUs) based at least in part on a combination of parameters. For example, the combination of parameters can include time and/or calendar information such as the time of day, day of year or time of season. The combination of parameters may include solar calendar information such as, for example, the direction of the sun relative to the facility and/or target (e.g., IGUs). The direction of the sun relative to the facility and/or target (e.g., IGUs) may be determined by the controller (e.g., MC) based at least in part on time and/or calendar information, e.g., together with information known about the geographical location of the facility (e.g., building) on Earth and the direction that the target (e.g., IGUs) faces (e.g., in a North-East-Down coordinate system). The combination of parameters also can include exterior and/or interior environmental conditions. For example, the outside temperature (external to the building), the inside temperature (within a room adjoining the target IGUs), or the temperature within the interior volume of the IGUs. The combination of parameters may include information about the weather (for example, whether it is clear, sunny, overcast, cloudy, raining or snowing). Parameters such as the time of day, day of year, and/or direction of the sun can be programmed into and tracked by the control system (e.g., the MC therein). Parameters (such as the outside temperature, inside temperature, and/or IGU temperature) can be obtained from sensors in, on or around the building or sensors integrated with the target (e.g., on or within the IGUs). At times the target can comprise a sensor. Examples of algorithms, routines, modules, or other means for generating IGU tint values are described in U.S. patent application Ser. No. 13/772,969, filed Feb. 21, 2013 and titled CONTROL METHOD FOR TINTABLE WINDOWS, and in PCT Patent Application No. PCT/US15/029675, filed May 7, 2015 and titled CONTROL METHOD FOR TINTABLE WINDOWS, each of which is hereby incorporated by reference in its entirety.
In some embodiments, at least one (e.g., each) device (e.g., ECD) within each IGU is capable of being tinted, e.g., responsive to a suitable driving voltage applied across the EC stack. The tint may be to (e.g., virtually) any tint state within a continuous tint spectrum defined by the material properties of the EC stack. However, the control system (e.g., the MC therein) may be programmed to select a tint value from a finite number of discrete tint values (e.g., tint values specified as integer values). In some such implementations, the number of available discrete tint values can be at least 2, 4, 8, 16, 32, 64, 128 or 256, or more. For example, a 2-bit binary number can be used to specify any one of four possible integer tint values, a 3-bit binary number can be used to specify any one of eight possible integer tint values, a 4-bit binary number can be used to specify any one of sixteen possible integer tint values, a 5-bit binary number can be used to specify any one of thirty-two possible integer tint values, and so on. At least one (e.g., each) tint value can be associated with a target tint level (e.g., expressed as a percentage of maximum tint, maximum safe tint, and/or maximum desired or available tint). For didactic purposes, consider an example in which the MC selects from among four available tint values: 0, 5, 10 and 15 (using a 4-bit or higher binary number). The tint values 0, 5, 10 and 15 can be respectively associated with target tint levels of 60%, 40%, 20% and 4%, or 60%, 30%, 10% and 1%, or another desired, advantageous, or suitable set of target tint levels.
In some embodiments, in some implementations the MC and the NC are implemented as a master controller application and a network controller application, respectively, executing within respective physical computers or other hardware devices. For example, each of the master controller application and the network controller application can be implemented within the same physical hardware. Each of the master controller application and the network controller application can be implemented as a separate task executing within a single computer device that includes a multi-tasking operating system such as, for example, an operating system based at least in part on a Linux® kernel or another suitable operating system.
In some embodiments, the master controller application and the network controller application can communicate via an application programming interface (API). In some embodiments, the master controller and network controller applications communicate over a loopback interface. By way of reference, a loopback interface is a virtual network interface, implemented through an operating system, which enables communication between applications executing within the same device. A loopback interface is typically identified by an IP address (often in the 127.0.0.0/8 address block in IPv4, or the 0:0:0:0:0:0:0:1 address (also expressed as: 1) in IPv6). For example, the master controller application and the network controller application can each be programmed to send communications targeted to one another to the IP address of the loopback interface. In this way, when the master controller application sends a communication to the network controller application, or vice versa, the communication does not need to leave the computer.
In some embodiments wherein the MC and the NC are implemented as master controller and network controller applications, respectively, there are generally no restrictions limiting the available protocols suitable for use in communication between the two applications. This generally holds true regardless of whether the master controller application and the network controller application are executing as tasks within the same or different physical computers. For example, there is no need to use a broadcast communication protocol, such as BACnet, which limits communication to one network segment as defined by a switch or router boundary. For example, the oBIX communication protocol can be used in some implementations for communication between the MC and the NCs.
In some embodiments, each of the NCs is implemented as an instance of a network controller application executing as a task within a respective physical computer. In some embodiments, at least one of the computers executing an instance of the network controller application also executes an instance of a master controller application to implement the MC. For example, while only one instance of the master controller application may be actively executing in the network system at any given time, two or more of the computers that execute instances of network controller application can have an instance of the master controller application installed. In this way, redundancy is added such that the computer currently executing the master controller application is no longer a single point of failure of the entire system. For example, if the computer executing the master controller application fails or if that particular instance of the master controller application otherwise stops functioning, another one of the computers having an instance of the master network application installed can begin executing the master controller application to take over for the other failed instance. In some embodiments, more than one instance of the master controller application may execute concurrently. For example, the functions, processes, or operations of the master controller application can be distributed to two (or more) instances of the master controller application.
In some embodiments, when the MC determines to tint one or more IGUs, the MC writes a specific tint value to the AV in the NC associated with the one or more respective WCs that control the target IGUs. For example, the MC may generate a primary tint command communication including a BACnet ID associated with the WCs that control the target IGUs. The primary tint command also can include a tint value for the target IGUs. The MC may direct the transmission of the primary tint command to the NC using a network address such as, for example, an IP address or a MAC address. Responsive to receiving such a primary tint command from the MC through the upstream interface, the NC may unpackage the communication, map the BACnet ID (or other first protocol ID) in the primary tint command to one or more CAN IDs (or other second protocol IDs), and write the tint value from the primary tint command to a first one of the respective AVs associated with each of the CAN IDs.
In some embodiments, the NC then generates a secondary tint command for each of the WCs identified by the CAN IDs. Each secondary tint command may be addressed to a respective one of the WCs by way of the respective CAN ID. For example, each secondary tint command also can include the tint value extracted from the primary tint command. The NC may transmit the secondary tint commands to the target WCs through the downstream interface via a second communication protocol (for example, via the CANOpen protocol). In some embodiments, when a WC receives such a secondary tint command, the WC transmits a status value back to the NC indicating a status of the WC. For example, the tint status value can represent a “tinting status” or “transition status” indicating that the WC is in the process of tinting the target IGUs, an “active” or “completed” status indicating that the target IGUs are at the target tint state or that the transition has been finished, or an “error status” indicating an error. After the status value has been stored in the NC, the NC may publish the status information or otherwise make the status information accessible to the MC or to various other authorized computers or applications. In some embodiments, the MC requests status information for a particular WC from the NC based at least in part on intelligence, a scheduling policy, or a user override. For example, the intelligence can be within the MC or within a BMS. A scheduling policy can be stored in the MC, another storage location within the network system, or within a cloud-based system.
In some embodiments, the NC handles some of the functions, processes, or operations that are described above as being responsibilities of the MC. In some embodiments, the NC can include additional functionalities or capabilities not described with reference to the MC. For example, the NC may also include a data logging module (or “data logger”) for recording data associated with the IGUs controlled by the NC. In some embodiments, the data logger records the status information included in each of some or all of the responses to the status requests. For example, the status information that the WC communicates to the NC responsive to each status request can include a tint status value (S) for the IGUs, a value indicating a particular stage in a tinting transition (for example, a particular stage of a voltage control profile), a value indicating whether the WC is in a sleep mode, a tint value (C), a set point voltage set by the WC based at least in part on the tint value (for example, the value of the effective applied voltage VEff), an actual voltage level VAct measured, detected or otherwise determined across the ECDs within the IGUs, an actual current level IAct measured, detected or otherwise determined through the ECDs within the IGUs, and various sensor data, for example, collected from photosensors or temperature sensors integrated on or within the IGUs. The NC 500 may collect and queue status information in a messaging queue like RabbitMC, ActiveMQ or Kafka and stream the status information to the MC for subsequent processing such as data reduction/compression, event detection, etc., as further described herein.
In some embodiments, the data logger within the NC collects and stores the various information received from the WCs in the form of a log file such as a comma-separated values (CSV) file or via another table-structured file format. For example, each row of the CSV file can be associated with a respective status request, and can include the values of C, S, VEff, VAct and IAct as well as sensor data (or other data) received in response to the status request. In some implementations, each row is identified by a timestamp corresponding to the respective status request (for example, when the status request was sent by the NC, when the data was collected by the WC, when the response including the data was transmitted by the WC, or when the response was received by the NC). In some embodiments, each row also includes the CAN ID or other ID associated with the respective WC.
In some embodiments, each row of the CSV file includes the requested data for all of the WCs controlled by the NC. The NC may sequentially loop through all of the WCs it controls during each round of status requests. In some embodiments, each row of the CSV file is identified by a timestamp (for example, in a first column), but the timestamp can be associated with a start of each round of status requests, rather than each individual request. In one specific example, columns 2-6 can respectively include the values C, S, VEff, VAct and IAct for a first one of the WCs controlled by the NC, columns 7-11 can respectively include the values C, S, VEff, VAct and IAct for a second one of the WCs, columns 12-16 can respectively include the values C, S, VEff, VAct and IAct for a third one of the WCs, and so on and so forth through all of the WCs controlled by the NC. The subsequent row in the CSV file may include the respective values for the next round of status requests. In some embodiments, each row also includes sensor data obtained from photosensors, temperature sensors, or other sensors integrated with the respective IGUs controlled by each WC. For example, such sensor data values can be entered into respective columns between the values of C, S, VEff, VAct and IAct for a first one of the WCs but before the values of C, S, VEff, VAct and IAct for the next one of the WCs in the row. Each row can include sensor data values from one or more external sensors, for example, positioned on one or more facades or on a rooftop of the building. The NC may send a status request to the external sensors at the end of each round of status requests.
In some embodiments, the NC translates between various upstream and downstream protocols, for example, to enable the distribution of information between WCs and the MC or between the WCs and the outward-facing network. For example, the NC may include a protocol conversion module responsible for such translation or conversion services. The protocol conversion module may be programmed to perform translation between any of a number of upstream protocols and any of a number of downstream protocols. For example, such upstream protocols can include UDP protocols such as BACnet, TCP protocols such as oBix, other protocols built over these protocols as well as various wireless protocols. Downstream protocols can include, for example, CANopen, other CAN-compatible protocol, and various wireless protocols including, for example, protocols based at least in part on the IEEE 802.11 standard (for example, WiFi), protocols based at least in part on the IEEE 802.15.4 standard (for example, ZigBee, 6LoWPAN, ISA100.11a, WirelessHART or MiWi), protocols based at least in part on the Bluetooth standard (including the Classic Bluetooth, Bluetooth high speed and Bluetooth low energy protocols and including the Bluetooth v4.0, v4.1 and v4.2 versions), or protocols based at least in part on the EnOcean standard (ISO/IEC 14543-3-10).
In some embodiments, the NC uploads the information logged by the data logger (for example, as a CSV file) to the MC on a periodic basis, for example, every 24 hours. For example, the NC can transmit a CSV file to the MC via the File Transfer Protocol (FTP) or another suitable protocol over an Ethernet data link 316. The status information may be stored in a database or made accessible to applications over the outward-facing network.
In some embodiments, the NC includes functionality to analyze the information logged by the data logger. For example, an analytics module can be provided in the NC to receive and/or analyze the raw information logged by the data logger (e.g., in real time). In real time may include within at most 15 seconds (sec.), 30 sec., 45 sec., 1 minute (min), 2 min., 3 min. 4 min., 5 min, 10 min., 15 min. or 30 min from receipt of the logged information by the data logger, and/or from initiation of the operation (e.g., from receipt and/or from start of analysis). In some embodiments, the analytics module is programmed to make decisions based at least in part on the raw information from the data logger. In some embodiments, the analytics module communicates with the database to analyze the status information logged by the data logger after it is stored in the database. For example, the analytics module can compare raw values of electrical characteristics such as VEff, VAct and IAct with expected values or expected ranges of values and flag special conditions based at least in part on the comparison. For example, such flagged conditions can include power spikes indicating a failure such as a short, an error, or damage to an ECD. The analytics module may communicate such data to a tint determination module or to a power management module in the NC.
In some embodiments, the analytics module filters the raw data received from the data logger to more intelligently or efficiently store information in the database. For example, the analytics module can be programmed to pass only “interesting” information to a database manager for storage in the database. For example, interesting information can include anomalous values, values that otherwise deviate from expected values (such as based at least in part on empirical or historical values), or for specific periods when transitions are happening. Examples of data manipulation (e.g., filtering, parsing, temporarily storing, and efficiently storing long term in a database) can be found in PCT Patent Application No. PCT/US15/029675 (Attorney Docket No. VIEWP049X1WO) filed May 7, 2015 and titled CONTROL METHOD FOR TINTABLE WINDOWS that is hereby incorporated by reference in its entirety.
In some embodiments, a database manager module (or “database manager”) in the control system (e.g., in the NC) is configured to store information logged by the data logger to a database on a periodic basis, for example, at least every hour, every few hours, or every 24 hours. The database can be an external database such as the database described above. In some embodiments, the database can be internal to the controller (e.g., the NC). For example, the database can be implemented as a time-series database such as a Graphite database within the secondary memory of the controller (e.g., of the NC) or within another long term memory within the controller (e.g., the NC). For example, the database manager can be implemented as a Graphite Daemon executing as a background process, task, sub-task or application within a multi-tasking operating system of the controller (e.g., the NC). A time-series database can be advantageous over a relational database such as SQL because a time-series database is more efficient for data analyzed over time.
In some embodiments, the database can collectively refer to two or more databases, each of which can store some or all of the information obtained by some or all of the NCs in the network system. For example, it can be desirable to store copies of the information in multiple databases for redundancy purposes. The database can collectively refer to a multitude of databases, each of which is internal to a respective controller (e.g., NC), e.g., such as a Graphite or other times-series database. It can be beneficial to store copies of the information in multiple databases such that requests for information from applications including third party applications can be distributed among the databases and handled more efficiently. For example, the databases can be periodically or otherwise synchronized, e.g., to maintain consistency.
In some embodiments, the database manager filters data received from the analytics module to more intelligently and/or efficiently store information, e.g., in an internal and/or external database. For example, the database manager can be programmed to store (e.g., only) “interesting” information to a database. Interesting information can include anomalous values, values that otherwise deviate from expected values (such as based at least in part on empirical or historical values), and/or for specific periods when transitions are happening. More detailed examples of how data manipulation (e.g., how raw data can be filtered, parsed, temporarily stored, and efficiently stored long term in a database) can be found in PCT Patent Application No. PCT/US15/029675 (Attorney Docket No. VIEWPO49X1WO) filed May 7, 2015 and titled CONTROL METHOD FOR TINTABLE WINDOWS that is hereby incorporated by reference herein in its entirety.
In some embodiments, a status determination module of a target is included in the controller (e.g., the NC, the MC, or the WC), e.g., for calculating, determining, selecting, or otherwise generating status values for the target. For example, a tint determination module can be included in the controller (e.g., the NC, the MC, or the WC) for calculating, determining, selecting, or otherwise generating tint values for the IGUs. For example, the status (e.g., tint) determination module can execute various algorithms, tasks, or subtasks to generate tint values based at least in part on a combination of parameters. The combination of parameters can include, for example, the status information collected and stored by the data logger. The combination of parameters also can include time or calendar information such as the time of day, day of year or time of season. The combination of parameters can include solar calendar information such as, for example, the direction of the sun relative to the target (e.g., IGUs). The combination of parameters can include one or more characteristics of the enclosure environment that comprise gaseous concentration (e.g., VOC, humidity, carbon dioxide, or oxygen), debris, gas type, gas flow velocity, gas flow direction, gas (e.g., atmosphere) temperature, noise level, or light level (e.g., brightness). The combination of parameters can include the outside parameters (e.g., temperature) external to the enclosure (e.g., building), the inside parameter (e.g., temperature) within the enclosure (e.g., a room adjoining the target IGUs), and/or the temperature within the interior volume of the IGUs. The combination of parameters can include information about the weather (for example, whether it is clear, sunny, overcast, cloudy, raining or snowing). Parameters such as the time of day, day of year, and/or direction of the sun, can be programmed into and tracked by the control system (e.g., that includes the NC). Parameters such as the outside temperature, inside temperature, and/or IGU temperature, can be obtained from sensors in, on or around the building or sensors integrated on or within the IGUs, for example. In some embodiments, various parameters are provided by, or determined based at least in part on, information provided by various applications including third party applications that can communicate with the controller(s) (e.g., NC) via an API. For example, the network controller application, or the operating system in which it runs, can be programmed to provide the API.
In some embodiments, the target status (e.g., tint) determination module determines status (e.g., tint) value(s) of the target based at least in part on user overrides, e.g., received via various mobile circuitry (e.g., device) applications, wall devices and/or other devices. In some embodiments, the status (e.g., tint) determination module determines status (e.g., tint) values based at least in part on command(s) or instruction(s) received by various applications, e.g., including third party applications and/or cloud-based applications. For example, such third party applications can include various monitoring services including thermostat services, alert services (e.g., fire detection), security services and/or other appliance automation services. Additional examples of monitoring services and systems can be found in PCT/US2015/019031 (Attorney Docket No. VIEWP061WO) filed 5 Mar. 2015 and titled MONITORING SITES CONTAINING SWITCHABLE OPTICAL DEVICES AND CONTROLLERS that is incorporated herein by reference in its entirety. Such applications can communicate with the status (e.g., tint) determination module and/or other modules within the controller(s) (e.g., NC) via one or more APIs. Some examples of APIs that the controller(s) (e.g., NC) can enable are described in PCT Patent Application No. PCT/US15/64555 (Attorney Docket No. VIEWP073WO) filed Dec. 8, 2015 and titled MULTIPLE INTERFACING SYSTEMS AT A SITE, that is incorporated herein by reference in its entirety.
In some embodiments, the analytics module compares values of VEff, VAct and IAct as well as sensor data obtained in real time and/or previously stored within the database with expected values or expected ranges of values and flag special conditions based at least in part on the comparison. For example, the analytics module can pass such flagged data, flagged conditions or related information to a power management module. For example, such flagged conditions can include power spikes indicating a short, an error, or damage to a smart window (e.g., an ECD). In some embodiments, the power management module modifies operations based at least in part on the flagged data or conditions. For example, the power management module can delay status (e.g., tint) commands of a target until power demand has dropped, stop commands to troubled controller(s) (e.g., local controller such as WC) (and put them in idle state), start staggering commands to controllers (e.g., lower hierarchy controllers such as WCs), manage peak power, and/or signal for help.
In some embodiments, a controller (e.g., WC) or other network device includes a sensor or sensor ensemble. For example, a plurality of sensors or a sensor ensemble may be organized into a sensor module. A sensor ensemble may comprise a circuit board, such as a printed circuit board, e.g., in which a number of sensors are adhered or affixed to the circuit board. Sensor(s) can be removed from a sensor module. For example, a sensor may be plugged into and/or unplugged out of, the circuit board. Sensor(s) may be individually activated and/or deactivated (e.g., using a switch). The circuit board may comprise a polymer. The circuit board may be transparent or non-transparent. The circuit board may comprise metal (e.g., elemental metal and/or metal alloy). The circuit board may comprise a conductor. The circuit board may comprise an insulator. The circuit board may comprise any geometric shape (e.g., rectangle or ellipse). The circuit board may be configured (e.g., may be of a shape) to allow the ensemble to be disposed in frame portion such as a mullion (e.g., of a window). The circuit board may be configured (e.g., may be of a shape) to allow the ensemble to be disposed in a frame (e.g., door frame and/or window frame). The frame may comprise one or more holes, e.g., to allow the sensor(s) to obtain (e.g., accurate) readings. The circuit board may be enclosed in a wrapping. The wrapping may comprise flexible or rigid portions. The wrapping may be flexible. The wrapping may be rigid (e.g., be composed of a hardened polymer, from glass, or from a metal (e.g., comprising elemental metal or metal alloy). The wrapping may comprise a composite material. The wrapping may comprise carbon fibers, glass fibers, and/or polymeric fibers. The wrapping may have one or more holes, e.g., to allow the sensor(s) to obtain (e.g., accurate) readings. The circuit board may include an electrical connectivity port (e.g., socket). The circuit board may be connected to a power source (e.g., to electricity). The power source may comprise renewable and/or non-renewable power source.
In some embodiments, an increase in the number and/or types of sensors may be used to increase a probability that one or more measured property is accurate and/or that a particular event measured by one or more sensor has occurred. In some embodiments, sensors of sensor ensemble may cooperate with one another. In an example, a radar sensor and/or lidar sensor of sensor ensemble may determine presence of a number of individuals in an enclosure. A processor (e.g., processor 615) may determine that detection of presence of a number of individuals in an enclosure is positively correlated with an increase in carbon dioxide concentration. In an example, the processor-accessible memory may determine that an increase in detected infrared energy is positively correlated with an increase in temperature as detected by a temperature sensor. In some embodiments, network interface (e.g., 650) may communicate with other sensor ensembles similar to sensor ensemble. The network interface may additionally communicate with a controller.
Individual sensors (e.g., sensor 610A, sensor 610D, etc.) of a sensor ensemble may comprise and/or utilize at least one dedicated processor. A sensor ensemble may utilize a remote processor (e.g., 654) utilizing a wireless and/or wired communications link. A sensor ensemble may utilize at least one processor (e.g., processor 652), which may represent a cloud-based processor coupled to a sensor ensemble via the cloud (e.g., 650). Processors (e.g., 652 and/or 654) may be located in the same building, in a different building, in a building owned by the same or different entity, a facility owned by the manufacturer of the window/controller/sensor ensemble, or at any other location. In various embodiments, as indicated by the dotted lines of
Individual sensors (e.g., sensor 610A, sensor 610D, etc.) may comprise an imaging system capable of capturing one or more images of visible light, infrared (IR), ultraviolet (UV), and/or radar reflections. In some embodiments, an imaging system may comprise an IR camera used to capture identifying gesture and/or gait recognition data of a user while preserving the user's privacy. The field of view (FOV) of the camera may be wide-angle (e.g., 30°, 45°, 60°, 90°, 120°, 145°, or 180°) or telephoto (12°, 8°, 6°, or 3°). The image resolution format of the camera may be, for example, 160×120, 240×180, 480p, 720p, 1080p, 1440p, 1.0 megapixel (MP), 1.3 MP, 2 MP, 3 MP, 5 MP, 8 MP, 12 MP, 16 MP, 20 MP, 33 MP, 2K, 4K, or 8K. An IR camera may perform thermal imaging between a low-temperature threshold (e.g., −50° C., −40° C., −30° C., −20° C., 0° C., 10° C., 20° C., 30° C., or 40° C.) and a high-temperature threshold (e.g., 100° C., 120° C., 150° C., 200° C., 300° C., 400° C., 500° C., or 600° C.). The spectral range of the IR camera may vary (e.g., 3 μm to 5 μm, or 8 μm to 14 μm), and the frame rate (e.g., frames per second (FPS)) may also vary (e.g., 15 FPS, 24 FPS, 30 FPS, 32 FPS, 48 FPS, 50 FPS, or 60 FPS). In some embodiments, an imaging system may comprise a radar imaging system to capture identifying gesture and/or gait recognition data of a user while preserving the user's privacy. A radar imaging system may scan a volume of space using, for example, one or more antenna arrays (e.g., a transmitter and receiver antenna arrays), and process the radar data from the scan to form a 2D image (e.g., from azimuth and elevation) or 3D image (e.g., from azimuth, elevation, and range). User motion may be determined using Doppler information from a single image (e.g., often called a “4D image”) and/or detected movement between images. Similar to a visible-light or IR camera, the FOV of the radar imaging system may be relatively wide (e.g., 30°, 45°, 60°, 90°, 120°, 145°, 180°, 270°, or 360°) or narrow (e.g., 12°, 8°, 6°, or 3°) in either or both azimuth and elevation. Range, too, may vary (e.g., a maximum depth of 10 m, 15 m, 25 m, 50 m, 100 m, or 300 m), as well as Doppler (e.g., ±1 m/s, 5 m/s, 10 m/s, 25 m/s, 50 m/s, 75 m/s, 100 m/s, 130 m/s, 150 m/s, 175 m/s, or 200 m/s). Resolution of the radar imaging system may vary in either or both azimuth and elevation (e.g., 0.5°, 0.75°, 1°, 1.5°, 2°, 3°, 5°, or 10°), range (which may be dependent on range: e.g., 10 cm @ 10 m versus 50 cm @ 300 m), and Doppler (e.g., 0.01 m/s, 0.05 m/s, 0.1 m/s, 0.5 m/s, 1 m/s, 1.5 m/s, 2 m/s, 3 m/s, 5 m/s, or 10 m/s). The radar imaging system may use any of a variety of frequencies or frequency ranges, including those used for cellular communication (e.g., 800 MHz, 1900 MHz, 2.1 GHz, 2.3 GHz, 2.6 GHz, 3.5 GHz, 4.8 GHz, 6 GHz, 26 GHz, 28 GHz, 40 GHz, 66 GHz, or 71 GHz), Wi-Fi (1000 MHz, 2.4 GHz, 3.6 GHz, 4.9 GHz, 5 GHz, 5.9 GHz, 6 GHz and 60 GHz), imaging radar-specific frequencies (e.g., 4 GHz, 8 GHz, 77 GHz, 94 GHz, 160 GHz, 300 GHz, 580 GHz, or 600 GHz), or other wireless technologies, including those described hereafter. In some embodiments, an imaging system may comprise a lidar sensor used to capture external behavior data, including identifying gesture and/or gait recognition data of a user while preserving the user's privacy. The lidar sensor may comprise traditional lidar, microelectromechanical system (MEMS)-based lidar, solid-state lidar, or the like. The field of view (FOV) of the lidar sensor may be wide-angle (e.g., 30°, 45°, 60°, 90°, 120°, 145°, 180° or 360°) or narrow (12°, 8°, 6°, or 3°), which may be based, at least in part, on the underlying technology (e.g., traditional lidar vs. solid-state lidar). In some embodiments, the FOV may be dynamic, based on one or more detected objects and/or regions of interest. Horizontal FOV may be different than vertical FOV. The resolution (e.g., angular resolution) in horizontal and/or vertical directions may be, for example, 3°, 2°, 1°, 0.5°, 0.1°, 0.05°, or 0.01°, which may be selected based on a distance at which a target (e.g., person) is expected to be from the lidar sensor. (Higher resolutions may be used for targets at farther distances.) The frame rate of each lidar scan may vary (e.g., 2.5 Hz, 5 Hz, 10 Hz, 15 Hz, 20 Hz, 30 Hz, or 60 Hz).
In some embodiments, sensor data is exchanged among various network devices and controllers. The sensor data may also be accessible to remote users (e.g., inside or outside the same building) for retrieval using personal electronic devices, for example. Applications executing on remote devices to access sensor data may also provide commands for controllable functions such as tint commands for a window controller. An example window controller(s) is described in PCT Patent Application No. PCT/US16/58872, titled CONTROLLERS FOR OPTICALLY-SWITCHABLE DEVICES, filed Oct. 26, 2016, and in U.S. patent application Ser. No. 15/334,832, titled CONTROLLERS FOR OPTICALLY-SWITCHABLE DEVICES, filed Oct. 26, 2016, each of which is herein incorporate by reference in its entirety.
In some embodiments, the controller (e.g., NC) periodically requests status information from lower hierarchy controller(s) (e.g., from the WCs it controls). For example, the controller (e.g., NC) can communicate a status request to at least one (e.g., each) of the lower hierarchy controller(s) (e.g., from the WCs it controls) at a frequency of at least every few seconds, every few tens of seconds, every minute, every few minutes, or after any requested period of time. In some embodiments, at least one (e.g., each) status request is directed to a respective one of the lower hierarchy controllers (e.g., WCs) using the CAN ID or other identifier of the respective lower hierarchy controller(s) (e.g., WCs). In some embodiments, the controller (e.g., NC) proceeds sequentially through all of the lower hierarchy controllers (e.g., WCs) it controls during at least one (e.g., each) round of status acquisition. The controller (e.g., NC) can loop through at least two (e.g., all) of the lower hierarchy controllers (e.g., WCs) it controls such that a status request is sent to these lower hierarchy controllers (e.g., WCs) sequentially in the round of status acquisition. After a status request has been sent to a given lower hierarchy controller (e.g., WC), the upper hierarchy level controller (e.g., NC) may waits to receive the status information from one lower hierarchy controller (e.g., WC), e.g., before sending a status request to the next one of the lower hierarchy controller (e.g., WC) in the round of status acquisition.
In some embodiments, after status information has been received from all of the lower hierarchy controllers (e.g., WCs) that the upper hierarchy controller (e.g., NC) controls, the upper hierarchy controller (e.g., NC) performs a round of status change (e.g., tint) command distribution to the target (e.g., to the IGU). For example, in some implementations, at least one (e.g., each) round of status acquisition is followed by a round of tint command distribution, which is then followed by a next round of status acquisition and a next round of tint command distribution, and so on. In some embodiments, during a round of status (e.g., tint) command distribution to the controller of the target, the controller (e.g., NC) proceeds to send a tint command to the lower hierarchy controller (e.g., WC) that the higher hierarchy controller (e.g., NC) controls. In some embodiments, the hierarchy controller (e.g., NC) proceeds sequentially through all of the lower hierarchy controllers (e.g., WCs) it controls during the round of tint command distribution. In other words, the hither hierarchy (e.g., NC) controller loops through (e.g., all of) the lower hierarchy controllers (e.g., WCs) it controls such that a status (e.g., tint) command is sent to (e.g., each of) the lower hierarchy controllers (e.g., WCs) sequentially in the round of status (e.g., tint) command distribution to change the status of the target (e.g., change the tint state of the IGU).
In some embodiments, a status request includes one or more instructions indicating what status information is being requested from the respective lower hierarchy controller (e.g., local controller such as a WC). In some embodiments, responsive to the receipt of such a request, the respective lower hierarchy controllers (e.g., WC) responds by transmitting the requested status information to the higher hierarchy controller (e.g., NC) (e.g., via the communication lines in an upstream set of cables). In some other embodiments, each status request by default causes the lower hierarchy controllers (e.g., WC) to transmit a predefined set of information for the set of targets (e.g., IGUs, sensors, emitters, or media) it controls. The status information that the lower hierarchy controllers (e.g., WC) communicates to the upper hierarchy controller (e.g., NC) responsive to the status request, can include a (e.g., tint) status value (S) for the target (e.g., IGUs). For example, indicating whether the targets (e.g., IGUs) is undergoing a status change (e.g., tinting transition) or has finished a status change (e.g., tinting transition, or light intensity change). The tint status value S or another value can indicate a particular stage in a tinting transition (for example, a particular stage of a voltage control profile). In some embodiments, the status value S or another value indicates whether the lower hierarchy controller (e.g., WC) is in a sleep mode. The status information communicated in response to the status request also can include the status (e.g., tint) value (C) for the target (e.g., IGUs), for example, as set by the controller (e.g., MC or the NC). The response also can include a set point voltage set by the lower hierarchy controller (e.g., WC) based at least in part on the status (e.g., tint) value (e.g., the value of the effective applied VEff). In some embodiments, the response includes a near real-time actual voltage level VAct measured, detected, or otherwise determined across the ECDs within the IGUs (for example, via the amplifier and the feedback circuit). In some embodiments, the response includes a near real-time actual current level IAct measured, detected, or otherwise determined through the ECDs within the IGUs (for example, via the amplifier and the feedback circuit). The response also can include various near real-time sensor data, for example, collected from photosensors or temperature sensors integrated on or within the IGUs.
In some embodiments, external behavior recognition such as gait recognition or gesture control, and/or voice control, may be used to interact with a target (e.g., an optically switchable device). Such control methods may be more convenient compared to more conventional control methods, e.g., that may require a user to touch or otherwise physically interact with a particular component (e.g., switch, knob, keypad, touchscreen, etc.). Voice control may be beneficial for users, e.g., with certain disabilities.
In some embodiments, external behavior recognition and/or voice control is used to implement any type of manipulation of a target (e.g., any type of command on an optically switchable device). For example, external behavior recognition and/or voice control may be used to implement tinting commands for a target, or for a group or zone of targets. For example, the command may be for a single optically switchable device (e.g., “change window 1 to tint 4” or “make window 1 darker”), or for a group or zone of optically switchable devices (e.g., “change the windows in zone 1 to tint 4” or “make the windows in zone 1 darker” or “make the windows in zone 1 much darker,” etc.). The commands may relate to discrete optical states to which the relevant optically switchable device(s) should change (e.g., discrete tint levels, or other discrete optical states) or relative changes in the optical states of the optically switchable device(s) (e.g., darker, lighter, more reflective, less reflective, e.g., or “my office is too dark, please lighten it up” or “I want to run the projector,” (letting the system know to darken the room) or “it's hot in here” (letting the system know to darken the windows and block heat gain) etc.). Where relative changes are used, the control system may be designed and/or configured to implement incremental (e.g., step) changes (e.g., 10% darker or lighter) in the optical state of the optically switchable device to carry out the command. The degree of each incremental (e.g., step) change may be pre-defined. In some embodiments, the control system is designed and/or configured to implement incremental (e.g., step) changes of a size and/or degree specified by the user. Such command(s) may be modified by any relative words used in the command (e.g., “very” or “a little bit,” or “lighter” or “darker” etc.).
In some embodiments, voice control is also be used to set a schedule for the target (e.g., optically switchable device). For example, a user may direct the optically switchable device(s) to tint at particular times/days (e.g., “make the windows in zone 1 go to tint 4 at 2 μm Monday through Friday” or “the morning sun makes it hot in here” (letting the system know to tint the windows during the morning hours when the sun impinges on that side of the building) or “I can't see the mountains well in the afternoon” (letting the system know that the windows are tinted too much in the afternoon and to lighten them during the afternoon)). Similarly, voice control can be used to implement tinting rules for the optically switchable device (e.g., “tint the windows in zone 1 to tint 4 when it's sunny outside” or “tint the windows in this room if the temperature inside this room is above 70° F.”). In some embodiments, any rules that can be implemented on a network of optically switchable devices (including any other networked components such as thermostat, BMS, electronic device, etc.) can be initiated via voice control.
In some embodiments, voice control is implemented on various components of control architecture for the target (e.g., smart window system), e.g., onboard window controllers or other window controllers, NCs, MCs, wall switches (e.g., interfaces with control components) and/or a separate device that interfaces with any or all of the aforementioned devices and/or components.
In some embodiments, gesture control is used to control the target. The gesture control may or may not use a limited command set (e.g., at times due to a lesser number of movements that would need to be recognized compared to the more expansive dictionary of words that can be recognized when using voice control). For example, gesture control can be used to implement many types of commands. For example, gesture control can be used to indicate that a particular target (e.g., window) or group of targets (e.g., windows) should change their state (e.g., change to a lighter or darker state (or other optical states if non-electrochromic optically switchable devices are used)). The user may indicate the target(s) (e.g., window(s)) to be changed, e.g., by standing in front of the relevant target(s) (e.g., window(s)) and/or pointing to the relevant target(s) (e.g., window(s)). Indication of the target may trigger coupling of the gesture with the target. The user may indicate the desired change by raising or lowering their hands or arms, or by opening or closing their palms, for instance. A dictionary of recognized gestures may be created to define the types of commands that can be accomplished via gesture control. More expansive gesture dictionaries may enable finer, more complex control of the optically switchable devices. There may be some degree of tradeoff in terms of ease of use, with smaller gesture dictionaries being potentially easier for users to master.
In some embodiments, gait recognition is used to control the target. Gait recognition may be used to identify a user by his or her gait, which may then give the user access to a command set for controlling an object using voice and/or gesture control. Using gait recognition, for example, a user may be identified upon walking into a room or other area related to a facility. Based on the user's identification, a library of commands (e.g., voice and/or gesture) may be accessible to the user, based on the user's identity, preferences, security/access level, etc. for controlling a target device related to the room or other area. Preferences can include preferences regarding window tint level, temperature (e.g., heating and/or cooling preferences), lighting (e.g., brightness and/or color), humidity, carbon dioxide (CO2) levels, etc. of a room, preferences regarding one or more devices in a room and/or any other preset preference described herein. In some embodiments, user preferences may be stored by database local to a sensor ensemble (e.g., sensor ensemble 605), in a separate computer of the facility (e.g., processor 654), or in one or more computers separate from the facility (e.g., cloud 651). Once the user's identity is known, preferences corresponding to the user can then be accessed from the database. In some embodiments, once a user's preferences are known, they may be used in multiple facilities communicatively coupled with the one or more for facilities from which the user's preferences were obtained. For example, for embodiments in which user preferences are stored in the cloud (e.g., cloud 651) and/or a computer (e.g., processor 652) connected to the cloud, user identification information and preferences may be propagated to other cloud-connected facilities and/or sensor ensembles enabling a cloud-connected sensor ensemble (e.g., sensor ensemble 605) in a facility, separate from and original facility in which user preferences were initially obtained, to identify a user, give the user access to the user's command set for controlling one or more targets, and/or implement the user's preferences. Any changes that the user may make to the user's preferences may further be saved to the cloud and propagated to all other facilities/cloud-connected devices. In this manner, user preferences may be modified and/or updated over time using one or more facilities.
In some embodiments, the external behavior of the user (e.g., gait and/or gestures) may be detected using at least one sensor. The sensor may be communicatively coupled to the network. The sensor may be an imaging system (e.g., an imaging radar system, lidar sensor, and/or a camera such as a video camera or IR camera). The sensor(s) (e.g., camera, lidar, and/or radar) may be provided on any available device, and in some examples is provided as part of a wall unit, as part of a device that interfaces with a wall unit (e.g., a smartphone, tablet, or other electronic device), as part of a hand-held device (e.g., smartphone, tablet, or other electronic device), on an electrochromic window or frame, or as part of any other device that is configured to control an electrochromic or other optically switchable window. For example, a user may walk and/or gesture while holding, wearing, or otherwise moving a sensing device that is configured to sense movement, and/or acceleration, etc. The readings on the sensing device may be used to help determine the gait of the user and/or what gesture a user has made. The movement sensing device may include one or more accelerometers (e.g., 3-axis accelerometer), gyroscopes, magnetometers, imaging system, or the like (and may be included in a virtual reality (VR) interface, such as the Oculus Quest or Oculus Rift available from Facebook Technologies, LLC, of Menlo Park, California. The mobile circuitry may be, or be included in, a user controller, a character controller, and/or a player controller.
In some embodiments, the sensing device is a fitness device (e.g., any of various wearable devices from Fitbit Inc. or Jawbone, each in San Francisco, CA), watch (e.g., from Apple Inc. of Cupertino, CA or Pebble Technology Corporation in Palo Alto, CA), or similar wearable device. In some embodiments, relative positioning, velocity, acceleration, and/or Doppler effect is used to determine the gait of a user and/or changes in gesture as commands to change the status of the target. In some embodiments, image recognition software is used to identify a human user for gait recognition and/or determine changes in gesture as commands to change the status of the target. In some embodiments, facial recognition software is used to determine changes in facial expressions as commands to change the tint level of windows. The gesture may comprise facial or bodily gesture (e.g., of limbs or part of limbs). The gesture may comprise kinesthetic movement. The gesture may comprise a physical movement of a body part. The gesture may comprise a corporal, and/or anatomic movement. The movement may comprise a muscular movement. The movement may comprise a movement of one or more bones (e.g., by moving their adjoining muscle(s).
In some embodiments, a type of command that may be initiated via voice control is to turn off “listening mode.” The sound sensor (e.g., listening device) may be operatively (e.g., communicatively) coupled to the network. When listening mode is on, the device that listens for commands is able to pick up oral commands. When listening mode is off, the device that listens for commands is not able to pick up, hear, and/or record such commands. For example, the device that listens for commands may be part of a (e.g., window) controller, IGU, wall device, and/or another electronic device (e.g., phone, tablet, etc.). A user may request to turn listening mode off for increased privacy, and/or energy savings, etc. In some cases, the user may request that listening mode turn off for a specified time period (e.g., the duration of a meeting), for example. In order to turn listening mode back on, the user may press a button/touchscreen (e.g., on the device that listens for commands, on the window controller, IGU, wall device, or other electronic device) or otherwise indicate that listening mode should turn back on. Devices may indicate when listening mode is on and/or off. In one example, one or more lights (e.g., LEDs) may indicate whether listening mode is on or off. The light may be turned on to indicate that listening mode is on, and off to indicate that listening mode is off (or vice versa). In another example, a first light or light color may indicate that listening mode is on, and a second light or light color may indicate that listening mode is off. In another example, devices can use an audio cue, e.g., may emit a tone, e.g., periodically, as a reminder to the user that listening mode is inactive (or active). In certain implementations, listening mode may be deactivated for a period of time (e.g., for at least about 1 minute, 10 minutes, 30 minutes, 1 hour, 2 hour, 3 hours, 1 day, etc.), after which listening mode may automatically be reactivated. The period of time over which listening mode remains deactivated may be chosen by the user, or may be preset, for example. In some embodiments, listening mode is activated by default. Listening mode may be on unless it is turned off (e.g., permanently turned off, or turned off for a period of time, as mentioned herein). In some embodiments, the default setting is that listening mode is off (e.g., listening mode does not activate unless a command is received to turn listening mode on).
In some embodiments, where gesture command is used, the user can control whether a relevant device that interprets gesture commands is in a “watching mode.” Like the listening mode, the watching mode can be turned on and off. When a device is in watching mode, it is able to sense and interpret gesture commands, for example. When the watching mode is off, the device is not able to sense, record, and/or process gesture commands. Details provided herein related to listening mode may similarly apply to watching mode. The device that interprets the gesture may or may not be part of the control system. The gesture interpreting device may comprise a circuitry (e.g., may comprise a processor). The gesture interpreting device may be communicatively coupled to the network and/or to the control system. The gestures may be interpreted with respect to a virtual image of the enclosure in which the controllable target (e.g., an IGU, a sensor, a light, or a media) is disposed in. The gestures may be interpreted with respect to a target it is coupled to (e.g., pointed at).
In some embodiments, one or more voice commands are used to ask a question to the system controlling the target (e.g., optically switchable device (or some component on the network on which the optically switchable device is installed)). The questions may relate directly to the target (e.g., actuator, or optically switchable device), or more generally, to any target (e.g., optically switchable device) or group of targets (e.g., devices) communicatively coupled to (e.g., on) the network, for example. For instance, a user may ask what the current optical state is for a particular optically switchable device (e.g., “what's the tint level of window 1?”). Similarly, a user may ask what the upcoming behavior will be for a particular optically switchable device (e.g., “when is the next time the windows in my office will begin to get darker?”). The questions may also relate to any other information to which the network has access. For instance, a user may ask about weather data (e.g., temperature data, cloud data, precipitation data, forecast data, etc.), location data (e.g., “where am I?” or “how do I get from here to the nearest printer/exit/bathroom/etc.”), access data (e.g., “am I allowed to control the tint level of the windows in this room?”), etc. A user may ask about any environmental characteristic of the enclosure (e.g., as delineated herein). A user may ask for an explanation of why the target (e.g., optically switchable device) is performing in a certain way. In one example, a user might ask, “why is window 1 tinting?” and the system may explain in response to the query, “clouds expected to clear in 20 minutes, tinting in anticipation of bright sun.” This feature may be particularly useful in cases where the optically switchable device is programmed to execute rules that might not be immediately observable and/or understandable to a user. The answer may be provided visually (e.g., on a screen), as a printed material, or aurally (e.g., through a speaker).
In some embodiments, a voice command is used to control the degree of privacy in the enclosure (e.g., room), e.g., with respect to (e.g., wireless) communications. In some embodiments, optically switchable windows are patterned to include one or more antenna that may be used to block or allow particular wavelengths to pass through the windows. When activated, these patterned antennae can provide increased security/privacy by blocking cell phone communications, Wi-Fi communications, etc. Examples of patterned antennae and related privacy considerations can be found in PCT Application No. PCT/US15/62387, filed Nov. 24, 2015, and titled WINDOW ANTENNAS that is incorporated herein by reference in its entirety.
In some embodiments where voice and/or gesture control are used, one or more dictionaries are defined. For voice control, the dictionaries may define a set of words and/or phrases that the system is configured to interpret/understand. Similarly, for gesture control, the dictionaries may define a set of gestures that the system is configured to interpret/understand. Dictionaries may be tiered, e.g., given a command in a first level dictionary, a new dictionary at a second level may be initiated for receiving commands, and once received, yet another level dictionary may be actuated. In this way, individual dictionaries need not be overly complex, and the end user can quickly get to the command structure they desire. In some embodiments, (e.g., when the target is a media) the gestures are interpreted as cursor movement on a media projection.
Examples of words or phrases that may be defined include names/identifications for each optically switchable device or group of devices (e.g., “window 1,” “group 1,” “zone 1,” etc.). Such names/identifications may also be based at least in part on the location of the optically switchable devices. In this respect, the dictionaries may be defined to include words that identify optically switchable devices based at least in part on location (e.g., “first floor,” or “break room,” or “east-facing”), and/or words that provide a relation between the user (or some other person) and the optically switchable device being identified (e.g., “my office,” “the left window,” or “Deepa's room”).
In some embodiments, the dictionaries also define words related to the desired commands that can be instructed. For example, the dictionaries may include words like “tint,” “clear,” “clearest,” “darker,” “darkest,” “lighter,” “lightest,” “more,” “less,” “very,” “a little,” “tint level,” “tint1,” “tint2,” etc. Any words likely to be used by a person when instructing the optically switchable device when using verbal commands may be included in the dictionary. In cases where the system is configured to allow a user to set a schedule or rules for the behavior of the optically switchable device, the dictionary or dictionaries can include any words needed to understand such commands (e.g., “Monday,” “Tuesday through Friday,” “morning,” “afternoon,” “bedtime,” “sunrise,” “if,” “then,” “when,” “don't,” “cloudy,” “sunny,” “degrees,” “someone,” “no one,” “movement,” “only,” etc.). Similarly, in cases where the system is configured to allow a user to ask a question, the dictionary or dictionaries can include any words needed to understand the types of questions the system is designed to answer.
In some embodiments, there is a tradeoff between larger dictionaries, which may enable finer control, more natural and/or flexible commands, and more complex functions (e.g., answering any question where the answer is available on the internet), compared to smaller dictionaries, which may be easier for people to master, and which may enable faster and/or more local processing. Smaller dictionaries may be used in a tiered format, where access to successive dictionaries is afforded by a user providing the proper voice or gesture command in one dictionary in order to be allowed access to the next dictionary.
In some embodiments, a single dictionary may be used. In other embodiments, two or more dictionaries may be used, and the dictionary that is used at a particular time depends on what type of command, or what portion of a command a user is trying to convey. For example, a first dictionary may be used when a user is identifying which optically switchable device they wish to control, and a second dictionary may be used when the user is identifying what they want the optically switchable device to do. The first dictionary could include any words needed to identify the relevant optically switchable device, while the second dictionary could include any words needed to interpret what the user wants the optically switchable device to do. Such contextual dictionaries can provide a limited sub-set of words that the system is configured to understand and/or interpret whenever the particular dictionary is being used. This may make it easier to interpret a user's commands.
In some embodiments, one or more dictionaries may be tailored to particular users. The dictionaries for defining and/or determining which electrochromic window(s) a user desires to switch may be limited based at least in part on which windows the user is authorized to switch, for instance. In one example, user A is allowed to switch windows 1-5, while user B is allowed to switch windows 6-10. The dictionary or dictionaries used to transcribe and/or interpret commands from user A may be limited to identifying windows 1-5, while the dictionary or dictionaries used to transcribe and/or interpret commands from user B may be limited to identifying windows 6-10.
In some embodiments, each dictionary includes certain keywords that allow the user to navigate through the system more easily. Such keywords may include phrases such as “help,” “back,” “go back,” “previous,” “undo,” “skip,” “restart,” “start over,” “stop,” “abort,” etc. When a user requests help, the system may be configured to communicate to the user (e.g., visually and/or aurally) the words, phrases, commands, windows, etc. that the system is currently configured to accept/understand based at least in part on the dictionary that is being used at a given time. For instance, if a user requests help while the system is accessing a dictionary that defines the different windows available for switching, the system may communicate that the available inputs at that time are, e.g., “window 1,” “window 2, “window 3,” “group 1,” etc.
In some embodiments, the system acts to ensure that a user is authorized to make a particular command before the command is executed. This can prevent unauthorized users from making changes to the optically switchable devices. One setting in which this is particularly valuable is conference rooms, where there may be many people present at once. In such cases, it may be desirable to ensure that people who do not have authority to change the optical state of the optically switchable devices are prevented from doing so. This can reduce the risk that the optically switchable devices will change based at least in part on overheard (typically non-relevant) comments made by those in the room. Another setting in which this feature may be valuable is a commercial office space, where it may be desired that individual people can each control a limited number of optically switchable devices near their workspaces, for instance. In one example, a (e.g., each) person may be authorized to control the target (e.g., optically switchable window(s)) in their particular office, or on their particular floor, etc. For example, it may be beneficial to ensure that the (e.g., only) people who are able to initiate a change in the target (e.g., optical transitions) via voice or gesture command are authorized to do so.
In some embodiments, authorization is done by having a user “log in” to the system to identify himself or herself. This may be done by logging into an application on an electronic device (e.g., smartphone, tablet, etc.), by keying in a code, electronically recognizing a code, by fingerprinting, eye pattern identification, facial identification, or voicing a passcode, etc. In another example, voice recognition may be used to confirm the identity of a user. In a further example, facial recognition, fingerprint scanning, retinal scanning, or other biometric-based methods may be used to confirm the identity of a user. In another example, gait recognition can be used to identify a user. In some embodiments, gait recognition may be used in conjunction with one or more additional authentication techniques. Different authorization procedures may be best suited for different applications and/or contexts. In a particular example, a user may be automatically authorized. Gait recognition may provide such automatic authorization where a sensor comprising an imaging system used for gait recognition may be situated at a location (e.g., in a building) enabling the imaging system to capture a sufficient amount of video or number of successive images for gait recognition. This may be at or adjacent to a location in a building at which authorization is used. For example, a gait recognition imaging system may be set up to capture video in a hallway leading up to a door, enabling a user to be authorized via gait recognition to provide access at the door (e.g., unlocking the door or activating an additional sensor at or near the door for additional authentication). According to some embodiments, automatic authorization may be based at least in part on a physical authorization token (e.g., an RFID badge, a BLE beacon, UWF beacon, etc. having appropriate identification information), and the proximity of the physical authorization token to a sensor that reads the token. The sensor may be provided on an optically switchable device or adjacent thereto (e.g., in a frame portion of the IGU such as in a mullion), on a controller in communication with the optically switchable device, on a wall unit in communication with the optically switchable device, etc. The verification may occur locally (e.g., on the sensor that reads the token, on an optically switchable device, on a controller, on a wall unit, etc.), and/or in the cloud.
In some embodiments, authorization occurs whenever it is needed, and authorization may expire after a set amount of time has passed, or after the user has been idle for a set amount of time (e.g., after 24 hours, or after 1 hour, or after 10 minutes). The time period used for auto-logging out may depend on the setting in which the target (e.g., windows) are installed or projected. For example, whether the target(s) (e.g., windows) are in a public area or a private area). In some cases, authorization may not expire until a user logs out (e.g., using any available method including, but not limited to, orally requesting a logout, pressing a logout button, etc.). In some embodiments, authorization occurs each time a command is made. In some embodiments, authorization occurs in stages even when interpreting a single command. In a first authorization stage, it may be determined whether the user has authorization to make any changes on the network, and in a second authorization stage, it may be determined whether the user has authorization to make the particular change that the user has requested and/or initiated.
In some embodiments, the authorization process is used to limit the dictionaries used to interpret the voice and/or gesture commands. For example, the dictionary or dictionaries for a particular user may exclude one or more specified targets (e.g., optically switchable devices (or groups/zones of such devices)) that the user is not authorized to control. In one example, a user may be only authorized to control the optically switchable devices in zone1 and zone 2, so the dictionary or dictionaries used to interpret commands for this user may include “zone 1” and “zone 2” while excluding “zone 3.” Any other words needed to interpret and/or understand the command may also be included in the dictionary.
In some embodiments, a control system (e.g., voice control and/or external behavior control) includes an authentication module which is used to practice the authorization and/or security techniques discussed herein. For example, the authorization module may be used to ensure that the person giving the command is authorized to make the command. The authentication module may comprise a blockchain procedure and/or embedded encryption key(s). The blockchain procedure may comprise (e.g., peer-to-peer) voting. The encryption key(s) may be linked to a target (e.g., a device). The authentication module may be designed to ensure that only authorized devices can connect to a given network, facility, and/or service. The module may compare the optically switchable device identified in the command to a list of optically switchable devices that the user is authorized to control. In cases where a user tries to control an optically switchable device that they are not authorized to control, the authentication module may be configured to notify the user (e.g., visually, in print, and/or aurally) that they are not authorized to control the relevant optically switchable device. In other cases, no action is taken when an un-authorized command is given (e.g., no notification to the user, and no change to the target status (e.g., no switching of the optically switchable device)). The authentication may consider the identification of the user and/or other employee data such as rank, seniority, certification, education, and/or departmental affiliation. The identification of the user may be provided to the authentication module, e.g., via a facility entry tag of the user. The authentication module may be required to limit access to sensitive medical information, dangerous manufacturing machinery, and/or any restricted information. Examples of authentication (e.g., using blockchain procedure) can be found in PCT patent application serial number PCT/US20/70123 that is incorporated herein by reference in its entirety.
The skeleton extraction/pose estimation module 702 may comprise hardware and/or software components (e.g., of a sensor, controller, or computer) configured to receive one or more images (e.g., successive images or video frames) of one or more users from an imaging system and extract the skeleton and pose of each of the one or more users in the one or more images. In particular, one or more skeleton extraction algorithms can be used to identify various physical features of each user (e.g., head, torso, arms, legs, hands, and feet) and create a 2D or 3D representation, or skeleton, of these features. The relative position of each of these identified features (known as “keypoints”) represents the pose of the user. A 3D skeleton may be extracted in embodiments where 3D information is obtained from the imaging system (e.g., from imaging radar, lidar sensor, and/or multiple cameras). In some embodiments, a skeleton is a compact representation of a user that includes multiple lines or polygons representing multiple respective features of the user (e.g., head, torso, arms, legs, hands, and feet) as well as joints where features connect (e.g., where an arm or leg attaches to the torso) and joints within features (e.g., a knee of a leg or elbow of an arm). In some embodiments, finer features of a user such as eyes, eyebrows, nose, mouth, fingers, etc. may be extracted during the skeleton extraction process. In some embodiments, these finer features may be used for gesture recognition. Skeleton extraction and pose estimation algorithms may involve image processing and/or other computer vision (CV) techniques such as shape analysis, factorization, and object classification/detection. In some embodiments, skeleton extraction/pose estimation may leverage machine learning (ML) or other forms of artificial intelligence (AI), including Convolutional Neural Networks (CNNs) and deep learning-based approaches. Skeleton extraction and pose estimation algorithms may utilize a “autumn-up” approach (in which keypoints in an image are identified and grouped into skeletons) or a “top-down” approach (in which object detection is used to identify objects and key points are extracted from the objects). Skeleton extraction/pose detection may be performed by products such as OpenPose (available to license from Carnegie Melon University), Cascaded Pyramid Network (CPN, by Yilun Chen et al.), AlphaPose (an open-source pose tracker maintained by the Shanghai Jiao Tong University Machine Vision and Intelligence Group (MVIG)), and High Resolution Net (HRNet, an open-source project developed by Microsoft Research Asia and University of Science and Technology of China). The skeleton extraction/pose estimation module 702 may include software libraries for pose estimation, which may be based on regular camera images. For embodiments in which infrared cameras, lidar, or imaging radar are used, the software libraries can be modified (e.g., adapting object detection and/or other CV processing used to perform the pose extraction) to adapt to infrared images, lidar scans, and/or radar images.
Pose tracking module 704 may comprise hardware and/or software components (e.g., of a sensor, controller, or computer) configured to receive track the pose of each of one or more skeletons in an image across a plurality of successive images (e.g., video frames) captured over a period of time. Pose tracking may use any of a variety of algorithms (e.g., mapping nearest-neighbor key points, predicting action/movement, etc.) to map a skeleton in one image to the corresponding skeleton in a successive image. Some embodiments comprise assigning a unique instance ID for each keypoint or user across multiple images. In some embodiments, pose tracking may be performed by algorithms that also perform skeleton extraction/pose estimation. In such embodiments, the pose tracking module 704 may be combined with the skeleton extraction/pose estimation module 702.
The external behavior learning module 706 may comprise hardware and/or software components (e.g., of a sensor, controller, or computer) configured to receive poses for one or more users, tracked overtime, to determine one or more characteristic external behaviors exhibited by each user that can later be used to identify the user. External behaviors can include a series of one or more intentional and unintentional poses of a user, such as a gait or gesture. Aspects of these external behaviors such as timing, pose, and user dimensions (e.g., height and width) can be used to identify one or more external behaviors unique to an individual. As with the other modules illustrated in
Data analysis using ML may be performed by a machine-based system (e.g., comprising a circuitry) that may use known data (e.g., positive and/or negative datasets) to train one or more models for pose extraction, pose tracking, and/or external behavior learning, which may be used to subsequently analyze new data in the field. The model may be implemented using circuitry, such as the circuitry of a processor, and/or software. Data analysis by one or more of the modules in
The uniqueness of an external behavior can be determined when extracting the external behavior data for a user. For example, over the course of a plurality of video frames a user may perform actions that result in a skeleton corresponding to the user undergoing a series of poses during those video frames. These poses may be compared with poses from other external behavior data of the user and other users to determine (1) whether the user commonly exhibits this external behavior, and (2) whether this external behavior is unique (e.g., based on at least a subset of the series of poses). To do so, ML models may be trained using a positive dataset (e.g., videos of a user exhibiting the external behavior) and a negative dataset (e.g., videos of the user not exhibiting during the external behavior, videos of other users exhibiting a similar behavior, etc.). For example, a gait recognition model can be trained using a positive data set of videos of the user walking and a negative dataset of videos of the user not walking and/or videos of other users (e.g., walking or not walking).
Images used for training an external behavior learning module may be obtained in different ways. In some embodiments, users may provide identification or other authentication means. A user providing identification or other authentication may include, for example, a user presenting a fob or other physical token, providing biometric input, and/or other identification. In some embodiments, identification of a user may be performed via sensor or device input from captured sensor information of the user, without active user input. This can include, for example, facial recognition (e.g., from a camera image), voice recognition, or other user identification means. In some embodiments, sensor input (e.g., indicative of user height/shape) can be fused with other input to determine the identity of the user. In some embodiments, device settings provided by the user such as thermostat settings, lighting settings, window tint settings, etc. may be compared with user preferences and used as an input to identify a particular user. For example, if facial recognition does not provide the identity of a user above a threshold level of confidence, a comparison of device settings made by the with known preferences of the user may be made and, if the device settings match the known preferences, the identity of the user may then exceed the threshold level of confidence.
Once the identity of the user is determined (e.g., above a threshold level of confidence), video of the user may be obtained and used (e.g., using the process 800) to train the external behavior learning module. Once the external behavior learning module is able to identify the user (e.g., above a threshold level of confidence) based on a unique external behavior exhibited by the user, the learning module may be used to subsequently identify the user using the unique external behavior.
As used herein, the term “identity of the user” refers to an identifier unique to the user individual. This can include a name, employee ID, or other information that may make the user identifiable to other humans or systems other than the external behavior extraction system. In other embodiments, however, the external behavior extraction system may (e.g., randomly) assign a unique identifier to a user and use IR cameras, lidar, imaging radar, and/or similar data to address privacy concerns and help minimize storing any information that would make the user identifiable outside the external behavior extraction system.
At block 902, the method 900 may begin with detecting a triggering event. As noted, an external behavior recognition system may be deployed in a facility and may be communicatively coupled with various systems and devices, including cameras, card (or other physical token) readers, sensors, HVAC systems, etc. Thus, any of a variety of triggering events can be detected to initiate the method 900. In some embodiments, the triggering event may comprise the authentication of a user. This may include, for example, the user presenting an identification or access card at a card reader.
At block 904, video of the user is in capture, responsive to detecting the triggering event at block 902. In some embodiments, the video may be situated to capture a particular external behavior. For example, an external behavior recognition system set up to perform gait recognition may be obtained from video of a camera set up to capture footage of the user walking. In a particular example, the triggering event detected at block 902 may comprise a user presenting an access card to unlock a door, and the video of the user captured at block 904 is captured from a camera set up to capture video of the user walking through the doorway of the unlocked door.
At block 906, external behavior data is extracted from the video. This external behavior data may be extracted, for example, using the process 800 of
At block 910, a determination is made of whether the user identity is known or not. Again, the identity may comprise an identifier unique to the user, but may not necessarily be used outside the external behavior recognition system to identify the user. If no such identity is known, the functionality at 912 may be performed, where a new identifier is created. This new identifier may comprise a generated number or code (e.g., generated via random or sequential generation algorithms), unique to the user. Once the new identifier is created, or if the user identity/identifier is known at block 910, the external behavior data may then be associated with the identity/identifier of the user, at block 914. This can allow the subsequent external behavior data of the user to be used with stored data (associated with the identity/identifier) to train an external behavior learning module to identify unique external behavior of the user.
At block 916, a determination is made of whether a threshold amount of data associated with the identity/identifier is reached. This may comprise, for example, obtaining a threshold amount of external behavior data of the user to create a positive data set to accurately train an external behavior learning module to identify the user from external behavior data. According to some embodiments, this threshold amount may be determined using historical models to determine a minimum amount of external behavior data needed to train the external behavior learning module to identify the user. In some embodiments, an ongoing or iterative process may be used in which an ML model of the external behavior learning module is trained until the model is able to identify the user using external behavior data with a threshold accuracy (e.g., 80%, 90%, 95%, 97%, 99%, 99.5%, or 99.9% accuracy on positive and/or negative datasets). At block 918, once the threshold amount of data associated with the identity/identifier of the user is reached, the ML model of the external behavior learning module is trained using stored external behavior data associated with the identity/identifier, and the process can then stop. The method 900 may be repeated as needed (e.g., with every new detected trigger event) until the ML model of the external behavior learning module is sufficiently trained to obtain the desired accuracy of identifying the user from external behavior data. Once sufficiently trained, the ML model can then be deployed for purposes of identifying the user. An example process of identification is illustrated in
The method begins at block 1102, where a plurality of successive images of a user of the facility is captured with an imaging system of the facility. The imaging system may comprise a camera, an infrared camera, lidar sensor, or an imaging radar system. The imaging system may be operatively coupled with a controller. Capturing the plurality of successive images of the user may be responsive to a triggering event. In some embodiments, the triggering event may comprise detection of the user at a location of the facility. In some embodiments, imaging system comprises the camera or the IR camera, and the plurality of successive images comprise video frames.
The functionality at block 1104 comprises obtaining, from the plurality of images, external behavior data of the user, wherein the external behavior data is representative of one or more physical actions taken by the user during the capturing of the plurality of images. The one or more physical actions taken by the user may comprise a gait or gesture of the user. Obtaining the external behavior data the user may comprise extracting a respective pose of the user from each image of the plurality of successive images.
At block 1106, the functionality comprises determining an identity of the user based at least in part on the external behavior data of the user determining a unique identifier associated with the user. Determining the identity of the user may be further based on one or more device inputs received from the user. The one or more device inputs comprise a temperature setting, a window tint setting, and/or a lighting setting. The lighting setting comprises a brightness and/or color of light. Some embodiments may comprise generating the identity of the user, where the identity comprises a unique code or number associated with the user. In some embodiments, determining the identity of the user comprises using machine learning to determine the identity. Such embodiments may further include training a machine learning model used to perform the machine learning. Training the machine learning model may comprise using previously-obtained sets of images of the user taking the one or more physical actions as a positive dataset. The previously-obtained sets of images may be obtained by the imaging system. Determining the identity of the user may be further based on sensor information regarding the user. The sensor information may comprise information indicative of a sound made by the user, dimensions of the user, and/or biometric information regarding the user.
At block 1108, the functionality comprises implementing environment customizations associated with the identity of the user. Implementing environment customizations may comprise controlling an environmental aspect using one or more building systems. The one or more building systems may comprise a device ensemble having a housing that encloses the one or more devices that comprise: (i) sensors, (ii) a transceiver, or (iii) a sensor and an emitter. The device ensemble may be disposed in a fixture of the facility, or is attached to a fixture of the facility. The one or more building systems may comprise a tintable window. The tintable window may comprise an electrochromic window. The device ensemble may comprise the imaging system. Implementing environment customizations may comprise adjusting a temperature, window tint, and/or lighting within the facility.
In some embodiments where commands are provided by external behavior, such as gestures, a mobile circuitry, or a sensor (e.g., of an imaging system) may be used instead of (or in addition to) a microphone, in order to perceive and record the user's command. The mobile circuitry may be communicatively coupled to the network that is communicatively coupled to a digital twin of the enclosure in which the target is disposed. Instead of a voice recognition module, a gesture recognition module may be employed for analyzing the mobile circuitry and/or sensor (e.g., imaging system) data. For example, a user may be positioned within a field of view of a camera so that external behavior (e.g., gesture commands) of the user can be captured which are carried out according to a desired control action to be taken in connection with controllable targets (e.g., devices) such as tintable windows. For example, movements of the user can be captured by the mobile device manipulated by the user (e.g., moved by the user) which are carried out according to a desired control action to be taken in connection with controllable targets (e.g., devices) such as tintable windows.
In some embodiments, the access point is connected to a switch to accomplish network communication between the control device of a user (e.g., a mobile circuitry) and a control unit for the target (e.g., window, media, or other appliance) to receive a command. For example, the switch may be connected to a router and/or the control unit. The connections between the different elements may be wired and/or wireless, as appropriate for a particular application. For example, the access point may be a wireless access point, and the connection between the access point and the device may be wireless. In some embodiments, the device may be any number of electronic devices configured to control a status of a target (e.g., such as a media, or the electrochromic windows). The router may include firewall protection to enhance security. The control unit may be a window controller, NC, or MC. If the control unit is not a window controller, it may relay instructions to relevant window controllers over the network, for example.
In some embodiments, the interactive systems to be controlled by a user include media (e.g., visual and/or audio content) for display, e.g., to building occupants. The display may include stills or video projection arrangements. The display may include transparent organic light-emitting devices (TOLED). The display may be integrated as a display construct with window panel(s) (e.g., frame(s)). Examples of display constructs can be found in U.S. provisional patent application Ser. No. 62/975,706 filed on Feb. 12, 2020, titled “TANDEM VISION WINDOW AND MEDIA DISPLAY,” that is incorporated herein in its entirety.
In some embodiments, a display construct is coupled with a viewing (e.g., a tintable viewing) window. The viewing window may include an insulated glass unit (IGU). The display construct may include one or more glass panes. The display (e.g., display matrix) may comprise a light emitting diode (LED). The LED may comprise an organic material (e.g., organic light emitting diode abbreviated herein as “OLED”). The OLED may comprise a transparent organic light emitting diode display (abbreviated herein as “TOLED”), which TOLED is at least partially transparent. The display may have at its fundamental length scale 2000, 3000, 4000, 5000, 6000, 7000, or 8000 pixels. The display may have at its fundamental length scale any number of pixels between the aforementioned number of pixels (e.g., from about 2000 pixels to about 4000 pixels, from about 4000 pixels to about 8000 pixels, or from about 2000 pixels to about 8000 pixels). A fundamental length scale may comprise a diameter of a bounding circle, a length, a width, or a height. The fundamental length scale may be abbreviated herein as “FLS.” The display construct may comprise a high resolution display. For example, the display construct may have a resolution of at least about 550, 576, 680, 720, 768, 1024, 1080, 1920, 1280, 2160, 3840, 4096, 4320, or 7680 pixels, by at least about 550, 576, 680, 720, 768, 1024, 1080, 1280, 1920, 2160, 3840, 4096, 4320, or 7680 pixels (at 30 Hz or at 60 Hz). The first number of pixels may designate the height of the display and the second pixels may designates the length of the display. For example, the display may be a high resolution display having a resolution of 1920×1080, 3840×2160, 4096×2160, or 7680×4320. The display may be a standard definition display, enhanced definition display, high definition display, or an ultra-high definition display. The display may be rectangular. The image projected by the display matrix may be refreshed at a frequency (e.g., at a refresh rate) of at least about 20 Hz, 30 Hz, 60 Hz, 70 Hz, 75 Hz, 80 Hz, 100 Hz, or 120 Hertz (Hz). The FLS of the display construct may be at least 20″, 25″, 30″, 35″, 40″, 45″, 50″, 55″, 60″, 65″, 80″, or 90 inches (″). The FLS of the display construct can be of any value between the aforementioned values (e.g., from about 20″ to about 55″, from about 55″ to about 100″, or from about 20″ to about 100″).
In some embodiments, at least a portion of a window surface in a facility is utilized to display the various media using the glass display construct. The display may be utilized for (e.g., at least partial) viewing an environment external to the window (e.g., outdoor environment), e.g. when the display is not operating. The display may be used to display media (e.g., as disclosed herein), to augment the external view with (e.g., optical) overlays, augmented reality, and/or lighting (e.g., the display may act as a light source). The media may be used for entertainment and non-entertainment purposes. The media may be used for video conferencing. For example, the media may be used for work (e.g., data analysis, drafting, and/or video conferencing). For example, the media may be used for educational, health, safety, purchasing, monetary, or entertainment purposes. The media may present personnel not at the enclosure in which the media display is disposed (e.g., remote employees). The media may present personnel at the enclosure in which the media display is disposed. For example, the media display may mirror the personnel (e.g., and their actions such as in real time) in the enclosure in which the media display and the local personals are disposed. The media may be used as a coaching tool by mirroring the local personnel. For example, the mirroring media may serve as a fitness coaching tool, a speech coaching tool, a posture coaching tool, and/or a behavioral coaching tool. The media may present personnel at the enclosure in which the media display is disposed and remote personnel, e.g., in a collage, overlayed, and/or bifurcated display. The media may be manipulated (e.g., by utilizing the display construct). Utilizing the display construct can be direct or indirect. Indirect utilization of the media may be using an input device such as an electronic mouse, or a keyboard. The input device may be communicatively (e.g., wired and/or wirelessly) coupled to the media. Direct utilization may be by using the display construct as a touch screen using a user (e.g., finger) or a contacting device (e.g., an electronic pen or stylus).
In some embodiments, the media may be displayed by a transparent media display construct. The transparent display construct that is configured to display media, may be disposed on, or coupled (e.g., attached) to, a window, a door, a wall, a divider, or to any other architectural element of a facility. The architectural element may be a fixture or a non-fixture. The architectural element (e.g., window, wall, or divider) may be static or mobile (e.g., a moving window or door). The architectural element may comprise a tintable window. The architectural element may comprise a tintable substance (e.g., an optically switchable device such as an electrochromic device). The optically switchable device may alter its transparency, absorbance, or color, e.g., at least in the visible spectrum. A user may control the usage of the media and/or tint state of the architectural element, e.g., separately or as linked to each other. A user in one enclosure looking out of the enclosure through the transparent media display, may optionally see both the media, and the external environment of the enclosure through the media display.
Embodiments described herein relate to vision windows with a tandem (e.g., transparent) display construct. In certain embodiments, the vision window is an electrochromic window. The electrochromic window may comprise a solid state and/or inorganic electrochromic (EC) device. The vision window may be in the form of an IGU. When the IGU includes an electrochromic (abbreviated herein as “EC”) device, it may be termed an “EC IGU.” The EC IGU can tint (e.g., darken) a room in which it is disposed and/or provide a tinted (e.g., darker) background as compared to a non-tinted IGU. The tinted IGU can provide a background preferable (e.g., necessary) for acceptable (e.g., good) contrast on the (e.g., transparent) display construct. In another example, windows with (e.g., transparent) display constructs can replace televisions (abbreviated herein as “TVs”) in commercial and residential applications. Together, the (e.g., transparent) display construct and EC IGU can provide visual privacy glass function, e.g. because the display can augment the privacy provided by EC glass alone.
In some embodiments, the display construct comprises a hardened transparent material such as plastic or glass. The glass may be in the form of one or more glass panes. For example, the display construct may include a display matrix (e.g., an array of lights) disposed between two glass panes. The array of lights may include an array of colored lights. For example, an array of red, green, and blue colored lights. For example, an array of cyan, magenta, and yellow colored lights. The array of lights may include light colors used in electronic screen display. The array of lights may comprise an array of LEDs (e.g., OLEDs, e.g., TOLEDs). The matrix display (e.g., array of lights) may be at least partially transparent (e.g., to an average human eye). The transparent OLED may facilitate transition of a substantial portion (e.g., greater than about 30%, 40%, 50%, 60%, 80%, 90% or 95%) of the intensity and/or wavelength to which an average human eye senses. The matrix display may form minimal disturbance to a user looking through the array. The array of lights may form minimal disturbance to a user looking through a window on which the array is disposed. The display matrix (e.g., array of lights) may be maximally transparent. At least one glass pane of the display construct may be of a regular glass thickness. The regular glass may have a thickness of at least about 1 millimeters (mm), 2 mm, 3 mm, 4 mm, 5 mm, or 6 mm. The regular glass may have a thickness of a value between any of the aforementioned values (e.g., from 1 mm to 6 mm, from 1 mm to 3 mm, from 3 mm to about 4 mm, or from 4 mm to 6 mm). At least one glass pane of the display construct may be of a thin glass thickness. The thin glass may have a thickness of at most about 0.4 millimeters (mm), 0.5 mm, 0.6 mm, 0.7 mm, 0.8 mm, or 0.9 mm thick. The thin glass may have a thickness of a value between any of the aforementioned values (e.g., from 0.4 mm to 0.9 mm, from 0.4 mm to 0.7 mm, or from 0.5 mm to 0.9 mm). The glass of the display construct may be at least transmissive (e.g., in the visible spectrum). For example, the glass may be at least about 80%, 85%, 90%, 95%, or 99% transmissive. The glass may have a transmissivity percentage value between any of the aforementioned percentages (e.g., from about 80% to about 99%). The display construct may comprise one or more panes (e.g., glass panes). For example, the display construct may comprise a plurality (e.g., two) of panes. The glass panes may have (e.g., substantially) the same thickness, or different thickness. The front facing pane may be thicker than the back facing pane. The back facing pane may be thicker than the front facing pane. Front may be in a direction of a prospective viewer (e.g., in front of display construct 101, looking at display construct 101). Back may be in the direction of a (e.g., tintable) window (e.g., 102). One glass may be thicker relative to another glass. The thicker glass may be at least about 1.25*, 1.5*, 2*, 2.5*, 3*, 3.5*, or 4* thicker than the thinner glass. The symbol “*” designates the mathematical operation of “times.” The transmissivity of the display construct (that including the one or more panes and the display matrix (e.g., light-array or LCD)) may be of at least about 20%, 30%, 35%, 40%, 45%, 50%, 60%, 70%, 80%, or 90%. The display construct may have a transmissivity percentage value between any of the aforementioned percentages (e.g., from about 20% to about 90%, from about 20% to about 50%, from about 20% to about 40%, from about 30% to about 40%, from about 40% to about 80%, or from about 50% to about 90%). A higher transmissivity parentage refers higher intensity and/or broader spectrum of light that passes through a material (e.g., glass). The transmissivity may be of visible light. The transmissivity may be measured as visible transmittance (abbreviated herein as “Tvis”) referring to the amount of light in the visible portion of the spectrum that passes through a material. The transmissivity may be relative to the intensity of incoming light. The display construct may transmit at least about 80%, 85%, 90%, 95%, or 99% of the visible spectrum of light (e.g., wavelength spectrum) therethrough. The display construct may transmit a percentage value between any of the aforementioned percentages (e.g., from about 80% to about 99%). In some embodiments, instead of an array of lights, a liquid crystal display is utilized.
In some embodiments, diverse types of interfaces are employed for providing user control of interactive targets (e.g., systems, devices, and/or media). The interactive targets can be controlled, e.g., using control interface(s). The control interface may be local and/or remote. The control interface may be communicated through the network. The control system may be communicatively coupled to the network, to which the target(s) are communicatively coupled. An example of a control interface comprises manipulating a digital twin (e.g., representative model) of a facility. For example, one or more interactive devices (e.g., optically switchable windows, sensors, emitters, and/or media displays) may be controlled using a mobile circuitry. The mobile circuitry may comprise a gaming-type controller (e.g., a pointing device) or a virtual reality (VR) user interface. When an additional new device is installed in the facility (e.g. in a room thereof) and is coupled to the network, the new target (e.g., device) may be detected (e.g., and included into the digital twin). The detection of the new target and/or inclusion of the new target into the digital twin may be done automatically and/or manually. For example, the detection of the new target and/or inclusion of the new target into the digital twin may be without requiring (e.g., any) manual intervention.
In some embodiments, a digital twin comprises a digital model of the facility. The digital twin is comprised of a virtual three dimensional (3D) model of the facility. The facility may include static and/or dynamic elements. For example, the static elements may include representations of a structural feature of the facility and the dynamic elements may include representations of an interactive device with a controllable feature. The 3D model may include visual elements. The visual elements may represent facility fixture(s). The fixture may comprise a wall, a floor, wall, door, shelf, a structural (e.g., walk-in) closet, a fixed lamp, electrical panel, elevator shaft, or a window. The fixtures may be affixed to the structure. The visual elements may represent non-fixture(s). The non-fixtures may comprise a person, a chair, a movable lamp, a table, a sofa, a movable closet or a media projection. The visual elements may represent facility features comprising a floor, wall, door, window, furniture, appliance, people, and/or interactive target(s)). The digital twin may be similar to virtual worlds used in computer gaming and simulations, representing the environment of the real facility. Creation of a 3D model may include the analysis of a Building Information Modeling (BIM) model (e.g., an Autodesk Revit file having *.RVT format), e.g., to derive a representation of (e.g., basic) fixed structures and movable items such as doors, windows, and elevators. The 3D mode may comprise architectural details related to the design of the facility, such as a 3D model, elevation details, floor plans, and/or project settings related to the facility. The 3D model may comprise annotation (e.g., with two dimensional (2D) drafting element(s)). The 3D model may facilitate access to information from a model database of the facility. The 3D model may be utilized for planning and/or tracking various stages in the lifecycle of the facility (e.g., facility concept, construction, maintenance and/or demolition). The 3D model may be updated during the lifecycle of the facility. The update may be periodically, intermittently, on occurrence of an event (e.g., relating to the structural status of the facility), in real time, on availability of manpower, and/or at a whim. The digital twin may comprise the 3D model, and may be updated in relation to (e.g., when) the 3D model of the facility is updated. The digital twin may be linked to the 3D model (e.g., and thus linked to its updates). In real time may include within at most 15 seconds (sec.), 30 sec., 45 sec., 1 minute (min), 2 min., 3 min. 4 min., 5 min, 10 min., 15 min. or 30 min. from the occurrence of a change in the enclosure (e.g., a change initiated by the user).
In some embodiments, the digital twin (e.g., 3D model of the facility) is defined at least in part by using one or more sensors (e.g., optical, acoustic, pressure, gas velocity, and/or distance measuring sensor(s)), to determine the layout of the real facility. Usage of sensor data can be used exclusively to model the environment of the enclosure. Usage of sensor data can be used in conjunction with a 3D model of the facility (e.g., (BIM model) to model the environment of the enclosure. The BIM model of the facility may be obtained before, during, and/or after the facility has been constructed. The BIM model of the facility can be updated (e.g., manually and/or using the sensor data) during operation of the facility (e.g., in real time). In real time may include, during occurrence of a change of, or in, the facility. In real time may include within at most 2 h, 4 h, 6 h, 8 h, 12 h, 24 h, 36 h, 48 h, 60 h, or 72 h from the occurrence of a change of, or in, the facility.
In some embodiments, dynamic elements in the digital twin include target (e.g., device) settings. The target setting may comprise (e.g., existing and/or predetermined): tint values, temperature settings, and/or light switch settings. The target settings may comprise available actions in media displays. The available actions may comprise menu items or hotspots in displayed content. The digital twin may include virtual representation of the target and/or of movable objects (e.g., chairs or doors), and/or occupants (actual images from an imaging system or from stored avatars). In some embodiments, the dynamic elements can be targets (e.g., devices) that are newly plugged into the network, and/or disappear from the network (e.g., due to a malfunction or relocation). The digital twin can reside in any circuitry (e.g., processor) operatively coupled to the network. The circuitry in which the digital circuitry resides may be in the facility, outside of the facility, and/or in the cloud. In some embodiments, a two-way link is maintained between the digital twin and a real circuitry. The real circuitry may be part of the control system. The real circuitry may be included in the MC, NC, floor controller, local controller, or in any other node in a processing system (e.g., in the facility or outside of the facility). For example, the two-way link can be used by the real circuitry to inform the digital twin of changes in the dynamic and/or static elements so that the 3D representation of the enclosure can be updated, e.g., in real time. In real time may include, during occurrence of a change of, or in, the enclosure. In real time may include within at most 15 seconds (sec.), 30 sec., 45 sec., 1 minute (min), 2 min., 3 min. 4 min., 5 min, 10 min., 15 min. or 30 min. from the occurrence of a change in, the enclosure. The two-way link may be used by the digital twin to inform the real circuitry of manipulative (e.g., control) actions entered by a user on a mobile circuitry. The mobile circuitry can be a remote controller (e.g., comprising a handheld pointer, manual input buttons, or touchscreen).
In some embodiments, one or more mobile circuitry devices of a user are aligned with (e.g., linked to) the virtual 3D “digital twin” model of the facility (or any portion thereof), e.g., via WiFi or other network connections. The mobile circuitry may comprise a remote (e.g., mobile) control interface. The mobile circuitry may include a pointer, gaming controller, and/or virtual reality (VR) controller. For example, the mobile circuitry may have no interaction with the physical facility, e.g., other than forwarding network communications via the aligned communication channel to and/or from the digital twin. The user interaction may not be direct and/or physical with any device being controlled in the enclosure. The user interaction of the user with the target may be indirect. The interaction of the user with the target may be devoid of tactile touch, optical ray projection, and/or vocal sound. The control actions taken by the user to control the target may be based at least in part on a relative position of the digital circuitry manipulated by a user, relative to the modeled space in the digital twin (e.g., virtual movement within the modeled enclosure). The control actions taken by the user to control the target may be not based on (e.g. and are oblivious to) the spatial relationship between the user and the digital twin. For example, a user may use a remote control pointing device, and point to a presentation portion. The presentation may be displayed on a TOLED display construct disposed in the line of sight between a user and a window (e.g., smart window). The coupling between the mobile circuitry and the target may be time based and/or may be action based. For example, the user may use the point the remote controller to the presentation, and by this couple with the presentation. The coupling may initiate on pointing in a duration that exceeds a duration threshold. The coupling may initiate by clicking the remote controller while pointing. The user may then point to a position that triggers a dropdown menu in the presentation. The dropdown menu may be visible (i) when the pointing may exceed a time threshold (ii) when the user presses button(s) on the remote controller (e.g., action based), and/or (iii) when the user performs a gesture (e.g., as disclosed herein). The user may then choose from the menu. The choice may be initiated (i) when the pointing may exceed a time threshold (ii) when the user presses button(s) on the remote controller (e.g., action based), and/or (iii) when the user performs a gesture (e.g., as disclosed herein). The actions of the user done in conjunction with the mobile circuitry (e.g., remote controller) may be communicated to the network, and thereby to the digital twin that is in turn communicated to the target. And thus, the user may indirectly communicate with the target through the digital twin. The mobile circuitry (e.g., remote controller) may be located with respect to the enclosure at one time, at time intervals, and/or continuously. Once a relative location of the mobile circuitry (e.g., remote controller) with the enclosure is determined, the user may use the remote controller anywhere (e.g., inside the enclosure, or outside of the enclosure). Outside of the enclosure may comprise in the facility or outside of the facility. For example, a conference room may establish its relative location with a remote controller. Thereafter, a user may use the relatively located remote controller to manipulate a light intensity of a light bulb disposed in the conference room while in the conference room, or while outside of the conference room (e.g., from home).
In some embodiments, the mobile circuitry (e.g., remote controller) can control a (e.g., any) interactive and/or controllable target (e.g., device) in the facility or any portion thereof, as long as (i) the target and (ii) the mobile circuitry (e.g., remote controller) are communicatively coupled to the digital twin (e.g., using the network). For example, the facility may comprise interactive targets comprising one or more sensors, emitters, tintable windows, or media displays, which devices are coupled to a communication network. In some embodiments, the user interacts with the digital twin from within the facility or from an (e.g., arbitrary) location outside the facility. For example, a remote controller device can comprise a virtual reality (VR) device, e.g., having a headset (e.g., a binocular display) and/or a handheld controller (e.g., motion sensor with or without input buttons). The mobile circuitry may comprise an Oculus Virtual Reality Player Controller (OVRPlayerController). In some embodiments, a remote control interface may be used which provides (i) visual representation to the user of the digital twin for navigation in the virtual facility, and/or (ii) user input actions for movement within the 3D model. The user input actions may include (1) pointing to an intended interactive target to be controller (e.g., to alter status of the target), (2) gestures, and/or (3) button presses, to indicate a selection action to be taken with the mobile circuitry (e.g., remote controller). The remote controller may be used to manipulate an interactive target by pointing towards them (e.g., for coupling), gesturing in other directions, and/or pressing one or more buttons operatively coupled to the mobile circuitry (e.g., buttons disposed on an envelope of the mobile circuitry). Interfacing between the mobile circuitry and the digital twin may not be carried out through a screen depicting the digital twin. Interfacing between the user and the digital twin may not be carried out through a screen showing the digital twin. Interfacing between the mobile circuitry and the digital model may not require (e.g., any) optical sensor as facilitator). Some embodiments employ a different mode of input from augmented reality applications that operate through interaction with a screen (e.g., by using an optical sensor such as a camera).
In some embodiments, a mobile circuitry (e.g., handheld controller) without any display or screen is used, which display or screen may depict a digital representation of the enclosure and/or the target. For example, instead of virtual navigation within the enclosure by the user, the actual location of the user can be determined in order to establish the location of the user in the digital twin, e.g., to use as a reference in connection with a pointing action by the user. For example, the mobile circuitry (e.g., handheld controller) may include geographic tracking capability (e.g., GPS, UWB, BLE, and/or dead-reckoning) so that location coordinates of the mobile circuitry can be transmitted to the digital twin using any suitable network connection established by the user between the mobile circuitry and the digital twin. For example, a network connection may at least partly include the transport links used by a hierarchical controller network within a facility. The network connection may be separate from the controller network of the facility (e.g., using a wireless network such as a cellular network).
In some embodiments, a user may couple to a requested target. The coupling may comprise a gesture using the mobile circuitry. The coupling may comprise an electronic trigger in the mobile circuitry. The coupling may comprise a movement, pointing, clicking gesture, or any combination thereof. For example, the coupling may initiate at least in part by pointing to the target for a period of time above a threshold (e.g., that is predetermined). For example, the coupling may initiate at least in part by clicking a button (e.g., a target selection button) on a remote controller that includes the mobile circuitry. For example, the coupling may initiate at least in part by moving the mobile circuitry towards a direction of the target. For example, the coupling may initiate at least in part by pointing a frontal portion of the mobile circuitry in a direction of the target (e.g., for a time above a first threshold) and clicking a button (e.g., for a time above a second threshold). The first and second thresholds can be (e.g., substantially) the same or different.
In the example of
In some embodiments, the mobile circuitry may be directional in at least two directions. For example, the mobile circuitry may have a front direction and a back direction. For example, the mobile circuitry may be able to distinguish between at least two, three, four, five, or six spatial directions. The directions may comprise up, down, front, back, right, or left. The directions may comprise north, south, east, and west. The directions may be relative directions, e.g., relative to the previous position of the mobile circuitry. The directions may be absolute directions (e.g., within a measurable error range). The directions may be in accordance with a Global Positioning System (GPS). Coupling of the mobile circuitry (e.g., remote controller) and the target (e.g., media projection) may comprise pointing a front direction of the mobile circuitry towards the target, e.g., for a time above a threshold. Using a network communication route from the remote controller to the digital twin, an intersection between the mobile circuitry and the target may be mapped digitally in the digital twin. The intersection may be from the tracked location of the mobile circuitry (e.g., handheld controller) along a digital ray indicated by pointing direction, e.g., to identify any requested interactive target (e.g., device and/or control element on a device). In the example shown in
In some embodiments, social interaction and/or communication is provided via the digital twin. When the digital twin is coupled to a communication network, it (e.g., intrinsically) allows for a social experience where remote participants join the facility and interact with targets (e.g., devices or media) therein via the digital twin. The concept of the digital twin may enable multi-user participation in manipulating an interactive target disposed in an enclosure; whether the participants are in the enclosure or not, and/or whether the participants are local or remote. For example, a plurality of users may access (e.g., interact with) the digital twin at the same time in a way that is perceptible to the other users. For example, when users employ VR headsets with visual displays and audio communication, they may see and/or hear one another in the virtual space represented by the digital twin. For example, when users employ video conferencing tools, they may see and/or hear one another in the virtual space represented by the digital twin. For example, a tracked user may be represented as an avatar placed within the corresponding location in the digital twin and displayed to other users. The avatar may be generic and/or may include photographic data that may be stored in advance or captured during an interaction of the user with the digital twin (e.g., using a camera or other personal identifier). The personal identifier may comprise facial recognition, fingerprint scanning, retinal scanning, or other biometric-based methods used to confirm an identity of a user.
Digital twin 1810 is accessible to a user 1813 via a communication link 1812 between digital twin 1810 and/or user interface equipment. For example, the user interface equipment can include a VR headset 1814 and a VR handheld controller 1815. Another user 1821 accesses digital twin 1810 at the same time via a communication link 1820. User 1821 may have a VR headset 1822 and a VR handheld controller 1823. In some embodiments, the digital twin 1810 may include dynamic elements for the room containing the group meeting 1806 (e.g., representations of persons seated around a conference table, representations of remote participants at virtual locations to which they have navigated within the VR model, and/or instantaneous views of the media content being displayed in the room). Digital twin 1810 may provide for exchanging audio signals captured by microphones (e.g., disposed in the room and/or the VR equipment) for reproduction for the other participants.
In some embodiments, network communication among a controller (e.g., MC), digital twin, user mobile circuitry (e.g., remote controller), and local interactive devices includes mono- or bi-directional messaging capability. For example, a combination of local area networks and/or wide area networks with appropriate gateways may be configured to facilitate (i) exchanging messages, (ii) updating of a digital twin, and/or (ii) user remote interaction with a target (e.g., for remotely controlling the interactive target). The messages may be relevant to a status change of the target, and/or to users of a meeting (without or with relation to the target, without or with relation to the enclosure in which the target is disposed, and with or without relation to the subject matter of the meeting). The controller may be configured (e.g., by appropriate software programming) to interact with the digital twin. The interaction may be for providing data identifying changes to static elements and the states of dynamic elements included in the digital twin. The digital twin may be configured to provide (i) intuitive capabilities to manipulate a target remotely, (ii) a virtual reality experience to at least one user to navigate a virtual 3D model of the enclosure, (iii) to investigate various dynamic states in the digital twin, and/or (iv) to exchange interactive (e.g., control) actions (e.g., events) related to the target, which actions are initiated by at least one user, e.g., via a virtual-reality interface. The remote manipulation may or may not comprise an electromagnetic and/or acoustic beam directed from the remote controller to the target. In some embodiments, remote manipulation may be devoid of an electromagnetic and/or acoustic beam directed from the remote controller to the target. In some embodiments, the communication coupling of the remote controller with the target may be (e.g., only) through the network that is communicatively coupled to the digital twin. In some embodiments, the communication coupling of the remote controller with the target may be (e.g., only) through the digital twin (e.g., using the network as a communication pathway that communicatively coupled the target, the digital twin, and the remote controller (comprising the mobile circuitry). The communication coupling may comprise wired and/or wireless communication. The digital twin may be configured to process a user input event, e.g., (i) to identify whether it corresponds to a valid command related to the target (e.g., from a predetermined list of valid control actions of the target) and/or (ii) to forward valid commands (e.g., to at least one controller or directly to the target) for manipulating the target (e.g., manipulating a state of the target that is manipulatable). In some embodiments, at least one controller monitors its ongoing exchange of data and/or commands with the local interactive target, e.g., to collect and/or forward updated information for the digital twin. The updated information may include any dynamic change of state, e.g., resulting from remote event(s) initiated by the user(s).
In some embodiments, messaging sequences include one or more data messages and one or more command messages exchanges between (i) one or more local targets and the processor, (ii) the processor and the digital twin, and/or (iii) the digital twin and the mobile circuitry. For example, a processor (e.g., a controller such as a MC) may send a data message to the digital twin when one or more new targets join the network from time to time. The data may represent new static and/or dynamic elements for inclusion in the digital twin 3D model of the facility. The data may represent changes in a (e.g., system) state for a dynamic element of a target.
In some embodiments, the mobile circuitry and the digital twin exchange one or more messages that enable a user to control (including to monitor and/or alter) operation of real targets (e.g., by manipulating their virtual twin elements in digital twin). For example, a user may activate their mobile circuitry (e.g., a remote gaming controller such as a VR headset and handheld VR controller (e.g., a point and click button)) to create a link with the digital twin. In some embodiments, upon an initial connection the digital twin and mobile circuitry exchange data messages with data for displaying a simulated scene in the digital twin, e.g., according to a default starting position. For example, a virtual simulation may begin at an entrance to the enclosure, or at any other point of interest (e.g., chosen by a user). In some embodiments when the user is actually located in the enclosure being represented, the starting position may correspond to the current location of the user (e.g., an initiate message may provide geographic coordinates of a GPS-equipped user remote controller). Data or commands within messages between the mobile circuitry and the digital twin may include navigation actions (resulting in updated views being returned from the digital twin) and/or control actions (e.g., point and click) to indicate a desired change in an alterable state of a target.
In some embodiments, the digital twin validates a received control action, e.g., by mapping the control action to an indicated location in the digital twin and/or checking against a list of valid actions. For example, the digital twin may only send a message to the processor (e.g., controller) when the control action event of the user corresponds to an identifiable and authorized interaction. When a valid interaction is found, a command message may be transmitted from the digital twin to the processor (e.g., controller), and forwarded to the affected target. After executing the command, one or more acknowledgement messages may propagate back to the digital twin and the 3D model of the digital twin may optionally be updated accordingly. For example, after executing a change in a tint value of an IGU, the digital twin model of the IGU may be adjusted to show a corresponding change in tint level.
In the example of
In the example of
In some embodiments, digital twin 1901 validates control actions by mapping the control action to an indicated location in the 3D model and/or checking against a list of valid actions. When a valid control action event is detected, digital twin 1901 may send a command message 1913 to processor and/or controller 1900 to identify the corresponding target and the corresponding change of state (e.g., toggling of an identified lighting circuit, or selection of a menu item in a projected display of a laptop presentation). A command message 1914 may be transmitted from processor and/or controller 1900 to the affected target 1902. After executing the command, target 1902 may send an acknowledgement message 1915 to processor and/or controller 1900. If the change is among the dynamic elements included in the digital twin, then processor and/or controller 1900 may send an update dynamic element message 1916 to digital twin 1901. If the current simulation being viewed by the user includes the dynamic element, then an update view message 1917 may be sent to remote controller 1903, e.g., to provide new data adjusted for the new dynamic state.
At time, it may be requested and/or advantageous to reduce (e.g., eliminate) direct contact between a user and a target apparatus (e.g., surface of the target apparatus). For example, reducing direct user interaction between the user and a target apparatus may reduce a risk of pathogen infection (e.g., fungi, virus, and/or bacteria), which pathogen resides in the (e.g., surface) of the device. The pathogen may be contagious and/or disease causing. The target apparatus may be an interactive target. The target apparatus may be disposed in an enclosure. The target apparatus may be a third party apparatus. The target apparatus may be a service device (e.g., a device offering service(s) to a user).
In some embodiments, the target apparatus is operatively coupled to a network. The network is operatively coupled, or includes, a control system (e.g., one or more controllers such as a hierarchal control system). In some embodiments, a mobile circuitry of a user is paired to a target apparatus (e.g., service device). The target apparatus may receive an identification tag when operatively (e.g., communicatively) coupled to the network (e.g., and to the control system). The target apparatus may be operatively coupled to a mobile circuitry through the network (e.g., using indirect coupling). The coupling between the mobile circuitry and the target apparatus may be through an application of the facility and/or of the target apparatus. There may not be a requirement for a physical proximity between the target apparatus and the mobile circuitry (e.g., and the user). The target apparatus may be selected using information related to a location of the user and/or the mobile circuitry of the user. The user may be located at a distance of at most 50 meters (m), 25 m, 10 m, 5 m, 2 m, or 1.5 m from the target apparatus. The user may be located at a distance between any of the above mentioned distances from the target apparatus (e.g., from about 50m to about 1.5m, from about 50m to about 25m, from about 25m to about 1.5m). The distance between the user and the target apparatus may be larger than the distance requires for pairing between devices (e.g., Bluetooth type pairing). There may be no need for any physical proximity between the user (and/or the mobile circuitry of the user), and the target apparatus (e.g., service device). The user may select the target apparatus (e.g., service device) from a list (e.g., dropdown menu). The user may be required to operatively coupled the mobile circuitry to the network to which the target apparatus is coupled. The communication between the mobile circuitry and the service device can be mono-directional (e.g., from the mobile circuitry to the target apparatus, or vice versa), or bidirectional between the target apparatus and the mobile circuitry (e.g., through the network). One user may control one or more target apparatuses (e.g., service device). One target apparatus may be controlled by one or more users. A plurality of users may send requests to one target apparatus, which requests may be placed in a que (e.g., based on a prioritization scheme such as time of receipt, urgency, and/or user seniority).
In some embodiments, the target apparatus is identified by the network upon connection to the network (which connection may be wired and/or wireless). The target apparatus may be identified via an identification code (e.g., RFID, QR-ID, barcode). In some embodiments, the identification code is not a visible (e.g., scannable) identification code. The identification code may comprise non-contact identification (e.g., electromagnetic and/or optical). The optically recognized identification may be a machine-readable code, e.g., consisting of an array of black and white squares or lines (e.g., barcode or a Quick Response (QR) code). The electromagnetic identifier may comprise radio-frequency identification (RFID). The RFID may be ultra-high frequency RFID. The identifier may comprise a transponder (e.g., RF transponder), a receiver, a transmitter, or an antenna. The identifier may be passive or active (e.g., transmit electromagnetic radiation). The identifier may comprise near field communication (NFC).
In some embodiments, a user may control the target apparatus (e.g., service device). For example, a user may control mechanical, electrical, electromechanical, and/or electromagnetic (e.g., optical and/or thermal) actions of the target apparatus. For example, the user may control a physical action of the target apparatus. For example, the user may control if the target apparatus is turned on or off, if any controllable compartment thereof is open or closed, direct directionality (e.g., left, right, up, down), enter and/or change settings, enable or deny access, transfer data to memory, reset data in the memory, upload and/or download software or executable code to the target apparatus, cause executable code to be run by a processor associated with and/or incorporated in the target apparatus, change channels, change volume, causing an action to return to a default setting and/or mode. The user may change a set-point stored in a data set associated with the target apparatus, configure or reconfigure software associated with the target apparatus. The memory can be associated with and/or be part of the target apparatus.
In some embodiments, the target apparatus is operatively (e.g., communicatively) coupled to the network (e.g., communication, power and/or control network) of the enclosure. Once the target apparatus becomes operatively coupled to the network of the enclosure, it may be part of the targets controlled via the digital twin. The new target (e.g., third party target) may offer one or more services to a user. For example, the target (e.g., target apparatus) may be a dispenser. The dispenser may dispense food, beverage, and/or equipment, upon a command. The service device may include media players (e.g., which media may include music, video, television, and/or internet), manufacturing equipment, medical device, and/or exercise equipment. The target apparatus may comprise a television, recording device (e.g., video cassette recorder (VCR), digital video recorder (DVR), or any non-volatile memory), Digital Versatile Disc or Digital Video Disc (DVD) player, digital audio file player (e.g., MP3 player), cable and/or satellite converter set-top box (“STBs”), amplifier, compact disk (CD) player, game console, home lighting, electrically controlled drapery (e.g., blinds), tintable window (e.g., electrochromic window), fan, HVAC system, thermostat, personal computer, dispenser (e.g., soap, beverage, food, or equipment dispenser), washing machine, or dryer. In some embodiments, the target apparatus excludes entertainment an entertainment device (e.g., a television, recording device (e.g., video cassette recorder (VCR), digital video recorder (DVR), or any non-volatile memory), Digital Versatile Disc or Digital Video Disc (DVD) player, digital audio file player (e.g., MP3 player), cable and/or satellite converter set-top box (“STBs”), amplifier, compact disk (CD) player, and/or game console). The command may be initiated by contacting the target, or by communicating (e.g., remotely) with the target. For example, a user may press a button on the target apparatus to dispense item(s) (e.g., food, beverage, and/or equipment). For example, a user may interact with the target apparatus through usage of the mobile circuitry. The mobile circuitry may comprise a cellular phone, a touchpad, or a laptop computer.
In some embodiments, the network may be a low latency network. The low latency network may comprise edge computing. For example, at least one (e.g., any) controller of the (e.g., hierarchal) control system can be a part of the computing system. For example, at least one (e.g., any) circuitry coupled to the network can be a part of the computing system. Latency (e.g., lag or delay) may refer to a time interval between a cause and its effect of some physical change in the system being observed. For example, latency and physically be a consequence of the limited velocity which any physical interaction can propagate. For example, latency may refer to a time interval between a stimulation and a response to the stimulus. For example, the latency may refer to a delay before a transfer of data begins following an instruction for transfer of the data. The network may comprise fiber optics. The latency may be at least about 3.33 microseconds (μs), or 5.0 μs for every kilometer of fiber optic path length. The latency of the network may be at most about 100 milliseconds (ms), 75 ms, 50 ms, 25 ms, 10 ms, 5 ms, 4 ms, 3 ms, 2 ms, 1 ms, or 0.5 ms. The latency of the network may be of any value between the aforementioned values (e.g., from about 100 ms to about 0.5 ms, from about 100 ms to about 50 ms, from about 50 ms to about 5 ms, or from about 5 ms to about 0.5 ms). The network may comprise a packet-switched network. The latency may be measured as the time from the source sending a packet to the destination receiving it (e.g., one way latency). The latency may be measured one-way latency from source to destination plus the one-way latency from the destination back to the source (e.g., round trip latency).
In some embodiments, the mobile circuitry includes an application related to the target apparatus (e.g., third party device). The application may depict one or more service options offered by the target apparatus. For example, if the target apparatus is a beverage dispenser, the application may offer a selection of the various beverage options offered by the target apparatus, that are available to the user. For example, if the target apparatus is a food dispenser, the application may offer a selection of the various food options offered by the target apparatus, that are available to the user. For example, if the target apparatus is a mask dispenser, the application may offer dispensing of one mask option that is available to the user.
In some embodiments, a user is locatable in the enclosure (e.g., facility such as a building). The user can be located using one or more sensors. The user may carry a tag. The tag may include radio frequency identification (e.g., RFID) technology (e.g., transceiver), Bluetooth technology, and/or Global Positional System (GPS) technology. The radio frequency may comprise ultrawide band radio frequency. The tag may be sensed by one or more sensors disposed in the enclosure. The sensor(s) may be disposed in a device ensemble. The device ensemble may comprise a sensor or an emitter. The sensor(s) may be operatively (e.g., communicatively) coupled to the network. The network may have low latency communication, e.g., within the enclosure. The radio waves (e.g., emitted and/or sensed by the tag) may comprise wide band, or ultra-wideband radio signals. The radio waves may comprise pulse radio waves. The radio waves may comprise radio waves utilized in communication. The radio waves may be at a medium frequency of at least about 300 kilohertz (KHz), 500 KHz, 800 KHz, 1000 KHz, 1500 KHz, 2000 KHz, or 2500 KHz. The radio waves may be at a medium frequency of at most about 500 KHz, 800 KHz, 1000 KHz, 1500 KHz, 2000 KHz, 2500 KHz, or 3000 KHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 300 KHz to about 3000 KHz). The radio waves may be at a high frequency of at least about 3 megahertz (MHz), 5 MHz, 8 MHz, 10 MHz, 15 MHz, 20 MHz, or 25 MHz. The radio waves may be at a high frequency of at most about 5 MHz, 8 MHz, 10 MHz, 15 MHz, 20 MHz, 25 MHz, or 30 MHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 3 MHz to about 30 MHz). The radio waves may be at a very high frequency of at least about 30 Megahertz (MHz), 50 MHz, 80 MHz, 100 MHz, 150 MHz, 200 MHz, or 250 MHz. The radio waves may be at a very high frequency of at most about 50 MHz, 80 MHz, 100 MHz, 150 MHz, 200 MHz, 250 MHz, or 300 MHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 30 MHz to about 300 MHz). The radio waves may be at an ultra-high frequency of at least about 300 kilohertz (MHz), 500 MHz, 800 MHz, 1000 MHz, 1500 MHz, 2000 MHz, or 2500 MHz. The radio waves may be at an ultra-high frequency of at most about 500 MHz, 800 MHz, 1000 MHz, 1500 MHz, 2000 MHz, 2500 MHz, or 3000 MHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 300 MHz to about 3000 MHz). The radio waves may be at a super high frequency of at least about 3 gigahertz (GHz), 5 GHz, 8 GHz, 10 GHz, 15 GHz, 20 GHz, or 25 GHz. The radio waves may be at a super high frequency of at most about 5 GHz, 8 GHz, 10 GHz, 15 GHz, 20 GHz, 25 GHz, or 30 GHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 3 GHz to about 30 GHz).
In some embodiments, the identification tag of the occupant comprises a location device. The location device (also referred to herein as “locating device”) may compromise a radio emitter and/or receiver (e.g., a wide band, or ultra-wide band radio emitter and/or receiver). The locating device may include a Global Positioning System (GPS) device. The locating device may include a Bluetooth device. The locating device may include a radio wave transmitter and/or receiver. The radio waves may comprise wide band, or ultra-wideband radio signals. The radio waves may comprise pulse radio waves. The radio waves may comprise radio waves utilized in communication. The radio waves may be at a medium frequency of at least about 300 kilohertz (KHz), 500 KHz, 800 KHz, 1000 KHz, 1500 KHz, 2000 KHz, or 2500 KHz. The radio waves may be at a medium frequency of at most about 500 KHz, 800 KHz, 1000 KHz, 1500 KHz, 2000 KHz, 2500 KHz, or 3000 KHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 300 KHz to about 3000 KHz). The radio waves may be at a high frequency of at least about 3 megahertz (MHz), 5 MHz, 8 MHz, 10 MHz, 15 MHz, 20 MHz, or 25 MHz. The radio waves may be at a high frequency of at most about 5 MHz, 8 MHz, 10 MHz, 15 MHz, 20 MHz, 25 MHz, or 30 MHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 3 MHz to about 30 MHz). The radio waves may be at a very high frequency of at least about 30 Megahertz (MHz), 50 MHz, 80 MHz, 100 MHz, 150 MHz, 200 MHz, or 250 MHz. The radio waves may be at a very high frequency of at most about 50 MHz, 80 MHz, 100 MHz, 150 MHz, 200 MHz, 250 MHz, or 300 MHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 30 MHz to about 300 MHz). The radio waves may be at an ultra-high frequency of at least about 300 kilohertz (MHz), 500 MHz, 800 MHz, 1000 MHz, 1500 MHz, 2000 MHz, or 2500 MHz. The radio waves may be at an ultra-high frequency of at most about 500 MHz, 800 MHz, 1000 MHz, 1500 MHz, 2000 MHz, 2500 MHz, or 3000 MHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 300 MHz to about 3000 MHz). The radio waves may be at a super high frequency of at least about 3 gigahertz (GHz), 5 GHz, 8 GHz, 10 GHz, 15 GHz, 20 GHz, or 25 GHz. The radio waves may be at a super high frequency of at most about 5 GHz, 8 GHz, 10 GHz, 15 GHz, 20 GHz, 25 GHz, or 30 GHz. The radio waves may be at any frequency between the aforementioned frequency ranges (e.g., from about 3 GHz to about 30 GHz).
In some embodiments, the locating device facilitates location within an error range. The error range of the locating device may be at most about 5 meters (m), 4m, 3m, 2m, 1m, 0.5m, 0.4m, 0.3m, 0.2m, 0.1m, or 0.05m. The error range of the locating device may be any value between the aforementioned values (e.g., from about 5m to about 0.05m, from about 5m to about 1m, from about 1 m to about 0.3m, and from about 0.3m to about 0.05m). The error range may represent the accuracy of the locating device.
In some embodiments, a user seeks a service from a target apparatus that is a service device. The user may approach the service device, and open an application related to the facility (or services offered by and/or in the facility) on his mobile circuitry (e.g., handheld processor). The mobile circuitry may be operatively coupled (e.g., wirelessly) to the network. In parallel, and/or as a consequence to the opening of the application, the network may ascertain a location of the user. The location of the user may be ascertained via the mobile circuitry and/or via a tag carried by the user. The tag may transmit (e.g., emit) an identification of the user and/or the location of the user. The mobile circuitry can be a hand-held mobile circuitry (e.g., a cellular phone, laptop computer, tablet computer, gaming controller, virtual reality controller, or any other remote controller). The transmission may be sensed by one or more sensors disposed in the enclosure. Ascertaining a location of the user, the application may eligible targets (e.g., service devices) in a vicinity of the user. The user may select a requested target from the eligible targets presented by the application. Selection of the service device may allow opening its interface (e.g., and thus allow selection of its services). The user may select a requested service. The user selection may be transmitted to the service device through the network, and the service device may fulfil the request of the user. In this manner, the user is not required to physically contact the service device to perform service selection. The user may then retrieve the fulfilled service. Alternatively, the user may disable the location of the service, and select the service device that is remote, to fulfil a request. The user may or may not view (e.g., in the application) a digital twin of the enclosure in which the service device is disposed. The user may employ gesture control to operate the service device. For example, the user may employ his mobile circuitry to point to a service choice visible on the service device, which service choice may be translated by the control system to a choice selection.
For example, a user seeks a café late drink from an automatic coffee dispenser that can prepare espresso, macchiato, cappuccino, café late, and mocha. The user approaches the coffee dispenser and opens a facility application on his cellular phone that is coupled to the facility network. In parallel, and/or as a consequence to the opening of the application, the network can ascertain a location of the user. The location of the user may be ascertained via the cellular phone of the user and/or via an identification tag carried (e.g., ID tag) by the user (e.g., tag that allows entry to the facility). The tag may transmit (e.g., emit) the identification of the user and/or the location of the user. The transmission may be sensed by one or more sensors disposed in the facility. Ascertaining a location of the user, the application may eligible targets (e.g., service devices) in a vicinity of the user. The user may select the coffee dispenser from the eligible targets presented by the application. In one option, selection of the coffee dispenser may allow opening an interface to allow selection between espresso, macchiato, cappuccino, café late, and mocha drinks. The user may select a café late. The user selection may be transmitted to the coffee dispenser through the network, and the coffee dispenser may fulfil the café late drink request of the user. In this manner, the user is not required to physically contact the coffee dispenser to perform service selection of the café late drink. The user may then retrieve the café late drink without contacting the coffee dispenser. In another option, selection of the coffee dispenser may allow viewing the room in which the coffee dispenser is located as a digital twin. The user may point the cellular device at a coffee drink option shown on the coffee dispenser. This gesture may be transmitted to the control system via the network and translated by the control system to a choice selection. The user selection may be transmitted to the coffee dispenser through the network, and the coffee dispenser may fulfil the café late drink request of the user. In this manner, the user is not required to physically contact the coffee dispenser to perform service selection of the café late drink. The user may then retrieve the café late drink without contacting the coffee dispenser.
In some examples, there are various target apparatuses (e.g., machines) of the same type in a facility. For example, several printers, several coffee machines, or several food dispensers. A user may send a request to a target apparatus type. The specific target apparatus of that type executing the request may be the one closest to the user. The location of the user may be ascertain via the network (e.g., using facial recognition and/or ID tag). The control system may use the location of the user to identify a specific target apparatus of the requested type for executing the requested task. A user may override such recommendation of the control system. A user may request a specific target apparatus to execute the task. Certain target apparatuses may be dedicate to certain groups of user (e.g., departments). There may be a hierarchy in the permission provided to users to use the service apparatuses. The hierarchy may depend on the location, rank, department, of the user. The hierarchy may depend on the date and time at which the request is made, and/or requested execution time of the request. The groups of users may be identified by the control system. The group of users may be identified according to their activities at work and/or outside of work. Members of the group may be informed of other group members and/or of existence of the group. At times, certain functions may be informed of the group and/or its members (e.g., human resources, management, and/or facilities). For example, in case of a fire in the facility, a group of firefighters in the facility may be informed. For example, in case of an emergency in the facility, a group of medical professionals in the facility may be informed.
In some embodiments, the user toggles between gesture control mode and tap control mode. In the gesture control mode, the user can utilize the mobile circuitry to point the mobile circuitry at the target apparatus in space. In the tap control mode, the user is not required to point the mobile circuitry at the target apparatus in space, but select options related on the target apparatus, which options appear on the mobile circuitry for selection (e.g., via a dropdown menu). The selection between options presented on the mobile circuitry can be by using a touchscreen of the mobile circuitry, and/or scrolling through the options such as by using scroll functionality implemented in the mobile circuitry (e.g., represented by arrows).
In some embodiments, the interactive target is operatively coupled to the network via a computing interface. The computing interface may comprise an application programming interface (API). The countering interface may define interactions between multiple software and/or hardware intermediaries. The computing interface may identify requests that can be made, how to make those requests, the data formats that should be used, and/or any particular conventions to follow. The computing interface may provide extension mechanisms to allow a user extension of existing functionality. For example, an API can be specific to a target, or it can be designed using an industry standard (e.g., to ensure interoperability). When a user requests a service (e.g., via the computing interface) from a service device via the mobile circuitry and/or via gesture control, the message is sent to the server (e.g., as part of the control system), the service device may be informed, and may pick the request from a server queue, process the service request, and deploys (e.g., provides) the service to be picked by the user. Examples of communication interface, messaging, and control can be found in U.S. provisional patent application Ser. No. 63/000,342 filed on Mar. 26, 2020, titled “MESSAGING IN A MULTI CLIENT NETWORK,” which is incorporated herein by reference in its entirety.
In the example of
In some embodiments, target apparatus(es) (e.g., service device(s)) can be discovered within a range from a user (e.g., using the network and the control system). In some embodiments, target apparatus(es) (e.g., service device(s)) can be discovered within a range from a target apparatus. The user range and the apparatus range can intersect. The range can be referred to herein as a “discovery range,” for example, a service apparatus discovery range. A target apparatus can be discovered by a user when the target apparatus discovery range intersects with the user discovery range. For example, a target apparatus can be discovered by a user when the user is in the target apparatus discovery range. The discovery can be using the network. The discovery can be displayed in a mobile circuitry (e.g., cellular phone) of the user. The range can be specific to a target apparatus, target apparatus type, or a set of target apparatus types. For example, a first range can be for manufacturing machines, a second range can be for media displays, and a third range can be for food service machines. The range can be specific to an enclosure, or to a portion of the enclosure. For example, a first discovery range can be for a lobby, a second discovery range can be for a cafeteria, and a third discovery range can be for an office or for a group of offices. The range can be fixed or adjustable (e.g., by a user, a manager, a facility owner, and/or a lessor). A first target apparatus type may have a different discovery range from a second target apparatus type. For example, a larger control range can be assigned for light switches, and shorter for beverage service devices. The larger control range can be of at most about 1 meter (m), 2m, 3m, or 5m. The shorter control range can be of at most about 0.2 m, 0.3m, 0.4m, 0.5m, 0.6m, 0.7m, 0.8m, or 0.9m. A user may detect (e.g., visually and/or using a list) devices within relevant use range of the user. Visually may comprise using icons, drawings, and/or a digital twin of the enclosure (e.g., as disclosed herein). Usage of discovery ranges may facilitate focusing (e.g., shortening) a list of target apparatuses relevant for the user to control, e.g., and prevent the user from having to select from a long list of (e.g., largely irrelevant) target apparatuses (e.g., service devices). Controlling the range can be using a position of the user (e.g., using a geolocation device such as one comprising UWB technology), and target apparatus paring (e.g., Wi-Fi pairing) to the network. The range of discovery be unconstrained by a rage dictated by direct device-user paring technology (e.g., Bluetooth pairing range). For example, when the user is located far from the target apparatus, the user may be able to couple with the target apparatus even if the device is out of the direct device-user paring technology range (e.g., user range). The third party target apparatus selected by the user may or may not incorporate a technology for direct device-user pairing technology.
In some embodiments, pulse-based ultra-wideband (UWB) technology (e.g., ECMA-368, or ECMA-369) is a wireless technology for transmitting large amounts of data at low power (e.g., less than about 1 millivolt (mW), 0.75 mW, 0.5 mW, or 0.25 mW) over short distances (e.g., of at most about 300 feet (′), 250′, 230′, 200′, or 150′). A UWB signal can occupy at least about 750 MHz, 500 MHz, or 250 MHz of bandwidth spectrum, and/or at least about 30%, 20%, or 10% of its center frequency. The UWB signal can be transmitted by one or more pulses. A component broadcasts digital signal pulses may be timed (e.g., precisely) on a carrier signal across a number of frequency channels at the same time. Information may be transmitted, e.g., by modulating the timing and/or positioning of the signal (e.g., the pulses). Signal information may be transmitted by encoding the polarity of the signal (e.g., pulse), its amplitude and/or by using orthogonal signals (e.g., pulses). The UWB signal may be a low power information transfer protocol. The UWB technology may be utilized for (e.g., indoor) location applications. The broad range of the UWB spectrum comprises low frequencies having long wavelengths, which allows UWB signals to penetrate a variety of materials, including various building fixtures (e.g., walls). The wide range of frequencies, e.g., including the low penetrating frequencies, may decrease the chance of multipath propagation errors (without wishing to be bound to theory, as some wavelengths may have a line-of-sight trajectory). UWB communication signals (e.g., pulses) may be short (e.g., of at most about 70 cm, 60 cm, or 50 cm for a pulse that is about 600 MHz, 500 MHz, or 400 MHz wide; or of at most about 20 cm, 23 cm, 25 cm, or 30 cm for a pulse that is has a bandwidth of about 1 GHz, 1.2 GHz, 1.3 GHz, or 1.5 GHz). The short communication signals (e.g., pulses) may reduce the chance that reflecting signals (e.g., pulses) will overlap with the original signal (e.g., pulse).
In some embodiments, an identification (ID) tag of a user can include a micro-chip. The micro-chip can be a micro-location chip. The micro-chip can incorporate auto-location technology (referred to herein also as “micro-location chip”). The micro-chip may incorporate technology for automatically reporting high-resolution and/or high accuracy location information. The auto-location technology can comprise GPS, Bluetooth, or radio-wave technology. The auto-location technology can comprise electromagnetic wave (e.g., radio wave) emission and/or detection. The radio-wave technology may be any RF technology disclosed herein (e.g., high frequency, ultra-high frequency, super high frequency. The radio-wave technology may comprise UWB technology. The micro-chip may facilitate determination of its location within an accuracy of at most about 25 centimeters, 20 cm, 15 cm, 10 cm, or 5 cm. In various embodiments, the control system, sensors, and/or antennas are configured to communicate with the micro-location chip. In some embodiments, the ID tag may comprise the micro-location chip. The micro-location chip may be configured to broadcast one or more signals. The signals may be omnidirectional signals. One or more component operatively coupled to the network may (e.g., each) comprise the micro-location chip. The micro-location chips (e.g., that are disposed in stationary and/or known locations) may serve as anchors. By analyzing the time taken for a broadcast signal to reach the anchors within the transmittable distance of the ID-tag, the location of the ID tag may be determined. One or more processors (e.g., of the control system) may perform an analysis of the location related signals. For example, the relative distance between the micro-chip and one or more anchors and/or other micro-chip(s) (e.g., within the transmission range limits) may be determined. The relative distance, know location, and/or anchor information may be aggregated. At least one of the anchors may be disposed in a floor, ceiling, wall, and/or mullion of a building. There may be at least 1, 2, 3, 4, 5, 8, or 10 anchors disposed in the enclosure (e.g., in the room, in the building, and/or in the facility). At least two of the anchors may have at least of (e.g., substantially) the same X coordinate, Y coordinate, and Z coordinate (of a Cartesian coordinate system).
In some embodiments, a window control system enables locating and/or tracking one or more devices (e.g., comprising auto-location technology such as the micro location chip) and/or at least one user carrying such device. The relative location between two or more such devices can be determined from information relating to received transmissions, e.g., at one or more antennas and/or sensors. The location of the device may comprise geo-positioning and/or geolocation. The location of the device may an analysis of electromagnetic signals emitted from the device and/or the micro-location chip. Information that can be used to determine location includes, e.g., the received signal strength, the time of arrival, the signal frequency, and/or the angle of arrival. When determining a location of the one or more components from these metrics, a localization (e.g., using trilateration such as triangulation) module may be implemented. The localization module may comprise a calculation and/or algorithm. The auto-location may comprise geolocation and/or geo-positioning. Examples of location methods may be found in PCT Patent Application serial number PCT/US17/31106 filed on May 4, 2017 titled “WINDOW ANTENNAS,” which is incorporated herein by reference in its entirety.
In some embodiments, the position of the user may be located using one or more positional sensors. The positional sensor(s) may be disposed in the enclosure (e.g., facility, building, or room). The positional sensor may be part of a sensor ensemble or separated from a sensor ensemble (e.g., standalone positional sensor). The positional sensor may be operatively (e.g., communicatively) coupled to a network. The network may be a network of the facility (e.g., of the building). The network may be configured to transmit communication and power. The network may be any network disclosed herein. The network may extend to a room, a floor, several rooms, several floors, the building, or several buildings of the facility. The network may operatively (e.g., to facilitate power and/or communication) couple to a control system (e.g., as disclosed herein), to sensor(s), emitter(s), antenna, router(s), power supply, building management system (and/or its components). The network may be coupled to personal computers of users (e.g., occupants) associated with the facility (e.g., employees and/or tenants). At least part of the network may be installed as the initial network of the facility, and/or disposed in an envelope structure of the facility. The users may or may not be present in the facility. The personal computers of the users may be disposed remote from the facility. The network may be operatively coupled to other devices in the facility that perform operations for, or associated with, the facility (e.g., production machinery, communication machinery, and/or service machinery). The production machinery may include computers, factory related machinery, and/or any other machinery configured to produce product(s) (e.g., printers and/or dispensers). The service machinery may include food and/or beverage related machinery, hygiene related machinery (e.g., mask dispenser, and/or disinfectant dispensers). The communication machinery may include media projectors, media display, touch screens, speakers, and/or lighting (e.g., entry, exit, and/or security lighting).
In some embodiments, at least one device ensemble includes at least one processor and/or memory. The processor may perform computing tasks (e.g., including machine learning and/or artificial intelligence related tasks). In this manner the network can allow low latency (e.g., as disclosed herein) and faster response time for applications and/or commands. In some embodiments, the network and circuitry coupled thereto may form a distributed computing environment (e.g., comprising CPU, memory, and storage) for application and/or service hosting to store and/or process content close to the user's mobile circuitry (e.g., cellular device, pad, or laptop).
In some embodiments, the network is coupled to device ensemble(s). The device ensemble may perform (e.g., in real time) sensing and/or tracking of occupants in an enclosure in which the device ensemble is disposed (e.g., in situ), e.g., (i) to enable seamless connectivity of the user's mobile circuitry to the network and/or adjustment of network coupled machinery to requirements and/or preferences of the user, (ii) to identify the user (e.g., using facial recognition, speech recognition, and/or identification tag), and/or (iii) to cater the environment of the enclosure according to any preferences of the user. For example, when a meeting organizer enters into an allocated meeting room, the organizer may be recognized by one or more sensors (e.g., using facial recognition and/or ID tag), presentation of the organizer may appear on screens of the meeting room and/or of screens of processors of the invitees. The screen may be controlled (e.g., remotely by the organizer or invitees, e.g., as disclosed herein). The invitees can be in the meeting room, or remote. The organizer can connect to an assistant via the network. The assistant can be real or virtual (e.g., digital office assistant). The organizer can place one or more requests to the assistant, which requests may be satisfied by the assistant. The requests may require communication and/or control using the network. For example, the request may be retrieval of a file and/or file manipulation (e.g., during the meeting). The request may be altering a function controlled by the control system (e.g., dim the lights, cool the room environment, sound an alarm, shut doors of the facility, and/or halt operation of a factory machinery). The assistant (e.g., digital assistant) may take notes during the meeting (e.g., using speech recognition), schedule meetings, and/or update files. The assistant may analyze (e.g., read) emails and/or replies to them. An occupant may interact with the assistant in a contactless (e.g., remote) manner, e.g., using gesture and/or voice interactions (e.g., as disclosed herein).
As shown in the example of
In the example shown in
In some cases, a small cell system is made available to a building, at least in part, via one or more antennas. Examples of antennas, sky sensor, and control systems can be found in U.S. patent application Ser. No. 15/287,646, filed Oct. 6, 2016, which is incorporated herein by reference in its entirety.
In some embodiments, the target apparatus is operatively coupled to the network. The network may be operatively (e.g., communicatively) coupled to one or more controllers. The network may be operatively (e.g., communicatively) coupled to one or more processors. Coupling of the target apparatus to the network may allow contactless communication of a user with the target apparatus using a mobile circuitry of the user (e.g., through a software application installed on the mobile circuitry). In this manner, a user need not directly communicatively couple and decouple from the service device (e.g., using Bluetooth technology). By coupling the target apparatus to the network to which the user is communicatively coupled (e.g., through the mobile circuitry of the user), a user may be communicatively couple to a plurality of target apparatuses simultaneously (e.g., concurrently). The user may control at least two of the plurality of target apparatuses sequentially. The user may control at least two of the plurality of target apparatuses simultaneously (e.g., concurrently). For example, a user may have two applications of two different target apparatuses open (e.g., and running) on his mobile circuitry, e.g., available for control (e.g., manipulation).
In some example, the discovery of target apparatus by a user is not restricted by a range. The discovery of target apparatus by a user can be restricted by at least one security protocol (e.g., dangerous manufacturing machinery may be available only to permitted manufacturing personnel). The security protocol can have one or more security levels. The discovery of target apparatus by a user can be restricted by apparatuses in a room, floor, building, or facility in which the user is located. The user may override at least one (e.g., any) range restriction and select the target apparatus from all available target apparatuses.
In some embodiments, the target apparatus is communicatively coupled to the network. The target device may utilize a network authentication protocol. The network authentication protocol may open one or more ports for network access. The port(s) may be opened when an organization and/or a facility authenticates (e.g., through network authentication) an identity of a target apparatus that attempts to operatively couple (and/or physically couples) to the network. Operative coupling may comprise communicatively coupling. The organization and/or facility may authorize (e.g., using the network) access of the target apparatus to the network. The access may or may not be restricted. The restriction may comprise one or more security levels. The identity of the target apparatus can be determined based on the credentials and/or certificate. The credentials and/or certificate may be confirmed by the network (e.g., by a server operatively coupled to the network). The authentication protocol may or may not be specific for physical communication (e.g., Ethernet communication) in a local area network (LAN), e.g., that utilizes packets. The standard may be maintained by the Institute of Electrical and Electronics Engineers (IEEE). The standard may specify the physical media (e.g., target apparatus) and/or the working characteristics of the network (e.g., Ethernet). The networking standard may support virtual LANs (VLANs) on a local area (e.g., Ethernet) network. The standard may support power over local area network (e.g., Ethernet). The network may provide communication over power line (e.g., coaxial cable). The power may be direct current (DC) power. The power may be at least about 12 Watts (W), 15 W, 25 W, 30 W, 40 W, 48 W, 50 W, or 100 W. The standard may facilitate mesh networking. The standard may facilitate a local area network (LAN) technology and/or wide area network (WAN) applications. The standard may facilitate physical connections between target apparatuses and/or infrastructure devices (hubs, switches, routers) by various types of cables (e.g., coaxial, twisted wires, copper cables, and/or fiber cables). Examples of network authentication protocols can be 802.1X, or KERBEROS. The network authentication protocol may comprise secret-key cryptography. The network can support (e.g., communication) protocols comprising 802.3, 802.3af (PoE), 802.3at (PoE+), 802.1Q, or 802.11s. The network may support a communication protocol for Building Automation and Control (BAC) networks (e.g., BACnet). The protocol may define service(s) used to communicate between building devices. The protocol services may include device and object discovery (e.g., Who-Is, I-Am, Who-Has, and/or I-Have). The protocol services may include Read-Property and Write-Property (e.g., for data sharing). The network protocol may define object types (e.g., that are acted upon by the services). The protocol may define one or more data links/physical layers (e.g., ARCNET, Ethernet, BACnet/IP, BACnet/IPv6, BACnet/MSTP, Point-To-Point over RS-232, Master-Slave/Token-Passing over RS-485, ZigBee, and/or LonTalk). The protocol may be dedicated to devices (e.g., Internet of Things (IoT) devices and/or machine to machine (M2M) communication). The protocol may be a messaging protocol. The protocol may be a publish—subscribe protocol. The protocol may be configured for messaging transport. The protocol may be configured for remote devices. The protocol may be configured for devices having a small code footprint and/or minimal network bandwidth. The small code footprint may be configured to be handled by microcontrollers. The protocol may have a plurality of quality of service levels including (i) at most once, (ii) at least once, and/or (iii) exactly once. The plurality of quality of service levels may increase reliability of the message delivery in the network (e.g., to its target). The protocol may facilitate messaging (i) between device to cloud and/or (ii) between cloud to device. The messaging protocol is configured for broadcasting messages to groups of targets such as target apparatuses (e.g., devices), sensors, and/or emitters. The protocol may comply with Organization for the Advancement of Structured Information Standards (OASIS). The protocol may support security schemes such as authentication (e.g., using tokens). The protocol may support access delegation standard (e.g., OAuth). The protocol may support granting a first application (and/or website) access to information on a second application (and/or website) without providing the second with a security code (e.g., token and/or password) relating to the first application. The protocol may be a Message Queuing Telemetry Transport (MQTT) or Advanced Message Queuing Protocol (AMQP) protocol. The protocol may be configured for a message rate of at least one (1) message per second per publisher. The protocol may be configured to facilitate a message payload size of at most 64, 86, 96, or 128 bytes. The protocol may be configured to communicate with any device (e.g., from a microcontroller to a server) that operates a protocol compliant (e.g., MQTT) library and/or connects to compliant broker (e.g., MQTT broker) over a network. Each device (e.g., target apparatus, sensor, or emitter) can be a publisher and/or a subscriber. A broker can handle millions of concurrently connected devices, or less than millions. The broker can handle at least about 100, 10000, 100000, 1000000, or 10000000 concurrently connected devices. In some embodiments, the broker is responsible for receiving (e.g., all) messages, filtering the messages, determining who is interested in each message, and/or sending the message to these subscribed device (e.g., broker client). The protocol may require internet connectivity to the network. The protocol may facilitate bi-directional, and/or synchronous peer-to-peer messaging. The protocol may be a binary wire protocol. Examples of such network protocol, control system, and network can be found in U.S. provisional patent application Ser. No. 63/000,342 filed Mar. 26, 2020 titled “MESSAGING IN A MULTI CLIENT NETWORK,” which is incorporated herein by reference in its entirety.
Examples of network security, communication standards, communication interface, messaging, coupling of devices to the network, and control can be found in U.S. provisional patent application Ser. No. 63/000,342, and in PCT patent application serial number PCT/US20/70123 filed Jun. 4, 2020, titled “SECURE BUILDING SERVICES NETWORK,” each of which is incorporated herein by reference in its entirety.
In some embodiments, the network allows a target apparatus to couple to the network. The network (e.g., using controller(s) and/or processor(s)) may let the target apparatus join the network, authenticate the target apparatus, monitor activity on the network (e.g., activity relating to the target apparatus), facilitate performance of maintenance and/or diagnostics, and secure the data communicated over the network. The security levels may allow bidirectional or monodirectional communication between a user and a target apparatus. For example, the network may allow only monodirectional communication of the user to the target apparatus. For example, the network may restrict availability of data communicated through the network and/or coupled to the network, from being accessed by a third party owner of a target apparatus (e.g., service device). For example, the network may restrict availability of data communicated through the network and/or coupled to the network, from being accessed by the organization and/or facility into data relating to a third party owner and/or manufacturer of a target apparatus (e.g., service device).
In some embodiments, the control system is operatively coupled to a learning module. The learning module may utilize a learning scheme, e.g., comprising artificial intelligence. The learning module may be learn preference of one or more users associated with the facility. Users associated with the facility may include occupants of the facility and/or users associated with an entity residing and/or owning the facility (e.g., employees of a company residing in the facility). The learning modules may analyze preference of a user or a group of users. The learning module may gather preferences of the user(s) as to one or more environmental characteristic. The learning module may use past preference of the user as a learning set for the user or for the group to which the user belongs. The preferences may include environmental preference or preferences related to a target apparatus (e.g., service machine, and/or production machine).
In some embodiments, a control system conditions various aspects of an enclosure. For example, the control system may condition an environment of the enclosure. The control system may project environmental preferences of the user and condition the environment to these preferences in advance (e.g., at a future time). The preferential environmental characteristic(s) may be allocated according to (i) user or group of users, (ii) time, (iii) date, and/or (iv) space. The data preferences may comprise seasonal preferences. The environmental characteristics may comprise lighting, ventilation speed, atmospheric pressure, smell, temperature, humidity, carbon dioxide, oxygen, VOC(s), particulate matter (e.g., dust), or color. The environmental characteristics may be a preferred color scheme or theme of an enclosure. For example, at least a portion of the enclosure can be projected with a preferred theme (e.g., projected color, picture or video). For example, a user is a heart patient and prefers (e.g., requires) an oxygen level above the ambient oxygen level (e.g., 20% oxygen) and/or a certain humidity level (e.g., 70%). The control system may condition the atmosphere of the environment for that oxygen and humidity level when the heart patient occupant is in a certain enclosure (e.g., by controlling the BMS).
In some embodiments, a control system may operate a target apparatus according to preference of a user or a group of users. The preferences may be according to past behavior of the user(s) in relation to the target apparatus (e.g., settings, service selection, timing related selections, and/or location related selections). For example, a user may refer coffee late with 1 teaspoon of sugar at 9 am from the coffee machine near his desk at a first location. The coffee machine at the first location may automatically generate a cup of such coffee at 9 am in the first location. For example, a user group such as a work-team prefers to enter a conference room having a forest background, with a light breeze at 22° C. The control system may control project the forest background (e.g., on a wall and/or on a media screen), adjust the ventilation system to have a light breeze, and adjust the HVAC system for 22° C. in every conference room when this group is holding a meeting. The control system may facilitate such control by controlling the HVAC system, projector, and/or media display.
In some embodiments, the control system may adjust the environment and/or target apparatus according to hierarchical preferences. When several different users (e.g., of different groups) are gathered in an enclosure, which users have conflicting preferences, the control system may adjust the environment and/or target apparatus according to a pre-established hierarchy. The hierarchy may comprise jurisdictional (e.g., health and/or safety) standards, health, safety, employee rank, activity taking place in the enclosure, number of occupants in the enclosure, enclosure type, time of day, date, season, and/or activity in the facility.
In some embodiments, the control system considers results (e.g., scientific and/or research based results) regarding environmental conditions that affect health, safety and/or performance of enclosure occupants. The control system may establish thresholds and/or preferred window-ranges for one or more environmental characteristic of the enclosure (e.g., of an atmosphere of the enclosure). The threshold may comprise a level of atmospheric component (e.g., VOC and/or gas), temperature, and time at a certain level. The certain level may be abnormally high, abnormally low, or average. For example, the controller may allow short instances of abnormally high VOC level, but not prolonged time with that VOC level. The control system may automatically override preference of a user if it contradicts health and/or safety thresholds. Health and/or safety thresholds may be at a higher hierarchical level relative to a user's preference. The hierarchy may utilize majority preferences. For example, if two occupants of a meeting room have one preference, and the third occupant has a conflicting preference, then the preferences of the two occupants will prevail (e.g., unless they conflict health and/or safety considerations).
In some embodiments, a user expresses at least one preference environmental characteristic(s) and/or target apparatus, which preference constitutes an input. The input may be by manual input (including tactile, voice and/or gesture command). A past expression of preference (e.g., input) may be registered in a database and linked to the user. The user may be part of a group of users. The group of users may be any grouping disclosed herein. The preference of the user may be linked to the group to which the user belongs. The user may enter an enclosure at a prescheduled time. The environmental characteristic(s) of the enclosure may be adjusted to the user preference (i) when the user was scheduled to enter the enclosure and/or (ii) when one or more sensors sense presence of the user in the enclosure. The environmental characteristic(s) of the enclosure may be return to a default level and/or adjusted to another's preference (i) when the scheduled presence of the user in the enclosure terminates and/or (ii) when one or more sensors sense absence of the user in the enclosure. The target apparatus may be adjusted to the user preference (i) when the user was scheduled to use the target apparatus and/or (ii) when one or more sensors sense presence of the user near the target apparatus (e.g., within a predetermined distance threshold). The target apparatus may return to default setting or be adjusted to another's preference (i) when the scheduled use of the target apparatus by the user ends and/or (ii) when one or more sensors sense absence of the user near the target apparatus (e.g., within a predetermined distance threshold).
In some examples, a target apparatus is a tintable window (e.g., an electrochromic window). In some embodiments, a dynamic state of an electrochromic window is controlled by altering a voltage signal to an electrochromic device (ECD) used to provide tinting or coloring. An electrochromic window can be manufactured, configured, or otherwise provided as an IGU. IGUs may serve as the fundamental constructs for holding electrochromic panes (also referred to as “lites”) when provided for installation in a building. An IGU lite or pane may be a single substrate or a multi-substrate construct, such as a laminate of two substrates. IGUs, especially those having double- or triple-pane configurations, can provide a number of advantages over single pane configurations; for example, multi-pane configurations can provide enhanced thermal insulation, noise insulation, environmental protection and/or durability when compared with single-pane configurations. A multi-pane configuration also can provide increased protection for an ECD, for example, because the electrochromic films, as well as associated layers and conductive interconnects, can be formed on an interior surface of the multi-pane IGU and be protected by an inert gas fill in the interior volume of the IGU.
In some embodiments, a tintable window exhibits a (e.g., controllable and/or reversible) change in at least one optical property of the window, e.g., when a stimulus is applied. The stimulus can include an optical, electrical and/or magnetic stimulus. For example, the stimulus can include an applied voltage. One or more tintable windows can be used to control lighting and/or glare conditions, e.g., by regulating the transmission of solar energy propagating through them. One or more tintable windows can be used to control a temperature within a building, e.g., by regulating the transmission of solar energy propagating through them. Control of the solar energy may control heat load imposed on the interior of the facility (e.g., building). The control may be manual and/or automatic. The control may be used for maintaining one or more requested (e.g., environmental) conditions, e.g., occupant comfort. The control may include reducing energy consumption of a heating, ventilation, air conditioning and/or lighting systems. At least two of heating, ventilation, and air conditioning may be induced by separate systems. At least two of heating, ventilation, and air conditioning may be induced by one system. The heating, ventilation, and air conditioning may be induced by a single system (abbreviated herein as “HVAC). In some cases, tintable windows may be responsive to (e.g., and communicatively coupled to) one or more environmental sensors and/or user control. Tintable windows may comprise (e.g., may be) electrochromic windows. The windows may be located in the range from the interior to the exterior of a structure (e.g., facility, e.g., building). However, this need not be the case. Tintable windows may operate using liquid crystal devices, suspended particle devices, microelectromechanical systems (MEMS) devices (such as micro-shutters), or any technology known now, or later developed, that is configured to control light transmission through a window. Examples of windows (e.g., with MEMS devices for tinting) are described in U.S. patent application Ser. No. 14/443,353 filed May 15, 2015, titled “MULTI-PANE WINDOWS INCLUDING ELECTROCHROMIC DEVICES AND ELECTROMECHANICAL SYSTEMS DEVICES,” that is incorporated herein by reference in its entirety. In some cases, one or more tintable windows can be located within the interior of a building, e.g., between a conference room and a hallway. In some cases, one or more tintable windows can be used in automobiles, trains, aircraft, and other vehicles, e.g., in lieu of a passive and/or non-tinting window.
In some embodiments, the tintable window comprises an electrochromic device (referred to herein as an “EC device” (abbreviated herein as ECD, or “EC”). An EC device may comprise at least one coating that includes at least one layer. The at least one layer can comprise an electrochromic material. In some embodiments, the electrochromic material exhibits a change from one optical state to another, e.g., when an electric potential is applied across the EC device. The transition of the electrochromic layer from one optical state to another optical state can be caused, e.g., by reversible, semi-reversible, or irreversible ion insertion into the electrochromic material (e.g., by way of intercalation) and a corresponding injection of charge-balancing electrons. For example, the transition of the electrochromic layer from one optical state to another optical state can be caused, e.g., by a reversible ion insertion into the electrochromic material (e.g., by way of intercalation) and a corresponding injection of charge-balancing electrons. Reversible may be for the expected lifetime of the ECD. Semi-reversible refers to a measurable (e.g., noticeable) degradation in the reversibility of the tint of the window over one or more tinting cycles. In some instances, a fraction of the ions responsible for the optical transition is irreversibly bound up in the electrochromic material (e.g., and thus the induced (altered) tint state of the window is not reversible to its original tinting state). In various EC devices, at least some (e.g., all) of the irreversibly bound ions can be used to compensate for “blind charge” in the material (e.g., ECD).
In some implementations, suitable ions include cations. The cations may include lithium ions (Li+) and/or hydrogen ions (H+) (i.e., protons). In some implementations, other ions can be suitable. Intercalation of the cations may be into an (e.g., metal) oxide. A change in the intercalation state of the ions (e.g. cations) into the oxide may induce a visible change in a tint (e.g., color) of the oxide. For example, the oxide may transition from a colorless to a colored state. For example, intercalation of lithium ions into tungsten oxide (WO3-y (0<y≤˜0.3)) may cause the tungsten oxide to change from a transparent state to a colored (e.g., blue) state. EC device coatings as described herein are located within the viewable portion of the tintable window such that the tinting of the EC device coating can be used to control the optical state of the tintable window.
In some embodiments, an IGU includes two (or more) substantially transparent substrates. For example, the IGU may include two panes of glass. At least one substrate of the IGU can include an electrochromic device disposed thereon. The one or more panes of the IGU may have a separator disposed between them. An IGU can be a hermetically sealed construct, e.g., having an interior region that is isolated from the ambient environment. A “window assembly” may include an IGU. A “window assembly” may include a (e.g., stand-alone) laminate. A “window assembly” may include one or more electrical leads, e.g., for connecting the IGUs and/or laminates. The electrical leads may operatively couple (e.g. connect) one or more electrochromic devices to a voltage source, switches and the like, and may include a frame that supports the IGU or laminate. A window assembly may include a window controller, and/or components of a window controller (e.g., a dock).
In some embodiments, (e.g., each of the) first and/or the second panes 2504 and 2506 are transparent and/or translucent to light, e.g., in the visible spectrum. For example, (e.g., each of the) first and/or second panes 2504 and 2506 can be formed of a glass material (e.g., an architectural glass or other shatter-resistant glass material such as, for example, a silicon oxide (SOx)-based glass material. The (e.g., each of the) first and/or second panes 2504 and 2506 may be a soda-lime glass substrate or float glass substrate. Such glass substrates can be composed of, for example, approximately 75% silica (SiO2) as well as Na2O, CaO, and several minor additives. However, the (e.g., each of the) first and/or the second panes 2504 and 2506 can be formed of any material having suitable optical, electrical, thermal, and mechanical properties. For example, other suitable substrates that can be used as one or both of the first and the second panes 2504 and 2506 can include other glass materials as well as plastic, semi-plastic and thermoplastic materials (for example, poly(methyl methacrylate), polystyrene, polycarbonate, allyl diglycol carbonate, SAN (styrene acrylonitrile copolymer), poly(4-methyl-1-pentene), polyester, polyamide), and/or mirror materials. In some embodiments, (e.g., each of the) first and/or the second panes 2504 and 2506 can be strengthened, for example, by tempering, heating, or chemically strengthening.
In
In the example of
Configuration examples of IGUs are described in U.S. Pat. No. 8,164,818, issued Apr. 24, 2012 and titled ELECTROCHROMIC WINDOW FABRICATION METHODS (Attorney Docket No. VIEWP006), U.S. patent application Ser. No. 13/456,056 filed Apr. 25, 2012 and titled ELECTROCHROMIC WINDOW FABRICATION METHODS (Attorney Docket No. VIEWP006X1), PCT Patent Application No. PCT/US2012/068817 filed Dec. 10, 2012 and titled THIN-FILM DEVICES AND FABRICATION (Attorney Docket No. VIEWP036WO), U.S. Pat. No. 9,454,053, issued Sep. 27, 2016 and titled THIN-FILM DEVICES AND FABRICATION (Attorney Docket No. VIEWP036US), and PCT Patent Application No. PCT/US2014/073081, filed Dec. 13, 2014 and titled THIN-FILM DEVICES AND FABRICATION (Attorney Docket No. VIEWP036X1WO), each of which is hereby incorporated by reference in its entirety.
In the example shown in
In some embodiments, the selection or design of the electrochromic and counter electrode materials generally governs the possible optical transitions. During operation, in response to a voltage generated across the thickness of the EC stack (for example, between the first and the second TCO layers), the electrochromic layer transfers or exchanges ions to or from the counter electrode layer to drive the electrochromic layer to the desired optical state. To cause the EC stack to transition to a transparent state, a positive voltage may be applied across the EC stack (for example, such that the electrochromic layer is more positive than the counter electrode layer). In some embodiments, in response to the application of the positive voltage, the available ions in the stack reside primarily in the counter electrode layer. When the magnitude of the potential across the EC stack is reduced or when the polarity of the potential is reversed, ions may be transported back across the ion conducting layer to the electrochromic layer causing the electrochromic material to transition to an opaque state (or to a “more tinted,” “darker” or “less transparent” state). Conversely, in some embodiments using electrochromic layers having different properties, to cause the EC stack to transition to an opaque state, a negative voltage is applied to the electrochromic layer relative to the counter electrode layer. For example, when the magnitude of the potential across the EC stack is reduced or its polarity reversed, the ions may be transported back across the ion conducting layer to the electrochromic layer causing the electrochromic material to transition to a clear or “bleached” state (or to a “less tinted”, “lighter” or “more transparent” state).
In some implementations, the transfer or exchange of ions to or from the counter electrode layer also results in an optical transition in the counter electrode layer. For example, in some implementations the electrochromic and counter electrode layers are complementary coloring layers. More specifically, in some such implementations, when or after ions are transferred into the counter electrode layer, the counter electrode layer becomes more transparent, and similarly, when or after the ions are transferred out of the electrochromic layer, the electrochromic layer becomes more transparent. Conversely, when the polarity is switched, or the potential is reduced, and the ions are transferred from the counter electrode layer into the electrochromic layer, both the counter electrode layer and the electrochromic layer become less transparent.
In some embodiments, the transition of the electrochromic layer from one optical state to another optical state is caused by reversible ion insertion into the electrochromic material (for example, by way of intercalation) and a corresponding injection of charge-balancing electrons. For example, some fraction of the ions responsible for the optical transition may be irreversibly bound up in the electrochromic material. In some embodiments, suitable ions include lithium ions (Li+) and hydrogen ions (H+) (i.e., protons). In some other implementations, other ions can be suitable. Intercalation of lithium ions, for example, into tungsten oxide (WO3-y(0<y≤˜0.3)) causes the tungsten oxide to change from a transparent state to a blue state.
In some embodiments, a tinting transition is a transition from a transparent (or “translucent,” “bleached” or “least tinted”) state to an opaque (or “fully darkened” or “fully tinted”) state. Another example of a tinting transition is the reverse (e.g., a transition from an opaque state to a transparent state). Other examples of tinting transitions include transitions to and from various intermediate tint states, for example, a transition from a less tinted, lighter or more transparent state to a more tinted, darker or less transparent state, and vice versa. Each of such tint states, and the tinting transitions between them, may be characterized or described in terms of percent transmission. For example, a tinting transition can be described as being from a current percent transmission (% T) to a target % T. Conversely, in some other instances, each of the tint states and the tinting transitions between them may be characterized or described in terms of percent tinting; for example, a transition from a current percent tinting to a target percent tinting.
In some embodiments, a voltage applied to the transparent electrode layers (e.g. across the EC stack) follows a control profile used to drive a transition in an optically switchable device. For example, a window controller can be used to generate and apply the control profile to drive an ECD from a first optical state (for example, a transparent state or a first intermediate state) to a second optical state (for example, a fully tinted state or a more tinted intermediate state). To drive the ECD in the reverse direction—from a more tinted state to a less tinted state—the window controller can apply a similar but inverted profile. In some embodiments, the control profiles for tinting and lightening can be asymmetric. For example, transitioning from a first more tinted state to a second less tinted state can in some instances require more time than the reverse; that is, transitioning from the second less tinted state to the first more tinted state. In some embodiments, the reverse may be true. Transitioning from the second less tinted state to the first more tinted state can require more time. By virtue of the device architecture and materials, bleaching or lightening may not necessarily (e.g., simply) the reverse of coloring or tinting. Indeed, ECDs often behave differently for each transition due to differences in driving forces for ion intercalation and deintercalation to and from the electrochromic materials.
In
In
In
The voltage control profile 2600 illustrated and described with reference to
In some embodiments, the applied voltage or current profiles are interrupted for relatively short durations of time to provide open circuit conditions across the device. While such open circuit conditions are in effect, an actual voltage or other electrical characteristics can be measured, detected, or otherwise determined to monitor how far along an optical transition has progressed, and in some instances, to determine whether changes in the profile are desirable. Such open circuit conditions also can be provided during a hold stage to determine whether a holding voltage VHold should be applied or whether a magnitude of the holding voltage VHold should be changed. Examples related to controlling optical transitions is provided in PCT Patent Application No. PCT/US14/43514 filed Jun. 20, 2014 and titled CONTROLLING TRANSITIONS IN OPTICALLY SWITCHABLE DEVICES, which is hereby incorporated by reference in its entirety.
In one or more aspects, one or more of the functions described herein may be implemented in hardware, digital electronic circuitry, analog electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents thereof, or in any combination thereof. Certain implementations of the subject matter described in this document also can be implemented as one or more controllers, computer programs, or physical structures, for example, one or more modules of computer program instructions, encoded on a computer storage media for execution by, or to control the operation of window controllers, NCs, and/or antenna controllers. Any disclosed implementations presented as or for electrochromic windows can be more generally implemented as or for switchable optical devices (including windows, mirrors, etc.).
Various modifications to the embodiments described in this disclosure may be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the spirit or scope of this disclosure. Thus, the claims are not intended to be limited to the implementations shown herein but are to be accorded the widest scope consistent with this disclosure, the principles and the novel features disclosed herein. Additionally, a person having ordinary skill in the art will readily appreciate, the terms “upper” and “lower” are sometimes used for ease of describing the figures, and indicate relative positions corresponding to the orientation of the figure on a properly oriented page, and may not reflect the proper orientation of the devices as implemented.
Certain features that are described in this specification in the context of separate implementations also can be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation also can be implemented in multiple implementations separately or in any suitable sub combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub combination or variation of a sub combination.
Similarly, while operations are depicted in the drawings in a particular order, this does not necessarily mean that the operations are required to be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Further, the drawings may schematically depict one more example processes in the form of a flow diagram. However, other operations that are not depicted can be incorporated in the example processes that are schematically illustrated. For example, one or more additional operations can be performed before, after, simultaneously, or between any of the illustrated operations. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.
While preferred embodiments of the present invention have been shown, and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. It is not intended that the invention be limited by the specific examples provided within the specification. While the invention has been described with reference to the afore-mentioned specification, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. Furthermore, it shall be understood that all aspects of the invention are not limited to the specific depictions, configurations, or relative proportions set forth herein which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the invention described herein might be employed in practicing the invention. It is therefore contemplated that the invention shall also cover any such alternatives, modifications, variations, or equivalents. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.
This application claims the benefit of U.S. Provisional Patent Application Ser. No. 63/265,907, filed Dec. 22, 2021, entitled “EXTERNAL BEHAVIOR RECOGNITION IN AN ENCLOSURE”. This application also claims the benefit of, and is a continuation-in-part of, International Patent Application Serial No. PCT/US20211027418, filed Apr. 15, 2021, entitled “INTERACTION BETWEEN AN ENCLOSURE AND ONE OR MORE OCCUPANTS”. This application also claims the benefit of U.S. Provisional Patent Application Ser. No. 63/170,245, filed Apr. 2, 2021, entitled “DISPLAY CONSTRUCT FOR MEDIA PROJECTION AND WIRELESS CHARGING”. This application also claims the benefit of U.S. Provisional Patent Application Ser. No. 63/154,352, filed Feb. 26, 2021, entitled “DISPLAY CONSTRUCT FOR MEDIA PROJECTION AND WIRELESS CHARGING”. This application also claims the benefit of, and is a continuation-in-part of, U.S. patent application Ser. No. 17/249,148, filed Feb. 22, 2021, entitled “CONTROLLING OPTICALLY-SWITCHABLE DEVICES”. Each of the above recited patent applications is entirely incorporated herein by reference.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2022/016829 | 2/17/2022 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63265907 | Dec 2021 | US | |
| 63170245 | Apr 2021 | US | |
| 63154352 | Feb 2021 | US | |
| 63080899 | Sep 2020 | US | |
| 63010977 | Apr 2020 | US | |
| 63052639 | Jul 2020 | US | |
| 62327880 | Apr 2016 | US | |
| 61624175 | Apr 2012 | US | |
| 62248181 | Oct 2015 | US | |
| 61289319 | Dec 2009 | US | |
| 62085179 | Nov 2014 | US | |
| 62085179 | Nov 2014 | US | |
| 62248181 | Oct 2015 | US | |
| 61289319 | Dec 2009 | US | |
| 62305892 | Mar 2016 | US | |
| 62370174 | Aug 2016 | US | |
| 62370174 | Aug 2016 | US | |
| 62305892 | Mar 2016 | US | |
| 62551649 | Aug 2017 | US | |
| 62426126 | Nov 2016 | US | |
| 62607618 | Dec 2017 | US | |
| 62523606 | Jun 2017 | US | |
| 62507704 | May 2017 | US | |
| 62506514 | May 2017 | US | |
| 62490457 | Apr 2017 | US | |
| 62666033 | May 2018 | US | |
| 62607618 | Dec 2017 | US | |
| 62523606 | Jun 2017 | US | |
| 62507704 | May 2017 | US | |
| 62506514 | May 2017 | US | |
| 62490457 | Apr 2017 | US | |
| 62607618 | Dec 2017 | US | |
| 62523606 | Jun 2017 | US | |
| 62507704 | May 2017 | US | |
| 62506514 | May 2017 | US | |
| 62490457 | Apr 2017 | US | |
| 62911271 | Oct 2019 | US | |
| 62952207 | Dec 2019 | US | |
| 62975706 | Feb 2020 | US | |
| 63085254 | Sep 2020 | US | |
| 63170345 | Apr 2021 | US | |
| 63154352 | Feb 2021 | US | |
| 63115842 | Nov 2020 | US | |
| 61652021 | May 2012 | US | |
| 61289319 | Dec 2009 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16096557 | Oct 2018 | US |
| Child | 17249148 | US | |
| Parent | 16462916 | May 2019 | US |
| Child | 16946947 | US | |
| Parent | 16082793 | Sep 2018 | US |
| Child | 16462916 | US | |
| Parent | 13479137 | May 2012 | US |
| Child | 14468778 | US | |
| Parent | 13049750 | Mar 2011 | US |
| Child | 13479137 | US | |
| Parent | 13479137 | May 2012 | US |
| Child | 14468778 | US | |
| Parent | 13049750 | Mar 2011 | US |
| Child | 13479137 | US | |
| Parent | 16608157 | Oct 2019 | US |
| Child | 16950774 | US | |
| Parent | 16664089 | Oct 2019 | US |
| Child | 17083128 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17249148 | Feb 2021 | US |
| Child | 18546520 | US | |
| Parent | 14391122 | Oct 2014 | US |
| Child | 16096557 | US | |
| Parent | 16946947 | Jul 2020 | US |
| Child | 14391122 | US | |
| Parent | 14951410 | Nov 2015 | US |
| Child | 16082793 | US | |
| Parent | 14468778 | Aug 2014 | US |
| Child | 14951410 | US | |
| Parent | 12971576 | Dec 2010 | US |
| Child | 13049750 | US | |
| Parent | 13449248 | Apr 2012 | US |
| Child | 14951410 | US | |
| Parent | 14401081 | Nov 2014 | US |
| Child | 13449248 | US | |
| Parent | 16462916 | May 2019 | US |
| Child | 16946947 | US | |
| Parent | 16082793 | Sep 2018 | US |
| Child | 16462916 | US | |
| Parent | 14468778 | Aug 2014 | US |
| Child | 16082793 | US | |
| Parent | 12971576 | Dec 2010 | US |
| Child | 13049750 | US | |
| Parent | 13449248 | Apr 2012 | US |
| Child | 14951410 | US | |
| Parent | 16950774 | Nov 2020 | US |
| Child | 17249148 | US | |
| Parent | 16608157 | Oct 2019 | US |
| Child | 16608157 | US | |
| Parent | 17083128 | Oct 2020 | US |
| Child | 17249148 | US | |
| Parent | 16608159 | Oct 2019 | US |
| Child | 16664089 | US |
