System, method and apparatus for remote monitoring

Description

TECHNICAL FIELD

The present disclosure relates to monitoring systems and in particular to a web-based integrated monitoring system for small spaces.

BACKGROUND

Home security and monitoring systems require professional installation and require professional management and monitoring. For small spaces such as apartments, the installation of a monitoring system is not practical due to the investment required to install the system and the on-going expense to monitor and maintain the monitoring system. In addition, in the rental apartment market, landlords do not want the additional expense of a monitoring system and renters do not want to install as system as they cannot take it with them. Existing solutions require multiple components to be installed and provide limited controllability and monitoring capability. Accordingly, an improved monitoring system remains highly desirable.

INCORPORATION BY REFERENCE

Each patent, patent application, and/or publication mentioned in this specification is herein incorporated by reference in its entirety to the same extent as if each individual patent, patent application, and/or publication was specifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-IF show views of a monitoring unit, under an embodiment.

FIG. 2A is a system overview of the monitoring system, under an embodiment.

FIG. 2B is a system overview of the monitoring system 200A, under an alternative embodiment.

FIG. 3 is a hardware block diagram of the monitoring system, under an embodiment.

FIG. 4 is a method of operation of the monitoring unit, under an embodiment.

FIG. 5 shows a block diagram of the monitoring unit firmware, under an embodiment.

FIGS. 6A-6B is an example monitoring system dashboard user interface on a mobile device, under an embodiment.

FIG. 7 is an example monitoring system night stand mode user interface on a mobile device, under an embodiment,

FIG. 8 is an example monitoring system status screen on a mobile device, under an embodiment.

FIG. 9 is an example live streaming user interface display on a mobile device, under an embodiment.

FIG. 10 is an example live streaming multi-view user interface display on a mobile device, under an embodiment.

FIG. 11 shows monitoring system accessory control on a live streaming user interface on a mobile device, under an embodiment.

FIGS. 12A-12C show control and scheduling screens for the monitoring unit on a mobile device, under an embodiment.

FIGS. 13A-13B show a user interface for configuring rules for the monitoring system unit on a mobile device, under an embodiment.

FIG. 14A is a front perspective view of the monitoring unit with a detachable stand, under an embodiment.

FIG. 14B is a rear perspective view of the monitoring unit with a detachable stand, under an embodiment.

FIG. 15A is a front perspective view of the monitoring unit with a detachable wall bracket, under an embodiment.

FIG. 15B is a rear perspective view of the monitoring unit with a detachable wall bracket, under an embodiment.

DETAILED DESCRIPTION

Embodiments of a monitoring system are described herein. The monitoring system is configured for use in smaller spaces, for example, but is not so limited. The architecture of the monitoring system includes numerous layers that operate in concert to provide security, notification, video streaming, and home automation functions, as described in detail herein.

FIGS. 1A-1F show views of a monitoring unit 100, under an embodiment. FIG. 1A shows a front view of the monitoring unit 100. The monitoring unit 100 integrates multiple devices and/or operations into a compact form factor to provide monitoring functions. The monitoring unit 100 comprises a lens (e.g., fish-eye lens, etc.) with a camera 102 that operates to view a wide target area from a single location. A motion sensor 104 is included to detect motion within the target area. The monitoring unit 100 of an embodiment includes one or more environmental sensors for monitoring parameters of the local environment. For example, the monitoring unit 100 includes an ambient light sensor 106 and/or a temperature and humidity sensor 114. The monitoring unit 100 also includes an indicator 108 (e.g., LED indicator, etc.) to provide visual status on the operation of the monitoring unit 100. A microphone 110, a speaker 112, and a siren 116 are also included to detect noises in the environment, provide feedback, allow two way communications and alert noises. A stand 118 is coupled or connected to the monitoring unit 100 but may be removed for wall mounting applications.

FIG. 1B is a perspective view of the monitoring unit 100, under an embodiment. FIG. 1C is a rear view of the monitoring unit 100, under an embodiment. FIG. 1D is a side view of the monitoring unit 100, under an embodiment. FIG. 1E is a top view of the monitoring unit 100, under an embodiment. FIG. 1F is a bottom view of the monitoring unit 100, under an embodiment.

FIG. 2A is a system overview of the monitoring system 200, under an embodiment. The monitoring system 200 includes the monitoring unit 100 along with one or more additional components as appropriate to an installation of the monitoring unit 100. For example, the monitoring unit 100 of an embodiment is coupled to a wide area network (e.g., the Internet) via a Wi-Fi router and/or cellular data coupling or connection. The system also includes an application for installation on a user's smartphone or tablet, and a web application accessible from any computer. Additionally, the system 200 includes a cloud-based back-end supporting the mobile application, web application, and device firmware.

FIG. 2B is a system overview of the monitoring system 200A, under an alternative embodiment. The monitoring system 200A includes a plurality of monitoring units 100-1/100-N (collectively referred to as 100) along with one or more additional components as appropriate to an installation of the monitoring units 100. For example, the monitoring units 100 of an embodiment are coupled to a wide area network (e.g., the Internet) via a Wi-Fi router and/or cellular data coupling or connection. The system also includes an application for installation on a user's smartphone or tablet, and a web application accessible from any computer. Additionally, the system 200A includes a cloud-based back-end supporting the mobile application, web application, and device firmware of the monitoring units 100. The monitoring system 200A comprising a plurality of monitoring units is described in detail herein.

The monitoring unit is installed in a user's home (e.g., on the wall, using the provided wall-mount, on a flat surface, etc.) and is powered by a power source. The power source of an embodiment includes a Direct Current (DC) wall adapter, for example, but is not so limited. Battery-backup is available to ensure the security aspects of the system remain functional if there is a loss of power. The monitoring unit of an embodiment uses encryption for all outgoing and incoming data. A user can set up multiple monitoring units to monitor and secure several areas.

Users interact with the environment in which a monitoring unit is installed in a number of ways. The monitoring unit's mobile and web applications show current and historical environmental readings of its surroundings by reporting on temperature, humidity, ambient light and sound. This information is periodically uploaded by monitoring unit and stored in the cloud infrastructure using a time-series database. It is presented using current values and graphs in the Vitals section of the mobile and web application.

The monitoring unit's internal sensors provide a wealth of information or data about the device and its surroundings, for use in security and notification scenarios. The data provided includes but is not limited to information representing one or more of the following: unexpected motion within monitoring unit's field of view (FOV); temperature changes within the space; humidity changes within the space; physical movement of monitoring unit (due to vibrations or tampering); loud, unexpected sounds; and changes in ambient light.

The monitoring unit of an embodiment can be coupled or connected to various remote or peripheral devices or sensors. The monitoring unit includes a home area network (HAN) radio, or personal area network (PAN), to control and receive information from paired accessories such as sensors, switches, and dimmers. HAN-connected accessories can be controlled by way of security rules, a programmable schedule and internal sensor triggers such as ambient light and temperature. The HAN devices include but are not limited to IEEE 802.11 Wireless Local Area Network devices, and IEEE 802.15 Wireless Personal Area Network devices, for example Wi-Fi, Zigbee and/or Z-wave based devices, but are not so limited.

The monitoring unit includes couplings or connections among a variety of remote components like remote sensors and other devices at the premises, and supports discovery, installation and configuration of the remote devices coupled or connected to the system, as described in detail herein. The monitoring unit uses this self-generated sub-network to discover and manage the remote devices at the premises. The monitoring unit thus enables or forms a separate wireless and/or wired network, or sub-network, that includes some number of devices and is coupled or connected to the LAN or WAN of the host premises. The monitoring unit sub-network can include, but is not limited to, any number of other devices like wired devices, wireless devices, sensors, cameras, actuators, interactive devices, WiFi devices, and security devices to name a few. The monitoring unit manages or controls the sub-network separately or privately from other communications and transfers data and information between components of the sob-network and the LAN and/or WAN, but is not so limited.

The monitoring unit also provides a coupling or connection to a central monitoring station (CMS) data for remote monitoring of the premises. The data of an embodiment is provided to the CMS and to the remote device, but in other embodiments is provided to one of the remote device and the CMS. Under this embodiment, one or more monitoring units are coupled to the CMS via a network (e.g., one or more of WiFi, LAN, WAN, cellular, etc.). Alternatively, the monitoring units are coupled to the CMS via the network (e.g., WAN, cellular, etc.) and an intermediate server or device (e.g., remote server, etc.). In operation, the monitoring unit transmits collected data and information to the CMS based upon a user-selected state of the monitoring unit. The data transmitted by the monitoring unit includes data of the monitoring unit as well as data of and data received from devices coupled to the monitoring unit via the local sub-network. The monitoring unit automatically delivers data of one or more onboard and/or coupled devices to the CMS. The interactions and notifications between the monitoring unit and the remote CMS of an embodiment are controlled or managed by the mobile application running on the mobile device. As such, the user interface presented by the mobile application provides controls for enabling or disabling remote monitoring by the CMS; for example, a user can activate monitoring at the CMS via the mobile application when leaving town on a trip, and can deactivate CMS monitoring upon his/her return. The monitoring unit and remote server therefore provide a mechanism to activate and deactivate monitoring by the remote CMS.

An embodiment of this mechanism is an Application Programming Interface (API) using an interface technology such as REST or SOAP, for example, to send monitoring activation and deactivation messages to the CMS and to receive acknowledgements from the CMS. Other embodiments such as the client application, monitoring device, or remote server utilize the user selection to enable/disable the delivery of activity messages to the CMS, where the CMS is always available and uses the presence of messages to trigger monitoring periods. The current invention also anticipates the integration of the CMS billing system into the service to enable on-demand billing of monitoring services, and/or to offer time-based monitoring of the system (e.g. the CMS monitoring is active for a specific period of time).

Users can place the monitoring unit into one of a plurality of security modes (e.g., Home, Away, Vacation, Off) using the mobile application, thereby activating and deactivating the various security preferences (defined as rules). Other security modes, such as ‘CMS Monitoring mode’ for example, can be utilized as well to effectuate different behaviors for the device and/or for its monitoring. Rules and other configurations may be stored on the monitoring unit's firmware and as such do not require a centralized server environment. In another embodiment these rules and configurations are stored on a remote server or backed up to a remote server to facilitate replacement of a defective unit.

The monitoring unit's mobile application allows users to set rules for each security mode pertaining to notifications, home-automation actions and alarms based on a set of scenarios, Under a scenario, the monitoring unit's various sensors (both internal and externally paired) can alert a user to activity within their environment, using data from sensors. The notification options of an embodiment include but are not limited to mobile push, SMS messages, telephone calls, and electronic mail to name but a few.

Under another scenario, sensors and their corresponding actions are configured by way of the mobile application. The monitoring unit can also leverage the use of externally paired HAN sensors to drive actions and notifications. The HAN sensors can include one or more of thermostats, door sensors, actuators, door locks, garage openers, window sensors, light dimmers or switches, to name a few.

The monitoring unit under yet another scenario allows rules associated with sensors (whether externally-paired or internal) to control connected appliances by way of paired HAN dimmers and switches. Furthermore, the monitoring unit can control the state of HAN-connected appliances by way of a configurable schedule, based on time and/or sunrise/sunset based on installed location.

The monitoring unit allows a user to set up notification-only rules that are outside the scope of any security modes. These rules can result in mobile push notifications derived from the same sensors that trigger security mode rules.

The monitoring unit can alert the surrounding environment to a potential breach of security by way of a very loud siren, driven by rules associated with sensors, both internal and externally paired. The siren can also be triggered by external parties such as the CMS and/or the user from a remote device. This capability allows a remote entity to interact with the device to warn occupants or deter an intruder. Moreover, the monitoring unit's system receives weather and other environmental information. This can influence rules and also provide additional environmental status to the mobile and web applications.

The monitoring unit allows users to connect to a live stream of video and audio via their mobile application or any remote device. This video is captured and streamed using a very wide field-of-view (FOV) lens, allowing the user to electronically pan, tilt, and zoom within their space. Additionally, multiple angles of the captured live stream can be viewed at once, in a segmented fashion. Each segment represents a distinct view of the monitoring unit's surroundings and the direction and zoom level chosen by the user are retained when a user returns to the live stream.

Conventional video cameras using a wide angle lens and local on-camera ‘de warping’ removed the distortion imparted by the wide angle lens locally on the camera processor to produce a flattened image, and then streamed portions or all of the flattened image to a remote device. In these systems the remote device displayed the de-warped video, but had no ability to simulate the raw video data being presented by the lens. These conventional systems therefore were optimized for lower-end remote devices that were not capable of advanced video processing.

In contrast to these conventional camera technologies, the monitoring unit described herein comprises ‘Immersive 3D video streaming’, which transmits lens-warped video data collected at the camera to the remote device where it is de-warped. In an embodiment, the raw lens-warped video data collected at the camera is transmitted or streamed to the remote device in a highly compressed format; in various alternative embodiments the warped video data collected at the camera can be clipped or processed in some manner before being compressed and transmitted or streamed. Regardless of any pre-processing technique applied to the video data collected at the camera, the embodiments described herein transmit or stream warped video data from the monitoring unit to the remote device, and the remote device performs de-warping of the video data. However, other alternative embodiments can de-warp the video data at the monitoring unit prior to transmitting the data stream to a remote device.

The local processor of the remote device manipulates the received video data to provide an optimal user experience. A key distinction in this approach is the ability to rely upon the high performance video decoding and three-dimensional (3D) manipulation capabilities present in state of the art remote devices, which include but are not limited to smart phones, tablet computers, personal computers, and other mobile and/or portable processor-based devices. Generally, the immersive 3D video streaming process executing on the remote device decodes and decrypts the video stream to a raw video frame buffer, creates a 3D space that emulates the specific lens geometry of the camera, and maps the video frame buffer to the 3D space providing an ‘immersive 3D video view’ that allows the remote device to zoom, pan, and move around in the 3D space giving the perception of being ‘inside the lens looking around’.

The monitoring unit of an embodiment generates a Immersive 3D video stream using components comprising a lens with a wide-angle geometry, as described in detail herein, that ‘warps’ or distorts the video to obtain the wide-angle view. The monitoring unit includes an image encoder that encodes the video image into a compressed streaming format. The monitoring unit of an embodiment stores the compressed streaming format with warped video to a local storage device coupled to the monitoring unit. Alternatively, the monitoring unit stores the compressed streaming format with warped video at a remote server or other remote processing component or memory to which it is coupled via a network coupling (e.g., LAN, WAN, Internet, etc.). Alternative embodiments may use other devices (e.g., a local Digital Video Recorder-DVR, etc.) to accomplish the process of encoding, compression, and storage separately from the monitoring device itself.

The monitoring unit streams the compressed video to a remote device (e.g., smart phones, tablet computers, personal computers, other mobile and portable pro based devices, etc.). The monitoring unit of an embodiment streams the compressed video directly to the remote device. Alternatively, however, the monitoring unit streams the compressed video to the remote device via an intermediate server (e.g., relay server, intermediate DVR, etc.).

The remote device decompresses the received compressed video stream. The remote device decompresses the video stream and then further processes the resulting decompressed video images using data of the camera lens geometry, more specifically the wide-angle geometry (de-warping) of the camera lens. For example, the remote device of an embodiment decompresses the video stream using a software codec (e.g. FFMPEG) executing on a processor. The remote device of an alternative embodiment decompresses the video stream using a hardware codec. The remote device uses 3D rendering technology to map the warped video to a 3D space replicating the lens geometry. The data of the lens geometry used by the remote device to process the received video stream is used by the mobile application, under an embodiment, and is one or more of received dynamically from a remote server or monitoring unit, included in a mapping table at the mobile device, and known a priori, but is not so limited. In an alternative embodiment the lens geometry is specified as part of the data interchange associated with the video feed setup. In yet another alternative embodiment the mobile application stores data of a plurality of known lens geometries associated with the camera types supported by the application.

The remote device ‘maps’ the decompressed warped image to the 3D space representing the lens geometry and displays this 3D view using a display that is a component of or coupled to the remote device. The remote device includes a user interface that enables a user to ‘move’ around the environment of the monitoring unit by panning and zooming around the 3D space and the mapped video image. The user interface of an embodiment is generated by the mobile application, but is not so limited. The user interface of an embodiment enables a user to navigate the 3D space using pinching gestures and swiping gestures when the remote device includes a touchscreen display. Additionally, the remote device enables playing of and interacting with stored video clips generated by the monitoring unit, where the stored video clips are stored at local storage or remote server in the same way.

By way of example in an embodiment, the process for a remote device to receive the Immersive 3D video stream from the monitoring unit comprises the remote device creating a tunnel (e.g., Secure Sockets Layer (SSL), etc.) to the monitoring unit by coupling or connecting to an external port that was configured by the monitoring unit (e.g., using Universal Plug and Play (UPnP), etc.). The monitoring unit encodes the raw video image (e.g., 1280×1070) using an encoder application (e.g., H.264 with High Profile, etc.). The monitoring unit sends the encoded video stream to the mobile device using the tunnel (e.g., sends video stream using RTP tunneled in RTSP).

The mobile device of this example embodiment includes a multimedia data library (e.g., FFmpeg library, etc.) that decodes packets of the video stream (e.g., Real-time Transport Protocol (RTP) packets of H.264 stream) to an image buffer (e.g., a YUV color space image) in memory. The size of the memory buffer of an example is 1280×1070, but is not so limited. The mobile device creates an a virtual surface (e.g., Open Graphics Library (OpenGL)) through an API for rendering vector graphics. The virtual surface of this example embodiment is YUV for rendering the pan/zoomed image, where the image size is based on the mobile device, but the embodiment is not so limited. The mobile device user interface includes controls (e.g., pinch with fingers, zoom with fingers, etc.) for selecting a position of the rendered image on the display of the mobile device. Based on the selected position on the image, the mobile device takes a portion of the decoded image (e.g., YUV) and executes a de-warping and scaling algorithm (e.g., Open (IL) to produce a rendered subset image into the image surface.

Users interact with their premises during a live-streaming session by controlling appliances and speaking into their space through the monitoring unit's built-in two-way audio functionality, which streams live audio from the mobile application to the monitoring unit's speaker. The monitoring unit supports two-way voice sessions under numerous embodiments. For example, a remote device of an embodiment initiates a two-way voice session with one or more monitoring units at a premises. Similarly, a third-party monitoring station initiates a two-way voice session with one or more monitoring units at a premises. Additionally, the monitoring unit provides live video contemporaneously with or as a component of the two-way voice session with the remote device and/or third party monitoring station. The two-way voice sessions include sessions over a WAN (e.g., Internet Protocol (IP) via WiFi, etc.) and/or sessions over a cellular network (e.g., cellular voice, IP data, etc.), but are not so limited.

The monitoring unit can record video of events associated with rules triggered by internal and externally paired sensors. The monitoring unit of an embodiment continuously records video and audio in a loop, enabling it to report on an event by presenting footage before and after it occurs. Users can review these recorded events on the mobile and web applications and perform electronic pan, tilt, and zoom operations within the captured video, as though they were streaming it in real time.

The monitoring unit records a longer video if subsequent events happen in rapid succession. A single video is created which encapsulates the several events in question. The system understands that this particular video maps to several events, and vice-versa.

The monitoring unit allows users to record video in an on-demand fashion, triggered from the mobile and web applications. As with event-driven footage, users can perform electronic pan, tilt, and zoom operations within the captured on-demand video recordings via the mobile and web applications. The monitoring unit also includes smart sound detection, allowing captured loud sounds to be fed to characterization software which helps identify it, (e.g. a smoke alarm, barking dog, etc.).

The monitoring unit periodically transmits a heartbeat signal or message to the cloud or other network infrastructure, allowing the user to be notified when the monitoring unit disconnects from the Internet. The user is also notified when the monitoring unit reconnects and continues to post heartbeat information. If connectivity issues arise and events trigger video recording, the monitoring unit saves videos locally and queues them for later upload, when connectivity resumes.

The cloud infrastructure of an embodiment comprises one or more of a number of components. The components include for example, front-end web services the expose or include an Application Programming Interface (API) for both the mobile application and the monitoring unit firmware. This public traffic is encrypted and authentication is performed against strong firmware and user credentials. Additionally, back-end databases include user account information and settings, and monitoring unit configuration and time-series sensor and event data. Back-end in-memory databases house real-time sensor history, cached data and heartbeat information.

The cloud infrastructure components also include notification services and worker services. The notification services send information to users by way of direct and third-party assisted methods. These include email, mobile push notifications, SMS and voice calls. The worker services process uploaded content, check in-memory real-time data for connected devices' heartbeats, trigger notifications, start on-demand recordings for users and perform additional infrastructure and product-related functions. File storage services providing fast and reliable disk space for the entire infrastructure.

Additionally, the cloud infrastructure components include infrastructure backup services, multi-site disaster recovery for database, worker and web services, and redundancy for each component. High availability for all worker and web services is provided by way of application-level clustering and load balancing for incoming web service requests from mobile apps and firmware.

Multiple monitoring units are installed at various independent locations at a premises, and when so installed function as a distributed premises security system, under an embodiment and as described herein. The monitoring units of this collective installation automatically discover and couple to each other and share data over a device network (e.g., IP, WiFi, wired connection, Z-Wave, etc.) that is separate and independent from a LAN and/or WAN to which the monitoring units are coupled at the premises. In an embodiment the monitoring units utilize the LAN (Wifi, Ethernet, etc.) to couple to each other and share data. In another embodiment the monitoring units utilize a WAN (such as a cellular or broadband network) to couple to each other and share data. In yet another embodiment the monitoring devices are installed in physically remote locations (such as a home and an office) and are coupled via a WAN, but can still share data and form a distributed security network. The monitoring units thereby combine logically to form an integrated monitoring or security network at the premises or between premises. Each monitoring unit includes an automatic installation process for adding and removing itself from this integrated network. The monitoring units are configured to repeat at least one message between devices in the integrated security network formed by the coupling of the devices. When the installation includes multiple monitoring units, the collection of monitoring units are controlled or monitored from a single mobile application and are associated and managed with a single user account. Similarly, the collection of monitoring units is coupled to a remote service and monitored at the CMS.

FIG. 3 is a hardware block diagram of the monitoring system, under an embodiment. The monitoring unit's hardware architecture comprises but is not limited to one or more of the following components: ARM System-on-chip 302; DDR Memory 304; Flash storage 306; Home area network RF module with antenna 308; Wi-Fi (or local area network technology) module with antenna 310; Cellular data module with antenna 312 if provisioned; Camera system comprising a multi-megapixel CMOS sensor and very wide FOV lens 314; Audio system comprising a microphone 316 and speaker 318; Alarm siren 320; Passive infrared (PIR) motion sensor with a very side FOV Fresnel lens 322; Temperature sensor 324: Relative humidity sensor 326; Accelerometer 328; Ambient light sensor 330; Power system 332 comprising a DC jack 334 and battery compartment 336; RGB LED indicator 338; Push-button 340.

FIG. 4 is a method of operation of the monitoring unit, under an embodiment. The monitoring unit's firmware is based upon an operating system such as Linux, for example, but is not so limited. Specialized software or applications along with this operating system provides the services, API and functionality for set up and use of the monitoring unit's features in concert with the cloud infrastructure and mobile and web applications.

During the user's initial setup of monitoring unit, the following tasks are performed by the firmware:

- a. The monitoring unit's firmware boots.
- b. Since no existing device information is present, the monitoring unit creates a Wi-Fi access point for setup functions.
- c. User launches the mobile application and after creating an account using their information begins the setup process.
- d. User connects to monitoring unit's Wi-Fi access point and submits Wi-Fi credentials for their home network.
- e. The monitoring unit attempts to connect with the home network using the provided Wi-Fi credentials,
- f. The monitoring unit registers itself to the cloud back-end, associates with the current user and attempts to open ports on the user's Internet router (for incoming connections) using Universal Plug and Play (UPNP) or Network Address Translation (NAP) Port Mapping Protocol (PMP), depending on the type of router present.
- g. Once fully connected, the monitoring unit turns off its Wi-Fi access point and begins normal operation.
- h. In the cases where a new Wi-Fi router is present, the monitoring unit has moved to a new environment, or connectivity to the existing router fails, the monitoring unit can accept new Wi-Fi credentials in a similar fashion to the initial setup process.

Embodiments described herein include a setup or enrollment process that comprises determining geolocation of the monitoring unit during installation at the premises. The monitoring unit of an embodiment incorporates a Wifi module (processor and radio (802.11)), and during enrollment the monitoring unit puts the WiFi module into ‘Access Point mode’. The mobile device running the mobile application described in detail herein enrolls as a WiFi client to the monitoring unit access point. The mobile application then provides new WiFi credentials (e.g., service set identification (SSID), password (optional), etc.) to the monitoring unit via the Wifi access point; subsequently, the mobile application automatically switches the mobile device over to the same WiFi SSID, or the user manually switches the mobile device to that SSID using a network configuration utility. Upon receipt of the new WiFi credentials, the monitoring unit automatically switches its WiFi processor to enroll as a client at the new WiFi SSID (using the optional password). Either the monitoring unit or the mobile application initiates a process to store the WiFi credentials on a remote server or other remote device. The monitoring unit of an embodiment restores the WiFi credentials from a remote server, but the remote server of an alternative embodiment initiates restoration of the Wifi credentials of the monitoring unit.

The mobile application of an embodiment provides numerous operations, but is not so limited. For example, the mobile application provides a user interface that enables a user to switch the monitoring unit to the access point mode in order to change the SSID. The mobile application provides authentication directly to the camera (e.g. username, password, etc.). Alternatively, the mobile application provides authentication against a remote server.

The mobile application provides to one or more monitoring units location information corresponding to the monitoring unit installation, where the location information corresponding to the monitoring unit is location data determined at the mobile device. The monitoring unit then provides its location data to the remote server. Alternatively, the mobile application provides the location data of the monitoring unit installation directly to a remote server or other remote device. The monitoring unit of an embodiment includes an administrative tool that provides information about numerous monitoring units and their respective physical locations.

In an alternative embodiment the monitoring unit is temporarily coupled or connected via a physical connector (e.g. a USB cable) to a mobile device running the mobile application. In this embodiment the mobile application delivers the Wifi SSID and password over the wired connection, and the monitoring device then switches to the Wifi access point as described above.

Generally, the monitoring unit's operating state comprises but is not limited to the following:

- a. Sensor polling is running and receiving raw data from sensors,
- b. The rules engine is running and can interface with sensors,
- c. The audio and video service and RTSP server are running and are ready to accept incoming connections, record footage in a loop and detect loud sounds,
- d. The PIR motion sensor service is running and able to detect movement within the monitoring unit's FOV,
- e. Automated tasks run at their pre-defined intervals and perform, but are not limited to, one or more of the following: maintain contact or communication between the monitoring unit and the cloud back-end and ensure incoming ports remain open on the user's Internet router; check for updates to the monitoring unit's firmware; post status updates about the current environment around the monitoring unit; post heartbeats periodically to inform the cloud backend of the monitoring unit's state.
  
  Sensors and Rules

The sensor polling service reads from internal sensors (e.g., temperature, humidity, ambient light, acceleration/motion, etc.) and sends the data to the rules engine. It can also receive a signal from any other part of the firmware to force an immediate read of the sensors. All sensor data is sent to the rules engine.

The PIR motion sensor service reads from the PIR software driver directly, but is not so limited. The motion sensor, which implements a Bessel filter in order to eliminate false positives, issues a message to the Rules engine if a threshold for motion is exceeded.

When loud sound above a predefined threshold is detected, a signal is passed to the rules engine. When appropriate, the loud sound in question is passed through characterization software to help identify it, (e.g. a smoke alarm, barking dog, etc.).

The rules engine loads a list of rules for notifications for the current security mode (home, away, or vacation) from a database into memory. The rules engine also loads any HAN control rules that can be controlled via schedule, ambient light, temperature or any other sensors, be they internal or external. Notification-only rules are processed in parallel to mode-based security rules.

The rules engine saves the data with a timestamp in the monitoring unit's firmware database. The data is also sent to each active rule/control in order to determine what action, if any, should be taken (e.g. turn on an appliance, sound the siren, notify the user etc.).

Audio and Video

The audio and video service is responsible for streaming media, saving to a file and detecting loud sounds.

For saving footage to a file, audio and video are encoded and placed into a circular buffer of a certain time. This enables the monitoring unit to capture video and audio “before” an event has occurred. This queue is operating when the system is on but is not so limited.

For streaming, video and audio are encoded and served via RTP/RTSP to a user's mobile application. The streaming is encrypted and supports multiple clients at once,

FIG. 5 shows a block diagram of the monitoring unit firmware, under an embodiment. The mobile application is the user's interface to the monitoring unit. The mobile application is executed on a smartphone or other personal or mobile electronic device. Within the application, the user's account is created, security and notification rules are defined, environmental readings are displayed, live streaming takes place and other settings are submitted to the cloud back-end and the monitoring unit's firmware. The application also serves as a tool to set up the monitoring unit's hardware, enabling the monitoring unit to pair with the user's home Wi-Fi network.

Key functions are accessed from the application's tab bar at the bottom of the screen, once the user has logged into their account.

FIGS. 6A-6B show an example monitoring system dashboard user interface on a mobile device, under an embodiment. The Dashboard provides an at-a-glance view of the monitoring unit's system and provides access to one or more of the following functions: the monitoring unit's current security mode; the temperature near the monitoring unit; the monitoring unit's Wi-Fi signal strength; current status of HAN-connected accessories; weather alerts, if present; access to Events, Recordings, and Settings; access to the Night Stand.

FIG. 7 is an example monitoring system Night Stand mode user interface on a mobile device, under an embodiment. The user can activate the Night Stand mode (available from the Dashboard), providing access to the various HAN-connected control accessories, a clock, weather information, and a panic button. Pressing the panic button activates the monitoring unit's siren.

FIG. 8 is an example monitoring system status screen on a mobile device, under an embodiment. The Vitals section displays monitoring unit's internal sensor readings and external weather information in an easy-to-understand format. Historical information is displayed using graphs allowing the user to see trends for each reading.

FIG. 9 is an example live streaming user interface display on a mobile device, under an embodiment. Tapping or selecting the center icon of the mobile application's tab bar launches live streaming and couples or connects the user securely to the monitoring unit's built-in wide FOV camera. The user can then pan, tilt, and zoom the live stream of monitoring unit's surroundings, control connected HAN plug-in modules (e.g. turning on a light) and stream audio from the microphone of their mobile device to the user's space by way of monitoring unit's built-in two-way audio feature.

FIG. 10 is an example live streaming multi-view user interface display on a mobile device, under an embodiment. Users can also choose to switch to a multi-view version of the live stream, which allows them to look at different areas of their space at the same time. This is achieved by presenting the user with several smaller views of the pan, tilt, and zoom video. The video image is segmented allowing multiple areas within the field of view of the camera to be isolated. The image captured by the camera has wide angle viewing area such as provided by a fish-eye lens. The image maybe de-warped and post-processed to provide a more viewable image. The monitoring unit remembers the last-used settings of the live stream, including the direction the user was looking and the zoom level of the video.

FIG. 11 shows monitoring system accessory control on a live streaming user interface on a mobile device, under an embodiment. If the user has paired HAN control accessories to monitoring unit, they can be accessed and controlled from within the live video screen. This allows a user to turn an appliance on or off and see the results in real time, if desired.

FIGS. 12A-12C show control and scheduling screens for the monitoring unit on a mobile device, under an embodiment. The Controls section allows configuration and control of HAN-connected accessories. These accessories can be configured via one or more of a timed schedule, sunrise/sunset, ambient light level, and temperature.

FIGS. 13A-13B show a user interface for configuring rules for the monitoring system unit on a mobile device, under an embodiment. The rules section allows the user to set security-related actions for motion, loud sound and temperature change triggers. Actions can be set for each security mode and include but are not limited to one or more of the following: record video and audio of the event; notifications; push message; electronic mail; phone call; SMS message; notification to a user's Trusted Circle members; the sounding of the monitoring unit's built-in siren; control of any connected HAN switches.

Additionally, notification-only options are present which allow the user to be informed of events outside the scope of the current security mode.

Additional functionality may be provided by the camera such as motion detection and ambient light detection. The processor may use image processing to determine characteristics of the image for use in motion detection, face recognition or light detection. In addition the microphone may be used for voice recognition function of the monitoring unit.

FIG. 14A is a front perspective view of the monitoring unit with a detachable stand, under an embodiment. FIG. 14B is a rear perspective view of the monitoring unit with a detachable stand, under an embodiment.

FIG. 15A is a front perspective view of the monitoring unit with a detachable wall bracket, under an embodiment. FIG. 15B is a rear perspective view of the monitoring unit with a detachable wall bracket, under an embodiment.

Embodiments described herein include a monitoring unit comprising a camera. The monitoring unit comprises a network interface. The monitoring unit comprises a processor coupled to the camera and the network interface. The monitoring unit comprises at least one application executing on the processor. The processor receives an image from the camera. The processor receives sensor data from at least one sensor coupled to the processor. The processor generates an alert based upon a change in at least one of the image and the sensor data. The alert is sent via the network interface to a mobile device.

Embodiments described herein include a monitoring unit comprising: a camera; a network interface; a processor coupled to the camera and the network interface; and at least one application executing on the processor, wherein the processor receives an image from the camera, wherein the processor receives sensor data from at least one sensor coupled to the processor, wherein the processor generates an alert based upon a change in at least one of the image and the sensor data, wherein the alert is sent via the network interface to a mobile device.

The monitoring unit of an embodiment comprises at least one a memory device coupled to the processor.

The monitoring unit of an embodiment comprises at least one communication module coupled to the processor.

The at least one communication module comprises a home area network (HAN) radio frequency (RF) module.

The at least one communication module comprises a Wi-Fi module.

The executing of the at least one application generates an enrollment process.

The enrollment process automatically places the WiFi module into an Access Point mode.

The mobile device comprises a mobile application, wherein the mobile application enrolls as a client to the Access Point.

The mobile application provides WiFi credentials to the processor via the Access Point.

At least one of the mobile application and the processor initiate storage of the WiFi credentials on a remote server.

At least one of the remote server and the processor restore the WiFi credentials from the remote storage device.

The mobile application provides authentication against at least one of the processor and a remote server.

The processor automatically switches the WiFi module to enroll as a client using the WiFi credentials.

The mobile application automatically switches the mobile device to enroll using the WiFi credentials.

The mobile application provides a user interface that includes at least one control for switching the processor to the Access Point mode to change the WiFi credentials.

The mobile application provides to a device location information corresponding to installation of the monitoring unit.

The device comprises a remote server.

The device comprises at least one of the monitoring unit and at least one additional monitoring unit.

The monitoring unit of an embodiment comprises an administrative application that provides information about at least one monitoring unit that includes the location information.

The at least one communication module comprises a local area network (LAN) module.

The at least one communication module comprises a cellular data module.

The at least one communication module is coupled to a remote device to communicate with the remote device.

The at least one communication module is coupled to a remote device to communicate over an Internet Protocol (IP) channel.

The at least one communication module is coupled to a remote device to communicate over a cellular channel.

The communication comprises a two-way voice session with the remote device.

The communication comprises a data session, wherein video images are transmitted during the data session.

The remote device comprises the mobile device.

The remote device comprises a central monitoring station.

The communication module automatically establishes a coupling with the at least one sensor.

The communication module automatically establishes a coupling with a local area network. (LAN) at the premises.

The at least one application transfers data between at least one device on the LAN.

The communication module forms a sub-network at the premises.

The sub-network is a private network.

The at least one sensor is coupled to the sub-network.

Devices couple to the sub-network and communicate over the sub-network, wherein the devices include at least one of wireless devices, wired devices, and IP devices.

The monitoring unit of an embodiment comprises a remote server including a user account coupled to the processor.

The camera comprises an image sensor and a lens.

The camera comprises a lens including a wide-angle geometry, wherein the camera generates images including warped images, wherein the camera generates the images using a wide-angle view mapped to the geometry of the lens from collected images.

The camera comprises an encoder that encodes collected images to generate a processed data stream.

The processed data stream is a compressed video stream.

The monitoring unit of an embodiment comprises memory coupled to the camera, wherein the camera stores to the memory the processed data stream that includes warped video.

The memory is local to the camera.

The memory is remote to the camera.

The camera streams the processed data stream to a remote device, wherein the remote device comprises at least one of a mobile device and a server.

The camera streams the processed data stream directly to the remote device.

The camera streams the processed data stream to the remote device via at least one intermediary device.

Remote device processes the processed data stream using knowledge of a wide-angle geometry of the lens.

The processing comprises decompressing the processed data stream.

The remote device comprises a software codec, wherein the software codec decompresses the processed data stream.

The remote device comprises a hardware codec, wherein the hardware codec decompresses the processed data stream.

The processing comprises using three-dimensional (3D) rendering and mapping warped video to a 3D space representing at least a portion of the lens geometry.

The processing comprises displaying a 3D view of the collected images via a display coupled to the remote device.

The remote device comprises a user interface comprising control gestures for navigating around the 3D view presented via the display.

The navigating comprises at least one of panning and zooming around the 3D view presented via the display, wherein the control gestures comprise at least one of pinching gestures and swiping gestures.

The camera comprises at least one of a video camera and an imaging camera.

The camera comprises a CMOS sensor and very wide FOV lens.

The monitoring unit of an embodiment comprises an audio system.

The monitoring unit of an embodiment comprises an alarm siren.

The at least one sensor comprises a motion sensor.

The motion sensor comprises a passive infrared (PIR) motion sensor with a very wide FOV Fresnel lens.

The at least one sensor comprises an environmental sensor.

The environmental sensor comprises at least one of a temperature sensor and a humidity sensor.

The at least one sensor comprises an accelerometer.

The at least one sensor comprises an ambient light sensor.

The monitoring unit of an embodiment comprises a power system.

The monitoring unit of an embodiment comprises at least one indicator coupled to the processor.

The at least one application generates at least one notification.

The at least one notification comprises one or more of a push message, an electronic mail, a telephone call, a Short-Message-Service (SMS) message, a notification to at least one contact.

The monitoring unit is coupled to one or more accessories.

The accessories are controlled by at least one of a timed schedule, a sunrise/sunset event, an ambient light level, and a temperature.

The monitoring unit of an embodiment comprises a rules engine executing on the processor.

The monitoring unit of an embodiment comprises a mobile application installed on the mobile device.

The mobile application generates a user interface presented on the mobile device, wherein the user interface provides access to at least one of the image and the sensor data.

The at least one application is at least one of accessed and controlled using the user interface.

The at least one sensor is controlled via the user interface.

The monitoring unit of an embodiment comprises at least one actuator coupled to the processor, wherein the at least one actuator is controlled via the user interface.

The monitoring unit of an embodiment comprises a heartbeat signal generated by the processor and transmitted to a remote device.

The monitoring unit of an embodiment comprises at least one remote server coupled to the network interface.

The coupling comprises at least one of a wide area network and a cellular network.

The at least one remote server comprises a central monitoring station.

The processor transmits to the central monitoring station at least one of the image and the sensor data.

The processor transmits to the central monitoring station a message comprising information representing at least one of the image and the sensor data.

The mobile device comprises a mobile application.

The mobile application comprises an interface for enabling and disabling remote monitoring by the central monitoring station.

The mobile application comprises an interface for controlling characteristics of the message and transmission of the message.

Embodiments described herein include a monitoring unit comprising a plurality of sensors. The plurality of sensors includes an image sensor. The monitoring unit comprises a network interface. The monitoring unit comprises a processor coupled to the plurality of sensors and the network interface. The monitoring unit comprises at least one application executing on the processor. The processor receives sensor data from the plurality of sensors. The processor generates an alert based upon a change in the sensor data. The alert is sent via the network interface to a mobile device associated with a user.

Embodiments described herein include a monitoring unit comprising: a plurality of sensors, wherein the plurality of sensors include an image sensor; a network interface; a processor coupled to the plurality of sensors and the network interface; and at least one application executing on the processor, wherein the processor receives sensor data from the plurality of sensors, wherein the processor generates an alert based upon a change in the sensor data, wherein the alert is sent via the network interface to a mobile device associated with a user.

Embodiments described herein include a system for remote monitoring. The system comprises a monitoring unit at a premises. The monitoring unit comprises a processor coupled to a plurality of sensors. The plurality of sensors includes an image sensor. The processor includes at least one application executing on the processor. The processor receives sensor data from the plurality of sensors and generates monitoring unit data. The system includes a server and a database located remote to the premises and coupled to the monitoring unit via a network coupling. The server receives the sensor data and the monitoring unit data and stores the sensor data and the monitoring unit data in the database. The server provides access to the sensor data and the monitoring unit data via a mobile device.

Embodiments described herein include a system for remote monitoring, the system comprising: a monitoring unit at a premises, the monitoring unit comprising a processor coupled to a plurality of sensors, wherein the plurality of sensors include an image sensor, wherein the processor includes at least one application executing on the processor, wherein the processor receives sensor data from the plurality of sensors and generates monitoring unit data; and a server and a database located remote to the premises and coupled to the monitoring unit via a network coupling, wherein the server receives the sensor data and the monitoring unit data and stores the sensor data and the monitoring unit data in the database, wherein the server provides access to the sensor data and the monitoring unit data via a mobile device.

The processor generates an alert based upon a change in at least one of the sensor data and the monitoring unit data, wherein the alert is sent to the mobile device.

The system of an embodiment comprises at least one communication module coupled to the processor.

The at least one communication module comprises a home area network (HAN) radio frequency (RF) module.

The at least one communication module comprises a Wi-Fi module.

The executing of the at least one application generates an enrollment process.

The enrollment process automatically places the WiFi module into an Access Point mode.

The mobile device comprises a mobile application, wherein the mobile application enrolls as a client to the Access Point.

The mobile application provides WiFi credentials to the processor via the Access Point.

At least one of the mobile application and the processor initiate storage of the WiFi credentials on the server.

At least one of the server and the processor restore the WiFi credentials from the server.

The mobile application provides authentication against at least one of the processor and a server.

The processor automatically switches the WiFi module to enroll as a client using the WiFi credentials.

The mobile application automatically switches the mobile device to enroll using the WiFi credentials.

The mobile application provides a user interface that includes at least one control for switching the processor to the Access Point mode to change the WiFi credentials.

The mobile application provides to a device location information corresponding to installation of the monitoring unit.

The device comprises the server.

The device comprises at least one of the monitoring unit and at least one additional monitoring unit.

The system of an embodiment comprises an administrative application that provides information about at least one monitoring unit that includes the location information.

The at least one communication module comprises a local area network (LAN) module.

The at least one communication module comprises a cellular data module.

The at least one communication module is coupled to a remote device to communicate with the remote device.

The at least one communication module is coupled to a remote device to communicate over an Internet Protocol (IP) channel.

The at least one communication module is coupled to a remote device to communicate over a cellular channel.

The communication comprises a two-way voice session with the remote device.

The communication comprises a data session, wherein video images are transmitted during the data session.

The remote device comprises the mobile device.

The remote device comprises a central monitoring station.

The communication module automatically establishes a coupling with the plurality of sensors.

The communication module automatically establishes a coupling with a local area network (LAN) at the premises.

The at least one application transfers data between at least one device on the LAN.

The communication module forms a sub-network at the premises.

The sub-network is a private network.

The plurality of sensors are coupled to the sub-network.

Devices couple to the sub-network and communicate over the sub network, wherein the devices include at least one of wireless devices, wired devices, and IP devices.

The server includes a user account.

The image sensor comprises a camera including a lens.

The camera comprises an encoder that encodes collected images to generate a processed data stream.

The processed data stream is a compressed video stream.

The system of an embodiment comprises memory coupled to the camera, wherein the camera stores to the memory the processed data stream that includes warped video.

The memory is local to the camera.

The memory is remote to the camera.

The camera streams the processed data stream to a remote device, wherein the remote device comprises at least one of the mobile device and the server.