Artificial intelligence (AI) may be used to identify people, packages, or situations requiring attention. Much hardware and software may be devoted to analyzing video data insofar as facial 6 recognition, etc., is concerned. Typically, such functionality is performed on a large scale through and on a network. However, given the ever-increasing capabilities of computing devices, a need exists to provide AI information processing at a local level.
Features, aspects, and advantages are described below with reference to the drawings, which are intended to illustrate, but not to limit, the invention. In the drawings, like reference characters denote corresponding features consistently throughout similar embodiments.
Mobile computing devices, such as smartphones, contain an artificial intelligence engine that may be used, which would otherwise represent unused or under-utilized computing power.
Artificial intelligence may be used to identify people and things connected to a doorbell camera and/or surveillance camera. Further, there are the paradigms of Cloud and Edge (Cloud referring to on-demand computer system resources and Edge referring to distributed data storage and processing of information near a source of data), used with processing video information.
In the case of streaming video, using inference, things may be inferred from video and/or audio information using artificial intelligence (AI). AI inference may also be referred to as computer vision or machine learning, and there are well-known ways to accomplish it. More specifically, a neural processor may use a doorbell camera and/or surveillance camera video and/or audio data to infer certain things about that data. The AI inference may inform whether any faces or human bodies or packages or pets are present in recorded camera image(s) or identified in recorded audio. Further, the AI inference may be used in theft situations where, for example, a package is stolen from a residence, etc. The resulting inference information may be reported back to a user through the Cloud on their mobile device. A relatively long amount of time elapses between getting a live call to receiving metadata about that live call. For instance, the inference may identify a person, a package, a type of vehicle, etc.
In some cases, Cloud computing may perform the AI inference.
Beyond the paradigms of Cloud and Edge, there is the Fog. The Fog refers to devices that are connected to the Edge. Instead of making inferences at the Cloud or on the Edge, inferences may be determined locally on a mobile phone (e.g., in the Fog or at the “Far” Edge). Smartphones now typically contain neural processors, and that resource may be used advantageously in connection with artificial intelligence programs used to identify objects, events, etc. Using the neural processor that may already exist in a phone also dispenses with having to place such capability at the doorbell location. Newer phones, in particular, are being sold with neural processors that are positioned near regular general-purpose processors.
Further, processing capability is more likely to be upgraded at the mobile phone than upgrading doorbell processing hardware. AI inference at the mobile computing device will be referred to herein as “In the Pocket,” the Pocket representing a place where a mobile phone May be typically found. Making these inference determinations locally at a mobile computing device provides an advantage. For instance, a mobile phone likely has much higher processing power than that available at the Edge and because its use probably involves less latency than the Cloud. Further, processing In the Pocket (i.e., in the Fog) is cheaper to use than processing in the Cloud because every time a function is used in the Cloud, that incurs a fee. By comparison, a mobile phone providing similar functionality would not incur a processing fee.
Further, there may be an advantage in connection with real-time latency since a mobile phone may be close to a device for which AI inference is required. Accordingly, a mobile phone may be connected directly to such a device (e.g., a doorbell camera) through a local area network such as Wi-Fi instead of over the Internet at large. Even in instances where a mobile phone is very remote from a device and communicating over the Internet at large, although In the Pocket inference may not necessarily offer an advantage in latency over Cloud or Edge inference, its advantage in cost remains.
An additional advantage of performing AI inference in the Fog is that Pocket Inference provides privacy assurance that doesn't require reliance on the word of a service provider, etc. For instance, Cloud providers may claim that users can trust them not to divulge personal information, e.g., showing a video to others who come to a front door, etc. Further, they may make claims such as, “We're not going to look at your information.” With pocket inference, such trust is unnecessary as that video will not lie in the Cloud since Pocket Inference doesn't allow a Cloud provider to see a copy of the surveillance images or video.
Distributed computing may also be used with the AI inference performed on a mobile computing device. Consequently, two or more devices may be used to perform a required AI inference. For instance, AI engine capability may be borrowed from a mobile phone in a network in connection with a blockchain detailing such an occurrence for accounting, billing, awarding credit among participating smartphone owners, etc. Further, it is contemplated that older smartphones without a neural processor may borrow the neural processing capability of another smartphone. In addition, the AI inference duties may be shared between a mobile device and, for instance, a personal computer, tablet, etc. In other words, the AI inference task may be off-loaded to another device or a private server (or somewhere close to the Edge) under the doorbell system owner's control. The application herein applies not only to video that streams in real-time but also to audio. Many Internet of Things (IoT) applications concern sending telemetry data up to the Cloud, where it gets processed (in connection with decision making and drawing inferences). This can be done at the Edge or in the Fog. Many Open Source libraries are available concerning computer vision and analysis that may be downloaded to a mobile computing device, such as a smartphone, and used there.
None of the steps described herein is essential or indispensable. Any of the steps can be adjusted or modified. Other or additional steps can be used. Any portion of any of the steps, processes, structures, and/or devices disclosed or illustrated in one embodiment, flowchart, or example in this specification can be combined or used with or instead of any other portion of any of the steps, processes, structures, and/or devices disclosed or illustrated in a different embodiment, flowchart, or example. The embodiments and examples provided herein are not intended to be discrete and separate from each other.
The section headings and subheadings provided herein are non-limiting. The section headings and subheadings do not represent or limit the full scope of the embodiments described in the sections to which the headings and subheadings pertain. For example, a section titled “Topic 1” may include embodiments that do not pertain to Topic 1, and embodiments described in other sections may apply to and be combined with embodiments described within the “Topic 1” section.
The various features and processes described above may be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. In addition, certain methods, events, states, or process blocks may be omitted in some implementations. The methods, steps, and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states relating thereto can be performed in other sequences that are appropriate. For example, described tasks or events may be performed in an order other than the order specifically disclosed. Multiple steps may be combined in a single block or state. The example tasks or events may be performed in serial, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed example embodiments. The example systems and components described herein may be configured differently than described. For example, elements may be added to, removed from, or rearranged compared to the disclosed example embodiments.
Conditional language used herein, such as, among others, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements, and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment. The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Conjunctive language such as the phrase “at least one of X, Y, and Z,” unless specifically stated otherwise, is otherwise understood with the context as used in general to convey that an item, term, etc. may be either X, Y, or Z. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y, and at least one of Z to each be present.
The term “and/or” means that “and” applies to some embodiments and “or” applies to some embodiments. Thus, A, B, and/or C can be replaced with A, B, and C written in one sentence and A, B, or C written in another sentence. A, B, and/or C means that some embodiments can include A and B, some embodiments can include A and C, some embodiments can include B and C, some embodiments can only include A, some embodiments can include only B, some embodiments can include only C, and some embodiments can include A, B, and C. The term “and/or” is used to avoid unnecessary redundancy.
While certain example embodiments have been described, these embodiments have been presented by way of example only and are not intended to limit the scope of the inventions disclosed herein. Thus, nothing in the foregoing description is intended to imply that any particular feature, characteristic, step, module, or block is necessary or indispensable. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the inventions disclosed herein.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2022/081983 | 12/19/2022 | WO |