Patent Application
20060190419
Bunn et al, U.S. patent application Ser. No. 10/626,888 (filed Jul. 25, 2003), “Voice, Lip-reading, Face and Emotion Stress Analysis, Fuzzy Logic Intelligent Camera System”
Bunn et al, U.S. patent application (filed Dec. 6, 2004, number not yet assigned), “Data Analysis Algorithms for a Voice, Lip-reading, Face, Emotion, Intoxication Impairment and Violent Behavior Stress Analysis, Fuzzy Logic Intelligent Camera System.”
Not Applicable
Not Applicable
Video surveillance for security of things, places, and people has long been a major area of patenting of methods, systems, techniques, and technology. Analyses of data from security video cameras for security surveillance are well known. Lemelson, in 1991 U.S. Pat. No. 5,067,012, reveals a method and system for scanning and inspecting video camera images for automated recognition of objects, and Higashimura et al, in 2002 U.S. Pat. No. 6,747,554, reveal a network surveillance unit and means for recording video security camera images which can be related to local alarm signals with methods and means for storing, viewing and distributing these data via internet WEB communications. Alexander et al, in 2005 U.S. Pat. No. 6,839,731, reveal shared network methods and means for sharing information from databases via internet WEB communications with secure access implementation of ID cards and PIN numbers, video data surveillance, voice recognition and such high security systems to maintain secure information communications.
The well-known video surveillance technology for recognition of objects, such as revealed in the Lemelson 1991 U.S. Pat. No. 5,067,012 for detecting objects, has been applied to detecting and recognizing specific persons and tracking their movement by head appearance and movement of subjects as revealed in the Darrell et al 2002 U.S. Pat. No. 6,445,810. Refining observations to a singular subject, video surveillance methods have been applied to the actual movement of body parts of a subject, such as tracking eye movement in the Strachan 1999 U.S. Pat. No. 5,980,041 by reflecting infrared light from a hologram off the retina of the subject and using triangulation of the reflected light for development of physiological measurement tools for eye focus and movement, and in the Harman 2002 U.S. Pat. No. 6,459,446 by reflecting infrared light off the cornea of the subject and using multiple cameras to track eye movement for development of viewing technology for 3-D video.
Significantly advancing the video surveillance technology, Bunn et al, in 2003 U.S. patent application Ser. No. 10/626,888, teach a voice, lip-reading, face and emotion stress fuzzy logic intelligent camera system which analyzes digital video data to automatically detect stress on people, animals or things for the purpose of recognizing facial or body appearance or movement or speech which could indicate stress on, or danger or a threat or potential of danger or threat to or from persons, animals, actions, activities, or things. Bunn et al, in 2003 U.S. patent application Ser. No. 10/626,888, contemplate including detecting and estimating intoxication and impairment levels by alcohol or drugs of subjects observed and linking this detection to identification of the observed subject for facial and voice recognition as well as identification by ID card, photo ID and the like. Bunn et al, in 2004 U.S. patent application (Filed Dec. 6, 2004, number not yet assigned), further teach the fuzzy logic algorithms that permit detection and interpretation of the features describing the facial or body or speech or appearance or movement noted in U.S. Ser. No. 10/626,888.
The preferred embodiment of the present invention is focused on the integration of the video surveillance technology prior art whether referred to herein or otherwise for the purpose of detection of intoxication, drunken and impaired behavior including the identification of subjects and the possible prevention of underage drinking in a localized establishment or place by means of a system that we call SoberCam™, and the sharing of such information throughout a communications network of central database systems and participating networked groups, systems or agencies alerting security personnel, systems and agencies for appropriate response in a distributed system we call LastCall™ Network.
A significant problem exists with most of these conventional embodiments of video surveillance and security types of systems in that they acquire very large and unwieldy volumes of data. Surveillance systems in the prior art view, observe, record and process many details of the images from video cameras and systems but do not deal well with the control and limitation of the data contained in the video data stream whether in analogue or digital format. With the modern state of the art, digital video recording (DVR) systems and high-speed, high-resolution cameras can generate 1.5 terabytes of data in 15 minutes.
This invention deals with the ways and means for overcoming this digital glut.
A preferred embodiment of the invention herein combines software, neural logic, fuzzy logic, neural networks and artificial intelligence to monitor, analyze and select data bits that occur when a pre-determined algorithm or electronic signature is activated. The algorithm acts as a switch that signals the system to discard irrelevant data while saving selected items. In practice, 99.9983% of the usual video surveillance data will be discarded and only 0.0017% retained for security personnel attention. One method of achieving this reduction in “real time” is to have the system buffering data for a short time while analyzing it and upon being activated, the system would record the buffer data incoming data until again activated to stop recording. The buffer would need to be large and the system processing speed fast enough so that non of the desired data are lost.
In another preferred embodiment of the invention, the algorithm selects and retains images of an individual subject only when there are legally valid grounds for doing so on the basis of just cause. All other images can be discarded.
In another preferred embodiment of the invention, the algorithm and image assessment system can use the monitoring and intelligence capabilities of the software to adjust to ever-increasing speeds and resolution improvements in the video camera systems thereby maintaining the minimum data storage levels as the camera technology advances.
In effect, our intelligent camera acts as a video surveillance data analyzer for 99.9% of the time and as a conventional surveillance camera system storing images for 0.1% of the time. In a preferred embodiment of the invention, the algorithms could be fine-tuned to a specific surveillance application and scene being observed, such that the analyzer could remove 99.9983% of the images and thus retain only 0.0017% for a reduction of data storage of nearly a factor of 60,0000.
Bunn et al, patent application U.S. Ser. No. 10/626,888 filed Jul. 25, 2003, teaches algorithms such as staggering, drug-taking and dealing, violence, threatening movements, throwing objects and related anti-social activities that can trigger intelligent camera systems to automatically recognize these occurrences and notify the appropriate security personnel. These video data recordings are of sufficient resolution and frame speed that can be matched by the existing DVR data acquisition and storage and image database management systems of the day.
A preferred embodiment of the invention goes further and uses high-resolution, high-speed video camera systems with different algorithms to measure fine resolution characteristics of observed subjects such as, but not limited to, measuring pupil dilation of the eyes, sweating, blushing, and other bio-behavioral aspects at the onset, and notes changes in these aspects thereafter and calibrates them to levels of impairment, intoxication and behavioral changes. In this application, the subjects being observed in effect provide their own basic database standards against which to measure change. This we call the SoberCam™ application.
The SoberCam™ camera system envisioned in a preferred embodiment of the invention will also use algorithms to scan at high-resolution and high-speed video surveillance large venues such as entertainment arenas, sporting fields of play and the like for which the system and algorithms can establish virtual barriers to detect incursions into selected restricted areas. Camera resolution is such as to detect a person moving from higher levels in the venue to close proximity to the entertainment or playing surfaces or areas for which algorithms will detect and notice and command the system to monitor and record this movement for later analyses or identification. Potentially rowdy spectators can also be similarly identified and noted and images recorded. In a preferred embodiment of the invention, subsequent escalation of activities by the observed subjects can be further analyzed by the algorithms to detect hotspots of potentially threatening or violent behavior and the system can alert security personnel for appropriate action to be taken.
In a preferred embodiment of the invention, the SoberCam™ observations and database-stored imagery can be recalled for the venue in question as an information source of video evidence to support legal actions as needed.
In a preferred embodiment of the invention, the information stored in the system's databases of images, video data and algorithm results can be shared with other groups, entertainment and sporting venues, related clubs, bars and the like as a pre-emptive warning of local, nearby neighborhood, inter-city, nation-wide or international potential threat, disruption or problem whether at the same time, other times or other sites. This sharing of such information we call LastCall™ Network.
In a preferred embodiment of the invention, the occurrence for example of intoxication by an observed subject at say nightclub A can trigger storage of video data and by using facial recognition of subjects entering the nightclub A at later times and comparing these to the recorded database can permit the system to recognize previous trouble makers and alert security to take appropriate action. This would be LastCall™ Network operating on a restricted local basis.
In a preferred embodiment of the invention to illustrate an example, in which the information from the above occurrence at nightclub A is shared with Nightclub B and Nightclub C in the neighborhood this would be LastCall™ Network operating on the citywide basis. A further example is if the above occurrence happened at a sporting venue in City A and is shared with a sporting venue in City B this would be LastCall™ Network that could be inter-city or nation-wide or international depending on where they are located.
In a preferred embodiment of the invention, SoberCam™ could be used to prevent underage drinking in which the system would use technology of ID cards such as but not limited to student, health, driver's license, social security, credit and such like cards, scanning of both magnetic strip or smart card information and imbedded picture ID. The system would use facial recognition to assess if the subject being observed on site is the same as the ID picture and information and whether the subject appears to be under drinking age. Features such as lack of wrinkles and non-existence of beard, and balding and sagging neck skin or frequency and timber of voice are not perfect by can be indicators of relative age. If the analysis algorithms suspect underage, the system can inform security personnel to investigate and if appropriate to take action to deny entrance to a drinking establishment or area. This information could be shared via the LastCall™ Network to participating groups, drinking establishments and the like, thereby further assisting the prevention of underage drinking.
In another preferred embodiment of the invention, utilizing wide-angle and zooming narrow angle video camera technology with high-resolution and high-speed capabilities the SoberCam™ system utilizing illumination, such as but not limited to, infrared directed from the camera location towards subjects could view the reflected infrared light from the eyes of the subjects such as at a sporting event arena to detect the number of subjects looking specifically in the direction of the camera. This would be similar to the “animal in the headlights” example of a cat looking towards your oncoming vehicle at night, where light from the headlights will reflect off the animal's retina directly back to the vehicle such that the driver may see only the two bright spots of light reflecting from the cat's eyes. Algorithms in the SoberCam™ system could simply count the number of bright spots and divide by 2 to get a good approximation of the number of subjects looking directly at the camera and light source.
This retina reflection of light application is not strictly tracking eye movement of the subjects but rather detecting at any instant, the number of subjects looking in a specific direction. If the camera and light source are located near where an advertisement such as on an illuminated sign or billboard or electronic display such as a “JumboTron” the system in this embodiment of the invention could measure the effectiveness of what is being displayed such as an advertisement. In effect the system algorithms are measuring crowd response to displayed images or messages, which could collect statistics that could be interpreted to assess effectiveness of the display, or the message or advertisement. If the system is observing subjects entering a venue and the system detects the subjects watching a billboard and thus not watching where they are walking, say down stairways or aisles, this could provide evidence of the subject's responsibility should an accident such as tripping occur.
In another preferred embodiment of the invention, the collection of video data over time provides the SoberCam™ system with information from which to derive statistically important conclusions. Effectiveness of advertising noted above is one such conclusion. Changes in the actions of subjects over time can permit statistical conclusion of the onset of intoxication or impairment of the observed subjects, or the escalation of violence, or the occurrence of a health condition such as seizure or heart attack.
Sharing of information in databases via the LastCall™ Network application need not be limited to those data selected by the SoberCam™ intelligent video surveillance camera system but can incorporate related databases such as personal identification information, legal or criminal activities, actions or convictions, health and drug or alcohol information, suspected terrorist activities and the like. Sharing of all such information permits the algorithms to detect problems or potential problems quickly, automatically allowing the system to notify the authorities and security personnel to take appropriate action.
The descriptions that follow are provided so as to enable any person skilled in the art to make and use the invention, and sets forth some modes presently contemplated by the inventors of carrying out their invention. Various modifications, however, will remain readily apparent to those skilled in the art, since the generic principles of the present invention have been defined herein.
Data for processing by the algorithms revealed in this invention are obtained from observations by an intelligent Camera System means incorporating the use of, but not limited to, sensors, input-output devices such as a surveillance camera means with incorporated local controller, 101, with associated illumination means, 102, and listening audio means, 103, and video means, 104, functions and full pan, tilt and zoom computer-controlled motion for monitoring of a given scene, situation, place, thing, persons or environment. A unique aspect of the Camera System means is the incorporation of the new technologies of high-resolution, low noise-level, low light-level, high-speed digital camera systems which permits the algorithms to perform in the real-world environment of nightclubs, bars and large venue arenas as well as they perform in the laboratory monochromatic calibration environments. It is these modern enabling technologies that give rise to the development of the algorithmic means of this patent.
In a preferred embodiment, the algorithms of this invention could be analyzing movements of a person or persons and their activity, 110, with a plurality of sensors and input-output means such as but not limited to audio and video, the data from which can be communicated by wired, 105, or wireless, 106, to a local facility, 107, so that the intelligent analysis means, located at the local, 107, or central, 116 facilitiy employing the algorithms of this patent can interpret those person or persons and/or activities, their conditions or drug or alcohol induced impairments and possibility for potential threat from the subject's appearance, movements and actions. Observations of subjects and identification credentials such as magnetic or intelligent ID cards, driver's license or photo ID, heath ID can also permit identification of the subject by using but not limited to algorithmic comparisons to earlier observation databases of audio, visual and speech and text information to which the facilities are connected via the Internet WEB, 111, or by hardwired land or telephonic, 112, or wireless links, 117 to other News MultiMedia, 113, Government, 114 and Associated, 115 databases. Analyses results can be sent out through the WEB 111, or land links, 112, or by wireless, 117, to computers and hand held devices, 109 or to the cellular network and cell phone units, 108.
A unique aspect of the fuzzy logic algorithmic Exceptions Data Engine means of this invention is its ability to learn from the data collected from these observations, 201, and from data in and collected for the comparison databases. The process of analyzing these data to determine an exception, 202, creates the definable occurrences of exceptions that can be used to eliminate unwanted data, 203, 204, and 205. Depending upon selectable criteria that define the exceptions, 203, the elimination of data at any give time could range from 100%, no recording, through to 0% with recording of all observations. A typical embodiment of this invention could result in elimination of 99.9983% of the observations, reducing database information storage, 206, by a factor of nearly 60,000 while efficiently citing and reporting exceptions to decision makers, 207, permitting appropriate action to be taken, 208.
For example, if the subject under observation is a young person the analyses, 202, with face recognition could compare photo ID and general facial appearance, 203, to determine that the person may be underage. If available, related databases could be queried, 204, to see if birth date information such as given on a driver's license could confirm the subject's age. In any event the Exceptions Data Engine would have queried, 205, and detected the occurrence of the exception and stored that occurrence in the exceptions databas, 206, that the subject may be underage and informed the decision makers, 207, to take appropriate action, 208, such as to deny that person access to a drinking establishment, area or venue.
In a preferred embodiment of this invention, the observations can include but are not limited to observing from a few to large crowds of subjects who have been illuminated by a lighting means, 102, located near the intelligent camera means, 101, from which data vision analyses for an exception request to the Eyes analysis, 203, by the Exceptions Data Engine could detect the number of subjects within view, who are looking in the camera direction. This analysis could employ detection of the reflection from the retinas of the subjects' eyes that if looking in the direction of the camera and the light source located there, would appear as bright spots. In darkened locations such as sports or entertainment venues, using infrared illumination, which is not visible to the subjects, would not be invasive and would permit the subjects' pupils to remain more open and hence increase the reflected light resulting in brighter and more easily detected reflections. With sufficient camera speed and resolution technology, the individual eyes of each subject would be resolved to create two such bright spots and the analyses could determine how many subjects were looking towards the camera and light source. Such information could be recorded as an exception, 206 and passed to inform decision makers, 207, for use to measure response of subjects to whatever was at the location of the camera such as advertising, video displays, security information, entertainment and such like.
These database means and facilities, whether incorporated into the camera means or located elsewhere, can include local and remote databases including but not limited to: the Multi-Media, 113, such as print including newspapers, radio and TV; the Government, 114, such as criminal activity/conviction, or incarceration, or driver's license identification, or terrorist activity; and the Associated data systems, 115, such as medical/mental health, or education, and the like. Health information, in particular could be critical in understanding the actions, emotions and motions of persons to recognize the differences between drunkenness, heart attack, diabetic coma, epileptic seizure and the like. These databases as part of the Camera System means can be linked via the WEB, hardwired, telephony or wireless means for access, analyses by the algorithmic means revealed in this patent.
In a preferred embodiment of the invention, the SoberCam™ Local fuzzy logic algorithm system means, learning by the Camera System means can result from a plurality of fuzzy logic algorithmic analyses incorporated into the SoberCam™ Intelligence Engine illustrated in
In a preferred embodiment of the invention, for the above example of a subject who appears underage, the related databases, 304, may contain a driver's license information and photo ID that could confirm the Exceptions Data Engine facial recognition analysis of the subject and SoberCam™ Intelligence Engine could identify that the subject indeed was younger than legal drinking age. In this example, related database searches, 304, in a health database could indicate the subject has a serious heart condition and in a legal database could indicate there is an outstanding arrest warrant for the subject. The analysis system means, 303, so learns and updates the occurrences databases, 305, that this person currently under surveillance observation is underage, has an outstanding arrest warrant and the fuzzy logic algorithmic system means reports to the security systems and personnel, 306. In this example the decision makers, 307, would be advised to deny the person access to drinking areas, venues or establishments and to immediately inform the police for appropriate action. Prevention of underage drinking is a unique aspect of this invention. Assisting police is another unique aspect of this invention.
The SoberCam™ Intelligence Engine includes a plurality of computer analysis techniques and technologies, software, firmware and hardware methods and designs including but not limited to recording and storage and retrieval of data, video pattern recognition, facial recognition, body action recognition, stress analysis of facial appearance and movement, stress analysis of body appearance and movement, emotional condition stress analysis from facial and/or speech and/or body action, surrounding environment condition assessment, voice stress analysis, voice recognition, voice speech recognition to text, lip reading recognition of speech and conversion to text, deep extraction of information and meaning from text or multi-media information, identification ID and photo ID input-output data analyses and the like.
Many of these techniques and technologies have been noted in the background to this invention, but what is unique in this invention is that we reveal an Exceptions Data Engine means for massive reduction of stored data observations and the SoberCam™ Intelligence Engine automated learning and decision analysis for the detection and understanding of a threat or potential threat or condition of or by a person or persons or animals or objects, by their actions, or their appearance, or their impairment intoxication, or their personal information and history, or any combination of these.
In a preferred embodiment of this invention, we reveal a method and means to significantly utilize the above Exceptions Data Engine and SoberCam™ Intelligence Engine processing and analyses of these video, audio, input-output and sensor data through a centralized Vision storage, retrieval and analysis facility. Unique to this embodiment of the invention, this facility provides the capability of networked sharing of Image, Vision and related data Information directed to fighting underage drinking, preventing drunk driving, and preventing the escalation of threatening actions or situations. We call this the LastCall™ Network of Fuzzy Logic Algorithmic System Means.
The LastCall™ Network allows permitted members of the network such as Venue C, 401, Club B, 409, and Club C, 404, each reducing data via their individual Exceptions Data Engine, 402, to access the Shared LastCall™ Communications Network, 403, to exchange data and information with a Central Processing and Storage Facility, 405. The central processing and storage facility operating the SoberCam™ Intelligence Engine, 408, can store, analyze, interpret and categorize these data and analyses results in the associated databases, 406, as described above, and report to the decision makers to take appropriate action, 407.
Unique to the LastCall™ Network is its ability to permit access to the network members approved at various levels, to access results and exception occurrences of local area, citywide, national, or international information depending on their level of access permission. This increases the effectiveness and utility of the exceptions data and extends the reach of the LastCall™ Network Fuzzy Logic Algorithmic System Means from the local to the citywide to the National and to the International scene.
In a preferred embodiment of the invention, the participating members of the Network could be automatically updated with recent exception occurrences from the local area, citywide, national or international databases. The members could have this information sent to their local systems, and could have it recorded, displayed, noted to wireless cellular phones or personal data assistants (PDA's) and the like. In the above example of the young person who is under legal drinking age and with an outstanding arrest warrant as detected by the SoberCam™ Intelligent Engine, the person could possibly slip away from the establishment where they were detected. The person could then attempt to enter a nearby establishment also on the Network that could have already informed them to be on the lookout for the person thereby assisting the security personnel in advance. If the person appears, the Network would identify the person and collaborate to the security personnel of the problem with this person so immediate action could be taken. If the person is a security threat, wide dissemination of the information through the Network could help to prevent a national or international security threat.
We have indicated just some but not all of the examples of preferred embodiments, applications and uses of the Algorithm Analysis System means revealed in this invention that would come to mind of a person or persons skilled in the art of security systems.
