Patent Application
20170251183
Many imaging devices are commonly deployed in modern facilities, such as retail outlets, schools, factories, and larger, fenced facilities. However, there are so many imaging devices, monitoring streams of images and video from these devices is challenging. Prior efforts have been made to identify, and bring attention to, images and video streams where there are changes, people or objects loitering, and to identify certain individuals. However, such solutions are focused on monitoring what is changing or what is present within one or more images or streams.
Various embodiments herein each include at least one of systems, methods, and software for identification and imaging of Point of Sale (POS) terminal-proximate event occurrences. One such embodiment is in the form of a method that includes detecting an event occurrence and selecting at least one digital image captured with regard to a time-location of the detected event. This method further includes providing each of the at least one selected digital images to at least one location.
Another method embodiment includes detecting, by a product scanner, an event occurrence. This method then selects at least one digital image captured with regard to a location of the detected event where each of the at least one digital images is captured by digital imaging devices present within or coupled to the product scanner. This method may then provide each of the at least one selected digital images to at least one location.
Another embodiment in the form of a produce scanner includes at least one imaging device integrated within the product scanner or electrically coupled thereto, at least one network interface device, at least one processor, and at least one memory device. The product scanner of such embodiments includes an event detection module stored as instructions in the at least one memory device that are executable by the at least one processor to perform data processing activities. The data processing activities may include monitoring data conditions within the product scanner in view of event detection rules where each event detection rule include at least one data condition, which when satisfied, indicates an event occurrence. The data processing activities also include selecting at least one digital image captured by one of the at least one imaging device. The selected image may be selected with regard to a location of and upon an occurrence of a detected event. The data processing activities further include transmitting, via the at least one network interface device, each of the at least one selected digital images to at least one network location.
Various embodiments herein each include at least one of systems, methods, and software for identification and imaging of terminal-proximate event occurrences. Such embodiments utilize images and video captured by one or both of imaging devices of and coupled to product scanners deployed at checkout terminals for purposes of monitoring certain events that may take place, such as scanning of a product, failure of a product scanner to scan a product when presented within a scan field, a mismatch between a scanned barcode and one or more properties of the item presented for scanning, an electronic article security (EAS) sensor violation, and the like. These images may be utilized not only for security purposes for fraud and other crime detection and monitoring, but also to assist a retail outlet operator in identifying data missing in a product database such as when a product barcode is not found in a product database. These and other embodiments are described herein with reference to the figures.
In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments in which the inventive subject matter may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice them, and it is to be understood that other embodiments may be utilized and that structural, logical, and electrical changes may be made without departing from the scope of the inventive subject matter. Such embodiments of the inventive subject matter may be referred to, individually and/or collectively, herein by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept if more than one is in fact disclosed.
The following description is, therefore, not to be taken in a limited sense, and the scope of the inventive subject matter is defined by the appended claims.
The functions or algorithms described herein are implemented in hardware, software or a combination of software and hardware in one embodiment. The software comprises computer executable instructions stored on computer readable media such as memory or other type of storage devices. Further, described functions may correspond to modules, which may be software, hardware, firmware, or any combination thereof. Multiple functions are performed in one or more modules as desired, and the embodiments described are merely examples. The software is executed on a digital signal processor, ASIC, microprocessor, or other type of processor operating on a system, such as a personal computer, server, a router, or other device capable of processing data including network interconnection devices.
Some embodiments implement the functions in two or more specific interconnected hardware modules or devices with related control and data signals communicated between and through the modules, or as portions of an application-specific integrated circuit. Thus, the exemplary process flow is applicable to software, firmware, and hardware implementations.
Product scanners, also referred to herein as scanners, utilized in modern retail outlets now include one to many digital imaging devices (i.e., cameras) integrated therein and even other digital imaging devices that may be coupled thereto. An example of such a product scanner is the 7879 product scanner available from NCR Corporation of Duluth, Ga. Various embodiments herein utilize digital imaging devices of product scanners and other digital imaging devices that may be coupled thereto, such as by a Universal Serial Bus (USB) connection, for purposes of security monitoring. However, as mentioned above, environments where product scanners are deployed often include many imaging devices that each provide images and video. The result can be an overload of images and videos to monitor for fraud, theft, and other crimes. Various embodiments herein identify occurrences of events that have a possibility of involving a crime or other events of interest.
Some embodiments enable the use of a product scanner as a security monitoring device to enable remote monitoring of the checkout process at point-of-sale terminals, whether they be self-service checkout (SSCO) or cashier-assisted checkout terminals. Some embodiments monitor the checkout process at a terminal as well as activity around the terminal.
But utilizing images and video captured by imaging devices of product scanners is not straight forward and presents several challenges. For example, images are often captured at a relatively high-frame rate by the various imaging devices, such as at a rate of 40 frames per second or more. While such a high frame rate may be well suited for product barcode scanning, such a rapid frame rate can present challenges in some embodiments when the images or video are to be provided via a network to a remote monitoring station, network storage device, network video recorder (NVR), and the like. Some such systems may only be capable of receiving four to eight frames per second in order to manage network bandwidth and computer storage requirements.
In addition to these challenge, certain events of interest, for example seeing each product's barcode in an image, can happen too quickly for the image and video streams to capture as frames of interest. A typical scanned product may only appear in the scanner field of view for four frames on each camera, and only two frames on a single camera may contain a readable, legible barcode. This challenge can be further compounded by the relatively low frame rate of networked systems discussed above. For example, if the remote video system can receive only four randomly selected frames per second, the chances of any image of a scanned product may be 40 percent, and the chances of capturing a readable barcode image may be five percent. This is not useful for security auditing purposes.
Additionally, because remote security video systems are not tightly integrated with POS systems, it can be challenging to correlate images and video captured by imaging devices of a product scanner with POS activities. Even with tight integration, time-of-day clock inaccuracies and other system challenges make it difficult to marry the POS data with the video streaming data.
Thus, various embodiments herein not only utilize imaging devices of and coupled to product scanners for security monitoring but some such embodiments also buffer and then time-shift scanner frames-of-interest to ensure the video streams include them. Some embodiments further mark these scanner frames-of-interest with event information for later analysis. Some further embodiments may also adjust the video stream frame rate and other camera parameters in response to scanner events-of-interest.
The SSCO monitoring terminal 210, upon receipt of an image or video from one of the SSCO terminals 202, 204, 206, 208 may present the image or video to enable personnel to view activities and event occurrences as they happen. An employee at the SSCO monitoring terminal 210 may utilize the images to ensure a crime or product substitution fraud is not occurring, perform further actions to ensure a product presented is properly scanned and priced, and the like.
Furthermore, the various components included in
Moreover, the methods, POS terminal 304, and scanner 308 presented herein and below may include all or some combination of the components shown in the context of the checkout station 300. Further, although a checkout station 300 is illustrated as including a scanner 308, the scanner 308 may be a standalone element or an element of other systems, devices, and terminals in other embodiments. Examples of other terminal-types that may include a scanner 308 are self-service terminals (SSTs) beyond just SSCO terminals, POS terminals, clerk operated and self-service library checkout stations, time-keeping terminals, and the like. Additionally, references to POS terminals herein and in the claims are general references to both POS and SSCO terminals, unless expressly stated otherwise.
The methods of some embodiments are programmed as executable instructions stored in memory and/or non-transitory computer-readable storage media and executed on one or more processors associated with the components and devices herein.
The checkout station 300 includes one or more POS displays 302 that present information of a POS system 304 coupled to the one or more POS displays 302. Information presented by the one or more POS displays 302 includes information relevant in a retail context and with regard to operation of the checkout station 300. The checkout station 300 also includes the scanner 308.
The scanner 308 may be referred to as a barcode scanner or product scanner as those are the tasks most commonly associated with such devices. During operation of the checkout station 300, items are placed within a scan field of the scanner 308. One or more scanning devices 318 of the scanner 308, such as one or more cameras and laser scanners then scan a barcode and information read therefrom is communicated to the POS system 304. The POS system 304 then uses that data to identify the item placed within the scan field of the scanner 308 and performs an additional function. The additional function may include a price lookup and addition of the item to a list of items to be purchased, which may be presented on the one or more POS displays 302.
The scanner 308 may include one or more scan fields, such as two scan fields of bi-optic scanners that are commonly seen in grocery and discount retail outlets. Each scan field may include one or more imaging devices 318, such as cameras. In some embodiments, a total of four or more cameras may be included in one or both of the scan fields, combined. Additionally, one or more other cameras may be directly or indirectly coupled to the scanner 308. In addition to the scanning devices 318, the scanner 308 may include various other components. The various other components may include an integrated scale 310 such as may be used in a grocery outlet to weigh items such as produce and one or both of a speaker 312 and display lighting 316 to output audio a visual signal such as signals of (un)successful scans. The scanner 308 may also include scan field lighting 320 that may be turned on and off and adjusted based on a detected presence of an item to be scanned or upon identification of an event occurrence where the scan field lighting 320 may be turned off to enable imaging devices 308 to capture a full field of view rather than just a product within the scan field. The scanner 308 also includes one or more of at least one of a hardware communication interface 314 or a software or firmware communication interface 332 that enables to scanner to communicate with the POS system 304 communication interface 306 and, in some embodiments, directly over a wired or wireless network.
During typical operation, the scanner 308 is operated according to instructions executed on a processor 322. The processor may be a single or multi-core ASIC, digital signal processor, microprocessor, or other type of processor. The instructions may be firmware 326 or software 330 stored in one or more memories 324. The one or more memories 324 may be or include volatile and non-volatile memories, write-protected memories, write-once memories, random access memory (RAM), read only memories (ROM), and other memory and data storage types and devices. The additional function may also include software for identification and imaging of event occurrences. The additional functions may also include software implementing one or more fraud-detection process that declares security events in view of one or more fraud-detection thresholds at the checkout station 300. Security events may freeze user interaction capabilities with the POS system 304 and scanner 308 that must be cleared before proceeding, sending a security message to another terminal or a computing or communication device of an employee having security or supervision responsibilities, and the like. At the same time, such security events may also identify an event for which one or more images may be captured or retrieved from previously captured images for transmission via the communication interface 332 to a network location for review, storage, recording, and the like.
Instructions of software 307 that control operation of the POS system 307 may be stored in a memory device of the POS system and may include processes for identification and imaging of event occurrence. However, in other embodiments, instructions of processes for identification and imaging of event occurrences is present within the software 330 stored in the memory 324 of the scanner 308.
Instructions that control operation of the scanner 308 may be stored in firmware 326 or as software 330 in memory 324. These instructions are typically executed according configuration settings stored in the memory 324. The instructions may execute in view of configuration settings 328, which may also configure operation of the scanner 308 and the various components therein. For example, the configuration settings 308 may configure speaker 312 volume, display lighting 316 outputs, scan field lighting 320 brightness, decoding algorithm of the scanning device(s) 318 and the instructions, one or more communication protocols used to communicate data from the scanner 308 to the POS system 304, such as via a wired or wireless communication interface 306 of the POS system 304, scale 330 operating parameters (e.g., unit of measure as pounds or kilograms), among other configuration settings the particular scanner 308 of an embodiment may include. In some embodiments, the configuration settings 328 may include a firmware version, a software version, and the like. Thus, when a configuration is set or updated, the setting or updating of the configuration settings 328 may include population and updates of any of the configuration settings 328 of a particular embodiment, including an update to firmware and software present on the scanner. The software 330 may also, as have mentioned above, include one or more processes for identification and imaging of event occurrences.
In some embodiments, the processes for identification and imaging of event occurrences are storing the memory 324 in the form of an event detection module. Instructions of an event detection module, in some embodiments include instructions that are executable by the at least one processor 322 to perform data processing activities. The data processing activities may include monitoring data conditions within the product scanner in view of event detection rules. In some embodiments, each event detection rule includes at least one data condition, which when satisfied, indicates an event occurrence. Examples of such a data conditions may include activation of an EAS sensor coupled to the scanner 308 or the POS system 307, a product scan, a failure to scan a presented product, a stable weight or sale weight signal from the scale 310, the scanner 308 waking upon from an inactive mode as triggered by a motion sensor, a cashier login to the POS terminal 304, an “item not on file” result on the scanner 308 or the POS system 304, and the like.
In some embodiments, upon an event detection rule being satisfied by a data condition, the data processing activities may further include selecting at least one digital image captured by one of the at least one imaging device 318 where the selected image is selected with regard to a location of the detected event. The location may be specified in an event detection rule, such as a scan field camera upon a product scan event occurrence. Subsequently, the data processing activities include transmitting, via the communication interface 332, each of the at least one selected digital images to at least one network location. The data processing activities may include additional data processing activities that implement the various other methods and processes described herein.
Events that may be detected based on event detection rules can be virtually any thing for which data may be received or inferred based on other data within one or both of the scanner 308 and the POS system 304, depending on where the particular embodiment is implemented. For example, the following is a list of events that may be detected by event detection rules. Note that a frame of interest is a video frame that captures an event of interest. Unless otherwise noted, the scanner 308 or POS system 304 can detect events based on data or though various image analysis techniques that are known in the art.
When an event occurrence is detected according an event detection rule, one or more images and video segments are gathered. Conventional video security systems simply send the latest frame available when it is time to send the next video frame. Frames are rigidly scheduled according to the target video frame rate. In some embodiments herein, the scanner 308 identifies not only an occurrence of an event of interest, but also associated frames of interest, and queues the frame(s) of interest for the next scheduled video frame(s). For example, when an NVR system or other video recording system records a limited number of frames per second, some embodiments herein identify a frame or frames that include a best view or resolution of a particular item in the frame(s), such as a barcode, to ensure the barcode is legible and that that frame(s), is actually received by the NVR or other video recording system.
Some embodiments also include manipulation of camera parameters and other parameters, such as lighting, that may impact how an image or video is captured. Some such parameters, for example, may include one or more of the following:
Some embodiments further include binding metadata to particular images or frames of interest. This metadata may include:
The POS system 304 logs contain some of this data, but because of time clock inaccuracies and system integration challenges, it can be difficult to match the POS log data with the video stream images. Thus, binding the metadata to the image resolves that difficulty.
The metadata may be bound to an image or frames of video in many different ways. For example:
The method 400 includes detecting 402 an event occurrence and selecting 404 at least one digital image captured with regard to a location of the detected event. The method 400 then provides 406 each of the at least one selected digital images to at least one location. Some embodiments further include adding metadata to at least one of the selected digital images. The metadata in such embodiments typically include data with regard to the detected event occurrence.
In some embodiments of the method 400, providing 406 each of the at least one selected digital images to at least one location includes placing each of the at least one digital images in a transmission queue for transmission to each of the at least one locations in a next available data transmission window. In some of such embodiments and some other embodiments, providing 406 each of the at least one selected digital images includes transmitting each of the at least one selected digital images via a data network to at least one of a POS terminal monitoring station, a network data storage location, a premises video monitoring system, and an NVR.
Each of the at least one digital images are typically captured by digital imaging devices present within or coupled to a product scanner. In some embodiments, at least one of the digital imaging devices captures video and at least one selected digital image is a portion of video captured by the video-capturing digital imaging device.
In some embodiments the method 400 includes receiving digital images from each of the digital imaging devices and buffering digital images in an image buffer for each of the digital imaging devices for a most recent period, such as 10, 15, or 30 seconds, 1 or a greater number of minutes, and the like. In some such embodiments, selecting 404 at least one digital image with regard to a location of the detected event includes selecting at least one digital image from the buffered digital images received from a particular digital imaging device that is correlated with the location of the detected event, the digital image selected from the buffered digital images captured at a time when the event was detected.
In some embodiments of the method 400, an event is detected 403 upon satisfaction of one or more event rules stored within a memory of the product scanner, each event rule including at least one data condition monitored by the product scanner. The at least one event rule in such embodiments monitors a data condition with regard to data received from another device coupled to the product scanner.
Each of such event rules typically includes data identifying one or more of the digital imaging devices from which the at least one digital image is to be selected upon satisfaction of the respective event rule. In one such embodiment, upon detection 402 of the event occurrence, the method 400 includes identifying at least one digital imaging device to capture an image based on the satisfied event rule. Next the method 400 may modify at least one parameter of the digital imaging device prior to capturing at least one image from a normal condition to an event image capturing condition and, upon capturing at least one image, returning the digital imaging device to the normal condition.
In some embodiments, the method 500 includes detecting 502, by a product scanner, an event occurrence. The method 500 may then select 504 at least one digital image captured with regard to a location of the detected event. Each of the at least one digital images are captured by digital imaging devices present within or coupled to the product scanner. The method 500 then provides 506 each of the at least one selected digital images to at least one location.
In some embodiments, the method 500 further includes receiving, by the product scanner, digital images from each of the digital imaging devices and buffering, in a memory device of the product scanner, digital images in an image buffer for each of the digital imaging devices for a most recent period. In some such embodiments, selecting 504 at least one digital image with regard to a location of the detected event includes selecting at least one digital image from the buffered digital images received from a particular digital imaging device that is correlated with the location of the detected event. The digital image selected 504 in such embodiments is selected 504 from the buffered digital images captured at a time when the event was detected. In a further embodiment, an event is detected 502 upon satisfaction of one or more event rules stored within a memory of the product scanner. Each event rule typically includes at least one monitored data condition within the product scanner or data received by the product scanner. Each event rule may include data identifying one or more of the digital imaging devices from which the at least one digital image is to be selected 504 upon satisfaction of the respective event rule.
It will be readily understood to those skilled in the art that various other changes in the details, material, and arrangements of the parts and method stages which have been described and illustrated in order to explain the nature of the inventive subject matter may be made without departing from the principles and scope of the inventive subject matter as expressed in the subjoined claims.
