Currently retailers deploy employees to manually check missing items on shelves, which is inefficient, time consuming, and fraught with oversight errors. Even when retailers rely on transactional data from their point-of-sale (POS) systems for in-store inventory, the data does not update in real time and can be inconsistent with the actual shelf inventory. These problems cause restocking operations to be slow and often result in popular items missing from shelves for longer than is necessary.
The retailers experience lost opportunity sales when the shelves are not adequately stocked. Customers may frequent different stores associated with different retailers resulting in a loss of customer loyalty. Customers may become frustrated and post adverse comments on social media, which may result in potential or existing customers, who did not experience any out-of-stock items, intentionally avoiding the retailer's store or the retailer altogether.
Thus, timely and accurately restocking items on shelves of a store is of significant import to the retailer. Even a one-time lapse in timely restocking by a retailer can have long lasting negative consequences for that retailer.
In various embodiments, methods and a system for computer vision shelf auditing are presented. A machine-learning model (MLM) is trained on images of store shelves to identify items and shelf dimensions or pixel coordinates of empty spaces on the shelves that are associated with the items. Real-time images of the shelves are provided to the MLM and item identifiers for the items, empty space identifiers for the corresponding items, and empty space dimensions or pixel coordinates for the empty space identifiers are provided as output from the MLM. For each item identifier and empty space dimensions or pixel coordinates, a total quantity that needs to be restocked on a corresponding shelf with the corresponding item is calculated. A real-time report is sent to store personnel or published to a website monitored by the store personnel for the personnel to restock the shelves in near real time at the store. This ensures that store shelves are continuously and regularly restocked.
According to an aspect, a method of computer vision shelf auditing is presented. A real-time image of a shelf in a store is provided to a machine-learning model (MLM) as input. An empty space identifier for an empty space on the shelf and an item identifier for an item associated with the empty space are received as output from the MLM. A quantity of the item is determined, the quantity corresponds to the empty space and is based on known dimensions for a single item associated with the item identifier. A shelf identifier for the shelf, the empty space identifier for the empty space, the item identifier for the item, and the quantity are reported for real-time restocking of the item on the shelf of the store.
As stated above, timing restocking of items on shelves of stores can have a significant impact on the business of a store. A store can experience customer turnover when shelves are not properly restocked with the frustrated customers going to other stores to find their items and a chance of the customers never frequenting the store again.
As will be demonstrated herein and below, a machine-learning model is trained on images and/or a video stream taken of shelves of the store for purposes of recognizing the items and identifying missing items by empty spaces identified on the shelves from the images/video. An alert or a report is sent to staff of the store identifying the missing items and the corresponding shelves. Additionally or alternatively, the alert or report is published on a website monitored by store personnel in real time. This permits timely and real-time item restocking at the store, which the industry has been unable to be achieve by just relying on the accuracy of transaction records and the diligence of staff who periodically perform manual inspection of store shelves.
Furthermore, the various components (that are identified in the
As used herein, the terms “staff,” “staff member,” “employee,” “store personnel,” “personnel,” and/or “store employees” can be used interchangeably and synonymously herein and below. These terms refer to the individual to which item restocking alerts or reports are sent to or monitored by.
System 100 includes a cloud 110 or a server 110 (hereinafter just “cloud 110”), a retail store 120, and retail servers 130. Cloud 110 includes at least one processor 111 and a non-transitory computer-readable storage medium 112, which includes executable instructions for a retail manager 113, a restocking manager 114, a trainer 115, one or more machine-learning models (MLMs) 116, and an alert manager 117. Medium 112 also includes a restocking data store 118. The instructions when executed by processor 111 cause processor 111 to perform operations relevant to 113-117 as discussed herein and below.
Each retail store 120 includes cameras 121, shelves with items stocked thereon, and terminals 123. It is noted that each retail store 120 can also include a store server with a processor, a medium, and executable instructions, which are executed by the processor for purposes of interacting and reporting transaction data from transaction performed on terminals 123.
Each retail server 130 includes at least one processor 131 and a non-transitory computer-readable storage medium 132, which includes executable instructions for a store manager 133 and an Application Programming Interface (API) 135. Medium 132 can also include planograms for each of stores of the corresponding retailer. The executable instructions when executed by processor 131 cause processor 131 to perform operations relevant to store manager 133 and API 135 as discussed herein and below.
Initially, retail manager 113 interacts with API 135 to receive planograms 134 for stores 120 of the retailer. Store manager 133 configures cameras 121 of the stores for sending or making available, in a file or data store accessible to cloud 110, images of video captured in real time by cameras 121. The images or video streams are streamed to memory buffers, files, or data stores accessible to cloud 110.
Trainer 115 uses a store's planogram 134 and metadata associated with the images or video stream, to label features in the images. The metadata can include a camera identifier, a camera location, a retailer identifier, a store identifier, and shelf identifiers associated with shelves 122 of the store 120 that the field-of-view of the corresponding camera 121 captures in the corresponding image or video stream. The store's planogram provides Item identifiers for each item, shelf identifiers for each shelf 122, dimensions on the corresponding shelf that each type of an item having a same item identifier appears on the corresponding shelf 122, and optionally a total item count for each type of item on a given shelf 122. This enables trainer 115 to produce training images or training video for each shelf 122 of each store 120. The training images or training video are labeled within input training features by trainer 115, the input training features labeled include a retailer identifier for the retailer, a store identifier for the store 120, shelf identifiers for the shelves in the corresponding training image or training video, item type identifiers for item types on each shelf 122, item identifiers for the items associated with each item type, and the shelf dimensions for a given shelf 122 associated with a given item type identifier that occupies the corresponding shelf dimensions. The labeled shelf dimensions, for a given shelf of a given item type associated with a same item identifier, are obtained from the planogram by trainer 115. The retailer identifier, store identifier, camera identifier, and shelf identifiers are obtained from the metadata associated with a given training image or a given frame of the training video. The labeled training images or training video are used as features provided as input with the training images or training video to MLM 116 during a training session.
The trainer 115 also provides as input, during a training session with MLM 116, a control file or control data structure associated with the shelves and the items, The control file or control data structure includes image coordinates for a given item type on a given shelf. That is, for a given type of item all having a same item identifier and a given image captured by a given camera 121, the pixel coordinates of the items having the item type on the shelf within the image are noted in a control file/data structure. This gives the pixel coordinate for the top of the items on the shelf, the leftmost pixel coordinate for the items on the shelf, the height of the items expressed as a bottom coordinate for the items on the shelf, and the width of the items expressed as a rightmost coordinate for the items on the shelf. The four coordinates allow the MLM 116 to disambiguate between two different item spaces based on the pixel coordinates in the control file/data structure and based on the identified pixel coordinates associated with a given identified empty space in an image. This file/data structure can be maintained per store 120 and its planogram 134 and/or per shelf of each store 120 and its planogram 134. The four-pixel coordinates allows the MLM 116 to determine when detected empty space pixel coordinates are within or enclosed by the empty space and if so, associate the corresponding item identifier provided in the control file or data structure.
An example, control file or control data structure 180 is shown in
When the training image or training video are missing items in a given location, trainer 115 labels the dimensions and/or pixel coordinates of the empty area on the corresponding shelf and the item identifier that should have been in the empty area. These additional labels associated with the dimensions and/or pixel coordinates of the empty area and the corresponding item identifiers are labeled as the expected output from a MLM 116 when provided a training image or training video of a given shelf with the labeled input features and when provided the control file/data structure 180.
Trainer 115 then trains MLM 116 on a set of acquired training images or training video that includes each of the items on the shelves 122 in different orientations on their corresponding shelves 122 with the labeled input features, the labeled expected output for each training image, and the control file/data structure 180 when an empty space was detected in the training image or training video. The MLM 116 configures weights and an algorithm that when provided a given image of a given shelf 122 with an empty space and the control file/data structure 180 provides as output an item identifier for the missing item associated with the empty space and the dimensions and/or pixel coordinates of the empty space on the corresponding shelf 122 After training, MLM 116 is configured to receive the labeled input features with a corresponding image or video of a given shelf 122, to receive the control file/data structure 180, and to provide as output shelf identifiers, the item identifiers associated with any empty spaces present in the corresponding image or video along with the dimensions and/or pixel coordinates of each empty space for each item identifier.
When trainer 115 determines through a testing set of images or video that MLM 116 has reached an acceptable level of accuracy in predicting, from detected empty spaces in images of video of shelves 122, the item identifiers and corresponding shelf dimensions and/or pixel coordinates for each item identifier that is empty, trainer 115 releases MLM 116 for real-time restocking predictions. Once trainer 115 has released MLM 116 for real-time restocking predictions, retail manager 113 obtains real-time images and video from the cameras 121 of stores 120 for a given retailer. Retail manager 113 interacts with each retail server's API 135 to obtain the corresponding store planograms 134. Retail manager 113 then labels each image or video received from cameras 121 with the corresponding input features, which include the retailer identifier, store identifier, shelf identifiers, item type identifier per shelf identifier, and item identifier for the corresponding item type.
Retail manager 113 provides the control file/data structure 180 and images or video with the labeled input features as input to MLM 117. MLM 117 returns as output for a shelf identifier for a shelf represented in the images or video, an empty space identifier for each empty space detected on each shelf, the item identifier associated with each empty space, and the shelf dimensions and/or pixel coordinates of each empty space.
In an embodiment, retail manager 113 modifies the original image or video with labels that visually distinguish each empty space with an item name for the item identifier. Each empty space is also linked to or associated with known shelf dimensions and/or pixel coordinates that each item identifier is supposed to occupy on its corresponding shelf. In an embodiment, the known shelf dimensions are obtained from the corresponding planogram 134. The modified images and the shelf dimensions and/or pixel coordinates for each item associated with an empty space are stored by retail manager 113 in restocking data store 118.
Alert manager 117 identifies the entries made by retail manager 113 in restocking data store 118 and generates real-time report outs (e.g., reports and/or alerts) to store personnel of the corresponding store. Alert manager 117 interacts with a retail server's API 135 to obtain contact details for the store personnel and sends a real-time report or alert to personnel-operated devices that displays a restocking report and includes item identifiers and names for items in need of restocking along with a quantity of each item that should be restocked. Optionally, the restocking report includes the modified images for the shelves 122 with the item-named empty spaces and corresponding needed item identifiers.
Retail manager 113 receives the image 140 or video frame 140 in real time from a file, memory buffer, or data store that camera 121 streams to. Retail manager 113 obtains the metadata associated with the image 140 or video frame 140 and identifies the retailer. Retail manager 113 interacts with API 135 to obtain the planogram 134 for the corresponding store 120 and labels input features of the image 140 or video frame 140. Retail manager 113 uses the planogram 134 to obtain the control file/data structure 180 and provides the control file/data structure 180, the input features, and image 140 or video frame 140 as input to MLM 116. MLM 116 returns as output a shelf identifier for each shelf that includes an empty space, an empty space identifier for each empty space identified in the image 140 or video frame 140, dimensions and/or pixel coordinates of each empty space, and an item identifier associated with each empty space.
In an embodiment, retail manager 113 modifies the image 140 or video frame 140 to create a modified image 160 depicted in
Alert manager 117 receives notice that restocking data store 118 was updated with the modified image 160 and interacts with a corresponding retail server's API 135 to obtain contact information for store restocking personnel on duty at store 120. Alternatively or additionally, alert manager 117 posts a report, notification, and/or alert along with the modified image 160 to a website monitored by store personnel of store 120.
In an embodiment, Alert manager 117 counts the items of each unique item identifier that appear on the shelf in the modified image 160 and uses known dimensions and/or pixel coordinates associated with a missing item type to identify the missing quantity of the corresponding item using the size of the corresponding empty space for the missing item. In an embodiment, alert manager 117 uses a total count of items for a given item identifier on a given shelf that is noted in the corresponding planogram 134 to determine a missing count of the items needed for the corresponding empty space in the modified image without using the size of the corresponding empty space.
In an embodiment, 113-118 are provided over a cloud processing environment as a network-based service to store devices of store 120. This network-based service can also be provided to retailer devices such as retail server 130 through the cloud processing environment. The devices can include mobile devices, tablets, terminals, servers, wearable processing devices, etc.
The above-referenced embodiments and other embodiments are now discussed with reference to
In an embodiment, the item restocking shelf auditor executes cloud 110 or server 110. In an embodiment, the item restocking shelf auditor executes on a specific retail server 130.
In an embodiment, the item restocking shelf auditor is one, all, or some combination of 113, 114, 115, 116, and/or 117. In an embodiment, item restocking shelf auditor presents another, and in some ways, an enhanced processing perspective from that which was discussed above with 113-118 of system 100.
At 210 (shown in
In an embodiment, at 211 (shown in
In an embodiment, at 212 (shown in
At 220 (shown in
In an embodiment, at 221 (shown in
At 230 (shown in
In an embodiment of 212 and 230, at 231 (shown in
At 240 (shown in
In an embodiment, at 241 (shown in
In an embodiment, at 242 (shown in
In an embodiment of 231 and 240, at 243 (shown in
In an embodiment, the device that executes shelf restocking manager is cloud 110 or server 110. In an embodiment, the device that executes the shelf restocking manager is a specific retail server 130.
In an embodiment, the shelf restocking manager is all of, or some combination of, 113, 114, 115, 116, 117, and/or method 200. The shelf restocking manager presents another and, in some ways, an enhanced processing perspective from that which was described above for 113-118 of system 100 and/or method 200.
At 310 (shown in
In an embodiment, at 311 (shown in
In an embodiment of 311 and at 312 (shown in
In an embodiment of 312 and at 313 (shown in
At 320 (shown in
At 330 (shown in
In an embodiment of 313 and 330, at 331 (shown in
At 340 (shown in
In an embodiment, at 341 (shown in
In an embodiment of 341 and at 342 (shown in
At 350 (shown in
In an embodiment, at 351 (shown in
In an embodiment, at 352 (shown in
In an embodiment of 313 and 330, at 331 (shown in
It should be appreciated that where software is described in a particular form (such as a component or module) this is merely to aid understanding and is not intended to limit how software that implements those functions can be architected or structured. For example, modules are illustrated as separate modules, but can be implemented as homogenous code, as individual components, some, but not all of these modules can be combined, or the functions can be implemented in software structured in any other convenient manner.
Furthermore, although the software modules are illustrated as executing on one piece of hardware, the software can be distributed over multiple processors or in any other convenient manner. The above description is illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of embodiments should therefore be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
In the foregoing description of the embodiments, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting that the claimed embodiments have more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Description of the Embodiments, with each claim standing on its own as a separate exemplary embodiment.