Patent Application
20230206507
The present invention relates to a processing apparatus, a processing method, and a program.
Patent Document 1 discloses a technique for recognizing a food material in each group of time-series images capturing an inside of a refrigerator, determining, based on a result of the recognition, whether the food material is moved, determining the food material of which a not-moved period is exceeds a threshold value as a left-over item, and issuing a warning.
[Patent Document 1] Japanese Patent Application Publication No. 2016-57022
There is a store in which a sell-by date for a product is set for a purpose of preventing an accident and the like. In such a store, it is necessary for a store clerk to have a means for recognizing a product of which a sell-by date is passed, a product of which a sell-by date is close, and the like. For example, when a label on which various kinds of information such as a freshness date, an expiry date, and the like are printed is attached to each product, a means of checking a label on each product on display can be used. However, it may not be possible to attach a label to a product such as a side dish that is not individually packaged, such as a croquette, a yakitori, and the like, and a fresh food. Also, work to check a label on each product on display is very time-consuming. Patent Document 1 does not disclose such problems in a store and a solution thereof.
An object of the present invention is to provide a technique for recognizing a status of a sell-by date of each product on display.
According to the present invention, a processing apparatus including:
an acquisition unit that acquires a plurality of time-series images including a display area;
a product detection unit that detects a product from each of a plurality of the time-series images;
an identity determination unit that determines identity of products detected from the images that are different from each other;
a display time management unit that manages a display time of each detected product, based on a result of the detection and a result of the identity determination; and
an output unit that outputs information related to the display time of each detected product
is provided.
Further, according to the present invention, a processing method including,
by a computer:
Further, according to the present invention, a program causing a computer to function as:
an acquisition unit that acquires a plurality of time-series images including a display area;
a product detection unit that detects a product from each of a plurality of the time-series images;
an identity determination unit that determines identity of products detected from the images that are different from each other;
a display time management unit that manages a display time of each detected product, based on a result of the detection and a result of the identity determination; and
an output unit that outputs information related to the display time of each detected product
is provided.
According to the present invention, a technique for recognizing a status of a sell-by date of each product on display is achieved.
In a store where a processing apparatus according to the present example embodiment is used, sell-by dates for at least some products are determined based on a display time of the products (time for which the product is on display). Specifically, the sell by date is a timing when the display time exceeds an upper limit value.
The at least some products of which sell-by dates are managed based on the display time may be a product to which a label cannot be attached, for example, such as a croquette, and a yakitori, or may be a product to which a label can be attached. Further, the product is, for example, food, but is not limited thereto, and may be another kind of product such as a household electric appliance and an article of stationary.
A product of which a sell-by date is passed or a product of which a sell-by date is close is recalled from a display shelf, or is sold at a discounted price.
When detecting a product from each of a plurality of time-series images including a display area of the product, the processing apparatus according to the present example embodiment determined identity of products detected from the images that are different from each other, and manages, based on a result of the detection and a result of identity determination, a display time of each product displayed in the display area. Further, the processing apparatus outputs information related to the display time of each product on display.
A store clerk can recognize, based on the information output from the processing apparatus, a status of a sell-by date of each product on display, specifically, “whether the sell-by date is passed”, “whether the sell-by date is close”, and the like.
Next, one example of a hardware configuration of the processing apparatus is described. Each functional unit of the processing apparatus is achieved by any combination of hardware and software, mainly including a central processing unit (CPU) of any computer, a memory, a program loaded into the memory, a storage unit (capable of storing a program that has been stored in advance from a stage of shipping an apparatus, as well as a program downloaded from a storage medium such as a compact disc (CD), a server on the Internet, or the like) such as a hard disk storing the program, and an interface for network connection. Further, it is understood by a person skilled in the art that there are various modification examples of a method and an apparatus for achieving the functional unit.
The bus 5A is a data transmission path for the processor 1A, the memory 2A, the peripheral circuit 4A, and the input/output interface 3A to mutually transmit and receive data. The processor 1A is, for example, an arithmetic processing apparatus such as a CPU, and a graphics processing unit (GPU). The memory 2A is, for example, a memory such as a random access memory (RAM), and read only memory (ROM). The input/output interface 3A includes an interface for acquiring information from an input apparatus, an external apparatus, an external server, an external sensor, a camera, and the like, and an interface for outputting information to an output apparatus, an external apparatus, an external server, and the like, and the like. The input apparatus is, for example, a keyboard, a mouse, a microphone, a physical button, a touch panel, and the like. The output apparatus is, for example, a display, a speaker, a printer, a mailer, and the like. The processor 1A is capable of issuing an instruction to each module, and performing an arithmetic operation, based on a result of arithmetic operation by each module.
The acquisition unit 11 acquires a plurality of time-series images including a display area. The display area is an area where a product is displayed, and is achieved by, for example, a product display shelf and the like. In a store in which the processing apparatus 10 is used, a camera is installed in a position and an orientation from which a product displayed in the display area (preferably, all products displayed in the display area) is captured.
The camera may continuously (for example, during business hours) capture a moving image, or may regularly capture a still image at a time interval greater than a frame interval of the moving image. The acquisition unit 11 acquires an image generated by such a camera.
The product detection unit 12 detects a product from each of a plurality of time-series images. Further, the product detection unit 12 outputs, as a detection result, position information (hereinafter, “product position information”) indicating a position in the image and product identification information (a product code, a product name, and the like) for each of the detected products.
The product detection can be achieved by using any well-known product recognition technique. For example, the product detection unit 12 detects an object area (an area where an object is present) by using a well-known object detection technique. The product detection unit 12 may detect, as the object area, an area (for example, a rectangular area) including an object and surroundings of the object. Otherwise, the product detection unit 12 may detect, as the object area, an area shaped along an outline of an object and where only the object is present. The latter can be achieved, for example, by using a method of detecting a pixel area where a detection target is present, which is called a semantic segmentation or an instance segmentation.
Further, the product detection unit 12 recognizes, by using a well-known product recognition technique (keypoint matching, deep learning, or the like), a product included in an image of the object area.
The product position information is indicated by, for example, coordinates of a two-dimensional coordinate system set on an image. The product position information may be indicated by one point in the image, or may be indicated by an area occupying a part of the image. For example, the product position information may be information indicating a representative point (a center, a barycenter, or the like) of the above-described object area, or may be information indicating the above-described object area (a partial area in the image).
The identity determination unit 13 determines identity of products detected from images different from each other. Specifically, the identity determination unit 13 determines whether a product detected from a first image is identical to a product detected from a second image. Herein, identical means that the products are the same individual. Specifically, when the products are separate individuals, the products are not identical as defined herein, even when pieces of the product identification information match with each other. In the following, an example of processing of determining identity is described.
The identity determination unit 13 is capable of determining identity of products detected from each of images different from each other, based on positions, in the image, of the products. When the positions in the image of the products each detected from each of a plurality of the images are the same or a difference between the positions is within a threshold value, the products each detected from each of the plurality of images can be determined to be an identical product. In case of the example, two images whose difference in image generation timing (capture timing) is equal to or less than a reference time become targets of identity determination processing (the above-described first image and second image). The reference time is a design matter, but becomes a value being small to some extent.
When the product position information is indicated by one point in an image, the identity determination unit 13 may use that “a distance between points is equal to or less than a threshold value”, as a condition for determining that the products are identical. Further, when the product position information is occupied by a partial area in the image, the identity determination unit 13 may use that “areas that overlap each other occupy equal to or more than a reference percentage of an area indicated by each piece of product position information”, as the condition for determining that the products are identical. The condition for determining that the products are identical exemplified herein is merely one example, and is not limited there to.
Note that, the identity determination unit 13 may use that “the above-described condition is satisfied and pieces of the product identification information match with each other”, as a condition for determining that the products are identical.
The identity determination unit 13 is capable of determining identity of products detected from each of images that are different from each other, based on feature values (shapes, sizes, colors, and the like) of appearances of the products. Even when the products are of the same kind (pieces of the product identification information match with each other), an individual difference may appear in appearance of the products. The identity determination unit 13 is capable of determining identity of the products, based on such an individual difference in appearance.
The display time management unit 14 manages, based on a result of detection by the product detection unit 12 and a result of identity determination by the identity determination unit 13, a display time of each product detected by the product detection unit 12.
For example, the display time management unit 14 is capable of starting management of a display time of a product, among products detected from an image, that is not determined to be identical to any one of products detected from an image being previous in time-series to the image, as a product newly displayed in a display area. Further, the display time management unit 14 is capable of ending management of a display time of a product, among products detected from an image, that is not determined to be identical to any one of products detected from an image being subsequent to the image.
Herein, a specific example of processing performed by the display time management unit 14 is described with reference to
As illustrated in
Further, when a product not being determined to be identical to any one of the products managed in the display time management information is present among the products detected from the first image (Yes in S12), the display time management unit 14 starts management of a display time of the product (S13). Specifically, the display time management unit 14 registers product identification information and product position information of the product in association with a new serial number, and also registers, as a display start time, a generation timing (capture timing) of the first image or a timing near the generation timing of the first image.
Note that, when there is no product managed in the display time management information at a timing of determination in S11, the display time management unit 14 determines that a product not being determined to be identical to any one of the products managed in the display time management information is present among the products detected from the first image (Yes in S12), and starts management of a display time of all the products detected from the first image (S13).
On the other hand, when a product not being determined to be identical to any one of the products managed in the display time management information is not present in the products detected from the first image (No in S12), the display time management unit 14 does not execute processing in S13.
Further, when a product not being determined to be identical to any one of the products detected from the first image is present among the products managed in the display time management information (Yes in S14), the display time management unit 14 ends management of a display time of the product (S15). For example, the display time management unit 14 may delete information of the product from the display time management information. Otherwise, the display time management unit 14 may register, in association with the product, information indicating that management of the display time is ended.
Note that, the above-described “information indicating that management of a display time is ended” may be time information or date and time information indicating the generation timing (capture timing) of the first image or a timing near the generation timing of the first image. By registering such information, a display start time and a display end time of each of a plurality of products can be recorded. Further, the information can be used in marketing and the like.
On the other hand, when a product not being determined to be identical to any one of the products detected from the first image is not present among the products managed in the display time management information (No in S14), the display time management unit 14 does not execute processing in S15.
Further, when a product being determined to be identical to any one of the products detected from the first image is present among the products managed in the display time management information (Yes in S16), the display time management unit 14 updates product position information of the product managed in the display time management information to product position information of the product detected from the first image (S17).
On the other hand, when a product being determined to be identical to any one of the products detected from the first image is not present among the products managed in the display time management information (No in S16), the display time management unit 14 does not execute processing in S17.
Note that, order of determination in S12, S14 and S16 is not limited to the illustrated example.
Returning to
The “information related to a display time” to be output may be “information indicating an approximation of a display time”. For example, the display time is divided into a plurality of groups such as “less than Ml”, “equal to or more than M1 and less than M2”, and “equal to or more than M2”. Note that, the number of the groups is any number equal to or more than two. Further, the “information indicating an approximation of a display time” becomes information indicating which group a current display time of each product belongs to.
Note that, a frame display reference as illustrated in
Further, the output unit 15 may highlight a product P of which the display time exceeds an upper limit value. There are various methods of highlighting, and for example, a frame F of the product P may be caused to blink, or may be displayed in a predetermined color (red). The upper limit value may be different for each product P.
The “information related to a display time” to be output may be “information indicating an approximation of a remaining time until a sell-by date”. The remaining time until a sell-by date (hereinafter, which may simply be referred to as a “remaining time”) is computed by subtracting the display time from the above-described upper limit value. For example, the remaining time is divided into a plurality of groups such as “less than L1”, “equal to or more than L1 and less than L2”, and “equal to or more than L2”. Note that, the number of the groups is any number equal to or more than two. Further, the “information indicating an approximation of a remaining time” becomes information indicating which group the remaining time of each product belongs to.
Note that, a frame display reference may be different for each product P. Further, the number of the above-described groups may also be different for each product P. Furthermore, the output unit 15 may highlight a product P of which the remaining time is below a lower limit value. There are various methods of highlighting, and for example, a frame F of the product P may be caused to blink, or may be displayed in a predetermined color (red).
The “information related to a display time” to be output may be a “display time” or a “remaining time until a sell-by date” itself.
Note that, in the above-described examples 1 to 3, the frame F may be substituted with another mark.
In the present example, the display time management unit 14 manages a display time of each product in a plurality of display areas, for each of the display areas. Further, the output unit 15 outputs, as information related to a display time, display area identification information in which a product of which a display time exceeds a report reference value is present. The display area identification information is information for identifying each of the plurality of display areas from each other. The report reference value may be different for each product.
As described above, according to the processing apparatus 10 of the present example embodiment, a display time of a product displayed in a display area can be managed based on a plurality of time-series images including the display area. Further, the processing apparatus 10 is capable of outputting information related to a display time of each product on display. A store clerk can recognize, based on the information output from the processing apparatus 10, a status of a sell-by date of each product on display, specifically, “whether the sell-by date is passed”, “whether the sell-by date is close”, and the like.
Further, as illustrated in
Note that, in the present description, “acquisition” includes at least one of: “retrieving data stored in another apparatus or a storage medium by an own apparatus (active acquisition)” based on a user input or based on a program instruction, for example, receiving data by making a request or query to another apparatus, reading data by accessing another apparatus or a storage medium, and the like; “inputting data output from another apparatus to an own apparatus (passive acquisition)”, based on a user input or based on a program instruction, for example, receiving data that are distributed (or transmitted, notified by a push notification, or the like), and selecting and acquiring data or information from received data or information; and “generating new data by editing data (converting the data into a text, reordering pieces of the data, extracting some pieces of the data, changing a file format, and the like) and acquiring the new data”.
While the invention of the present application has been described with reference to the example embodiment (and the examples), the invention of the present application is not limited to the above-described example embodiment (and the examples). Various modifications that can be understood by a person skilled in the art may be made in configuration and details of the invention of the present application within the scope of the invention of the present application.
A part or the entirety of the above-described example embodiment may be described as the following supplementary notes, bet is not limited thereto.
an acquisition unit that acquires a plurality of time-series images including a display area;
a product detection unit that detects a product from each of a plurality of the time-series images;
an identity determination unit that determines identity of products detected from the images that are different from each other;
a display time management unit that manages a display time of each detected product, based on a result of the detection and a result of the identity determination; and
an output unit that outputs information related to the display time of each detected product.
the output unit outputs, on the image, a processed image displaying, in association with each product detected from the image, information indicating the display time.
the output unit outputs the processed image displaying, in association with each product detected from the image, information according to a length of the display time.
the output unit outputs the processed image highlighting a product of which the display time exceeds an upper limit value.
the upper limit value is different for each product.
the identity determination unit determines identity of products detected from the images that are different from each other, based on positions of the products in the image.
the display time management unit starts management of the display time of a product, among products detected from the image, that is not determined to be identical to any one of products detected from the image being previous to the image, as a product newly displayed in the display area.
the display time management unit ends management of the display time of a product, among products detected from the image, that is not determined to be identical to any one of products detected from the image being subsequent to the image.
by a computer:
an acquisition unit that acquires a plurality of time-series images including a display area;
a product detection unit that detects a product from each of a plurality of the time-series images;
an identity determination unit that determines identity of products detected from the images that are different from each other;
a display time management unit that manages a display time of each detected product, based on a result of the detection and a result of the identity determination; and
an output unit that outputs information related to the display time of each detected product.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2020/021316 | 5/29/2020 | WO |
