Patent Application
20190005535
The present invention relates to a sales promotion system, a sales promotion method, a non-transitory computer readable medium storing a sales promotion program, and a shelf system and, particularly, to a sales promotion system, a sales promotion method, a non-transitory computer readable medium storing a sales promotion program, and a shelf system using product and customer images.
A variety of sales promotions are carried out in stores and the like where many products are displayed. For example, information on products is provided to customers through pamphlets and the like placed in stores, or salesclerks offer customers a discount based on their reactions to the products.
Patent Literatures 1 to 5, for example, are known as related art.
PTL1: Japanese Unexamined Patent Publication No. 2013-054539
PTL2: Japanese Unexamined Patent Publication No. 2012-038035
PTL3: Japanese Unexamined Patent Publication No. 2009-048430
PTL4: Japanese Unexamined Patent Publication No. 2009-003701
PTL5: Japanese Unexamined Patent Publication No. 2006-184387
For example, according to the techniques disclosed in related art like Patent Literature 1, when a customer's face or hand comes close to an infrared sensor of an electronic shelf label terminal, the display content on a display screen of the electronic shelf label terminal is changed.
However, because the technique of the related art simply detects the approach of a customer by an infrared sensor or the like, it is not possible to detect the fact that the customer has taken an interest in a product. Therefore, provision of information is often done by simply placing stationary posters, pamphlets, samples and the like. Further, in the technique of the related art, a timely promotion cannot be carried out at the moment when a customer is thinking about whether to buy a product, and it is not possible to provide information or offer a discount in accordance with the customer's action regarding the product as salesclerks do in stores.
Thus, the technique disclosed in the related art has a problem that it is difficult to carry out effective promotions in accordance with a customer's action regarding a product.
In light of the above, an exemplary object of the present invention is to provide a sales promotion system, a sales promotion method, a non-transitory computer readable medium storing a sales promotion program, and a shelf system capable of carrying out more effective promotions.
A sales promotion system according to an exemplary aspect of the present invention includes an image information acquisition means for acquiring input image information on an image taken of a presentation area where a product is presented to a customer, a tracking means for tracking an action of the customer regarding the product based on the input image information, and a promotion information output means for outputting promotion information related to the product corresponding to the tracked customer's action to the customer.
A sales promotion method according to an exemplary aspect of the present invention includes acquiring input image information on an image taken of a presentation area where a product is presented to a customer, tracking an action of the customer regarding the product based on the input image information, and outputting promotion information related to the product corresponding to the tracked customer's action to the customer.
A non-transitory computer readable medium storing a sales promotion program according to an exemplary aspect of the present invention causes a computer to perform a sales promotion process including acquiring input image information on an image taken of a presentation area where a product is presented to a customer, tracking an action of the customer regarding the product based on the input image information, and outputting promotion information related to the product corresponding to the tracked customer's action to the customer.
A shelf system according to an exemplary aspect of the present invention includes a shelf placed to present a product to a customer, an image information acquisition means for acquiring input image information on an image of the product and the customer taken, a tracking means for tracking an action of the customer regarding the product based on the input image information, and a promotion information output means for outputting promotion information related to the product corresponding to the tracked customer's action to the customer.
According to the exemplary aspects of the present invention, it is possible to provide a sales promotion system, a sales promotion method, a non-transitory computer readable medium storing a sales promotion program, and a shelf system capable of carrying out more effective promotions.
Prior to describing exemplary embodiments, an overview of the characteristics of exemplary embodiments is given hereinbelow.
As shown in
As described above, in the exemplary embodiment, a customer's action regarding a product is tracked, and promotion information is output to the customer based on the tracked customer's action regarding the product. For example, by tracking the hand, the line of sight and the like of a customer as a customer's action, it is possible to detect the fact that the customer has become interested in the product or the like, which enables effective promotions in accordance with the customer's interests.
A first exemplary embodiment is described hereinafter with reference to the drawings.
As shown in
The 3D (three-dimensional) camera 210 is an imaging device (distance image sensor) that takes an image of and measures a target and generates a distance image (distance image information). The distance image (range image) contains image information which is an image of a target taken and distance information which is a distance to a target measured. For example, the 3D camera 210 is Microsoft Kinect (registered trademark) or a stereo camera. By using the 3D camera, it is possible to recognize (track) a target (a customer's action or the like) including the distance information, and it is thus possible to perform highly accurate recognition.
As shown in
Note that, although an example in which the 3D camera 210 is used as a device that takes images of the product shelf 300 and the customer 400 is described below, it is not limited to the 3D camera and may instead be a general camera (2D camera) that outputs only images taken. In this case, tracking is performed using the image information only.
Each of the facial recognition camera 220 and the in-store camera 230 is an imaging device (2D camera) that takes and generates an image of a target. The facial recognition camera 220 is placed at the entrance of a store or the like, takes an image of a face of a customer who comes to the store and generates a facial image to recognize the customer's face. The in-store camera 230 is placed at a plurality of positions in a store, takes an image of each section in the store and generates an in-store image to detect the customer traffic flow in the store. Note that each of the facial recognition camera 220 and the in-store camera 230 may be a 3D camera. By using a 3D camera, it is possible to accurately recognize the customer's face or the customer's moving route.
The information presentation device 240 is a device that presents (outputs) the promotion information generated by the sales promotion device 100 to a customer. The way of presentation (output) to a customer is not limited, and it may be a display of letters and images on a display device, audio output through a speaker or the like. The sales promotion device 100 is placed at a position where the customer 400 can view (hear) it.
As shown in
As shown in
Each element in the sales promotion device 100 may be formed by hardware or software or both of them, and may be formed by one hardware or software or a plurality of hardware or software. For example, the product information DB 150 and the customer information DB 160 may be storage devices connected to an external network (cloud). Each function (each processing) of the sales promotion device 100 may be implemented by a computer including CPU, memory and the like. For example, a sales promotion program for performing a sales promotion method (sales promotion process) according to the exemplary embodiments may be stored in a storage device, and each function may be implemented by executing the sales promotion program stored in the storage device on the CPU.
This sales promotion program can be stored and provided to the computer using any type of non-transitory computer readable medium. The non-transitory computer readable medium includes any type of tangible storage medium. Examples of the non-transitory computer readable medium include magnetic storage media (such as floppy disks, magnetic tapes, hard disk drives, etc.), optical magnetic storage media (e.g. magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R/W, and semiconductor memories (such as mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory), etc.). The program may be provided to a computer using any type of transitory computer readable medium. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium cart provide the program to a computer via a wired communication line such as an electric wire or optical fiber or a wireless communication line.
The distance image analysis unit 110 acquires a distance image generated by the 3D camera 210, tracks a detection target based on the acquired distance image, and recognizes its action. In this exemplary embodiment, the distance image analysis unit 110 mainly tracks and recognizes a customer's hand, a customer's line of sight, and a product picked up by a customer. The distance image analysis unit 110 refers to the product information DB 150 to recognize a product contained in the distance image. Note that, although the distance image analysis unit 110 detects a customer's action in this example, it may detect a customer's facial expression (pleasure, surprise etc.), heart rate and the like based on the distance image and present the promotion information in accordance with the facial expression or heart rate. Further, a microphone may be mounted on a 3D camera, and a customer's voice input to the microphone may be recognized by a voice recognition unit. For example, based on the recognized voice, the feature (the loudness, pitch, tempo etc. of a voice) of a customer's conversation may be detected, and the promotion information may be presented in accordance with the feature of the conversation.
The customer recognition unit 120 acquires a facial image of a customer generated by the facial recognition camera 220 and recognizes a customer contained in the acquired facial image by referring to the customer information DB 160. The flow analysis unit 130 acquires an in-store image generated by the in-store camera 230, analyzes the moving history of a customer in the store based on the acquired in-store image and detects the traffic flow (moving route) of the customer.
The presentation information generation unit 140 generates promotion information to be presented to a customer based on detection results of the distance image analysis unit 110, the customer recognition unit 120 and the flow analysis unit 130, and outputs the generated promotion information to the information presentation device 240. The promotion information is information for promoting the sales (purchase) of a product, and it is information that encourages a customer to determine to purchase a product. The presentation information generation unit 140 refers to the product information DB 150 and the customer information DB 160, and generates, as the promotion information, information related to the product which the distance image analysis unit 110 detects has been touched by the customer, information on a product related to the customer recognized by the customer recognition unit 120, and information on a product related to the customer traffic flow analyzed by the flow analysis unit 130.
Note that, although an example where the promotion information in accordance with an action is generated and output to (displayed on) the information presentation device is described in this exemplary embodiment, the generated promotion information may be output to another device or system for further use of the promotion information. For example, the promotion information may be stored as a customer's wish list in the customer information DB or another database, and the promotion information may be provided to the customer through a network.
The product information DB (product information storage unit) 150 stores product related information that is related to products placed in a store. The product information DB 150 stores, as the product related information, product identification information 151, characteristic/simultaneous buying (simultaneous selling) information 152, push information 153, comparison information 154 and the like.
The product identification information 151 is information for identifying a product (product master), and it includes a product code, a product name, a product type, product label image information (image) and the like. The characteristic/simultaneous buying information 152 is characteristic information indicating the characteristics of a product and information (recommendation information, suggestion information) for suggesting to a customer the simultaneous buying of another product with the product, and it includes a product's material or place of origin, food to be eaten in combination with the product, a recipe that uses the product as ingredients, a coordinated product and the like.
The push information 153 is information for encouraging a customer to determine to purchase a product, and it includes, for example, discount information such as a discount coupon. The comparison information 154 is information indicating a comparison of the product with another product, and it includes information about a comparison of the product's, for example, characteristics, price and the like with those of the other product.
The customer information DB (customer information storage unit) 160 stores customer related information that is related to customers who come to a store. The customer information DB 160 stores, as the customer related information, customer identification information 161, attribute information 162, preference information 163, history information 164 and the like.
The customer identification information 161 is information for identifying a customer, and it includes a customer's membership ID, name, address, birth date, facial image information (image) and the like. The attribute information 162 is information indicating the attributes of a customer, and it includes for example, the age, gender, occupation and the like of the customer.
The preference information 163 is information indicating the preferences of a customer, and it includes, for example, a hobby, favorite food, color, music, movie and the like of the customer. The history information 164 is information about the history of a customer, and it includes, for example, a product purchase history, a store visit history, an in-store moving history, a contact history (access history) such as picking up/looking at a product and the like.
The distance image acquisition unit 111 acquires a distance image containing a customer and a product which is taken and generated by the 3D camera 210. The region detection unit 112 detects a region of each part of a customer or a region of a product contained in the distance image acquired by the distance image acquisition unit 111.
The hand tracking unit 113 tracks the action of a customer's hand detected by the region detection unit 112. The hand action recognition unit 114 recognizes the customer's action regarding a product based on the hand action tracked by the hand tracking unit 113. For example, when a customer brings the palm of his/her hand toward his/her face while holding the product, the hand action recognition unit 114 determines that the customer has picked up and looked at the product. In the case where, when a product is held in the customer's hand, the hand is hidden behind the product and an thus image of it is not taken by the camera, the hand action recognition unit 114 may detect the position, direction or change of the product being held and thereby determine that the customer has picked up the product.
The sight line tracking unit 115 tracks the action of the customer's line of sight (eye) detected by the region detection unit 112. The sight line action recognition unit 116 recognizes the customer's action regarding a product based on the action of the customer's line of sight (eye) detected by the sight line tracking unit 115. When a product is placed in the direction of the line of sight, the sight line action recognition unit 116 determines that the customer has looked at the product.
The product tracking unit 117 tracks the action (state) of a product detected by the region detection unit 112. The product tracking unit 117 tracks the product which the hand action recognition unit 114 has determined that the customer has picked up or the product which the sight line action recognition unit 116 has determined that the customer has looked at. The product recognition unit 118 identifies which product corresponds to the product tracked by the product tracking unit 117 by referring to the product information DB 150. The product recognition unit 118 compares the label of the detected product with the image information of the label of the product identification information 151 stored in the product information DB 150 and performs matching to thereby recognize the product. Further, the product recognition unit 118 stores the relationship between placement positions on a shelf and products into the product information DB 150, and identifies the product based on the product picked up by the customer or the position of the shelf on which the product looked at by the customer is placed.
A sales promotion method (sales promotion process) that is performed in the sales promotion system (sales promotion device) according to this exemplary embodiment is described hereinafter with reference to
As shown in
After that, the customer picks up a product placed on the shelf (S103). Then, the 3D camera 210 in the vicinity of the shelf takes an image of the customer's hand, and the sales promotion device 100 recognizes the action of the customer's hand and a product type by using the distance image of the 3D camera 210 (S104). Specifically, the distance image analysis unit 110 in the sales promotion device 100 tracks the distance image of an image of the customer's hand (line of sight) and the product, and detects the action that the customer has picked up the product (the customer looks at the product) and detects the product that matches the product picked up by the customer by referring to the product information DB 150, and thereby recognizes the product picked up by the customer (the product looked at by the customer).
Then, the sales promotion device 100 presents the promotion information (relevant information) such as product information or discount of the product recognized in S104 to the information presentation device 240 (S105). The customer then reads the presented information (provided information) presented on the information presentation device 240 and thereby determines whether to purchase the product he/she has picked up (or looked at) (S106). Specifically, the presentation information generation unit 140 in the sales promotion device 100 presents the promotion information related to the recognized product by referring to the product information DB 150 and the customer information DB 160. Further, the presentation information generation unit 140 presents, as the promotion information, information about the product related to the customer recognized in S102 and further presents information about the product related to the customer traffic flow analyzed in S108, which is described later.
After that, the customer moves to another section (S107). Then, the in-store camera 230 takes an image of the customer's movement between sections, and the sales promotion device 100 grasps the purchase behavior in another section (S108). Specifically, the flow analysis unit 130 in the sales promotion device 100 analyzes the customer's moving history based on the images of a plurality of sections and detects the customer traffic flow and thereby grasps the purchase behavior of the customer. Then, the processing after S103 is repeated, and when the customer picks up a product in a section to which he/she has moved, information related to the product on the basis of the customer traffic flow (purchase behavior) is presented to the customer.
As shown in
Then, the hand tracking unit 113 tracks the action of the customer's hand detected in S203 (S204). The hand tracking unit 113 tracks the hone structure of the customer's hand and its vicinity and detects the action of the finger or palm of the hand based on the image and the distance contained in the distance image.
After that, the hand action recognition unit 114 extracts the feature of the action of the hand based on the action of the hand tracked in S204 (S205), and recognizes the action of the customer's hand on the product, which is the action of holding the product or the action of looking at the product, based on the extracted feature (S206). The hand action recognition unit 114 extracts the direction, angle, and change in movement of the finger or the palm (wrist) as a feature amount. For example, the hand action recognition unit 114 detects that the customer is holding the product from the angle of the finger, and when the direction of the normal to the palm is toward the face, it detects that the customer is looking at the product. Further, the state of holding a product or the state of picking up and looking at a product may be learned in advance, and the action of the hand may be identified by comparison with the learned feature amount.
After S203, the sight line tracking unit 115 tracks the action of the customer's line of sight detected in S203 (S207). The sight line tracking unit 115 tracks the bone structure of the customer's face and its vicinity and detects the action of the face, eye and pupil based on the image and the distance contained in the distance image.
After that, the sight line action recognition unit 116 extracts the feature of the action of the line of sight based on the action of the line of sight tracked in S207 (S208), and recognizes the action of the customer's line of sight on the product, which is the action that the customer is looking at the product, based on the extracted feature (S209). The sight line action recognition unit 116 extracts the direction, angle, and change in movement of the face, eye and pupil as a feature amount. For example, the sight line action recognition unit 116 detects the direction of the light of sight based on the action of the face, eye and pupil and detects whether the direction of the line of sight is toward the product or not. Further, the state of looking at a product may be learned in advance, the action of the line of sight may be identified by comparison with the learned feature amount.
After S203, the product tracking unit 117 tracks the action (state) of the product detected in S203 (S210). Further, the product tracking unit 117 tracks the product determined that the customer picks up in S206 and the product determined that the customer looks at in S209. The product tracking unit 117 detects the orientation, position and the like of the product based on the image and the distance contained in the distance image.
Then, the product recognition unit 118 extracts the feature of the product tracked in S210 (S211) and, based on the extracted feature, recognizes the corresponding product from the product information DB 150 (S212). The product recognition unit 118 extracts the letters or image of the label on the product as a feature amount. For example, the product recognition unit 118 compares the extracted feature amount of the label with the feature amount of the label in the product information DB 150 and retrieves the product where the feature amount matches and thereby identifies the product. Further, in the case where the relationship between placement positions on the shelf and products is stored in the product information DB 150, the position on the shelf of the product which the customer picks up or looked at is acquired, and the position of the shelf is retrieved from the product information DB 150 to thereby detect the matching product.
As shown in
Specifically, the presentation information generation unit 140 acquires customer information (S302) and acquires flow information (S303). The presentation information generation unit 140 acquires the customer information from the customer recognition unit 120 so as to provide information on the basis of a customer recognition result by the facial image of the facial recognition camera 220 to the customer. Further, the presentation information generation unit 140 acquires the flow information from the flow analysis unit 130 so as to provide information on the basis of a customer traffic flow analysis result by the in-store image of the in-store camera 230 to the customer. Note that, before recognizing a customer, before a customer moves, and when each information is not necessary, the acquisition of customer information and the acquisition of flow information may be omitted.
Then, the presentation information generation unit 140 generates the promotion information (product related information) and presents it to the customer in S304 to S306. Note that S304 to S306 show one example of information to be presented, and other promotion information may be presented. The presentation of S304 to S306 may be done at the same time or in an arbitrary order. Further, information that is arbitrary selected among S304 to S306 may be presented or may be presented together with other information.
In S304, the presentation information generation unit 140 refers to the product information DB 150 and acquires the characteristic/simultaneous buying information 152 related to the target product which the customer picks up/looks at, generates the acquired characteristic/simultaneous buying information 152 as the presentation information and outputs it to the information presentation device 240.
For example, the presentation information generation unit 140 generates information such as area of production or raw material of a product which a customer picks up, food to be eaten in combination with a product which a customer picks up (cheese that is good with wine when wine is picked up), food from similar area of production or raw material, a recipe that uses the product as ingredients (pizza when cheese is picked up), ingredients and cookware included in this recipe (pizza crust or topping, oven range etc.), coordinated products (clothes, shoes, belt, bag etc.) and the like. Further, for the related product, information that instructs a route or guides the way of shopping in the store.
Further, among the characteristic/simultaneous buying information 152, adapted information such as information narrowed down by the customer information acquired in S302 or information narrowed down by the flow information acquired in S303 is used as the presentation information. For example, the provided information is adapted by the recognized customer attribute such as a gender and an age or the member customer ID, and the provided information is adapted by findings of the purchase or interests in another section by the customer traffic flow analysis. For example, information related to diet, allergy information and health may be presented based on the attributes, preferences, history and the like. When a plurality of digital signage are placed in a store, for example, the promotion information may be presented in the plurality of digital signage in conjunction along the recognized customer traffic flow.
In S305, the presentation information generation unit 140 refers to the product information DB 150 and acquires the push information 153 related to the target product which the customer picks up/looks at, generates the acquired push information 153 as the presentation information and outputs it to the information presentation device 240. Particularly, when a customer keeps looking at a product for a specified period of time, it is assumed that the customer is thinking about whether to purchase the product, and it is preferred to present the push information (encouraging information).
For example, the presentation information generation unit 140 generates a discount proposal such as “50% discount for the second item”, a discount coupon, word-of-mouth information, ranking information and the like for the product which the customer picks up. Information of a discount coupon may be displayed, and the coupon may be made available by cooperation with a smartphone. A barcode of a product may be read through a reader device of a shopping cart or the like, and the current subtotal is displayed, and further the amount after the discount may be displayed.
Further, as in S304, among the push information 153, adapted information such as information narrowed down by the customer information acquired in S302 or information narrowed down by the flow information acquired in S303 is used as the presentation information.
In S306, the presentation information generation unit 140 refers to the product information DB 150 and acquires the comparison information 154 related to the target product which the customer picks up/looks at, generates the acquired comparison information 154 as the presentation information and outputs it to the information presentation device 240.
For example, when the customer picks up two products, the presentation information generation unit 140 generates information that compares the characteristics, functions, price, discount and ranking of those two products. In the case where there is a product whose purchase is to be recommended among those two products, the push information such as discount information is presented in the same manner as in S305. When it is detected that the customer is looking at the display of a best-before date on a product label, discount information of the older one (the one with an earlier best-before date) may be displayed.
Further, as in S304, among the comparison information 154, adapted information such as information narrowed down by the customer information acquired in S302 or information narrowed down by the flow information acquired in S303 is used as the presentation information.
As described above, in this exemplary embodiment, the customer's hand motion is observed by the 3D camera placed at the potion from which a product shelf and a client (shopper) in front of the shelf can view to recognize which product the customer picks up. Then, on an information presentation device (display device) adjacent to the product, the promotion information such as information about the product, a discount proposal or related recommendation information is displayed.
Because it is possible to precisely grasp the hand motion by the 3D camera and thereby grasp the action on a product such as touching the product, picking up the product or rotating the product to read the label, the promotion information adapted to the motion can be displayed timely (real time). Specifically, because it is possible to accurately detect that the customer becomes interested in the product based on such an action, the promotion can be carried out effectively for the customer who becomes interested in the product, thereby promoting the purchase of the product and increasing the sales.
For example, when it is detected that a customer has two products in both hands and compares them, comparison information of the both products, presentation of a coupon for the product desired to be sold or the like may be displayed to thereby encourage (push) the customer who is thinking about whether to purchase the product to determine the purchase of the product.
Further, by adapting the promotion information with the customer information recognized using the facial recognition camera or adapting the promotion information with the customer traffic flow analyzed using the in-store camera, it is possible to present more effective information.
A second exemplary embodiment is described hereinafter with reference to the drawings. In this exemplary embodiment, an example where the first exemplary embodiment is applied to one shelf system is described.
As shown in
The presentation information generation unit 140 generates promotion information to be presented to a customer based on a detection result of the distance image analysis unit 110, and outputs the generated promotion information to the information presentation device 240. The presentation information generation unit 140 refers to the product information DB 150 and generates, as the promotion information, information related to the product which the distance image analysis unit 110 detects that a customer has touched.
When the customer performs the corresponding action, the presentation information generation unit 140 generates promotion information (product related information) in S304 to S306 and presents it to the customer. Although the generation of information on the basis of customer information and flow information is not done in this exemplary embodiment, which is different from the first exemplary embodiment, the other operation is the same as that in the first exemplary embodiment.
Specifically, the presentation information generation unit 140 acquires the characteristic/simultaneous buying information 152 related to the target product which the customer picks up/looks at, generates the presentation information such as food to be eaten in combination and outputs it to the information presentation device 240 (S304). Further, the presentation information generation unit 140 acquires the push information 153 related to the target product which the customer picks up/looks at, generates the presentation information such as a discount coupon and outputs it to the information presentation device 240 (S305). Further, the presentation information generation unit 140 acquires the comparison information 154 related to the target product which the customer picks up/looks at, generates the presentation information such as information about the comparison in characteristics and the like between two products held in both hands and outputs it to the information presentation device 240 (S306).
As described above, in this exemplary embodiment, the main elements in the first exemplary embodiment are included in one product shelf. It is thereby possible to accurately detect a customer's action on a product and display the promotion information in accordance with the customer's action in a timely manner.
Further, because this exemplary embodiment can he implemented with one product shelf only, a device or a system other than the shelf is not required. It is thus possible to easily introduce this system even in a store where there is no advanced system such as a POS system or a network.
It should be noted that the present invention is not limited to the above-described exemplary embodiment and may be varied in many ways within the scope of the present invention.
While the invention has been particularly shown and described with reference to exemplary embodiments thereof, the invention is not limited to these embodiments. It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the claims.
This application is based upon and claims the benefit of priority from Japanese patent application No. 2013-185129, filed on Sep. 6, 2013, the disclosure of which is incorporated herein in its entirety by reference.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2013-185129 | Sep 2013 | JP | national |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14916723 | Mar 2016 | US |
| Child | 16125893 | US |
