Patent Application
20160110791
The present disclosure relates to a sensor-based environment, and more specifically, to providing an adaptive personal experience within such an environment using a determined field of view of the person.
To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation. The illustrations referred to here should not be understood as being drawn to scale unless specifically noted. Also, the drawings are often simplified and details or components omitted for clarity of presentation and explanation. The drawings and discussion serve to explain principles discussed below, where like designations denote like elements.
Aspects of the current disclosure relate to an integrated environment capable of providing a personalized, automated, and adaptive experience for a person within the environment. A number of different sensor devices may be employed within the environment, and networked with various computing devices such as point-of-sale (POS) terminals, digital signage, servers, and mobile or handheld computing devices to provide a seamless integration of mobile technologies and e-commerce into a traditional experience.
Using one or more visual sensors within the environment, a retailer or other provider may determine a person's field of view, and may compile personal behaviors and determine personal preferences. This data may then be used to provide timely, tailored recommendations in real-time to the person in order to more effectively influence their experience. While generally discussed within the context of a shopping environment, such as a retail store or other commercial environment, it is contemplated that the techniques disclosed herein may be applied to other environments (some non-limiting examples include libraries, museums, classrooms, hospitals, etc.) to provide an adaptive experience for persons included therein.
Terminals 105 generally include any structure that is capable of receiving input from customers and/or producing output to customers within the environment 100. The terminals 105 may include computing systems, portions of computing systems, or devices controllable by computing systems. In one example, a terminal may include a computing device that is communicatively coupled with a visual display and audio speaker(s), as well as being communicatively coupled with one or more input devices. In another example, a terminal may include a visual display and associated driver hardware, but a computing device coupled to the terminal and providing data for display is disposed separately from the terminal. In some embodiments, terminals 105 may be implemented as standalone devices, such as a kiosk disposed on the store floor or monolithic device disposed on a shelf or platform. In some embodiments, terminals 105 may be integrated partially or wholly with other components of the environment 100, such as input or output devices included with shelving or other structural components in the environment (e.g., components used for product display or storage). In some embodiments, terminals 105 may be modular and may be easily attachable and detachable to elements of the environment 100, such as the structural components.
Generally, terminals 105 may be distributed throughout the environment 100 and may enhance various phases of the shopping experience for customers. For example, terminals 105 may include digital signage 108 disposed throughout the environment, such as included in or near aisles, endcaps, displays, and/or shelving in the environment. A customer may view and/or interact with the digital signage 108 as he/she moves through the store environment. The digital signage may be included in a static display or may be movable, such as including digital signage within a customer's shopping cart or basket. Terminals 105 may also include POS terminals 106 that provide a checkout functionality, allowing the customer to complete his/her shopping transaction (e.g., make payment for selected items). In some embodiments, terminals 105 may provide an integrated functionality. For example, the terminals may function in one mode as digital signage, and when engaged by a customer, the terminals function as a POS terminal.
Servers 1101, 1102 generally include processors, memory, and communications capabilities, and may perform various computing tasks to support the commercial operation of the environment 100. Servers 1101, 1102 communicate using various wired and/or wireless communications methods with terminals 105, sensors 120, and with other networked devices such as user devices 140 and other devices 150. Servers 1101, 1102 generally execute computer program code in which input data is received from networked devices, the input data is processed and/or stored by the servers, and output data is provided to networked devices for operation of the environment 100.
Sensors 120 may include various sensor devices, such as video sensors 125, audio sensors 130, and other sensors 135. The other sensors 135 generally include any sensors that are capable of providing meaningful information about customer interactions with the environment, e.g., location sensors, weight sensors, and so forth. Sensors 120 may be deployed throughout the environment 100 in fixed and/or in movable locations. For example, sensors 120 may be statically included in walls, floors, ceilings, displays, or other devices, or may be included in shopping carts or baskets capable of being transported around the environment. In one embodiment, sensors 120 may include adjustable position sensor devices, such as motorized cameras attached to a rail, wire, or frame. In one embodiment, sensors 120 may be included on one or more unmanned vehicles, such as unmanned ground vehicles (UGVs) or unmanned aerial vehicles (UAVs or “drones”). Sensors 120 may also include sensor devices that are included in user devices 140 or other devices 150 (which in some cases may include body-worn or carried devices). User devices 140 and other devices 150 may include passive or actively-powered devices capable of communicating with at least one of the networked devices of environment 100. One example of a passive device (which may be worn or carried) is a near-field communication (NFC) tag. Active devices may include mobile computing devices, such as smartphones or tablets, or wearable devices such as a Google Glass™ interactive eyepiece (Glass is a trademark of Google Inc.). The user devices 140 generally denotes ownership or possession of the devices by customers, while the other devices 150 denotes ownership or possession by the retailer or other administrator of the environment 100. In some cases, other devices 150 may be carried by employees and used in the course of their employment. User devices 140 and other devices 150 may execute applications or other program code that generally enables various features provided by the servers and/or other networked computing devices.
Memory 210 may include a variety of computer-readable media selected for their size, relative performance, or other capabilities: volatile and/or non-volatile media, removable and/or non-removable media, etc. Memory 210 may include cache, random access memory (RAM), storage, etc. Storage included as part of memory 210 may typically provide a non-volatile memory for the networked computing devices (e.g., servers 1101, 1102), and may include one or more different storage elements such as Flash memory, a hard disk drive, a solid state drive, an optical storage device, and/or a magnetic storage device. Memory 210 may be included in a single computing device or may represent an aggregation of memory included in networked devices. Memory 210 may include a plurality of modules 211 for performing various functions described herein. The modules 211 include program code that is executable by one or more of the processors 205. As shown, modules 211 include user identification 212, item identification 214, advertising 216, recommendations 218, virtual cart 220, assistance 222, security 224, power management 226, gaming 228, audit 230, loyalty program 232, and inventory 234. The modules 211 may also interact to perform certain functions. For example, a loyalty program module 232 during operation may make calls to user identification module 212, item identification module 214, advertising module 216, and so forth. The person of ordinary skill will recognize that the modules provided here are merely non-exclusive examples; different functions and/or groupings of functions may be included as desired to suitably operate the shopping environment. Memory 210 may also include customer profiles 236 and customer images 238, which may be accessed and/or modified by various of the modules 211. In one embodiment, the customer profiles 236 and customer images 238 may be stored on the servers 1101, 1102 or on a separate database.
Input/output (I/O) 250 may include a number of different devices interfacing with various computing devices and with the shopping environment. I/O 250 includes sensors 120, which have been described above. I/O 250 may further include input devices 252 and output devices 254 that are included to enhance the shopping experience for customers. In one embodiment, terminals 105, user devices 140, and other devices 150 may include visual displays and/or audio speakers (examples of the output devices 254), and various input devices 252 (such as cameras, keyboards or keypads, touchscreens, buttons, inertial sensors, etc.). I/O 250 may further include wired or wireless connections to an external network 256 using I/O adapter circuitry. Network 256 may include one or more networks of various types, including a local area or local access network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet). In one embodiment, various networked computing devices of the system 200 are interconnected using a LAN, and one or more computing devices (e.g., servers 1101, 1102, user devices 140) include connections to the Internet.
Environment 300 also includes a number of kiosks (or terminals) 305. Generally, kiosks 305 may be configured for performing customer checkout and/or other shopping functions. Each kiosk 305 may each include computing devices or portions of computing systems, and may include various I/O devices, such as visual displays, audio speakers, cameras, microphones, etc. for interacting with the customer. In some embodiments, a customer 340 may have a mobile computing device, such as smartphone 345, that communicatively couples with the kiosk 305 for completing the checkout transaction. In one embodiment, the mobile computing device may execute a store application that is connected to the networked computing systems (e.g., through servers 1101, 1102), or may be directly connected to kiosk 305 through wireless networks within the environment (e.g., over Wi-Fi or Bluetooth). In one embodiment, the mobile computing device may couple to the kiosk 305 when brought within range, e.g., using Bluetooth or NFC.
Environment 300 also includes one or more shelving units 303 having shelves 310 that support various store items 315. Though not shown, multiple shelving units 303 may be disposed in a particular arrangement in the environment 300, and the space between adjacent shelving units may form aisles through which customers may travel. In some embodiments, the shelving unit 303 may include visual sensors or other sensor devices or I/O devices. The sensors or devices may couple to a customer's smartphone 345 and/or other networked computing devices (including servers) within the environment 300. For example, the front portions 320 of shelves 310 may include video sensors oriented outward from the shelving unit 303 to capture customer interactions with items 315 on the shelving unit 305, and the data from the video sensors may be provided to back-end servers for storage and/or analysis. In some embodiments, portions of the shelving unit 303 (such as the front portions 320 of shelves 310) may include indicator lights or other visual display devices or audio output devices that are used to communicate with a customer.
I/O 250 includes output devices 254 and sensors 120. Output devices 254 include one or more devices for presenting information to customers and include audio output devices 440 and visual output devices 445. The audio output devices 440 may include conventional audio speakers having any suitable form factor (e.g., in a stereo, headphones, etc.), as well as devices using alternative methods of producing sound to a customer, such as bone conduction transducers in a worn device. Visual output devices 445 may include visual displays and various visual indicators such as light emitting diodes (LEDs). Other output devices 450 may provide information to customers through tactile feedback (e.g., haptic devices) or other sensory stimuli. Sensors 120 may include visual sensors 455 which may be carried or worn sensors 460, and distributed sensors 465 that are disposed throughout the shopping environment. Other sensors 470 may be included that are suitable for collecting information about a customer and his/her interactions within the shopping environment. Examples of other sensors 470 may include infrared (IR) sensors, thermal sensors, weight sensors, capacitive sensors, magnetic sensors, sonar sensors, radar sensors, lidar sensors, and so forth.
The visual sensors 455 may be used to capture one or more images 415 of the customer and/or the shopping environment, which may include views from various perspectives (e.g., a customer-worn visual sensor, static or movable visual sensors at various locations in the environment). The images 415 may be stored in memory 210, and may be individually or collectively processed to determine information about customers in the environment and their respective interactions with items in the environment.
Memory 210 includes one or more programs 435 that collectively receive data about customers and the shopping environment, process the received data, and transmit information to customers in order to influence the customers' shopping experiences. Programs 435 may include program code to determine a customer's field of view at a given time, including which items are included in the field of view. In one embodiment, the customer's field of view may be determined directly. For example, a body-worn device may include a visual sensor (i.e., a worn visual sensor 460) that, when the device is worn, gives the visual sensor an orientation that is similar to the orientation of customer's head or eyes (e.g., a forward-looking camera). Images captured from the worn visual sensor may generally reflect the customer's field of view.
In some embodiments, the customer's field of view may be estimated (or determined indirectly) using other sensor measurements. In one embodiment, the customer's field of view may be estimated by determining the orientation of one or both of the customer's eyes. Eye orientation may be determined using worn visual sensors 460 (e.g., an inward-facing camera on a head-worn device) and/or distributed visual sensors 465 (e.g., capturing images of the customer's face and image processing to determine an eye orientation). In other embodiments, the customer's field of view may be estimated by determining the position and/or orientation of the customer's head and/or body using various visual sensor measurements. The customer's field of view may be represented in any suitable data format, such as an image or as coordinate data (e.g., Cartesian, polar, spherical coordinates).
While it is possible that a single visual sensor 455 may be used to determine a customer's field of view, several embodiments employ a combination of a plurality of visual sensors 455 to determine a field of view. These embodiments may be preferred as providing additional data to support a more accurate estimate of the field of view. Additionally, the plurality of visual sensors used to determine a field of view may include visual sensors selected from different categories (worn sensors 460, distributed sensors 465) to provide additional robustness to the collected data.
Programs 435 may also include program code to identify one or more items included within the customer's field of view. The identification process may be performed directly or estimated. One example of direct identification is performing image processing on images collected from a worn, forward-looking camera to visually identify one or more items. Estimating items within a customer's field of view may require combining sensor data with known information about the shopping environment, such as the items in the environment (item data 420) and their relative arrangement or layout within the environment (location data 425).
Programs 435 may also include program code to present information to customers based on the identified one or more items. In some embodiments, the information may be used to influence the customer's shopping experience. The information presented to the customer may include information about the identified items (e.g., nutritional data, pricing, a customer's purchase history of the items, etc.), information encouraging the purchase of identified items (e.g., bringing particular items to the customer's attention, touting the items' features, offering discounts or other promotions on the items, etc.), and information encouraging the purchase of alternatives to the identified items (e.g., highlighting differences of the items, offering discounts, etc.).
To present relevant and persuasive information in real-time to a particular customer, programs 435 may access and analyze various additional data in memory 210 that is related to the customer and perhaps other customers of the shopping environment. In one embodiment, programs 435 may analyze shopping data 430 collected from previous shopping experiences for the particular customer and/or for other customers or groups of customers. For example, shopping data 430 may include customer views and the items included therein, progressions of customer views (showing a customer's interest over time), selection or purchase history for items, and so forth. While shopping data 430 may be compiled and used to generate information to present to customers and influence their shopping experiences in real-time, the shopping data 430 may also be used by the retailer or administrator of the shopping environment to modify the layout of the environment. The shopping data 430 may help the retailer identify trends in customer shopping, and to optimize placement of items within the environment to improve customer sales.
System 400 may also present information to customers based on their personal preferences 405. The preferences 405 may generally be stored in a corresponding customer profile 236, and may reflect preferences that are explicitly specified by a customer, or may be determined based on the customer's historical shopping behavior (e.g., included in shopping data 430). For example, a customer may have an allergy to a particular ingredient, and the customer may enter this allergy information in preferences 405, e.g., using a mobile phone app for the retailer. Accordingly, the system 400 when determining which information to present to the customer may present information that highlights items within the customer's field of view that include the ingredient, and may further suggest alternative items that do not include the ingredient.
A customer's shopping history may also be used to determine customer preferences 405. For example, the customer's determined fields of view and purchase history from shopping data 430 may be processed to deduce which items, combinations of items, and/or aspects of items are preferred by the customer. For example, preferred aspects might include preferred brands, costs, quantities, sizes, ingredients, nutritional properties (e.g., calorie content, fat, sugar, vitamins, minerals, etc.), and so forth. For example, the customer may specify a preference for low-fat foods, and the system may determine recommended items based on the items included in the customer's field of view and the customer's preferences. This may include suggesting a particular item within the field of view for purchase (or alternatively, an item located outside the field of view) and/or advising the customer about the item's properties vis-à-vis the customer's preferences (e.g., reporting fat content).
Of course, a customer's preferences may be included as a logical combination of a plurality of these aspects (e.g., a customer prefers Brand X items to Brand Y, so long as the cost of the Brand X item is no more than 150% of the Brand Y item). In some embodiments, other customers' shopping data may also be used to deduce a particular customer's preferences. Of course, the preferences may be dynamically updated to identify whether deduced preferences are accurate or not. The dynamic updating may be caused by the customer's explicit indication and/or by the customer's shopping patterns following the deduced preference. For example, the system 400 may deduce that a customer has a categorical preference for Brand X items over similar Brand Y items. However, the customer's historical shopping data indicated that the customer looked at a Brand X item (e.g., field of view data) before selecting and purchasing a similar Brand Y item (e.g., field of view data and/or purchase history data). The system in response may adapt or may entirely remove the deduced (and determined inaccurate) preference.
In some embodiments, system 400 may present information to customers that is also based on other programs 410. Examples of programs 410 may include fitness, nutrition, or health goals, money management goals, etc. In some embodiments, the programs 410 themselves may be integrated into a store application and accessible by the customer's mobile computing device or wearable device. In other embodiments, the store application may interface with applications from other providers to determine the customer's goals and present appropriate information during the shopping experience. For example, the system 400 could include a nutrition-oriented program, and may make suggestions for more nutritious items to a customer who is looking at junk food items (e.g., candy).
Within the scene 600 are defined a customer's field of view 615 and an area 605 outside the customer field of view. In one embodiment, the customer's field of view 615 may be represented by an image captured from a forward-looking camera. While shown as generally rectangular, the customer field of view 615 may have any suitable alternative shape and size. For example, the customer's actual vision may encompass a significantly larger area, but determining the field of view for purposes of the shopping environment may include applying a threshold or weighting scheme that emphasizes areas that are closer to the center of a customer's vision. Of course, data provided by various visual sensors and/or other sensors may be used to make these determinations.
The field of view 615 may include a plurality of fully included items 620, as well as a plurality of partially included items 625. When determining which items to identify as “included” in the field of view, certain embodiments may categorically include or exclude the partially included items 625. An alternative embodiment may rely on image processing to determine whether a partially included item 625 should be identified as included. For example, if the processing cannot recognize the particular item with a certain degree of confidence, the item may be excluded. In another alternative embodiment, partially included items 625 may be included, and the amount of item inclusion (e.g., the percentage of surface area of the item included) may be used to determine a customer focus or calculate a customer interest score, which are discussed further below.
Items that are fully or partially included in the customer field of view 615 may be recognized by performing image processing techniques on images captured by various visual sensors. For example, images that include the items may be compared against stock item images stored in a database or server. To aid image processing, items may also include markers or distinctive symbols, some of which may include item identification data such as barcodes or quick response (QR) codes. Of course, other processing techniques may be employed to recognize a particular item, such as textual recognition, determining the item's similarity to adjacent items, and so forth.
Scene 650 of
In one embodiment, the relative position and/or orientation of portions of the customer's body may be determined. In one embodiment, the position and orientation of the customer's eyes 680 may be determined. For example, eye position within the environment may be determined in Cartesian coordinates (i.e., determining x, y, and z-direction values) and eye orientation may be represented by an angle a defined relative to a reference direction or plane (such as horizontal or an x-y plane corresponding to a particular value of z). In other embodiments, other portions of the customer's body may (also) be used to determine the field of view, such as the position and orientation of the customer's head 665, or of one or both shoulders 670. In other embodiments, the customer's interaction with the shelving unit 603 by extending her arm 675 may also be captured in one or more images, and the direction of the extended arm may be used to determine her field of view.
Of course, embodiments may use combinations of various aspects of the scene to determine the customer's field of view. In some embodiments, the combinations may be weighted; for example, data showing a customer 660 reaching out her arm 675 towards a specific item may be weighted more heavily to determine her field of view than the orientation of her shoulders. In some embodiments, the weights may be dynamically updated based on the customer's shopping behaviors following an estimate of the customer's field of view. For example, if a customer reached for (or selected) an item that was not included in the determined field of view, the system may adjust the relative weighting in order to accurately capture the customer's field of view. This adjustment may include determining correlation values between particular captured aspects of the scene to the selected item; for example, the customer's head may be partly turned towards the selected item, but their eye orientation may generally be more closely tied to the selected item. In some embodiments, the correlation values may be more useful where one or more aspects of the scene cannot be determined (e.g., the system may be unable to determine eye orientation for a customer wearing sunglasses, non-optimal visual sensor positioning, etc.).
Scene 685 of
Certain additional aspects depicted in scene 685 and captured in images may be used to estimate a customer's field of view. In one example, the orientation of customer 660A may be estimated using the relative position of his/her shoulders 670. As shown, a line connecting the two shoulders may be compared to a reference direction or plane (e.g., parallel to the length of shelving unit 603A) and represented by an angle β. In another example, the orientation of customer 660B may be estimated using the orientation of his/her head 665, comparing a direction of the customer's head to a reference direction or plane, which may be represented by an angle γ. Images may also capture a customer 660C interacting with the shelving unit 603B, and the position and/or orientation of the customer's arm 675 may be used to determine the customer's field of view.
Items 715A-D are included on a shelf 710 of a shelving unit 703. The determined field of view 705 of a customer may include different groups of items at different times, and the progression of the customer's field of view over time may help determine which item(s) within the field of view are specifically being focused on by the customer. Generally, the customer's focus on a particular item may indicate that the item is merely attracting the customer's attention, or that the customer is deciding whether or not to purchase the item. Either way, understanding the object of a customer's focus may help retailers or suppliers to improve packaging and placement of items or to influence the customer's shopping experience in real-time.
View 1 illustrates a field of view 705 that includes items 715A, 715B, and a portion of 715C. In one embodiment, items 715A and 715B may be included in a customer focus determination due to the items' full inclusion within the field of view 705. Conversely, item 715C may be excluded from a customer focus for being only partially included in the field of view 705. In an alternative embodiment, all three items may be included by virtue of being at least partially included in the field of view 705. In some embodiments, a customer focus on particular items may be a time-based determination. For example, if the customer's field of view 705 remained relatively steady during a preset amount of time (e.g., 5 or 10 seconds), such that both items 715A, 715B remained within the field of view during this time, the computing system may determine that the customer is focused on items 715A-C. In some embodiments, the particular item(s) must continuously remain in the field of view 705 during the preset amount of time (e.g., remain across several samples of the field of view during this time).
View 2 illustrates a field of view 705 that includes item 715B, and portions of items 715A and 715C. View 2 could represent the same field of view as View 1 at a later time. In some embodiments, the customer's focus may be determined to include item 715B but not 715A. For example, the system could determine that 715B is the lone customer-focused item based on its relatively central position within the field of view 705, or perhaps based on the changes to the field of view from View 1 (i.e., a shift away from previously fully included item 715A). In one embodiment, item 715B must also remain within the field of view 705 for the predetermined amount of time to be considered as a customer-focused item. Of course, other methods are possible to identify one or more relatively prominent items within a field of view to determine a customer's focus, such as determining of the percentage of the item that is included within the field of view, determining the percentage of the field of view occupied by the item, and so forth. In alternative embodiments, the customer-focused items may still include item 715A and/or item 715C.
View 3 illustrates a field of view 705 that includes item 715B, and portions of items 715A and 715C. View 3 differs from View 2 in that the field of view 705 is “closer” to the shelving unit 703 and items 715 in View 3 than in View 2. View 3 could represent the same field of view as View 2 at a later time, e.g., as the customer moves towards the shelving unit 703. Here, the system could determine that 715B is a customer-focused item based on its relatively central position within the field of view 705 and/or based on the changes to the field of view from View 2 (i.e., item 715B occupies an increased percentage of the field of view 705).
View 4 also illustrates a field of view 705 that includes item 715B, and portions of items 715A and 715C. View 4 could represent the same field of view as View 3 at a later time. In View 4, the customer has selected item 715B and is holding the item in his/her hand 725. Here, the system could determine that 715B is a customer-focused item based on its relatively central position within the field of view 705, based on the changes to the field of view from View 3 (e.g., item 715B occupies an increased percentage of the field of view 705), and/or based on the presence of a portion 730 of the customer's hand 725 included in the field of view 705.
In some embodiments, a customer interest score may be determined for items included within the customer's field of view. The customer interest score is generally time-based, and score may be used to timely interject desired information in order to influence a customer's shopping decisions (e.g., whether to buy a particular item) in real-time. In one embodiment, a customer interest score is calculated and updated concurrent with determining a customer focus, and the customer interest score for an item exceeding a threshold value may be used to determine that the customer has focused on the item. The customer interest score may also be adjusted based on various other aspects shown in the Views 1-4, such as position within the field of view (e.g., a central position may mean more interest), percentage occupied within the field of view (e.g., a larger percentage may mean more interest), customer selection and/or manipulation of the item, and so forth. Of course, the calculation of a customer's interest score may reflect their particular shopping behaviors (e.g., a customer is determined to be 95% likely to pick up and manipulate a customer-focused item within their field of view, so the mere fact of picking up an item might not indicate an increased level of customer interest in the item).
After determining that a customer has focused on a particular item, the customer interest score may generally decrease over time as it becomes more likely that the customer will not purchase the particular item. To maintain or improve customer interest, and to thereby increase the likelihood that the customer will purchase the item, the system may provide information about the item at desired times (corresponding to customer interest scores). This may include providing information about the items or alternatives, and/or offering promotional pricing on the item (or an alternative item). In one embodiment, to provide more effective promotions (e.g., more customer-tailored and more timely), and to prevent customer abuse of a time-based promotional pricing scheme, the customer shopping history may include data regarding the number of promotions already offered to the customer, the length of time (or customer interest score) before offering the promotion, the customer utilization rate of the previous promotions, etc.
Presentation 815 indicates a customer preference for a second item 810 to the currently viewed or focused item 805. Presentation 815 generally suggests that the customer may also prefer the second item 810. The customer preference may be determined based on compiled previous shopping data for other customers. The shopping data may include separate statistics for the two items, for example, based on historical purchase data for each item. The shopping data may also include compiled information that directly interrelates the two items. For example, in previous shopping experiences, other customers also focused on item 805 (e.g., based on determined fields of view) but ultimately declined to purchase item 805 in favor of item 810.
Presentation 820 indicates that a promotion is available for the second item 810. Presentation 820 may reflect a sales promotion or prices already in effect (e.g., a reporting function), or the promotion may be dynamically generated to encourage the customer to purchase item 810. For example, manufacturers or stores may wish to promote item 810 over item 805, and the promotion may be presented to the customer in real-time. As discussed above, in some embodiments, the promotion may be determined, as well as the timing of its presentation to the customer, based on a determined level of customer interest in an item.
Presentation 825 indicates that item 805 would have a negative impact on the customer's goals. The presentation 825 may be based on customer profile data such as customer preferences and/or programs. Examples of programs may include fitness, nutrition, or health goals, money management goals, etc. Although not shown here, one embodiment may present the information to the customer upon which the negative impact determination was made. If the system determines that item 805 is incompatible with specified goals or preferences, or that better alternative items exist consistent with the customer's goals/preferences, the system may recommend one or more alternative items for presentation to the customer.
As shown, a single item 805 is included within a customer's field of view 705, and item 810 is located outside the field of view. Of course, similar techniques may be applied to a field of view or a customer focus that includes more than one item. In cases where the item 805 (already included in the field of view 705) is recommended for presentation to the customer, the system may display various indicators that highlight the item 805 to the user. For example, if the field of view 705 represents the customer view through a head-worn computing device such as wearable computing device 500, the system may display colors, shapes, etc. that visually enhance the item 805, or may reduce the visibility of other items in the field of view (e.g., gray out). The system may also display location information of the item 805, for example, by illuminating one or more of the display elements 835a-b nearest to the item 805.
In cases where an item is recommended for presentation that is not disposed within the field of view 705 (e.g., item 810), the system may output directional information from the current field of view 705 to the item. For example, the system may display an arrow 830 overlay indicating the relative direction from the field of view 705 or item 805 to the target item 810. The system may also provide directional information using one or more of the display elements 835, 836. For example, the display elements 835, 836 may be illuminated in sequence from left to right to guide the customers' eye toward item 810. Of course, the various types of information discussed here (e.g., highlighting, location, and directional) may additionally or alternately be provided by non-visual means to the customer, such as audio or haptic outputs.
At block 915, one or more first items that are included within the customer's field of view are identified. In one embodiment, identifying items may include performing image processing on a captured image that reflects the customer's field of view (e.g., from a forward-looking camera). In another embodiment, identifying items may include referencing coordinate data determined for the customer's field of view against a known store layout that includes item location information. In such an embodiment, image processing may not be necessary to identify the one or more first items.
At block 925, at least one second item is selected for presentation to the customer. The selection is based on the identified one or more first items. In one embodiment, at least one of the second item(s) may be included in the identified first items. In other embodiments, each of the second item(s) may be different than the identified first items. The selection may be based on decisional logic for recommending a particular item for sale, based on historical shopping data for the customer (and/or other customers) and/or customer preferences or programs.
At block 935, information related to the at least one second item is presented to the customer. Presenting information may be done in any suitable manner, such as displaying textual or graphical information in a visual overlay, displaying directional information to the customer, or outputting sound or other feedback. The information may be related to the properties of the second item (e.g., price, nutrition information, etc.) or to historical shopping data, promotions, or other customer-specific preferences or programs. The timing of the presentation may be determined based on customer interest scores or other timing considerations that are collectively intended to influence a customer's shopping purchases.
At block 955, a customer interest score is determined for the at least one customer-focused item. The customer interest score may also be time-based, and may be influenced by various characteristics of the interaction of the customer with the item, which may be observed through the determined customer field of view. In one embodiment, the customer interest score may be determined concurrently with determining the customer focus on an item. In one embodiment, the customer-focused item is determined based on the customer interest score for that item reaching a predetermined threshold value. In another embodiment, the customer-focused item is determined by the item's score that exceeds scores for other items within the field of view.
At block 965, the content of the information to present to the customer is determined based on the determined customer interest score. Determining the content may include determining whether to present information for the customer-focused item or for an alternate item, whether or not to present a promotion for the item or the alternate item, determining an amount of the promotion, and so forth.
The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
In the preceding, reference is made to embodiments presented in this disclosure. However, the scope of the present disclosure is not limited to specific described embodiments. Instead, any combination of the following features and elements, whether related to different embodiments or not, is contemplated to implement and practice contemplated embodiments. Furthermore, although embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment is not limiting of the scope of the present disclosure. Thus, the following aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the appended claims except where explicitly recited in a claim(s). Likewise, reference to “the invention” shall not be construed as a generalization of any inventive subject matter disclosed herein and shall not be considered to be an element or limitation of the appended claims except where explicitly recited in a claim(s).
Aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module,” or “system.”
The present disclosure may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.
Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
Embodiments of the disclosure may be provided to end users through a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as a computing capability that provides an abstraction between the computing resource and its underlying technical architecture (e.g., servers, storage, networks), enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Thus, cloud computing allows a user to access virtual computing resources (e.g., storage, data, applications, and even complete virtualized computing systems) in “the cloud,” without regard for the underlying physical systems (or locations of those systems) used to provide the computing resources.
Typically, cloud computing resources are provided to a user on a pay-per-use basis, where users are charged only for the computing resources actually used (e.g., an amount of storage space consumed by a user or a number of virtualized systems instantiated by the user). A user can access any of the resources that reside in the cloud at any time, and from anywhere across the Internet. In context of the present disclosure, a user may access applications (e.g., a retail store app for a mobile computing device) or related data (e.g., compiled shopping data) available in the cloud. Doing so allows a user to access this information from any computing system attached to a network connected to the cloud (e.g., the Internet).
While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
This application claims benefit of U.S. provisional patent application Ser. No. 62/064,323, filed Oct. 15, 2014, entitled “Integrated Shopping Environment,” which is herein incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62064323 | Oct 2014 | US |
