Patent Application
20130243332
The present application claims priority to Patent Application No. 201210068720.2, filed on Mar. 15, 2012, with the State Intellectual Property Office of the People's Republic of China.
Conventionally, when a customer enters a shopping mall, a clerk steps forward to ask the customer what to buy. However, such behavior usually annoys the customer. Thus, a method and system that is able to detect a product/goods that is interesting to a user in a more implicit manner may help a shop clerk to offer the interesting product/goods and associated sale information in a more effective way.
In one embodiment, a method for estimating an object of interest is provided. Visual information of a customer's face is obtained. Pupil location information indicative of at least a location of a pupil of an eye of the customer is determined based on the visual information. A field of view of the customer is determined based on the visual information. Then a focal point of the customer is determined based on the pupil location information, the field of view, and a predetermined focus condition. An object of interest of the customer is estimated based on the focal point. Information associated with the object is provided to the customer.
In another embodiment, an apparatus for estimating an object of interest is provided. The apparatus includes a visual information obtaining module, a pupil location information determining module, a field-of-view determining module, a focal point determining module, and a control module. The visual information obtaining module is configured for obtaining visual information of a customer's face. The pupil location information determining module is configured for determining pupil location information indicative of at least a location of a pupil of an eye of the customer based on the visual information. The field-of-view determining module is configured for determining a field of view of the customer based on the visual information. The focal point determining module is configured for determining a focal point of the customer based on the pupil location information, the field of view of the customer, and a predetermined focus condition. The control module is configured for estimating an object of interest of the customer based on the focal point and providing information associated with the object to the customer.
In yet another embodiment, a system comprising a plurality of sub-systems connected via a network is provided. A first sub-systems of the plurality of sub-systems comprises a visual information obtaining module, a pupil location information determining module, a field-of-view determining module, a focal point determining module, a control module, a collecting module, and a sharing module. The visual information obtaining module is configured for obtaining visual information of a customer's face. The pupil location information determining module is configured for determining pupil location information indicative of at least a location of a pupil of an eye of the customer based on the visual information. The field-of-view determining module is configured for determining a field of view of the customer based on the visual information. The focal point determining module is configured for determining a focal point of the customer based on the pupil location information, the field of view of the customer, and a predetermined focus condition. The control module is configured for estimating an object of interest of the customer based on the focal point and providing information associated with the object to the customer. The collecting module is configured for collecting statistics with respect to the object. The sharing module is configured for facilitating sharing of the statistics with respect to the object among the plurality of sub-systems via the network.
Additional benefits and novel features will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings or may be learned by production or operation of the disclosed embodiments. The benefits of the present embodiments may be realized and attained by practice or use of various aspects of the methodologies, instrumentalities and combinations set forth in the detailed description set forth below.
Features and benefits of embodiments of the claimed subject matter will become apparent as the following detailed description proceeds, and upon reference to the drawings, wherein like numerals depict like parts. These exemplary embodiments are described in detail with reference to the drawings. These embodiments are non-limiting exemplary embodiments, in which like reference numerals represent similar structures throughout the several views of the drawings, and wherein:
Reference will now be made in detail to the embodiments of the present teaching. While the present teaching will be described in conjunction with these embodiments, it will be understood that they are not intended to limit the present teaching to these embodiments. On the contrary, the present teaching is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the present teaching as defined by the appended claims.
Furthermore, in the following detailed description of the present teaching, numerous specific details are set forth in order to provide a thorough understanding of the present teaching. However, it will be recognized by one of ordinary skill in the art that the present teaching may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present teaching.
At 101, visual information of a customer's face may be obtained. The customer may be in a supermarket seeking for a product/goods. The visual information of the customer's face naturally contains visual information of at least an eye of the customer. The visual information may include image information obtained by, e.g., a photo camera or video information obtained by, e.g., a video camera.
At 102, pupil location information may be determined based on the visual information. The pupil location information may indicate at least a location of a pupil of an eye of the customer.
At 103, a field of view of the customer may be determined based on the visual information.
At 104, a focal point of the customer may be determined based on the pupil location information, the field of view of the customer, and a predetermined focus condition. The focal point may be a point, in the field of view, that the customer focuses his/her view at. The predetermined focus condition may be a condition that needs to be met before a focal point of the customer can be affirmed.
At 105, an object may be estimated based on the affirmed focal point. The object may be a product/goods that the customer is interested in.
Then at 106, information associated with the object may be provided to the customer. In case that the object is a product/goods interesting to the customer, the information may include price, features, and/or sales information of the product/goods. The information may be provided by a clerk in person. The information may also be provided by an apparatus via a displaying device in the supermarket or a displaying device with the customer. Thus, the customer can get desirable information associated with a product/goods that may interest him/her, without being disturbed by a clerk.
The method for estimating an object of interest in accordance with an embodiment of the present teaching can be applied in shopping malls, supermarkets, etc. to collect data/information for ordinary customers' interested goods. For example, in accordance with the method shown in
At 201, visual information of a customer's face may be obtained.
In one embodiment, the visual information includes one or more frames of image data. In one embodiment, the visual information can be obtained from a video signal of pupils of eyes of the customer via a camera. In an alternate embodiment, in order to reduce computational complexity and amount of information to be stored, a grayscale camera can be used to capture images for the pupils of the eyes. The visual information can also be obtained from another type of camera. For example, an infrared camera can be used to provide the visual information so as to avoid biases caused by light during an image capturing process. Furthermore, both a grayscale camera and an infrared camera can be used together to obtain more accurate visual information.
At 202, one or more frames of image data may be captured from the visual information.
In one embodiment, the images can be processed using a variety of methods, e.g., image compression, image enhancement, image restoration, image segmentation, etc. By processing the images, valid image data can be obtained for analysis. During the image processing, irrelevant image data can be removed. M frames of image data may be captured from the valid image data. M may be a natural number and be greater than or equal to one. For example, M may be six or seven.
At 203, pupil location information of the customer may be determined based on predetermined pupil-movement sub-areas and the captured frames of image data.
Because of a characteristic of eye, e.g., persistence of vision, an afterimage can be persist for about one twenty-fourth of a second on the retina of an eye. Six or seven frames of image data per second can be captured to analyze a movement of a pupil of an eyeball, in one embodiment. Each frame of image data may correspond to a position of the pupil of the eyeball. These six or seven frames of image data may be in a order according to which they were captured, so as to obtain sequential changes of the positions of the pupil.
In accordance with an embodiment of the present teaching, location information for both pupils of two eyes of a customer may be determined. Because the process for determining pupil location information may be the same for two pupils, one pupil of one eye will be illustrated and described for example below. As shown in
At 204, a range of movement of pupils of the customer may be determined based on the visual information.
Because a radian of rotation of a human eyeball is in a limited range, the range of movement of the pupil in the eyeball can be determined based on the visual information. Thus, a range of movement of both pupils of the customer can be determined accordingly. In one embodiment, the range of movement of the pupil may be indicated by accurate values or values with a permissible deviation.
At 205, a field of view of the customer may be generated based on the range of movement of the pupils. For example, a movement coverage area that covers the range of movement of the pupils can be calculated based on the range of movement. The movement coverage area can be used to define the field of view of the customer.
At 206, an estimated focal point in the field of view of the customer may be determined based on the pupil location information.
In accordance with an embodiment of the present teaching, an estimated focal point is determined based on mapping points in the field of view from both pupils of two eyes of the customer. Because the process for determining mapping points may be the same for two pupils, one pupil of one eye will be illustrated and described for example below. In one embodiment, when a location of the pupil is mapped in the field of view of the customer, the point, representing the location of the pupil, in the field of view is referred to as a mapping point of the location of the pupil (e.g., a point of gaze). In other words, if the location of the pupil is superimposed onto the field of view of the customer, the point, representing the location of the pupil in the field of view of the customer may be the mapping point. The estimated focal point may be determined based on two mapping points of both pupils of the customer. For example, the estimated focal point can be the middle point of the two mapping points in the field of view.
Taking a television (TV) screen as an example of the field of view of an eye ball, a process is described in accordance with
At 207, the estimated focal point may be affirmed as the focal point of the customer if the estimated focal point meets a predetermined focus condition.
In one embodiment, the predetermined focus condition includes a predetermined time condition, and/or a predetermined frequency condition.
The predetermined time condition may be met if the estimated focal point stays in a sub-area in the field of view of the customer for at least a predetermined time period. For example, a predetermined time period may be set to be three seconds. An estimated focal can be affirmed as a focal point if a residence time of the estimated focal point in the field of view of the customer is three seconds or more. In other words, a sub-area in the field of view of the customer may be defined as a focal point if the estimated focal point stays in the sub-area for at least three seconds.
The predetermined frequency condition may be met if the estimated focal point falls in a sub-area at a frequency that is greater than a predetermined frequency threshold. For example, a predetermined frequency threshold may be set to be two times/min. An estimated focal point can be affirmed as a focal point if the estimated focal point in the field of view of the customer appears more than two times in one minute. In other words, a sub-area in the field of view of the customer can be defined as a focal point if the estimated focal point falls in the sub-area at a frequency that is greater than two times per minute.
At 208, an object of interest of the customer may be estimated based on the affirmed focal point. In addition, information associated with the object may be provided to the customer (not shown in
The method for estimating an object of interest, in accordance with an embodiment of the present teaching can be applied in many places, e.g., shopping malls, supermarkets, etc. It is important and desirable to understand demands of customers in many places. The conventional way that a clerk steps forward to ask a customer is often considered as a disturb to the customer. Therefore, an eyeball movement tracking system implementing an exemplary method of the present teaching can be applied in the shopping malls to avoid disturbing the customers, and also can obtain information for the customers' shopping demands conveniently and accurately. Furthermore, the method of the present teaching can also be used to provide goods information corresponding to the focal point, e.g., styles, prices, discounts of the goods, and information about whether there are updated and new arrivals of the goods, to registered users. The registered users may be, e.g., registered customers of a supermarket, a shopping mall, or other places. If a customer is not a registered user, the eyeball movement tracking system may capture and collect information with respect to the focal point of the customer.
In one embodiment, if a registered user (or a registered account) is bound to a specific terminal, the related information for the goods may be transmitted to the specific terminal. In one embodiment, the specific terminal may be a customer-held terminal, e.g., a portable computer, mobile phone, or other receiving devices.
In one embodiment, a customer can become a registered user of a store by downloading related application software provided by the store to a customer-held terminal and registering to be a member of a service that provides goods information. It can be determined that whether a customer is a registered user by a comparison of obtained visual information of the customer with stored visual information of registered users, or by recognizing an identity of the customer using an ID (identification) device.
The visual information obtaining module 510 may obtain visual information of a customer's face, which naturally includes one or two eyeballs.
The pupil location information determining module 520 may determine pupil location information indicative of at least a location of a pupil of an eyeball based on the visual information.
The field-of-view determining module 530 may determine a field of view of the customer based on the visual information.
The focal point determining module 540 may determine a focal point based on the pupil location information, the field of view of the customer, and a predetermined focus condition.
The control module 550 may estimate an object of interest of the customer based on the focal point and provide information associated with the object to the customer.
The storage 560 may store the predetermined focus condition and/or the information associated with the object.
The predetermined focus condition may include a predetermined time condition, and/or a predetermined frequency condition.
The pupil location information determining module 520 may further include an image capturing unit 621 that captures one or more frames of image data from the visual information, and include a pupil location information determining unit 622 that determines pupil location information for at least a pupil of an eyeball based on predetermined pupil-movement sub-areas and the frames of image data. In one embodiment, the frames of image data includes at least six frames of image data.
The field-of-view determining module 530 may further include a range determining unit 631 that determines a range of movement of the pupils based on the visual information, and include a field-of-view generating unit 632 that generates data indicative of the field of view of the customer based on the range of the movement of the pupils.
The focal point determining module 540 may further include a mapping unit 641 that determines an estimated focal point in the field of view of the customer based on the pupil location information, and include a focal point affirming unit 642 that affirms an estimated focal point as a focal point of the customer if the estimated focal point meets the above mentioned predetermined focus condition.
The apparatus 500, in the exemplary embodiment shown in
The collecting module 670 may be configured for collecting statistics with respect to the object of interest.
The estimating module 680 may be configured for estimating a level of interest of customers with respect to the object based on the statistics with respect to the object.
The sharing module 690 may be configured for sharing the statistics with respect to the object among multiple entities for enhancing a supply of the object. When the object is a goods interesting to the customer, the multiple entities may include chain stores of a supermarket selling the goods, manufacturers of the goods, suppliers of the goods, and/or distributors of the goods. The multiple entities may be connected via a local area network or Internet.
In the exemplary embodiment, at least one of the multiple sub-systems includes all modules in the apparatus 500, as shown in
In addition, the sub-system 701 may facilitate sharing of the statistics with respect to the object among the multiple sub-systems 701, 702, 703, 704 in the system 700, via the network 710. In one embodiment, the system 700 further comprises a server 720 connected to the network 710. The server 720 may be configured for controlling the sharing of the statistics among the sub-systems 701, 702, 703, 704 in the system 700. For example, the server 720 may receive the statistics with respect to the object from the sub-system 701 and provide the statistics to other sub-systems 702, 703, 704 in the system 700.
The network 710 may be a local area network or Internet. Each of the sub-systems 701, 702, 703, 704 may be located in an entity that is associated with the object. When the object is a goods interesting to the customer, statistics of the goods can be shared among the entities for enhancing a supply of the goods.
The computer 800, for example, includes COM ports 802 connected to and from a network connected thereto to facilitate data communications. The computer 800 also includes a central processing unit (CPU) 804, in the form of one or more processors, for executing program instructions. The exemplary computer platform includes an internal communication bus 806, program storage and data storage of different forms, e.g., disk 808, read only memory (ROM) 810, or random access memory (RAM) 812, for various data files to be processed and/or communicated by the computer, as well as possibly program instructions to be executed by the CPU. The computer 800 also includes an I/O component 814, supporting input/output flows between the computer and other components therein such as user interface elements 816. The computer 800 may also receive programming and data via network communications.
Hence, aspects of the method of estimating an object of interest, as outlined above, may be embodied in programming. Program aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of executable code and/or associated data that is carried on or embodied in a type of machine readable medium. Tangible non-transitory “storage” type media include any or all of the memory or other storage for the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide storage at any time for the software programming.
All or portions of the software may at times be communicated through a network such as the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another. Thus, another type of media that may bear the software elements includes optical, electrical, and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software. As used herein, unless restricted to tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.
Hence, a machine readable medium may take many forms, including but not limited to, a tangible storage medium, a carrier wave medium or physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, which may be used to implement the system or any of its components as shown in the drawings. Volatile storage media include dynamic memory, such as a main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that form a bus within a computer system. Carrier-wave transmission media can take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer can read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.
Those skilled in the art will recognize that the present teachings are amenable to a variety of modifications and/or enhancements. For example, although the implementation of various components described above may be embodied in a hardware device, it can also be implemented as a software only solution—e.g., an installation on an existing server. In addition, the units of the host and the client nodes as disclosed herein can be implemented as a firmware, firmware/software combination, firmware/hardware combination, or a hardware/firmware/software combination.
While the foregoing description and drawings represent embodiments of the present teaching, it will be understood that various additions, modifications and substitutions may be made therein without departing from the spirit and scope of the principles of the present teaching as defined in the accompanying claims. One skilled in the art will appreciate that the teaching may be used with many modifications of form, structure, arrangement, proportions, materials, elements, and components and otherwise, used in the practice of the teaching, which are particularly adapted to specific environments and operative requirements without departing from the principles of the present teaching. The presently disclosed embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the teaching being indicated by the appended claims and their legal equivalents, and not limited to the foregoing description.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201210068720.2 | Mar 2012 | CN | national |
