This application relates generally to analysis of mental states and more particularly to sharing affect data across a social network.
People spend a tremendous amount of time on the internet, much of that including the viewing and interacting with web pages including pages for social networks. The evaluation of mental states is key to understanding individuals and the way in which they react to the world around them and this world more and more include the virtual world. Mental states run a broad gamut from happiness to sadness, from contentedness to worry, from excitement to calmness, as well as numerous others. These mental states are experienced in response to everyday events such as frustration during a traffic jam, boredom while standing in line, impatience while waiting for a cup of coffee, and even as people interact with their computers and the internet. Individuals may become rather perceptive and empathetic based on evaluating and understanding others' mental states but automated evaluation of mental states is far more challenging. An empathetic person may perceive another's being anxious or joyful and respond accordingly. The ability and means by which one person perceives another's emotional state may be quite difficult to summarize and has often been communicated as having a “gut feel.”
Many mental states, such as confusion, concentration, and worry, may be identified to aid in the understanding of an individual or group of people. People can collectively respond with fear or anxiety, such as after witnessing a catastrophe. Likewise, people can collectively respond with happy enthusiasm, such as when their sports team obtains a victory. Certain facial expressions and head gestures may be used to identify a mental state that a person is experiencing. Limited automation has been performed in the evaluation of mental states based on facial expressions. Certain physiological conditions may provide telling indications of a person's state of mind and have been used in a crude fashion as in polygraph tests.
Analysis of people, as they interact with the internet and various media, may be performed by gathering mental states through evaluation of facial expressions, head gestures, and physiological conditions. Some of the mental state analysis may then be shared across a social network. A computer implemented method for communicating mental states is disclosed comprising: collecting mental state data of an individual; analyzing the mental state data to produce mental state information; and sharing the mental state information across a social network. The method may further comprise electing, by the individual, to share the mental state information. The method may further comprise presenting the mental state information to the individual, prior to the electing. The mental state data may be collected over a period of time and the mental state information that is shared is a reflection of a mood for the individual. The mood may include one of a group comprising frustration, confusion, disappointment, hesitation, cognitive overload, focusing, being engaged, attending, boredom, exploration, confidence, trust, delight, and satisfaction. The sharing may include posting mental state information to a social network web page. The method may further comprise uploading the mental state information to a server. The method may further comprise distributing the mental state information across a computer network. The mental state data may include one of a group comprising physiological data, facial data, and actigraphy data. A webcam may be used to capture one or more of the facial data and the physiological data. The facial data may include information on one or more of a group comprising facial expressions, action units, head gestures, smiles, brow furrows, squints, lowered eyebrows, raised eyebrows, and attention. The physiological data may include one or more of electrodermal activity, heart rate, heart rate variability, skin temperature, and respiration. The method may further comprise inferring of mental states based on the mental state data which was collected. The method may further comprise identifying similar mental states within the social network. The mental states may include one of a group comprising frustration, confusion, disappointment, hesitation, cognitive overload, focusing, being engaged, attending, boredom, exploration, confidence, trust, delight, and satisfaction. The method may further comprise communicating an image of the individual with the mental state information that is being shared. The image of the individual may be from a peak time of mental state activity. The image may include a video. The method may further comprise restricting distribution of the mental state information to a subset of the social network. The method may further comprise sharing aggregated mental state information across the social network. The mental state data may be collected as the individual interacts with a web-enabled application. The web-enabled application may be one of a group comprising a landing page, a checkout page, a webpage, a website, a video on the web-enabled application, a game on the web-enabled application, a trailer, a movie, an advertisement, and a virtual world. The method may further comprise forwarding a reference to the web-enabled application as a part of the sharing of the mental state information. The reference may include a URL and a timestamp. The forwarding may include an image of material from the web-enabled application. The forwarding may include a video of material from the web-enabled application. The sharing may be part of a rating system for the web-enabled application. The mental state data may be collected using a biosensor.
In some embodiments, a computer program product embodied in a non-transitory computer readable medium for communicating mental states may comprise: code for collecting mental state data of an individual; code for analyzing the mental state data to produce mental state information; code for electing, by the individual, to share the mental state information; and code for sharing the mental state information across a social network. In embodiments, a system for sharing mental states may comprise: a memory for storing instructions; one or more processors attached to the memory wherein the one or more processors are configured to: collect mental state data of an individual; analyze the mental state data to produce mental state information; receive an instruction, from the individual, to elect to share the mental state information; and share the mental state information across a social network.
In some embodiments, a computer implemented method for communicating mental states comprises: receiving mental state information on of an individual; inferring mental states for the individual based on the mental state information which was received; and sharing the mental states which were inferred across a social network.
Various features, aspects, and advantages of numerous embodiments will become more apparent from the following description.
The following detailed description of certain embodiments may be understood by reference to the following figures wherein:
The present disclosure provides a description of various methods and systems for analyzing people's mental states as they interact with websites, web-enabled applications, and/or other features on the internet with the result being shared across a social network. Social networking has become more and more a part of everyday life with a society which constantly connected through the Internet. Communication is accomplished by email, postings, texting, short messages, and the like but communication of emotions has remained a challenge. By performing mental state analysis and then communicating those mental states across a social network, virtual communication becomes much more attuned to the person. The communication is not limited to explicit postings and instead allows communication of emotion. Mental states may include emotional states and/or cognitive states. Examples of emotional states include happiness or sadness. Examples of cognitive states include concentration or confusion. Observing, capturing, and analyzing these mental states can yield significant information about people's reactions that far exceed current capabilities in website type analytics.
A challenge solved by this disclosure is the collection and analysis of mental states of an individual to produce mental state information that may be shared across a social network. Mental state data may be collected from an individual while performing specific tasks or over longer periods of time. Mental state data may include physiological data from sensors, facial data from a webcam, or actigraphy data. The mental state data may be analyzed to create mental state information. Mental state information may include moods, other mental states, mental state data or mental state information derived or inferred from mental state data. Mental states of the individual may include frustration, confusion, disappointment, hesitation, cognitive overload, focusing, being engaged, attending, boredom, exploration, confidence, trust, delight, and satisfaction or other emotions or cognitive states. Mental state information may relate to a specific stimulus, such as reacting to a web-enabled application, or may be a mood, which may relate to a longer period of time and may indicate, for example, a mental state for a day.
The individual may be given an opportunity to share their mental state with others. If the individual opts-in for sharing, their mental state may be shared over a social network. The mental state may be shared over a social network by posting mood information on a social media or social network web page. The mental state shared may be an overall mood or may be a reaction to a specific stimulus. If the mental state is a reaction to a specific stimulus, a reference to the stimulus, such as a web-enabled application, may be shared. The reference may include a uniform reference locator (URL) and/or a timestamp. An image of the individual corresponding to their mood may be posted along with the mental state. Other individuals on the social network having a similar mental state may be identified to the individual. And in some cases the mental states of an individual's contacts on the social network may be aggregated and shared on the social network.
A record button 120 may be included to record the webcam view 110. The record button 120 may be part of the “opting in” by the individual in the webcam view 110 where permission is obtained for observing mental state information and sharing this information. The record button 120 may be moused over to explain the purpose of the record button 120. The record button 120 may be clicked in order to start the recording. The record button may be clicked again to stop the recording. In some embodiments, recording may be accomplished based on sensing context. Recording can automatically begin as viewing or interaction begins with a specific web-enabled application. Recording can automatically end at a specific point in time or as a web-enabled application reaches it's ending point. One such example is a series of video trailers that may be viewed. Recording of the webcam view can begin and end with the start and termination of each video trailer. In embodiments, permission may be granted for recording of the webcam view for certain contexts of operation. Further, the context may be recorded as well as the webcam view.
A chart button 130 may be used to display analytics of the information collected while the webcam was recording. The chart button 130 may be moused over to explain the purpose of the button. The chart button 130 may be clicked on to display a chart such as that shown in
The flow 300 may continue with analyzing the mental state data 320 to produce mental state information. While mental state data may be raw data such as heart rate, mental state information may include information derived from the raw data. The mental state information may include the mental state data. The mental state information may include valence and arousal. The mental state information may include the mental states experienced by the individual. Some embodiments may include inferring of mental states based on the mental state data which was collected.
The flow 300 may continue with uploading mental state information 330 to a server. The server may be remote from the user and may be a host to data used by a social network, but in other embodiments the server may be separate from the social network's computer system and be used for storage for mental state information as well as other functionality. In some cases, an image may be communicated 340 to the server with the mental state information. The image may be of the individual as the mental state data was being collected and may be representative of the mental state information. In other embodiments, the image may be captured or identified in advance to represent a particular mental state. The flow 300 may continue with presenting the mental state information to the individual 350, prior to electing to share. Some embodiments may allow the user to make the election before the presenting. In some embodiments the mental state data, the mental state information, or a subset of the mental state information may be presented to the individual. In some embodiments there may be no presentation. The mental state information may be presented to the individual in various ways such as a textual description of a mood, an image obtained of the individual or from the individual, a graph such as shown in
The flow 300 may continue with electing, by the individual, to share the mental state information 360 or mental states. The individual may choose to restrict distribution 362 of the mental state information. The individual may choose to share all or a portion of the mental state data and mental state information. The individual may choose to share with an individual, a group of people, or across a social network, such as restricting distribution of the mental state information to a subset of a social network. In embodiments, mental state information may be shared to others whom the network may recommend. In some embodiments, a reference to a web-enabled application may be forwarded 364 to the selected group or subgroup. In some embodiments, the forwarding is accomplished by selecting a “like” type button on a web page. The reference may include information on a video, trailer, e-book, web site, movie, advertisement, television show, streamed video clip, video game, computer game, or the like. The reference may include a timestamp, page number, web page URL, or the like to identify a portion of the reference. The forwarding may include a Twitter™ message, text, SMS, or the like. A URL or short-URL may be included when the reference is forwarded 364. The flow 300 may continue with sharing mental state information 370. The sharing may include transmission of data from an individual's client computer to a server which retains mental state information. The sharing may include a web link, a web-enabled application reference, or a web-enabled application. The mental state information may be communicated from the server to an individual 380. Alternatively, there may be peer-to-peer sharing of mental state information from a first individual to a second individual. Some embodiments may include sharing the mental state information across a social network 382. Mental states may be communicated via Facebook™, LinkedIn™, MySpace™, Twitter™, Google+™, or other social networking site.
The flow 400 continues with inferring mental states 440 for the individual based on the mental state information which was received. The mental states that may be inferred include frustration, confusion, disappointment, hesitation, cognitive overload, focusing, being engaged, attending, boredom, exploration, confidence, trust, delight, and satisfaction. In some embodiments, collective mental states may be inferred for a group of people. The flow 400 continues with sharing the mental states which were inferred across a social network 450. Some embodiments may include identifying similar mental states within the social network 452. The group of people that may be searched to identify similar mental states may vary according to the embodiment. Some embodiments may only search an individual's direct contact list while others may search an extended contact list such as including the contacts of the individual's contacts, or an even more extended group going out several levels of contact's contacts. In other embodiments, only a group that has been specifically created to share mental state information may be searched while other embodiments may search outside of the individual's extended network to help identify people that may be interesting to the individual and may be potential new contacts.
Multiple individuals can have their mental states collected and their mental state information distributed across a computer network 460 for various purposes. These mental states can be aggregated together and the combined mental state evaluation can be posted or propagated to others. A webmaster may collect affect data and mental state information. This data and/or information can be tagged to the website controlled by the webmaster and therefore the mental states can be associated with the web-enabled application. Further, aggregated responses can be used to evaluate the viral potential of a web-enabled application, such as a video or game. The aggregation may take various forms in various embodiments but examples may include creating an aggregate mood of an individual's contact on a social network, creating aggregate mental state information of the people that have viewed a movie trailer, tabulating a percentage of a particular group having a particular mental state, or any other method of aggregating mental state information. Flow 400 may finish by sharing aggregated mental state information across a social network 470.
Analysis of action units, gestures, and mental states 550 may be accomplished using the captured images of the person 520. The action units may be used to identify smiles, frowns, and other facial indicators of mental states. The gestures, including head gestures, may indicate interest or curiosity. For example, a head gesture of moving toward the electronic display 510 may indicate increased interest or a desire for clarification. Based on the captured images, analysis of physiological data may be performed. Respiration, heart rate, heart rate variability, perspiration, temperature, and other physiological indicators of mental state can be observed by analyzing the images. So in various embodiments, a webcam is used to capture one or more of the facial data and the physiological data.
Mental states may be inferred based on physiological data, such as physiological data from the sensor 612. Mental states may also be inferred based on facial expressions and head gestures observed by a webcam or a combination of data from the webcam along with data from the sensor 612. The mental states may be analyzed based on arousal and valence. Arousal can range from being highly activated, such as when someone is agitated, to being entirely passive, such as when someone is bored. Valence can range from being very positive, such as when someone is happy, to being very negative, such when someone is angry. Physiological data may include electrodermal activity (EDA) or skin conductance or galvanic skin response (GSR), accelerometer readings, skin temperature, heart rate, heart rate variability, and other types of analysis of a human being. It will be understood that both here and elsewhere in this document, physiological information can be obtained either by biosensor 612 or by facial observation. Facial data may include facial actions and head gestures used to infer mental states. Further, the data may include information on hand gestures or body language and body movements such as visible fidgets. In some embodiments these movements may be captured by cameras or by sensor readings. Facial data may include the tilting the head to the side, leaning forward, a smile, a frown, as well as many other gestures or expressions.
Electrodermal activity may be collected in some embodiments and may be collected continuously, every second, four times per second, eight times per second, 32 times per second, or on some other periodic basis. The electrodermal activity may be recorded. The recording may be to a disk, a tape, onto flash memory, into a computer system, or streamed to a server. The electrodermal activity may be analyzed 630 to indicate arousal, excitement, boredom, or other mental states based on changes in skin conductance. Skin temperature may be collected on a periodic basis and may be recorded. The skin temperature may be analyzed 632 and may indicate arousal, excitement, boredom, or other mental states based on changes in skin temperature. The heart rate may be collected and recorded. The heart rate may be analyzed 634 and a high heart rate may indicate excitement, arousal or other mental states. Accelerometer data may be collected and indicate one, two, or three dimensions of motion. The accelerometer data may be recorded. The accelerometer data may be used to create an actigraph showing an individual's activity level over time. The accelerometer data may be analyzed 636 and may indicate a sleep pattern, a state of high activity, a state of lethargy, or other state based on accelerometer data. The various data collected by the biosensor 612 may be used along with the facial data captured by the webcam.
Some embodiments may include the ability for a user to select a particular type of mental state information for display using various buttons or other selection methods. In the example shown, the smile mental state information is shown as the user may have previously selected the Smile button 840. Other types of mental state information that may be available for user selection in various embodiments may include the Lowered Eyebrows button 842, Eyebrow Raise button 844, Attention button 846, Valence Score button 848 or other types of mental state information, depending on the embodiment. The mental state information displayed may be based on physiological data, facial data, and actigraphy data. An Overview button 849 may be available to allow a user to show graphs of the multiple types of mental state information simultaneously.
Because the Smile option 840 has been selected in the example shown, smile graph 850 may be shown against a baseline 852 showing the aggregated smile mental state information of the plurality of individuals from whom mental state data was collected for the rendering 810. Male smile graph 854 and female smile graph 856 may be shown so that the visual representation displays the aggregated mental state information on a demographic basis. The various demographic based graphs may be indicated using various line types as shown or may be indicated using color or other method of differentiation. A slider 858 may allow a user to select a particular time of the timeline and show the value of the chosen mental state for that particular time. The slider may show the same line type or color as the demographic group whose value is shown.
Various types of demographic based mental state information may be selected using the demographic button 860 in some embodiments. Such demographics may include gender, age, race, income level, or any other type of demographic including dividing the respondents into those respondents that had a higher reaction from those with lower reactions. A graph legend 862 may be displayed indicating the various demographic groups, the line type or color for each group, the percentage of total respondents and or absolute number of respondents for each group, and/or other information about the demographic groups. The mental state information may be aggregated according to the demographic type selected.
The social network content 1000 may include mental state information section 1050. The mental state information section 1050 may allow for posting mental state information to a social network web page. It may include mental state information that has been shared by the individual or may include mental state information that has been captured but not yet shared, depending on the embodiment. In at least one embodiment, a mental state graph 1052 may be displayed to the individual showing their mental state information while viewing a web-enabled application, such as the graph of
Some embodiments may include a mental state score 1056. In some embodiments, the mental state data is collected over a period of time and the mental state information that is shared is a reflection of a mood for the individual in a mental state score 1056. The mental state score may be a number, a sliding scale, a colored scale, various icons or images representing moods or any other type of representation. In some embodiments, the mental state score 1056 may emulate a “mood ring” as was popular back in the 1970's. Various moods may be represented, including, but not limited to, frustration, confusion, disappointment, hesitation, cognitive overload, focusing, being engaged, attending, boredom, exploration, confidence, trust, delight, and satisfaction.
Some embodiments may include a section for aggregated mental states of friends 1058. This section may include an aggregated mood of those friends shown in the friends section 1030 that have opted to share their mental state information. Other embodiments may include aggregated mental states of those friends that have viewed the same web-enabled application as the individual and may allow the individual to compare their mental state information in the mental state graph 1052 to their friends' mental state information 1058. Other embodiments may display various aggregations of different groups.
The server 1130 may have an internet connection for receiving mental states or collected mental state information 1131 and have a memory 1134 which stores instructions and one or more processors 1132 attached to the memory 1134 to execute instructions. The server 1130 may receive mental state information collected from a plurality of people as they interact with a rendering from the client computer 1120 or computers, and may analyze the mental state data to produce mental state information. The server 1130 may also aggregate mental state information on the plurality of people who interact with the rendering. The server 1130 may also associate the aggregated mental state information with the rendering and also with the collection of norms for the context being measured. In some embodiments the server 1130 may also allow user to view and evaluate the mental state information that is associated with the rendering, but in other embodiments, the server 1130 may send the aggregated mental state information 1141 to a social network 1140 to be shared, distributing the mental state information across a computer network. This may be done to share the mental state information across a social network. In some embodiments the social network 1140 may run on the server 1130.
Each of the above methods may be executed on one or more processors on one or more computer systems. Embodiments may include various forms of distributed computing, client/server computing, and cloud based computing. Further, it will be understood that for each flow chart in this disclosure, the depicted steps or boxes are provided for purposes of illustration and explanation only. The steps may be modified, omitted, or re-ordered and other steps may be added without departing from the scope of this disclosure. Further, each step may contain one or more sub-steps. While the foregoing drawings and description set forth functional aspects of the disclosed systems, no particular arrangement of software and/or hardware for implementing these functional aspects should be inferred from these descriptions unless explicitly stated or otherwise clear from the context. All such arrangements of software and/or hardware are intended to fall within the scope of this disclosure.
The block diagrams and flowchart illustrations depict methods, apparatus, systems, and computer program products. Each element of the block diagrams and flowchart illustrations, as well as each respective combination of elements in the block diagrams and flowchart illustrations, illustrates a function, step or group of steps of the methods, apparatus, systems, computer program products and/or computer-implemented methods. Any and all such functions may be implemented by computer program instructions, by special-purpose hardware-based computer systems, by combinations of special purpose hardware and computer instructions, by combinations of general purpose hardware and computer instructions, and so on. Any and all of which may be generally referred to herein as a “circuit,” “module,” or “system.”
A programmable apparatus which executes any of the above mentioned computer program products or computer implemented methods may include one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, programmable devices, programmable gate arrays, programmable array logic, memory devices, application specific integrated circuits, or the like. Each may be suitably employed or configured to process computer program instructions, execute computer logic, store computer data, and so on.
It will be understood that a computer may include a computer program product from a computer-readable storage medium and that this medium may be internal or external, removable and replaceable, or fixed. In addition, a computer may include a Basic Input/Output System (BIOS), firmware, an operating system, a database, or the like that may include, interface with, or support the software and hardware described herein.
Embodiments of the present invention are not limited to applications involving conventional computer programs or programmable apparatus that run them. It is contemplated, for example, that embodiments of the presently claimed invention could include an optical computer, quantum computer, analog computer, or the like. A computer program may be loaded onto a computer to produce a particular machine that may perform any and all of the depicted functions. This particular machine provides a means for carrying out any and all of the depicted functions.
Any combination of one or more computer readable media may be utilized. The computer readable medium may be a non-transitory computer readable medium for storage. A computer readable storage medium may be electronic, magnetic, optical, electromagnetic, infrared, semiconductor, or any suitable combination of the foregoing. Further computer readable storage medium examples may include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, Flash, MRAM, FeRAM, or phase change memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
It will be appreciated that computer program instructions may include computer executable code. A variety of languages for expressing computer program instructions may include without limitation C, C++, Java, JavaScript™, ActionScript™, assembly language, Lisp, Perl, Tcl, Python, Ruby, hardware description languages, database programming languages, functional programming languages, imperative programming languages, and so on. In embodiments, computer program instructions may be stored, compiled, or interpreted to run on a computer, a programmable data processing apparatus, a heterogeneous combination of processors or processor architectures, and so on. Without limitation, embodiments of the present invention may take the form of web-based computer software, which includes client/server software, software-as-a-service, peer-to-peer software, or the like.
In embodiments, a computer may enable execution of computer program instructions including multiple programs or threads. The multiple programs or threads may be processed more or less simultaneously to enhance utilization of the processor and to facilitate substantially simultaneous functions. By way of implementation, any and all methods, program codes, program instructions, and the like described herein may be implemented in one or more thread. Each thread may spawn other threads, which may themselves have priorities associated with them. In some embodiments, a computer may process these threads based on priority or other order.
Unless explicitly stated or otherwise clear from the context, the verbs “execute” and “process” may be used interchangeably to indicate execute, process, interpret, compile, assemble, link, load, or a combination of the foregoing. Therefore, embodiments that execute or process computer program instructions, computer-executable code, or the like may act upon the instructions or code in any and all of the ways described. Further, the method steps shown are intended to include any suitable method of causing one or more parties or entities to perform the steps. The parties performing a step, or portion of a step, need not be located within a particular geographic location or country boundary. For instance, if an entity located within the United States causes a method step, or portion thereof, to be performed outside of the United States then the method is considered to be performed in the United States by virtue of the entity causing the step to be performed.
While the invention has been disclosed in connection with preferred embodiments shown and described in detail, various modifications and improvements thereon will become apparent to those skilled in the art. Accordingly, the spirit and scope of the present invention is not to be limited by the foregoing examples, but is to be understood in the broadest sense allowable by law.
This application claims the benefit of U.S. provisional patent applications “Sharing Affect Data Across a Social Network” Ser. No. 61/414,451, filed Nov. 17, 2010, “Using Affect Within a Gaming Context” Ser. No. 61/439,913, filed Feb. 6, 2011, “Recommendation and Visualization of Affect Responses to Videos” Ser. No. 61/447,089, filed Feb. 27, 2011, “Video Ranking Based on Affect” Ser. No. 61/447,464, filed Feb. 28, 2011, “Baseline Face Analysis” Ser. No. 61/467,209, filed Mar. 24, 2011, and “Mental State Analysis of Voters” Ser. No. 61/549,560, filed Oct. 20, 2011. Each of the foregoing applications is hereby incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
61414451 | Nov 2010 | US | |
61439913 | Feb 2011 | US | |
61447089 | Feb 2011 | US | |
61447464 | Feb 2011 | US | |
61467209 | Mar 2011 | US | |
61549560 | Oct 2011 | US |