The present invention is generally directed to systems and methods for providing haptic effects in response to deformation of a flexible cover for an electronic device.
Handheld electronic devices, such as smartphones, tablets, and e-book readers, are frequently used with covers that protect their screens and can be peeled off while in use. It is increasingly common for such devices to be able to detect the state of the cover and take an action based on the detected state. For example, many screen covers cause the display screen of the electronic device to turn off when the cover is closed. It is expected that covers for electronic devices will eventually be instrumented so that gestures, such as bending the cover or one of the corners of the cover, or peeling the cover off of the electronic device, can be detected and used for user input.
It is desirable to improve a user's interactions with a cover of an electronic device by providing additional information to the user.
According to an aspect of the invention, there is provided a system that includes an electronic device comprising a display screen, a cover configured to cover the display screen, a sensor configured to sense an input gesture that includes deformation and/or movement of the cover relative to the electronic device, and a processor configured to determine an action for the electronic device to perform based on the input gesture, to determine a haptic effect to generate based on the input gesture and/or the action for the electronic device to perform, and to initiate the action. The system also includes a haptic output device configured to generate the haptic effect.
In an embodiment, the sensor is selected from the group consisting of: a bend sensor, a light sensor, a pressure sensor, a contact sensor, a potentiometer, an angular sensor, an angular velocity sensor, an accelerometer, and a magnetic sensor.
In an embodiment, the haptic output device includes an actuator configured to generate vibrations.
In an embodiment, the cover includes the actuator.
In an embodiment, the system includes a hinge that connects the cover to the electronic device, and the hinge includes the actuator.
In an embodiment, the haptic output device is embedded in the cover and includes an electro-rheological or magneto-rheological fluid to control a stiffness of the cover.
In an embodiment, the haptic output device includes an electrostatic friction device configured to generate a friction effect at a surface of the cover. In an embodiment, the surface of the cover faces the electronic device when the cover covers the display screen.
According to an aspect of the invention, there is provided a method that includes sensing an input gesture comprising deformation and/or movement of a cover for an electronic device relative to the electronic device with a sensor, determining an action for the electronic device to perform based on the input gesture with a processor, determining a haptic effect to generate based on the input gesture and/or the action for the electronic device to perform with the processor, initiating the action for the electronic device to perform with the processor, and generating the haptic effect with a haptic output device during and/or after the sensing of the input gesture.
In an embodiment, the haptic effect is generated during the sensing of the input gesture.
In an embodiment, the haptic effect is generated after the sensing of the input gesture.
In an embodiment, the haptic effect is generated after the action is completed to confirm completion of the action.
In an embodiment, the haptic effect includes resisting peeling of the cover away from the electronic device.
In an embodiment, the haptic effect includes creating a texture when the cover is peeled away from the electronic device.
In an embodiment, the haptic effect includes changing a stiffness of the cover.
In an embodiment, the haptic effect includes a tactile sensation on a surface of the cover facing the electronic device when the cover covers the electronic device.
In an embodiment, the haptic effect includes a tactile sensation on a top surface of the cover opposite a surface facing the electronic device when the cover covers the electronic device.
In an embodiment, the method includes determining whether the action can be initiated, and the haptic effect includes resisting the deformation when the action cannot be initiated.
In an embodiment, the haptic effect includes actuating at least one hinge located along at least one crease in the cover to fold the cover at the crease.
In an embodiment, the haptic effect includes restricting movement of the cover to a predetermined angle relative to the electronic device based on the action.
According to an aspect of the invention, there is provided a cover for an electronic device. The cover includes a sensor configured to sense an input gesture comprising deformation and/or movement of the cover relative to the electronic device, and a haptic output device configured to generate a haptic effect based on the input gesture.
In an embodiment, the cover includes a processor configured to determine an action for the electronic device to perform based on the input gesture, to determine the haptic effect to generate based on the input gesture and/or the action for the electronic device to perform, and to initiate the action.
These and other aspects, features, and characteristics of the present invention, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention. As used in the specification and in the claims, the singular form of “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise.
The components of the following Figures are illustrated to emphasize the general principles of the present disclosure and are not necessarily drawn to scale. Reference characters designating corresponding components are repeated as necessary throughout the Figures for the sake of consistency and clarity.
The cover 120 may be configured to protect the electronic device 110 when in the closed position. In an embodiment, the cover 120 may be configured to receive the electronic device 110 so when the electronic device 110 sits inside the cover 120 and the cover 120 is in the closed position, the cover 120 substantially or completely surrounds the electronic device 110. In an embodiment, the cover 120 may include at least one hinge 122 that is configured to releasably connect or attach the cover 120 to the electronic device 110 via a magnet or other mechanism that allows the cover 120 to be releasably connected to the electronic device 110. The cover 120 may be transparent or partially transparent such that at least a portion of the cover 120 allows the display screen 112 to display images to the user when the cover 120 is in a closed position. In an embodiment, the cover 120 may be a transparent or opaque cover that is used as an interactive layer for the user to provide input gestures to the electronic device 110. In an embodiment, the cover 120 is flexible so that the user may bend, fold, twist, roll or otherwise deform the cover 120 and a manner that the cover 120 may be used as an interactive interface for applications running on the underlying electronic device 110.
Upon detection of an input gesture in the form of deformation of the cover 120, the electronic device 110 may then perform an action based on the input gesture that has been detected, such as turning on the display screen 112 or changing the information displayed on the display screen 112. The action to be performed may be dependent on the context, and may be programmed as part of an application or the operating system of the electronic device 110.
As schematically illustrated in
The processor 130 may be a general-purpose or specific-purpose processor or microcontroller for managing or controlling the operations and functions of the system 100. For example, the processor 130 may be specifically designed as an application-specific integrated circuit (“ASIC”) to control output signals to a user of the input/output devices 150 to provide haptic feedback or effects to the user. The processor 130 may be configured to decide, based on predefined factors, what haptic feedback or effects are to be generated based on a haptic signal received or determined by the processor 130, the order in which the haptic effects are generated, and the magnitude, frequency, duration, and/or other parameters of the haptic effects. The processor 130 may also be configured to provide streaming commands that can be used to drive the haptic output device 180 for providing a particular haptic effect. The processor 130 may be part of the electronic device 110 or the cover 120. In some embodiments, more than one processor 130 may be included in the system 100, with each processor 130 configured to perform certain functions within the system 100. For example, in an embodiment, a first processor may be part of the electronic device 110 and a second processor may be part of the cover 120. An embodiment of the processor 130 is described in further detail below.
The memory device 140 may include one or more internally fixed storage units, removable storage units, and/or remotely accessible storage units. The various storage units may include any combination of volatile memory and non-volatile memory. The storage units may be configured to store any combination of information, data, instructions, software code, etc. More particularly, the storage units may include haptic effect profiles, instructions for how the haptic output device 180 of the input/output devices 150 is to be driven, and/or other information for generating haptic feedback or effects. The memory device 140 may be part of the electronic device 110 or the cover 120.
The bus and/or communications network 160 may be configured to allow signal communication between the various components of the system 100 and also to access information from remote computers or servers through another communications network. The communications network may include one or more of a wireless communications network, an Internet, a personal area network (“PAN”), a local area network (“LAN”), a metropolitan area network (“MAN”), a wide area network (“WAN”), etc. The communications network may include local radio frequencies, cellular (GPRS, CDMA, GSM, CDPD, 2.5G, 3G, 4G LTE, etc.), Ultra-WideBand (“UWB”), WiMax, ZigBee, and/or other ad-hoc/mesh wireless network technologies, etc.
The sensor 170 is configured to detect the deformation and/or movement of the cover 120 with respect to the electronic device 110. The sensor 170 may be part of the cover 120, the electronic device 110, or both the cover 120 and the electronic device 110. The deformations may include bending, folding, rolling, or twisting the cover 120, or peeling the cover 120 off of the electronic device 110 from an edge or corner. The movements may include sliding the cover 120 relative to the electronic device 110, or separating the cover 120 from the electronic device 110. The sensor 170 may include: a bend sensor (e.g., FSR, strain gauge, fiber optic sensor), a light sensor, a pressure sensor, a contact sensor, a mechanical switch that may detect when the cover 120 is moved off the electronic device 110, a potentiometer or other angular sensor in the hinge 122, an accelerometer, a gyroscope or other angular velocity sensor, a magnetic sensor (e.g., magnetic markers on the cover 120), etc.
Additional sensors 170 may be included in the system 100. For example, in an embodiment, the electronic device 110 may include a proximity sensor configured to sense the location of a user input element, such as the user's hand or a part of the user's hand, such as a finger, or a stylus, to the display screen 112, which may be part of a touch sensitive device. In an embodiment, the cover 120 may include a touch sensitive device configured to sense touch inputs provided by the user on the cover 120 so that the user may interact with the electronic device without opening the cover 120 or taking the cover 120 off of the electronic device 110. In an embodiment, the electronic device 110 and/or the cover 120 may include a pressure sensor configured to measure the pressure applied to a touch location at the touch sensitive device. In an embodiment, the system 100 may include a temperature, humidity, and/or atmospheric pressure sensor configured to measure environmental conditions. In an embodiment, the system 100 may include a physiological sensor configured to capture a user's physiological signals, such as heart rate, etc. In an embodiment, the system 100 may include image sensors and/or a camera configured to capture a user's facial expressions and associated biometric information.
The haptic output device 180 is configured to provide haptic feedback to the user of the system 100 and may be part of the cover 120 or the electronic device 110. In an embodiment, the cover 120 and the electronic device 110 may each include at least one haptic output device 180. The haptic output device 180 may be powered by an energy source provided on the electronic device 110 and/or the cover 120.
The haptic feedback provided by the haptic output device 180 may be created with any of the methods of creating haptic effects, such as vibration, deformation, kinesthetic sensations, electrostatic or ultrasonic friction, etc. In an embodiment, the haptic output device 180 may include an actuator, for example, an electromagnetic actuator such as an Eccentric Rotating Mass (“ERM”) in which an eccentric mass is moved by a motor, a Linear Resonant Actuator (“LRA”) in which a mass attached to a spring is driven back and forth, or a “smart material” such as piezoelectric materials, electro-active polymers or shape memory alloys, a macro-composite fiber actuator, an electro-static actuator, an electro-tactile actuator, and/or another type of actuator that provides a physical feedback such as vibrotactile feedback. A suitable haptic output device 180 may be selected based on its size and/or output. For example, piezoelectric materials and/or electro-active polymers, as represented by 180A in
The haptic output device 180 may include non-mechanical or non-vibratory devices such as those that use electrostatic friction (“ESF”), ultrasonic friction (“USF”), or those that induce acoustic radiation pressure with an ultrasonic haptic transducer, or those that use a haptic substrate and a flexible or deformable surface, or those that provide thermal effects, or those that provide projected haptic output such as a puff of air using an air jet, and so on. Multiple haptic output devices 180 may be used to generate different haptic effects, which may be used to create a wide range of effects such as deformations, vibrations, etc. For example, pulses of vibration may give the illusion that a flexible surface is more flexible than it really is, that it is made of a specific material, such as wood or plastic, or that it is bending at discrete points (i.e. detents). Additional examples of haptic effects that may be generated by the haptic output device 180 are discussed below.
It should be appreciated that although modules 132, 134, 136 and 138 are illustrated in
The sensor module 132 is configured to receive a signal from the sensor 170 that is generated when the sensor 170 detects deformation of the cover 120, which may be used as an input gesture by the user. The sensor module 132 is programmed to analyze raw sensor data received from the sensor 170 and determine whether a deformation of the cover 120 has occurred above a predetermined threshold to indicate an input gesture by the user. For example, the sensor module 132 may determine that a single corner of the cover 120 has been bent based on the readings from one or more sensors 170 on the cover 120 by an amount that indicates an input gesture has been made. The sensor module 132 may also determine that the cover 120 has been peeled off the display screen 112 by an amount that exposes pixels starting at a certain column on the display screen 112, which correlates to a particular input gesture. Upon determining that the sensed deformation of the cover 120 is above a predetermined threshold that indicates an input gesture is being made by the user, the sensor module 132 may output a signal to the input module 134 for further processing.
The input module 134 is configured or programmed to receive a signal from the sensor module 132 that is generated when the sensor module 132 detects a deformation of the cover 120 that is above a predetermined threshold indicating the user is making an input gesture. For example, the user may indicate an input by bending the cover 120 at one corner thereof, or by folding a part of the cover 120 away from the electronic device 110, or by twisting a part of the cover 120, or by a sequence of bending, folding and/or twisting parts of the cover 120. The input module 134 may be programmed with a look-up table that provides a correlation of the signals received from the sensor module 132 to predefined input gestures. After the input gesture has been identified, the input module 134 is configured or programmed to send a signal to the determination module 136 for further processing.
The determination module 136 is configured or programmed to determine what type of action is to be taken by the system 100 according to the signal received from the input module 134, and what type of haptic feedback is to be generated by the haptic output device 180. The determination module 136 may be programmed with a library of input gesture and function information that is available to the system 100 and corresponding haptic effect, if any, so that the determination module 136 may determine a corresponding output. In addition to sending a signal to command a particular action to be taken by the electronic device 110, the determination module 136 may also output a signal to the haptic output device control module 138 so that a suitable haptic effect may be provided to the user. For example, an input gesture involving bending a corner of the cover 120 may correspond to the electronic device 110 retrieving e-mail from a server, and the haptic effect may indicate how many new e-mails have been retrieved from the server.
The haptic output device control module 138 is configured or programmed to determine a haptic control signal to output to the haptic output device 180, based on the signal generated by the determination module 136. Determining the haptic control signal may include determining one or more parameters that include an amplitude, frequency, duration, etc., of the haptic feedback that will be generated by the haptic output device 180 to provide the desired effect to the user, based on all inputs to the system 100.
As indicated above, the haptic output device 180 may be used to produce haptic effects while the user is performing the input gesture or after the input gesture has been completed. The haptic feedback may, for example, provide information about the ongoing input gesture and its effect. The haptic feedback may also be used to confirm that an input gesture has been accepted.
In an embodiment, the haptic feedback may be generated while the user performs an input gesture with the cover 120. For example, in an embodiment, the cover 120 may be configured to resist peeling or produce a realistic texture when peeled off of the electronic device 110. This may, for example, replicate the feel of tape, static attraction, the creaking of a hinge, or the texture of a material such as wood being bent. In an embodiment, the cover 120 may be configured to change stiffness to replicate water, sand, or other materials. In an embodiment, the cover 120 may be configured to produce tactile sensations on one or both of its surfaces, for example, using electrostatic frictional effect.
The haptic feedback may be used to confirm which of many possible input gestures has been provided by the user. A different action may, for example, be triggered by pulling on the cover 120 at different locations along its edge, and a haptic texture may be used to indicate which of these actions has been selected.
The haptic output device 180 may also be used to indicate or restrict which input gestures can be performed. For example, in an embodiment, the cover 120 may be configured to resist an input gesture, such as peeling, when the associated function is not available. The cover 120 may also be configured to resist peeling at specific locations, such as when specific content is about to be revealed.
In an embodiment, the cover 120 may be configured to allow folding along specific creases 124 with actuated hinges 180D illustrated in
The haptic feedback may be delivered through a variety of haptic output devices, some of which may make use of the unique properties of the cover 120, such as the presence of the hinges 122. The cover 120 may be used to deliver haptic feedback while located on the front or the back of the electronic device 110. A third-party cover in accordance with embodiments of the invention may provide the system 100 with haptic feedback even though the electronic device 110 is not configured to generate haptic feedback. Haptic feedback may be applied through the hinges 122 of the cover 120, either where the cover 120 couples to the electronic device 110, or in the folds 124 of the cover 120.
In an embodiment illustrated in
Different mechanisms may be used to produce an attraction or repulsion force between the cover 120 and the electronic device 110, and affect how the cover 120 reacts to being pulled open. As illustrated in
In embodiments of the invention, the haptic effect may be in the form of changing a stiffness of the cover 120, as schematically illustrated in
In an embodiment illustrated in
In an implementation of embodiments of the invention illustrated in
In an implementation of embodiments of the invention illustrated in
In an implementation of embodiments of the invention, a user may use an e-book that has a flexible overlay. The user may bend the corners of the overlay slightly to flip to the next page of the e-book. The user may bend and hold to flip through multiple pages. As the user does so, the user may feel realistic haptic effects that remind him/her of manipulating a real book.
In an implementation of embodiments of the invention, a user may use a drawing application on his/her desktop computer. The monitor of the computer may have an overlay that allows the user to peek at the image he/she is working on before and after a certain operation is applied. Haptic effects may be generated to remind the user of the static feel of a sheet of plastic acetate. When the operation cannot be previewed, the overlay may resist peeling to clearly indicate that the option is not available.
In an embodiment illustrated in
The embodiments described herein represent a number of possible implementations and examples and are not intended to necessarily limit the present disclosure to any specific embodiments. Various modifications can be made to these embodiments as would be understood by one of ordinary skill in the art. Any such modifications are intended to be included within the spirit and scope of the present disclosure and protected by the following claims.
This application is a continuation of prior U.S. patent application Ser. No. 14/858,148, filed on Sep. 18, 2015, which is hereby incorporated by reference in its entirety for all purposes.
Number | Name | Date | Kind |
---|---|---|---|
6297838 | Chang et al. | Oct 2001 | B1 |
7289121 | Balakrishnan et al. | Oct 2007 | B1 |
7443380 | Nozawa | Oct 2008 | B2 |
20040008191 | Poupyrev et al. | Jan 2004 | A1 |
20060238494 | Narayanaswami et al. | Oct 2006 | A1 |
20060274036 | Hioki et al. | Dec 2006 | A1 |
20070247422 | Vertegaal et al. | Oct 2007 | A1 |
20080303782 | Grant et al. | Dec 2008 | A1 |
20090066663 | Chang et al. | Mar 2009 | A1 |
20090219247 | Watanabe et al. | Sep 2009 | A1 |
20100011291 | Nurmi et al. | Jan 2010 | A1 |
20100066685 | Cain et al. | Mar 2010 | A1 |
20100117975 | Cho | May 2010 | A1 |
20100141605 | Kang et al. | Jun 2010 | A1 |
20110021272 | Grant | Jan 2011 | A1 |
20110057873 | Geissler et al. | Mar 2011 | A1 |
20110227822 | Shai | Sep 2011 | A1 |
20110261021 | Modarres et al. | Oct 2011 | A1 |
20120115422 | Tziortzis et al. | May 2012 | A1 |
20120127088 | Pance | May 2012 | A1 |
20120169609 | Britton | Jul 2012 | A1 |
20120188153 | Tziortzis et al. | Jul 2012 | A1 |
20120260220 | Griffin | Oct 2012 | A1 |
20120268412 | Cruz-Hernandez et al. | Oct 2012 | A1 |
20130009882 | Salmela | Jan 2013 | A1 |
20130050112 | Vanhelle | Feb 2013 | A1 |
20130076614 | Ive | Mar 2013 | A1 |
20130088339 | Lim et al. | Apr 2013 | A1 |
20130093660 | Hirsch et al. | Apr 2013 | A1 |
20130201115 | Heubel | Aug 2013 | A1 |
20130298039 | Mestre | Nov 2013 | A1 |
20130328825 | Brown | Dec 2013 | A1 |
20140085230 | Sato et al. | Mar 2014 | A1 |
20140268517 | Moon | Sep 2014 | A1 |
20150055308 | Lim et al. | Feb 2015 | A1 |
20150177838 | Bae | Jun 2015 | A1 |
Entry |
---|
Any copies of information that are not included with this Information Disclosure Statement can be found in U.S. Appl. No. 14/858,148. |
Extended European Search Report dated Dec. 16, 2016 in corresponding European Patent Application No. 16 189 157.7. |
Rendl, C. et al., “FlexSense: A Transparent Self-Sensing Deformable Surface,” Novel Hardware I, UIST'14, Oct. 5-8, 2014, pp. 129-138. |
Extended European Search Report issued in European Application No. 16189157.7, dated Dec. 16, 2016. |
Rendl et al., “FlexSense: A Transparent Self-Sensing Deformable Surface,” Novel Hardware I, UIST'14, Oct. 5-8, 2014, pp. 129-138. |
Number | Date | Country | |
---|---|---|---|
20180321752 A1 | Nov 2018 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 14858148 | Sep 2015 | US |
Child | 16023727 | US |