Patent Grant
10380783
The present invention relates to an electronic device whose display screen has flexibility, a storage medium which stores a program for performing a display in the electronic device, a program for performing a display in the electronic device, and a method for performing a display in the electronic device.
In recent years, portable information terminals such as mobile phones or smartphones, portable music reproducing devices, portable game machines, and other consumer portable electronic devices with high performance which include display devices have become so widespread as to change the lifestyle of the people of today.
Development of such portable electronic devices is further promoted; as well as multifunctionality of the devices, development in terms of external features, such as a reduction in, size or weight, is also actively pursued. In addition, as next-generation portable electronic devices, flexible electronic devices in the form of a sheet are being actively researched and developed.
The e-book reader disclosed in Patent Document 1 has a flexible housing and a positional sensor and a bending sensor which detect a bended part of the e-book reader. Contents displayed on a display portion are switched on the basis of results of the detection by the positional sensor and the bending sensor, whereby a user can control the display without using a mouse or a button owing to the flexibility of the housing.
[Patent Document 1] Japanese Published Patent Application No. 2010-157060
Meanwhile, in, such a flexible portable electronic device, even though operation utilizing the flexibility is possible, displayed still images or moving images are flat, where the flexibility of the display screen and the electronic device which includes the display screen is not sufficiently utilized. In this manner, the product planning has failed to achieve a synergistic effect of the flexibility of the display screen and the multifunctionality of the electronic device, which makes the electronic device including a flexible display portion less attractive to users.
In view of the above, an object of one embodiment of the present invention is to provide an electronic device which displays an object (a body) on a flexible display screen in accordance with a three-dimensional shape of the display screen by utilizing the flexibility of the display screen.
Another object of one embodiment of the present invention is to provide a program for displaying an object on a flexible display screen in accordance with a three-dimensional shape of the display screen by utilizing the flexibility of the display screen.
Thus, one embodiment of a structure of the invention disclosed in this specification is an electronic device which includes a display portion including a flexible display device displaying an object on a display screen; a detection portion detecting positional data of a given part of the display screen; and an arithmetic portion calculating a three-dimensional shape of the display screen on the basis of the positional data and computing motion of the object to make the object be displayed to move according to a given law in accordance with the calculated three-dimensional shape of the display screen.
Another embodiment of a structure of the invention disclosed in this specification is a computer-readable storage medium storing a program for making an electronic device which includes a flexible display device displaying an object on a display screen execute a first step of detecting positional data of a given part of the display screen; a second step of calculating a three-dimensional shape of the display screen on the basis of the positional data; a third step of computing motion of the object to make the object be displayed to move according to a given law in accordance with the calculated three-dimensional shape of the display screen; and a fourth step of displaying the computed motion of the object on the display screen.
A further embodiment of a structure of the invention disclosed in this specification is a program for making an electronic device which includes a flexible display device displaying an object on a display screen execute a first step of detecting positional data of a given part of the display screen; a second step of calculating a three-dimensional shape of the display screen on the basis of the positional data; a third step of computing motion of the object to make the object be displayed to move according to a given law in accordance with the calculated three-dimensional shape of the display screen; and a fourth step of displaying the computed motion of the object on the display screen.
A still further embodiment of a structure of the invention disclosed in this specification is a displaying method which uses an electronic device including a flexible display device displaying an object on a display screen and includes a step of detecting positional data of a given part of the display screen; a step of calculating a three-dimensional shape of the display screen on the basis of the positional data; a step of computing motion of the object to make the object be displayed to move according to a given law in accordance with the calculated three-dimensional shape of the display screen; and a step of displaying the computed motion of the object on the display screen.
An electronic device according to one embodiment of the present invention includes, in its category, a portable electronic device such as a mobile phone, a PHS, a smartphone, a personal computer, a personal digital assistant (PDA), a tablet PC, a laptop PC, a mini computer, an e-book reader (electronic paper), an electronic dictionary, an electronic notebook, a calculator, a navigation system, a digital photo frame, an image reproducing device, an IC recorder, or a portable game machine.
In a flexible display screen and an electronic device which includes the flexible display screen, a detection portion in which a plurality of sensors are provided in matrix, for example, is provided directly under a display device including the flexible display screen. Positional data detected by the sensors and the like are combined and a curved shape of a display portion is calculated. An object is displayed to move according to a given law in accordance with the curved shape of the display screen (a three-dimensional shape of the display screen).
Note that a “three-dimensional shape of the display screen” refers to a physical form of the display screen which is altered (e.g., curved or bended) by application of external force on the electronic device or the display device, which can be defined by three-dimensional space coordinates. Thus, the three-dimensional shape also includes, in its category, a two-dimensional shape before or after deformation (i.e., a flat shape without a curve).
“An object is displayed to move according to a given law in accordance with a three-dimensional shape of the display screen” means that, for example, in the case where the display screen is warped downward, the object (body) displayed on the display screen is displayed to move in accordance with force imaginarily applied to the object, which is, for example, force in the natural world such as gravity. The object may be a solid (e.g., a dice, a fallen leaf, or a marble) or liquid (e.g., water). Further, all things in nature including gas and powder can be defined as the object. Accordingly, a user can feel a sense of reality as if the object exists on the surface of the electronic device.
Moreover, the object may be a living thing. For example, in electronic games or the like, the object can be a character in the form of a person, an animal, a plant, or an imaginary creature.
A “given law” refers to a law which can be expressed by a given equation. For example, it refers to a physical law of the natural world, specifically a law which defines motion of a body. More specific examples are laws, which can be expressed by equations of motion visually descriptive of mechanics, of gravity, attractive force, frictional force, air resistance, motion of a rigid or fluid body, and the like. However, a “given law” is not limited to physical laws of the natural world. For example, to enhance a visual effect, the given law may be a law which is accentuated with deviation from a physical law of the natural world, or a law violating the natural law, by which an object floats defying gravity, for example. In addition, such an equation does not necessarily represent a physical law of the natural world exactly, and may be a pseudo equation or an equation simplified.
The detection portion has sensors which are provided in a plurality of given positions for calculation of a three-dimensional shape of the display screen of the electronic device. For example, it is possible to provide a plurality of positional sensors in matrix in the vicinity of the display screen as the sensors so that the positional sensors detect relative positional coordinates. It is also possible to provide a plurality of acceleration sensors in matrix in the vicinity of the display screen as the sensors so that the acceleration sensors detect relative change in acceleration of each part, which accompanies deformation of the display screen. The sensors are not limited to the above ones and can be sensors to which, for example, a mechanical, electromagnetic, thermal, acoustic, or chemical means is applied as long as the sensors can detect parameters necessary for the calculation of a three-dimensional shape of the display screen. For example, an acceleration sensor, an angular velocity sensor, a vibration sensor, a pressure sensor, a gyroscope sensor, or the like can be used as the sensors. Alternatively, these sensors may be combined to be used.
Note that the present invention includes, in its scope, a method, hardware (e.g., an electronic device, a computer, a semiconductor device, or a storage medium), a system, a program, software, and the like in which the operations in the present invention are achieved.
An electronic device can be provided which displays an object on a flexible display screen in accordance with a three-dimensional shape of the display screen by utilizing the flexibility of the display screen.
A program for displaying an object on a flexible display screen in accordance with a three-dimensional shape of the display screen by utilizing the flexibility of the display screen can be provided.
Thus, a user interface (UI) can be provided which can offer a user a sense of reality as if an object displayed on the display screen exists on the surface of the electronic device.
Hereinafter, embodiments of the invention disclosed in this specification will be described with reference to the accompanying drawings. Note that the invention disclosed in this specification is not limited to the following description, and it is easily understood by those skilled in the art that modes and details can be variously changed without departing from the spirit and the scope of the invention. Therefore, the invention disclosed in this specification is not construed as being limited to the description of the following embodiments.
In this embodiment, one mode of a structure of an electronic device which has a flexible display screen and one mode of a method for performing a display on the display screen are described with reference to
(Structure of the Electronic Device)
An example of a structure of the electronic device which has a flexible display screen according to one embodiment of the present invention is described with reference to
The display screen 102 is a part of a display device which displays images including a still image and a moving image. As the display device including the display screen 102, there are a light-emitting device in which each pixel includes a light-emitting element typified by an organic light-emitting element (OLED); a liquid crystal display device; an electronic paper performing a display in an electrophoretic mode, an electronic liquid powder (registered trademark) mode, or the like; a digital micromirror device (DMD); a plasma display panel (PDP); a field emission display (FED); a surface conduction electron-emitter display (SED); a light-emitting diode (LED) display; a carbon nanotube display; a nanocrystal display; a quantum dot display; and the like. The display screen according to one embodiment of the present invention is a part of any of these display devices, and a flexible display device is used as the display device including the display screen.
In this embodiment, a touch panel with which data can be input by an instruction means such as a finger or a pen is provided as an input means on the display screen 102. Since the touch panel is provided, a region for a keyboard on the electronic device becomes unnecessary and thus the display screen can be provided in a large region. Moreover, since data can be input with a pen or a finger, a user-friendly interface can be obtained. Although the touch panel may be of any of various types such as a resistive type, a capacitive type, an infrared ray type, an electromagnetic induction type, and a surface acoustic wave type, a resistive type or a capacitive type is particularly preferable since the display screen 102 according to one embodiment of the present invention can be curved.
Since the display screen 102 is flexible, the housing 101 also needs to have deformability. The housing 101 is preferably formed using an elastic resin material, a plastically deformable metal material, a combination thereof, or the like. For example, a pressed metal plate can be used for the four corners of the housing 101 and a plastic molded body can be used for the other parts. Note that although not shown, it is also possible to use a material which is not flexible for the housing 101 when only the display device including the display screen 102 has flexibility and a space is provided between the display device and the housing 101. In this case, the housing 101 may be partly pleated, for example, so that the housing 101 can be stretched and shrunk as the display screen 102 is curved.
As illustrated in
The four corners of the electronic device 100 are rounded. Since the electronic device 100 has its four corners rounded, concentration of stress at edges of the four corners due to bending or twisting can be relieved, which can lead to an improvement in durability of the display device and the electronic device 100.
Further, so that the electronic device 100 can be easily curved, the thickness of the electronic device 100 is preferably as small as possible in a range which ensures a certain strength.
The shape of the electronic device 100 which is illustrated in
The home button 103 is provided on the lower middle part of the upper surface of the housing of the electronic device 100. When the home button 103 is pressed, a home screen is displayed on the display screen 102. Further, the electronic device 100 may be configured such that main power supply of the electronic device 100 is turned off with a press of the home button 103 for a given time. A structure may also be employed in which a press of the home button 103 brings the device which is in a sleep mode out of the sleep mode. Besides, the home button can be used as a switch for starting a variety of functions, for example, in accordance with the length of time for pressing or by pressing the home button at the same time as another button. When, the home button 103 has a variety of functions as described above, the buttons actually provided on the housing 101 can be reduced in number so as to simplify the structure and the design of the electronic device 100. The number of buttons provided on the housing 101 is preferably made as small as possible so that the electronic device 100 according to one embodiment of the present invention, which has flexibility, becomes less trouble-prone and has high reliability.
The undersurface (bottom surface) of the electronic device 100 is illustrated in
A volume control button 104 and a mute button 105 are provided on the upper right corner of the side surface of the electronic device 100. A speaker 107 for outputting sound is provided on the lower left corner of the side surface of the electronic device 100. The speaker 107 outputs various kinds of sound, examples of which are sound set for predetermined processing such as startup sound of an operating system (OS), sound from sound files executed in various applications, such as music from music reproduction application software, and an incoming e-mail alert. Specifically, in the electronic device 100 according to one embodiment of the present invention, the speaker 107 may output sound in response to a curve of the display screen 102, or may output sound in response to move of an object, which will be described later.
In addition, the volume of sound output from the speaker 107 can be adjusted with the volume control button 104. Sound from the speaker 107 can be silenced instantaneously by a press of the mute button 105. Although not shown, a connector for outputting sound to a device such as headphones, earphones, or a headset may be provided together with or instead of the speaker 107 for outputting sound.
A microphone 106, which can be used for sound input and recording, is provided on the lower right corner of the side surface of the electronic device 100. Further, a sleep button 108 is provided on the upper left corner of the side surface of the electronic device 100. The electronic device 100 can be put into the sleep mode when the sleep button 108 is pressed, in which case power can be saved by stopping given functions such as a display on the display screen 102 with the main power supply on. In this manner, consumption of power of a battery incorporated in the electronic device 100 can be suppressed.
When a variety of input-output devices are locally provided on the four corners of the electronic device 100 as described above, parts which do not have flexibility can be concentrated in the four corners, which allows the electronic device 100 to have flexibility as a whole. In the case where a non-flexible member is used for the four corners, the structural strength of the electronic device 100 increases and the usability of the electronic device 100 can be enhanced. Therefore, for the four corners of the housing 101 of the electronic device 100, a non-flexible member which is different from materials of the other parts is preferably used.
In contrast, a camera 109 is provided on the upper middle part of the undersurface of the electronic device 100. Images obtained with the use of the camera 109 can be displayed on the display screen 102. Note that providing the camera 109 on the middle part allows the user to know where the camera 109 is positioned even in the case where the user operates the electronic device 100 while seeing its upper surface. Since the electronic device 100 according to one embodiment of the present invention has flexibility, it is possible to capture blur-free images by standing the electronic device 100 which is curved into a U shape on a table or the like, without using a fixture such as a tripod.
Note that although not shown, the electronic device 100 may include a terminal for connecting an external memory drive. Examples of the external memory drive are storage medium drives such as an external hard disk drive (HDD), a flash memory drive, a digital versatile disk (DVD) drive, a DVD-recordable (DVD-R) drive, a DVD-rewritable (DVD-RW) drive, a compact disc (CD) drive, a compact disc recordable (CD-R) drive, a compact disc rewritable (CD-RW) drive, a magneto-optical (MO) disc drive, a floppy disk drive (FDD), and a nonvolatile solid state drive (SSD) device different from the above flash memory drive. Although the electronic device 100 has the touch panel on the display screen 102, a keyboard may be provided on the housing 101 instead of the touch panel or may be externally added.
The above-described electronic device 100 has flexibility as illustrated in
The above structure of the electronic device 100 is merely an example and some of the components can be omitted; for example, the keyboard 170 can be omitted, in which case a virtual keyboard using the touch panel 169 and the display device 158 is produced with software. Moreover, a component other than the above may be added to the structure.
For the processor 151, a microprocessor such as a digital signal processor (DSP) or a graphics processing unit (GPU) can be used in addition to a central processing unit (CPU). The processor 151 interprets and executes instructions from various programs to process various kinds of data and control programs.
Note that a thin film transistor in which a channel formation region includes an oxide semiconductor can be used for the processor 151. Because the transistor has extremely low off-state current, by using the transistor as a switch for holding electric charge (data) flowed into a memory element, a long data retaining period can be ensured. By utilizing the above characteristics for a register of the processor 151 or the like, the processor 151 can operate only when needed and the contents of the previous processing can be stored in the memory element in the rest of the time, so that normally off computing can be performed; thus, power consumption of the electronic device can be reduced.
The main memory 152 is used as a main memory device.
A dynamic random access memory (DRAM), for example, is used for the RAM 180, and a memory space as a workspace for the processor 151 is virtually allocated and used. As illustrated in
In the ROM 181, a basic input/output system (BIOS) 186, firmware, and the like for which rewriting is not needed are stored. Further, display portion physical property data 187 including physical parameters of the display screen 102 having flexibility according to one embodiment of the present invention, and sensor characteristics data 188 concerning the characteristics of the sensors 156 which detect positional data can be stored in the ROM 181 in advance. As the ROM 181, a mask ROM, a one time programmable read only memory (OTPROM), or an erasable programmable read only memory (EPROM) can be used. As an EPROM, an ultra-violet erasable programmable read only memory (UV-EPROM) which can erase stored data by irradiation with ultraviolet rays, an electrically erasable programmable read only memory (EEPROM), a flash memory, and the like can be given.
The auxiliary memory 154 incorporated in the electronic device 100 functions as an auxiliary memory device. The auxiliary memory 154 is a storage medium having a larger capacity than the main memory 152, and is connected to the system bus 150 through the memory controller 153. The memory controller 153 functions as an interface which controls reading and writing of data from and into the auxiliary memory 154, for example. For the auxiliary memory 154, a storage medium drive such as a hard disk drive (HDD) or a nonvolatile solid state drive (SSD) device can be used, for example.
Note that although being incorporated in the electronic device 100 in
The sensors 156 detect parameters necessary for calculating a three-dimensional shape of the display screen of the electronic device 100. For example, as the sensors 156, a plurality of positional sensors which can specify a positional relation relative to each other can be provided in matrix in the vicinity of the display screen so that the positional sensor detects positional data relative to the other positional sensors. It is also possible to provide a plurality of acceleration sensors in matrix in the vicinity of the display screen as the sensors 156 so that the acceleration sensors detect relative change in acceleration of each part, which accompanies deformation of the display screen. The sensors 156 are not limited to the above ones and can be sensors to which, for example, a mechanical, electromagnetic, thermal, acoustic, or chemical means is applied as long as the sensors can detect parameters necessary for the calculation of a three-dimensional shape of the display screen. For example, an acceleration sensor, an angular velocity sensor, a vibration sensor, a pressure sensor, a gyroscope sensor, or the like can be used as the sensors. Alternatively, these sensors may be combined to be used. Note that the sensors 156 may be incorporated in touch sensors provided on the display screen 102. By combining the touch sensors and the positional sensors into one component, the number of parts can be reduced, which can contribute to a reduction in the thickness of the electronic device 100.
The sensor controller 155 is an interface which performs centralized control of the plurality of sensors 156. The sensor controller 155 supplies power from the power supply 160 to the plurality of sensors 156, and receives input from the sensors 156, converts it into a control signal, and outputs the signal to the system bus 150. The sensor controller 155 may handle errors made by the sensors 156 or may calibrate the sensors 156.
The display device 158 is connected to the system bus 150 through the display controller 157. As the display device 158, a display device having flexibility and selected from a light-emitting device in which each pixel includes a light-emitting element typified by an organic light-emitting element (OLED); a liquid crystal display device; an electronic paper performing a display in an electrophoretic mode, an electronic liquid powder (registered trademark) mode, or the like; a digital micromirror device (DMD); a plasma display panel (PDP); a field emission display (FED); a surface conduction electron-emitter display (SED); a light-emitting diode (LED) display; a carbon nanotube display; a nanocrystal display; a quantum dot display; and the like is used. In response to drawing instructions input from the processor 151 through the system bus 150, the display controller 157 controls the display device 158 so that a predetermined image is displayed on the display screen 102 of the display device 158.
The power supply 160 supplies power to a plurality of components of the electronic device 100. As the power supply 160, for example, one or more primary batteries or secondary batteries are included. In the case of indoor use or the like, an alternate-current (AC) power supply may be used as an external power supply. Particularly in the case of using the electronic device 100 separately from the external power supply, it is favorable that the power supply have a large charge/discharge capacity which allows the electronic device 100 to be used for a long time. When the power supply 160 is charged, a charger separate from the electronic device 100 may be used. Further, since the electronic device 100 in this embodiment is flexible, it is preferable that the power supply 160 be also flexible. As a secondary battery having such a feature, for example, a lithium ion secondary battery and a lithium ion polymer secondary battery can be given. It is preferable that a laminate package be used as a jacket of the battery so that the battery has flexibility.
Further, although not shown, the power supply 160 may have a power supply management device (battery management unit: BMU). The BMU collects data on cell voltage or cell temperatures of the battery, monitors overcharge and overdischarge, controls a cell balancer, handles a deterioration state of the battery, calculates the remaining battery power (state of charge: SOC), and controls detection of a failure, for example.
The power supply controller 159 controls transmission of power from the power supply 160 to each component through the system bus 150 or a power supply line. The power supply controller 159 has a power converter or an inverter with a plurality of channels, a protection circuit, and the like. Further, the power supply controller 159 has a function of reducing power consumption. For example, after detection of no input to the electronic device 100 for a given period, the power supply controller 159 lowers clock frequency or stops input of clocks of the processor 151, stops operation of the processor 151 itself, or stops rotation of the HDD, thereby reducing the power consumption. Such a function is performed with the power supply controller 159 alone or the power supply controller 159 interlocking with the processor 151.
The communication interface (I/F) 162 is connected to the system bus 150 through the communication controller 161. The communication controller 161 and the communication I/F 162 control, in response to instructions from the processor 151, a connection signal for connecting the electronic device 100 to a computer network, and transmit the signal to the computer network. Accordingly, communication can be performed by connecting the electronic device 100 to a computer network such as the Internet (which is an infrastructure of the World Wide Web (WWW)), an intranet, an extranet, a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), or a global area network (GAN).
In the case where communication between the electronic device 100 and another device is performed without a transmission line, i.e., wirelessly, a radio-frequency (RF) circuit may be provided in the communication interface (I/F) 162 so that an RF signal is transmitted and received. The RF circuit performs conversion between an electromagnetic signal and an electric signal in a frequency band which is set by a national law, and performs communication with another communication device wirelessly with the use of the electromagnetic signal. Several tens of kilohertz to several tens of gigahertz is a practical frequency band which is generally used. The RF circuit includes an RF circuit portion and an antenna which are compatible with a plurality of frequency bands; the RF circuit portion can include an amplifier, a mixer, a filter, a digital signal processor (DSP), an RF transceiver, or the like. In the case of performing wireless communication, it is possible to use, as a communication protocol or a communication technology, a communications standard such as Global System for Mobile Communication (GSM) (registered trademark), Enhanced Data Rates for GSM Evolution (EDGE), Code Division Multiple Access 2000 (CDMA2000), or Wideband Code Division Multiple Access (W-CDMA), or a communications standard developed by IEEE such as Wireless Fidelity (Wi-Fi) (registered trademark), Bluetooth (registered trademark), or ZigBee (registered trademark).
Further, in the case where the electronic device 100 is used as a phone for a telephone conversation, the communication controller 161 and the communication I/F 162 control, in response to instructions from the processor 151, connection signals for connecting the electronic device 100 to a telephone line, and transmit the signal to the telephone line.
The speaker 164, the sound output connector 165, and the microphone 166, which are responsible for sound, are connected to the sound controller 163 to be connected to the processor 151 through the system bus 150. The sound controller 163 generates analog sound signals audible to the user in response to instructions from the processor 151, and outputs the signals to the speaker 164 or the sound output connector 165. Sound data input to the microphone 166 is converted into a digital signal in the sound controller 163 and processed in the sound controller 163 and the processor 151. To the sound output connector 165, a sound output device such as headphones, earphones, or a headset is connected and sound generated in the sound controller 163 is output to the device.
One or more switches provided on the housing (hereinafter referred to as the housing switches 168 for convenience), the touch panel 169 provided in the vicinity of the display screen 102, the keyboard 170 provided on the housing 101, the camera 171 provided on the undersurface of the housing 101, and the external port 172 to which other input components can be connected are controlled by the input interface 167, and the input interface 167 is connected to the processor 151 and the like through the system bus 150.
The housing switches 168 correspond to, for example, the home button 103, the volume control button 104, the mute button 105, the sleep button 108, and the like which are illustrated in
The touch panel 169 is provided on the display screen 102 and can be used as an input device by which data is input with an instruction means such as a finger or a pen. Since the touch panel 169 is provided, a region for a keyboard on the electronic device becomes unnecessary and thus the display screen can be provided in a large region. Moreover, since data can be input with a pen or a finger, a user-friendly interface can be obtained. Although the touch panel 169 may be of any of various types such as a resistive type, a capacitive type, an infrared ray type, an electromagnetic induction type, and a surface acoustic wave type, a resistive type or a capacitive type is particularly preferable since the display screen 102 according to one embodiment of the present invention can be curved. Note that the touch panel 169 may include the sensors 156 to serve as one component. This structure allows the number of parts to be reduced and contributes to a reduction of the thickness of the electronic device 100.
The vibration motor 174 is connected to the system bus 150 through the output interface 173. In response to instructions from the processor 151, the output interface 173 controls vibration time and the like and makes the vibration motor 174 vibrate. Accordingly, the electronic device 100 is made to vibrate and the vibration can be used as haptic effects to the user at the time of reception of e-mails or execution of applications such as computer games. The vibration motor 174 may be made to vibrate in accordance with the degree of a curve of the display screen 102; for example, when the flexibility of the display screen 102 is limited to a certain degree, the vibration motor 174 can be used to alert the user that the threshold value is exceeded. Although not shown, as well as the vibration motor 174, a variety of output devices with which the user can perceive by the use of the five senses can be connected to the output interface 173. For example, a light-emitting device for showing an operation status of the electronic device 100, an aroma diffuser which releases fragrance in response to vibration, or the like can be connected to the output interface 173.
(Functions of the Electronic Device)
Next,
The display portion 201 includes the display device 158 and the display controller 157 described with reference to
The detection portion 202 includes the sensors 156 and the sensor controller 155 described with reference to
The arithmetic portion 203 includes the processor 151 described with reference to
After calculating the three-dimensional shape of the display screen 102, the arithmetic portion 203 makes the object move on the field in accordance with the calculated three-dimensional shape of the display screen 102.
The storage portion 204 includes the main memory 152, the auxiliary memory 154, and the memory controller 153 described with reference to
As illustrated in
As illustrated in
The law which defines move of an object refers to a law which can be expressed by a given equation which defines the move of the object. For example, it refers to a physical law of the natural world, specifically a law which defines motion of a body. More specific examples are laws, which can be expressed by equations of motion visually descriptive of mechanics, of gravity, attractive force, frictional force, air resistance, motion of a rigid or fluid body, and the like. However, a “given law” is not limited to physical laws of the natural world. For example, to enhance a visual effect, the given law may be a law which is accentuated with deviation from a physical law of the natural world, or a law violating the natural law, by which an object floats defying gravity, for example. In addition, such an equation does not necessarily represent a physical law of the natural world exactly, and may be a pseudo equation or an equation simplified. Data on a law which defines move of an object is a group of equations which form the basis for simulation of the motion of the object. Although not shown, the data on the law which defines the move of the object can also be stored in the storage portion 204 as a structured list, table, or database.
The arithmetic portion 203 refers to the physical quantities, positional data, and the like which are associated with the object ID and field ID stored in the storage portion 204 and the three-dimensional shape of the display screen 102 which is calculated on the basis of data from the detection portion 202, and simulates motion of the object on the basis of the data on the law which defines the move of the object so that the object moves according to the above law. In other words, in response to change in the shape of the display screen 102, the arithmetic portion 203 calls the object data, the field data, the data on the law which defines the move of the object, and the like as well as data on the change in the shape to the workspace, inserts parameters in equations of the data on the law which defines the move of the object, and calculates the motion of the object. While the calculation of the motion of the object is performed until the motion of the object substantially subsides, the detection portion 202 monitors further change in the shape of the display screen 102 and modifies the motion of the object immediately on detecting change in the shape.
As described above, modification which accompanies the change in the shape of the display screen 102 is repeatedly performed and simulation continues until the motion of the object subsides (i.e., the motion of the object stops), whereby the realistic motion of the object in accordance with the deformation of the display screen 102 can be felt by the user. The motion of the object which is simulated by the arithmetic portion 203 is output to the display portion 201 and displayed on the display screen 102. Note that it is preferable that change in the shape of the display screen 102 have a given threshold value and the arithmetic portion 203 perform computation only when the threshold value is exceeded. In that case, an increase in the calculation amount of the arithmetic portion 203 can be suppressed and the motion of the object can be stopped at a predetermined level.
The input portion 205 includes the microphone 166, the housing switch 168, the touch panel 169, the keyboard 170, the camera 171, the external port 172, the sound controller 163, and the input interface 167 which are described with reference to
The output portion 206 includes the speaker 164, the sound output connector 165, the sound controller 163, the vibration motor 174, and the output interface 173 which are described with reference to
(Example of Operation of the Electronic Device)
Next, an example of operation of the electronic device 100 is described with reference to
The object 301 in
In
Applying various laws to the motion of the object 301 as described above allows the user to feel a sense of reality.
Note that all of the physical quantities and the like of the object 301 and the field are not necessarily referred to, and only part of them may be used for reference. When only part is referred to, burdens on the arithmetic portion 203 can be reduced. Similarly, only part of the group of equations which are stored in the storage portion 204 may be used in the simulation. Although only one object 301 is displayed on the field in
In the above manner, data on the three-dimensional shape of the display screen 102 is calculated using the sensors 156 and the processor 151, whereby the object 301 which moves according to the shape can be displayed. Accordingly, the user can feel a sense of reality as if the object 301 displayed on the display screen 102 exists on the surface of the electronic device 100.
Note that although the deformation of the display screen 102 is a condition for starting move of the object 301 here, the condition is not limited thereto. For example, when acceleration sensors are provided as the input interface 167 in the electronic device 100, detection of acceleration by the acceleration sensors may be the condition for starting move of the object 301. The object 301 can be made to move in a direction in accordance with inclination detected by the acceleration sensors. In addition, when the microphone 166 is used as the input interface 167, the object 301 can be made to move in accordance with the volume of input sound, for example. Alternatively, the object 301 may be made to move in a direction in accordance with a direction input with the keyboard 170. Further, the object 301 may be reset to its initial position by a press of the home button 103, which is provided on the housing 101 of the electronic device 100.
(Procedure for Display Processing of Object)
Next, a procedure for display processing of an object in the electronic device 100 according to one embodiment of the present invention is described with reference to
Accordingly, the procedure for display processing of an object which is described below is executed by an application program. The program is stored in a computer-readable storage medium such as the auxiliary memory 154 or the main memory 152.
Further, although the program corresponds to software here, such a processing means can also be an electronic circuit or mechanical hardware.
First, an application for displaying an object in the electronic device 100 according to one embodiment of the present invention is started (S001). Here, the application program or the like stored in the auxiliary memory 154 is loaded into the main memory 152. Note that starting of the application may be set to be performed at the same time as startup of the operating system.
Then, initial conditions related to the object and the field are set (S002 and S003). The initial conditions related to the object and the field may be set so that conditions at the time of close of the application in the previous use are transferred, or may be reset to the default settings in every use. Moreover, the user may be allowed to select any of these settings.
The initial conditions of the object set the object ID 251 which is the object data 250 stored in the storage portion 204, display positions, and the like. Further, the initial conditions of the field set the field ID 261 which is the field data 260 stored in the storage portion 204, and the like. Specifically, the setting refers to reading of the object ID 251 and the object physical quantity 253 and the like associated therewith from the auxiliary memory 154, and storing of them in the main memory 152. Similarly, parameters for the field data 260 are also set.
An order in which the setting of the initial conditions related to the object (S002) and the setting of the initial conditions related to the field (S003) are performed is not limited to the above. The setting of the initial conditions related to the field may be performed first, or the setting of the initial conditions related to the object (S002) and the setting of the initial conditions related to the field (S003) may be performed at the same time.
Then, with the plurality of sensors 156 which are provided in the vicinity of the display screen 102, positional data in the respective positions is obtained (S004). After that, on the basis of the positional data obtained with the plurality of sensors 156, the three-dimensional shape of the display screen 102 is calculated in the arithmetic portion 203 (S005). In the case where the shape of the display screen 102 has already been obtained with another system or application after starting of the electronic device 100 and before starting of the application, the three-dimensional shape of the display screen 102 may be calculated using the obtained data on the shape.
Then, the shape of the display screen 102 calculated in Step S005 is compared with the initial state of the object, and whether to make the object move is determined (S006). For example, in the case where the initial position of the object which is set in Step S002 corresponds to a position having an inclination that exceeds a predetermined threshold with respect to a horizontal plane due to a curved, shape of the display screen 102, the arithmetic portion 203 determines that move of the object is necessary. In contrast, in the case where the initial position of the object corresponds to a position parallel to the horizontal plane although the display screen 102 has a curved portion, the object is at rest, and thus the arithmetic portion 203 determines that move of the object is not necessary. When the arithmetic portion 203 determines that move of the object is necessary, the processing shifts into an object moving mode 510 (or an object moving mode 520 which will be described as another example) (S007). When the arithmetic portion 203 determines that move of the object is not necessary, the processing shifts into an object standby mode 530 (S008).
When the object standby mode 530 is exited, whether to close the application is judged in Step S009; in the case where the conditions are met (i.e., in the case where an application closing flag which will be described later is “1”), the application is closed (S010), and in the case where the conditions are not met (i.e., in the case where the application closing flag which will be described later is “0”), the processing returns to Step S004.
At the time of closing the application, various kinds of data stored in the main memory 152 is preferably stored in the auxiliary memory 154. Accordingly, the data can be used as the initial settings in the next starting of the application.
After the object moving mode starts (S020), the motion of the object is determined according to the shape of the display screen which has already been calculated (S021). Here, the motion of the object is determined before Δt.
The motion of the object is simulated according to a predetermined law in the arithmetic portion 203. The simulation is executed using the physical quantity of the object, the physical quantity of the field, and the like which are set in Step S002 and Step S003. Through this simulation, the moving direction, speed, and the like of the object are determined. For the simulation of the moving direction, speed, and the like of the object, the field data 260 on the associated field ID 261 is also referred to. For example, in the case where the field physical quantity 263 of the field ID 261 which is selected includes data on an elastic material such as rubber, the motion of the object which bounces after falling into a hollow formed by curving the display screen 102 is simulated. Data on the simulated motion of the object is stored in the main memory 152 of the storage portion 204.
Simulation for displaying rotation, deformation, or the like of the object may be performed to further provide the user with a sense of reality. However, when the simulation is complicated, the arithmetic portion 203 becomes overloaded so that delay in processing of the arithmetic portion 203 inhibits an appropriate display; as a result, a sense of reality offered to the user is diminished. In view of this, it is preferable that computation be simplified or a database of computation results be made, for example, so as to avoid a slowdown in the computation speed.
Then, according to the motion of the object which has been determined by the simulation, the motion of the object is displayed on the display screen 102 for a period of Δt (S022). The motion of the object is affected by coefficient of friction, air resistance, gravity, or the like depending on the field physical quantity 263. Note that motion of the object includes stop of the object. Therefore, Step S023 includes a motion sequence from stop of the object to move of the object, and that from move of the object to stop of the object.
Then, the three-dimensional shape of the display screen 102 is recalculated (Step S023). Specifically, in this step, data stored in the storage portion 204, which is data on the previously calculated three-dimensional shape of the display screen 102, is rewritten with data on the three-dimensional shape of the display screen 102 which is newly obtained by a recalculation sequence 511 for the three-dimensional shape of the display screen.
Here, the recalculation sequence 511 for the three-dimensional shape of the display screen is described with reference to
By the recalculation sequence 511 for the three-dimensional shape of the display screen, the data on the previously calculated three-dimensional shape of the display screen 102 is updated by being overwritten with the newly obtained data on the three-dimensional shape of the display screen 102. Alternatively, without overwriting, an address different from that of the storage portion 204 in which the data on the previously calculated three-dimensional shape of the display screen 102 is stored may be specified to store the newly obtained data.
Further, at the same time as the recalculation sequence 511 for the three-dimensional shape of the display screen, data input with an input device such as a touch panel may be read from the input interface 167 in Step S023.
Then, for update of data on the motion of the object which is stored in the main memory 152, the motion of the object is simulated again (S024). The simulation of the motion of the object may be performed without exception; alternatively, it is possible to perform the simulation of the motion of the object only in the case where there is a difference between the data on the previously calculated three-dimensional shape of the display screen 102 and the data on the three-dimensional shape of the display screen 102 recalculated in Step S023 when these pieces of data are compared with each other. In this case, the amount of the change preferably has a predetermined threshold value for the comparison between the pieces of data on the three-dimensional shape, in which case noise or burdens due to unnecessary computation by the arithmetic portion 203 in a region unnoticeable to the user can be reduced. That is, it is preferable that it be determined that the three-dimensional shape of the display screen 102 has been changed only when the amount of the change of the data on the recalculated three-dimensional shape of the display screen 102 from the data on the previously calculated three-dimensional shape of the display screen 102 exceeds the predetermined threshold value, and the processing proceed to Step S024; when there is no change, the processing proceed to Step S025.
Next, in Step S025, whether the object is in motion or at rest is determined. In the object moving mode 510, simulation of the motion of the object is performed every period of Δt. Therefore, in some cases, the results of the simulation in Step S021 show that the object after Δt is in motion. Further, depending on the results of the recalculation of the three-dimensional shape of the display screen 102 in Step S023, and the like, another motion of the object might occur by the simulation in Step S024. In this case, it is determined that the object is not at rest but in motion, so that the processing returns to Step S022.
In contrast, in the case where it is determined that the object is at rest, the object moving mode 510 is exited (S026).
As described above, a display is performed with the period of the motion of the object divided into periods each having a length of Δt, and the simulation is repeated with the shape of the display screen changed, whereby the motion of the object can be modified and displayed each time, which can have the user feel a sense of reality.
In the processing in the above-described object moving mode 510, time for displaying the motion of the object is divided and simulation for the next display is performed after an intended display is finished; meanwhile, in processing in the object moving mode 520, the motion of the object is displayed for a given period and in this period, simulation for the next display is performed.
When the object moving mode 520 starts (S050), the motion of the object is determined by simulation on the basis of data on the three-dimensional shape of the display screen 102 which has already been calculated (S051).
Then, the motion determined by the simulation is displayed for a given period (here, Δt), and during the period Δt, the motion of the object in the next period Δt is simulated. The display of the motion of the object is executed in accordance with a display sequence 521 for the motion of the object, and the next motion of the object is executed in accordance with a computation sequence 522 for the motion of the object (S052).
Here, the display sequence 521 for the motion of the object is described with reference to
Next, the computation sequence 522 for the motion of the object is described with reference to
With the sensors 156, positional data of given parts of the display screen 102 is obtained (S073). Then, the three-dimensional shape of the display screen 102 is calculated on the basis of the obtained positional data (S074). In accordance with data on the calculated three-dimensional shape of the display screen 102, the motion of the object for the period Δt is simulated and determined (S075). After that, updating t by addition oft is performed (S076).
Steps S073 to S076 described above are repeated until t becomes Δt (S077). Here, the period Δt is the same as the period Δt which is determined in the display sequence 521 for the motion of the object. By synchronization between the sequences, the sequences are concurrently performed during the same period (Δt). That is, in the display sequence 521 for the motion of the object, while the motion of the object is displayed, the motion of the object for the next period can be determined in advance by the computation sequence 522 for the motion of the object.
In the computation sequence 522 for the motion of the object, when the shape of the display screen 102 is changed during the period Δt, the determined motion of the object is modified by being overwritten. Note that in
When t becomes Δt as a result of the operation repeated as described above, the computation sequence 522 for the motion of the object ends (S078).
After Step S052, as illustrated in
As described above, in the object moving mode 520 illustrated in
Next, the object standby mode 530 is described.
When the object standby mode starts (S030), in Step S031, the application closing flag which shows whether or not the application is closed and a mode exiting flag which shows whether or not the object standby mode 530 is exited are initialized. For example, both of the flags are set to “0”.
Then, whether or not the display screen 102 is deformed is checked with the use of the sensors 156 (S032) by comparing the data on the three-dimensional shape of the display screen 102 which has already been calculated in Step S005, Step S042, or Step S074 with the data on the three-dimensional shape of the display screen 102 which is recalculated in Step S032 (S032). Note that the amount of the change preferably has a predetermined threshold value for the comparison between the pieces of data on the three-dimensional shape, in which case noise or burdens due to unnecessary computation by the arithmetic portion 203 in a region unnoticeable to the user can be reduced. That is, it is preferable that it be determined that the three-dimensional shape of the display screen 102 has been changed only when the amount of the change of the data on the recalculated three-dimensional shape of the display screen 102 from the data on the previously calculated three-dimensional shape of the display screen 102 exceeds the predetermined threshold value, and the processing proceed to Step S034.
In the case where it is determined that the three-dimensional shape of the display screen 102 has been changed, the mode exiting flag is set to “1” (S034). In the case where there is no change in the shape of the display screen 102, whether or not there is input from the input interface 167 is then checked. The input from the input interface 167 here is limited to input that affects the motion of the object. Similarly, when there is input, the mode exiting flag is set to “1” (S034). In the case where there is neither change in the shape of the display screen 102 nor the input from the input interface 167, the processing proceeds to the next step while the mode exiting flag remains “0”.
In Step S035, whether or not there is notification of an instruction for closing the application is checked. When there is the notification, the application closing flag for the application is set to “1” (S036). The instruction for closing the application includes, for example, an instruction for closing the application from the user through the processor 151 and an instruction for closing the application from another application, a program, or an operating system. In the case where there is not the notification of the instruction for closing the application, the application closing flag remains “0”, which is the initial value.
The flag is judged in Step S037. That is, in the case where one or both of the mode exiting flag and the application closing flag are set to “1”, the object standby mode is exited (S038). In contrast, in the case where both of the flags are “0”, the object standby mode persists. In other words, the processing returns to Step S032.
As described above, in the object standby mode 530, an input state is monitored by looping through checking of whether or not there is given input. When there is either kind of input, the object standby mode 530 is exited and the processing proceeds to Step S009.
This embodiment can be combined with any of the other embodiments as appropriate. Note that the present invention includes, in its scope, a method, hardware (e.g., an electronic device, a computer, a semiconductor device, or a storage medium), a system, a program, software, and the like in which the operations in the present invention are achieved.
Although the object 301 which is a spherical solid is described in Embodiment 1 as an example of an object, an object is not limited thereto. In this embodiment, the case where the object is designed to resemble liquid is described with reference to
The object 303 in
The object 303 is defined as liquid by the object physical quantity 253 in the object data 250. The arithmetic portion 203 performs simulation by referring to data or a parameter stored in the object physical quantity 253 and indicating that the object is liquid. Simulation in which the object 303 moves like real liquid according to a law of fluid mechanics (hydraulics) is performed, whereby the motion of the object is displayed as if liquid flows from higher to lower areas. As the data or parameter indicating that the object is liquid, for example, a value of density, viscosity, a compaction property, surface tension, or the like may be prepared as a physical quantity, or for simplicity, the degree of a physical quantity such as viscosity may be divided into several levels to be prepared as several pieces of data. Further, an image set of liquid in various forms is preferably stored in the object image 254 in advance so as to provide the user with visual recognition of the liquid flowing.
A physical quantity and the like are set also for the field; for example, the liquid object moves while encountering friction determined for the field.
Applying various laws to the motion of the liquid object 303 as described above allows the user to feel a sense of reality.
Note that all of the physical quantities and the like of the object 303 and the field are not necessarily referred to, and only part of them may be used for reference. When only part is referred to, burdens on the arithmetic portion 203 can be reduced. Similarly, only part of the group of equations which are stored in the storage portion 204 may be used in the simulation. Although only one object 303 is displayed on the field in
In the above manner, data on the three-dimensional shape of the display screen 102 is calculated using the sensors 156 and the processor 151, whereby the liquid object 303 which moves according to the shape can be displayed. Accordingly, the user can feel a sense of reality as if the liquid object 303 displayed on the display screen 102 exists on the surface of the electronic device 100.
This embodiment can be combined with any of the other embodiments as appropriate. Note that the present invention includes, in its scope, a method, hardware (e.g., an electronic device, a computer, a semiconductor device, or a storage medium), a system, a program, software, and the like in which the operations in the present invention are achieved.
In this embodiment, an electronic device in which a touch panel is used as an input device included in an input portion and the object can be controlled by input from the touch panel is described with reference to
With such a structure in which the electronic device 100 includes the touch panel in the input portion, the procedure for display processing of the object is the same as that described in Embodiment 1 in principle. However, the conditions for starting the motion of the object are access to the object through the touch panel as well as detection of deformation of the display screen 102, which is described in Embodiment 1. Therefore, in the procedure for processing described in Embodiment 1, the steps for determining move of an object (S006, S025, and S053) involve checking of whether or not there is detection of input with the touch panel.
In a similar manner, in the object moving mode 510 illustrated in
Note that although the example in which the touch panel is used for moving the object 302 is described in this embodiment, the object 302 may be made to move using, other than the touch panel, any of a variety of input devices which can be included in the input portion 205; for example, a housing switch provided on the housing 101, a keyboard which is incorporated in or externally provided to the electronic device 100, a pointing device (e.g., a mouse or a controller) which is connected to the external port, a microphone for sound input, and the like can be used. When a microphone is used, for example, the motion of the object 302 can be appropriately adjusted to be displayed depending on the volume of input sound (see
As described above, the input portion 205 can include a variety of input devices with which the object 302 displayed on the display screen 102 can be made to move, so that the user can feel a sense of reality as if the object exists on the surface of the electronic device 100.
This embodiment can be combined with any of the other embodiments as appropriate. Note that the present invention includes, in its scope, a method, hardware (e.g., an electronic device, a computer, a semiconductor device, or a storage medium), a system, a program, software, and the like in which the operations in the present invention are achieved.
In this embodiment, the case where processing for displaying an object on the display screen is executed at the same time as a different application is described with reference to
In
Although text-editing application software is illustrated as the different application software in
An object 306a and an object 306b according to one embodiment of the present invention are displayed on the display screen 102. The object 306a and the object 306b move in response to input from the sensors or the input portion described in Embodiments 1 to 3. Here, the object 306a and the object 306b are set so that the objects are not allowed to move in the entire region of the display screen 102 and cannot enter given regions. That is, in
Further, as illustrated in
In this manner, when not only an object and a field but also a variety of images displayed on the display screen 102, such as icons or windows of application software have physical quantities to be treated as bodies, the motion of the object can be more realistic.
Note that a target which is provided with a physical quantity and treated as a body is not limited to an image displayed on the display screen and may be ends (four sides) of the display screen, a finger of a user which is detected with the touch panel, or the like.
In order to achieve such a display, images such as icons or windows can be regarded as second objects and object IDs, object shapes, object physical quantities, or the like can be set in advance. The conditions of the second objects may be incorporated in simulation of the motion of the object by the arithmetic portion.
Alternatively, images such as icons or windows may be defined as parts of a field and integrated into field data to be reflected in the simulation of the motion of the object.
Note that not all the images such as the icons or windows are necessarily regarded as bodies; only predetermined icons or windows may be regarded as bodies. Alternatively, the object may be displayed moving on these images so that the images are regarded as a background for the object without being regarded as bodies. It is also possible to allow the user to decide whether to regard these images as bodies.
In the case of performing such processing of the motion of the object which incorporates images such as icons or windows of application software or ends of the display screen as bodies, a display of these images and the like is controlled by the operating system; thus, the processing needs to interlock with the operating system. Therefore, by incorporating the procedure for display processing of the object into the operating system as a module, a series of steps of display processing of the object can be executed efficiently.
This embodiment can be combined with any of the other embodiments as appropriate. Note that the present invention includes, in its scope, a method, hardware (e.g., an electronic device, a computer, a semiconductor device, or a storage medium), a system, a program, software, and the like in which the operations in the present invention are achieved.
In this embodiment, display processing in which a given region of a curved display screen is selected and an object moves only in the region is described with reference to
In
In
Such a display of an object can be used for a home screen or a screensaver of the operating system or various kinds of application software.
To perform the above display of an object, the procedure for processing which is described in Embodiment 1 can be used. However, in the object moving mode of the procedure for processing, a region in which an, object can move needs to be defined in accordance with the three-dimensional shape of the display screen 102 as appropriate. Thus, in the object moving mode, the procedure for processing which is illustrated in
In this embodiment, the region in which the object can move is defined as a given height of the display screen 102. For example, when the middle position of the display screen is set as a position of reference, a two-dimensional space positioned above (or below) the position of reference by a given height can be defined as the region in which the object can move. Alternatively, a three-dimensional space positioned above (or below) the position of reference by a given range of heights can be defined as the region in which the object can move. Such a two-dimensional or three-dimensional space may be defined as a value in advance, or a given space including a portion in which an object is displayed may be defined as a region in which the object can move.
After the region in which the object can move is determined, motion of the object in the region is determined in Step S082. In simulation of the motion of the object at this time, data on the region which is determined in Step S081 is used as boundary conditions. Subsequent steps of processing for a display and the like (S083 to S089) are similar to those in the object moving mode 510 which is illustrated in
This embodiment can be combined with any of the other embodiments as appropriate. Note that the present invention includes, in its scope, a method, hardware (e.g., an electronic device, a computer, a semiconductor device, or a storage medium), a system, a program, software, and the like in which the operations in the present invention are achieved.
In this embodiment, a computer game which utilizes a display of an object moving in accordance with the shape of the display screen is described with reference to
Under the circumstances, as illustrated in
Here, by setting the river 309 as the above-described object, water flow in the river 309 interlocks with deformation of the display screen 102, whereby the user can feel a sense of reality. In addition, by calculation of the shape of the display screen 102 using the sensors 156, the range of motion of the characters can be controlled in accordance with the shape.
To achieve such a computer game, the procedures for display processing of objects which are described in any of Embodiments 1 to 5 can be applied to bodies including characters in the computer game. The procedure for display processing of an object may be incorporated in a computer program as a module, or may be executed while interlocking with an operating system in which the procedure for display processing of an object is incorporated. Such a program, a module, or the like may be stored in a computer-readable storage medium such as the auxiliary memory 154 or the main memory 152, or alternatively such a processing means may be achieved in the form of an electronic circuit or mechanical hardware.
As described above, the game can proceed depending on change in the shape of the display screen 102, so that the user can enjoy reality of the computer game.
This embodiment can be combined with any of the other embodiments as appropriate. Note that the present invention includes, in its scope, a method, hardware (e.g., an electronic device, a computer, a semiconductor device, or a storage medium), a system, a program, software, and the like in which the operations in the present invention are achieved.
100: electronic device, 101: housing, 102: display screen, 103: home button, 104: volume control button, 105: mute button, 106: microphone, 107: speaker, 108: sleep button, 109: camera, 150: system bus, 151: processor, 152: main memory, 153: memory controller, 154: auxiliary memory, 155: sensor controller, 156: sensor, 157: display controller, 158: display device, 159: power supply controller, 160: power supply, 161: communication controller, 162: communication I/F, 163: sound controller, 164: speaker, 165: sound output connector, 166: microphone, 167: input interface, 168: housing switch, 169: touch panel, 170: keyboard, 171: camera, 172: external port, 173: output interface, 174: vibration motor, 180: RAM, 181: ROM, 182: operating system, 183: application program, 184: program module, 185: program data, 186: BIOS, 187: display portion physical property data, 188: sensor characteristics data, 201: display portion, 202: detection portion, 203: arithmetic portion, 204: storage portion, 205: input portion, 206: output portion, 250: object data, 251: object ID, 252: object shape, 253: object physical quantity, 254: object image, 255: default of an object, 260: field data, 261: field ID, 262: field shape, 263: field physical quantity, 264: field image, 265: default of a field, 301: object, 302: object, 303: object, 304: icon, 305: window, 306: object, 306a: object, 306b: object, 307: object, 308a: region, 308b: region, 309: river, 310a: character, and 310b: character.
This application is based on Japanese Patent Application serial no. 2012-109132 filed with Japan Patent Office on May 11, 2012, the entire contents of which are hereby incorporated by reference.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2012-109132 | May 2012 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5844547 | Minakuchi et al. | Dec 1998 | A |
| 6375572 | Masuyama et al. | Apr 2002 | B1 |
| 6641482 | Masuyama et al. | Nov 2003 | B2 |
| 7223173 | Masuyama et al. | May 2007 | B2 |
| 7601066 | Masuyama et al. | Oct 2009 | B1 |
| 7730250 | Nanmoto | Jun 2010 | B2 |
| 8015490 | Ogikubo | Sep 2011 | B2 |
| 8462125 | Birnbaum et al. | Jun 2013 | B2 |
| 8466873 | Vertegaal et al. | Jun 2013 | B2 |
| 8504936 | Gimpl et al. | Aug 2013 | B2 |
| 8527892 | Sirpal et al. | Sep 2013 | B2 |
| 8587417 | Birnbaum et al. | Nov 2013 | B2 |
| 8588860 | Sirpal et al. | Nov 2013 | B2 |
| 8599106 | Gimpl et al. | Dec 2013 | B2 |
| 8638301 | Birnbaum et al. | Jan 2014 | B2 |
| 8648825 | Sirpal et al. | Feb 2014 | B2 |
| 8659565 | Sirpal et al. | Feb 2014 | B2 |
| 8666455 | Song | Mar 2014 | B2 |
| 8864583 | Lotzer | Oct 2014 | B1 |
| 8866602 | Birnbaum et al. | Oct 2014 | B2 |
| 8922531 | Lee | Dec 2014 | B2 |
| 8958848 | Shin et al. | Feb 2015 | B2 |
| 8976112 | Birnbaum et al. | Mar 2015 | B2 |
| 9063571 | Birnbaum et al. | Jun 2015 | B2 |
| 9134803 | Birnbaum et al. | Sep 2015 | B2 |
| 9229532 | Birnbaum et al. | Jan 2016 | B1 |
| 9323378 | Choe | Apr 2016 | B2 |
| 9477309 | Birnbaum et al. | Oct 2016 | B2 |
| 9497305 | Youn et al. | Nov 2016 | B2 |
| 9607422 | Leedom | Mar 2017 | B1 |
| 9612662 | Birnbaum et al. | Apr 2017 | B2 |
| 9692865 | Youn et al. | Jun 2017 | B2 |
| 9785238 | Birnbaum et al. | Oct 2017 | B2 |
| 9900414 | Youn et al. | Feb 2018 | B2 |
| 9910496 | Birnbaum et al. | Mar 2018 | B2 |
| 9952820 | Anderson | Apr 2018 | B2 |
| 10019061 | Birnbaum et al. | Jul 2018 | B2 |
| 20040008191 | Poupyrev et al. | Jan 2004 | A1 |
| 20040029640 | Masuyama et al. | Feb 2004 | A1 |
| 20060062983 | Irvin et al. | Mar 2006 | A1 |
| 20070178974 | Masuyama et al. | Aug 2007 | A1 |
| 20070247422 | Vertegaal | Oct 2007 | A1 |
| 20090002328 | Ullrich et al. | Jan 2009 | A1 |
| 20090036213 | Masuyama et al. | Feb 2009 | A1 |
| 20090219247 | Watanabe | Sep 2009 | A1 |
| 20090325698 | Masuyama et al. | Dec 2009 | A1 |
| 20100017759 | Birnbaum et al. | Jan 2010 | A1 |
| 20100045705 | Vertegaal et al. | Feb 2010 | A1 |
| 20100072467 | Yamazaki et al. | Mar 2010 | A1 |
| 20100164888 | Okumura et al. | Jul 2010 | A1 |
| 20100214243 | Birnbaum et al. | Aug 2010 | A1 |
| 20100229089 | Narita | Sep 2010 | A1 |
| 20100311502 | Miller et al. | Dec 2010 | A1 |
| 20110078571 | Asbahr et al. | Mar 2011 | A1 |
| 20110134144 | Moriwaki | Jun 2011 | A1 |
| 20110227822 | Shai | Sep 2011 | A1 |
| 20110312419 | Masuyama et al. | Dec 2011 | A1 |
| 20120004032 | Masuyama et al. | Jan 2012 | A1 |
| 20120139834 | Han | Jun 2012 | A1 |
| 20130009882 | Salmela | Jan 2013 | A1 |
| 20130093660 | Hirsch | Apr 2013 | A1 |
| 20130127748 | Vertegaal et al. | May 2013 | A1 |
| 20130162556 | Yu | Jun 2013 | A1 |
| 20130222276 | Kim | Aug 2013 | A1 |
| 20130293481 | You | Nov 2013 | A1 |
| 20130321373 | Yoshizumi | Dec 2013 | A1 |
| 20140085184 | Vertegaal et al. | Mar 2014 | A1 |
| 20140173471 | Maenpaa | Jun 2014 | A1 |
| 20140189506 | Birnbaum et al. | Jul 2014 | A1 |
| 20150309611 | Vertegaal et al. | Oct 2015 | A1 |
| 20170168579 | Birnbaum et al. | Jun 2017 | A1 |
| 20170224140 | Vertegaal et al. | Aug 2017 | A1 |
| 20180004295 | Birnbaum et al. | Jan 2018 | A1 |
| 20180260029 | Birnbaum et al. | Sep 2018 | A1 |
| 20180348870 | Birnbaum et al. | Dec 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 101211325 | Jul 2008 | CN |
| 101557432 | Oct 2009 | CN |
| 101763143 | Jun 2010 | CN |
| 101782804 | Jul 2010 | CN |
| 1895774 | Mar 2008 | EP |
| 2109295 | Oct 2009 | EP |
| 2202624 | Jun 2010 | EP |
| 2287710 | Feb 2011 | EP |
| 2723107 | Apr 2014 | EP |
| 2741177 | Jun 2014 | EP |
| 3130983 | Feb 2017 | EP |
| 3206381 | Aug 2017 | EP |
| 3220616 | Sep 2017 | EP |
| 3236337 | Oct 2017 | EP |
| 2001-170358 | Jun 2001 | JP |
| 2004-046792 | Feb 2004 | JP |
| 2006-330082 | Dec 2006 | JP |
| 2007-048237 | Feb 2007 | JP |
| 2009-205565 | Sep 2009 | JP |
| 2010-157060 | Jul 2010 | JP |
| 2010-211211 | Sep 2010 | JP |
| 2011-044128 | Mar 2011 | JP |
| 2011-528476 | Nov 2011 | JP |
| 2012-0128833 | Nov 2012 | KR |
| 200745986 | Dec 2007 | TW |
| 201118648 | Jun 2011 | TW |
| WO-2010009145 | Jan 2010 | WO |
| WO-2010009149 | Jan 2010 | WO |
| WO-2010009152 | Jan 2010 | WO |
| WO-2010009157 | Jan 2010 | WO |
| WO-2010009163 | Jan 2010 | WO |
| WO-2010088477 | Aug 2010 | WO |
| WO-2011005318 | Jan 2011 | WO |
| WO-2012044801 | Apr 2012 | WO |
| Entry |
|---|
| Author: Khalilbeigi et al., Title: FoldMe: Interacting with Double-sided Foldable Displays, Date: Feb. 2012, Publisher: ACM (Year: 2012). |
| International Search Report (Application No. PCT/JP2013/062792) dated Jul. 9, 2013. |
| Written Opinion (Application No. PCT/JP2013/062792) dated Jul. 9, 2013. |
| Chinese Office Action (Application No. 201380024451.2) dated May 31, 2017. |
| Taiwanese Office Action (Application No. 106105946) dated Apr. 24, 2018. |
| Number | Date | Country | |
|---|---|---|---|
| 20190172248 A1 | Jun 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15713729 | Sep 2017 | US |
| Child | 16250176 | US | |
| Parent | 13875400 | May 2013 | US |
| Child | 15713729 | US |
