Patent Application
20140359410
This application claims priority under 35 U.S.C. §119(a) to a Korean patent application filed on May 31, 2013 in the Korean Intellectual Property Office and assigned Ser. No. 10-2013-0062699, the entire disclosure of which is incorporated herein by reference.
1. Field of the Invention
The present invention relates generally to a method and an apparatus for data processing and, more particularly, to a method and an apparatus for gesture-based data input.
2. Description of the Related Art
Various apparatuses, or types of electronic devices, capable of performing communication and data processing, include, for example cellular communication terminals, Personal Digital Assistants (PDA), electronic organizers, smartphones, and tablet Personal Computers (PCs). These electronic devices have evolved into multifunctional devices that integrate various functions. For example, one such electronic device integrates various functions including voice and video telephony functions, a messaging function that includes Short Message Service/Multimedia Message Service (SMS/MMS) and email, a navigation function, a document editing function, a photographing function, a broadcast playback function, a multimedia (video and audio) playback function, an Internet access function, a messenger function, and a Social Networking Service (SNS) function.
With the introduction of touchscreens to electronic devices, it is possible to overcome limitations of traditional input means and further facilitate a user's manipulation of an electronic device. The user is capable of providing an input through a touch gesture with a finger or an electronic pen (e.g., stylus pen) on the touchscreen. For example, if the user makes a handwriting motion with the electronic pen, the electronic device stores the input as a drawing object (e.g., a drawing board function) or analyzes the stroke data (e.g., coordinates) to convert the handwriting motion to text or a figure (e.g., a writing recognition or figure recognition function).
The present invention has been made to address at least the above problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of the present invention provides a gesture-based data processing method and apparatus that are capable of facilitating various types of data input on an editable page displayed on a screen of an electronic device.
Another aspect of the present invention provides a gesture-based data processing method and apparatus that are capable of sensing a hovering gesture made over the screen of the electronic device.
An additional aspect of the present invention provides a gesture-based data processing method and apparatus that are capable of displaying a guide to prompt user input in response to a hovering gesture when an editable page is displayed on the screen of the electronic device. A further aspect of the present invention provides a gesture-based data processing method and apparatus that are capable of displaying a guide in response to a hover-in event when an editable page is displayed on the screen of the electronic device, and displaying input data in response to a hover-out event.
Another aspect of the present invention provides a gesture-based data processing method and apparatus that are capable of improving user convenience and device usability by optimizing an environment for supporting data input on an editable page displayed on the screen of the electronic device.
In accordance with an aspect of the present invention, a method is provided for data processing in an electronic device. A user gesture is detected by the electronic device. A guidance object is displayed on a screen of the electronic device in response to the user gesture. Data input by a user is received. The data input by the user is displayed with the guidance object on the screen, when the data is received.
In accordance with another aspect of the present invention, an apparatus is provided that includes a control unit that detects a user gesture, and receives data input by a user. The apparatus also includes a screen that displays a guidance object in response to the user gesture, and displays the data input by the user with the guidance object when the data is received.
In accordance with another aspect of the present invention, a non-transitory computer-readable storage medium is provided that stores one or more programs, which when executed implement the steps of: detecting a user gesture; displaying a guidance object on a screen in response to the user gesture; receiving data input by a user; and displaying the data input by the user with the guidance object on the screen, when the data is received.
Another aspect of the invention provides a computer program comprising instructions arranged, when executed, to implement a method in accordance with any one of the above-described aspects. A further aspect provides machine-readable storage storing such a program.
The above and other aspects, features, and advantages of the present invention will be more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which:
Embodiments of the present invention are described in detail with reference to the accompanying drawings. The same or similar components may be designated by the same or similar reference numerals although they are illustrated in different drawings. Detailed description of constructions or processes known in the art may be omitted to avoid obscuring the subject matter of the present invention.
Embodiments of the present invention provide an electronic device and a method for supporting the inputting of data. Specifically, a method and an apparatus are provided for processing data that is input with an application capable of recognizing various types of data (e.g., text, drawing, figure, and document) on an editable page, according to embodiments of the present invention.
In various embodiments of the present invention, the application may be any type of application that supports handwriting and drawing input functions, such as, for example, office, memo, email, message, travel organizer, drawing, web browser, and document editing applications. In various embodiments of the present invention, the application may be a type of application that is capable of providing an editable page and receiving data through the editable page.
According to various embodiments of the present invention, when using an application for inputting data to the electronic device, an application-specific guide appears or disappears in response to a user gesture. In various embodiments of the present invention, the user gesture may include, for example, user interaction in order for the electronic device to display the guide (e.g., a guidance object of the present invention). For example, the user gesture may include at least one of a hovering gesture detectable by the touchscreen and a hand gesture detectable by an infrared sensor, an illuminance sensor, or camera module.
According to various embodiments of the present invention, when an application page is displayed by the electronic device, a guidance object designated for the application appears in response to a hover gesture and disappears in response to removal of the hover gesture. The hover gesture is made with a specific input tool (e.g., a user's finger, an electronic pen, such as stylus pen).
According to various embodiments of the present invention, when an application page is displayed by the electronic device, a guidance object designated for the application appears in response to a hand gesture made by a user (or a certain object in replacement of user's hand), and disappears with the removal of the hand gesture (e.g., moving a hand out of a sensing area of the sensor). The hand gesture is detected by a sensor (e.g., an infrared sensor, an illuminance sensor), or a camera module (e.g., a front camera facing the same direction as a surface of the screen).
According to various embodiments of the present invention, a certain guidance object appears in response to a hover gesture such that the user is capable of inputting data (e.g., text, a drawing, a figure, and a document) and checking the data input process in real time when the hover gesture is removed. Similarly, the guidance object may appear in response to a hand gesture of the user, such that the user is capable of inputting data (e.g., text, a drawing, a figure, and a document), and checking the data input status in real time by removing the hand gesture while the guidance object is displayed.
In the following, a description of embodiments of the present invention is directed to a representative case in which the electronic device determines whether to display the guide in response to a hovering gesture. However, embodiments of the present invention are not limited thereto, and can be implemented with any type of user gesture (including, for example, a hand gesture) detectable by means of various sensors of the electronic device.
In various embodiments of the present invention, if an input gesture is detected within a certain distance or range of the electronic device, before contact of a certain input tool (e.g., a hovering gesture), the electronic device interprets this input gesture as a pre-action for writing or drawing. The electronic device displays a guidance object (e.g., a guidance line, a background design, and a background image) on the page of the screen to assist in a writing or drawing action of the user. In various embodiments of the present invention, the guidance object appears when the input tool is within a predetermined distance or range of the device (e.g., a hover gesture), and disappears when the input tool is removed from that distance or range. In various embodiments of the present invention, the gesture-based data processing method can be implemented in such a way that the guidance object is shown and hidden according to a distance of the input tool from the device.
In various embodiments of the present invention, the electronic device may recognize presence of an input tool (e.g., a user's finger or an electronic pen) by measuring an amount of electric current within a certain range without the input tool contacting the capacitive/resistive touchscreen. In various embodiments of the present invention, the hovering gesture denotes an event in which a certain input tool enters into a predetermined range from the electronic device. The hovering gesture is used to determine whether to show or remove a guidance object on the page. In various embodiments of the present invention, the hovering gesture may be transferred through an Application Program Interface (API).
In various embodiments of the present invention, the guidance object denotes a guide in a certain form that appears in response to the hover gesture in order to assist in inputting data. For example, the guidance object may appear in the form of horizontal lines spaced at regular intervals on the page for guiding the user in writing letters aligned horizontally. The guidance object may also appear as gridlines, a figure, or a background image, which assist the user in inputting data for a decorative purpose.
As shown in
The radio communication unit 110 is capable of communication through at least one of cellular communication, Wireless Local Area Network (WLAN) communication, short range communication, a location positioning systems (e.g., Global Positioning System (GPS)), and broadcast reception. For example, the radio communication unit 110 includes a cellular communication module 111, a WLAN communication module 113, a short range communication module 115, a location positioning module 117, and a broadcast reception module 119.
The cellular communication module 111 is capable of communicating radio signals with at least one of a base station of a cellular communication network, an external device, and various servers (e.g., integration server, provider server, content server, Internet server, cloud server). The radio signals may carry the voice telephony data, video telephony data, and text/multimedia message data.
The WLAN module 113 is responsible of establishing a WLAN link with an Access Point (AP) or another electronic device and capable of being embedded in the electronic device or implemented as an external device. There is various radio Internet access technologies available such as Wireless-Fidelity (Wi-Fi), Wireless Broadband (WiBro), World Interoperability for Microwave Access (WiMAX), and High Speed Downlink Packet Access (HSDPA). The WLAN module 113 is capable of establishing a WLAN link with another electronic device to transmit and receive various guidance objects selected by the user to and from the other electronic device. The WLAN module 113 is also capable of establishing WLAN link with various servers to receive various guidance objects. The WLAN module 113 may be in on-state always or turned on according to the user setting or input.
The short range communication module 115 is responsible for the short range communication of the electronic device. There are various short range communication technologies available such as Bluetooth, Bluetooth Low Energy (BLE), Radio Frequency Identification (RFID), Infrared Data
Association (IrDA), Ultra Wideband (UWB), ZigBee, and Near Field Communication (NFC), etc. The short range communication module 115 is capable of transmitting and receiving various guidance objects to and from another electronic device according to the user's intension in the state connected to another electronic device. The short range communication module 115 may be in on-state always or turned on according to the user setting or input.
The location positioning module 117 is responsible for positioning the location of the electronic device, and Global Positioning System (GPS) module is one of the representative location positioning modules. The location positioning module 117 collects accurate distance and time information from at least three base stations and performs triangulation based on the acquired information to acquire 3-Dimentional (3D) location information with latitude, longitude, and altitude. The location positioning module 117 is also capable of calculating the location information based on the signals from three or more satellites in real time. The location information of the electronic device can be acquired using various methods.
The broadcast reception module 119 receives broadcast signals (e.g., TV broadcast signal, radio broadcast signal, and data broadcast signal) and/or information on the broadcast (e.g., broadcast channel information, broadcast program information, and broadcast service provider information) from an external broadcast management server through a broadcast channel (e.g., satellite broadcast channel, and terrestrial broadcast channel).
The input unit 120 generates a signal for controlling operation of the electronic device in response to user input. The input unit 120 may include, for example, a keypad, a dome switch, a touch pad (capacitive/resistive), a jog wheel, a jog switch, or a sensor (e.g., a voice sensor, a proximity sensor, a luminance sensor, an acceleration sensor, or a gyro sensor). The input unit 120 can be implemented with external buttons and/or virtual button on the touchscreen 130. The input unit 120 is capable of receiving user input for executing an application capable of editing a page, and generating a signal corresponding to the user input. The user input unit 120 is also capable of receiving user input for configuring a guidance object to be displayed on a specific page, or selecting or modifying the guidance object on the page. The user input unit 120 is further capable of generating a signal corresponding to the user input.
The touchscreen 130 is an input/output means responsible for input and display functions, simultaneously, and includes a display panel 131 and a touch panel 133. In an embodiment of the present invention, the touchscreen 130 is capable of detecting a user's touch gesture by means of the touch panel 133 displaying a screen (e.g., an application execution screen (page screen), a page screen having a guidance object, an outbound call processing screen, a messenger screen, a game screen, or a gallery screen). The touchscreen 130 is also capable of generating an input signal corresponding to the touch gesture. Then the control unit 180 identifies the touch gesture and controls execution of an action corresponding to the touch gesture. The touchscreen 130 is also capable of detecting a hovering gesture while displaying an editable page, and generating an input signal corresponding to the hovering gesture.
The display panel 131 is capable of displaying (output) information processed by the electronic device in one or more of several modes (e.g., telephony mode, photographing mode). In an exemplary case that the electronic device operates in the telephony mode, the display panel 131 is capable of displaying a User Interface (UI) or Graphic UI (GUI) related to the telephony mode. In an exemplary case that the electronic device operates in the video telephony mode or a camera shooting mode, the display panel 131 is also capable of displaying the image or UI/GUI with captured and/or received picture. The display unit 131 is capable of displaying an execution screen (e.g., a page screen) corresponding to the application and shows or hides a specific guidance object in response to a hovering gesture made by the user on the page screen. For example, the display panel 131 is capable of showing the guidance object designated for the corresponding page according to the hovering gesture by the user, and removing the guidance object from the page according to removal of the hovering gesture by the user. The display panel 131 is also capable of showing the data input by the user overlapped with the guidance object on the page, and removing the guidance object when the hover gesture is removed such that only the user input data is displayed. The display panel 131 is also capable of supporting a display mode switching function for switching between a portrait mode and a landscape mode.
The display panel 131 can be implemented as any one of a Liquid Crystal Display (LCD), a Thin Film Transistor LCD (TFT LCD), a Light Emitting Diode (LED), an Organic LED (OLED), an Active Matrix OLED (AMOLED), a flexible display, a bended display, and a 3-Dimensional (3D) display. The display panel 131 can be implemented as a transparent or semitransparent display panel through which light penetrates.
The touch panel 133 is placed on the display panel 131 to detect a touch gesture of the user on the surface of the touch screen 130 (e.g., a single touch gesture, a multi-touch gesture, a photographing gesture, and a data input gesture). If the touch gesture is detected on the surface of the touchscreen 130, the touch panel 133 extracts the coordinates of the position of the touch gesture and transfers the coordinates to the control unit 170. The touch panel 133 generates a signal corresponding to the touch gesture to the control unit 170. The control unit 170 is capable of executing a function according to the signal transmitted by the touch panel 133, based on the position where the touch gesture is detected.
The touch panel 133 is capable of detecting the hovering gesture at a predetermined range from the surface of the touchscreen 130, and generating a signal corresponding to the detected hovering gesture. The touch panel 133 is capable of measuring an amount of electric current within a specific distance from the touchscreen 130 although there is no contact between an input tool and the touchscreen 130, and recognizing the tool and movement and retreat of the tool. The control unit 180 is capable of analyzing the hovering gesture represented by the signal from the touch panel 133, and executing a function (e.g., showing or removing the guidance object) corresponding to the analyzed hovering gesture.
The touch panel 133 is capable of receiving user input for executing an application capable of editing a page, configuring the guidance object to be presented on the page, or selecting or changing the guidance object while the page is displayed. The touch panel 133 is also capable of generating the input signal corresponding to the user input.
The touch panel 133 can be configured to converts the pressure applied at a specific position of the display panel 131, or the change of capacitance at a specific position of the display panel 131, to an electrical input signal. The touch panel 133 is capable of measuring the pressure of the touch input as well as the position and size of the touch. If a touch input is detected, the touch panel 133 generates corresponding signal(s) to a touch controller (not shown). The touch controller (not shown) is capable of processing the signal(s) and transferring the corresponding data to the control unit 180. In this way, the control unit 100 is capable of checking the touched area on the touchscreen 130.
The audio processing unit 140 is capable of transferring an audio signal input from the control unit 180 to a speaker (SPK) 141, and transferring an audio signal received through a microphone (MIC) 143 to the control unit 180. The audio processing unit 140 processes the voice/sound data received from the control unit 180 so as to output the audio signal through the speaker 141 in the form of audible sound wave, and processes the audio signal received through the microphone 143 to generate a digital signal for transfer to the control unit 180.
The audio processing unit 140 sends the audio signal received from the control unit 170 to the speaker (SPK) 141 and sends the audio signal such as voice input through the microphone (MIC) 143 to the control unit 180. The audio processing unit 140 is capable of processing the voice/sound data to output an audible sound wave through the speaker 141 and processing the audio signal including voice to generate a digital signal to the control unit 180.
The audio processing unit 140 is capable of outputting voice/sound data corresponding to the hovering gesture, while the page is displayed, under the control of the control unit 180. For example, the audio processing unit 140 is capable of outputting specific voice/sound data in showing the guidance object when the hovering gesture is detected, and different voice/sound data in hiding the guidance object when the hovering gesture is no longer detected, under the control of the control unit 180. The audio processing unit 140 is capable of receiving the voice data commanding to change (or select) the guidance object on the page, and transferring the voice data to the control unit 180. The audio processing unit 140 is also capable of receiving the voice data commanding to show or hide the guidance object while the page is displayed, and transferring the voice data to the control unit 180.
The speaker 141 is capable outputting audio data received by means of the radio communication unit 110 in any one of the above-described modes of the electronic device, and stored in the storage unit 150. The speaker 141 is also capable of outputting audio signal related to a function executed in the electronic device (e.g., application execution, guidance object presentation, guidance object hiding, inbound call reception, outbound call placing, data input, picture shooting, and media content playback).
The microphone 143 is capable of processing the sound input in any one of the above-described modes of the electronic device, to generate electric audio data. The microphone 143 can be implemented with various noise cancellation algorithms for removing noise generated in receiving the outside sound.
The storage unit 150 stores programs associated with information processing and control functions of the control unit 180, and temporarily stores input/output data (e.g., guidance object, contact information, document data, picture data, messages, chat data, and media contents including audio and video). The storage unit 150 is also capable of storing information of usage frequencies (e.g., an application usage frequency, a data usage frequency, a guidance object usage frequency, and multimedia content usage frequency), weights, and priorities. The storage unit 150 is also capable of storing data of various patterns of vibration and sound effects that are output in response to touch inputs made on the touchscreen 130.
The storage unit 150 stores an Operating System (OS) of the electronic device and application programs for controlling touchscreen-based input and display operations, showing and hiding the guidance object on an application page, overlapping the guidance object and the data input by the user. The storage unit 150 also semi-persistently or temporarily stores data generated in association with the application programs. The storage unit 150 is also capable of storing various settings for processing data in response to the hovering gesture of the user (e.g., guidance object display and various data input and display). The settings information may include mappings between guidance objects and pages. The settings information also may further include information on whether to store a link to the guidance object referenced in storing the data input together on the application page.
The storage unit 150 can be implemented with a storage medium of at least one of a flash type memory, a hard disk type memory, a micro type memory, a card type memory (e.g., a Secure Digital (SD) type memory and an eXtream Digital (XD) card type memory), a Random Access Memory (RAM), a Dynamic RAM (DRAM), a Static RAM (SRAM), a Read-Only Memory (ROM), a Programmable ROM (PROM), an Electrically Erasable PROM (EEPROM), a Magnetic RAM (MRAM), a magnetic disk memory, and an optical disk type memory. The electronic device can interoperate on the Internet with a web storage working as the storage unit 150.
The interface unit 160 provides an interface for external devices that are connectable to the electronic device. The interface unit 160 is capable of transferring data or power from the external devices to the internal components of the electronic device, and transferring internal data to the external devices. For example, the interface unit 160 can be provided with wired/wireless headset port, external charging port, wired/wireless data port, memory card slot, identity module slot, audio input/output port, video input/output port, earphone jack, etc.
The camera module 170 is responsible for a photographing function of the electronic device. The camera module 170 is capable of capturing a still or motion image. The camera module 170 is capable of outputting video data to the display unit 131 and the control unit 180. The camera module 170 may include an image sensor (or camera sensor) for converting an optical signal to an electric signal, and may also include an video signal processor for converting the electronic signal received from the image sensor to digital video data. The image sensor can be embodied as a Charge-Coupled Device (CCD) sensor or a Complementary Metal-Oxide-Semiconductor (CMOS) sensor. The camera module 170 is capable of providing an image processing function for supporting photo capture according to various options set by the user (e.g., zooming), a screen aspect ratio, and/or effects (e.g., a sketch effect, a mono effect, a sepia effect, a vintage effect, and a mosaic effect).
The control unit 180 controls overall operations of the electronic device. For example, the control unit 180 is capable of controlling voice telephony, data communication, and video telephony functions. The control unit 180 includes a data processing module 182 that processes data in response to the hovering gesture of the user on the application page, according to an embodiment of the present invention. In an embodiment of the present invention, the data processing module 182 can be implemented in the control unit 180 or as a separate element from the control unit 180. In various embodiments of the present invention, the data processing module 182 may include a page manager 184, a guide manager 186, and an object manager 188. The page manager 184, the guide manager 186, and the object manager 188 are described in more detail hereinafter.
The page manager 184 of the control unit 180 is capable of generating an editable page for receiving a user input. For example, the page manager 184 of the control unit 180 is capable of generating a page of a specific application (e.g., a document editing application, an email application, and a web browser).
The guide manager 186 of the control unit 180 is also capable of processing to show or remove the guidance object on the application page in response to the hovering gesture. For example, the guide manager 186 of the control unit 180 is capable of showing at least one guidance object designated for the corresponding page in response to the hovering gesture in viewing the application page. The guide manager 186 of the control unit 180 is also capable of processing to remove the guidance object on the application page when the hovering gesture is removed. Specifically, the guide manager 186 of the control unit 180 is capable of toggling between showing and removing the guidance object on the application page in response to the hovering gesture.
The object manager 188 of the control unit 180 is further capable of displaying data with or without the guidance object in response to the user input. For example, the object manager 188 of the control unit 180 is capable of displaying the data with the guidance object in response to the hover gesture, and without the guidance object when the hover gesture is removed.
The operations of the control unit 180 are described in greater detail below in the operation and control method of the electronic device with reference to accompanying drawings.
According to an embodiment of the present invention, the control unit 180 is capable of controlling various operations associated with the typical functions of the electronic device, as well as the above-described functions. The control unit 180 is capable of receiving input signals corresponding to various touch-based events supported on the touch-based input interface (e.g., touchscreen 130) and controlling functions in response to the input signal. The control unit 180 is also capable of controlling transmission/reception of various data through a wired or wireless communication channel.
The power supply 190 supplies power from an external power source or from an internal power source to internal components of the electronic device.
As described above, the electronic device according to one of various embodiments of the present invention can be implemented with a computer-implemented page manager 184 for displaying a page of receiving user input, a computer-implemented guide manager 186 for showing or hiding guidance object on the page in response to a hovering event, and a computer-implemented object manager 188 for displaying data with the presence or absence of the guidance object in response to the user input.
As described above, the electronic device, according to an embodiment of the present invention, can be any type of information communication and multimedia device equipped with at least one of Application Processor (AP), Graphic Processing Unit (GPU), and Central Processing Unit (CPU). For example, the electronic device can be any of a cellular communication terminal operating with various communication protocols corresponding to the communication systems, a tablet PC, a smartphone, a digital camera, a Portable Multimedia Player (PMP), a Media Player (e.g., an MP3 player), a portable game console, a Personal Digital Assistant (PDA), etc. Also, the gesture-based control method, according to an embodiment of the present invention can be applied to various display devices, such as, for example, a digital Television (TV), Digital Signage (DS), a Large Format Display (LFD), a laptop computer, and a desktop computer.
The gesture-based data processing method, according to an embodiment of the present invention can be implemented in software, hardware, or combination of both hardware and software, and stored in a computer-readable storage medium. In the case of a hardware implementation, the gesture-based data processing method, according to an embodiment of the present invention, can be implemented with at least one of Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electrical units which perform certain tasks.
Embodiments of the present invention can be implemented by the control unit 170 itself. In the case of being implemented in software, the procedures and functions described in the embodiments of the present invention can be implemented with software modules (e.g., page manager 184, guide manager 186, and object manager 188). The software modules are capable of performing at least one of the above-described functions and operations.
The storage medium can be any type of computer-readable storage media storing the program commands of displaying an editable page for receiving a user input, showing and hiding guidance object on the page in response to the user gesture, and displaying the data with the presence or absence of the guidance object in response to the user input.
Referring to
The control unit 180 detects a hovering gesture over the touchscreen 130, in step 203. For example, the user may make a hovering gesture with a certain input tool, by entering within a predetermined range from the surface of the touchscreen 130, while the page is displayed, such that the control unit 180 detects the hovering gesture.
The control unit 180 controls the display of a guide in response to the hovering gesture, in step 205. For example, the control unit 180 is capable of displaying a predetermined guidance object on the page in response to the hovering gesture. The control unit 180 is capable maintaining the display of the guidance object as long as the hovering gesture is maintained and, at this time, the input tool may be of hovering or contacting the surface of the touchscreen 130 to input data (e.g., a text object or a drawing object).
The control unit 180 detects user input, in step 207. For example, the control unit 180 is capable of detecting a signal corresponding to the user input for entering certain data (e.g., a text object or a drawing object) on the page while the guidance object is present on the page. In various embodiments of the present invention, the user input may be made with at least one user gesture (e.g., writing and drawing gestures).
The control unit 180 controls the display of an object corresponding to the user input, in step 209. For example, the control unit 180 is capable of displaying the object corresponding to the user input (e.g., a text object or a drawing object) along with the guidance object.
The control unit 180 detects removal of the hovering gesture, in which the input tool is removed from the surface of the touchscreen 130 and the above-described range from the surface of the touchscreen 130, in step 211. For example, the user may make lifting the input tool beyond a predetermined distance from the touchscreen 130 while the guidance object is present.
The control unit 180 controls the display of the object without the guidance object in response to detection of removal of the hovering gesture, in step 213. For example, if the hovering gesture is removed while the guidance object is shown on the page, the control unit 180 controls the display of only the object corresponding to the user input (e.g., the text object or the drawing object) without the guidance object.
Although not shown in
As shown in
With a page in the state shown in
With a page in the state shown in
With a page in the state shown in of
As described above with reference to
If the input tool 400 is detected on a drawing-recognition page, the guidance object (e.g., horizontal lines) 300 may appear semi-transparently on the page. The guidance object assists the user to write text as aligned horizontally. If the user lifts the input tool 400 beyond a predetermined range from the screen, only the text 500 input by the user is displayed. The guidance object (e.g., horizontal lines) 300 is provided for the purpose of writing-assistance, and thus, is ignored when the input data is stored or transmitted. For example, the input data may be stored in itself or along with the page as the background. According to an embodiment of the present invention, however, the input data may be stored as combined with the guidance object, according to the user's intention.
As shown in
While the screen is in the state shown in
While the screen is in the state shown in
As described with reference to
For example, if the input tool is detected within a distance above the screen displaying a drawing application page, the guidance object (e.g., background sketch) 320 appears on the page semi-transparently. Then the user is capable of drawing an object in detail with the assistance of the guidance object (e.g., background sketch) 320. If the user lifts the input tool 400 out of a predetermined distance from the screen displaying the page, i.e. hover-out gesture is detected, the guidance object (e.g., background sketch) is hidden such that the object drawn by the user is left visible. In various embodiments, the guidance object (e.g., background sketch) 320 appears and disappears dynamically in response to the hover-in and hover-out gestures made on the screen displaying the page of the application capable of recognizing various data input gestures made with the input tool 400.
Since the input tool 400 has to approach the screen in order for the user to draw an object on screen, it is possible to configure the hover-in gesture detectable and provide a background sketch on the page semi-transparently in response thereto. The user is capable of drawing an object in detail with the assistance of the background sketch. If the input tool 400 hovers out of the distance range from the screen, the background sketch disappears such that the user can check object drawn until then intuitively.
According to an embodiment of the present invention, the guidance object provision mode for providing the guidance object (e.g., background sketch) based on the hovering gesture, can be turned on/off with a menu option. Also, it is possible to bring a figure or drawing template provided by the application, or a picture taken from the web browser or gallery application or captured by the camera module, as the background sketch while the guidance object provision mode is on. According to an embodiment of the present invention, the guidance object appearing in the guidance object provision mode may be any type of data stored in memory devices connected to the electronic device. For example, the guidance object may be data stored in the internal memory of the electronic device or received from an external storage (e.g., a content server or a cloud server) connected through a network.
In the embodiment of present invention illustrated in
As shown in
With a screen in the state as shown in
With a screen in the state shown in
With a screen in the state shown in
As described with reference to
According to various embodiments of the present invention, the to guidance object 340 is provided to notify the user of active regions allowing for drawing or writing input or inactive regions not allowing for user input with a guide. According to an embodiment of the present invention, a certain document format to be filed in a public office or exit/entry process can be provided with active and inactive fields. Accordingly, if the hover-in event is detected, the electronic device dims out the inactive fields in order to prevent the user from attempting data input to the inactive fields.
As shown in
With a screen that is in the state shown in
With a screen that is in the state shown in
If a figure representing a certain quick command is drawn while the user maintains the hovering gesture with the input tool 400, the electronic device retrieves the quick command corresponding to the figure and executes the function indicated by the quick command with the display of the execution screen. For example, if the figure object input is made by the user to as shown in
For example, the user is capable of removing the hovering gesture with the input tool 400 above the touchscreen 130, as shown in
As described with reference to
For example, the user may encounter the situation in which the user has no idea of what to do in attempting to use a function of the electronic device. In various embodiments of the present invention, the electronic device can be configured to provide the user with information on the manipulation tips, available functions of application, quick commands, help information (e.g., text or pictogram-based tutorial) etc., in association with the hovering gesture. Specifically, the electronic device is capable of semi-transparently providing the guidance object 360 to assist the user in making a follow-up action upon detection of a hovering gesture.
The user may also encounter a situation in which the user has no idea of what to do with the electronic device when operating an application with functions executable according to the writing recognition or gesture recognition-based quick commands. For example, the user may not know that “?” is designated for search command execution and “@” is designated for email function execution in association with the use of the electronic device. According to an embodiment of the present invention, the electronic device is capable of detecting the hovering gesture made with an input tool above an application page screen, and providing the user with the information on the actions available to be taken in response to the hovering gesture.
According to various embodiments of the present invention, the electronic device can be configured such that a certain guidance object designated for the corresponding application appears in response to the hover-in event and disappears in response to the hover-out event. In various embodiments, the guide display mode may be on/off according to the user settings. At least one guidance object can be designated per page in various applications in the guide display mode according to the user settings.
The user is capable of configuring various types of guidance objects through a guidance object setting menu screen provided by the electronic device. For example, the guidance object is capable of being designed in the form of any one of a figure, a symbol, a letter (or letter string), an image, a template, and a visual effect (e.g., engraving, embossing, shading, and dimming effects). The guidance object can be generated using a picture captured by the camera unit 170, stored in the storage unit 150, or received from a server, and defined in a new form with a certain tool according to the user's intention.
When a plurality of guidance objects are mapped to a specific page, a selection window listing the guidance objects can be provided in response to the hovering gesture, such that the user selects one of the guidance objects.
Referring to
The control unit 180 detects the hovering gesture occurring above the touchscreen 130, in step 703. For example, the user is capable of making the hovering gesture within a predetermined distance from the surface of the touchscreen 130 while the page is displayed, and the control unit 180 is capable of detecting the hovering gesture.
Upon detection of the hovering gesture, the control unit 180 determines whether the guide display mode has been turned on, in step 705. For example, if the hovering gesture is detected, the control unit 180 determines whether to display a predetermined guidance object depending on the configuration of the guide display mode.
If it is determined that the guide display mode has been turned off, the control unit 180 performs a corresponding operation, in step 707. For example, the control unit 180 may ignore the hovering gesture, or may perform a specific operation corresponding to the hovering gesture when the guide display mode has been turned off. For example, the control unit 180 may call for a predetermined menu in response to the hovering gesture, or may execute a function (e.g., zooming operation) mapped to the hovering gesture.
If it is determined that the guide display mode has been turned on, the control unit 180 determines whether there is a guidance object corresponding to the page, in step 709. For example, the control unit 180 may determine to display the guidance object when the guide display mode has been turned on, and may determine whether there is at least one guidance object designated for the corresponding page.
If it is determined that there is no guidance object corresponding to the page, the control unit 180 displays a guidance object configuration menu, in step 711, and configures a guidance object in response to user input with respect to the configuration menu, in step 713. For example, the control unit 180 is capable of displaying a menu window having menu items for configuring a guidance object. The user is capable of performing selection of a guidance object to be mapped to the page in the menu window.
According to an embodiment of the present invention, if a guidance object is to be acquired through capturing an image, the user is capable of activating the camera module 170 by selecting a camera item from the menu window and taking the captured image as the guidance object. If a guidance object is to be acquired from a gallery application, the user is capable of executing a gallery application by selecting a gallery item from the menu window and selecting one of the pictures (or images) listed by the gallery application for use as a guidance object. If a guidance object is to be acquired by selecting a figure, the user is capable of selecting a figure list item from the menu window to display a figure list and select a figure from the figure list for use as a guidance object. If a guidance object is to be acquired through web search, the user is capable of selecting a web search item from the menu window to search the web for the related images, figures, letters, symbols, etc., and select one of the found objects for use as a guidance object. Specifically, according to embodiments of the present invention, the guidance object can be acquired from the inside of the electronic device or the outside of the electronic device (e.g., from servers and other electronic devices).
If it is determined that there is a guidance object corresponding to the page, in step 709, the control unit 180 determines whether there is a single guidance object corresponding to the page, in step 715.
If it is determined that there is only one guidance object corresponding to the page, the methodology proceeds to step 721. If it is determined that there are multiple guidance objects corresponding to the page, the control unit 180 may display a selection window listing the guidance objects corresponding to the page, in step 717, such that the user selects one of the guidance objects, in step 719. Although the description is directed to the case where one guidance object is designated for one page in an embodiment of the present invention, the present invention is not limited thereto. According to various embodiments of the present invention, one or more guidance objects can be designated for one page according to the user's selection (or configuration).
When a guidance object is configured at step 713, when it is determined that there is a single guidance object corresponding to the page at step 715, or when a guidance object is selected from the selection window at step 719, the control unit 180 controls the display such that the guidance object appears on the page, in step 721. For example, the control unit 180 is capable of processing the corresponding operation in response to the hovering gesture, and then displaying the corresponding guidance object on the page. The input tool may be in the state of hovering above the screen or being in contact with the surface of the touchscreen 130 for inputting data (e.g., a text object or a drawing object).
The control unit 180 detects a user input, in step 723. For example, the control unit 180 is capable of detecting a signal corresponding to the user input of data (e.g., a text object or a drawing object) on the page having the guidance object. In an embodiment of the present invention, the signal corresponding to the user input may include at least one user gesture (e.g., a writing input or a drawing input).
The control unit 180 controls the display of an object corresponding to the user input, in step 725. For example, the control unit 180 is capable of simultaneously displaying the guidance object designated for the page and the object corresponding to the user input (e.g., a text object or a drawing object).
The control unit 180 detects removal of the hovering gesture, in step 727, in which the input tool is lifted from the surface of the touchscreen 130 and beyond a predetermined distance from the touchscreen 130, while the guidance object is visible.
The control unit 180 displays the data object without the guidance object, in response to removal of the hovering gesture, in step 729. For example, if the hovering gesture is removed while the guidance object is visible, the control unit 180 controls the display such that only the data object (e.g., a text object or a drawing object) is displayed without the guidance object on the page.
Although not depicted in
Although not depicted in
Although not depicted in
In the above-described embodiments of the present invention, the descriptions have been directed to the case where the guidance object is displayed based on the user's hovering gesture. According to an embodiment of the present invention, however, the present invention can be implemented with hand gestures detectable with various sensors (e.g., an infrared sensor, an illuminance sensor, and/or a camera module (oriented in the same direction as the surface of touchscreen)).
According to an embodiment of the present invention, when displaying the page in response to the user request, the electronic device is capable of activating at least one sensor (e.g., at least one of an infrared sensor, an illuminance sensor, and a camera module). If an input event in which an object enters into a sensing range of the corresponding sensor is detected (e.g., a hand gesture), the electronic device is aware that the user is preparing the writing or drawing action. The electronic device displays the guidance object (e.g., a guide line, a background sketch, or a background image) to assist in the writing or drawing action of the user. In an embodiment of the to present invention, if a hand gesture (e.g., a hand gesture in which the hand enters within the sensing range of the sensor) is detected by a currently running sensor, the guidance object appears on the page screen. If the hand gesture stops (e.g., the hand moves out of the sensing area of the sensor), the guidance object disappears on the page screen. Specifically, in an embodiment of the present invention, the guidance object appears and disappears in real time according to the presence/absence of detection of the hand gesture.
As described above, the gesture-based data processing method and apparatus of the present invention is capable of toggling the visibility of a guidance object according to the user gesture (e.g., hovering gesture and hand gesture).
The gesture-based data processing method and apparatus of the present invention is also capable of showing a guidance object on an editable page of the application of the electronic device in response to a hovering event to assist the user to input data efficiently.
Also, the gesture-based data processing method and apparatus of the present invention is capable of showing a guide in response to a hover-in event in the state that an editable page is displayed on the screen of the electronic device, displaying data input with the assistance of the guide as overlaid thereon, and hiding the guide in response to a hover-out event to display only the input data with the absence of the guide. The gesture-based data processing method and apparatus of the present invention allows the user to make writing, drawing, or document editing input accurately with the assistance of the guide (e.g., guidance object preconfigured in various forms) appearing in response to the hover-in action of a certain input tool (e.g., user's finger and electronic pen). For example, the user may write text along the horizontal line appearing on the page in response to the hover-in event, draw with the assistance of a background sketch appearing in response to the hover-in event, or editing a document with the assistance of indication marks appearing in response to the hover-in event.
The gesture-based data processing method and apparatus of the present invention is further capable of assisting the user to write text as aligned along the horizontal guide lines and drawing a figure with the assistance of guide, resulting in improvement of writing and drawing recognition rate.
The gesture-based data processing method and apparatus of the present invention is also capable of providing various types of background sketches in response to a hovering gesture triggered by a certain input tool (e.g., user's finger and electronic pen), such that the user draws a picture delicately and accurately with the assistance of the background sketch. The background sketch disappears in response to removal of the hovering gesture triggered by the input tool such that only the picture drawn by the user is displayed. Accordingly, the user is capable of checking the drawing progress by removing the hovering gesture whenever necessary while drawing the picture.
Also, the gesture-based data processing method and apparatus of the present invention is capable of presenting various drawing tools in the form of figures or picture templates in the hover-in state of the input tool in order to assist the user's input while checking the input result with the hover-out gesture.
Furthermore, the gesture-based data processing method and apparatus of the present invention is capable of assisting the user to make an accurate and delicate input on an editable page of the application executed in the electronic device, resulting in improvement of user convenience and device usability and competitiveness. The gesture-based data processing method and apparatus according to any of the various embodiments of the present invention can be applied to any of all types of electronic devices capable of processing data (e.g., data input and display) as well as portable terminals (such as smartphone, tablet computer, PDA, and digital camera).
In various embodiments of the invention, the modules may be implemented as any or combination of software, firmware, hardware, and/or any combination thereof. Also, some or all of the modules may be implemented as an entity capable of executing the functions of the individual modules identically.
In various embodiments of the present invention, a plurality of operations may be performed sequentially, repeatedly, or in parallel. Also, some of the operations may be omitted or replaced by other operations.
While the invention has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2013-0062699 | May 2013 | KR | national |
