The present disclosure generally relates to mobile or handheld devices capable of sensing motions and processing graphics. More particularly, but not by way of limitation, the present disclosure relates to a handheld device, with an application running thereon, serves as an air mouse for controlling TV and displaying graphical information on the TV.
Currently, home TV systems generally include set-top boxes (STBs) for decoding video streams, storing electronic program guide (EPG), and performing other video-related functions. Upcoming 4K (Ultra High Definition, 3840×2160 pixels) contents and new user interfaces (UIs) require more powerful STBs. However, upgrading existing STBs or purchasing new STBs can be expensive for the end users. Another component in existing home TV systems is a remote control, which acts as a man-machine interface. A typical remote control includes many buttons, either physical buttons or virtual buttons displayed on a touch-screen. A user presses a button (e.g., up, down, play, stop, etc.) on the remote control to make a selection on the TV. With this type of remote control, a user has to look down at the remote control when making selections, which can be inconvenient sometimes, for example, when the user watches TV in a dark room.
Meanwhile, handheld devices such as smartphones (e.g., iPhone™), tablets (e.g., iPad™), and other portable devices (e.g., iTouch™) are getting more powerful. They are typically equipped with motion sensors (e.g., accelerometers, gyroscopes) and powerful graphics processors. Also, the broadband connection at home is getting faster with Wi-Fi and other technologies. Hence, it is possible and desirable to use handheld devices to enhance the existing home TV systems and user viewing experiences.
The disclosed handheld device is configured to run an air mouse application (App) thereon. With the air mouse App running, the handheld device functions as an air mouse. The air mouse may or may not have buttons or keys displayed on the handheld device. A user holds the handheld device and makes a motion or a gesture such as a twist, a tilt, a shake, a swipe, a tap, multiple quick taps, a push down (or pressing down), an up-down move, a left-right (or side-to-side) move, and other type of motions. The handheld device senses the motion, translates the motion to a command, and passes the command to a TV. In an embodiment, the command is passed to the TV through an STB.
The handheld device keeps a mapping between the TV's display screen and the handheld device's own screen, which may be an internal virtual screen or an actual display screen. If the TV's screen resolution and the handheld's screen resolution are different, a mapping between the two is used for both the X and the Y axes. In an embodiment, the mapping is linear. A symbol of the air mouse may be displayed on the TV. The air mouse's position on the handheld device and the air mouse's position on the TV are synchronized and may be updated periodically. The air mouse's position on the handheld device (and hence its position on the TV) is determined by the handheld device based on motion sensing. The position of the air mouse is passed directly to the TV, or indirectly to the TV through an STB.
The disclosed air mouse is capable of voice control. A user may make a voice command, such as “channel up,” “channel down”, “mute,” and so on. The air mouse receives a voice command, translates the voice command to a TV command, and passes the TV command to a TV. In an embodiment, the TV command is passed to the TV through an STB.
With the air mouse (the handheld device running the air mouse App), the user does not have to look at the screen of the handheld device. The user only needs to look at the TV and issues a command through motion, gesture, or voice. This greatly enhances the user's viewing experiences when watching TV. For example, when a user watches TV in a dark room, he or she may not be able to see, or simply does not want to get distracted with, the buttons on a remote control.
The disclosed handheld device is configured to run an EPG application (App) thereon, which provides an EPG function to a user. The handheld device retrieves or otherwise gets the EPG from a TV service provider, and stores it in the handheld device. The handheld device also gets the resolution of the user's TV, and adjusts the formats (e.g., sizes) of the UI of the EPG before passing it to the TV for proper display. The EPG is updated periodically from a cloud server based on the location of the handheld device. With the EPG App, the handheld device becomes a personalized portable EPG for the user. Any action to the EPG by the user is processed by the handheld device and the corresponding UI changes may be displayed on a TV. A user may use the air mouse App to navigate and select highlighted item(s) on the EPG. In an embodiment, the UI of the EPG is passed to a TV through an STB.
The accompanying drawings illustrate embodiments of the systems and methods disclosed herein and together with the description, serve to explain the principles of the present disclosure.
For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosure is intended. Any alterations and further modifications to the described devices, systems, methods, and any further application of the principles of the present disclosure are fully contemplated as would normally occur to one having ordinary skill in the art to which the disclosure relates. For example, the features, components, and/or steps described with respect to one embodiment may be combined with the features, components, and/or steps described with respect to other embodiments of the present disclosure to form yet another embodiment of a device, system, or method according to the present disclosure even though such a combination is not explicitly shown. Further, for the sake of simplicity, in some instances the same reference numerals are used throughout the drawings to refer to the same or like parts.
The system 10 further includes a handheld device 18. In the present embodiment, the handheld device 18 includes a display screen 20. In various embodiments, the screen 20 may be a touch screen, such as a single touch or multi-touch screen. In another embodiment, the handheld device 18 does not include a display screen, but is capable of maintaining a virtual screen internally, such as in a memory, by its graphics processor. In the following discussion, unless otherwise specified, screen 20 refers to either an actual display screen, which a user may see or touch, or a virtual screen, which is not visible to a user but nonetheless exists internally in the handheld device 18. The screen 20 has a certain size and resolution, such as 4.7 inches with 750×1334 pixels, or another size and resolution.
Referring back to
The provider system 12 and the handheld device 18 are connected through a link 34 which may be the Internet. The TV 14, the STB 16, and the handheld device 18 are typically located in a room such as in home or in a hotel room. The handheld device 18 and the STB 16 are connected through a link 26 which may be a Wi-Fi network where the handheld device 18 and the STB 16 have the same IP subnet. The handheld device 18 and the media player 22 are connected through a link 28 which may be a Wi-Fi network where the handheld device 18 and the media player 22 have the same IP subnet. In an embodiment, the links 26 and 28 may be in the same Wi-Fi network. The STB 16 and the streaming network 24 are connected through a link 30 which may be the Internet.
In embodiments, the handheld device 18 is configured to run one or more applications (App) thereon. An App is a computer program or software (a set of computer-executable instructions) designed to run on the handheld device 18. An App may be pre-installed on the handheld device 18 or installed through an application distribution platform, such as Apple Inc.'s App Store, Google Inc.'s Google Play, or Microsoft Windows Phone Store. An App may be stored in a storage media, such as the memory 29, of the handheld device 18. In at least one embodiment, the handheld device 18 is configured to run an air mouse App. With the air mouse App running, the handheld device 18 functions as an air mouse for the TV 14. In an embodiment, the air mouse does not display buttons or keys on the handheld device 18. Rather, it detects user commands through motion sensing. For example, a user may hold the handheld device 18 and make a motion such as a twist, a tilt, a shake, a lateral move, or other type of movement. Such motion can be detected by the handheld device 18 with or without a display screen. Or, the user may make a gesture on the screen 20 (which is a touch screen in this case) such as a swipe, a tap, multiple quick taps, a push down (or pressing down), or other type of gestures. The handheld device 18 senses the motion or the gesture, translates the motion or the gesture into a command, and passes the command to the TV 14. In an embodiment, the command is passed to the TV 14 through the STB 16. To further this embodiment, a thin client is installed on the STB 16 which enables the STB 16 to process the commands from the handheld device 18 such as changing channels, turning to a specified channel, powering on/off, and so on. The STB 16 then sends corresponding commands along with proper graphics (such as EPG or a user interface (UI)) to the TV 14.
In various embodiments, the air mouse App may provide a user with a set of predefined motion and command pairings, or the user may configure a particular motion for a particular command based on user preferences. The following motion and command pairings are non-limiting examples that the air mouse App may include or provide.
(1) Command: Change a channel
(2) Command: Change volume
(3) Command: Pause
(4) Command: Fast forward and rewind if applicable
(5) Command: Move an air mouse symbol
(6) Command: Mute
(7) Command: Swap a channel
(8) Command: Show menu
(9) Command: Point to the next object while the TV 14 is in menu or EPG mode
In an embodiment, the handheld device 18 keeps a mapping between the TV 14's display screen and the handheld device 18's screen 20 (a touch screen or a virtual screen as discussed above). If the TV 14's screen resolution and the screen 20's resolution are different, a linear mapping between the two is used for both the X and the Y axes. A symbol of the air mouse may be displayed on the TV 14. The air mouse's position on the handheld device 18 and the air mouse symbol's position on the TV 14 are synchronized and may be updated periodically, for example, every one second. The air mouse's position on the handheld device 18 (and hence its position on the TV 14) is determined by the handheld device 18 based on motion sensing and the screen mapping above. The position of the air mouse is passed directly to the TV 14, or indirectly to the TV 14 through the STB 16. This is different from existing TV remote controls or existing air mouse (“existing devices”). The existing devices only determine a movement of the mouse, such as up, down, left, or right; but not the position of the mouse on the TV. The movement of the mouse is communicated to a STB (or a smart-TV having STB functions built in), which then determines the position of the mouse symbol on the TV. In contrast, the disclosed handheld device 18 and the air mouse App determine both the movement and the position of the mouse, which advantageously simplifies the implementation of the STB 16.
In an embodiment, the air mouse App enables other applications running on the handheld device 18 to be displayed on the TV 14. The user interfaces (UI) of these other applications are designed or developed for the screen 20, and are mapped to the TV 14's screen through the air mouse App. This enables these other applications to be developed independent of the resolution of the TV 14.
Referring to
At operation 44, the handheld device 18 acquires some of the TV 14's information, including the resolution of the TV 14, from the STB 16 through the link 26. At operation 44, the handheld device 18 further calculates a linear mapping factor M between the TV 14's resolution and the screen 20's resolution. When running various applications (such as the air mouse App above), the handheld device 18 uses the linear mapping factor M to map its user interface (graphics or mouse position) to the TV 14's display screen.
The linear mapping factor M includes a linear factor, mx, for the X axis, and another linear factor, my, for the Y axis. Therefore, it may be denoted as M=(mx, my). In an embodiment, the linear mapping factor M is calculated by dividing the TV 14's resolution with the screen 20's resolution for the X axis and the Y axis respectively. For example, if the TV 14 and the screen 20 have the same resolution (e.g., both are 1080p), then the linear mapping factor M is (1, 1). If the TV 14's resolution is 4K and the screen 20's resolution is 1080p, then the linear mapping factor for the X axis is mx=3840/1920=2, and the linear mapping factor for the Y axis is my=2160/1080=2. Therefore, the linear mapping factor M is (2, 2). Various other methods of calculating the linear mapping factor M are possible. When running an application, the handheld device 18 may multiply an object's coordinates in the screen 20 by the linear mapping factor M to get the object's coordinates on the TV 14, which is subsequently sent to the TV 14 through the STB 16.
At operation 46, while running the air mouse App discussed above, the handheld device 18 gets its mouse position in the screen 20. In an embodiment, the mouse position may be initially set to the upper left corner of the screen 20. Alternatively, the initial mouse position may be set to lower right corner or another point of the screen 20. The mouse position is subsequently updated according to the methods discussed in the present disclosure, such as sensing a user's motions or recognizing a user's voice commands which is discussed below. At operation 48, the handheld device 18 calculates the corresponding air mouse's position in the TV 14's display screen by applying the linear mapping factor M as discussed above.
At operation 50, the handheld device 18 checks if there is any change in UI due to a movement of the mouse. If there is no change, at operation 52, the handheld device 18 sends the adjusted mouse position to the STB 16 which subsequently sends it to the TV 14 for displaying. If there is some change in UI, at operation 54, the handheld device 18 sends the changed UI and the adjusted mouse position to the STB 16, which subsequently sends them to the TV 14 for proper displaying. The above operations 46-54 repeat for as long as there is any movement of the air mouse, which is detected by the handheld device 18 through motion sensing or voice recognition as discussed below.
In an embodiment, the air mouse App is capable of processing voice commands. A user may make a voice command, such as “channel up,” “channel down”, “mute,” and so on. The air mouse App receives the voice command, translates the voice command to a TV command, and sends the TV command to the TV 14. In the present embodiment, the TV command is passed to the TV 14 through the STB 16. To further this embodiment, the handheld device 18 and the operating system running thereon have voice recognition capability, such as Siri of Apple Inc.'s iOS.
With the air mouse App running, the user does not have to look at the screen 20 of the handheld device 18. The user only needs to look at the TV 14 and issues commands through motion, gesture, or voice, as discussed above. This provides advantages over existing remote controls or mouse applications that require a user to press buttons or keys on a device. The disclosed air mouse App greatly enhances the user's viewing experiences when watching TV. For example, when a user watches TV in a dark room, he or she may not be able to see, or simply does not want to get distracted with, the buttons on a remote control. With the disclosed air mouse App, the user simply leans back and controls the TV 14 through the handheld device 18 while looking at the TV 14.
In another embodiment, the handheld device 18 is configured to run an EPG application (App) thereon, which provides an EPG function to a user.
At operation 82, the TV service provider system 12 gets program guide and related information such as commentary, as well as video on Demand (VOD), from third parties. The TV service provider system 12 may compile an EPG and broadcast it to its subscribers. At operation 84, the handheld device 18 retrieves or otherwise gets the EPG from the TV service provider system 12 through the link 34, and stores the EPG in the handheld device 18 (e.g., in the internal memory 29 of the handheld device 18). The EPG may include channel lineup, VOD, commentary, etc. The EPG is updated periodically from a cloud server based on the location of the handheld device 18. At operation 86, the handheld device 18 creates a user interface (UI) of the EPG. In an embodiment, the UI of the EPG is further adjusted based on a linear mapping between the resolution of the user's TV (e.g., the TV 14) and the resolution of the screen 20. For example, the linear mapping may be the same as discussed with respect to
With the EPG App, the handheld device 18 becomes a personalized portable EPG for the user. Any action to the EPG by the user is processed by the handheld device 18 and the corresponding UI changes can be displayed on a TV at home or at some other locations (e.g., in a hotel room). A user may use the air mouse App above to navigate and select highlighted item(s) on the EPG.
The disclosed EPG App provides advantages over existing TV systems where an STB gets the EPG and stores it in the STB. With existing TV systems, the EPG is stored locally at home and is not portable. A user may make changes in the STB's EPG. But the changed EPG is stored in the STB at home. When the user travels away from home, he or she cannot carry the STB and hence has no access to his or her personal EPG. With the disclosed EPG App, a user has a portable personalized EPG stored conveniently in his or her handheld device. Further, the EPG App can be configured to obtain (using push-down, pull-into, or other techniques) the latest EPG from the TV service provider system 12 based on the location of the handheld device 18 which may have one or more positioning sensors (e.g., GPS receiver) in an embodiment. In some cases, TV programs differ in different locations and in different time zones. For example, the TV channels in a hotel may be a subset of available channels at home. When a user stays in the hotel and uses his or her handheld device to navigate the TV channels therein, the user's EPG in the handheld device is automatically updated for that location and only available TV channels will be displayed. This greatly enhances the user's experiences.
The disclosed EPG App provides other advantages. With the EPG App, the handheld device 18 can offload much work traditionally performed by the STB 16. The handheld device 18 can obtain the EPG and display the EPG on the TV 14. It may also display the EPG on its screen 20. In an embodiment, the handheld device 18 may run the air mouse App and the EPG App concurrently. The EPG App causes the TV 14 to display the EPG and the air mouse App allows the user to navigate, highlight, and select channels in the EPG by looking at the TV 14. This allows the design of the STB 16 to be simplified. For example, the STB 16 only needs to perform decoding and decryption for Content Access (CA) or Digital Right Management (DRM). If CA or DRM is not involved, the audio/video streaming can be directly performed by the handheld device 18 running the disclosed air mouse App and certain other applications. This allows new contents to be developed without changing existing STB.
The foregoing has outlined features of several embodiments. Those skilled in the art should appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and/or achieving the same advantages of the embodiments introduced herein. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that they may make various changes, substitutions and alterations herein without departing from the spirit and scope of the present disclosure.
This application claims the benefit of U.S. Prov. No. 62/207,026 entitled “Handheld Devices and Applications for TV,” filed Aug. 19, 2015, herein incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
62207026 | Aug 2015 | US |