Patent Application
20200125200
The present invention relates to an apparatus and method for using a touchscreen to control and monitor a remote computer, which is connected via an Ethernet connection.
The combination of a client application running on a first computer, such as a standard Microsoft Windows application running on a personal computer (PC), which is used to control a second computer running a server application would be known to one skilled in the art as ‘remote management’ system.
The inventors investigated the use of existing remote management systems for suitability as a user interface for commercial and residential automation control products, such as those offered for sale by Crestron Electronics, Inc. During this investigation, the inventors discovered that existing remote management systems such as Microsoft Remote Desktop proved to be unsuitable for such an application for several reasons. Based on this research, the inventors were able to determine that in order to be suitable for use with commercial and residential automation control products, a remote management system would need the following characteristics:
One of the existing remote management systems, Virtual Network Computing (VNC) is an open source computer program that provides a graphical desktop sharing system that uses the Remote Frame Buffer protocol (RFB) to remotely control another computer.
Today, with LAN speeds of 1 GBps or higher, HDMI over LAN provides the possibility of connecting a relatively large number of screens through the same local area network with HDMI, VGA and DVI connections. (Also known as: ‘VGA over LAN’ or ‘DVI over LAN’).
It is to be understood that both the general and detailed descriptions that follow are exemplary and explanatory only and are not restrictive of the invention.
Just like the traditional VGA/HDMI/DP monitor or USB touchscreen, eTouchscreen client needs to be able to connect to the PC and display the output contents from PC once the touchscreen and PC are powered up and physically connected directly or indirectly via an Ethernet switch. In addition to supporting most of the Remote Management features, more efforts are needed to make the eTouchscreen behave like a traditional monitor plus other advanced Ethernet specific features such as waking up the PC from deep sleep (S3) and hibernate (S4) states via WoL. The first application of eTouchscreen solution would be turning the Mercury or TSW touch screen into an eTouchscreen for Rigel PC to server as the Rigel controlling console.
Crestron Remote enables an IT help desk to remotely view Crestron device's display output and control it via a secure channel. Advantageously, the inventive features that are incorporated into Crestron Remote can be incorporated into many products running Windows or Android operating systems.
Crestron Remote is similar in function to the open source application VNC and Microsoft Remote Desktop. Crestron Remote, however, utilizes inventive technology and thus provides better performance. More specifically, Crestron Remote provides additional required features that neither VNC nor Microsoft Remote Desktop offers. More specifically, Crestron Remote provides the following features:
Features
Sleep states for a personal computer (PC) are defined by the Advanced Configuration and Power Interface (ACPI) specification as shown in the table below. System level power states are denoted S0-S5. Higher S-numbers indicate deeper levels of sleep. The table below helps define the states:
Support H.264 encoding and decoding and RTSP protocol for communication to be consistent with Crestron streaming standard
High number of frames-per-second (fps) and low latency
As indicated in Table 1, both VNC (Open Source software) and Microsoft Remote Desktop only partially meet the above requirements.
One skilled in the art would recognize that certain remote desktop functionality can be provided by Remote Desktop functionality as shown in the table below.
There is a long-felt need to provide additional remote desktop functionality, for example
HDMI and DP monitors are the most popular display devices. Most of these devices don't support touch. Some monitors support touch, but a separate USB connection is needed between the display and the PC. Both display and USB cables are limited to relative short length without noticeable signal drop.
The emerging USB touch screen offers both video display and touch support with a single USB cable. Again the USB cable is limited to short lengths unless extra hardware is used to extend the cable length.
The accompanying figures further illustrate the present invention. Exemplary embodiments are illustrated in reference figures of the drawings. It is intended that the embodiments and figures disclosed herein are to be considered to illustrative rather than limiting.
The components in the drawings are not necessarily drawn to scale, emphasis instead being placed upon clearly illustrating the principles of the present invention. In the drawings, like reference numerals designate corresponding parts throughout the several views.
The following is a list of the major elements in the drawings in numerical order.
touch screen (running client application 21)
11 Ethernet (network connection)
13 appliance computer (running server application 31)
21 client (computer program running on touch screen 11)
22 ‘smart’ communication channel (between client 21 and server 23)
23 server (computer program running on appliance computer 13)
26 pop-up menu (of audio-video device specific commands 27)
27 audio-video device specific commands (supported by appliance computer 13)
30 user interface (displayed on touch screen 11)
32 alias combo box (p/o user interface 30)
36 connect button (p/o user interface 30)
38 clear history button (p/o user interface 30)
50 first connection channel (between client 21 and server 23)
60 first connection channel (between client 21 and server 23)
115 remote display (p/o touch screen 11)
311 user name combo box (p/o user interface 30)
312 password combo box (p/o user interface 30)
341 IP address or host name combo box (p/o user interface 30)
342 IP port combo box (p/o user interface 30)
1010 (step of) sending user credentials from client to server
1020 (step of) authenticating user credentials (at server)
1030 (step of) generating a token and saving that token to registry
1035 (step of) disconnecting client
1040 (step of) sending authentication response to client (from server)
1510 (step of) receiving authentication request token (at server)
1520 (step of) finding token in registry (at server)
1530 (step of) deleting token from registry (at server)
1535 (step of) aborting connection (between client and server)
1540 (step of) sending authentication response to client (from server)
2010 (step of) receiving description command (at server)
2020 (step of) authenticating description command (at server)
2030 (step of) sending description command response
2035 (step of) aborting connection (between client and server)
2510 (step of) reading in a saved device list
2520 (step of) determining whether saved device list is empty
2530 (step of) creating a saved device list entry and assigning a host name to alias (for this entry)
2601 (step of) determining whether an alias has been assigned (for each list entry)
2602 (step of) assigning a host name to an alias (for each list entry)
3010 (step of) selecting an alias from the alias list
3020 (step of) registering an event (“selected-index-changed”)
3030 (step of) updating properties corresponding to alias
3510 (step of) typing in a new alias (by end-user)
3520 (step of) registering an event (“selected-index-changed”)
3530 (step of) determining whether new alias name is in alias list
3540 (step of) updating properties corresponding to alias
4010 (step of) clicking on connect button (by end-user)
4020 (step of) determining whether host name or IP address has been entered
4040 (step of) updating properties corresponding to entered host name or IP Address
4045 (step of) showing invalid IP address message
4050 (step of) determining whether alias combo box is empty
4060 (step of) setting host address as alias
4062 (step of) initiating connection (between client and server)
4070 (step of) determining whether alias is associated with existing device
4080 (step of) updating properties corresponding to existing device
4082 (step of) initiating connection (between client and server)
4085 (step of) determining whether address is associated with existing device
4090 (step of) updating device properties in device list
4091 (step of) updating alias name in alias combo box
4092 (step of) initiating connection (between client and server)
4095 (step of) creating new device entry in device list
4096 (step of) adding new alias name to alias combo box
4097 (step of) initiating connection (between client and server)
The present invention is generally implemented as part of a networked computer system including a touch screen device that is located remotely form a computer that is under its control. Hence, such an illustrative networked computer system and its operation will be described initially.
Unless the context clearly requires otherwise, throughout the description and the claims, the words ‘comprise’, ‘comprising’, and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to”.
The preferred embodiment of the present invention is described herein in the context of a wall-mounted touchscreen device controlling a, but is not limited thereto, except as may be set forth expressly in the appended claims.
Refer now to
Crestron Remote includes Crestron proprietary applications on both server (Crestron products) and client (help desk PCs and tablets) sides. Two parent applications, CrestronRemoteClient on client side and CrestronRemoteService on server side, are responsible for product access authentication, configuration, as well as mouse and keyboard controlling. Each parent application owns a child application responsible for streaming the device display output to the client.
More specifically,
Once connected by the client 11, the server 13 will send the client a list of product specific commands 27 it supports. The user can then trigger these commands 27 via a local pop-up menu 26 on the touch screen 11. These commands 27 are crucial to manage a locked-down consumer appliance PC where no system menu is exposed.
Intelligent logic is built into the client 21 and server 23 applications to ensure prompt connection and reconnection and provide informative end-user notification. The client 21 can be configured to connect to a target appliance computer 13 with a static IP address once the touch screen 11 is powered up. It can also discover the target appliance computer 13 based on its hostname via a proprietary (alias) discovery protocol. Advantageously, automatic connection is always guaranteed whether there is a direct cable link between the touch screen 11 and the appliance computer 13 or they are connected via an Ethernet switch on a local area network (LAN).
When the appliance computer 13 cold starts, the server application 23 broadcasts an online message to all devices that are connected to the appliance computer 13, such as by a LAN. Once the client application 21 gets this message and recognizes that it is from the preconfigured appliance computer 13, it will connect to it immediately.
When the appliance computer 13 is about to reboot, the server 23 will send a notification to the client 21. Upon receiving this message, the client 21 will display an informative message on the display 115 of the touch screen 11. Similarly, when the appliance computer 13 is going into sleep or hibernate state, the client 21 will put itself into sleep mode as well upon receiving this notification.
By accurately tracking the state of the server application 23 that is hosted on the appliance computer 13, the client application 21, running on the touch screen 11 can properly wake up the appliance computer 13. While the appliance computer 13 is standby mode, touching on the display screen 115 or activation of the occupancy sensor on the touch screen 11 will trigger a touch message being sent to the appliance computer 13 to wake it up. If the appliance computer 13 is in deep sleep (S3) or hibernate (S4) mode the client 21 would send a magic WoL packet to the appliance computer 13 to wake it up and then reconnect to the appliance computer 13. Note that WoL from hybrid shutdown or fully powered off states is not supported for an appliance computer 13 running Windows 8 or above because users of these operating systems expect zero power consumption and battery drain in shutdown state.
Refer now to
In one exemplary embodiment, the client user interface 30 will start up at the resolution of 1280×800 pixels. The user can maximize the application window to full screen and then restore it to the original size. The view aspect ratio of the target system output is always maintained. So, depending on size of the client window one may see black band on left and right sides or top and bottom sides.
All mouse and keyboard messages, collectively ‘touch events’ 24, targeted to the client application 21 will be captured and sent to the remote service 23 running on the appliance computer 13 for controlling the appliance computer 13. Upon receiving the touch events 24 (mouse and keyboard messages), the remote service 23 will inject them into the local operating system, which subsequently routes the messages to the current foreground window. To the foreground window, these injected input messages are no different from those triggered via an external physical mouse and keyboard. It should be noted that, in order to activate the remote mouse and keyboard controlling, the client application must have focus and the mouse activities must be within the client application 21 client area.
The client 21 captures all touch events 24 from the touch screen 11 and sends these events over to the server 23. These touch events 24 are then injected into the operating system of the appliance computer 13 by the server 23 and processed by the operating system in a similar fashion as it does with the local touch events.
Intelligent logic is built into both client 21 and server 23 applications to ensure prompt connection once the appliance computer 13 and/or touch screen 11 starts up or physically connected just like a traditional video monitor does. In addition, informative notifications are displayed on the touch screen 11 display 115 whenever appropriate to inform the user of the appliance computer 13 state, connection status, as well as trouble shooting suggestions.
The client 21 can wake up the appliance computer 13 from standby mode when the user touches on the touch screen 11 display 115 or alternatively, when for example, an occupancy sensor on the touch screen 11 is activated. In addition, the client 23 can also wake up the appliance computer 13 from deep sleep (S3) or hibernation (S4) states by sending the magic Wake-On-LAN (WoL) packet. Note that WoL from hybrid shutdown or fully powered off states is not supported for an appliance computer 13 running Windows 8 or above because the associated end-users expect zero power consumption and battery drain in shutdown state.
Refer now to
Refer now to
As shown in
In a preferred embodiment of the present invention, when the top-level connection port is changed, such as via a Crestron Toolbox console command “CRPort” then the RTSP over TCP port number will automatically be changed to 1 plus the newly changed port number. For example, if the CRPort command assigns a port number 50000, then the RTSP over TCP port number will automatically become 50001.
According to the present invention, authentication is performed using a scheme, which is based on the administrator (admin) credentials assigned to the product by an end-user. For example, the credentials can be based on either a local admin account or a domain account. The authentication credentials provided by the user are sent to the remote service via a secure channel. The authentication request and response packets are detailed byte-by-byte in Tables 2 and 3, shown below.
Likewise, capability of the remote service is determined using a request-response scheme. The capability request and response packets are detailed byte-by-byte in Tables 4 and 5, shown below.
The number of displays and the display names associated with the remote service is also determined using a request-response scheme. The display output request and response packets are detailed byte-by-byte in Tables 6 and 7, shown below.
Command lists are exchanged with the remote service by using a request-response scheme. The remote command list request and response packets are detailed byte-by-byte in Tables 8 and 9, shown below.
In a preferred embodiment, all TCP communications, via the Internet, are over a secure channel that complies with the Federal Information Processing Standard (FIPS). For intranet usage, non-secure RTSP over UDP can be used in order to reduce overhead associated with encryption and decryption.
Upon receiving the connected display outputs, collectively ‘video frames’ 25, from the appliance computer 13, the client application 21 will dynamically build all display outputs, on the remote display 115, into a top-level display output menu. Once a display output is selected from the menu by the end-user, it will be sent over to the device via a remote command packet. Advantageously, this will allow an end-user the ability to switch and remotely view and control different device outputs.
In one embodiment, the eTouchscreen server 23 is created as a Windows service so it will start once the Windows operating system starts up. The server 23 can be configured to capture the desired display output and the captured video frames 25 are then encoded and streamed out to the client. Both capture and encoding are done by hardware accelerated technologies. The encoded video frames 25 are transferred to the client 21 via the efficient RTSP protocol. Frame throttling mechanisms are built into both server 23 and client 21 to minimize latency.
The touch events 24 are sent to the server 23 from the client 21 via TCP/IP protocol. Depending on the Windows logon state, proper code path is executed to inject the touch events into the operating system for the display output.
Once received by the client 21, the display output video frames 25 are decoded and displayed full screen on the touch screen 11 via hardware accelerated APIs. The client 21 also captures all user touch events 24 and sends them to the server 23 via TCP/IP protocol for processing. In addition, smart logic is built into the client 21 to allow the user to issue audio-video device specific commands 27 to the server 23.
Configuration commands are exchanged with the remote service by using a request-response scheme. The remote command request and response packets are detailed byte-by-byte in Tables 10 and 11, shown below.
Similar to display outputs from the appliance computer 13, the client application 21 will dynamically build all supported commands received from the appliance computer 13 into a top-level Command menu. Once a command is selected from the audio-video device specific menu 26 it will be sent over to the appliance computer 13 via the above remote command packet. Advantageously, this will allow the user to issue audio-video device specific commands 27 to the appliance computer 13 in order to trigger special functionality.
Note that the audio-video device specific commands 27 are defined and processed by the appliance computer 13. The number of commands 27 and the functionalities attached to them are independent of the client application 21. To the client application 21, these commands are just a list of strings passing around between the client application and the appliance computer 13. Advantageously, this design allows new commands to be added to the appliance computer 13 to allow operation of new and different audio-video devices without re-deploying the client application 21.
In one illustrative embodiment of the present invention, the appliance computer 13 display output is captured with Windows API BitBlt. The captured video frame 25 is then encoded into H.264 format and transmitted to the client 21 side via real-time streaming protocol (RTSP).
In one alternate embodiment, where the touch screen 11 is running the Microsoft Windows operating system, upon receiving the captured video stream from the appliance computer 13, the (Crestron Remote) client 21 decodes it using hardware accelerated Microsoft Media Foundation technology. The decoded video frames 25 are then color space converted to red, green, blue, alpha (RGBA) format and rendered to the touch screen 11 display 115 via Microsoft DirectX APIs.
The server 23 can be configured to capture the desired display output of the appliance computer 13. The captured video frames 25 are then encoded and streamed out to the client 23. Once received by the client 23, the video frames 25 are decoded and displayed full-screen on the touch screen 11. The capture, encoding, and decoding are all done via hardware accelerated technologies to achieve low latency, best performance, and operation efficiency.
If for any reason the touch screen loses connection to the PC, the client application 21 will output an informative message on the touch screen 11 to help the end-user trouble shoot the problem. Once the problem is resolved, a recovery mechanism built into the client 21 will allow the touch screen 11 to quickly reconnect to the appliance computer 13.
To ensure that only the dedicated touch screen 11 can connect to the appliance computer 13, connection authentication is provided as shown in
In one embodiment, the Crestron eTouchscreen 11 offers both display output and touch support with a single category 5 (Cat 5) Ethernet cable 12. The distance between the touch screen 11 and appliance computer 13 is essentially unlimited, which is an installation advantage. By creating a virtual display output on the appliance computer 13 for connection by the eTouchscreen 11, a physical display connector is not needed, which advantageously simplifies hardware design and reduces cost for developing a future appliance computer 13.
Refer now to
To solve the aforementioned problems, the present invention is a unique system in which a remote touchscreen is able to fully emulate a computer monitor, keyboard, and mouse and additionally provide a unique ‘functions’ menu.
The following is a list of the acronyms used in the specification in alphabetical order.
ACPI Advanced Configuration and Power Interface (industry specification)
Admin administrator (typically credentials)
API application programming interface
AV audio-video
BitBlt bit block transfer (computer graphics data operation)
CPU central processing unit (typically an integrated circuit)
CRPort Crestron remote port (command)
DP DisplayPort
DSP digital signal processor
DVI Digital Visual Interface (specification for computer video)
FIPS Federal Information Processing Standard
fps frames per second
H.264 MPEG-4 Part 10, Advanced Video Coding
HDMI High-Definition Multimedia Interface
iOS operating system for Apple IPad and iPhone devices
LAN local area network
MPEG motion picture experts group (standards setting organization)
NUC next unit of computing (small form factor computer by Intel)
OS operating system
PC personal computer
RFB remote frame buffer (protocol)
RGBA red, blue, green, alpha (color format where ‘alpha’ indicates opacity).
RL ‘Room Link’ (Crestron brand name)
RTSP real-time streaming protocol
S0 sleeping state (awake)
S1 sleeping state
S2 sleeping state
S3 sleeping state
S4 sleeping state
S5 sleeping state (requires reboot to awake)
TCP terminal control protocol
TSW touch screen, wall (Crestron part number nomenclature)
UDP user datagram protocol
UI user interface
USB universal serial bus (physical layer specification)
VGA Video Graphics Array (specification for computer video)
VNC Virtual Network Computing
WoL Wake-Up on LAN (computer networking standard)
Alternate embodiments may be devised without departing from the spirit or the scope of the invention. For example, the remote touchscreen could also provide audio input and output functionality similar to the well-known speaker and microphone interfaces provided with a personal computer. Also, multiple clients can connect to the same appliance PC simultaneously.
Other advantages of the present invention can also be realized, such as the elimination of distance limits when using the inventive technology with HDMI over LAN zero clients.
