Patent Application
20200019253
The present invention relates to a touch type operation apparatus, an operation method of the same, and an information processing system using the touch type operation apparatus.
A touch type operation apparatus such as a touch panel display that comprises a touch sensor and is operated on an operation screen displayed on a display is known (refer to JP2014-535086A, corresponding to US2013/076649A1).
While the shape of the display panel or the touch sensor of the touch panel display is generally a flat surface panel, a flat surface panel having a partially curved shape is developed using a flat surface panel having flexibility as disclosed in JP2014-535086A. In addition to curving the flat surface as in JP2014-535086A, a touch panel display in which a touch sensor or a display panel is shaped as a three-dimensional solid shape such as a cylindrical shape is suggested (JP2015-228118A, corresponding to US2017/108946A1).
Considering various shapes of touch type operation apparatuses as in JP2014-535086A and JP2015-228118A, designing the shape based on human engineering can provide a user with comfortable operability or improve work efficiency. Particularly, as the shape of the touch sensor or the display panel becomes more complicated as in JP2015-228118A than the shape obtained by simply curving the flat surface as in JP2014-535086A, operability using the characteristics of the solid shape can be provided, or an application using the solid shape is considered.
In the development of the touch type operation apparatus having a solid shape, a new problem arises unlike the flat surface panel. Types of gesture operations providing an operation input with respect to the flat surface panel include a tap operation of tapping an icon on the operation screen, a swipe operation for switching a screen or a page, a pinch operation of enlarging and reducing the screen or the page, and the like. Since the shape of the touch sensor or the display panel is a two-dimensional plane, coordinate information in the flat surface is enough for specifying such general types of gesture operations in the flat surface panel.
In the case of a three-dimensional solid shape as disclosed in JP2015-228118A, outer peripheral surfaces are configured with a plurality of surfaces. Thus, simply with only the coordinate information in each flat surface, the type of gesture operation cannot be specified, and an operation command corresponding to the type of gesture operation cannot be determined. The touch type operation apparatus having a three-dimensional solid shape has a complicated method of specifying the gesture operation. Thus, both the touch type operation apparatus having a three-dimensional solid shape and an application program adapting to the touch type operation apparatus are developed together, and versatility is sacrificed. Versatility means that one touch type operation apparatus is used in a plurality of application programs, or conversely, one application program is used in a plurality of touch type operation apparatuses.
Along with advances in development of connected cars and Internet of things (IOT) connectable to the Internet, various apparatuses are connected to the Internet, and various types of application programs having different applications are expected to be used using one touch type operation apparatus. In addition to the application programs, the touch type operation apparatus is expected to have various solid shapes, and one application program is expected to be used in various types of touch type operation apparatuses. In order to effectively use various types of application programs and touch type operation apparatus in combination, a system that secures versatility is important.
An object of the present invention is to provide a touch type operation apparatus, an operation method of the same, and an information processing system using the touch type operation apparatus capable of securing versatility by facilitating adaptation of a plurality of application programs in a touch type operation apparatus having a three-dimensional solid shape.
In order to resolve the above object, a touch type operation apparatus of the present invention comprises a touch sensor, a solid shape information obtaining unit, a gesture specification information generation unit, an application program interface unit, and a display control unit. The touch sensor has a three-dimensional solid shape. The solid shape information obtaining unit accesses a storage unit storing solid shape information related to the solid shape of the touch sensor and obtains the solid shape information. The gesture specification information generation unit specifies a content of a gesture operation performed with respect to the touch sensor based on the solid shape information and generates gesture specification information representing the specified content of the gesture operation. The application program interface unit communicates with an application execution unit executing an application program which uses the touch sensor as an input apparatus based on the gesture specification information and receives response information from the application execution unit. The display control unit controls display of a display based on the received response information.
It is preferable that the application execution unit executes an application program obtained by external communication or executes an application program provided as an application service from a server.
It is preferable that the application program interface unit transmits the gesture specification information to the application execution unit.
It is preferable that the display is integrated with the touch sensor and has the same solid shape as the touch sensor, and the display control unit performs the display control by performing transformation processing corresponding to the solid shape of the display on the response information received from the application execution unit.
It is preferable that the solid shape information includes at least one of a shape ID for identifying the solid shape, a surface ID for identifying a plurality of outer peripheral surfaces constituting the solid shape, plane coordinate information in a case where the solid shape is unfolded in a plane, or three-dimensional coordinate information related to the solid shape, and the response information includes information related to a response screen displayed as a processing result of processing executed by the application execution unit based on the gesture specification information and display position information related to a display position of the response screen.
It is preferable that the solid shape information further includes outer peripheral surface attribute information representing whether the outer peripheral surface is a flat surface or curved surface.
It is preferable that the application program interface unit determines an operation command with respect to the application program based on the gesture specification information and the solid shape information and transmits the operation command to the application execution unit.
It is preferable that the display is integrated with the touch sensor and has the same solid shape as the touch sensor, and the display control unit performs the display control by performing transformation processing corresponding to the solid shape of the display on the response information received from the application execution unit.
It is preferable that the solid shape information includes at least one of a shape ID for identifying the solid shape, a surface ID for identifying a plurality of outer peripheral surfaces constituting the solid shape, plane coordinate information in a case where the solid shape is unfolded in a plane, or three-dimensional coordinate information related to the solid shape, and the response information includes information related to a response screen displayed as a processing result of processing executed by the application execution unit based on the operation command and display position information related to a display position of the response screen.
It is preferable that the solid shape information further includes outer peripheral surface attribute information representing whether the outer peripheral surface is a flat surface or a curved surface.
An information processing system of the present invention using a touch type operation apparatus is an information processing system comprising the touch type operation apparatus, and an application program execution unit that executes an application program which uses the touch type operation apparatus as an input apparatus, in which the touch type operation apparatus comprises a touch sensor, a solid shape information obtaining unit, a gesture specification information generation unit, an application program interface unit, and a display control unit. The touch sensor has a three-dimensional solid shape. The solid shape information obtaining unit accesses a storage unit storing solid shape information related to the solid shape of the touch sensor and obtains the solid shape information. The gesture specification information generation unit specifies a content of a gesture operation performed with respect to the touch sensor based on the solid shape information and generates gesture specification information representing the specified content of the gesture operation. The application program interface unit communicates with an application execution unit executing an application program which uses the touch sensor as an input apparatus based on the gesture specification information and receives response information from the application execution unit. The display control unit controls display of a display based on the received response information.
It is preferable that the information processing system further comprises an application information obtaining unit that accesses a server distributing the application program or a server providing the application program as an application service and obtains the application program or information related to the application program.
It is preferable that the solid shape information includes at least one of a shape ID for identifying the solid shape, a surface ID for identifying a plurality of outer peripheral surfaces constituting the solid shape, plane coordinate information in a case where the solid shape is unfolded in a plane, or three-dimensional coordinate information related to the solid shape, and the response information includes information related to a response screen displayed as a processing result of processing executed by the application execution unit based on an operation command and display position information related to a display position of the response screen.
It is preferable that the information processing system further comprises an API information provision unit that provides the solid shape information to an API publication server publishing application program interface information to a developer developing the application program.
An operation method of a touch type operation apparatus of the present invention is an operation method of a touch type operation apparatus comprising a touch sensor having a three-dimensional solid shape. The method comprises a solid shape information obtaining step of accessing a storage unit storing solid shape information related to the solid shape of the touch sensor and obtaining the solid shape information, a gesture specification information generation step of specifying a content of a gesture operation performed with respect to the touch sensor based on the solid shape information and generating gesture specification information representing the specified content of the gesture operation, a communication step of communicating with an application execution unit executing an application program which uses the touch sensor as an input apparatus based on the gesture specification information and receiving response information from the application execution unit, and a display control step of controlling display of a display based on the received response information.
According to the present invention, the application program interface unit communicating with the application execution unit based on the gesture specification information corresponding to the solid shape is comprised in the touch type operation apparatus. Thus, one touch type operation apparatus is easily adapted to a plurality of application programs, or conversely, one application program is easily adapted to a plurality of touch type operation apparatuses. Versatility can be secured.
In
The instrument panel 11 includes, for example, a dashboard 16 extending in an approximately horizontal direction to a passenger seat 15 side from a driving seat 14 side in which a steering wheel 13 is arranged, and a center console 17 arranged between the driving seat 14 and the passenger seat 15.
The center console 17 comprises a lower part 17B arranged between the seat surfaces of the driving seat 14 and the passenger seat 15 and an upper part 17A raised upward from the lower part. The upper part 17A is connected to the dashboard 16 in the upper end portion thereof. For example, decorative plates that smoothly connect both of the parts 17A and 17B are disposed in the upper part 17A and the lower part 17B of the center console 17. A surface 17C of each of the parts 17A and 17B is a stepless connected surface.
The console system 10 is configured with a touch panel display (hereinafter, simply referred to as the touch panel) 21, a speaker 23, a console device control unit 24, and a main control device 26. Besides, an operation unit (not illustrated) configured with a mechanical switch is connected to the console system 10. The touch panel 21 and the console device control unit 24 constitute the touch type operation apparatus according to the embodiment of the present invention.
As previously known, the touch panel 21 is a device that is configured with a thin display such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display and a touch sensor arranged on a screen of the display and can receive an input of an operation instruction by a touch operation through an operation screen displayed on the display. A display region of the touch panel 21 extends on the whole surface of the center console 17 and has a size and a shape extending to the lower part 17B from the upper part 17A of the center console 17. That is, the whole surface 17C of the center console 17 functions as the touch sensor and the display.
As illustrated in
For example, a map display screen displaying a map of the navigation system and the operation screen for operating various APs such as the AV system are displayed on the touch panel 21. In addition, contents videos such as television broadcasts and movies are played back, and videos and the like having a decorative display effect are displayed on the touch panel 21. The map display screen and the main operation screen are displayed in a main display region 21A disposed in the upper part 17A.
The speaker 23 outputs various types of audio such as guidance of the navigation system, music played back by the AV system, and operation guidance of each AP. The operation guidance includes a sound effect representing reception of an operation, audio of reading a title of a selected menu item, and the like.
The console device control unit 24 comprises a touch panel control unit 32, an AP interface unit 33, and a memory 34.
The touch panel control unit 32 controls the display of the touch panel 21 and receives an input of an operation instruction through the touch operation with respect to the touch panel 21. The touch panel control unit 32 inputs an operation signal representing the received operation instruction into the AP interface unit 33. In addition, the touch panel control unit 32 receives response information such as a process result in response to the operation instruction from the AP interface unit 33 and controls the display of the touch panel 21 depending on the response information. Accordingly, the AP uses the touch panel 21 as an input apparatus for the operation signal.
The AP interface unit 33 communicates with an AP execution unit 41 of the main control device 26 and receives the response information from the AP execution unit 41 based on the operation signal input from the touch panel control unit 32. The AP interface unit 33 transmits the received response information to the touch panel control unit 32 or the speaker 23. In a case where the response information received from the AP execution unit 41 is screen information, the AP interface unit 33 transmits the response information to the touch panel control unit 32. In a case where the response information is audio information, the AP interface unit 33 transmits the response information to the speaker 23. That is, the AP interface unit 33 is an interface that relays information necessary for the processes of APs between the touch panel control unit 32 and the AP execution unit 41.
The touch panel control unit 32 and the AP interface unit 33 are accessibly connected to the memory 34. As will be described below, the memory 34 stores information that is referred to in the execution of the process of each of the units 32 and 33.
The main control device 26 manages and controls each unit of the automobile including the console system 10. The main control device 26 comprises the AP execution unit 41, a memory 42, and an AP information obtaining unit 43. The AP execution unit 41 is implemented using a central processing unit (CPU) 45. The CPU 45 functions as the execution unit 41 for each AP by loading and executing various APs such as the navigation system, the AV system, and the air conditioning system. In addition, the touch type operation apparatus configured with the touch panel 21 and the console device control unit 24, and the main control device 26 including the AP execution unit 41 constitute an information processing system according to the embodiment of the present invention.
The AP execution unit 41 is communicably connected to the AP interface unit 33 through a connector. The AP execution unit 41 transmits the operation screen for operating each AP to the AP interface unit 33 in order to display the operation screen on the touch panel 21. The AP execution unit 41 receives the operation signal input through an operation on the operation screen using the touch panel 21 or the solid region 22 from the AP interface unit 33 and executes a process corresponding to the received operation signal.
For example, the content of the process of each AP is as follows. In a case where the navigation system is started, the AP execution unit 41 executes a navigation process as follows. As the navigation process, first, a current location display process of displaying the current location of the automobile is executed. In the current location display process, the AP execution unit 41 determines the current location of the automobile based on a GPS signal obtained by a global positioning system (GPS) device, not illustrated, and transmits the map display screen corresponding to the current location to the AP interface unit 33.
In addition, as the navigation process, the AP execution unit 41 performs a destination setting process of implementing a destination setting function. In the destination setting process, in a case where the AP execution unit 41 receives an execution request for a route search process from the current location to a destination as the operation signal of the navigation system from the AP interface unit 33, the AP execution unit 41 executes the route search process by referring to a map database and the like. A search result is transmitted to the AP interface unit 33 as the response information. In a case where the destination is set by selecting a route, the AP execution unit 41 starts a guidance process of guiding the automobile along the route. In the guidance process, the AP execution unit 41 constantly transmits guidance information such as a guidance display screen and audio guidance corresponding to the current location of the automobile to the AP interface unit 33.
In addition, the AP execution unit 41 executes an accommodation search process of implementing an accommodation search function as the process of the navigation system. In a case where the AP execution unit 41 receives a search request for accommodations (for example, a gas station, a convenience store, and a hotel) near the current location as the operation signal, the AP execution unit 41 executes the accommodation search process by referring to an accommodation information database and the like and transmits a search result to the AP interface unit 33 as the response information.
In addition, the AP execution unit 41 starts the AV system and executes an AV process as follows. The AV process includes a playback process of receiving television broadcasting or radio broadcasting and playing back the received broadcasting and a playback process of playing back music or videos recorded in recording media such as a compact disc (CD) and a digital versatile disc (DVD). In addition, the console system 10 comprises an auxiliary (AUX) terminal that is a terminal to which an external apparatus such as a portable music player is connected. The console system 10 can perform a process of outputting a music played back in the external apparatus from the speaker 23.
The AP execution unit 41 transmits a video and audio of the television broadcasting and the radio broadcasting, a video and audio of the recording media and the external apparatus, and the like to the AP interface unit 33 as playback information for output from the touch panel 21 and the speaker 23. In a case where the AP execution unit 41 receives a channel selection request for the television broadcasting or the radio broadcasting from the AP interface unit 33, the AP execution unit 41 switches to the requested channel. In addition, in a case where the AP execution unit 41 receives a volume change request from the AP interface unit 33, the AP execution unit 41 adjusts the volume of the speaker 23 to the specified volume through the AP interface unit 33.
The AP execution unit 41 starts the air conditioning system and executes an air conditioning process of controlling the air conditioning apparatus 25 in the automobile. In a case where the AP execution unit 41 receives an adjustment request for adjusting the airflow amount or the temperature from the AP interface unit 33, the AP execution unit 41 executes a process of adjusting the airflow amount or the temperature by transmitting a control signal to the air conditioning apparatus 25.
In addition, in a case where an operation of channel selection or volume adjustment in the AV process or airflow amount adjustment or temperature adjustment in the air conditioning process is performed, the AP execution unit 41 performs a process of switching the screen of the touch panel 21 or changing a display content. Specifically, the AP execution unit 41 transmits the screen information to be displayed to the AP interface unit 33 as the response information.
The AP execution unit 41 supports multitasking and can execute a plurality of APs in parallel at the same time. For example, in a case where the main control device 26 is started by switching ON a main switch such as an ignition switch of the automobile, the navigation system is started. Then, the AV system and the air conditioning system are started based on the operation instruction, and each AP is executed in parallel at the same time. The memory 42 of the main control device 26 stores program data of the AP loaded in the AP execution unit 41, reference information referred to by the AP execution unit 41 in the execution of the AP, and the like.
In addition, the AP execution unit 41 can execute an externally obtained AP in addition to the preinstalled AP. The main control device 26 comprises a communication function of connecting to the Internet 46 and can download various APs from an AP distribution server 47 connected to the Internet 46.
In the case of downloading the AP, the AP execution unit 41 displays an AP selection screen as the operation screen on the touch panel 21 and receives a selection request from a user of the AP to be downloaded through the AP interface unit 33. The AP execution unit 41 transmits the received selection request to an AP information obtaining unit 43. The AP information obtaining unit 43 connects to the Internet 46 through a wireless communication unit 44 configured with an antenna, a transfer control circuit, and the like, accesses the AP distribution server 47, and obtains the AP from the AP distribution server 47. The AP information obtaining unit 43 is also implemented by loading and executing an operation program functioning as the main control device 26 by the CPU 45.
A current location mark 51A displaying the current location of the automobile, a route 51B to the set destination, a distance 51C to the destination, a destination setting button 51D, and the like are displayed on the map display screen 51.
In the touch panel 21, in a case where the touch operation of tapping the destination setting button 51D is performed, the screen transitions to a destination setting screen (not illustrated) from the map display screen 51. An input box for the destination and a keyboard and the like for setting the destination from a telephone number, an address, an accommodation name, and the like are displayed on the destination setting screen. In a case where the destination is set, the navigation system performs a route search, and the searched route 51B is displayed on the map display screen 51.
An AV menu button 52A is an operation button that calls a main menu enabling the operation of all functions of the AV system. A volume adjustment button 52B is an operation button for adjusting the volume of the speaker 23. A music selection button 52C is an operation button for performing music selection of selecting a music to be played back such as switching the music being played back to the previous or subsequent music in the AV system. The volume adjustment and the music selection are functions having a high frequency of use among the functions of the AV system. While the volume adjustment and the music selection can be operated from the menu screen, the volume adjustment button 52B and the music selection button 52C are dedicated buttons disposed for simplifying the operation of a function having a high frequency of use.
In addition, an air conditioner (A/C) menu button 52D is an operation button that calls a main menu enabling the operation of all functions of the air conditioning system. A temperature adjustment button 52E is an operation button for adjusting a target temperature. In the same manner as the volume adjustment button 52B and the like of the AV system, the temperature adjustment button 52E is a dedicated button disposed for a function having a high frequency of use. While illustration is not provided, dedicated buttons of the air conditioning system include an airflow amount adjustment button for adjusting the airflow amount.
In addition, while illustration is not provided, a main menu button is present on the menu screen 52. In a case where the main menu button is operated, a main menu (not illustrated) for setting or operating all functions of the navigation system, the AV system, and the air conditioning system is displayed. In a case where the main menu is displayed, for example, the screen transitions from the map display screen 51.
As illustrated in
In addition, as illustrated in
The operation screen of the console system 10 is operated through the touch operation with respect to the touch panel 21 on which the operation screen is displayed. As described above, while the operation screen is mainly displayed in the main display region 21A, the console system 10 can be operated through the touch operation with respect to the solid region 22 in addition to the touch operation with respect to the main display region 21A.
As illustrated in
As described above, the truncated cone portion 22A is a protruding region and is included in a rough solid shape region. The rough solid shape region refers to a protruding region or a recessed region in which at least a part of a boundary with an adjacent region is defined by a step. The step refers to a step such that the boundary can be sensed (touched) by touching with a hand. Specifically, it is preferable that the step has a height greater than or equal to the thickness of a finger (approximately 10 mm).
As illustrated in
In the case of the solid region 22 comprising the rough solid shape region such as the truncated cone portion 22A, an operator can recognize the location of the truncated cone portion 22A or recognize which part of the truncated cone portion 22A is touched through sense of touch with the hand HD without using vision. The input of the operation instruction can be received in the solid region 22 through various gesture operations as will be described below. Thus, by using the solid region 22, for example, a driver sitting in the driving seat 14 can operate the AP without using vision.
The touch panel 21 including the solid region 22 comprises, for example, an electrostatic capacitive type touch sensor that can perform multipoint detection of detecting a plurality of touch positions in an operation region at the same time. For example, a three-dimensional solid shape touch sensor including the rough solid shape region such as the solid region 22 can be manufactured using a manufacturing method disclosed in JP2016-213435A (corresponding to US2018/057943A1). Details of materials and the manufacturing method disclosed in JP2016-213435A are briefly summarized in
As illustrated in
The plated layer 67 is a compound having a functional group and the like interacting with a plating catalyst and the like and is, for example, the carboxylic acid group or the cyano group. The substrate 66 is a substrate that can be molded in a three-dimensional shape. The substrate 66 is made of a transparent resin such as polyethylene terephthalate (PET). The plated layer 67 is formed on the substrate 66 using a printing method such as screen printing.
As illustrated in
As illustrated in
A feature of the manufacturing method illustrated in
In the manufacturing method illustrated in
As previously known, in the electrostatic capacitive type touch sensor, a very low voltage is applied to the signal line 68 formed in a matrix shape in the initial state. In a case where the finger of the hand HD comes into contact with the touch sensor, an electrostatic capacitance at the touch position changes. The signal line 68 having a change in voltage is specified by detecting the change in voltage corresponding to the change in electrostatic capacitance. Since the signal line 68 is arranged in a matrix shape, the signal line 68 in an X direction and the signal line 68 in a Y direction intersecting with each other at the touch position are specified, and the touch position is detected.
In addition, as illustrated in
While
For example, the double tap is assigned to an operation of calling the menu screen 52 and an operation of the current location display. For example, in a case where the double tap is performed on the solid region 22 in a state where the map display screen 51 is displayed as illustrated in
For example, the single tap is assigned to a function of confirming selection of a menu item. In a case where the single tap is performed in a state where a menu item of any function is selected in the menu screen 52, the selection of the menu screen is confirmed.
For example, the rotation operation is assigned to a menu item selection operation in the menu screen 52. In a case where the menu screen 52 is displayed, a cursor is displayed at the position of any menu item. In a case where the rotation operation is performed, the cursor sequentially moves, and a menu item is selected.
For example, the lateral stroke is assigned to the operation of the airflow amount adjustment and the temperature adjustment in the air conditioning system and the operation of the volume adjustment and the music selection in the AV system. In a case where the lateral stroke is performed after the selection of the menu item such as the temperature adjustment or the volume adjustment is confirmed in the menu screen 52, the temperature or the volume is changed.
For example, the vertical pinch is assigned to an operation of closing the menu screen 52. In a case where the vertical pinch is performed in a state where the menu screen 52 is displayed on the map display screen 51, the display of the menu screen 52 is finished, and the menu screen 52 is closed.
In the present example, the types of gesture operations with respect to the solid region 22 and the assignment of the function to each gesture operation are for illustrative purposes. Types and assignment other than described above are also considered. For example, the tap operation, a slide operation, and the like with respect to the flat surface portion 22B and not the truncated cone portion 22A may be set distinctively from the gesture operations with respect to the truncated cone portion 22A.
In
In a case where the touch operation is performed on the solid region 22, a detection signal corresponding to the touched touch position is input into the gesture determination unit 32A. The detection signal is the output from matrix wiring disposed in the operation region of the solid region 22 and configured with each signal line 68 in the X direction and the Y direction. The detection signal represents the coordinates of an intersection between each signal line 68 in the X direction and the Y direction and corresponding to the touch position. Thus, the gesture determination unit 32A can specify the touch position in the solid region 22 from the detection signal.
The memory 34 that is a storage unit stores solid shape information table 71. A correspondence between the touch position and each portion of the solid shape is recorded in the solid shape information table 71 depending on the solid shape of the solid region 22. The gesture determination unit 32A reads solid shape information by accessing the memory 34 and referring to the solid shape information table 71. The gesture determination unit 32A functions as a solid shape information obtaining unit that obtains the solid shape information related to the solid shape of the touch sensor by accessing the memory 34 which is a storage unit.
The gesture determination unit 32A functions as a gesture specification information generation unit that specifies the content of the gesture operation by referring to the solid shape information table 71 and generates gesture specification information representing the specified content of the gesture operation based on the detection signal.
First, the solid shape information includes a shape identification data (ID). The shape ID is information for identifying the solid shape included in the solid region 22. The solid shape of the solid region 22 is a truncated cone, and “01” is assigned as the shape ID representing the truncated cone.
In addition, coordinates specified as the touch position in the matrix wiring of the solid region 22 based on the detection signal are recorded in the solid shape information table 71. While the touch position is represented by an identification number “1 to 100” for simplification in
For example, the surface ID is information for identifying each surface in a case where the outer peripheral surface of the solid shape is configured with a plurality of surfaces. The solid region 22 has three surfaces including an upper surface and a side surface on the outer peripheral surface of the truncated cone portion 22A and a surface corresponding to the flat surface portion 22B. In the present example, “001”, “002”, and “003” are assigned to the surface ID of the upper surface of the truncated cone portion 22A, the surface ID of the side surface, and the surface ID of the flat surface portion 22B, respectively.
In addition to the surface ID, outer peripheral surface attribute information (hereinafter, simply referred to as the surface attribute information) is recorded in the solid shape information table 71. For example, the surface attribute information is information as to whether each surface constituting the outer peripheral surface is the upper surface or the side surface of the truncated cone. In addition, the surface attribute information includes information representing whether each surface is a flat surface or a bent surface. In addition, not only a bent surface having a high curvature such as the side surface of the truncated cone due to processing is present, but also a curved surface having a low curvature is formed by slightly curving the flat surface portion 22B. The surface attribute information also includes information as to whether each surface is a flat surface or a curved surface obtained by slightly curving a flat surface. In the case of using information indicating a bent surface or a curved surface, it is preferable to add information related to the curvature. In order to appropriately display an image on the bent surface or the curved surface, it may be desirable to perform an image correction process such as coordinate transformation corresponding to the curvature. Thus, in the case of using the information representing the bent surface as the surface attribute information, it is preferable to include the curvature in the surface attribute information.
The gesture determination unit 32A can specify the type of solid shape and the surface on which the touch operation is performed based on the identification number (coordinate information) of the touch position, the shape ID, the surface ID, and the surface attribute information. While one ID is assigned to one surface in the present example, one surface may be divided into a plurality of blocks, and an ID may be assigned to each block. By doing so, it is possible to specify the block to which the touch position belongs in one surface.
In a case where a gesture operation such as the rotation operation illustrated in
In addition, in the case of the tap operation illustrated in
As described above, the gesture determination unit 32A generates the gesture specification information representing the specified content of the gesture operation. The gesture specification information includes the shape ID of the solid region 22, the surface ID, the surface attribute information, the type of gesture operation, the coordinates of the touch position, and the like.
In
As illustrated in
In addition,
The gesture determination unit 32A transmits the gesture specification information to the AP interface unit 33.
In
The display control unit 32B controls the display of the touch panel 21 including the solid region 22 based on the response information received from the AP interface unit 33.
The AP execution unit 41 comprises a command determination unit 41A and a response unit 41B. The command determination unit 41A determines an operation command based on the gesture specification information received from the AP interface unit 33. The operation command is an operation signal of the AP assigned to various gesture operations based on the content of the gesture operation.
The memory 42 of the main control device 26 stores a command determination table 72 (72A, 72B, 72C, . . . ). The command determination table 72 is table data in which a correspondence between the gesture operation such as “single tap”, “double tap”, and “rotation operation” and the operation command is recorded. The operation command includes “menu call/current location display”, “menu selection”, “music selection” and the adjustment of “volume” in the AV system, and the adjustment of “airflow amount”, “temperature”, and the like in the air conditioning system.
For example, an operation command providing an instruction to perform “menu call” or “current location display” to the AP execution unit 41 is assigned to the gesture operation of “double tap”. In addition, an operation command of “menu selection” is assigned to the gesture operation of “rotation operation”. An operation command providing an instruction to perform “music selection” or the adjustment of “volume”, “airflow amount”, or “temperature” depending on the selection state of the menu items is assigned to the gesture operation of “lateral stroke”.
While the operation command is assigned depending on the type of gesture operation such as the double tap or the rotation operation in the present example, different operation commands may be assigned to the same type of gesture operation depending on the surface on which the touch operation is performed. In this case, for example, different operation commands are assigned to the tap operation with respect to the upper surface and the tap operation with respect to the side surface.
In addition, as illustrated in
In a case where the shape of the touch panel 21 is a three-dimensional solid shape, the content of the gesture operation including the touch position and the type of gesture operation is complicated unlike the gesture operation with respect to a flat surface panel. The content of the gesture operation also changes in a case where the solid shape varies. Thus, a plurality of command determination tables 72 are prepared depending on the solid shape of the touch panel 21. Each of the command determination tables 72A, 72B, and 72C corresponding to the solid shape is different information depending on the solid shape and thus, is included in the solid shape information.
The shape ID representing the solid shape of the touch panel 21 is recorded as preset information 76 in the memory 42. For example, the preset information 76 is set in a case where the touch panel 21 is mounted with the main control device 26 at the time of manufacturing the automobile. The AP execution unit 41 recognizes the solid shape of the touch panel 21 by referring to the preset information 76.
The command determination unit 41A determines the operation command by referring to the command determination table 72 based on the content of the gesture operation input from the gesture determination unit 32A. The AP execution unit 41 executes each process in each AP of the navigation system, the AV system, and the air conditioning system depending on the operation command determined by the command determination unit 41A.
The response unit 41B generates the response information with respect to the operation command by referring to the command determination table 72. The command determination table 72 includes information related to the response screen as the response information with respect to the operation command. For example, the response information stored in the command determination table 72 is link information to a storage destination of image data of the response screen. The response information is prepared depending on the solid shape of the touch panel 21.
Information related to the response screen for the truncated cone corresponding to the truncated cone portion 22A is recorded in the command determination table 72A.
Information related to the response screen for the truncated pyramid corresponding to the truncated pyramid portion 81A is recorded in the command determination table 72B. Information related to the response screen for the hemisphere corresponding to the hemisphere portion is recorded in the command determination table 72C.
In the case of a flat surface panel, only the displayed screen size varies even in a case where the touch panel is changed. Thus, a process of matching the screen size of each touch panel by performing an enlargement or reduction process using the same image data can be performed. Thus, data of one response screen may be enough.
However, for example, as illustrated in
Therefore, in the console system 10, the response screen corresponding to the solid shape is prepared. The response information includes information related to the response screen depending on the solid shape and, for example, display position information related to a display position of the response screen and indicating the manner of dividing the image of the response screen, the surface of the solid shape on which a part of the response screen is to be displayed, and the part of the response screen to be displayed.
In
In addition, the display control unit 32B adjusts the display position and the like of the response screen in a detailed manner based on the surface attribute information as to whether the surface constituting the flat surface portion 22B of the solid region 22 is a flat surface or a slightly curved surface. Accordingly, appropriate display corresponding to the attribute of the surface can be performed.
In addition, the console device control unit 24 performs control for outputting audio from the speaker 23 based on the audio information included in the response information.
The memory 42 stores a plurality of APs executable by the AP execution unit 41. Data of each AP includes the content of the command determination table 72 corresponding to the solid shape and specifically, includes the correspondence between the gesture operation and the operation command and information related to a user interface (UI) including the response screen with respect to the operation command. The information related to the UI includes, for example, UI1 in the case of a typical flat surface panel, UI2 for the truncated cone, UI3 for the truncated pyramid, and UI4 for the hemisphere.
For example, information related to the command determination table 72 corresponding to the solid shape is embedded in advance in the preinstalled AP. A developer of the AP develops an appropriate UI and the like based on the specifications of the solid shape of the solid region of the touch panel manufactured by each maker. Data of the UI and the like are embedded in advance in the AP. Each of the command determination tables 72A, 72B, 72C, . . . is loaded into the memory 42 in a case where the AP is installed on the main control device 26. Accordingly, one AP can be adapted to a plurality of solid shapes of the touch panel 21.
The memory 42 stores a plurality of AP1, AP2, . . . . While illustration is not provided, any of the APs comprises the command determination table 72 corresponding to a plurality of solid shapes.
Hereinafter, an effect of the above configuration will be described with reference to the flowcharts illustrated in
In the automobile in which the console system 10 is mounted, in a case where the main switch such as the ignition switch is operated, a power supply of the console system 10 is switched ON. In a case where the power supply is switched ON, a power supply of the main control device 26 including the AP execution unit 41 is switched ON as illustrated in
As illustrated in
As illustrated in
In
The AP execution unit 41 executes a process corresponding to the determined operation command (S270). The response unit 41B transmits the response screen corresponding to the solid shape and the operation command to the AP interface unit 33 by referring to the command determination table 72 (S280). In the present example, the response unit 41B transmits the response screen for the truncated cone.
In
The touch panel control unit 32 specifies the content of the gesture operation based on the solid shape information related to the touch panel 21, generates the gesture specification information representing the content, and communicates with the AP execution unit 41 based on the generated gesture specification information. The display of the touch panel 21 is controlled based on the response information from the AP execution unit 41. The touch panel control unit 32 comprises the AP interface unit 33 transmitting the gesture specification information generated depending on the solid shape to the AP execution unit 41. Thus, the AP execution unit 41 can perform an appropriate process depending on the solid shape. Thus, adaptation of the plurality of APs executed by the AP execution unit 41 is facilitated.
In addition, in a case where the touch panel control unit 32 comprises the AP interface unit 33 outputting the gesture specification information corresponding to the solid shape, information communicated between a device side such as the touch panel control unit 32 and the AP execution unit 41 is clarified. Thus, for the developer developing the AP, development of the AP matching various solid shapes of the touch panel is facilitated.
By comprising the AP interface unit 33 in the touch type operation apparatus, one touch type operation apparatus is easily adapted to a plurality of APs, or conversely, one AP is easily adapted to a plurality of touch type operation apparatuses. Versatility can be secured.
As described above, the AP execution unit 41 of the present example can execute the AP obtained by external communication in addition to the preinstalled AP. In this case, various APs and various solid shapes of the touch panel are used in combination. In such a case, the touch type operation apparatus of the present example is particularly effective.
In the present example, the AP execution unit 41 can execute the AP downloaded from the AP distribution server 47 through the AP information obtaining unit 43. Besides, for example, an application service provider (ASP) server providing an application service through the Internet 46 may be used as the AP distribution server 47. The AP information obtaining unit 43 may access the ASP server, and the AP execution unit 41 may execute the AP provided as the application service. That is, the AP information obtaining unit 43 and the AP execution unit 41 may be able to use a so-called cloud service.
In such a case, the AP information obtaining unit 43 does not download the whole data of the AP and install the AP on the main control device 26. The AP information obtaining unit 43 communicates with the ASP server in real time and sequentially obtains necessary data related to the AP during the process of the AP execution unit 41. The AP execution unit 41 executes a process based on the sequentially obtained data related to the AP. In this case, for example, the AP information obtaining unit 43 and the AP execution unit 41 as a platform for executing the AP are provided with the application service of the ASP server using a web browser.
In addition, the AP information obtaining unit 43 may obtain the AP depending on the solid shape. In this case, the AP information obtaining unit 43 transmits the preset shape ID to the AP distribution server 47. The AP distribution server 47 extracts the AP based on the shape ID and distributes the AP to the AP information obtaining unit 43. By doing so, the AP information obtaining unit 43 can download the AP corresponding to the solid shape of the touch panel 21. In addition, in the same manner as the provision of the application service from the ASP server, the main control device 26 can be provided with the application service corresponding to the solid shape by causing the AP information obtaining unit 43 to transmit the solid shape to the ASP server.
In addition, the installation of the AP may not be performed through the Internet 46. The AP may be installed on the main control device 26 through removable media such as a Universal Serial Bus (USB) memory or by performing short range wireless communication with a smartphone. The USB memory and the short range wireless communication are included in the external communication.
In addition, in the present example, for example, the shape ID for identifying the solid shape, the surface ID for identifying the outer peripheral surface constituting the solid shape, the surface attribute information representing the attribute of the outer peripheral surface, the coordinates of the touch position based on the wiring of the signal line 68 in a matrix shape, and the like are described as the solid shape information. Besides, any information that can specify the content of the gesture operation with respect to the touch sensor having the solid shape may be used as the solid shape information. For example, the plane coordinate information in the case of unfolding the solid shape in a plane, the three-dimensional coordinate information related to the solid shape, and the like may be used. In addition, the information which can specify the content of the gesture operation does not need to include all of the kinds of solid shape information and may include at least one thereof.
In a console system 90 of a second embodiment illustrated in
As illustrated in
In
In a case where the gesture determination unit 32A receives the detection signal from the touchpad 92 having a truncated cone solid shape, the gesture determination unit 32A specifies the content of the gesture operation by referring to the solid shape information table 71. The gesture specification information is generated. The gesture specification information is transmitted to the AP execution unit 41 from the AP interface unit 33. In a case where the AP execution unit 41 receives the gesture specification information as the operation signal, the AP execution unit 41 determines the operation command by referring to the command determination table 72. The AP execution unit 41 transmits the response information corresponding to the operation command to the AP interface unit 33. Since the touch panel 91 is a flat surface panel, the response information includes information related to the response screen for the flat surface. The touch panel control unit 93 controls the display of the touch panel 91 based on the received response information.
Even in the case of using a flat surface panel as the touch panel 91, the screen may be a curved surface by curving the flat surface panel. In that case, as described above, it is preferable to adjust information related to the response screen for the flat surface in a detailed manner such that appropriate display is performed on the curved surface.
A third embodiment illustrated in
A command determination unit 33A is disposed in the AP interface unit 33. The command determination table 72 is stored in the memory 34 on the device side including the solid region 22 and not in the memory 42 of the main control device 26. Only the command determination table 72A for the truncated cone corresponding to the shape of the solid region 22 is stored as the command determination table 72.
An AP including only a UI0 for the flat surface is installed on the main control device 26. The memory 42 stores a command/response information correspondence table 96 storing the correspondence between the operation command and the response information. The response information for the flat surface is recorded in the table 96. In the third embodiment, the AP execution unit 41 does not recognize the shape of the solid region 22 and executes processing regardless of the shape of the solid region 22 by regarding the solid region 22 as a touch panel of a typical flat surface panel.
In the flowcharts of the third embodiment illustrated in
In S151 in
In S250 in
In S170 in
The response screen corresponding to the solid shape is one kind of solid shape information. As described above, the solid region 22 of the touch panel 21 comprises the display 70 which is integrated with the touch sensor 69 and has the same solid shape as the solid shape of the touch sensor 69. Thus, in the third embodiment, the display control unit 32B performs display control by performing the screen transformation processing corresponding to the solid shape of the display 70 integrated with the touch sensor 69. For example, as illustrated in
In the third embodiment, the command determination and the screen transformation corresponding to the solid shape of the touch panel 21 are performed on the device side comprising the touch panel 21 and the touch panel control unit 32. The AP execution unit 41 does not recognize the solid shape of the touch panel 21. Thus, the AP can be developed without considering the solid shape of the touch panel 21. In addition, even in a case where the touch panel 21 has any solid shape, the existing AP can be used. In the same manner as the first embodiment, even in the third embodiment, one touch type operation apparatus is easily adapted to a plurality of APs, or conversely, one AP is easily adapted to a plurality of touch type operation apparatuses. Versatility can be secured.
A fourth embodiment illustrated in
A fifth embodiment illustrated in
A user 101 using car sharing feels comfortable in a case where an AP of a smartphone 102 used daily by the user 101 can be used in the automobile used in car sharing. The AP information obtaining unit 43 of the fifth embodiment comprises a function of communicating with the smartphone 102 of the user 101 and downloading the AP used in the smartphone 102 in a case where user authentication is performed in the automobile.
The user 101 using car sharing makes a reservation for car sharing on a reservation site provided by a car share management server 104 through the smartphone 102 and obtains authentication information. The user 101 unlocks the automobile using the obtained authentication information and enters the automobile. At this point, by consent of the user 101, the AP information obtaining unit 43 refers to the authentication information by accessing the smartphone 102 and confirms that the user 101 is the user 101 having a reservation for car sharing. The AP information obtaining unit 43 obtains a terminal ID of the smartphone 102 and obtains registered AP information by accessing a terminal management server 103 managing information related to the AP installed on the smartphone 102. The AP information obtaining unit 43 downloads the same AP as the obtained registered AP information by accessing the AP distribution server 47.
By doing so, the user 101 can use the daily used AP in the automobile used in car sharing. Not all APs used in the smartphone 102 can be used in the automobile. Thus, the solid shape information may be transmitted to the AP distribution server 47 from the AP information obtaining unit 43 based on the preset information 76, and the usable AP may be extracted using the solid shape information in the AP distribution server 47.
An information processing system of a sixth embodiment illustrated in
(Specification Information Included in Solid Shape Information)
In the case of providing the solid shape information as the API information, it is preferable to include specification information illustrated in
As illustrated in
As illustrated in
The specification information is also effective in a case where the AP does not recognize the solid shape of the touch sensor as in the third embodiment and the fourth embodiment. Even in a case where the solid shape varies like the truncated cone, the truncated pyramid, and the hemisphere, the AP appropriate for the solid shapes can be developed in a case where the number of operation regions to which the menu items are assigned does not change.
While an example in which the touch type operation apparatus is mounted in the automobile is described above, the apparatus in which the touch type operation apparatus is mounted may not be the automobile. For example, the touch type operation apparatus may be applied to home appliances such as a refrigerator. A case where a touch sensor or a touch panel having a three-dimensional solid shape is used in an operation panel of a refrigerator is considered.
In the home appliances such as a refrigerator, in a case where the AP is downloaded from the AP distribution server 47, the download may be performed as follows. For example, in the AP distribution server 47, the AP is registered by storing a correspondence between a model number of the refrigerator and the AP. The AP information obtaining unit 43 mounted in the refrigerator transmits the model number of the refrigerator to the AP distribution server 47 in the case of downloading the AP. The AP distribution server 47 extracts an AP usable in the refrigerator based on the received model number and distributes the AP to the AP information obtaining unit 43. By doing so, the AP corresponding to the solid shape of the operation panel of the refrigerator can be downloaded. In addition, the model number of the refrigerator may be read using a short range wireless communication function of the smartphone. The model number of the refrigerator may be transmitted to the AP distribution server 47 from the smartphone, and the AP may be downloaded. The AP may be transferred to the refrigerator from the smartphone.
In each embodiment described above, for example, a hardware structure of processing units executing various processes like the console device control unit 24, the touch panel control unit 32, the gesture determination unit 32A, the command determination unit 41A, the display control unit 32B, and the AP interface unit 33 correspond to various processors as follows.
The various processors include a central processing unit (CPU), a programmable logic device (PLD), a dedicated electric circuit, and the like. The CPU is a versatile processor functioning as various processing units by executing software (operation program) as previously known. The PLD is a processor such as a field programmable gate array (FPGA) capable of changing a circuit configuration after manufacturing. The dedicated electric circuit is a processor having a circuit configuration such as an application specific integrated circuit (ASIC) dedicatedly designed to execute a specific process.
One processing unit may be configured with one of the various processors or may be configured with a combination of two or more processors of the same kind or different kinds (for example, a plurality of FPGAs or a combination of the CPU and the FPGA). In addition, a plurality of processing units may be configured with one processor. A first example of configuring a plurality of processing units with one processor is such that one processor is configured with a combination of one or more CPUs and software, and the processor functions as a plurality of processing units. A second example is such that as represented by a system on chip (SoC) or the like, a processor that implements the function of the whole system including the plurality of processing units using one integrated circuit (IC) is used. Various processing units are configured using one or more of the various processors as a hardware structure.
Furthermore, the hardware structure of the various processors is more specifically circuitry in which circuit elements such as a semiconductor element are combined.
Touch type operation apparatuses disclosed in the following appendix can be perceived from the above disclosure.
“Appendix 1”
A touch type operation apparatus comprising a touch sensor having a three-dimensional solid shape, a solid shape information obtaining processor that accesses a storage unit storing solid shape information related to the solid shape of the touch sensor and obtains the solid shape information, a gesture specification information generation processor that specifies a content of a gesture operation performed with respect to the touch sensor based on the solid shape information and generates gesture specification information representing the specified content of the gesture operation, an application program interface processor that communicates with an application execution unit executing an application program which uses the touch sensor as an input apparatus based on the gesture specification information and receives response information from the application execution unit, and a display control processor that controls display of a display based on the received response information.
“Appendix 2”
An information processing system using a touch type operation apparatus, the system comprising the touch type operation apparatus, and an application program execution processor that executes an application program which uses the touch type operation apparatus as an input apparatus, in which the touch type operation apparatus comprises a touch sensor having a three-dimensional solid shape, a solid shape information obtaining processor that accesses a storage unit storing solid shape information related to the solid shape of the touch sensor and obtains the solid shape information, a gesture specification information generation processor that specifies a content of a gesture operation performed with respect to the touch sensor based on the solid shape information and generates gesture specification information representing the specified content of the gesture operation, an application program interface processor that communicates with an application execution unit executing an application program which uses the touch sensor as an input apparatus based on the gesture specification information and receives response information from the application execution unit, and a display control processor that controls display of a display based on the received response information.
The present invention is not limited to each embodiment described above and may employ various configurations without departing from the nature of the present invention. In addition, the above various embodiments and various modification examples can be appropriately combined. In addition, the present invention includes a program and a storage medium storing the program.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2017-065953 | Mar 2017 | JP | national |
This application is a Continuation of PCT International Application No. PCT/JP2018/005041 filed on 14 Feb. 2018, which claims priority under 35 U.S.C § 119(a) to Japanese Patent Application No. 2017-065953 filed on 29 Mar. 2017. The above application is hereby expressly incorporated by reference, in its entirety, into the present application.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2018/005041 | Feb 2018 | US |
| Child | 16583088 | US |
