The present invention relates to an information processing apparatus for generating a user interface required for a manipulation device to manipulate a device to be manipulated, its control method, and a program.
Various user interface adaptation techniques have been conventionally studied. For example, a technique that supports macro creation, sorts selection candidates, or estimates the next manipulation by learning user's previous manipulation histories is known.
On the other hand, in the Web world, a technique for applying contents according to the properties of devices that access the Web is known. For example, in Device Independence Activity (http://www.w3.org/2001/di/Activity) of W3C (World Wide Web Consortium, http://www.w3.org/), the format of profile information that describes device capabilities and user preferences, i.e., CC/PP (Composite Capabilities/Preferences Profile), and the specification of a protocol that makes negotiations between devices using that format are being laid down.
Upon presenting contents of a user interface, a technique for generating those contents by transforming a source file using a stylesheet in accordance with user preferences and device properties is known. For example, Japanese Patent Laid-Open No. 2001-154852 has proposed a technique that separately describes contents of a user interface on a stylesheet as presentation and interaction, and generates the contents of the user interface using XSLT. On the other hand, Japanese Patent Laid-Open No. 2001-344230 has proposed a technique that transforms elements of a logical document on the basis of rules which associates them with those of a style which designates a presentation method.
The user interface adaptation technique based on the prior art and, especially, the Web contents adaptation technique have as their principal object to mainly customize “display” such as the type of browser, window size, and the like with respect to the properties of a manipulation device, but they cannot customize logic or use modalities as the user interface in consideration of the properties of the manipulation device or user's properties/preferences.
For example, the above technique cannot cope with adaptation that selects a system-initiative flow for a user who is not familiar with manipulations or a user-initiative flow for a user who is familiar with manipulations. In consideration of adaptation in a multimodal user interface as a combination of a plurality of modalities such as a GUI (graphic user interface), speech, and the like, it is important to dynamically change use modalities (e.g., to provide a speech-based user interface to a vision-impaired person), but such adaptation cannot be made by the prior art.
The present invention has been made in consideration of the aforementioned problems, and has as its object to provide an information processing apparatus which can improve usability of manipulations to a device to be manipulated, its control method, and program.
According to the present invention, the foregoing object is attained by providing an information processing apparatus for generating a user interface required for a manipulation device to manipulate a device to be manipulated, comprising:
In a preferred embodiment, the source document, the transformation description components, and the contents of the user interface are described in an XML language.
In a preferred embodiment, the transformation description components and the transformation description are described as stylesheets.
In a preferred embodiment, the manipulation device is the information processing apparatus.
In a preferred embodiment, the manipulation device is independent from the device to be manipulated, and
In a preferred embodiment, the transmission means transmits the profile information input to the manipulation device to the information processing apparatus.
In a preferred embodiment, the transmission means transmits the profile information, which is generated based on a user's manipulation history on the manipulation device, to the information processing apparatus.
In a preferred embodiment, the manipulation device comprises acquisition means for acquiring the profile information corresponding to the user of the manipulation device, and
According to the present invention, the foregoing object is attained by providing a method of controlling an information processing apparatus for generating a user interface required for a manipulation device to manipulate a device to be manipulated, comprising:
According to the present invention, the foregoing object is attained by providing a program that implements control of an information processing apparatus for generating a user interface required for a manipulation device to manipulate a device to be manipulated, comprising:
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
Preferred embodiments of the present invention will now be described in detail in accordance with the accompanying drawings.
In this embodiment, a case will be exemplified below wherein a PDA is assumed as a manipulation device as a building component of an information processing system, and a copying machine is assumed as a device to be manipulated by that manipulation device.
Note that this embodiment will explain the manipulation device and device to be manipulated as independent devices. However, the present invention can be applied to a device which is configured by integrating the manipulation device and device to be manipulated.
In a manipulation device 100 shown in
Reference numeral 108 denotes a contents execution unit of a user interface (UI) that executes the contents of the user interface, which are transmitted from the device 200 to be manipulated and are described in a multimodal markup language (to be abbreviated as MMML hereinafter), using a multimodal browser that allows inputs/outputs by means of speech and GUI.
Reference numeral 109 denotes a speech input/output unit which inputs/outputs speech to the contents of the user interface executed by the contents execution unit 108. Reference numeral 110 denotes a display unit which displays the contents of the user interface executed by the contents execution unit 108.
Reference numeral 111 denotes a GUI input unit which makes a GUI input based on the contents of the user interface executed by the contents execution unit 108.
On the other hand, in the device 200 to be manipulated shown in
Reference numeral 105 denotes a source document selection unit, which selects a source document from a source document holding unit 106 using the profile information parsed by the profile information parsing unit 102. Reference numeral 106 denotes a source document holding unit which holds source document that describe flows of system-initiative manipulations, user-initiative manipulations, and the like.
Reference numeral 107 denotes a transformation nit (XSLT: XSL Transformation), which transforms a source document selected by the source document selection unit 105 into MMML as an XML description language of a multimodal user interface using a stylesheet generated by the stylesheet generation unit 103, and transmits the transformation result to the contents execution unit 108 of the manipulation device 100. Note that the multimodal user interface is a user interface that allows inputs/outputs by means of a plurality of modalities such as speech, GUI, and the like.
Note that the source document includes a description of abstract manipulation (UI) components, which are independent from concrete manipulation (UI) components (modalities (GUI component and speech input/output components)) required to implement the user interface of the manipulation device 100. The stylesheet includes a description that transforms abstract manipulation components in the source document into concrete manipulation components of the manipulation device 100.
The hardware arrangement of the manipulation device 100 will be described below using
Referring to
Reference numeral 204 denotes a liquid crystal display device (LCD) which displays text, images, and the like. Note that another display device such as a CRT or the like may be used in place of this LCD 204. Reference numeral 205 denotes a touch panel which implements various data inputs and manipulations of the user interface. Operations to this touch panel are made by the user's finger or a dedicated pan. In addition to this touch panel 205, a keyboard and pointing device may be equipped.
Reference numeral 206 denotes a loudspeaker, which outputs synthetic speech. Reference numeral 207 denotes a rewritable external storage device. In this embodiment, a hard disk drive (HDD) is used as the external storage device 207. The HDD 207 stores various programs including, e.g., a program of a browser or the like which implements the user interface.
Reference numeral 208 denotes a microphone used to make a speech input for speech recognition. Reference numeral 209 denotes a bus which interconnects the respective building components of the manipulation device 100.
The hardware arrangement of the device 200 to be manipulated will be described below using
Referring to
Reference numeral 304 denotes a liquid crystal display device (LCD) which displays text, images, and the like. Note that another display device such as a CRT or the like may be used in place of this LCD 304. Reference numeral 305 denotes an input device, which includes, e.g., a ten-key pad and buttons. In addition, a pointing device and keyboard may be used.
Reference numeral 306 denotes a printing device which comprises, e.g., a laser beam printing device. In addition, an ink-jet printing device or thermal transfer printing device may be used. Reference numeral 307 denotes a rewritable external storage device. In this embodiment, a hard disk drive (HDD) is used as the external storage device 307. The HDD 307 stores various data such as stylesheets, source documents, and the like.
Reference numeral 308 denotes a bus which interconnects the building components of the device 200 to be manipulated.
The process to be executed by the information processing system of this embodiment will be described below using
The manipulation device 100 acquires manipulation device/user information including property information indicating the properties of the manipulation device itself and user information associated with the properties and preferences of the user (step S401). This embodiment adopts a method of inputting this manipulation device information from an input window implemented by a GUI on the LCD 204 by the user himself or herself. The profile information generation unit 101 generates profile information, which describes, in XML, information such as a use language, the screen size of the manipulation device used, familiarity, disability information, and the like, as shown in
This profile information is transmitted to the device 200 to be manipulated as a POST request of an HTTP message (step S403).
Note that the profile information shown in
On the other hand, the device 200 to be manipulated receives the profile information from the manipulation device 100 (step S404). The profile information parsing unit 102 parses the profile information using an XML parser (step S405).
The source document selection unit 105 selects an appropriate source document from those which are held in the source document holding unit 106 and describe a flow shown in
Note that a source document is an XML document which uses XHTML as a container. This document describes data models and abstract user interface components (to be referred to as abstract UI components) using XForms (http://www.w3.org/TR/xforms/) as the specification of W3C, and an event-driven flow using XML Events (http://www.w3.org/TR/xml-events/) as the specification of W3C. As the contents of the flow, a user-initiative flow, system-initiative flow, mixed-initiative flow of them, and the like are known as typical interactive patterns, and they are adopted as flows.
On the other hand, the stylesheet holding unit 104 holds components of stylesheets (to be referred to as stylesheet components) used to transform abstract UI components included in a source document (interaction flow description) into concrete UI components (to be referred to as concrete UI components hereinafter). Note that each stylesheet component serves as a transformation description component used to transform a description of an abstract UI component in a source document into that of a concrete UI component.
For example, a source document describes an abstract UI component that selects one of a plurality of options like paper size selection of a copying machine using an element “select1 (XForms)”. A stylesheet used to transform this component into a concrete UI component such as a GUI pull-down menu, speech input, or the like is a stylesheet component held in the stylesheet holding unit 104 (
More specifically, the stylesheet holding unit 104 holds a set of such stylesheet components. The stylesheet generation unit 103 selects appropriate stylesheet components from the set of these stylesheet components in accordance with the profile information (step S407), as shown in
For example, if it is parsed that the manipulation device has a speech input/output function and the user of interest is a vision-impaired person, a stylesheet component that transforms into a concrete UI component (e.g., speech synthesis/recognition component) is selected. The selected stylesheets (transformation description components) are integrated to dynamically generate a final stylesheet (transformation description) (step S408).
Note that rules, i.e., which of stylesheet components is to be selected in accordance with the profile information, are described by the developer of the user interface. As the description method, the rules may be directly described using a programming language or may be declaratively described in a predetermined description format. However, the method itself falls outside the scope of the present invention, and a detailed description thereof will be omitted.
The transformation unit 107 transforms the source document selected in step S406 using the generated stylesheet and an XSLT processor (step S409). In this manner, the final contents of the user interface are dynamically generated (step S410). The generated contents of the user interface are transmitted to the manipulation device 100 (step S411).
The manipulation device 100 receives the contents from the device 200 to be manipulated (step S412). The manipulation device 100 executes the received contents of the user interface by the contents execution unit 108 (step S413). In this manner, the user interface required to manipulate the device 200 to be manipulated in accordance with the properties of the manipulation device 100 and user's properties/preferences can be implemented on the manipulation device.
An example of the source document of the device 200 to be manipulated will be explained below using
Especially,
In
(ii) is a description associated with an input of the number of copies, and an abstract UI component used to input a value is described using an element “input (XForms)”.
(iii) is a description associated with paper selection, and an abstract UI component used to select one of a plurality of options is described using an element “select1 (XForms)”.
Likewise, (iv) is a description associated with the magnification, (v) is a description associated with the double-sided setup, and (vi) is a description associated with the density setup.
<String-XXXX/> at the head of each of the descriptions (iii) to (vi) (e.g., <String-PaperSize/> in (iii)) describes an output of a character string, and is added under the assumption that such character string is transformed into that of an appropriate language in accordance with the use language.
Stylesheet components held in the stylesheet holding unit 104 will be described in detail below.
Especially,
For example, a template (i) in
A template (ii) in
Especially,
For example, “input” in (ii) in
Also, “select1” in (iii) in
Especially,
For example, when a GUI text box and speech input are bound to the setup of the number of copies, an action that activates a corresponding speech input (to set in an inputtable state) when the text box is clicked and focused is typical in the multimodal user interface.
The role of this stylesheet component is to generate a description of such synchronization. Since MMML of this embodiment defines to describe an event using XML Events, this stylesheet component becomes a set of templates that match respective abstract UI components of the source document and output descriptions of XML Events.
Note that (i) in
The stylesheet generation unit 103 selects appropriate stylesheet components from, e.g., the set of stylesheet components shown in
A dynamically generated stylesheet will be described with reference to
(i) in
With this parameter, text to be displayed on the screen, recognition lexical items, and synthetic speech are changed in accordance with the use language. <screen-height></screen-height>and <screen-width></screen-width> in
The screen size, caption, GUI form size, and the like of the browser are changed in accordance with these values. Parameters “gui”, “listen”, “speech”, and the like in (i) in
Since the profile information in
(ii) in
Likewise, if <asr></asr>and <tts></tts> in
By transforming the source document in
A description of the contents of the final user interface to be generated will be described below with reference to
(i) in
(ii) in
(iii) in
(iv) in
(v) in
That is, upon clicking a GUI form, the speech input component is activated to start speech recognition, and the speech recognition result is bound to parameters to fill the clicked GUI form with the recognition result. Also, the form can be filled by direct inputs or selection from a pull-down menu. In this way, a multimodal user interface that allows speech and GUI inputs can be implemented.
An example of a display window when the contents shown in
As described above, according to this embodiment, the device to be manipulated dynamically generates logic or use modalities as the user interface of the manipulation device in consideration of the properties of the manipulation device 100 and user properties/preferences, and the manipulation device can implement such user interface. Hence, a user interface which is appropriately customized for a user who is not familiar with operations or a user such as a vision-impaired person or the like can be provided, thereby improving the usability.
[Another Embodiment]
In the above embodiment, the information shown in
In such arrangement, when the contents of this user interface are executed using the multimodal browser, the user interface can be provided in Japanese. In this manner, the modalities and display can be changed in accordance with the profile information.
[Still Another Embodiment]
The above embodiment assumes a user who is a normal and has an average familiarity with the device 200 to be manipulated (copying machine), and has exemplified a case wherein a user-initiative UI that allows speech and GUI inputs and makes the user actively fill a text box and pull-down menu is presented on the manipulation device 100 (PDA). However, the present invention is not limited to such specific embodiment.
For example, a system-initiative UI in which the system makes inquiries about input items and fills items with answers of the user can be presented to a vision-impaired person or a user who has low familiarity, and especially a speech-based system-initiative UI to a vision-impaired person. For example, the following inquiries and answers are made:
In this case, the contents which implement the above inquiries and answers are described.
[Still Another Embodiment]
In the above embodiment, word speech recognition that accepts isolated words such as “five”, “A4”, and the like is assumed as the type of speech recognition in the manipulation device which executes the contents of the user interface. Alternatively, continuous speech recognition that accepts continuous speech as a combination of a plurality of words may be adopted. With this speech recognition, when the user utters “five copies, A4 to A3, darkest”, a plurality of corresponding fields can be simultaneously filled.
[Still Another Embodiment]
In the above embodiment, a typical stylesheet that transforms an abstract UI component into a concrete UI component, and an application-dependent stylesheet including styles of text and a form, captions, and the like are separated, and the typical stylesheet is described in an application-independent format that can be re-used in other devices other than the device 200 to be manipulated (copying machine), thus further reducing authoring cost.
For example, in
Portions other than style attributes are transformed first using a stylesheet shown in
[Still Another Embodiment]
In the above embodiment, the profile information is set by the user via the GUI implemented by the manipulation device 100. However, profile information may be dynamically generated based on user's manipulation histories of the manipulation device 100.
[Still Another Embodiment]
In the above embodiment, the profile information is manually set by the GUI via the GUI implemented by the manipulation device 100. Alternatively, a user ID used to specify the user may be input, and a user information database that manages profile information for each user ID may be accessed using that user ID to acquire profile information corresponding to the user ID.
Note that this user information database may be managed by, e.g., the manipulation device 100 or by a dedicated server which can be accessed by the manipulation device 100 via a wired/wireless network.
[Still Another Embodiment]
In the above embodiment, various programs that implement this embodiment are held in the ROM 203 of the manipulation device 100 and the ROM 303 of the device 200 to be manipulated. However, the present invention is not limited to such specific case. For example, these programs may be held in an external storage device that can be connected to the manipulation device 100 or device 200 to be manipulated (CD-ROM/R/RW drive, DVD-ROM/RAM/R/RW drive, ZIP drive, MO drive, or memory card (e.g., SD card, MM (multimedia) card, smart media, compact flash®) slot). Alternatively, dedicated hardware that implements various programs may be prepared.
The preferred embodiments of the present invention have been explained, and the present invention can be practiced in the forms of a system, apparatus, method, program, storage medium, and the like. Also, the present invention can be applied to either a system constituted by a plurality of devices, or an apparatus consisting of a single equipment.
Note that the present invention includes a case wherein the invention is achieved by directly or remotely supplying a program of software (a program corresponding to the flowchart shown in
Therefore, the program code itself installed in a computer to implement the functional process of the present invention using the computer implements the present invention. That is, the scope of the claims of the present invention includes the computer program itself for implementing the functional process of the present invention.
In this case, the form of program is not particularly limited, and an object code, a program to be executed by an interpreter, script data to be supplied to an OS, and the like may be used as long as they have the program function.
As a recording medium for supplying the program, for example, a floppy® disk, hard disk, optical disk, magnetooptical disk, MO. CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), and the like may be used.
As another program supply method, the program may be supplied by establishing connection to a home page on the Internet using a browser on a client computer, and downloading the computer program itself of the present invention or a compressed file containing an automatic installation function from the home page onto a recording medium such as a hard disk or the like. Also, the program code that forms the program of the present invention may be segmented into a plurality of files, which may be downloaded from different home pages. That is, the present invention includes a WWW server which makes a plurality of users download a program file required to implement the functional process of the present invention by the computer.
Also, a storage medium such as a CD-ROM or the like, which stores the encrypted program of the present invention, may be delivered to the user, the user who has cleared a predetermined condition may be allowed to download key information that decrypts the program from a home page via the Internet, and the encrypted program may be executed using that key information to be installed on a computer, thus implementing the present invention.
The functions of the aforementioned embodiments may be implemented not only by executing the readout program code by the computer but also by some or all of actual processing operations executed by an OS or the like running on the computer on the basis of an instruction of that program.
Furthermore, the functions of the aforementioned embodiments may be implemented by some or all of actual processes executed by a CPU or the like arranged in a function extension board or a function extension unit, which is inserted in or connected to the computer, after the program read out from the recording medium is written in a memory of the extension board or unit.
The present invention is not limited to the above embodiments and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, to appraise the public of the scope of the present invention, the following claims are made.
This application claims priority from Japanese patent Application No. 2003-324693 filed on Sep. 17, 2003, the entire contents of which are hereby incorporated by reference herein.
Number | Date | Country | Kind |
---|---|---|---|
2003-324693 | Sep 2003 | JP | national |