PROGRAMMATIC USER INTERFACE GENERATION BASED ON DISPLAY SIZE

BACKGROUND

Devices such as personal computers (PCs), laptops, slates, and phones offer a wide range of screen sizes. However, there is no established method for scaling a user interface (UI) across a large range of screen sizes, from very large displays down to smaller displays. It is with respect to this general technical area that the present application is directed.

SUMMARY

Non-limiting examples of the present disclosure describe programmatic generation of a user interface for display on a processing device. A display class is determined from a plurality of display classes based on a detected display size of a processing device on which the user interface is to display. Prior to instantiating a user interface window, a stored user interface definition is identified and interpreted. The stored user interface definition comprises at least one programmed command object. A displayed user interface is instantiated on the processing device, where the displayed user interface comprises at least one user interface element. The user interface element is programmatically generated by translating the programmed command object of the user interface definition into the user interface element based on operations set in accordance with the determined display class.

In other non-limiting examples, a user interface is generated that comprises a user interface control palette usable to control an application. A display class is determined from a plurality of display classes based on a detected display size of a processing device on which the user interface is to display. Prior to instantiating a user interface window, a stored user interface definition is identified and interpreted. The stored user interface definition comprises a plurality of groupings of programmed command objects. A displayed user interface is instantiated on the processing device, where the displayed user interface comprises at least one user interface control palette. The user interface control palette is programmatically generated by translating the plurality of groupings of programmed command objects of the user interface definition into a plurality of groupings of user interface elements based on operations set in accordance with the determined display class.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and/or advantages of examples will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive examples are described with reference to the following figures.

FIG. 1 is a block diagram illustrating an example of a computing device with which aspects of the present disclosure may be practiced.

FIGS. 2A and 2B are simplified block diagrams of a mobile computing device with which aspects of the present disclosure may be practiced.

FIG. 3 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced.

FIG. 4 is an exemplary method for managing user interface definitions with which aspects of the present disclosure may be practiced.

FIG. 5 is an exemplary method for presenting a user interface with which aspects of the present disclosure may be practiced.

FIG. 6 is a diagram illustrating an exemplary method for setting a layout of a user generated interface palette with which aspects of the present disclosure may be practiced.

FIG. 7 is a diagram illustrating display for exemplary processing devices of different sizes with which aspects of the present disclosure may be practiced.

FIG. 8 is a diagram illustrating user interface examples with which aspects of the present disclosure may be practiced.

FIG. 9 is a diagram illustrating user interface examples with which aspects of the present disclosure may be practiced.

DETAILED DESCRIPTION

Users of processing devices desire applications to be optimized in a form-factor manner. However, there is no established method of scaling a user interface (UI) across a large range of screen sizes with such an approach. Simply attempting to merge a large screen version of an application with a small screen version of an application creates complications. As an example, when large screen versions of applications are executed on devices having smaller display sizes, the UI gets too crowded and touch targets become too small. Additionally, another complication is that UIs are not traditionally scalable across devices having different display sizes. For instance, a user of a processing device may be viewing an application on a device having a smaller display size (e.g., mobile phone) and proceed to connect the device having the small screen display to a device having a larger display size (e.g., PC). Attempted resizing of an application across differing display sizes may drastically affect the display and operation of the UI for an application and/or UI control. In other cases where different versions of an application are developed (e.g., mobile version and desktop version), systems are typically unable to recognize that a UI is to be scaled to a different programmed version to account for display size changes. Other instances of building UI packages may incorporate a scaling model for large and small screen devices but are only able to show a single type of UI (e.g., phone version or slate version) once an application is installed. This may limit a user's ability to connect to large display screens and enjoy UI that takes advantage of available display space.

In non-limiting examples, systems and methods describe programmatic generation of user interfaces in a form factor manner that accounts for display size of the processing device upon which an application/UI is displayed. Programming operations applied generate multiple sets of UI controls and layouts to use and based on the size of a display, determine which of the UI controls and layout algorithms to apply when instantiating a user interface. Examples of the present disclosure comprise evaluation of display class information associated with an application UI at runtime of the application to identify a class of display (e.g., large screen/tablet/slate/phablet/phone, etc.). Display class information may be used to determine whether to display a UI optimized for larger screen devices, smaller screen devices or something in-between. One or more UI layouts may be generated for each of a plurality of display classes. When a display class is identified based on the detection of a display size of a processing device upon which an application/UI is running, a user interface may be displayed based on a UI layout associated with determined display class. Examples described enable a UI to be tailored at run-time in a form factor manner without requiring any changes to code or a user interface definition. Exemplary UI examples generated provide visually different experiences tailored to the processing device than an application/UI is executing upon. At the same time, examples preserve semantic meaning of commands/user interactions and advantages of a definition-based UI.

A number of technical advantages are achieved based on the present disclosure including but not limited to: improved scalability of UI for applications, consistent functionality for a UI that accounts for varying display sizes, visually appealing presentation of UI controls, enhanced processing capability across devices of varying display sizes including improved efficiency and usability for UI controls, improved efficiency in navigation and access to control content for applications/UIs, improved efficiency in programming application command objects, and improved user interaction with applications/UI controls, among other examples.

FIGS. 1-3 and the associated descriptions provide a discussion of a variety of operating environments in which examples of the invention may be practiced. However, the devices and systems illustrated and discussed with respect to FIGS. 1-3 are for purposes of example and illustration and are not limiting of a vast number of computing device configurations that may be utilized for practicing examples of the invention, described herein.

FIG. 1 is a block diagram illustrating physical components of a computing device 102, for example a mobile processing device, with which examples of the present disclosure may be practiced. In a basic configuration, the computing device 102 may include at least one processing unit 104 and a system memory 106. Depending on the configuration and type of computing device, the system memory 106 may comprise, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 106 may include an operating system 107 and one or more program modules 108 suitable for running software programs/modules 120 such as JO manager 124, other utility 126 and application 128. As examples, system memory 106 may store instructions for execution. Other examples of system memory 106 may store data associated with applications. The operating system 107, for example, may be suitable for controlling the operation of the computing device 102. Furthermore, examples of the invention may be practiced in conjunction with a graphics library, other operating systems, or any other application program and is not limited to any particular application or system. This basic configuration is illustrated in FIG. 1 by those components within a dashed line 122. The computing device 102 may have additional features or functionality. For example, the computing device 102 may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 1 by a removable storage device 109 and a non-removable storage device 110.

As stated above, a number of program modules and data files may be stored in the system memory 106. While executing on the processing unit 104, program modules 108 (e.g., Input/Output (I/O) manager 124, other utility 126 and application 128) may perform processes including, but not limited to, one or more of the stages of the operations described throughout this disclosure. Other program modules that may be used in accordance with examples of the present invention may include electronic mail and contacts applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, photo editing applications, authoring applications, etc.

Furthermore, examples of the invention may be practiced in an electrical circuit comprising discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic elements or microprocessors. For example, examples of the invention may be practiced via a system-on-a-chip (SOC) where each or many of the components illustrated in FIG. 1 may be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communications units, system virtualization units and various application functionality all of which are integrated (or “burned”) onto the chip substrate as a single integrated circuit. When operating via an SOC, the functionality described herein may be operated via application-specific logic integrated with other components of the computing device 502 on the single integrated circuit (chip). Examples of the present disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluidic, and quantum technologies. In addition, examples of the invention may be practiced within a general purpose computer or in any other circuits or systems.

The computing device 102 may also have one or more input device(s) 112 such as a keyboard, a mouse, a pen, a sound input device, a device for voice input/recognition, a touch input device, etc. The output device(s) 114 such as a display, speakers, a printer, etc. may also be included. The aforementioned devices are examples and others may be used. The computing device 104 may include one or more communication connections 116 allowing communications with other computing devices 118. Examples of suitable communication connections 116 include, but are not limited to, RF transmitter, receiver, and/or transceiver circuitry; universal serial bus (USB), parallel, and/or serial ports.

The term computer readable media as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, or program modules. The system memory 106, the removable storage device 109, and the non-removable storage device 110 are all computer storage media examples (i.e., memory storage.) Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article of manufacture which can be used to store information and which can be accessed by the computing device 102. Any such computer storage media may be part of the computing device 102. Computer storage media does not include a carrier wave or other propagated or modulated data signal.

Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

FIGS. 2A and 2B illustrate a mobile computing device 200, for example, a mobile telephone, a smart phone, a personal data assistant, a tablet personal computer, a phablet, a slate, a laptop computer, and the like, with which examples of the invention may be practiced. For example, mobile computing device 200 may be implemented to execute applications and/or application command control. Application command control relates to presentation and control of commands for use with an application through a user interface (UI) or graphical user interface (GUI). In one example, application command controls may be programmed specifically to work with a single application. In other examples, application command controls may be programmed to work across more than one application. With reference to FIG. 2A, one example of a mobile computing device 200 for implementing the examples is illustrated. In a basic configuration, the mobile computing device 200 is a handheld computer having both input elements and output elements. The mobile computing device 200 typically includes a display 205 and one or more input buttons 210 that allow the user to enter information into the mobile computing device 200. The display 205 of the mobile computing device 200 may also function as an input device (e.g., a touch screen display). If included, an optional side input element 215 allows further user input. The side input element 215 may be a rotary switch, a button, or any other type of manual input element. In alternative examples, mobile computing device 200 may incorporate more or less input elements. For example, the display 205 may not be a touch screen in some examples. In yet another alternative example, the mobile computing device 200 is a portable phone system, such as a cellular phone. The mobile computing device 200 may also include an optional keypad 235. Optional keypad 235 may be a physical keypad or a “soft” keypad generated on the touch screen display or any other soft input panel (SIP). In various examples, the output elements include the display 205 for showing a GUI, a visual indicator 220 (e.g., a light emitting diode), and/or an audio transducer 225 (e.g., a speaker). In some examples, the mobile computing device 200 incorporates a vibration transducer for providing the user with tactile feedback. In yet another example, the mobile computing device 200 incorporates input and/or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., a HDMI port) for sending signals to or receiving signals from an external device.

FIG. 2B is a block diagram illustrating the architecture of one example of a mobile computing device. That is, the mobile computing device 200 can incorporate a system (i.e., an architecture) 202 to implement some examples. In one examples, the system 202 is implemented as a “smart phone” capable of running one or more applications (e.g., browser, e-mail, calendaring, contact managers, messaging clients, games, and media clients/players). In some examples, the system 202 is integrated as a computing device, such as an integrated personal digital assistant (PDA), tablet and wireless phone.

One or more application programs 266 may be loaded into the memory 262 and run on or in association with the operating system 264. Examples of the application programs include phone dialer programs, e-mail programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, Internet browser programs, messaging programs, and so forth. The system 202 also includes a non-volatile storage area 268 within the memory 262. The non-volatile storage area 268 may be used to store persistent information that should not be lost if the system 202 is powered down. The application programs 266 may use and store information in the non-volatile storage area 268, such as e-mail or other messages used by an e-mail application, and the like. A synchronization application (not shown) also resides on the system 202 and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in the non-volatile storage area 268 synchronized with corresponding information stored at the host computer. As should be appreciated, other applications may be loaded into the memory 262 and run on the mobile computing device 200 described herein.

The system 202 has a power supply 270, which may be implemented as one or more batteries. The power supply 270 might further include an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the batteries.

The system 202 may include peripheral device port 230 that performs the function of facilitating connectivity between system 202 and one or more peripheral devices. Transmissions to and from the peripheral device port 230 are conducted under control of the operating system (OS) 264. In other words, communications received by the peripheral device port 230 may be disseminated to the application programs 266 via the operating system 264, and vice versa.

The system 202 may also include a radio interface layer 272 that performs the function of transmitting and receiving radio frequency communications. The radio interface layer 272 facilitates wireless connectivity between the system 202 and the “outside world,” via a communications carrier or service provider. Transmissions to and from the radio interface layer 272 are conducted under control of the operating system 264. In other words, communications received by the radio interface layer 272 may be disseminated to the application programs 266 via the operating system 264, and vice versa.

The visual indicator 220 may be used to provide visual notifications, and/or an audio interface 274 may be used for producing audible notifications via the audio transducer 225. In the illustrated example, the visual indicator 220 is a light emitting diode (LED) and the audio transducer 225 is a speaker. These devices may be directly coupled to the power supply 270 so that when activated, they remain on for a duration dictated by the notification mechanism even though the processor 260 and other components might shut down for conserving battery power. The LED may be programmed to remain on indefinitely until the user takes action to indicate the powered-on status of the device. The audio interface 274 is used to provide audible signals to and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 225, the audio interface 274 may also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. In accordance with examples of the present invention, the microphone may also serve as an audio sensor to facilitate control of notifications, as will be described below. The system 202 may further include a video interface 276 that enables an operation of an on-board camera 230 to record still images, video stream, and the like.

A mobile computing device 200 implementing the system 202 may have additional features or functionality. For example, the mobile computing device 200 may also include additional data storage devices (removable and/or non-removable) such as, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 2B by the non-volatile storage area 268.

Data/information generated or captured by the mobile computing device 200 and stored via the system 202 may be stored locally on the mobile computing device 200, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio 272 or via a wired connection between the mobile computing device 200 and a separate computing device associated with the mobile computing device 200, for example, a server computer in a distributed computing network, such as the Internet. As should be appreciated such data/information may be accessed via the mobile computing device 200 via the radio 272 or via a distributed computing network. Similarly, such data/information may be readily transferred between computing devices for storage and use according to well-known data/information transfer and storage means, including electronic mail and collaborative data/information sharing systems.

FIG. 3 illustrates one example of the architecture of a system for providing an application that reliably accesses target data on a storage system and handles communication failures to one or more client devices, as described above. Target data accessed, interacted with, or edited in association with programming modules 108, applications 120, and storage/memory may be stored in different communication channels or other storage types. For example, various documents may be stored using a directory service 322, a web portal 324, a mailbox service 326, an instant messaging store 328, or a social networking site 330, application 128, IO manager 124, other utility 126, and storage systems may use any of these types of systems or the like for enabling data utilization, as described herein. A server 320 may provide storage system for use by a client operating on general computing device 102 and mobile device(s) 200 through network 315. By way of example, network 315 may comprise the Internet or any other type of local or wide area network, and client nodes may be implemented as a computing device 102 embodied in a personal computer, a tablet computing device, and/or by a mobile computing device 200 (e.g., mobile processing device). Any of these examples of the client computing device 102 or 200 may obtain content from the store 316.

FIG. 4 is an exemplary method 400 for managing user interface definitions with which aspects of the present disclosure may be practiced. As an example, method 400 may be executed by an exemplary system such as shown in FIGS. 1-3. In examples, method 400 may be executed on a device comprising at least one processor configured to store and execute operations, programs or instructions. However, method 400 is not limited to such examples. In at least one example, method 400 may be executed (e.g., computer-implemented operations) by one or more components of a processing device or components of a distributed network, for instance, web service/distributed network service (e.g. cloud service). System components may be utilized to perform the operations described herein with respect to method 400.

In examples, method 400 may be performed in associated with an application. An application is a software component that executes on the processing device, interfacing with hardware and software components of the device. An application comprises one or more programs designed to carry out operations, and an application is associated with a UI. A user interface (Up provides interactive or non-interactive dialog windows, task panes, etc., for managing user interaction with an application or service. Examples of tasks performed by a UI comprise presenting information and receiving input from a user. An example UI may contain one or more elements/controls, or groups of controls, such as, for example, push buttons, radio buttons, check boxes, edit boxes, text labels, list boxes, etc. An example UI includes any number of controls or other elements, such as, for example, text, vector graphics, images, video, animations, and audio. In one example, a UI exists separately from an application with which an exemplary UI is to be used. Alternately may be either integrated with the source code, or included with standard resources used by the application, such as dynamic linked library (DLL) files, among other examples.

In examples, a UI may provide application command control though UI controls. User interface control is a graphical control element that interfaces with an application that executes on the processing device (e.g., memory, processor and functions of mobile device) and software components such as an operating system (OS), applications executing on a mobile device, programming modules, input methods (e.g., soft input panel (SIP)) and command container such as a pane or contextual menu, among other examples. As an example, an UI control is used to control execution of actions/commands for the application. An SIP is an on-screen input method for devices (e.g., text input or voice input), and a pane is a software component that assists function of other software running on the device such as the OS and other software applications, among other examples. In some examples, a UI control may be integrated within an application. For instance, a UI control may be able to be launched, closed, expanded or minimized when an application is launched, closed, expanded or minimized. In other examples, UI control is executable as its own application that interfaces with another application. For instance, UI control may be able to be launched, closed or minimized separately from the launching of an application that is controlled by the UI control.

Method 400 begins at operation 402 where a programmed command object is received or updated. A programmed command object is code that represents a command and/or command group that is to be rendered as a UI element. A programmed command object may be included in a UI definition that is used to translate the programmed command objects into displayable and usable portions of a UI. UI elements are the displayable and usable portions of the UI that act as controls to trigger actions/commands associated with a programmed command object. In operation 402, developers may create new programmed command objects or update existing command objects for inclusion in a user interface definition that can be used to define how programmed command objects/groupings of programmed command objects are presented for display. In examples, developers that create programmed command objects may focus on commanding/function of a programmed command object do not necessarily need to be specify how a command object may display as it is scaled across processing devices having varying display sizes. In that example, programmed command objects may be included in a UI definition data and programmatically generated as UI elements in a form factor manner using other programming operations (e.g., application programming interface (API)) that evaluate a UI definition data.

Once programming command objects are created or updated, flow may proceed to operation 404 where a programming command object is stored. In operation 404, data associated with programming command objects may be stored in one or more memories, storages, libraries, files, or databases, etc., for use in creation of a UI definition. In one example, stored program command objects may be stored on a processing device upon which an application/UI is executing. In another example, stored program command objects may be stored on one or more separate processing devices that may be used to manage applications/UIs and UI definition data, for example. In any example, storages for programmed command objects (e.g., database) are editable in order to add or delete controls or other elements, or to change the appearance or behavior of one or more controls or other elements.

Flow may proceed to operation 406 where a UI definition is created or updated. A UI definition is a collection of programmed command objects for commanding an application. The UI definition is data used to translate the programmed command objects into displayable and usable portions of an application user interface. User interface definitions may be data stored in any form, for example any type of file format including linked file libraries. In examples, a UI definition includes properties for each programmed command object, including, for example, position, dimension, visibility, text, colors, opacity, borders, accessibility information, enabled state, and other states commonly associated with conventional UI controls. In one example, programmed command objects specified by the UI definition files are stored (operation 408) in a library or database comprised of predefined controls and other elements, i.e., vector graphics, images, video, etc. as identified in operation 404. Alternately, a storage, memory, database etc. may contain pointers to some or all of the controls or other programmed command objects. Such an example is useful where command objects having a relatively large file size, such as a video clip, are to be included in a rendered UI window.

In addition, the UI definition may also reference one or more event handlers or “listeners” that are to be associated with particular controls, groups of controls, other elements, or entire UI windows so that the controls, elements, or UI windows are capable of interacting with an associated application. In alternate examples, these listeners are either stored along with the UI definitions, or in a separate listener memory, storage, file, database, etc. After reading the user interface descriptions, either at or during application run time, the controls or other elements described by the UI definition files are read from a library or database file of controls and elements, associated with the specified listeners, and then used to automatically instantiate extensible user interfaces.

Flow may proceed to operation 408 where user definitions are stored for use in generation of user interfaces for display. In one example, user interface definitions or descriptions may be stored (operation 408) separate from the application code with which the user interfaces are intended to be used. User interface definitions or definition files can be changed at any time prior to running an underlying application, or alternately, at any time prior to displaying the user interface to provide uniquely configured user interface windows which are then automatically instantiated when rendering the user interface windows. This concept offers several major advantages. For instance, changes to any of the user interface windows do not require editing and recompiling of the associated application source code. Consequently, the number of potential errors that may be introduced into an application are dramatically reduced because the application itself is not edited to modify the UI windows associated with that application. Further the use of separate UI definition files serves to allow for a common baseline application source code, regardless of the user interfaces that are associated with that application. Alternately, as noted above, the UI descriptions are stored/included (operation 408) in either the application code itself, or in one or more linked files, such as a DLL file, or other files which are included in the application as the application is compiled, rather than including the descriptions in separate UI definition files. Regardless of where the UI descriptions are located, UI descriptions are interpreted and treated in the same manner prior to automatic instantiation of the UI windows.

FIG. 5 is an exemplary method 500 for presenting a user interface with which aspects of the present disclosure may be practiced. As an example, method 500 may be executed by an exemplary system such as shown in FIGS. 1-3. In examples, method 500 may be executed on a device comprising at least one processor configured to store and execute operations, programs or instructions. However, method 500 is not limited to such examples. In at least one example, method 500 may be executed (e.g., computer-implemented operations) by one or more components of a processing device or components of a distributed network, for instance, web service/distributed network service (e.g. cloud service). System components may be utilized to perform the operations described herein with respect to method 500.

Method 500 begins at operation 502 where a display class is determined based on a detected display size of a processing device upon which an application/UI is to be displayed. Display class determination provides an abstraction for determining the size of a display. Operations (e.g., code) can query display class information to determine a UI instance to instantiate depending on the display size of the processing device that an application window is to execute upon. A display class can be defined for processing devices having display sizes that fall within the range associated with the display class. That is, display classes act as transition points for UI experiences. Display class is a value that is determined based a maximum display size. The value for display class may be in any form including numeric values and elements of speech, as examples. For instance, display classes may be set to correspond with different types of processing devices (e.g., laptops, PCs, tablets, phones, etc.) where an exemplary display class may be “<=Phone” or “<=Tablet”. In another example, display classes may be set based on numeric values. For example, a display class may be identified using numeric values (e.g., 0 to 3 inches). In any examples, display classes are used to classify processing devices in accordance with display size. For example, a display class may be set for processing devices having a display size falling in a range from 0 to 3 inches where another display class may be set for processing devices having a display size in a range from 3.1 to 5 inches, and so on. A range for values of display classes may fall between 0 and infinity. In one example, operations for display class determination are written in style of successive less than or equal to (<=) checks, with an else for everything greater than a defined display class. In this example, additional display class designations may be easily added without having to change operational code behavior. However, one skilled in the art will recognize that display class designations including minimum and/or maximum values for ranges of display classes can be defined in any possible way that can be useful in defining user interface interaction. In examples, a minimum value of a display class may be a value that is equal to or greater than a maximum value of a display class which is directly smaller than the display class being defined. For instance, as in an example above, a first display class may correspond to a range for devices having displays between 0 and 3 inches and a minimum value of a second display class may take into account a maximum value of the first display class (e.g., 3 inches) and set the minimum value of the second display class at 3.1 inches, for instance. Display classes may be changes over time based on programmer prerogative, analysis/testing/use cases, etc.

A plurality of display classes may be set and operation 502 may determine a display class from the plurality of available display classes. In one example, operation 502 occurs upon or before launching of a new application or UI window or when a display size change is detected such as when a display is moved to a display of a different size. However, one skilled in the art will recognize that detection of a display size and determination of a display class may occur at any time after powering on of a processing device. For instance, operation 502 may execute in a background of an operating system (OS) while other applications are operating in the foreground of the OS. This may minimize the delay in launching of a UI instance and improve an overall user experience.

In one example, operation 502 utilizes API accessing a shared library of data (e.g., dynamic link library (DLL) to determine a display class. As one example, exemplary operational code associated with a display class determination (e.g., display class event) is not limited to but may be similar to:

/** Interface to register against for display class change events */

struct IDisplayClassInformation : public Mso::IRefCounted

{

public:

/** Returns the event store for display class change events. This event store will be invoked-

Whenever the running application changes to a different display with a new display-Whenever

the active display changes its DPI */

virtual DisplayClassChangedEvent& DisplayClassChanged( ) = 0;

virtual DisplayClass GetCurrentDisplayClass( ) const = 0;

};

/** Get a DisplayClasslnformation reference on the active UI thread */

MSOCPPAPI_(Mso::TCntPtr<Mso::DisplayClassInformation::IDisplayClassInformation>)

MakeDisplayClassInformation( );.

A processing device may be any device comprising a display screen, at least one memory that is configured to store operations, programs, instructions, and at least one processor that is configured to execute the operations, programs or instructions such as an application/application command control. Display size is a measurement of viewable area for display on a processing device. As an example, display size is a measurement associated with active viewable image size of a processing device. In other examples, display size may be associated with a nominal size value. In one example, detecting of the display size comprises detecting a measurement value for screen diagonal of a display of a processing device. However, display size determination is not limited to such an example. Factors that may be evaluated to determine a display size include but are not limited to: screen diagonal of a display, dot density (e.g., dots per inch (DPI), pixel density (e.g., pixels per inch (PPI), physical size of a screen/display, use case distance of a display from a user, display length, and display width, among other examples. As an example, display size may be a measurement value associated with effective resolution of a display for a processing device. Measurement of effective resolution enables is an example of a value used to evaluate display form factors with a common metric, and enables UI scaling to be classified into different display classes. However, one skilled in the art will recognize that any common metric relative to display size can be applied in exemplary method 500. In alternative examples, other factors other than display size may impact UI adaptation. Examples include but are not limited to: processing device orientation, processing device operational mode (e.g., keyboard mode, touch mode, ink mode, etc.), application window size, screen aspect ratio, and screen effective resolution, among other examples.

In one example, operation 502 may comprise a program instruction or module that can identify and evaluate system specifications for a processing device such as a mobile device or personal computer. For instance, a programming instruction implemented in operation 502 identifies a type or version of the processing device and executes a fetch of data to identify system information of the processing device. In another example, a programming instruction or module may reference manufacturer specification information to determine a value associated with display size of a processing device.

Flow may proceed to operation 504 where a stored UI definition is identified and interpreted. Creating and storing of UI definitions are described in the description of method 400 of FIG. 4. A UI definition may define commanding objects and groupings of commanding objects to be laid out as UI elements when a UI is generated. As described above, a UI definition is data used to translate the programmed command objects into displayable and usable portions of an application user interface. In examples, operation 504 occurs at runtime of an application/UI prior to instantiating a user interface window for display.

In examples, operation 504 may comprise parsing the stored UI definition and generating an object model for UI commanding upon which an actual user interface can be laid out for display to a user. An object model is a collection of programmed command objects or groupings/classes through which programming operations can be used to parse, and manipulate the programmed command objects. The object model represents commanding objects and groupings of commanding objects that are to be rendered as UI. In examples describe herein, UI listeners may work with a generated object model and be notified of user interactions with programmed command objects.

Flow may proceed to operation 506 where a user interface in instantiated and displayed on a processing device in a form factor manner. Operation 506 may comprise generating a user interface that comprises one or more user interface elements. As identified above, UI elements are the displayable and usable portions of the UI that act as controls to trigger actions/commands associated with a programmed command object. In generating a displayable UI, operation 506 programmatically generates the UI element(s) by translating the programmed command object into the user interface element based on operations set in accordance with the determined display class. In examples, operation 506 may employ a component configured to creation a UI element that triggers an action/operation of the programmed command element, the component execution operations to create UI elements through programs/code/applications/APIs/modules, etc. One skilled in the art will recognize that UI elements may have varying levels of complexity depending on a type of UI element that is being created and the properties of that UI including but not limited to: actions, data and associations, button controls, combo boxes, galleries, color, effect, caching, triggers, text/font, height/width, style, alignment, focus, language, and margins, among other examples. In one example, UI elements may be controls generated in extensible application markup language (XAML). However, one skilled in the art will recognize that UI elements may be generated in any form including any programming language including markup languages, among other examples.

In examples, operation 506 generates a UI and presents on a display of a processing device based on display class rules (e.g., display class rule set) for interpreting the UI definition. A display class rule set is a set of pre-defined dynamic layout rules for display classes that account for a detected display size of a processing device running an application/UI. A display class rule set may be defined for generating a UI from a stored UI definition taking into account a display class designation as determined in operation 502. In examples, display class rule sets may be programmed to be associated with one or more display classes. In one example, a display class rule set may be programmed for each of a plurality of display classes. In other examples, a display class rule may be programmed to cover more than one display class. For instance, a display class rule set may be utilized to set a UI layout for small screen processing devices or large screen processing devices, being classifications that cover multiple display classes. Operation 506 generates and displays a UI comprising one or more UI elements by evaluating a UI definition in accordance with display class rules. In one example, a generated UI is an application command control that comprises a plurality of palettes (UI palettes) programmed for application control. A palette is a hierarchical organization of UI elements including a plurality of groupings of UI elements usable for UI command control. Further description regarding palettes is provided in the description of FIGS. 6-9.

Dynamic layout of the automatically instantiated UI windows is accomplished by applying the display class rule set to a UI definition. In one example, display class layout rules are provided in a UI layout rule field or database which may be read out at the time that a UI is being instantiated. In examples, layout rules for display classes may be separately maintained from stored UI definitions and application code. In that instance, programmers who create and update commanding objects do not need to provide consideration for scaling of programmed commanding objects across processing devices of differing display sizes. Dynamic layout rules for display classes may be stored in a storage, memory, database, etc, of a processing device upon which a UI is to be instantiated or one or more other processing devices usable to manage user interface programming. The display class layout rules define how a programmed command object may be translated to be generated as a UI element. That is, presentation of a programmed command object may vary depending on the determined display class associated with the processing device upon which the UI is to be displayed. In operation 506, translating of the programmed command object into the user interface element may comprise setting a size of the user interface element based on the operations corresponding with the determine display class. For example, a programmed command object may appear in a first size (e.g., number of pixels, length by width, etc.) when displayed upon a processing device having a large display (identified by display class determination) and the same programmed command object may appear in a different size when displayed upon a processing device having a small display (identified by display class determination). In operation 506, translating of the programmed command object into the user interface element may comprise setting a style of the user interface element. For example, a programmed command object may be presented as a drop-down menu in one instance (e.g., where display space may be limited) and the same programmed command object may be presented more robustly as a plurality of visible icons in another instance, depending on the layout rules defined for the display class. In operation 506, translating of the programmed command object into the user interface element may comprise setting a layout of the user interface element based on the operations corresponding with the determine display class. For example, a grouping of programmed command objects may display only in a first arrangement in one instance and the arrangement may be different in another instance. That is, layout for each displayable UI window is interpreted in light of such display class rules.

Consequently, in examples, the layout rules for any user interface can be changed, edited, or otherwise updated any time prior to rendering the UI window to provide uniquely configured user interface windows. This may be accomplished without the need to modify the underlying application code in any way. Display class layout rules allow for dynamic resizing and repositioning of UI windows, with a corresponding real-time automatic dynamic adjustment of UI items within already rendered UI windows.

Flow may proceed to decision operation 508 where it is determined whether a UI definition is updated. In one example, display class layout rules provide for the automatic addition or removal of one or more UI elements from a displayed. UI window based either on user interaction with the UI window, or automatic programmatic interaction with the IA window by one or more active applications. That is, in examples, user of a UI may make changes or update a UI definition and a UI may be dynamically adjusted to account for such changes/updates. Alternatively, developers/programmers of a UI may make changes or update a UI definition and a UI may be dynamically adjusted to account for such changes updates. If operation 508 determines that a displayed definition has not been updated, flow branches NO and processing of method 500 ends. In an example where operation 508 determines that a UI definition has changed, flow may return to operation 504 where the UI definition is identified and interpreted. Flow may proceed to operation 506 where a displayed UI is instantiated based on the updated UI definition and the display class rules.

FIG. 6 is a diagram illustrating an exemplary method 600 for setting a layout of a generated user interface palette with which aspects of the present disclosure may be practiced. As an example, method 600 may be executed by an exemplary system such as shown in FIGS. 1-3. In examples, method 600 may be executed on a device comprising at least one processor configured to store and execute operations, programs or instructions. However, method 600 is not limited to such examples. In at least one example, method 600 may be executed (e.g., computer-implemented operations) by one or more components of a processing device or components of a distributed network, for instance, web service/distributed network service (e.g. cloud service). System components may be utilized to perform the operations described herein with respect to method 600.

In examples associated with the present disclosure, a displayed UI (e.g., application command control) hosts a majority of an application's command set, organized in a hierarchical structure of individual palettes. An exemplary UI may comprises a plurality of palettes (UI palettes) programmed for application control. A palette is a hierarchical organization of UI elements including a plurality of groupings of UI elements usable for UI command control. Exemplary palettes for application command control are illustrated in FIGS. 8-9 and further described in the description of FIGS. 7-9. Further, UI palettes may be programmed to dynamically interact with an application and display simultaneously with applications and/or user interface components such as an SIP or on screen keyboard.

Flow of method 600 begins at operation 602 where a layout of a UI palette is set or changed based on the display class rules as applied to a UI definition. In examples, operation 602 may occur when a UI is being generated for display on a processing device. Dynamic layout of the automatically instantiated UI windows is accomplished by applying a display class rule set to a UI definition as described in the description of FIG. 5. Setting (operation 602) a layout for a UI palette may further comprise operations such as operations shown in operations 604-610.

In operation 604, a maximum number of rows of UI elements for a palette are determined. Setting (operation 602) the layout of UI palette may comprise determining (operation 604) a maximum number of rows of UI elements for a UI palette displayed by the UI. Operation 604 enables an UI developer to best layout palettes and UI elements for users given the limited space that may available on processing devices. In operation 604, scaling for display of UI elements in UI palettes may be a programming operation associated with a display class rule set. Operation 604 determines a maximum number of rows of commands for a palette in association with the display class rule set which is based on the display size of the processing device. Operation 602 may utilize such data in determining an optimal layout for a UI palette.

In operation 606, a maximum number of UI elements per row may be determined for a UI palette. Setting (operation 602) the layout of UI palette may comprise determining (operation 606) a maximum number of UI elements per row for a UI palette displayed by the UI. Operation 606 enables an UI developer to best layout palettes and UI elements for users given the limited space that may available on processing devices. In operation 606, scaling for display of UI elements in UI palettes may be a programming operation associated with a display class rule set.

As an example, display class rule sets may act as scaling plateaus. For instance, evaluation of a display size of a processing device may determine that a diagonal of a display for a processing device is four inches wide, and the display of the processing device has a resolution that is 360 pixels in width by 640 pixels in length. Thus, the display class rule set may determine that for a display having an effective pixel width of 360 pixels, a maximum of 8 rows should be displayed each having a maximum of 4 commands per row. In another example, evaluation of the processing device may determine that a diagonal of a display for a processing device is 5.5 inches wide, and the display of the processing device has a resolution that is 512 pixels in width by 853 pixels in length. Thus, the application command control 402, based on its scaling plateaus, may determine to display up to 12 rows and up to 6 commands per row of commands in palettes that are displayed.

In operation 608, a height and/or width is determined for a UI palette that is being generated. Setting (operation 602) the layout of UI palette comprises determining a height/width for a UI palette displayed by the UI. In operation 608, scaling for display of UI elements in UI palettes may be a programming operation associated with a display class rule set. The determined height is an amount of space the UI palette takes up length-wise on a display of a processing device when a UI window is in an open state displaying the UI palette. The determined width is an amount of space the UI palette takes up from side to side on a display of a processing device when a UI window is in an open state displaying the UI palette. In one example, height/width values of a palette are set by a program developer of the UI and may be specified in the display class rule set for generating a form factor UI to display on a processing device. In one example, users of the UI may be prevented from resizing a UI palette. In another example, height/width of a palette may be adjustable, for example during operation by a user. In any example, height/width of a UI palette may be set for cases where the UI is displayed in different states (e.g., open state, minimized state, invisible state, etc.). In different states, a height/width of one palette or all palettes displayed may be set to a predetermined value. For instance, a palette height may be set to a percentage (e.g., 55%) of a height of a display for the processing device when the UI is operating in an open state. In another example, a height of a palette may be set to different percentage (e.g., smaller percentage) of a display when the UI is operating in a minimized state. In examples, a height of the palette may be set as a percentage or as a fixed number of pixels. Height or width values of a palette may also be conditional based on launching of other software components by the OS such an SIP, and panes. For example, conditions may be set such as when the palette sits above an SIP or below a pane, its height/width is equal to the remaining space for display of a UI palette on the processing device. When a software component such as an SIP or pane is closed or minimized, this may adjust the height and/or width of a UI palette.

In operation 610, a size, style and layout of UI elements may be determined for a UI palette. Setting (operation 602) the layout of UI palette may comprise setting (operation 610) of a size, style and layout of UI elements in a palette displayed by the UI. Description relates to setting of a size, style and layout of a UI element is described in operation 506 of FIG. 5. Operation 610 encompasses operations described in operation 506 as applied to generation of a UI palette. Operation 610 enables an UI developer to best layout palettes and UI elements for users given the limited space that may available on processing devices. In operation 610, scaling for display of UI elements in UI palettes may be a programming operation associated with a display class rule set. Setting (operation 610) of UI elements comprises setting a layout for at least one UI palettes such as a top-level palette and/or drill-in palette. Each of the top-level palettes and the drill-in palettes is a collection or grouping comprising one or more selectable UI elements. An exemplary top-level palette is shown in FIG. 8 and described in the description of FIG. 8. An exemplary drill-in palette is shown in FIG. 9 and described in the description of FIG. 9.

Operation 610 may utilize a UI definition in determining a layout of a UI palette. A UI definition may specify how programmed command objects can be grouped or nested and actual layout of the command objects as UI elements may be set by applying a display class rule set associated with a determined display class. In examples, a UI definition may comprise command grouping data being information relating to the grouping of command objects including associations between command objects. For example, text editing features such as bolding, underlining, italicization, superscript and subscript may be associated and commonly used. Ideally, a UI window would like to include all of these commonly used functions on the same UI palette. However, due to limitations on the screen size, certain commands may need to be separated. Command grouping data is information that identifies associations and what commands should or should not be separated from each other. For example, it may be determined that the maximum number of rows and commands allows displaying of text formatting commands including a superscript editing command in a top-level palette but would not also allow displaying of a subscript command. Using the command grouping data, it may be identified that from a functionality and/or usability standpoint, that it is best not to separate the superscript and subscript editing commands. For instance, a user who makes a subscript text edit may later look to make a superscript edit or visa-versa. Thus, in setting the layout of commands for palettes, operation 610 may take into account command grouping data of a UI definition to determine an optimal UI display window that accounts for the display size of a device upon which the UI is displaying. In one example, operation 610 may display a higher-level command for text editing in a top-level palette and the superscript and subscript editing commands may be included in a drill-in palette (child palette) of that top-level palette (parent palette) so they such commands remain associated.

Once a UI palette is set and instantiated, flow may proceed to decision operation 612 where it is determined whether a display size change has occurred. If a display size change is not detected flow branches NO and method 600 ends. Method 600 may be implemented/re-implemented any time a UI window is to be instantiated or a display size change is detected. If a display size change is detected (e.g., resolution change or connection of another device is detected) flow branches YES and returns back to operation 602 where a layout of palettes of a UI may be set or changed.

FIG. 7 is a diagram illustrating display for exemplary processing devices of different sizes with which aspects of the present disclosure may be practiced. Examples shown in FIG. 7 comprise processing devices having varying sizes and/or varying screen/display sizes, for example processing device 702, processing device 704, processing device 706 and processing device 708.

As shown in FIG. 7, a UI control (e.g., application command control) and an application/canvas are displayed in exemplary processing devices 702-708. An application command control and an application/canvas are examples of components of a UI with which the present disclosure may apply. In examples, the UI is programmed to efficiently scale itself to utilize display space of processing devices of different sizes and/or operating size of display windows. For example, presentation of a UI control and/or application/canvas may vary across the different processing devices 702-708. UI control and/or an application/canvas may be scaled according to a determined display class associated with a processing device.

An application/canvas is a portion of a display of a processing device that is designated for display of an application executing on the device. The application/canvas region is the application UI that shows effects implemented by actions executed via a UI control. That is, the application/canvas is the content consisting of but not limited to the pages in workspace or editable portions of an application.

A UI control (application command control) hosts a majority of an application's command set, organized in a hierarchical structure of individual palettes, chunks, and commands. Further, application command control may be programmed to dynamically interact with an application and display simultaneously with applications and/or user interface components such as a soft input panel (SIP) or on screen keyboard. In one example, a UI control may intelligently adapt based on content of an application (e.g., displayed or selected on an application canvas). A UI control comprises a plurality of palettes (command palettes) programmed for application control. In one example, palettes of an application command control comprise top-level palettes and drill-in palettes. Each of the top-level palettes and the drill-in palettes is a collection or grouping of rows comprising one or more selectable UI elements. As an example, a top-level palette may comprise a highest level grouping of UI elements including commands that are more frequently used/more likely to be used by users. A top-level palette may display command listings that can be drilled into and displayed in drill-in palettes. FIG. 8 illustrates an exemplary top-level palette of an application command control. A drill-in palette is a collection or grouping of commands that may be used less frequently/or likely to be used less frequently compared to the commands displayed on a top-level palette. As an example, drill-in palettes host over-flow commands that, due to constraints resulting from a limited amount of display space for an application command control, are not included in a top-level palette. FIG. 9 illustrates an exemplary drill-in palette of an application command control. Using a word processing application as an exemplary application, a top-level palette may comprise high-level UI elements for text editing, font editing, paragraph formatting, word finder, spell-check etc. that may be frequently used by users of an application. As an example, a drill-in palette for a word processing application may comprise sub-elements of such high-level commands of the top-level palette, for example, subscript or superscript commands for a font command/function. In examples, organization of palettes and UI elements may be editable, for example, where an element or a chunk of a palette can be pulled from one palette and added/displayed in another. For instance, an overflow command of a drill-in palette can be added to a top-level palette.

Examples of common components that make up a top-level palette or a drill-in palette include but are not limited to: a palette bar and palette title, palette switching feature (including one touch target that launches palette switcher from title of palette bar), UI element to dismiss palette (e.g., visual representation of ellipses), quick command elements (e.g., undo or redo), palette canvas comprising a plurality of elements, chunk UI elements (e.g., groupings of commands) and chunk dividers (e.g., dividing different groupings of commands), drill-in features to access drill-in palettes (when applicable).

In one example, palettes of a UI are presented in a vertical layout. For example, a top-level palette and a drill-in palette are vertically scrollable and comprise a collection of rows comprising one or more selectable UI elements. However, in other examples, setting of the layout of a palette may also comprise presenting UI elements in a horizontal layout where palettes are horizontally scrollable. In some examples, no limit is set on the scrollable height of a palette. Scrolling position may be kept on top-level palettes when switching between top-level palettes however scrolling position may or may not be kept for drill-in palettes. UI elements set and displayed may include labels identifying an element and may be configured to take up an entire row of a palette. In other examples, multiple elements may be displayed in one row of a palette. Scaling is applied to setting and displaying commands in palette rows. In some other examples, UI elements may not have labels, for example, commands that are well known or have images displayed that are well known to users. Separators or spacers (either horizontal or vertical depending on layout of palette) may be displayed to break up different elements or chunks of elements.

In FIG. 8, UI control is illustrated an exemplary top-level palette 802. In FIG. 9, UI control is illustrated an exemplary drill-in palette 902. For example, UI control shown in FIG. 9 is a drill-in palette 902 of the UI top-level palette 802 shown in FIG. 8. That is, UI top-level palette 802 is a parent palette of the UI drill-in palette 902 (e.g., child palette of the top-level palette). As shown in top-level palette 802, a row showing a “font formatting” command includes a caret indicative of a drill-in feature. When the drill-in feature is selected, a drill-in palette of drill-in palette 902 is displayed on a display of a processing device. For instance, in FIG. 9, font formatting sub-elements “superscript” and “subscript” are displayed. In this way, application command control and/or an application/canvas may be scaled in accordance with a determined display class associated with a processing device.

Reference has been made throughout this specification to “one example” or “an example,” meaning that a particular described feature, structure, or characteristic is included in at least one example. Thus, usage of such phrases may refer to more than just one example. Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more examples.

One skilled in the relevant art may recognize, however, that the examples may be practiced without one or more of the specific details, or with other methods, resources, materials, etc. In other instances, well known structures, resources, or operations have not been shown or described in detail merely to observe obscuring aspects of the examples.

While sample examples and applications have been illustrated and described, it is to be understood that the examples are not limited to the precise configuration and resources described above. Various modifications, changes, and variations apparent to those skilled in the art may be made in the arrangement, operation, and details of the methods and systems disclosed herein without departing from the scope of the claimed examples.

PROGRAMMATIC USER INTERFACE GENERATION BASED ON DISPLAY SIZE

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

International Classifications

Abstract

Description

Claims

PRIORITY

Provisional Applications (1)