Patent Application
20090119578
This invention relates to software and to development of such software. More particularly, this invention relates to a programming interface that facilitates use of a software platform by application programs and computer hardware.
Accompanying this specification is a set of three compact discs that stores a Software Development Kit (SDK) for the Microsoft® Windows® Code-Named “Longhorn” operating system. The SDK contains documentation for the Microsoft® Windows® Code-Named “Longhorn” operating system. Duplicate copies of each of these three compact discs also accompany this specification.
The first compact disc in the set of three compact discs (CD 1 of 3) includes a file folder named “lhsdk” that was created on Oct. 22, 2003; it is 586 Mbytes in size, contains 9,692 sub-folders, and contains 44,292 sub-files. The second compact disc in the set of three compact discs (CD 2 of 3) includes a file folder named “ns” that was created on Oct. 22, 2003; it is 605 Mbytes in size, contains 12,628 sub-folders, and contains 44,934 sub-files. The third compact disc in the set of three compact discs (CD 3 of 3) includes a file folder named “ns” that was created on Oct. 22, 2003; it is 575 Mbytes in size, contains 9,881 sub-folders, and contains 43,630 sub-files. The files on each of these three compact discs can be executed on a Windows®-based computing device (e.g., IBM-PC, or equivalent) that executes a Windows®-brand operating system (e.g., Windows® NT, Windows® 98, Windows® 2000, Windows® XP, etc.). The files on each compact disc in this set of three compact discs are hereby incorporated by reference.
Each compact disc in the set of three compact discs itself is a CD-R, and conforms to the ISO 9660 standard. The contents of each compact disc in the set of three compact discs is in compliance with the American Standard Code for Information Interchange (ASCII).
Also accompanying this specification is a compact disc that stores tables listing the files included on the set of three compact discs (CD 1 of 3, CD 2 of 3, and CD 3 of 3), hereinafter referred to as the file table content disc. A duplicate copy of the file table content disc also accompanies this specification.
The file table content disc, and files stored thereon, may be read using a Windows®-based computing device (e.g., IBM-PC, or equivalent) that executes a Windows®-brand operating system (e.g., Windows® NT, Windows® 98, Windows® 2000, Windows® XP, etc.). The file table content disc conforms to the International Standards Organization (ISO) 9660 standard, and the contents of the file table content disc are in compliance with the American Standard Code for Information Interchange (ASCII). The file table content disc includes the following three files:
Each of these three files contains a table that has more than 50 pages of text. The files on the file table content disc are hereby incorporated by reference.
The TABLE1.TXT file is a table listing all the files, including their names, sizes in bytes, and dates of creation, on the first compact disc in the set of three compact discs (CD 1 of 3). In the table, these files are listed in the order of creation date, creation time, size in bytes, and name.
The TABLE2.TXT file is a table listing all the files, including their names, sizes in bytes, and dates of creation, on the second compact disc in the set of three compact discs (CD 2 of 3). In the table, these files are listed in the order of creation date, creation time, size in bytes, and name.
The TABLE3.TXT file is a table listing all the files, including their names, sizes in bytes, and dates of creation, on the third compact disc in the set of three compact discs (CD 3 of 3). In the table, these files are listed in the order of creation date, creation time, size in bytes, and name.
Very early on, computer software came to be categorized as “operating system” software or “application” software. Broadly speaking, an application is software meant to perform a specific task for the computer user such as solving a mathematical equation or supporting word processing. The operating system is the software that manages and controls the computer hardware. The goal of the operating system is to make the computer resources available to the application programmer while at the same time, hiding the complexity necessary to actually control the hardware.
The operating system makes the resources available via functions that are collectively known as the Application Program Interface or API. The term API is also used in reference to a single one of these functions. The functions are often grouped in terms of what resource or service they provide to the application programmer. Application software requests resources by calling individual API functions. API functions also serve as the means by which messages and information provided by the operating system are relayed back to the application software.
In addition to changes in hardware, another factor driving the evolution of operating system software has been the desire to simplify and speed application software development. Application software development can be a daunting task, sometimes requiring years of developer time to create a sophisticated program with millions of lines of code. For a popular operating system such as various versions of the Microsoft Windows® operating system, application software developers write thousands of different applications each year that utilize the operating system. A coherent and usable operating system base is required to support so many diverse application developers.
Often, development of application software can be made simpler by making the operating system more complex. That is, if a function may be useful to several different application programs, it may be better to write it once for inclusion in the operating system, than requiring dozens of software developers to write it dozens of times for inclusion in dozens of different applications. In this manner, if the operating system supports a wide range of common functionality required by a number of applications, significant savings in applications software development costs and time can be achieved.
Regardless of where the line between operating system and application software is drawn, it is clear that for a useful operating system, the API between the operating system and the computer hardware and application software is as important as efficient internal operation of the operating system itself.
Over the past few years, the universal adoption of the Internet, and networking technology in general, has changed the landscape for computer software developers. Traditionally, software developers focused on single-site software applications for standalone desktop computers, or LAN-based computers that were connected to a limited number of other computers via a local area network (LAN). Such software applications were typically referred to as “shrink wrapped” products because the software was marketed and sold in a shrink-wrapped package. The applications utilized well-defined APIs to access the underlying operating system of the computer.
As the Internet evolved and gained widespread acceptance, the industry began to recognize the power of hosting applications at various sites on the World Wide Web (or simply the “Web”). In the networked world, clients from anywhere could submit requests to server-based applications hosted at diverse locations and receive responses back in fractions of a second. These Web applications, however, were typically developed using the same operating system platform that was originally developed for standalone computing machines or locally networked computers. Unfortunately, in some instances, these applications do not adequately transfer to the distributed computing regime. The underlying platform was simply not constructed with the idea of supporting limitless numbers of interconnected computers.
To accommodate the shift to the distributed computing environment being ushered in by the Internet, Microsoft Corporation developed a network software platform known as the “.NET” Framework (read as “Dot Net”). Microsoft®.NET is software for connecting people, information, systems, and devices. The platform allows developers to create Web services that will execute over the Internet. This dynamic shift was accompanied by a set of API functions for Microsoft's .NET™ Framework.
As use of the NET™ Framework has become increasingly common, ways to increase the efficiency and/or performance of the platform have been identified. The inventors have developed a unique set of API functions to allow for such increased efficiency and/or performance.
A programming interface provides functions for generating applications, documents, media presentations and other content. These functions allow developers to obtain services from an operating system, object model service, or other system or service. In one embodiment, the functions allow a developer to generate a graphical user interface.
The same numbers are used throughout the drawings to reference like features.
This disclosure addresses an application program interface (API) for a network platform upon which developers can build Web applications and services. More particularly, an exemplary API is described for operating systems that make use of a network platform, such as the .NET™ Framework created by Microsoft Corporation. The .NET™ Framework is a software platform for Web services and Web applications implemented in the distributed computing environment. It represents the next generation of Internet computing, using open communication standards to communicate among loosely coupled Web services that are collaborating to perform a particular task.
In the described implementation, the network platform utilizes XML (extensible markup language), an open standard for describing data. XML is managed by the World Wide Web Consortium (W3C). XML is used for defining data elements on a Web page and business-to-business documents. XML uses a similar tag structure as HTML; however, whereas HTML defines how elements are displayed, XML defines what those elements contain. HTML uses predefined tags, but XML allows tags to be defined by the developer of the page. Thus, virtually any data items can be identified, allowing Web pages to function like database records. Through the use of XML and other open protocols, such as Simple Object Access Protocol (SOAP), the network platform allows integration of a wide range of services that can be tailored to the needs of the user. Although the embodiments described herein are described in conjunction with XML and other open standards, such are not required for the operation of the claimed invention.
As used herein, the phrase application program interface or API includes traditional interfaces that employ method or function calls, as well as remote calls (e.g., a proxy, stub relationship) and SOAP/XML invocations.
It should be appreciated that in some of namespace descriptions below, descriptions of certain classes, interfaces, enumerations and delegates are left blank. More complete descriptions of these classes, interfaces, enumerations and delegates can be found in the subject matter of the compact discs that store the SDK referenced above.
Exemplary Network Environment
Web services 102 are accessible directly by other services (as represented by communication link 106) or a software application, such as Web application 110 (as represented by communication links 112 and 114). Each Web service 102 is illustrated as including one or more servers that execute software to handle requests for particular services. Such services often maintain databases that store information to be served back to requesters. Web services may be configured to perform any one of a variety of different services. Examples of Web services include login verification, notification, database storage, stock quoting, location directories, mapping, music, electronic wallet, calendar/scheduler, telephone listings, news and information, games, ticketing, and so on. The Web services can be combined with each other and with other applications to build intelligent interactive experiences.
The network environment 100 also includes representative client devices 120(1), 120(2), 120(3), 120(4), . . . , 120(M) that utilize the Web services 102 (as represented by communication link 122) and/or the Web application 110 (as represented by communication links 124, 126, and 128). The clients may communicate with one another using standard protocols as well, as represented by an exemplary XML link 130 between clients 120(3) and 120(4).
The client devices, referenced generally as number 120, can be implemented many different ways. Examples of possible client implementations include, without limitation, portable computers, stationary computers, tablet PCs, televisions/set-top boxes, wireless communication devices, personal digital assistants, gaming consoles, printers, photocopiers, and other smart devices.
The Web application 110 is an application designed to run on the network platform and may utilize the Web services 102 when handling and servicing requests from clients 120. The Web application 110 is composed of one or more software applications 130 that run atop a programming framework 132, which are executing on one or more servers 134 or other computer systems. Note that a portion of Web application 110 may actually reside on one or more of clients 120. Alternatively, Web application 110 may coordinate with other software on clients 120 to actually accomplish its tasks.
The programming framework 132 is the structure that supports the applications and services developed by application developers. It permits multi-language development and seamless integration by supporting multiple languages. It supports open protocols, such as SOAP, and encapsulates the underlying operating system and object model services. The framework provides a robust and secure execution environment for the multiple programming languages and offers secure, integrated class libraries.
The framework 132 is a multi-tiered architecture that includes an application program interface (API) layer 142, a common language runtime (CLR) layer 144, and an operating system/services layer 146. This layered architecture allows updates and modifications to various layers without impacting other portions of the framework. A common language specification (CLS) 140 allows designers of various languages to write code that is able to access underlying library functionality. The specification 140 functions as a contract between language designers and library designers that can be used to promote language interoperability. By adhering to the CLS, libraries written in one language can be directly accessible to code modules written in other languages to achieve seamless integration between code modules written in one language and code modules written in another language. One exemplary detailed implementation of a CLS is described in an ECMA standard created by participants in ECMA TC39/TG3. The reader is directed to the ECMA web site at www.ecma.ch.
The API layer 142 presents groups of functions that the applications 130 can call to access the resources and services provided by layer 146. By exposing the API functions for a network platform, application developers can create Web applications for distributed computing systems that make full use of the network resources and other Web services, without needing to understand the complex interworkings of how those network resources actually operate or are made available. Moreover, the Web applications can be written in any number of programming languages, and translated into an intermediate language supported by the common language runtime 144 and included as part of the common language specification 140. In this way, the API layer 142 can provide methods for a wide and diverse variety of applications.
Additionally, the framework 132 can be configured to support API calls placed by remote applications executing remotely from the servers 134 that host the framework. Representative applications 148(1) and 148(2) residing on clients 120(3) and 120(M), respectively, can use the API functions by making calls directly, or indirectly, to the API layer 142 over the network 104.
The framework may also be implemented at the clients. Client 120(3) represents the situation where a framework 150 is implemented at the client. This framework may be identical to server-based framework 132, or modified for client purposes. Alternatively, the client-based framework may be condensed in the event that the client is a limited or dedicated function device, such as a cellular phone, personal digital assistant, handheld computer, or other communication/computing device.
Developers'Programming Framework
The framework 132 encapsulates the operating system 146(1) (e.g., Windows®-brand operating systems) and object model services 146(2) (e.g., Component Object Model (COM) or Distributed COM). The operating system 146(1) provides conventional functions, such as file management, notification, event handling, user interfaces (e.g., windowing, menus, dialogs, etc.), security, authentication, verification, processes and threads, memory management, and so on. The object model services 146(2) provide interfacing with other objects to perform various tasks. Calls made to the API layer 142 are handed to the common language runtime layer 144 for local execution by the operating system 146(1) and/or object model services 146(2).
The API 142 groups API functions into multiple namespaces. Namespaces essentially define a collection of classes, interfaces, delegates, enumerations, and structures, which are collectively called “types”, that provide a specific set of related functionality. A class represents managed heap allocated data that has reference assignment semantics. A delegate is an object oriented function pointer. An enumeration is a special kind of value type that represents named constants. A structure represents static allocated data that has value assignment semantics. An interface defines a contract that other types can implement.
By using namespaces, a designer can organize a set of types into a hierarchical namespace. The designer is able to create multiple groups from the set of types, with each group containing at least one type that exposes logically related functionality. In the exemplary implementation, the API 142 is organized to include three root namespaces. It should be noted that although only three root namespaces are illustrated in
The presentation subsystem namespace 200 pertains to programming and content development. It supplies types that allow for the generation of applications, documents, media presentations and other content. For example, presentation subsystem namespace 200 provides a programming model that allows developers to obtain services from the operating system 146(1) and/or object model services 146(2).
The shell namespace 202 pertains to user interface functionality. It supplies types that allow developers to embed user interface functionality in their applications, and further allows developers to extend the user interface functionality.
The web services namespace 204 pertains to an infrastructure for enabling creation of a wide variety of applications, e.g. applications as simple as a chat application that operates between two peers on an intranet, and/or as complex as a scalable Web service for millions of users. The described infrastructure is advantageously highly variable in that one need only use those parts that are appropriate to the complexity of a particular solution. The infrastructure provides a foundation for building message-based applications of various scale and complexity. The infrastructure or framework provides APIs for basic messaging, secure messaging, reliable messaging and transacted messaging. In the embodiment described below, the associated APIs have been factored into a hierarchy of namespaces in a manner that has been carefully crafted to balance utility, usability, extensibility and versionability.
The file system namespace 206 pertains to storage. It supplies types that allow for information storage and retrieval.
In addition to the framework 132, programming tools 210 are provided to assist the developer in building Web services and/or applications. One example of the programming tools 210 is Visual Studio™, a multi-language suite of programming tools offered by Microsoft Corporation.
Root API Namespaces
As shown in
The presentation subsystem 200 is the root namespace for much of the presentation functionality of the API 142. A controls namespace 310 includes controls used to build a display of information, such as a user interface, and classes that allow a user to interact with an application. Example controls include “Button” that creates a button on the display, “RadioButton” that generates a radio-style button on the display, “Menu” that creates a menu on the display, “ToolBar” that creates a toolbar on the display, “Image” that generates an image on the display and “TreeView” that creates a hierarchical view of information.
Certain controls are created by nesting and arranging multiple elements. The controls have a logical model that hides the elements used to create the controls, thereby simplifying the programming model. The controls can be styled and themed by a developer or a user (e.g., by customizing the appearance and behavior of user interface buttons). Some controls have addressable components that allow an individual to adjust the style of individual controls. Additionally, the controls can be sub-classed and extended by application developers and component developers. The controls are rendered using vector graphics such that they can be resized to fit the requirements of a particular interface or other display. The controls are capable of utilizing animation to enhance, for example, the interactive feel of a user interface and to show actions and reactions.
Controls Namespace
The controls namespace 310 includes one or more panels, which are controls that measure and arrange their children (e.g., nested elements). For example, a “DockPanel” panel arranges children by docking each child to the top, left, bottom or right side of the display, and fills-in the remaining space with other data. For example, a particular panel may dock menus and toolbars to the top of the display, a status bar to the bottom of the display, a folder list to the left side of the display, and fills the rest of the space with a list of messages.
As mentioned above, System.Windows.Controls.Primitives is a namespace that includes multiple controls that are components typically used by developers of the controls in the System.Windows.Controls namespace and by developers creating their own controls. Examples of these components include “Thumb and RepeatButton”. “ScrollBar”, another component, is created using four repeat buttons (one for “line up”, one for “line down”, one for “page up”, and one for “page down”) and a “Thumb” for dragging the current view to another location in the document. In another example, “ScrollViewer” is a control created using two “ScrollBars” and one “ScrollArea” to provide a scrollable area.
The following list contains example classes exposed by the System.Windows.Controls namespace. These classes allow a user to interact with, for example, an application through various input and output capabilities as well as additional display capabilities.
The System.Windows.Controls namespace also contains various enumerations. The following list contains example enumerations associated with the System.Windows.Controls namespace.
“Position” is an example structure associated with the System.Windows.Controls namespace. A user of the Generator describes positions using this structure. For example: To start generating forward from the beginning of the item list, specify position (−1, 0) and direction Forward. To start generating backward from the end of the list, specify position (−1, 0) and direction Backward. To generate the items after the element with index k, specify position (k, 0) and direction Forward.
The following list contains example delegates associated with the System.Windows.Controls namespace.
Another namespace, System.Windows.Controls.Atoms, is a sub-namespace of the System.Windows.Controls namespace. System.Windows.Controls.Atoms includes associated controls, event arguments and event handlers. The following list contains example classes associated with the System.Windows.Controls.Atoms namespace.
The following list contains example delegates associated with the System.Windows.Controls.Atoms namespace.
A System.Windows.Controls.Primitives namespace is another sub-namespace of the System.Windows.Controls namespace. As mentioned above, the Primitives sub-namespace includes controls that are intended to be used as primitives by other more complex controls. The following list contains example classes associated with the System.Windows.Controls.Primitives namespace.
“IEnsureVisible” is an example interface associated with the System.Windows.Controls.Primitives namespace. IEnsureVisible is implemented on a visual to scroll/move a child visual into view.
The following list contains example enumerations associated with the System.Windows.Controls.Primitives namespace.
Documents Namespace
A documents namespace 312 is a collection of semantic and formatting elements that are used to create richly formatted and semantically rich documents. In one embodiment, an “element” is a class that is primarily used in conjunction with a hierarchy of elements (referred to as a “tree”). These elements can be interactive (e.g., receiving user input via keyboard, mouse or other input device), can render images or objects, and can assist with the arrangement of other elements. Example elements include a “Block” element that implements a generic block, a “Body” element that represents content that includes the body of a table, a “Cell” element that contains tabular data within a table, a “Header” element that represents the content included in the header of a table, and a “PageBreak” element that is used to break content across multiple pages.
The following list contains example classes exposed by the System.Windows.Documents namespace.
The following list contains example interfaces associated with the System.Windows.Documents namespace.
The following list contains example enumerations associated with the System.Windows.Documents namespace.
The following list contains example delegates associated with the System.Windows.Documents namespace.
Shapes Namespace
A shapes namespace 314 is a collection of vector graphics elements that is used to create images and objects. The use of vector graphics elements allows the elements to be easily resized to fit the requirements of a particular interface or display device. Example elements include an “Ellipse” element that draws an ellipse, a “Line” element that draws a straight line between two points, a “Rectangle” element that draws a rectangle, and a “Polygon” element that draws a polygon as a connected series of lines that form a closed shape.
The following list contains example classes exposed by the System.Windows.Shapes namespace.
Data Namespace
A data namespace 316 includes classes and interfaces used to bind properties of elements to data sources, data source classes, and data-specific implementations of data collections and views. These classes and interfaces are also used to handle exceptions in data entry and allow runtime creation of a user interface based on information in various data sources. Data can be displayed in textual form or can be utilized to change the formatting of the display, such as displaying dollar amounts in red if they are negative. Example classes include a “Bind” class that represents a binding declaration object that manages bindings between a dynamic property user interface and source data, and an “XmlDataSource” class that serves as a data source for data binding to XML content nodes.
Object-oriented applications typically represent data by classes that define both the value of a piece of data and the operations that can be performed on that data. The term “data item” refers to one such object. Applications can handle individual data items or collections of data items. They may use data items in three ways: (a) converting data from external sources such as file systems, remote servers, databases, etc. into the corresponding in-memory data items, and converting modified data items back into the form expected by these sources; (b) operating on the data items using a combination of data-centric and application-centric logic; (c) presenting the data embodied by the data items to the user through a user interface. Data namespace 316 provides support for the first and third of these tasks.
The first task, obtaining data from external sources, is supported by “data source” objects. A data source object is typically defined as a page-wide or application-wide resource, and serves as the gateway to the data. Data sources implement an IDataSource interface, which defines a standard mechanism by which the classes in the data namespace get access to the data. A particular data source object implements logic for retrieving the actual data by using mechanisms appropriate for the particular source. In one embodiment, the data namespace includes four data source classes:
A data source class is responsible for retrieving data from an external source and converting it into one or more data items suitable for use by the binding classes. If a collection of data items is needed, the application can use any of the standard collection classes from the Net Framework such as Array, ArrayList, Hashtable, etc., any of the data-centric collection classes from the System.Data namespace such as Dataset, or a data-centric collection class from the data namespace such as ArrayListDataCollection. The latter classes support change notifications; i.e., when the application changes the collection by adding an item, removing an item, sorting the collection, etc., the collection sends a notification. The binding classes listen for these notifications and automatically update the user interface to reflect the change.
Once the data has been converted to in-memory data items, the application can perform computations using the items and can modify the items as a result of the computations. These actions are performed using a combination of data-centric operations (defined by the data item classes) and application-centric operations (defined by the application itself). The actions may be initiated by the application automatically, or in response to an action of the user. Special support or cooperation from the data namespace is not necessary, thereby providing a clean separation of logic and presentation within the application.
The third data-related task, presenting the data through the user interface, is supported by the “binding” classes of the data namespace. These classes enable the application to describe the correspondence (binding) between a data item property (the source) and a user interface property (the target). The term data-binding (or simply binding) refers to the establishment of such a correspondence. For example, an application may choose to data-bind the Text property of a Textbox control to the CustomerName property of a data item. Having done so, the control will automatically display the customer's name, updating the display whenever the application changes the data item, and updating the data item whenever the user types a new name into the control.
This kind of correspondence is described using the Bind class, and implemented using the Binding class. Any number of UI properties can share the same description (Bind), but each property has its own unique Binding that holds the state for that particular instance. The description includes the following information about the desired correspondence:
All bindings except one-time rely on getting notified when the data property changes, so that the corresponding change can be made to the user interface. The binding classes recognize the IPropertyChange interface (from the System.ComponentModel namespace) as one way of implementing the required notifications.
The following tables list the members exposed by the System.Windows.Data namespace.
Media Namespace
A media namespace 318 provides various media classes. Application developers as well as component developers may use these classes to develop various presentation functionality. Example classes in media namespace 318 include an “ImageEffect” class that permits certain imaging effects (e.g., blur and grayscale), and a “Brush” class that provides a mechanism for filling an area using solid colors, gradients, images, video, and the like.
The media namespace 318 includes a sub-namespace System.Windows.Media.Animation that includes services that allow a developer to animate properties and coordinate a set of animations with a set of timelines. An animation is an object that changes a value over a period of time. Animation effects include moving an object on the display, and changing the size, shape, or color of an object. Multiple animation classes are provided to implement various animation effects. Effects can be achieved by associating an animation with an element's property value. For example, to create a rectangle that fades in and out of view, one or more animations are associated with the opacity property of the rectangle.
The media namespace 318 also includes a sub-namespace System.Windows.Media.TextFormatting that provides various text services. For example, a “TextFormatter” text engine provides services for breaking text lines and formatting text presented on a display. “TextFormatter” is capable of handling different text character formats and paragraph styles as well as handling international text layout.
The following tables list example members exposed by the System.Windows.Media namespace.
Media Integration Layer
A Media Integration Layer (MIL) provides an API for developers or programmers to accomplish possibly complex composition effects within their applications in a straightforward manner, while leveraging the graphics processing unit in a manner that does not adversely impact normal application performance. One aspect of the MIL provides the ability to combine different media types (e.g., 2D, 3D, Video, Audio, text, imaging and so forth) and animate them together smoothly and seamlessly.
The MIL provides a graphics architecture for multi-stage composition and a programming model that allows for functional parity at the programmatic and scripted interfaces. An API and script allows the creation of a retained structure or scene description that is composited when rendered, yet includes areas that have a more immediate-mode nature.
Via the interfaces, the MIL provides access to a data structure for storing visual information so that applications can take advantage of the graphics capabilities provided by the computer hardware. The interfaces are based on an element object model and a vector graphics markup language for using that element object model in a manner that allows program code developers to consistently interface with a scene graph data structure to produce graphics. The data structure may also be used for either directly rendering or for “compiling” the visual information so that it can be provided to a lower level graphics system for fast composition and animation.
The vector graphics element object model generally corresponds to shape elements and other elements including image and video elements that correlate with a scene graph object model of the scene graph. Markup may be parsed into data including elements in an element tree that is translated into the objects of a scene graph data structure. Other markup may be translated directly into data and calls that create the scene graph objects. The markup language provides distinct ways to describe an element, including a simple string format or complex property syntax, which may be named, enabling reuse in other locations in the markup.
An aspect of the MIL is the integration of animation and timing across the API set, providing animation as an inherent base-level concept. To facilitate smooth animation, the MIL provides s multiple-level graphics processing system and method (e.g., of an operating system). One such multiple-level graphics processing system comprises two components, including a tick-on-demand or slow-tick high-level component, and a fast-tick (e.g., at the graphics hardware frame refresh rate) low-level component. In general, the high-level, less frequent component performs computationally intensive aspects of updating animation parameters and traversing scene data structures, in order to pass simplified data structures to the low-level component. The low-level component operates at a higher frequency, such as the frame refresh rate of the graphics subsystem, to process the data structures into constant output data for the graphics subsystem. The low-level processing includes interpolating any parameter intervals as necessary to obtain instantaneous values to render the scene for each frame of animation.
Top level MIL objects include a visual tree, which is an object that contains the main content to be drawn. Controls will derive from visuals of the tree directly. Visuals are device and parent context independent. A render target is the device to which the visual is drawn. This object (e.g., screen) may have its own dirty or invalidation mechanism. Various render targets include a screen in a window, a Printer, a Metafile, a Surface, and a “Sub-window” which is a part of the scene that is drawn separately from the rest of the scene. Other drawing related objects include a Visual Renderer, comprising an object that is configured to draw a visual tree onto a render target, and a Display Scheduler object that knows when to draw the visual tree on to the render target. A Time Manager is a context object for a set of timing nodes, and is the object that the scheduler calls tick on.
A Visual API is provided, which is essentially a starting point for drawing via the media integration layer, and comprises multiple types of objects, including a VisualManager object, which connects a Visual Tree to a medium. The different types of VisualManagers (e.g., Screen, Printer, and Surface) are responsible for rendering a Visual Tree to their particular medium. A visual is where the programmer does the drawing; it is a node in the visual tree and provides a place for a program to draw.
The DrawingContext APIs present a context-based programming model for how to construct visual content that populates a Visual or are rendered to an ImageData. DrawingContext classes are provided, as well as the classes and entrypoints necessary to acquire a DrawingContext and enumerate the visual content in a RetainedVisual/DrawingVisual.
To enable mutability, there is provided a single set of types that derive from a common Changeable base class. Any type for which mutability is desired change may derive from the Changeable class. For example, in a graphics programming, the object model includes Brushes, Pens, Geometries, FloatAnimations, GradientStops, Segments, and so forth. An IsChangeable property specifies whether the changeable object can be modified or not, depending on its current value, which defines a state.
A brush is an object that represents a method to fill a plane. In addition to being able to fill a plane in an absolute way, brushes of the media integration layer are also able to adapt how they fill the plane relative to the size of the object that they are filling. Examples of types of brushes include SolidColorBrush, VisualBrush (which can reference a vector graphics resource/Visual), DrawingBrush, LinearGradient, RadialGradient, ImageBrush and NineGridBrush. Certain brush objects will have an idea of how they relate to the coordinate system when they are used, and an idea of how they relate to the bounding box of the geometry with which they are used. This size is based on from the object that the brush is filling. The Brush base class has a Transform, a general opacity, and a blend mode. Brush (and other object resources in Vector Graphics and the MIL API) objects are Changeables and are writable after they have been created, and follow the general Changeable pattern for how they behave after they are used in qualified use.
A Geometry class of objects can be used for clipping, hit-testing and rendering of 2D vector-based data with the Pen and Brush. The derived Geometry classes provide more specific building and enumeration semantics. A number of shape-specific Geometry types are provided, as well as a generalized PathGeometry that allows for explicit definition of more complex shaped Geometry. Geometry is an abstract base class. A GeometryCollection is a collection of multiple Geometry objects that have been combined using particular CombineMode operations on their defined area. This object allows easier building visual combinations of Geometry objects than is possible using strictly PathFigure objects within a PathGeometry.
ImageSource is an abstract class, comprising a basic building block for imaging. An ImageSource conceptually represents a single, constant set of pixels at a certain size and resolution. For example, an ImageSource may be a single frame in an image file that a Decoder could provide, or it may be the results of a transform that operates on a certain ImageSource of its own. An ImageSource is changeable, not because its own properties can be changed, but because the properties of its sub-classes can potentially be changed.
A Transform class of objects is provided for scaling, rotating, translating and skewing vector and raster graphics. The derived Transform classes provide friendly usage and enumeration semantics.
Effects provide a means to alter the visual contents of a scene in a rendering-centric manner. For example, VisualEffects (raster-based bitmap effects) operate on the image-based, fully composited representation of a portion of a scene. Effects are broken down into various types including VisualEffects, BlendModes and VectorEffects. VisualEffects can be used in a retained-mode scene by applying it to a sub-graph or an Element, or it can be used in the standalone image pipeline. BlendModes are a specific form of image-based effects, and can be applied to the retained mode scene in generally the same manner as VisualEffects. Blend modes perform a combination of the source and destination colors as the source is composited, e.g., multiply or add.
Hit testing is used to pick visuals in a scene, and operates by starting from the top of the control tree, and returning a control or set of controls by a point or geometry. A control can define whether it is hit or not with support services including rendered geometry, bounding box, out-of-band geometry (hit region), image opacity or mask, and its own logic. The control can return specific hit-related data on hit. The hit test mechanism can filter hit test results in an efficient manner. The hit test walk is a deep right to left walk of the visual tree, with hits reported through a callback in z-order, top-to-bottom fashion. When descending, the hit tester views the filtering in terms of element level relationships, for example, a canvas with shapes, or a dock panel with an inner canvas. When a hit occurs, the hit tester can either continue processing further hits (if any), or stop.
An animation system is provided, comprised of a timing control engine and a set of animation objects. The timing engine is a service that can be used by any objects that exhibit time-varying behaviors, e.g., animations and audio or video media objects. Animation objects implement a set of functions that map time spans into other data types, which are then used as inputs into other higher-level objects. Graphical animation is achieved by associating an animation collection with a rendering operation. Each animation used in a rendering operation may be run on a separate clock, referred to as a called a “timeline.” Once an animated primitive is drawn and animation parameters specified, the low-level rendering system takes care of redrawing the scene at regular intervals. Each time a frame is rendered the current value of the animations involved in the scene is computed, based on the elapsed time (in most cases measured by the system clock), and then the animated primitives are redrawn.
Various primitive types, color features and media support is also provided via the MIL. MediaData can be used to play any audio/video content.
Design Namespace
A design namespace 320 provides classes that enable the editing of forms and text, formatting data and cross-process data sharing. These classes provide an extensible framework for editing documents, applications, and other content. Design namespace 320 contains functionality on two levels: high-level functionality for application developers desiring pre-packaged ready-to-use editors for different types of information; and lower-level functionality for more advanced applications introducing its own types of data. Example pre-packaged ready-to-use editors offer plain text editing, rich text editing, forms (element layout) editing, and password input.
Design namespace 320 provides a flexible and extensible approach to organizing various functionality. Instead of providing a common fixed-feature editor, design namespace 320 provides a combination of services, behaviors and abstractions that allow developers to build specialized and easily-customizable information editing solutions. Design namespace 320 includes several customizable editing features, including: scoped services, stackable behaviors, editor-designer pattern, abstract designer for element editing, abstract text object model, adorner layer, design surface, generalized data transfer protocol, and extensible undo mechanism.
Scoped services allows the association of specific services with particular portions and sub-portions of application data. This mechanism also allows turning services on and off in different scopes. The main class providing a support for services is ServiceManager; individual services are implemented using IService and IScopedService interfaces.
The concept of “stackable behaviors” allows different behaviors to be activated in appropriate periods of time. In one embodiment, time-based behavior activation is nested, such that a sub-process can be started and finished while a more general process is temporary suspended and then properly restored. Rather than hard-coding a solution, the approach of stackable behavior solves this problem by allowing an integration of processes that do not have pre-defined knowledge about each other. The main class providing support for this mechanism is EditRouter; individual behaviors are subclasses of Editor and EditBehavior classes. Built-in types of editable data are supported by corresponding subclasses of EditBehavior such as TextEditor, ElementEditor and MoveBehavior.
The concept of “editor-designer pattern” allows separation between generic editing functionality and more specific sub-typed of editable information. The editor does not expect a specific data structure on which to operate. Instead, it assumes that data content is exposed through some abstract interface, hiding the implementation details, but sufficient for corresponding action of editing interaction. This pattern is particularly useful with elements (in forms editing) and for text (in rich text editing).
“Abstract designer for element editing” is an application of the “editor-designer pattern” for an area of element editing (e.g., forms editing). Various elements may have different internal structures, appearances and behaviors, but if they expose some standard “designers” for movement, rotation, or resizing, then they become editable by the element editor.
The abstract text model is another application of the “editor-designer pattern” for use in rich text editing. This abstraction allows different backing stores to participate in rich text editing, such as RTF, XHTML, plain text, XAML markup, syntactically highlighted source code, etc. Even though the semantic nature and internal structure of these types of texts can be different, by exposing itself via an abstract text model they allow the application of generic editing mechanisms. Additionally, this approach allows for text and data integration, such that text from different backing stores may contain pieces of each other, while providing a seamless editing experience.
The adorner layer is an unified tool that provides rich visual feedback during editing. The adorner layer provides uniformed support for many types of feedback. Powerful visuals may be dynamically attached and detached to elements and other portions of data to indicate potential user activity applicable to them. The design surface is a XAML control that pre-packages editing resources, such as the adorner layer and edit router. This simplifies middle-level editing development.
Editing functions such as cut/copy/paste and drag/drop are important in an integrated information environment, but can be difficult to support due to the complexity and incompatibility of different data types. The generalized data transfer protocol addresses this difficulty by providing for the combination of abstractions used for data extraction, data insertion, data dragging, and data conversion. This generalized data transfer protocol provides a powerful and flexible mechanism for application-data integration.
The extensible undo mechanism manages an undo stack by collecting data of various types. The undo manager includes protocols to handle undo operations with different data types.
The following tables list example members exposed by the System.Windows.Design namespace.
Input Namespace
An input namespace 322 includes an input manager that coordinates inputs received by the system. The input namespace 322 also includes classes that help manage and provide control for different input devices, such as a keyboard or a mouse. The input system allows applications to get information from input devices such as mice, keyboards, and styluses (pens). Most input functionality is in the form of properties, methods, and events on the UIElement, FrameworkElement, ContentElement, and ContentFrameworkElement classes in the System.Windows namespace. UIElement and ContentElement have similar input functionality, which is captured by the IInputElement interface. Similarly, FrameworkElement and ContentFrameworkElement, which derive from UIElement and ContentElement respectively, share many input APIs and both implement IFrameworkInputElement.
The input system provides full support for keyboards, mouse, and styles. Keyboard functionality includes keydown/up events, focus management and notification, accessing current key state, and converting keystrokes into input strings. Mouse functionality includes mouse position, clicks, movement, enter/leave events at element boundaries, mouse capture, hover and mouse wheel. Stylus functionality includes position and movement (both when the pen is touching the surface and when it is “in air”), taps/clicks, capture, and gesture recognition.
The Input namespace contains helper classes necessary for the above functionality, such as enumerations, events arguments, delegate signatures, etc. Additionally, the Input namespace includes the Keyboard, Mouse, and Stylus classes, which provide information relating to the current state of those devices. The input system provides ways of specifying mouse cursors on a particular element, and for an entire application.
The input system is integrated with Text Services Framework (TSF), which allows other input software to understand the textual context into which new text is being input. This is used by input method editors (IMEs), which allow users of East Asian languages to turn multiple keystrokes into a single character—the IME software communicates with TSF to identify the best interpretation of the keystrokes based on context and uses TSF to insert the text into the document. Similarly, speech recognition uses TSF to pick the best recognition and insert it into the document.
The input system offers automatic integration with TSF, in the form of a TextInput event. The Input namespace also offers APIs for describing the textual context, and controlling the conversion from raw keyboard input into textual input. The InputManager class provides filters and monitors, which allow a third party to observe the input stream and create new input events before the original input events have been fired. The InputManager also provides APIs for new devices to be plugged into the input system, using IInputProvider and InputReport.
The following tables list example members exposed by the System.Windows.Input namespace.
Navigation Namespace
A navigation namespace 324 provides a set of classes and services that allow the building of applications with navigation paradigms, such as a browser application. These classes and services permit the development of applications with customized navigation experiences. For example, when purchasing a product or service from an online merchant, clicking a “Back” button causes the application to display a different page that asks the user if they want to cancel or change their order. In another example, activating a “Refresh” button causes an application to retrieve new data instead of first reloading the application followed by retrieving the new data. The navigation namespace 324 also includes page functions that provide a mechanism for generating a hierarchy of questions that are presented to a user.
The following tables list example members exposed by the System.Windows.Navigation namespace.
Automation Namespace
An automation namespace 326 contains members used to support accessibility and user interface automation. The accessibility system includes a client side and a provider side. A system of tools includes client side automation tools including a client automation class for seeking user interface information. The client automation class includes event registration tools and logical element discovery tools. The set of tools further includes provider side automation tools for providing the client with user interface information. The provider side automation tools include an automation provider class having tools for providing the client with event information.
A client automation class provides UI automation methods for one or more clients. The client automation class contains methods that are not specific to any UI element. The client automation class may provide a method for obtaining a logical or raw element from a point, a window handle, or the desktop root element. The client automation class additionally may provide methods for finding a logical element based input criteria. The client automation class preferably also includes a method for registering and un-registering for event notifications. The automation class preferably also provides helper functions for loading proxy DLLs, retrieving the localized names of properties and control patterns and performing element comparisons. The client automation class also includes methods for clients to listen for events. The following tables list example members exposed by the System.Windows.Automation namespace.
Serialization Namespace
A serialization namespace 328 provides a parser that can load or save a hierarchy of objects (e.g., elements) from or to an XML file or a file with a binary representation. This process also sets properties associated with the objects and associates event handlers. Further, serialization namespace 328 provides functionality for serializing and de-serializing objects. Serialization is a process of taking a live, in-memory representation of objects, and producing a stream of data suitable for saving to a storage device (such as a disk) or sending across a network. Deserialization is the reverse process; i.e., taking a data stream and producing objects from the data stream. Deserialization is also referred to as “parsing”, and the code that implements parsing is known as the parser.
Both serialization and deserialization support multiple streaming formats, such as an XML format referred to as XAML or a binary format referred to as BAML. When the parser is used with a markup compiler, if can generate “CAML”, where the output is code to generate the serialized objects.
Serialization and deserialization support the full range of XAML features, including: standard syntax, compound properties, compact compound syntax, implicit child syntax, explicit children syntax, collection syntax with explicit collection tag, and collection syntax with implicit collection tag. Both serialization and deserialization are fully extensible. Classes can define how they wish to be serialized and deserialized through several mechanisms, such as:
A second extensibility mechanism is “designer hooks”, which allows another piece of code to plug into the serializer and deserializer, and insert and intercept attributes during parsing and serialization. This mechanism allows new markup features to be created, and allows custom processing of existing syntax. The markup compiler uses designer hooks to generate CAML.
Serialization and deserialization support a variety of objects (e.g., any class driving from System.Object). Classes driving from System.Windows.DependencyObject can define dependency properties, which get two extra features:
Interop Namespace
An interop namespace 330 provides a set of classes that enable interoperability with other operating systems or computing platforms. The following tables list example members exposed by the System.Windows.Interop namespace.
Forms.Interop Namespace
A forms.interop namespace 332 provides an element that allows an application to host a form control operation.
System.IO.CompoundFile Namespace
Another namespace, System.IO.CompoundFile (not shown in
System.Windows.Automation Namespace
A System.Windows.Automation namespace contains members used to support accessibility and user interface automation. The accessibility system includes a client side and a provider side. A system of tools includes client side automation tools including a client automation class for seeking user interface information. The client automation class includes event registration tools and logical element discovery tools. The set of tools further includes provider side automation tools for providing the client with user interface information. The provider side automation tools include an automation provider class having tools for providing the client with event information.
A client automation class provides UI automation methods for one or more clients. The client automation class contains methods that are not specific to any UI element. The client automation class may provide a method for obtaining a logical or raw element from a point, a window handle, or the desktop root element. The client automation class additionally may provide methods for finding a logical element based input criteria. The client automation class preferably also includes a method for registering and un-registering for event notifications. The automation class preferably also provides helper functions for loading proxy DLLs, retrieving the localized names of properties and control patterns and performing element comparisons. The client automation class also includes methods for clients to listen for events. The following tables list example members exposed by the System.Windows.Automation namespace.
System.Windows.Ink Namespace
A System.Windows.Ink namespace provides classes that support electronic ink processing systems. Electronic ink processing techniques are useful to a variety of software applications. These electronic ink processing techniques are particularly applicable to the analysis of electronic ink, including layout analysis, classification, and recognition of electronic ink. Certain electronic ink processing techniques allow the electronic ink to be processed asynchronously with regard to the operation of the software application implementing the techniques, so that the electronic ink can be processed without stopping or significantly delaying the operation of the software application. The software application can even continue to accept new electronic ink input while previous electronic ink input is being processed.
An application programming interface instantiates an ink analyzer object that receives document data for a document containing electronic ink content from a software application hosting the document and running on a first processing thread. The ink analyzer object then employs the first thread to make a copy of the document data, provides the copy of the document data to an electronic ink analysis process, and returns control of the first processing thread to the analysis process. After the analysis process has analyzed the electronic ink, the ink analyzer object reconciles the results of the analysis process with current document data for the document.
In particular embodiments, elements in a file or document may be described based upon their spatial position relative to each other. For example, both an electronic ink stroke and typewritten text may be described in terms of the same spatial coordinate system. Using spatial information to describe the elements of a document, the software application managing the document can maintain a data structure describing the relationship between its document elements. In particular, the software application can maintain a data structure both describing the class of the various document elements and defining associations between the various document elements. These associations can be defined, for example, as information used to link electronic ink stroke data or collections thereof to other elements in the electronic document (such as words, lines, paragraphs, drawings, table cells, etc.).
By describing document elements in a file or document data structure based upon their spatial position, document elements for a variety of file types can employ common techniques for identifying and manipulating their document elements. More particularly, a variety of software applications can describe document elements within a document based upon their spatial position and employ this spatial position referencing to use common electronic ink analysis methods. Still further, by specifying a particular region of a document for analysis, each software application can limit the analysis process to only desired elements within a document.
To analyze new electronic ink input into a document, the software application managing the document modifies a data structure associated with the document to include the new ink to be analyzed. The software application then provides this data structure (or relevant portions thereof) to an ink analysis tool, which copies some or all of the data structure for analysis (and operates on this copy of the data that is independent of the application program's document data structure). The ink analysis tool passes the copy to an analysis process, such as a parsing process (e.g., a layout analysis process and/or a classification process). The software application may resume its normal operation, including receiving new electronic ink input and/or other data, while the ink analysis process(es) is (are) being performed. In addition to receiving new electronic ink, any “other data” may be received by the application program, for example, data modifying a size, location, or content of existing ink, text, images, graphics, tables, flowcharts, diagrams, and the like; data adding additional text, images, graphics, tables, flowcharts, diagrams, and the like; data deleting existing text, images, graphics, tables, flowcharts, diagrams, and the like. After all desired analysis processes are completed, the analysis results are returned to the ink analysis tool.
An ink analysis tool performs a variety of functions to facilitate the processing of electronic ink. The ink analysis tool can be implemented as a programming interface, such as an API. Further, the ink analysis tool can be implemented as a group of software object routines and related information that can be called by a software application as needed to analyze ink in a document.
In one embodiment, an API embodying an implementation of an ink analysis tool (referred hereafter to as an Ink Analysis API), may contain two core classes. The first class is referred to as the “Analysis Context” class, and the second class is the “Ink Analyzer” class. The components of the “Analysis Context” class are used to create an analysis context object. The components of the “Ink Analyzer” class are then used to create and employ an object that provides a document independent analysis context object to an analysis process, determine when the analysis results have been generated, and reconcile the analysis results with the current state of a document.
The Analysis Context class is implemented by a host application to create an analysis context object, which serves as a proxy view onto the internal document tree of the software application. The analysis context object contains all unanalyzed ink data, and the analysis context object is used to identify which unanalyzed ink data should be analyzed. The analysis context object also contains information about previously analyzed ink. This previously analyzed ink may be used to decide how the currently unanalyzed ink should be analyzed, and itself may be modified in the course of analyzing the unanalyzed ink. Further, the analysis content object contains information about non-ink content of the document, which is used to properly classify ink as annotations to the non-ink content.
The Analysis Context class includes a constructor which, when called by the software application, creates the analysis context object. This class may also include various properties for the analysis context object, including the property entitled “Dirty Region {get;}.”. The Dirty Region property defines the portion of the document (and thus the portion of the analysis context object) that contains unanalyzed ink data.
The Analysis Context class may also include a property entitled “Rootnode {get;},” which identifies the topmost or root context node in the analysis context object. This root context node contains, as child context nodes, all other context nodes objects for the given analysis context object. In particular embodiments, the root context node should be of the context node type “Root.” In embodiments where the application implements its own analysis context object, the analysis context object may have other context nodes as siblings of the root context node, but components of the Ink Analyzer class may be limited to considering context nodes contained by the root context node.
The Analysis Context class may additionally include the property “Analysis Hints {get;},” which returns an array of analysis hint objects set by the software application. As will be discussed in more detail below, the analysis hint objects may contain any type of information that may assist the analysis process. This information may include, for examples, as factoids, guides, or a word list. It may also include information setting a language to be used for analysis, information designating the unanalyzed ink as handwritten text only or drawing only, or providing any kind of guidance to the parsing process, such as identifying the ink as lists, tables, shapes, flow charts, connectors, containers and the like.
In addition to these properties, the Analysis Context class may also include various methods that may be called by, e.g., the software application to have the analysis context object execute a task. For example, the Analysis Context class may include a method entitled “FindNode (Guid id).” Each node in the analysis context object has a globally unique identifier (or GUID), and this method will locate the node specified in the call anywhere in the analysis context object.
Like the Analysis Context class, the Ink Analyzer class also defines a public constructor that allows the software application to create an instance of the class (i.e., an ink analyzer object), along with various properties. For example, it may contain a property entitled “User Interface Context {get; set;},” which defines the processing thread to which the results of the analysis process are returned. This property allows the results to be synchronized to another object. For example, if this is set to the main form, parser results will be fired on the application main thread. It may also contain the property “AnalysisOptions AnalysisOptions {get; set;}”, which specifies various criteria that may be used for the analysis process. These criteria may include, for example, enabling text recognition, enabling the use of tables, enabling the use of lists, enabling the use of annotations, and enabling the use of connectors and containers.
The Ink Analyzer class will include various methods. For example, this class may include a method entitled “AnalysisRegion Analyze ( ).” This method starts a synchronous analysis process. Document element data is passed into this method, which describes the current state of the document and indicates what ink in the document needs to be analyzed. With some embodiments, the document element data can be provided as the analysis context object (i.e., AnalysisRegion Analyze (Analysis Context)), as noted above. Alternately, individual ink strokes may be passed to the analysis process, either using a reference to the strokes (i.e., AnalysisRegion Analyze (Strokes)) or referenced as a property of the Ink Analyzer object (e.g., InkAnalyzer.Strokes {get; set}) with no properties passed to the Analyze method.
Once the analysis process is complete, this method will return a reference to the document independent analysis context object that has been modified to contain the results of the analysis process. The method also returns an AnalysisRegion value describing the area in the document where results have been calculated.
The Ink Analyzer class may also include a method entitled “AnalysisRegion Analyze(AnalysisContext, waitRegion).” This method is the same as the synchronous Analysis Region Analyze (method discussed above, but it only has ink analyzed if results are needed in the specified waitRegion area. More particularly, the call to this method will identify the analysis context object for the document and the region of the analysis context object (referred to as the “waitRegion”) for which the analysis process should analyze synchronously. In certain embodiments, all other regions of the analysis context object will be ignored, unless the analysis process needs to analyze content in those regions in order to complete its analysis of the waitRegion. As discussed above, the analysis context object passed with this method contains a property called the “DirtyRegion” which describes the areas of the document that need analysis. By specifying a particular waitRegion, the software application may obtain the analysis results more quickly for one specific region of interest, rather than having all of the ink data in the document analyzed. When either of these Analyze methods is called, every available analysis process will be executed. Also, because these Analyze methods are synchronous calls, there is no need to execute the reconciliation process upon their conclusion, nor will an event fire once it is complete.
The Ink Analyzer class may also include a method entitled “BackgroundAnalyze(AnalysisContext).” This method starts the specified analysis operation, but does so on a separate background analysis thread. Thus, this method will return control to the main processing thread almost immediately while the actual analysis operation completes in the background. In particular, this method will return a value of “true” if the analysis process was successfully started. Again, the AnalysisContext value passed in to the method identifies the analysis context object for the document and indicates what ink in the document needs to be analyzed. Once the analysis operation has completed on the background thread, a Results event is raised to allow the software application to access the results. The event contains the results and the reconcile method which is used to incorporate the results back into the analysis context object for the current state of the document when the results are returned.
Each of these three Analyze methods in turn call the method “Clone” in an “Analysis Region” class. Using the “Clone” method, these Analyze methods create the independent document analysis context object that is subsequently modified by the analysis process to show the analysis results.
The Ink Analyzer class may also include a method entitled “Reconcile (AnalysisContext current, AnalysisResultsEventArgs resultArgs),” which the software application calls after receiving the results event which was caused by calling the BackgroundAnalyze(AnalysisContext) method. The Reconcile method compares the analysis results contained in the document independent analysis context object with the current version of the analysis context object maintained by the software application. This method identifies the nodes that need to be added and removed from the current version of the analysis context object, and identifies if any of the following properties of an existing node has changed: its recognition results, its location, the ink strokes associated with the node, or any other data associated with the results of the analysis operation. This method also writes these identified changes to the current version of the analysis context object. This method is sensitive to the ordering of context node ordering, such as the order of word context nodes on a line context node.
The analysis results (that is, the value of the property AnalysisResultsEventArgs) are passed back with this method, as they contain a public results structure and a private results structure. The public structure is returned so the software application can preview the changes that will occur in the reconcile stage. The private structure is included to prevent the software application from changing any of the analysis results before the reconcile process.
The Ink Analyzer class may also include the methods entitled “Recognizers RecognizersPriority( )” and “SetHighestPriorityRecognizer(recognizer).” When ink needs to be recognized, the appropriate recognizer will be used based on language and capabilities. Accordingly, the Recognizers RecognizersPriority( ) method returns the recognition processes in the order in which they will be evaluated by the Ink Analyzer object. The order is determined per system depending upon recognition processes are available, but can be overridden for the software application by calling the SetHighestPriorityRecognizer(recognizer) on the Ink Analyzer object. The InkAnalyzer will enumerate through this ordered list until an appropriate recognizer can be found. The SetHighestPriorityRecognizer(recognizer) method raises the priority of a recognition process. By raising the priority of a particular recognition process, that recognition process will be used if it matches the needed language and capabilities of the current recognition operation. In essence, the SetHighestPriorityRecognizer(recognizer) pushes the designated recognition process to the top of the list returned by the RecognizersPriority method.
The Ink Analyzer class may also contain a method entitled “AnalysisRegion Abort( ),” which may use an analysis context object as a parameter. This method allows for a foreground or background analysis operation to be terminated early. This method returns an analysis region that describes the area that was being analyzed prior to the abort. Thus, if the software application intends to continue the analysis operation at a later time, this region can be merged into the DirtyRegion of the analysis context object for the current state of the document. Still further, the Ink Analyzer class may include an event entitled “AnalysisResultsEventHandler,” which fires to the InkAnalyzer object as frequently as practical. More particularly, this event may fire between analysis processes and at least once every five seconds. This event can be used to provide the application with an update as to the status of an ongoing asynchronous analysis process (or processes).
The Ink Analysis API may also include classes in addition to the Analysis Context class and the Ink Analyzer class. For example, the Ink Analysis API may include a Context Node class. This class may include various components relating to the context nodes that make up the analysis context object and the document independent analysis context object, such as the property entitled “ContextNodeType Type {get;}.” Each context node has a type, and there is a specific set of rules that each type should adhere to. This includes rules such as what types of child context nodes are allowed and whether or not strokes may be directly associated to the context node or only via its child context nodes.
The possible types of context nodes may be defined in a ContextNodeTypes enumeration and may include, for example, the following types: An InkAnnotation node, which represents ink data annotating non-text data; an InkDrawing node which represents ink data forming a drawing, an InkWord node, which represents ink data forming a word, a Line node, which contains one or more InkWord nodes and/or TextWord nodes for words forming a line of text; ListItem node, which may contain Paragraph, Image, or like nodes expected in a list; and a List node, which contains one or more ListItem nodes each describing an entry in the list. The node types may also include a NonInkDrawing node, representing a non-ink drawing image; an Object node, representing data not covered by other values of the ContextNodeType enumeration; a Paragraph node, which contains a one or more Line nodes corresponding to lines forming a paragraph; a Picture or Image node, representing a picture image; a Root node, which serves as the topmost node in an analysis context object; a Table node, which contains nodes representing items making up a table; a TextBox node, representing a text box; a TextWord node; and an UnclassifiedInk node, corresponding to ink data that has not yet been classified. The node types may also include a Group node, for groups of other nodes, an InkBullet node for bullet items, a Row node for documents elements presented in a row of a table, and a Cell node for document elements presented in a cell of a table.
The Context Node class may also include the property entitled “GUID Id {get;},” which is a globally unique identifier for the current context node. In order to allow access to any desired context node, each context node within a single analysis context object should have unique identifier. This class may also include a property entitled “AnalysisRegion Location {get;},” which identifies the location in the document space where the relevant context node is actually positioned. An AnalysisRegion is a two-dimensional structure grouping one or more possibly disjoint rectangle like structures together. This class may also include the property entitled “StrokeCollection Strokes {get;}”, which identifies the ink strokes associated with the relevant context node. With particular embodiments, only leaf context nodes (such as Word, Drawing and Bullet nodes) are allowed by the Ink Analysis API to have strokes. The software application may use this property to reference the strokes at the leaf node level by all ancestor context nodes (e.g., the root node would contain a strokes reference to all strokes in the relevant analysis context object.)
Further, this class may include the property entitled “ContextNode ParentNode {get;},” which identifies the parent context node containing the relevant context node. In particular embodiments, context nodes are always created to depend from a parent context node, with the Root context node being a static member of an analysis context object. This class may also include the property “ContextNode[ ] SubNodes {get;}” which identifies all context nodes that are direct children of the relevant context node. That is, this property will only identify those children context nodes down one level in the analysis context object. For example, the value of this property for a Paragraph context node will only identify the line context nodes contained by the Paragraph node, and not the word context nodes that are children of the line context node.
This class may also include the property entitled “RecognitionResult RecognitionResult {get;},” which provides the recognition result as calculated by the relevant recognition analysis process or processes, as the RecognitionResult can represent more than one line of text from in more than one language. The Recognition Result is available for every context node in the document independent analysis context object even though the RecognitionData property which is set by the recognition analysis process and is used to create the RecognitionResult object might only be set at one level of the context node tree to avoid duplication of data. If the node does not have a RecognitionData associated with it, it will either merge the recognition results of all of its subnodes or extract the recognition result from its parent. This class may also include the property entitled “Stream RecognitionData {get; set;},” which is the persistent form of the RecognitionResult value. Again, the recognition analysis process produces the Stream RecognitionData value that is set on the relevant context node. The RecognitionResult object is then constructed based on this value.
The Context Node class may further include a property entitled “ContextLink[ ] Links {get;},” which provides an array of ContextLink objects. A ContextLink object describes an alternative relationship between two context nodes. While context nodes typically have a parent-child relationship with other context nodes, a ContextLink allows for an alternative relationship between context nodes. For example, a ContextLink may allow for a connection between two context nodes, anchoring of one context node to another context node, containment of one context node by another context node, or a desired type of link defined by the software application. ContextLinks may be added to this array by calling the AddLink method. Similarly, ContextLinks may be removed from this array by calling the DeleteLink method.
Still further, this class may include the properties “IsContainer {get;}” and “IsInkLeaf {get;}.” The property IsContainer {get;} has the value of “true” if the relevant context node is not a leaf context node (that is, if the relevant context node contains children context nodes and is thus considered a container context node), and has the value of “false” otherwise. The IsInkLeaf {get;} property has the value of “true” if the current context node is not a container context node, and has a value of “false” otherwise. That is, if the current context node does not contain any children context nodes, it is considered a leaf context node. In certain embodiments, an InkLeaf context node is expected to contain references to stroke data, whereas container context nodes do not have this restriction. Container context nodes may or may not reference stroke data, as designated by the software application.
The Context Node class may also contain the property “Rect RotatedBoundingBox {get; set;}.” The value of this property is calculated by a layout and classification analysis process. If the ink data associated with the relevant context node is written at an angle, then the bounds for the context node will still be horizontally aligned. The value of the RotatedBoundingBox property, however, will be aligned to the angle at which the ink data associated with the relevant context node was written. Still further, this class may include the property “ReClassifiable {get;},” which informs the InkAnalyzer if it is allowed to modify the values of the relevant context node.
In addition to these properties, the Context Node class may also include various methods. For example, this class may include a method entitled “ContextNode CreateSubNode(ContextNodeType type).” This method allows the creation of a child context node of a particular type. In one embodiment, this method may only allow valid child types of the relevant context node to be created, thereby preventing malformed data structures from being created. For example, this method may only allow a Line context node to create InkWord and TextWord child context nodes. This class may also contain a method entitled “void DeleteSubNode(ContextNode node),” which deletes the referenced child context node from the relevant analysis context object. However, in certain embodiments, if the referenced context node still contains strokes or child context nodes, then this method should full. Also, if the reference context node is not a direct child of the relevant context node, then this method should full. If a software application implements its own analysis context object and in turn employs this method, it does not delete non-empty context nodes or context nodes that are not direct children of the relevant context node to prevent malformed data structures in the analysis context object.
Additionally, this class may include the method “ContextNode[ ] HitTestSubNodes(AnalysisRegion region),” which returns an array of context node that are located in the specified region. However, only the immediate children nodes of this element are returned, not all descendants. The region is defined by the AnalysisRegion object, which may be a collection of one or more rectangles. In particular embodiments, if any part of a context node's location intersects the specified region, then that context node will be returned in the array. This method is employed to, for example, create the document independent analysis context object and to reconcile the analysis results with the analysis context object corresponding to the current state of the document. Thus, this method is frequently called, and should be optimized for fast repeated access by the InkAnalyzer object.
The Context Node class may also contain a method entitled “MoveStroke(Stroke stroke, ContextNode destination).” This method moves the association of a stroke from one leaf context node to another. In certain embodiments, this method is only used between leaf context nodes. It may also include a method entitled “MoveSubNodeToPosition(int OldIndex, int NewIndex),” which reorders the relevant context node with respect to its sibling context nodes. For example, if the document has three words on a line, e.g., word 1, word 2 and word 3, then their order is implied by the array of subnodes returned from the parent context node. This method allows their order to be changed, so that, relative to the relevant parent context node, word 1 is specified to be the last word on the line by moving the context node for word 1 from position one to position three.
Still further, this class may include the method entitled “AddLink(ContextLink link),” which adds a new ContextLink object to the current context node. In particular embodiments, the ContextLink object should contain a reference to the relevant context node in order for the ContextLink to be successfully added to the array of ContextLinks associated to the relevant context node. It may also contain the method entitled “DeleteLink(ContextLink link).” This method deletes or removes the specified ContextLink object from the array of ContextLinks for the relevant context node. In one embodiment, this method call always completes successfully, even if the ContextLink does not exist in the array of ContextLinks associated with the relevant context node.
The Ink Analysis API may also include an Analysis Hint class. As with many of the previously described classes, the Analysis Hint class may include a constructor, entitled “AnalysisHint( ),” which initializes an Analysis Hint object to an empty state. This class may also include a number of properties, including a property entitled “AnalysisRegion Location {get;}.” This property specifies the location in the document (as an AnalysisRegion) to which the AnalysisHint is applicable. For example, if the document is a free form note with a title section at the top of the page, then the application could set an AnalysisHint for the title region to specify that a horizontal line of ink is expected in that region. This Analysis Hint will help to increase the accuracy of an analysis process.
This class may also include a property entitled “string Factoid {get; set;},” which specifies a particular “factoid” that is to be used for the location in the document to which the AnalysisHint is applicable. Factoids provide hints to a recognition process as to an expected use of ink data (e.g., as regular text, digits, postal codes, filenames, and web URLs). This class may also include the properties entitled “RecognizerGuide Guide {get; set;}” and “OverrideLanguageld {get; set;}.” The RecognizerGuide Guide {get; set;} property specifies the writing guide that is to be applied to the location in the document to which the AnalysisHint is applicable. Writing guides may, for example, help improve the accuracy of a recognizer analysis process by specifying to the user and informing the recognizer analysis process where the user will write lines or characters. The OverrideLanguageld {get; set;} property specifies a Language Hint for the document to which the AnalysisHint is applicable. Setting a Language Hint causes the InkAnalyzer object to use the designated language instead of the language specified on the context node.
This class may also include a property entitled “PrefixText {get; set;},” which specifies the text that is written or typed prior to a line of ink that is to be recognized. Still further, this class may include a property entitled “RecognitionModes RecognitionFlags {get; set;},” which specifies particular type of modes a recognition process should respect at the location in the document to which the AnalysisHint is applicable. Additionally, this class may include a property entitled “SuffixText {get; set;},” which specifies the text that is written or typed after to a line of ink that is to be recognized, and a property entitled “WordList WordList {get; set;},” which specifies a particular set of words that should be used by a recognition analysis process. Word lists may be used when the expected recognition results are known before the user has actually written input data, such as a list of medical terms that are expected to be written inside a medical form.
Still further, this class may include a property entitled “WordMode {get; set}.” If this value is “true,” then the analysis process will bias itself to return a single word for the entire analysis region. It may also include a property entitled “Coerce {get; set},” which, if “true,” will force the analysis process to confine its result to any factoid or wordlist value set in the relevant hint. This class may also include a property entitled “AllowPartialDictionaryTerms {get; set}.” If this property value is “true,” then the recognition analysis process will be allowed to return partial words from its recognition dictionary.
In particular embodiments, the Ink Analysis API may further include an Analysis Region class. This class may include, for example, multiple constructors for constructing an AnalysisRegion object. For example, it may contain a first constructor for constructing an AnalysisRegion object having any region, a second constructor for constructing an AnalysisRegion object based upon the parameters for a two-dimensional rectangle, and a third constructor for constructing an AnalysisRegion object based upon four spatial coordinates. The default constructor may, for example, create an empty region. This class may also include a number of properties. For example, this class may include a property entitled “Rectangle Bounds {get;},” which retrieves the bounding rectangle for the AnalysisRegion, a property entitled “IsEmpty {get;},” which indicates whether the relevant AnalysisRegion object has an empty interior, and a property entitled “IsInfinite {get;},” which indicates whether the relevant AnalysisRegion is set to infinite or not.
This class may also include a number of methods, such as a method entitled “AnalysisRegion Clone( ),” which clones the relevant AnalysisRegion object. This class may also include a method entitled “Equals(AnalysisRegion otherRegion),” which tests whether the specified AnalysisRegion object (referred to as the otherRegion) is identical to the relevant AnalysisRegion object. This method returns a value of “true” if the interior of the specified Analysis Region object is identical to the interior of the relevant Analysis Region object, and otherwise returns a value of “false.”
This class may further include a method “Intersect(AnalysisRegion regionToIntersect),” which crops down the relevant AnalysisRegion object to the specified analysis region. Thus, the resulting AnalysisRegion object will only include areas that overlapped or intersected the specified analysis region. This class may also include the method entitled “Intersect(Rectangle rectangle),” which crops down the relevant AnalysisRegion object to the specified rectangle. Again, the resulting AnalysisRegion object will only include areas that overlapped or intersected the specified rectangle. It may also include the method entitled “MakeEmpty( ),” which initializes the relevant AnalysisRegion object to an empty interior, and the method entitled “MakeInfinite( ),” which sets the area occupied by the relevant AnalysisRegion to be infinite. It may further include various methods for uniting or separating differently defined areas, such as method entitled “Union(AnalysisRegion regionToUnion),” which specifies an AnalysisRegion object to union or add to the relevant AnalysisRegion object, and the method entitled “Union(Rectangle rectangle),” which unions a specified rectangle to the relevant Analysis Region object. With this method, the rectangle may be specified in terms of the coordinate space for the relevant AnalysisRegion object. Of course, this class may include numerous other methods for combining areas or extracting one area from another based upon any desired definition for the areas.
The Ink Analysis API may also have a Recognition Result class. As with many of the previously discussed classes, the Recognition Result class may include one or more constructors. For example, this class may include a constructor entitled “RecognitionResult(Stream lattice)”, which constructs a RecognitionResults object from a given recognition lattice. In particular embodiments, a recognition lattice is a serialized form of the results from a recognition process. This method may, for example, specify a recognition lattice as a byte array to be used for the construction of the relevant RecognitionResult object. It may also include a constructor entitled “RecognitionResult(ContextNode node),” which constructs a RecognitionResults object from a given context node. It may also include a constructor entitled “RecognitionResult(string Text, int StrokeCount),” which constructs a RecognitionResults object from a specified text value, which in turn is associated to a specified number of strokes and might be used for correction if the recognition process did not come up with an alternate recognition value corresponding to the actual handwritten ink data. Still further, this class may include a constructor entitled “RecognitionResult(RecognitionResult leftRecognitionResult, RecognitionResult rightRecognitionResult),” which constructs a RecognitionResults object by merging two existing Recognition Results objects together.
The Recognition Result class may also include one or more properties, such as a property entitled “StrokeCollection StrokeCollection {get;},” which provides an array of stroke indexes representing a collection of strokes that are contained in a single ink object, and a property entitled “RecognitionAlternate TopAlternate {get;},” which provides the best alternate of the recognition result. This class may also include the property entitled “RecognitionConfidence RecognitionConfidence {get;},” which provides the level of confidence (e.g., strong, intermediate, or poor) of the top alternate selection for the current results from a recognition analysis process, and a property entitled “string TopString {get;},” which returns the best result string of the analysis results from a recognition analysis process.
The Recognition Results class may also include a number of methods, such as a method entitled “public RecognitionAlternateCollection GetAlternateCollectionFromSelection (selectionStart, selectionLength, maximumAlternates),” which specifies a collection of alternates from a selection within the best result string of analysis results from a recognition analysis process, where each alternate corresponds to only one segment of ink. The input parameters for this method may include, for example, a value which specifies the start of the text selection from which the collection of alternates is returned, a value that specifies the length of the text selection from which the collection of alternates is returned, and a value that specifies the maximum number of alternates to return. This method may then return the RecognitionAlternateCollection collection of alternates from a selection within the best result string of the recognition result.
The Recognition Results class may further include a method entitled “RecognitionResult Merge(RecognitionResult left, string separator, RecognitionResult right).” This method may be used to create a new RecognitionResult object from a single string, resulting in a flat lattice, append a single string to the beginning or end of an existing RecognitionResult object, or concatenate a single string in between two existing RecognitionResult objects. This class may also include a method entitled “ModifyTopAlternate(RecognitionAlternate alternate),” which specifies the recognition result to be modified with a known alternate. With some embodiments, by default the best result string of the results of a recognition analysis process corresponds to the top alternate. However, this method can be used to specify that alternates other than the top alternate are used in the results of the recognition analysis process. If the new top alternate results in a different segmentation than the previous one, the ModifyTopAlternate method will automatically update the context nodes to reflect the changes. To retrieve the alternates that can be used to modify the recognition result, this method calls the GetAlternatesFromSelection method. This class may also have a method entitled “Stream Save( ),” which persistently maintains the relevant RecognitionResults object in the form of a recognition lattice. A recognition lattice is a serialized format used to express the results from a recognition process.
The Ink Analysis API may also have an Analysis Options enumerated type. This type may contain one or more fields that specify how ink data will be analyzed by an analysis process, such a field entitled “const AnalysisOptions Default,” which enables all available options for the analysis process. This field may, for example, enable text recognition, the use of tables, the use of lists, the use of annotations, the use of connectors and containers, and the use of intermediate results. This type may also include a field entitled “const AnalysisOptions EnableAnnotations,” which enables and disables the detection of annotations, a field entitled “const AnalysisOptions EnableConnectorsAndContainers,” which enables and disables the detection of connectors and containers, and a field entitled “const AnalysisOptions EnablelntermediateResults,” enables and disables the return of analysis results to the software application between the use of different, sequential analysis processes (e.g., between a parsing process and a subsequent recognition process). This type may also have a field entitled “const AnalysisOptions EnableLists,” which enables and disables the detection of lists, and a field entitled “const AnalysisOptions EnableTables,” which enables and disables the detection of tables. This enumerated type may further include a field entitled “const AnalysisOptions EnableTextRecognition,” which enables and disables a text recognition analysis process. However, if additional analysis processes are available (or different versions of the same analysis process), then this type may include additional AnalysisOptions accordingly.
Still further, the Ink Analysis API may include an AnalysisResultsEventArgs class. This class may have a constructor entitled “public AnalysisResultsEventArgs( ),” which creates a data structure that contains the analysis results and is returned to the software application when the AnalysisResults event is raised. This class may also include a property entitled “InkAnalyzer InkAnalyzer {get;},” which identifies the InkAnalyzer object that performed the analysis process.
The API may also have a Line class, which may be useful with some types of operating systems which recognize the use of a “Line” object representing a geometric line. This class may include a constructor, such as a constructor entitled “public Line(Point beginPoint, Point endPoint),” which creates a Line object. This class may also include various properties, such as a property entitled “public Point BeginPoint {get; set;},” which represents the beginning point of the line object and a property entitled “public Point EndPoint {get; set;},” which represents the ending point of the line object.
In addition to these classes, the Ink Analysis API may also contain a Recognition Alternate class. This class may include elements representing the possible word matches for segments of ink that are compared to a recognizer's dictionary. For example, this class may include a property entitled “Line Ascender {get;},” which provides the ascender line of a RecognitionAlternate object that exists on a single line (with a line being represented as two points), a property entitled “public Line Baseline {get;},” which provides the Baseline of a RecognitionAlternate object that exists on a single line, and a property entitled “Line Descender {get;},” which provides the descender line of a RecognitionAlternate object that exists on a single line. This class may also include a property entitled “RecognitionResult Extract {get;},” which provides a RecognitionResults object for the current RecognitionAlternate object. This property can be used, for example, to extract the RecognitionResult object for a word from the RecognitionResult object for a line containing that word.
It may also include the property entitled “Line Midline {get;},” which provides the midline for a RecognitionAlternate object that exists on a single line, a property entitled “StrokeCollection Strokes {get;},” which provides the collection of strokes that are contained in an ink object (that is, it provides a StrokeCollection representing the strokes that are associated to the RecognitionResult), and a property entitled “StrokeCollection[ ] StrokesArray {get;},” which provides a collection of strokes that are contained in one or more ink objects, representing the strokes that are associated with the RecognitionResult. This class also may include a property entitled “RecognitionConfidence RecognitionConfidence {get;},” which provides the level of confidence (e.g., strong, intermediate, or poor) that a recognition analysis process has determined in the recognition of a RecognitionAlternate object or of a gesture. For non-line nodes, the lowest RecognitionConfidence of the children of the relevant context nodes will be returned. It may also contain the property entitled “string RecognizedString {get;}” which specifies the result string of the alternate. Thus, for any context node above a word context node, the results string is concatenated together by this method. For example, a line node will contain a results string that in turn contains the results of all its children or word nodes. A paragraph node will then contain a results string that contains the results of all its children or line nodes.
The Recognition Alternate class may also contain one or more methods including, for example, a method entitled “StrokeCollection[ ] GetStrokesArrayFromTextRange(int selectionstart, int selectionlength),” which specifies a StrokeCollection from each ink object that corresponds to the known text range. This class may also contain a method entitled “StrokeCollection[ ] GetStrokesFromStrokesArrayRanges(StrokeCollection[ ] strokesArray),” which specifies the smallest collection of strokes that contains a known input collection of strokes and for which the recognizer can provide alternates. More particularly, the strokes are returned by an array of ink objects each containing an array of stroke indexes for the collection. The collection of ink strokes returned by this method may match the input collection, or it may be larger if the input collection matches only part of the smallest recognition result that includes all of the input strokes. This class may further include a method entitled “StrokeCollection GetStrokesFromStrokesRanges(StrokeCollection strokes),” which specifies the smallest collection of strokes that contains a known input collection of strokes and for which the recognizer can provide alternates, and a method entitled “StrokeCollection GetStrokesFromTextRange(int selectionstart, int selectionlength),” which specifies the StrokeCollection that corresponds to the known text range.
This class may further include a method entitled “void GetTextRangeFromStrokes(ref int selectionstart, ref int selectionend, StrokeCollection strokes),” which specifies the smallest range of recognized text for which the recognizer can return an alternate that contains a known set of strokes, and a method entitled “void GetTextRangeFromStrokesArray(ref int selectionstart, ref int selectionend, StrokeCollection[ ] strokesarray),” which specifies the smallest range of recognized text for which the recognizer can return an alternate that contains a known set of strokes. It also may have a method entitled “RecognitionAlternateCollection SplitWithConstantPropertyValue(GUID propertyType),” which returns a collection of alternates, which are a division of the alternate on which this method is called. Each alternate in the collection contains adjacent recognition segments which have the same property value for the property passed into the method. For example, this method can be used to obtain alternates that divide an original alternate by level of confidence boundaries (strong, intermediate, or poor) in the recognition result, line boundaries, or segment boundaries. It may further include a method entitled “byte[ ] GetPropertyValue(GUID propertyType),” which specifies the value of a known property of the alternate, such as the recognizer's confidence in the alternate. Not all recognition analysis processes will provide a value for all property types, however. Thus, this method provides the data for the types supported by the relevant recognition analysis process.
The Ink Analysis API may also include a Recognition Alternate Collection class. Like many of the classes discussed above, this class may include a constructor, entitled “RecognitionAlternateCollection( ),” for creating a RecognitionAlternateCollection object. This class may also include a number of properties, such as a property entitled “Count {get;},” which provides the number of objects or collections contained in a collection of alternate recognition values, a property entitled “IsSynchronized {get;},” which provides a value indicating whether access to the collection of alternate recognition values is synchronized with the software application (i.e., “thread safe”), and a property entitled “SyncRoot {get;},” which provides the object that can be used to synchronize access to the collection of alternate recognition values.
This class may also contain one or more methods, such as a method entitled “virtual void CopyTo(Array array, int index),” which copies all of the elements of the current collection of alternate recognition values to the specified one-dimensional array, starting at the specified destination array index, and a method entitled “IEnumerator IEnumerable.GetEnumeratoro,” which is a standard implementation of IEnumerable that enables callers to use the for each construct to enumerate through each RecognitionAlternate in the collection of alternate recognition values. This class may also include a method entitled “RecognitionAlternateCollectionEnumerator GetEnumerator( ),” which returns a RecognitionAlternateCollectionEnumerator that contains all of the objects within the collection of recognition alternate values. This method may be used, for example, to retrieve each object in the collection of recognition alternate values.
The Ink Analysis API may additionally include a Recognition Confidence enumeration and a Recognition Mode enumeration, each of which may contain one or more fields relating to a recognition analysis process. For example, the Recognition Confidence class may contain multiple fields, such as a field entitled “Intermediate,” indicating that the recognition analysis process is confident that the correct result is in the list of provided alternate recognition values, a field entitled “Poor,” which indicates that the recognition analysis is not confident that the result is in the list of provided alternate recognition values, and a field entitled “Strong,” which indicates that the recognition analysis process is confident that the best alternate in the alternate recognition values is correct.
Similarly, the Recognition Mode class may include fields that that specify how a recognition analysis process interprets electronic ink data and thus determines a recognition result string. For example, this class may include a field entitled “Coerce,” which specifies that the recognition analysis process coerce a recognition result based on a factoid that was specified for the context, and a field entitled “Line,” which specifies that the recognition analysis process treat the electronic ink data as a single line. This class also may include a field entitled “None,” which specifies that the recognition analysis process apply no recognition modes, and a field entitled “Segment,” which specifies that the recognition analysis process treat the electronic ink data as forming a single word or character. Still further this class may include a field entitled “TopInkBreaksOnly,” which disables multiple segmentation.
Still further, the Ink Analysis API may include a Context Link class, which defines how two context nodes may be linked together. The ContextLink object by itself represents which two context nodes are linked, the direction of the link, and the type of link. This class may include a property entitled “ContextNode SourceNode{get;},” which specifies the source context node that is being linked from another context node, a property entitled “ContextLinkType LinkType{get;},” which specifies the type of link relationship that exists between the source and destination context nodes, and a property entitled “CustomLinkType{get;},” which specifies that a custom link is being used. This situation would occur when an application decides to use the linking system of the Ink Analyzer API to represent application specific links beyond what the API can recognize.
This class may also include a property entitled “ContextNode DestinationNode {get;},” which specifies the destination context node that is being linked from another context node. There may be two constructors available to this class, which create a relationship between existing source and destination context nodes.
This class may also include an enumeration entitled “ContextLinkType enum,” which defines the type of relationship shared by two context nodes. These various link types may include, for example, an “AnchorsTo” type, which describes that one node is anchored to the other node. Both nodes can use the SourceNode or DestinationNode property based on the parsing situation. The link types may also include the type “Contains,” which describes that the one node contains the other node. With this relationship, the container node could be referenced as the SourceNode, while the containee node could be referenced as the DestinationNode. The link types may further include a “PointsTo” type, which describes that one node is pointing to another node. For this relationship, the node doing the pointing could be referenced as the SourceNode, while the node being pointed to could be referenced as the DestinationNode. Still further, the link types may have a “PointsFrom” type, which describes that one node is pointing from the other node. In this relationship, the node pointing away from the other node could be referenced as the SourceNode, while the node being pointed from could be referenced as the DestinationNode.
The link types may additionally include a “SpansHorizontally” type, which describes that one node runs the length horizontally of another node, and a “SpansVertically” type, which describes that one node runs the length vertically of another node. For these types, the node covering (strike out, underline, margin bar) the other node, usually written last, could be referenced as the SourceNode, while the node being spanned could be referenced as the DestinationNode. The link types may also include a “Custom” type, which describes that a custom link type has been used. When this value is used, the “CustomLinkType” property on the ContextLink object could provide more details as to the purpose of this link.
Application Model
A Windows Client integrates characteristics of the Web with characteristics of traditional desktop applications. The Application Model provides a framework for secure applications and simplifies development, deployment and maintenance of client applications. This framework provides a simple and consistent user experience. For example, local applications can leverage familiar browser-like features regardless of whether the application is hosted in the browser or is a standalone application, while retaining the benefits of executing on the local client. This framework allows users to leverage their familiarity with the Web, thereby increasing the user's comfort level and reducing the time required to learn to use a new application. The Application Model is part of the System.Windows namespace.
Applications utilizing the Application Model operate in a manner similar to Web pages. When a user browses to an application, the application is automatically installed without requiring user confirmation of the installation, rebooting the client system, or risking the malfunctioning of other applications. In one embodiment, applications download progressively, thereby providing an initial level of interactivity before the application is completely downloaded. Application updates are handled automatically and in a manner that is transparent to the user. Thus, the user always has access to the latest version of the application without having to explicitly perform an application upgrade.
Applications that use the Application Model run locally on the client system regardless of whether the client system is on-line (i.e., actively coupled to the Web) or off-line (i.e., not actively coupled to the Web). This allows an application to provide better performance than a server-based application that needs an active Web connection and continually exchanges data with the server across the Web. After an application is installed on a client system, the application can be accessed from a “Start” menu (like a traditional desktop application) or by navigating to the application (like a Web application). The Application Model contains three primary parts: application lifecycle management, an application framework and a navigation framework.
Two different types of applications are supported by the Application Model: an “on-line application” and a “managed application”. Applications utilizing the Application Model can execute in the browser or in a standalone top-level window. An “on-line application” is an application that executes from a server and is hosted in a browser. The application can be cached for offline access or the application may require certain on-line resources to execute properly. A “managed application” is available off-line and is installed on the client. The operating system services the managed application. An entry for the managed application can be added to the “Start” menu on the client. Applications can be downloaded progressively to allow the user to begin interacting with the application as it is being downloaded rather than delaying interaction until an installation process has finished.
Applications have an associated application manifest that describes the application's dependencies, such as additional libraries and resources needed to execute the application. An installer uses the application manifest to control downloading and installation of the application. A “trust manager” is invoked as part of the installation process. The trust manager uses the application manifest to determine what permissions are needed for the application to execute. The application manifest also specifies shell information, such as file associations and whether to add an entry to the Start menu as well as the icon and text for the entry.
Applications utilizing the Application Model include markup, code, resources and a manifest. An application is defined and scoped by its application object, which is a global object that persists in memory for the lifetime of each application session. The application object has knowledge of all the resources that belong to the application and provides a boundary between itself and other applications or external resources. The application framework uses the application object to identify, reference and communicate with the application. The application object is also used within the application to manage windows and resources, specify startup and shutdown behavior, handle configuration settings, specify visual styles for the application, share code, state and resources across navigations, and handle application-wide events.
A navigation framework supports navigation-based applications that leverage users' familiarity with navigation and journaling activities on the Web to provide a more familiar, consistent user experience on the client system, regardless of whether the application is hosted in the system browser or in a standalone top-level window. Journaling is the process used by the navigation framework to track navigation history. The journal allows users to retrace their steps backward and forward in a linear navigation sequence. Whether a navigation experience is hosted in the browser or in a standalone navigation window, each navigation is persisted in the journal and can be revisited in a linear sequence by using “forward” and “back” buttons or by invoking “go forward” and “go back” methods. Each navigation window has an associated journal.
A NavigationApplication class simplifies the task of creating navigation-based applications by providing properties and events related to navigation. The NavigationApplication class includes a startup property that specifies the page or element to which the system navigates when the application is first launched. This class also has a properties collection that allows an application developer to share global state information across pages without having to subclass the application, and supports data binding to these properties.
Exemplary Computing System and Environment
The exemplary computing environment 400 is only one example of a computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the computer and network architectures. Neither should the computing environment 400 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary computing environment 400.
The framework 132 may be implemented with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use include, but are not limited to, personal computers, server computers, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and so on. Compact or subset versions of the framework may also be implemented in clients of limited resources, such as cellular phones, personal digital assistants, handheld computers, or other communication/computing devices.
The framework 132 may be described in the general context of computer-executable instructions, such as program modules, being executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The framework 132 may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The computing environment 400 includes a general-purpose computing device in the form of a computer 402. The components of computer 402 can include, by are not limited to, one or more processors or processing units 404, a system memory 406, and a system bus 408 that couples various system components including the processor 404 to the system memory 406.
The system bus 408 represents one or more of several possible types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures can include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnects (PCI) bus also known as a Mezzanine bus.
Computer 402 typically includes a variety of computer readable media. Such media can be any available media that is accessible by computer 402 and includes both volatile and non-volatile media, removable and non-removable media.
The system memory 406 includes computer readable media in the form of volatile memory, such as random access memory (RAM) 410, and/or non-volatile memory, such as read only memory (ROM) 412. A basic input/output system (BIOS) 414, containing the basic routines that help to transfer information between elements within computer 402, such as during start-up, is stored in ROM 412. RAM 410 typically contains data and/or program modules that are immediately accessible to and/or presently operated on by the processing unit 404.
Computer 402 may also include other removable/non-removable, volatile/non-volatile computer storage media. By way of example,
The disk drives and their associated computer-readable media provide non-volatile storage of computer readable instructions, data structures, program modules, and other data for computer 402. Although the example illustrates a hard disk 416, a removable magnetic disk 420, and a removable optical disk 424, it is to be appreciated that other types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes or other magnetic storage devices, flash memory cards, CD-ROM, digital versatile disks (DVD) or other optical storage, random access memories (RAM), read only memories (ROM), electrically erasable programmable read-only memory (EEPROM), and the like, can also be utilized to implement the exemplary computing system and environment.
Any number of program modules can be stored on the hard disk 416, magnetic disk 420, optical disk 424, ROM 412, and/or RAM 410, including by way of example, an operating system 426, one or more application programs 428, other program modules 430, and program data 432. Each of the operating system 426, one or more application programs 428, other program modules 430, and program data 432 (or some combination thereof) may include elements of the programming framework 132.
A user can enter commands and information into computer 402 via input devices such as a keyboard 434 and a pointing device 436 (e.g., a “mouse”). Other input devices 438 (not shown specifically) may include a microphone, joystick, game pad, satellite dish, serial port, scanner, and/or the like. These and other input devices are connected to the processing unit 404 via input/output interfaces 440 that are coupled to the system bus 408, but may be connected by other interface and bus structures, such as a parallel port, game port, or a universal serial bus (USB).
A monitor 442 or other type of display device can also be connected to the system bus 408 via an interface, such as a video adapter 444. In addition to the monitor 442, other output peripheral devices can include components such as speakers (not shown) and a printer 446 which can be connected to computer 402 via the input/output interfaces 440.
Computer 402 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computing device 448. By way of example, the remote computing device 448 can be a personal computer, portable computer, a server, a router, a network computer, a peer device or other common network node, and so on. The remote computing device 448 is illustrated as a portable computer that can include many or all of the elements and features described herein relative to computer 402.
Logical connections between computer 402 and the remote computer 448 are depicted as a local area network (LAN) 450 and a general wide area network (WAN) 452. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.
When implemented in a LAN networking environment, the computer 402 is connected to a local network 450 via a network interface or adapter 454. When implemented in a WAN networking environment, the computer 402 typically includes a modem 456 or other means for establishing communications over the wide network 452. The modem 456, which can be internal or external to computer 402, can be connected to the system bus 408 via the input/output interfaces 440 or other appropriate mechanisms. It is to be appreciated that the illustrated network connections are exemplary and that other means of establishing communication link(s) between the computers 402 and 448 can be employed.
In a networked environment, such as that illustrated with computing environment 400, program modules depicted relative to the computer 402, or portions thereof, may be stored in a remote memory storage device. By way of example, remote application programs 458 reside on a memory device of remote computer 448. For purposes of illustration, application programs and other executable program components such as the operating system are illustrated herein as discrete blocks, although it is recognized that such programs and components reside at various times in different storage components of the computing device 402, and are executed by the data processor(s) of the computer.
An implementation of the framework 132, and particularly, the API 142 or calls made to the API 142, may be stored on or transmitted across some form of computer readable media. Computer readable media can be any available media that can be accessed by a computer. By way of example, and not limitation, computer readable media may comprise “computer storage media” and “communications media.” “Computer storage media” include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.
“Communication media” typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier wave or other transport mechanism. Communication media also includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.
Alternatively, portions of the framework may be implemented in hardware or a combination of hardware, software, and/or firmware. For example, one or more application specific integrated circuits (ASICs) or programmable logic devices (PLDs) could be designed or programmed to implement one or more portions of the framework.
Notionally, a programming interface may be viewed generically, as shown in
Aspects of such a programming interface may include the method whereby the first code segment transmits information (where “information” is used in its broadest sense and includes data, commands, requests, etc.) to the second code segment; the method whereby the second code segment receives the information; and the structure, sequence, syntax, organization, schema, timing and content of the information. In this regard, the underlying transport medium itself may be unimportant to the operation of the interface, whether the medium be wired or wireless, or a combination of both, as long as the information is transported in the manner defined by the interface. In certain situations, information may not be passed in one or both directions in the conventional sense, as the information transfer may be either via another mechanism (e.g. information placed in a buffer, file, etc. separate from information flow between the code segments) or non-existent, as when one code segment simply accesses functionality performed by a second code segment. Any or all of these aspects may be important in a given situation, e.g., depending on whether the code segments are part of a system in a loosely coupled or tightly coupled configuration, and so this list should be considered illustrative and non-limiting.
This notion of a programming interface is known to those skilled in the art and is clear from the foregoing detailed description of the invention. There are, however, other ways to implement a programming interface, and, unless expressly excluded, these too are intended to be encompassed by the claims set forth at the end of this specification. Such other ways may appear to be more sophisticated or complex than the simplistic view of
A Factoring
A communication from one code segment to another may be accomplished indirectly by breaking the communication into multiple discrete communications. This is depicted schematically in
B. Redefinition
In some cases, it may be possible to ignore, add or redefine certain aspects (e.g., parameters) of a programming interface while still accomplishing the intended result. This is illustrated in
C. Inline Coding
It may also be feasible to merge some or all of the functionality of two separate code modules such that the “interface” between them changes form. For example, the functionality of
D. Divorce
A communication from one code segment to another may be accomplished indirectly by breaking the communication into multiple discrete communications. This is depicted schematically in
E. Rewriting
Yet another possible variant is to dynamically rewrite the code to replace the interface functionality with something else but which achieves the same overall result. For example, there may be a system in which a code segment presented in an intermediate language (e.g. Microsoft IL, Java ByteCode, etc.) is provided to a Just-in-Time (JIT) compiler or interpreter in an execution environment (such as that provided by the Net framework, the Java runtime environment, or other similar runtime type environments). The JIT compiler may be written so as to dynamically convert the communications from the 1st Code Segment to the 2nd Code Segment, i.e., to conform them to a different interface as may be required by the 2nd Code Segment (either the original or a different 2nd Code Segment). This is depicted in
It is also noted that the above-described scenarios for achieving the same or similar result as an interface via alternative embodiments may also be combined in various ways, serially and/or in parallel, or with other intervening code. Thus, the alternative embodiments presented above are not mutually exclusive and may be mixed, matched and combined to produce the same or equivalent scenarios to the generic scenarios presented in
Although the invention has been described in language specific to structural features and/or methodological acts, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed invention.
This application is a continuation of U.S. application Ser. No. 10/694,080, filed Oct. 24, 2003, now pending. This application is a continuation-in-part of U.S. application Ser. No. 11/408,651, filed Apr. 21, 2006, now pending, which is a continuation of U.S. application Ser. No. 09/827,957, filed on Apr. 5, 2001, now abandoned, which is a continuation-in-part of U.S. application Ser. No. 09/670,791, filed on Sep. 27, 2000, now U.S. Pat. No. 6,873,337, which claims the benefit of U.S. Provisional application No. 60/195,593, filed on Apr. 6, 2000. This application is a continuation-in-part of U.S. application Ser. No. 10/839,908, filed May 6, 2004, now pending, which is a continuation of U.S. application Ser. No. 09/825,488, filed on Apr. 3, 2001, now U.S. Pat. No. 6,753,885, which claims the benefit of U.S. Provisional application No. 60/195,594, filed on Apr. 6, 2000.
| Number | Date | Country | |
|---|---|---|---|
| 60195593 | Apr 2000 | US | |
| 60195594 | Apr 2000 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10694080 | Oct 2003 | US |
| Child | 12207398 | US | |
| Parent | 09827957 | Apr 2001 | US |
| Child | 11408651 | US | |
| Parent | 09825488 | Apr 2001 | US |
| Child | 10839908 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11408651 | Apr 2006 | US |
| Child | 10694080 | US | |
| Parent | 09670791 | Sep 2000 | US |
| Child | 09827957 | US | |
| Parent | 10839908 | May 2004 | US |
| Child | 09670791 | US |
