Application-responsive markup language parser

Abstract

A markup language parser processes markup language data and provides processed data to an application. A reader is configured to tokenize the markup language data and store resulting tokens in a buffer associated with the reader. A scanner is configured to create string objects based on the tokens and to provide the created string objects to an application. The scanner is further configured to determine a need by the application for the string objects. For each at least some of the string objects created by the scanner, the scanner is configured to selectively create those string objects based on a determination of the need by the application for the string objects.

Description

TECHNICAL FIELD

The present invention is in the field of markup language parsers that provide processed markup language data to applications and, in particular, relates to a markup language parser that expends processing resources based at least in part on application request.

BACKGROUND

A markup language parser is, in a general sense at least, a program to determine the syntactic structure of a string of symbols in markup language. A markup language (or, perhaps more precisely, markup specification language) describes information (text or data), usually for storage, transmission, or processing by a program. The markup language typically does not specify what should be done with the data.

FIG. 1 illustrates a conventional markup language parser 100 from one simplistic point of view. In broad terms, the parser 100 processes markup language source from a file 106 and provides processed data for use by one or more applications 101. From the simplistic point of view illustrated in FIG. 1, the parser 100 can be considered to include two primary components—a reader 102 and a scanner 104.

The reader 102 reads the contents of the file 106 (including markup language statements which, in the example, are XML) to be processed and stores the contents into a buffer 108, typically of fixed predetermined size. If the size of the file 106 is more than the size of the buffer 108, then the buffer 108 is refreshed with the unread markup language data once the scanner 104 has processed the data that is currently in the buffer 108.

The reader 102 is configured to check for valid markup language characters, tokenize the markup language content (e.g., for XML in one example, to tokenize the markup language content to XMLNames, values and content), and provide the tokens to the scanner 104.

The scanner 104 is configured to process the tokens generated by the reader 102 and to provide string objects and/or values (generically, data 103) to the application 101 based on the tokens. For example, the scanner 104 may operate as a state machine. The string objects and/or values provided to the application 101 by the scanner 104 may be, for example, an XMLName (element name, attribute name), attribute value, element content, etc.

We now briefly discuss circumstances surrounding the conventional passing of data between the reader 102 and the scanner 104. The scanner 104 passes pointer objects to the reader 102. The pointer objects passed by the scanner 104 to the reader 102 are essentially just shells, to be populated by the reader 102. After processing by the reader 102, a pointer object points to a token in the buffer 108, and control is returned to the scanner 104. More particularly, the pointer object indicates an offset into the buffer 108 as well as the length of the token. Then, depending on the type of token being processed, the scanner 104 processes a populated pointer object to either create string objects or to copy data into a buffer 110 in the scanner 104.

It is desired to streamline the operation of the parser.

SUMMARY

A markup language parser processes markup language data and provides processed data to an application. A reader is configured to tokenize the markup language data and store resulting tokens in a buffer associated with the reader. A scanner is configured to create string objects based on the tokens and to provide the created string objects to an application.

The scanner is further configured to determine a need by the application for the string objects. For each at least some of the string objects created by the scanner, the scanner is configured to selectively create those string objects based on a determination of the need by the application for the string objects.

BRIEF DESCRIPTION OF FIGURES

FIG. 1 illustrates a conventional configuration of a markup language parser.

FIG. 2 illustrates a configuration of a markup language parser in which particular operations of the parser are carried out based on an indication from the application of need for the results of the operations.

FIG. 3 illustrates an alternate configuration of the FIG. 2 markup language parser.

DETAILED DESCRIPTION

It has been realized that, conventionally, the scanner of a markup language parser executes processing to create string objects or copy data into its buffer (depending upon the type of token being processed) even in situations where the application does not ask for or otherwise require a string object and/or data. In accordance with aspects of the invention, at least a portion of processing by the parser, which formerly was carried out “open loop,” is now carried out by the parser based on a request or other indication of need by the application, for the result of processing by the parser.

For example, in accordance with one aspect, the string objects are created by the scanner, to provide to the application(s), when requested by the application. In accordance with another aspect, the values are copied from the buffer in the reader, to the scanner for processing and providing to the application(s), based on an internal state of the scanner or the type of token being read.

We now discuss these aspects with reference to embodiments illustrated in FIG. 2 and FIG. 3.

Referring first to FIG. 2, like the conventional parser 100 illustrated in FIG. 1, the FIG. 2 parser 200 includes a reader 202 and a scanner 204. Like the buffer 108 in the FIG. 1 conventional parser 100, the reader 202 includes a buffer 208 into which contents of an XML file 206 are stored. Also like the FIG. 1 example, the reader 202 is configured to check for valid markup language characters in the markup language source 206 and to tokenize the markup language content.

Also like the scanner 104, the scanner 204 maintains a register 212 containing a pointer object that references back to the buffer 208 in the reader 202. For example, the reader 202 may have informed the scanner 204 of offset and length information pertaining to data relevant to a particular token, and the scanner 204 maintains the register 212 containing a pointer object that references back to this data.

Unlike the scanner 104, the scanner 204 typically does not (open loop) copy the pointed-to data from the reader buffer 208 into a buffer associated with the scanner 204 upon completion of processing of the pointer object (i.e., population) by the reader 202. Rather, when string objects are created by the scanner, the data is accessed directly from the reader buffer 208.

To avoid data being overwritten in the reader buffer 208 before it can be accessed by the scanner 204, the scanner 204 is notified when the referenced data in the buffer 208 may be no longer available (e.g., will be overwritten in the buffer 208). At this point, if it is determined by the scanner 204 that the referenced data will not be needed, then the referenced data need never be copied out of the buffer 208.

In one example, a “listener” mechanism is used to notify the scanner 204 when the referenced data in the buffer 208 may be no longer available. That is, the arrow 214 in FIG. 2 emanating from listener logic 213 indicates specification by the scanner 204 of a listener object to the reader 202. When the referenced data in the buffer 208 may be no longer available (e.g., when the reader 202 is about to load new data from the markup language document 206 into the buffer 208), the reader 202 gives a callback (arrow 216) to the listener logic 213 of the scanner 204.

At this point, the scanner 204 determines whether the referenced data may be needed, based on the internal state of the scanner 204 or the type of token that is being/was read. For example, if the state of the scanner 204 is such that the referenced data has already been processed or is past the state in which the referenced data would have been processed, then the referenced data is no longer needed. If the referenced data in the buffer 208 may be needed, then the referenced data is copied from the buffer 208 of the reader 202 into a buffer 210 of the scanner 204.

Furthermore, whereas in the FIG. 1 example, objects 103 are created without regard for need by the application(s) 101, in accordance with another aspect, the objects 203 (FIG. 2) are created when the application(s) 201 asks for data from the parser. Thus, for example, when an application 201 asks for data from the parser, the objects 203 are created using data from the buffer 208 (of the reader 202) or from the buffer 210 (of the scanner 204) as appropriate.

FIG. 3 illustrates an example where the scanner 304 maintains a pool 312 of pointer objects. This is particularly useful where, for example, a token may have multiple attributes in the buffer 308 of the reader 302. Each pointer object references, for example, a different one of the attributes in the buffer 308. The processing with respect to the pointer object references, and the data pointed to by the pointer objects, may be similar to that discussed above with reference to FIG. 2. As with the FIG. 2 example, at least some of the string objects 303 are created only when a need is indicated by the application. It is noted that, in the FIG. 3 example, a buffer like the buffer 210 of the scanner 204, in the FIG. 2 example, is not shown.

Put another way, in general, there may be two types of buffers, and there are two pools, one for each type of buffer. Both types of buffers are used to store attribute values. For example, in the following string:

• • attrNameOne=“Value1” the buffers may be used to store “Value1”.

With regard to the two types of buffers, a first type of buffer is used when the attribute is simple and all that is needed to get the value is the starting and ending offsets to the buffer. This is the type of buffer 212 illustrated in FIG. 2. That is, each pointer object in the buffer 212 is just a pointer to a set of characters in the reader buffer 208.

As discussed above, the first type of pointers are converted to string values and read into the scanner 204 in two scenarios. The first scenario is that an attribute is read by the application and the corresponding attribute value is requested by the application. The second scenario is that a callback 216 is received, indicating the buffer 208 is about to be written over. In the second scenario, the attribute value is copied into the buffer 210.

A second type of buffer (illustrated in the simplified view in FIG. 3) is used when attribute values are not simple, and more processing is applied before giving an attribute value to the application. The processing of the attribute value is applied when the scanner 304 is processing the buffer 308 in the reader (again, either due to the attribute being requested by the application 301 or due to a callbabck 316 being received indicating that the buffer 308 in the reader is about to be written over. The processed attribute values are stored in a byte array 311 in the scanner.

In this case, there is a one-to-one correspondence between the first type of object (in array 312) and the second type of object (in byte array 311). That is, each of the first type of object (in array 312) points to the buffer 308 in the reader 302. In this case, when a copy is made, the type one object points to a type 2 object (in byte array 311) in the scanner 304 that holds the actual values.

As described, a scanner of a markup language parser can omit execution of processing to create string objects or copy data into its buffer (depending upon the type of token being processed) in situations where the application does not ask for or otherwise require a string object and/or data.

Claims

1. A markup language parser that processes markup language data and provides processed data to an application, comprising: a reader configured to tokenize the markup language data and store resulting tokens in a buffer associated with the reader; a scanner configured to create string objects based on the tokens and to provide the created string objects to an application, wherein the scanner is further configured to determine a need by the application for the string objects; and for each at least some of the string objects created by the scanner, the scanner is configured to selectively create those string objects based on a determination of the need by the application for the string objects.

2. The markup language parser of claim 1, wherein for each of a selected plurality of tokens stored in the buffer associated with the reader, the scanner is configured to process a signal received by the scanner associated specifically with that token; and based on the signal, receive that token into the scanner from the reader.

3. The markup language parser of claim 2, wherein: the scanner is configured to hold a separate pointer pointing to each of the selected plurality of tokens; and the scanner is configured to receive each token into the scanner from the reader based by processing the pointer to that token.

4. The markup language parser of claim 2, wherein: the reader is configured to provide the signal, associated specifically with a token stored in the buffer associated with the reader, based on a condition that the token is to be overwritten in the buffer associated with the reader.

5. The markup language parser of claim 4, wherein: the scanner is configured to register, with the reader, pointers to the tokens in the buffer associated with the reader; and for a particular token stored in the buffer associated with the reader, the reader is configured to provide a signal, based on a condition that the token is to be overwritten in the buffer associated with the reader, only for tokens whose pointer is registered with the reader by the scanner.

6. The markup language parser of claim 5 wherein: the scanner being configured to register pointers to the tokens in the buffer associated with the reader includes, for each pointer, registering with the reader a listener object for that pointer; and wherein the signal the reader is configured to provide is a callback to the scanner from the reader.

7. The markup language parser of claim 1, wherein: the scanner being configured to receive a token into the scanner from the reader includes the scanner being configured to receive the token into a buffer associated with the scanner; and the scanner is configured to receive a token into the scanner from the buffer associated with the reader based on an indication of a need by the application for a string object, based on the token.

8. The markup language parser of claim 7, wherein: the scanner is further configured to receive a token into the scanner from the buffer associated with the reader based on an indication that the token is to be overwritten in the buffer associated with the reader.

9. The markup language parser of claim 8, wherein: the scanner maintains pointers to the tokens in the buffer associated with the reader.

10. The markup language parser of claim 9, wherein: the scanner registers the maintained pointers with the reader; and the reader notifies the scanner when a token in the buffer associated with the reader, corresponding to a registered maintained pointer, is to be overwritten in the buffer associated with the reader.

11. A method of operating a markup language parser to process markup language data and to provide processed data to an application, comprising: in a reader, tokenizing the markup language data and storing resulting tokens in a buffer associated with the reader; in a scanner, creating string objects based on the tokens and providing the created string objects to an application, wherein the scanner determines a need by the application for the string objects; and for each at least some of the string objects created by the scanner, the scanner selectively creates those string objects based on a determination of the need by the application for the string objects.

12. The markup language parser of claim 11, wherein for each of a selected plurality of tokens stored in the buffer associated with the reader, the scanner processes a signal received by the scanner associated specifically with that token; and based on the signal, receives that token into the scanner from the reader.

13. The markup language parser of claim 12, wherein: the scanner holds a separate pointer pointing to each of the selected plurality of tokens; and the scanner receives each token into the scanner from the reader based on processing the pointer to that token.

14. The markup language parser of claim 13, wherein: the reader is configured to provide the signal, associated specifically with a token stored in the buffer associated with the reader, based on a condition that the token is to be overwritten in the buffer associated with the reader.

15. The markup language parser of claim 14, wherein: the scanner is configured to register, with the reader, pointers to the tokens in the buffer associated with the reader; and for a particular token stored in the buffer associated with the reader, the reader is configured to provide a signal, based on a condition that the token is to be overwritten in the buffer associated with the reader, only for tokens whose pointer is registered with the reader by the scanner.

16. The markup language parser of claim 15, wherein: the scanner being configured to register pointers to the tokens in the buffer associated with the reader includes, for each pointer, registering with the reader a listener object for that pointer; and wherein the signal the reader is configured to provide is a callback to the scanner from the reader.

17. The markup language parser of claim 13, wherein: the scanner being configured to receive a token into the scanner from the reader includes the scanner being configured to receive the token into a buffer associated with the scanner; and the scanner is configured to receive a token into the scanner from the buffer associated with the reader based on an indication of a need by the application for a string object, based on the token.

18. The markup language parser of claim 14, wherein: the scanner is further configured to receive a token into the scanner from the buffer associated with the reader based on an indication that the token is to be overwritten in the buffer associated with the reader.

19. The markup language parser of claim 18, wherein: the scanner maintains pointers to the tokens in the buffer associated with the reader.

20. The markup language parser of claim 19, wherein: the scanner registers the maintained pointers with the reader; and the reader notifies the scanner when a token in the buffer associated with the reader, corresponding to a registered maintained pointer, is to be overwritten in the buffer associated with the reader.