Patent Application
20070033305
Application programs run on an operating system of a computer. When an application program requires data for processing, a request is sent to the operating system to retrieve the required data. The data is copied from a source file in storage such as a disk drive or other persistent storage device, via one or more input/output (I/O) buffers, into a buffer of the processor which is assigned to the application program. The application program can then read or modify the data in the buffer. Once the processing of the data or a part of the data is complete, the processed data is stored by writing it to a destination file in storage.
Many such file operations involve reading a file and writing it in another location without making any modifications to the original data. In this case, operating system and processor resources are needlessly occupied in transferring data to and from storage. Some operating systems implement special mechanisms to make this process more efficient. However, benefiting from such mechanisms can require modifications to applications and utilities which are based on standard interfaces.
Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings in which:
There will be described a method transferring data for use by a processor, the method comprising the steps of:
If in step c) a read access is detected then the data may be copied from the input data store to the current data store and, in response to a subsequent write operation on the current data store, the contents of the current data store copied to the output data store. Access to the current data store may be removed and the read access detected when an access fault is generated for the current data store. Once the data is transferred into the input data store then the store may be marked as busy or in-use.
Steps c) and d) may be triggered by a flag associated with the file from which the data is transferred. The data may be processed under the control of one or more application programs running on the processor. The input data store may be provided by one or more I/O read buffers, the output data store may be provided by one or more I/O write buffers and the current data store may be provided by one or more application buffers.
There will also be described apparatus for transferring data for use by a processor, the apparatus being operable to assign an input data store for holding input data for subsequent processing, an output data store for holding processed data and a current data store for holding data for use by the processor; transfer data for use by the processor from storage into the input data store; monitor the current data store for a read access by the processor; and if no read access is detected for the current data store then, in response to a write operation on the current data store, to save the contents of the input data store to storage.
Further embodiments provide a computer processor operable to transfer data from a source file into a read buffer for use by an application program; assign an application buffer to the application program; deny access by the application program to the application buffer; and if the application program issues a write on the application buffer without first having attempted to read the buffer then to save the data from the read buffer to a destination file.
An embodiment provides a method of transferring data from storage for use by an application program, the method comprising the steps of:
In step b) the permissions may be set so as to bar access to the application buffer and the access fault is a read fault. The method may comprise the further step of:
Some embodiments provide a program or a group of programs arranged to enable a programmable device or a group of programmable devices to carry out a method transferring data for use by a processor, the method comprising the steps of:
Some embodiments provide a program or a group of programs arranged to enable a programmable device or a group of programmable devices to provide apparatus for transferring data for use by a processor, the apparatus being operable to assign an input data store for holding input data for subsequent processing, an output data store for holding processed data and a current data store for holding data for use by the processor; transfer data for use by the processor from storage into the input data store; monitor the current data store for a read access by the processor; and if no read access is detected for the current data store then, in response to a write operation on the current data store, to save the contents of the input data store to storage.
Some embodiments provide a program or a group of programs arranged to enable a programmable device or a group of programmable devices to provide apparatus for transferring data for use by a processor, the apparatus being operable to transfer data for use by the processor from storage into the input data store and copying the data into the current data store; monitor the current data store for a write access by the processor; and if no write access is detected for the current data store then, in response to a write operation on the current data store, to save the contents of the input data store to a file in storage.
Some embodiments provide a program or a group of programs arranged to enable a programmable device or a group of programmable devices to carry out a method of transferring data from storage for use by an application program, the method comprising the steps of:
With reference to
In the normal mode, when the application program requires data from a source file 109 stored on the storage device 105, the program first opens the file for reading and receives a file descriptor (FWR) in return. The application then opens a destination file 111 to which to write the results of its processing and receives a further file descriptor (FDW). The file descriptors (FDR, FDW) provide the physical addresses of the files on the storage device 105.
The application program 107 is allocated a buffer (BUF) 113 for holding the data from the opened file 109 which it is processing. When the program issues a read instruction of a number of bytes on the buffer 113, the requested data is transferred, under the control of the kernel, from the source file 109 on the disk drive to an input/out read buffer (IOBR) 115 on the processor 103. From the IOBR the data is then copied into the application buffer (BUF) 113. Once the requested data is in the application buffer it can be read and modified by the application program. When the application program 107 issues a write operation of a number of bytes of the data in the buffer (BUF) the allocated contents of the BUF are copied into an input/output write buffer (IOBW) 117. From the IOBW the data is transferred, under the control of the kernel, to the destination file 111 on the storage device 105.
In the optimized mode, the kernel carries out further steps which can reduce the number of copy operations carried out by the processor 103 when transferring data between files on the storage device and the application buffer. The optimized mode is set by a system administrator. In the optimized mode, when providing data requested by an application program, the kernel only copies the data from the IOBR into the application buffer if the application program attempts to read the data from the application buffer. If the buffer is not read before it is written back to disk then the data from the read buffer IOBR is saved to disk as indicated by arrow 119 in
In the optimized mode, once data has been transferred into the read buffer IOBR, the application buffer is marked as “no access” and the buffer IOBR is marked as busy or in use. Marking the buffer as “no access” enables any read by the application on its buffer to be detected or monitored by the kernel as it will generate an access fault for that buffer. Denying access to the read buffer IOBR preserves its contents.
The processing carried out by the kernel in the normal and optimized modes will now be described in further detail with reference to
At step 207, the kernel checks whether it has been set to optimized or normal mode. If the kernel is operating in normal mode then processing moves to step 209 where the contents of the read buffer IOBR are copied into the application buffer and processing moves to step 307 in the flow chart of
Processing continues from step 301 either as a result of an access fault being detected by the kernel on the application buffer or by the application issuing a write instruction on the application buffer. If at step 301 the application issues a write instruction then processing moves to step 303. If an access fault is detected at step 301 then processing moves to step 307 where the contents of the read buffer IOBR are copied to the application buffer and the optimized copy flag for the application buffer is deactivated. Processing then moves to step 307 and waits for a write to be issued on the application buffer before moving to step 303.
At step 303, if the optimized copy flag is still active then processing moves to step 309 where the contents of the read buffer IOBR 115 are written to the destination file 111 and processing returns to step 205. If at step 303 the optimized copy flag was not active as processing passed though step 209 or has been deactivated when processing passed though step 305 then processing moves to step 311 where the contents of the application buffer 113 are written to the write buffer IOBW 117 before being written to the destination file 111 at step 313. Processing then returns to step 205 to await the next read issued by the application program. If at step 303 the optimized copy flag is activated then the contents of the read buffer IOBR are transferred directly to the destination file 111 and processing returns to step 205 as described above.
In the above arrangement, if the application program does not attempt to access the data in the application buffer then the optimized mode saves two buffer copies within the processor when compared to the normal mode of operation. In cases where data has just to be copied from one file (or device) to another, the optimizations can be applied implicitly by the kernel, without the application being aware of the modified operation. This is an advantage in cases where it is not possible to rebuild or rewrite the applications and is useful for systems designed for a wide variety of operating systems where it may be difficult to utilize special interfaces provided by an OS.
In another embodiment, the buffers IOBR, IOBW, BUF and the I/O operation ranges are of different sizes. The sizes of I/O buffers can be system and file system dependent, while BUF size can be application dependent. Additional structures are added to enable such fragmented I/O to be tracked. These structures are used to track the extent to which the I/O buffers IOBR, IOBW have been read or written to and also verification that the same application buffer BUF is used for the operations. This enables fragmented I/O to be handled while carrying out the processing according to the optimized or normal modes as described above with reference to
For the I/O buffers IOBR, IOBW:
For the application buffer BUF:
When the application program issues a read, it passes the input file descriptor (FDR), the buffer (BUF) and a read size (RDSZ) as arguments to the kernel. In response to this the kernel retrieves the file offset for the file from an internal data structure indexed by file descriptor and processes the read as set out in the following pseudo code:
In this code routine, if the temporary variable Temp_var>IOBR[i]—>Buffer_size, the kernel gets the next buffer IOBR[i+l] and update its fields until the requested RDSZ bytes are processed, at which point the kernel returns from read. If BUF is written to any time after the read and before a subsequent write on the application buffer, the Buffer_changed flag will be set to true, which will deactivate the optimized copy flag.
When the application issues a write operation on the application buffer, it passes the destination file descriptor (FDW), the buffer (BUF) and the write size (WRSZ) as arguments to the kernel. The kernel retrieves the file offset for the destination file from an internal data structure indexed by the file descriptor and also gets the relevant 10 Buffer, IOBW[i] that corresponds to this offset. The kernel then carries out the processing set out below in pseudo code:
In this code routine, if the variable Temp_var>IOBW[i]->Buffer_size, the kernel continues writing subsequent buffers until the write of WRSZ bytes is complete when the kernel returns from write. If the buffer has been changed then the optimized copy flag is deactivated and a normal copy is carried out rather than an optimized copy.
In some embodiments, in the optimized mode, data is transferred from storage into the read buffer IOBR and then copied into the application buffer BUF which is marked as read only. When the application attempts to read the contents of its application buffer no fault is generated. Only when the application program attempts to change the data in the buffer is a fault generated. In response to the fault, the kernel modifies the permissions to enable the write and deactivates the optimized copy flag. If no write fault is generated for the buffer before the application program issues a write on the buffer then the data from the read buffer IOBR is saved to the destination file in storage. This arrangement reduces the number of copies carried out in the processor by one, if the data in the application buffer remains unchanged.
In another embodiment, the kernel is arranged to monitor the performance of an application program and to dynamically select whether to apply the first technique described above where read faults are detected or the second technique where only write faults are detected. For example if the first technique is adopted for initial processing of the application program but more than half the reads in a predetermined period are not optimized then the kernel may switch to the second technique for the duration of the processing.
In a further embodiment, the file system used for the storage device 105 includes sparse files. A sparse file is a file that does not contain continuous data, but includes gaps or holes in the file where no data is present. The file systems that support sparse files do not allocate disk space for the holes, which conserves disk space. A normal read operation on a hole, however, succeeds, and returns a buffer full of zeros. Thus the fact that a file is sparse is transparent to the application program reading the file. During a normal copy operation, when the holes are read and written, they are converted to a normal file blocks, causing extra disk space to be used.
In this embodiment, each sparse block in the source file is flagged. When the kernel is operating in optimized mode as described above and reads a block from the source file which is flagged as sparse, the read does not result in any I/O operation. A read buffer IOBR is allocated, and marked as sparse. The kernel removes the access permissions for application buffer BUF as described above and the kernel returns from the read operation. If the application issues a write on the application buffer BUF without accessing its contents, the read buffer IOBR is marked for write and the write operation responds to the sparse flag by creating a gap or hole in the destination file. In this way, the sparse nature of the file is maintained if the application makes no change to the data. If the application buffer BUF is accessed or has been written to then write completes in a normal fashion as described above.
Using this approach, the sparse attribute of a file can be maintained across a read and write, and can be replicated at the destination. In cases where a sparse file is being backed up to a storage device such as a tape device, it is possible to convey the sparse nature of a file to the tape, thus enabling full re-creation of the file during a restore operation.
Embodiments may be applied to file streaming systems such as multi media file systems (MMFS) which host video and audio files for streaming download across networks. Such files are usually read from beginning to end in a continuous manner. The data read goes unchanged to a device such as a network interface card (NIC) for transmission to another computer or digital system. This arrangement can be treated as a copy where the destination is a NIC instead of a disk based file and therefore this data transfer process can be optimized. Since the application buffer BUF will be an unchanged copy of read buffer IOBR, it can be directly sent to the NIC, avoiding two extra buffer copies (IOBR to BUF and BUF to IOBW). Thus a MMFS using this approach can implicitly implement a copy optimization to provide an improved throughput.
In a further embodiments, instead of the kernel having a normal and an optimized mode, particular files or applications are marked as being suitable for optimized processing. In another embodiment, when a file system is mounted, a special option is provided that requests that all files residing on that file system be marked for copy optimization. In a further embodiment, an attribute is set on a file that causes it to be marked for copy optimization or when the file is opened, an option is passed to it for copy optimization to be set. In each of these cases, the copy optimization flag is set in the internal file system management data structures and used for subsequent operations.
It will be understood by those skilled in the art that the apparatus that embodies a part or all of the present invention may be a general purpose device having software arranged to provide a part or all of an embodiment of the invention. The device could be single device or a group of devices and the software could be a single program or a set of programs. Furthermore, any or all of the software used to implement the invention can be communicated via suitable transmission or storage means so that the software can be loaded onto one or more devices.
While the present invention has been illustrated by the description of the embodiments thereof, and while the embodiments have been described in considerable detail, it is not the intention of the applicant to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details representative apparatus and method, and illustrative examples shown and described. Accordingly, departures may be made from such details without departure from the spirit or scope of applicant's general inventive concept.
| Number | Date | Country | Kind |
|---|---|---|---|
| IN1100/CHE/2005 | Aug 2005 | IN | national |
