The present invention relates generally to software systems for electronic document management, and more specifically to a method and system for concurrently processing multiple large data files transmitted with a multipart format.
As it is generally known, many computer software programs operate in part by transmitting files from client software executing on a client computer system to server software executing on a server computer system. Many such systems operate over computer networks such as the Internet, for example in a World Wide Web (“Web”) service environment.
Document management systems are example of computer software systems that transfer files from client systems to server systems. Document management systems are used to manage electronic documents, and have become important applications for many users. Document management systems may be used by businesses or individuals to manage a wide variety of digital assets, such as documents, reports, invoices, forms, faxes, e-mails, audio, video and images, etc. Document management systems may, for example, include a database to organize the stored documents, and a search mechanism to quickly find specific documents.
Some existing document management systems enable a user to import document files from local sources on a client computer system, and then store the imported files onto a remote server system. Files that may be imported into such systems vary in size, and may be significantly large. Additionally, the number of users sharing such a system, and that may be concurrently importing files, may also be large. Previous solutions have uploaded all imported documents into server system memory, but that approach can have a negative impact on system performance. For example, poor performance may result from limited random access memory (RAM) space that can be allocated to a run time environment on the server. This limitation is present in systems such as those that employ run time environments such as the Java Virtual Machine (JVM). The resulting performance degradation may cause server systems to become unresponsive, and/or perform poorly when processing large documents.
In particular, Web applications generally consist of a browser program on a client computer system, operating as a front end for rendering content such as HTML and handling user interactions, and server side applications, such as a Java® Servlet, for handling data transmitted from the browser. When a need arises to upload file(s) from the browser to the server for further processing, scalability and performance are important considerations in these systems because of the potentially large number of concurrent users. Furthermore, the data files transmitted to the server may need to be accessible in a flexible way, in order to support on-demand retrieval and handling. Accordingly, the access and handling of the data files should not be tied to the sequential network I/O (“input/output”). Some technique for storing the uploaded data must be used that allows for decoupling of the files from the sequential network I/O. For relatively small sizes of transmitted data files, a memory buffer holding the entire file content may be sufficient in this regard. However, such an approach scales poorly when large amounts of file content are uploaded, or when there are large numbers of concurrent users, since the memory buffer size would have to increase in proportion to the uploaded data. In those cases the server system would become slow to respond due to heavy memory load ,or even crash. Moreover, such an approach becomes impractical if the file(s) being transferred have sizes in the hundreds of megabytes range.
For the above reasons and others, it would be desirable to have a new system for document management, that provides improved performance with regard to concurrently transferring large numbers of documents from a client system to a server system.
To address the above described and other shortcomings of previous systems, a new method and system are disclosed for concurrently processing multiple large data files transmitted from a client system to a server system using a multipart format is disclosed. An object-oriented approach to representing the multipart data is used on the server system. The disclosed system advantageously allows flexible handling of files with the disclosed object-oriented design, and is easy to scale to large numbers of concurrent users and large sized document files. The disclosed system can be applied to a variety of client-server systems requiring concurrent importing and processing of large files.
In the disclosed system, data files are transmitted from a client computer system to a server computer system in a multipart format. For example, the multipart format could be form data submitted through an HTML browser agent in “multipart/form-data” format, or an electronic e-mail message submitted through an Internet mail agent. Any specific type or kind of client computer system software may be used to provide the data files to the server computer system in the multipart format.
On the application server system, the disclosed system operates to parse the multipart data stream, and save each file's content part in a temporary file through a file system operating on the server system. The temporary files generated by the disclosed system are represented outside of main memory, for example in a secondary storage device such as a magnetic storage disk or the like. The disclosed system also creates a corresponding multipart container object, which may be stored in memory on the server system. The multipart container object includes all relevant information regarding the multipart format the data files were received in, including a reference to each temporary file, such as a file name. The container object further provides methods that allow consumer programs of the transferred files to open up the temporary files in the file stream on-demand, and that delete the temporary files when the consumer program closes them. In this way the disclosed system advantageously eliminates the need to load the entire contents of a transferred file into memory, and preserves the on-demand property of the transmitted data retrieval for stream based operations.
Through the multipart container data object of the disclosed system, retrieval of large document contents is decoupled from the network input stream, and the files transferred from a client system to a server system can be obtained by consumer software on-demand. The file size to be processed through the disclosed system is only limited by network transmission limitations, and by server file system space that is relatively easy to scale.
Thus there is disclosed a new system for document management, that provides improved performance with regard to concurrently transferring large numbers of documents from a client system to a server system.
In order to facilitate a fuller understanding of the present invention, reference is now made to the appended drawings. These drawings should not be construed as limiting the present invention, but are intended to be exemplary only.
As shown in
The application server computer system 12 receives the multipart formatted data stream 22, and a software process 28 operates to process the received multipart data stream. Processing of the received multipart formatted data stream includes saving 36 large data parts from the data stream, such as files contained within the data stream, into corresponding ones of the temporary files 32 contained within the file system 30. The file system 30 may advantageously store the temporary files within a secondary storage device, such as a magnetic disk or the like. Processing of the received multipart formatted data stream at the application server computer system further includes creating 38 a multipart container object 40. The multipart container object 40 may advantageously be stored in a high speed memory, such as a RAM (Random Access Memory), contained within the application server computer system. The multipart container object 40 is operable to read 42 the temporary files 32 from the file system 30, and to provide the contents of the temporary files 32 to the consumer processes 46 as part of a file input stream 48. Examples of consumer processes 46 include a database for permanently storing the files from the multipart formatted datastream, such as an electronic mail (e-mail) database, an indexing service for creating a search index for the contents of the files from the multipart formatted datastream, or another specific type of server process executing on the application server computer system. The multipart container object 40 is further operable to delete the temporary files 32 from the file system 30 in response to operations from the consumer processes 46 requesting that the files stored in them be closed. Operation of the components in the embodiment illustrated in
The client computer system 10 and application server computer system 12 may each, for example, include at least one processor, primary program storage, such as memory, for storing program code executable on the processor, secondary storage, such as one or more magnetic disks or other secondary storage devices, on which files, such as those files managed by the file system 30, may be stored, and one or more other input/output devices and/or interfaces, such as data communication and/or peripheral devices and/or interfaces. The client computer system 10 and application server computer system 12 may each further include appropriate operating system and or other run-time software.
At step 66, software on the application server computer system parses the received multipart formatted data stream to extract each file contained within the received multipart formatted data stream. Further at step 66, the files contained within the received multipart formatted data stream are stored in corresponding temporary files provided through a file system operating on the application server computer system. The temporary files may, for example, be stored on a secondary storage device, such as a magnetic disk, thus obviating the need to completely store all the received files in the main memory of the application server computer system.
At step 68, software on the application computer system writes a reference to each temporary file stored through the file system on the application server computer system into the multipart container object. Such references to the temporary files may, for example, consist of file names of the corresponding temporary files.
In step 70, a consumer process executing in the application server computer system, which may include any specific type of server application program, such as an indexing process, database program, e-mail application server, Web-based content management server, or other consumer process, operates to access the files received in the multipart formatted data stream from the client computer system by invoking a method provided by the multipart container object formed on the application server computer system. In this way, the consumer process accesses a file input stream provided by the multipart container object.
The consumer process refers to the files it consumes through file references stored in the container object. As described further below, and shown in
The multipart container object provides the contents of the temporary files to the consumer process as part of the file input stream at step 72. At step 74, the multipart container object processes a request from the consumer process to close the file input stream by, at least in part, deleting one or more of the temporary files previously provided to the consumer process through the file input stream.
In
For example, if a user on the client computer system selected two files “file1.txt” and “file2.gif”, agent software on the client computer system, such as the browser program, would construct the parts of the multipart formatted datastream 22 of
The public “InputStream” method allows a consumer process on the application server computer system to obtain the files contained in the received multipart formatted datastream through the multipart data object 100.
The multipart data object 100 may be used to store any metadata extracted from the multipart formatted datastream. In addition to the file names stored in the “filename” vector 102, such metadata may include any other relevant information describing the files extracted from the multipart formatted datastream. Such metadata may include, for example, the length, type, and/or other characteristics of the extracted files. Such information stored in the multipart data object 100 is also made available to the consuming processes on the application server computer system.
The multipart formatted datastream used to transmit submitted files from a client computer system to an application server computer system may, for example, conform to the multipart format outlined in RFC2045 (“Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies”, N. Freed and N. Borenstein, November 1996.). As noted above, such a multipart formatted datastream can, for example, consist of form data submitted through an HTML browser agent with “multipart/form-data” format. Alternatively, the multipart formatted datastream can consist of an electronic mail (“e-mail”) message or messages submitted through an Internet mail agent software program executing on the client computer system, and that follows IANA (Internet Assigned Numbers Authority) specifications found in “Assigned Numbers”, STD 2, RFC 1700, USC/ISI, J. Reynolds and J. Postel, October 1994.
Many advantages are provided by the disclosed system. These include removing the need to store complete files from a received datastream in main memory of an application server computer system while these files are accessed by one or more consuming processes. Additionally, the files in the received datastream are made available to consumer processes “on-demand”, in that they are available to be consumed as soon as they are received at the application server computer system. When uploading potentially large files, such as from a browser at a client computer to a server for further processing, the disclosed object oriented representation of the uploaded files decouples the sequential data of a received network input/output (I/O) stream from accesses to the received file data performed by consuming application server software processes. Moreover, the size of files processed through the disclosed system is only limited by the capabilities of the network, which are typically sufficient in this regard, and by server file system space, which is relatively easy to scale.
Those skilled in the art should readily appreciate that programs defining the functions of the present invention can be delivered to a computer in many forms; including, but not limited to: (a) information permanently stored on non-writable storage media (e.g. read only memory devices within a computer such as ROM or CD-ROM disks readable by a computer I/O attachment); (b) information alterably stored on writable storage media (e.g. floppy disks and hard drives); or (c) information conveyed to a computer through communication media for example using wireless, baseband signaling or broadband signaling techniques, including carrier wave signaling-techniques, such as over computer or telephone networks via a modem.
While the invention is described through the above exemplary embodiments, it will be understood by those of ordinary skill in the art that modification to and variation of the illustrated embodiments may be made without departing from the inventive concepts herein disclosed. Moreover, while the preferred embodiments are described in connection with various illustrative program command structures, one skilled in the art will recognize that they may be embodied using a variety of specific command structures.