1. Field of the Invention
The present invention is directed to technology for transferring messages, such as e-mails, over a network.
2. Description of the Related Art
Traditional systems for delivering messages over a communications network have several drawbacks when the messages are large. For example, large e-mails can cause traditional e-mail delivery systems to create network congestion that ties up costly bandwidth resources. An e-mail with large attachments is delivered in its entirety to all listed recipients, even though some recipients may not need the attachments introducing undesirable redundancy that wastes network bandwidth and data storage resources.
E-mail servers 14 and 18 receive and transmit e-mails over network 10. In one example, e-mail servers 14 and 18 support Simple Mail Transfer Protocol (SMTP). Servers 14 and 18 support different protocols in other instances. E-mail clients 12 and 16 provide user interfaces that allow users to compose, review, and manage e-mails. In operation, a user employs client 12 to compose an e-mail message with attachments addressed to client 16. Using client 12, the user issues a send command that sends the e-mail to server 14, which sends the e-mail over network 10 to server 18. Server 18 recognizes that the e-mail is addressed to client 16 and delivers it accordingly.
Server 14 sends the e-mail over network 10 as soon as bandwidth is available, regardless of whether the e-mail needs to be delivered immediately. This can lead to a waste of valuable bandwidth, if the e-mail recipient does not need the message until much later. The magnitude of this inefficiency is exacerbated as the size of the e-mail increases. Many e-mail system administrators do not even permit the transmission or reception of excessively large messages.
In one example, a product manager at a corporation needs to send a large volume of sales support materials to all members of a field sales forces. The materials consume 200 megabytes of electronic storage space. Many e-mail systems will not permit the transmission or reception of a 200 megabyte file, and e-mail accounts for the sales force members may not have sufficient resources for storing an e-mail of this size. This forces the product manager to make and mail a CD-ROM version of the materials for each sales force member. It would be significantly more desirable for the product manager to have the ability to send the large volume of materials via e-mail or some other form of electronic message delivery.
The present invention, roughly described, pertains to technology for managing the transfer of messages, such as e-mails, over a network. The messages can be transferred over the network through background delivery, which allows message transfers to be scheduled outside the flow of traditional e-mail transmissions. In many instances, this provides improved network bandwidth utilization and allows larger messages transfers.
In one embodiment, a proxy server resides between an e-mail client and an e-mail server to receive outbound e-mail messages sent by the e-mail client. The proxy server determines whether the outgoing e-mail should be scheduled for background delivery. In one embodiment, the proxy server makes this determination based on the size of the e-mail, including attachments. If the e-mail exceeds a predetermined size, the proxy server arranges for content from the e-mail to be delivered to the intended recipient through a scheduled background delivery. Otherwise, the proxy server forwards the e-mail to the e-mail server for traditional delivery.
The proxy server prepares e-mail content, such as attachments, for background delivery. The e-mail proxy server creates and packages one or more assets from the content. The e-mail proxy server sends the intended e-mail recipient a notice when the content is ready to be retrieved. The intended recipient causes a scheduling request to be issued for delivery of the content, and the request is serviced.
In one implementation, one or more forward proxies receive and service scheduling requests by arranging for content to be delivered in accordance with a specified bandwidth schedule. The forward proxy delivers the content at the latest time possible that conforms to the schedule. This enables the content delivery to be made at
Proxy server 20 is coupled between client 12 and server 14 to facilitate background delivery of electronic message content from client 12. Proxy Server 20 receives outgoing e-mails from client 12, including any attachments in the e-mail. In alternate embodiments, proxy server 20 may not receive all of the attachments—instead, receiving only descriptions of the attachments. Proxy server 20 determines whether e-mail should be delivered to the intended recipient through server 14 or background delivery. If server 14 is selected, proxy server 20 forwards the e-mail to server 14, which delivers the e-mail to the intended recipient as described above with respect to
Incoming messages from network 10 for e-mail client 12 pass through e-mail server 14, as described above with reference to
Proxy server 20 includes receiver 22, splitter 24, and messenger 26, which are each coupled to file system 28 and database 30. File system 28 maintains content slated for background delivery, and database 30 holds meta data associated with electronic message content. Database 30 may be any type of database, and is a relational database in one embodiment. Receiver 22 receives outgoing e-mail from client 12 and stores it in file system 28. Receiver 22 determines whether background delivery is to be employed for each e-mail and updates meta data in database 30 with delivery instructions.
Splitter 24 uses the meta data in database 30 to identify e-mails in file system 28 that are slated for background delivery. Splitter 24 prepares content from the e-mails for background delivery. Splitter 24 also creates notifications for intended e-mail recipients to indicate that content is ready for retrieval. Splitter 24 updates the meta data in database 30 to indicate that notifications are ready to be sent.
Messenger 26 employs meta data in database 28 to identify e-mails in file system 28 that need to be sent. These e-mails include notifications of available content for background delivery and e-mails not meeting the criteria for background delivery. In one embodiment, messenger 26 forwards the e-mails, including notifications, to server 14 for forwarding over network 10. In an alternate embodiment, messenger 26 forwards notifications over network 10 without the use of server 14.
Content transfer client 36, forward proxy, 34, and content proxy 32 are used to transfer content from file system 28 to client 16 in a background delivery. E-mail client 16 receives a notification that content is read to be received from file system 28. Client 16 issues a data transfer request for the content to content transfer client 36, which issues a scheduling request for the content to forward proxy 34. If forward proxy 34 does not have a copy of the content, forward proxy 34 submits a scheduling request for the content to content proxy 32. Content proxy 32 retrieves the content from file system 28 and delivers the content to forward proxy 34, which sends the content to e-mail client 16 through content transfer client 36. In one embodiment, content transfer client 36 is implemented in program code running on the same system as client 16.
In further embodiments, all or multiple e-mail clients have an associated proxy server and content transfer client. Each e-mail client configured in this manner is able to send and receive content through background delivery.
In determining whether to arrange for background delivery (step 52), proxy server 20 determines whether the received electronic message satisfies a predetermined characteristic. In one embodiment, the predetermined characteristic is the content associated with the electronic message exceeding a size limit. In other embodiments, different characteristics or combinations of characteristics can be employed, such as the network identity of the intended recipient. In a further embodiment, all electronic messages undergo background delivery—eliminating the need for step 52.
If it is determined not to arrange for background delivery (step 52), receiver 22 enters meta data into database 30 indicating that the electronic message is not to be delivered via background delivery (step 64). Messenger 26 sees the meta data indication and forwards the electronic message to server 18 for delivery to the intended recipient (step 66).
If it is determined to arrange for background delivery (step 52), receiver 22 enters meta data into database 30 indicating that the electronic message is to be delivered via background delivery (step 54). Splitter 24 sees the meta data entry and identifies the electronic message as slated for background delivery (step 56). Splitter 24 proceeds to prepare content associated with the electronic message for background delivery (step 58). Greater details regarding the preparation of the contents will be provided below.
After the content is prepared for background delivery, proxy server 20 sends the intended recipient, such as client 16, a notice (step 60). The notice informs client 16 that the content is ready for retrieval. In one implementation, the notice contains content transfer client 36 for use in retrieving the content. Greater details regarding the notice will be provided below.
In one embodiment, proxy server 20 provides status regarding delivery of the content (step 62). This status includes an e-mail in one embodiment and a posting to a user accessible web page in another embodiment. The status allows the sender of the electronic message to track the status of the message.
In one implementation, each scheduling request includes a bandwidth schedule indicating required delivery specifications for the contents, such as required time for receipt and minimum delivery bandwidth requirements. Forward proxy 34 and content proxy 32 ensure that the requirements from the bandwidth schedule are satisfied, unless this is not possible. In one implementation, forward proxy 34 and content proxy 32 provide the content at the latest time possible that still satisfies the bandwidth schedule. Greater details regarding the performance of receiving and servicing scheduling requests are provided below.
In some instances, forward proxy 34 may receive multiple requests for the same data. Except for the first request, forward proxy 34 can deliver the content from its local memory. This can occur when multiple e-mail clients are the intended recipients of the same content. The intended recipient of the content can also forward the notification relating to the content to another entity. The entity can use the notification to retrieve the content in the same way as the original intended recipient. In one implementation, client 16 forwards the content notification from proxy server 20 in the same way any e-mail is forwarded using e-mail server 18.
Splitter 24 also creates a launch asset (step 92). One implementation of a launch asset contains information from the body of the original electronic message and links corresponding to assets in the package. Alternatively, separate links are not provided for each asset. Instead, the launch asset identifies the package assets and contains links that allow a user to accept all assets, reject all assets, or ignore the assets at this time. Some implementations of the launch asset also include program code for implementing content transfer client 36. One version of the launch asset is a Hypertext Markup Language (HTML) document.
Finally, splitter 24 updates meta data in database 30 (step 94). This update indicates that the content has been converted to package assets and is ready to be delivered. In one embodiment, the meta data update sets a lifetime for the package, indicating how long the package will be stored in file system 28.
Messenger 26 transmits the notice, including the notification message and attached launch asset (step 104). Messenger 26 transmits the notice to e-mail server 14 for delivery to the intended recipient over network 10 in one embodiment. Alternatively, messenger 26 transmits the notice directly over network 10 to the intended recipient. Messenger 26 also updates the meta data in database 30 to indicate that the notice has been sent (step 106).
When the notice arrives at the intended content, recipient such as e-mail client 16, the user opens the notice. As described above, one version of the notice's launch asset provides the user with links associated with each asset in the package maintained on file system 28. Selecting a link initiates a process in content transfer client 36 for issuing a scheduling request to retrieve the content asset associated with the link. Alternatively, the launch asset may only allow the user to accept, reject, or ignore all assets. In a further embodiment, a user may not be presented with options in the launch asset. Instead, the launch asset directs content transfer client 36 to automatically initiate the content retrieval process. If the user does not have a content transfer client, the user can install one provided in the launch asset.
Topology server 120 correlates forward proxies 34, 130, and 132 to content transfer clients 36, 126, and 128. When a content transfer client needs to retrieve content, the content transfer client looks to topology server 120 to determine the forward proxy that should receive a scheduling request for the content. This is illustrated by the flowchart in
In
Embodiments of the invention have been described above with reference to a proxy server. In alternate embodiments, a client proxy can also be employed. The client proxy runs on the same computer system as the e-mail client and performs the operations described above for receiver 22—allowing receiver 22 to be eliminated from proxy server 20. The client proxy begins buffering outgoing e-mail messages, until it can determine whether background delivery is desired for the message. If background delivery is not desired, the client proxy uploads the stored portion of the message and the remainder of the message to e-mail server 12. Otherwise, the client proxy uploads the message to proxy server 20 for storage in file system 28.
Alternatively, e-mail client 12 can include an add-in that integrates with e-mail client 12 to determine whether to implement background delivery for an electronic message. This is similar to the client proxy, except that the close integration prevents unnecessary grouping and encoding of attachments. As described above, e-mail clients frequently group and encode attachments—requiring proxy server 20 to decode and separate the attachments. The client add-in makes the background delivery determination prior to attachments being grouped and encoded. If background delivery is desired, the client add-in delivers the attachments to proxy server 20 without grouping and encoding—saving proxy server 20 the processing burden of decoding and separating attachments. Using a client add-in also allows for background delivery system options to be presented directly in the e-mail client's graphical user interface.
The following description relating to
Network 10 connects receiver node 210, sender node 220, and intermediary nodes 230 and 240. In this example, sender 220 is transferring data to receiver 210 through intermediaries 230 and 240. The data can include a variety of information such as text, graphics, video, and audio. Receiver 210 is a computing device, such as a personal computer, set-top box, or Internet appliance, and includes transfer module 212 and local storage 214. Sender 220 is a computing device, such as a web server or other appropriate electronic networking device, and includes transfer module 222. In further embodiments, sender 220 also includes local storage. Intermediaries 230 and 240 are computing devices, such as servers, and include transfer modules 232 and 242 and local storages 234 and 244, respectively.
Transfer modules 212, 222, 232, and 242 facilitate the scheduling of data transfers in accordance with the present invention. In the case of a virtual node, the transfer module for a non-member node that does not exchange scheduling communications is maintained on the virtual node. The virtual node can share the required scheduling information with the non-member node in certain embodiments.
The transfer module at each node evaluates a data transfer request in view of satisfying various objectives. Example objectives include meeting a deadline for completion of the transfer, minimizing the cost of bandwidth, a combination of these two objectives, or any other appropriate objectives. In one embodiment, a transfer module evaluates a data transfer request using known and estimated bandwidths at each node and known and estimated storage space at receiver 210 and intermediaries 230 and 240. A transfer module may also be responsive to a priority assigned to a data transfer. Greater detail regarding transfer module scheduling operations appears below.
Admission control module 310 receives user requests for data transfers and determines the feasibility of the requested transfers in conjunction with scheduling module 320 and routing module 330. Admission control module 310 queries routing module 330 to identify possible sources of the requested data. Scheduling module 320 evaluates the feasibility of a transfer from the sources identified by routing module 330 and reports back to admission control module 310.
Execution module 340 manages accepted data transfers and works with other modules to compensate for unexpected events that occur during a data transfer. Execution module 340 operates under the guidance of scheduling module 320, but also responds to dynamic conditions that are not under the control of scheduling module 320.
Slack module 350 determines an amount of available resources that should be uncommitted (reserved) in anticipation of differences between actual (measured) and estimated transmission times. Slack module 350 uses statistical estimates and historical performance data to perform this operation. Padding module 360 uses statistical models to determine how close to deadlines transfer module 300 should attempt to complete transfers.
Priority module 370 determines which transfers should be allowed to preempt other transfers. In various implementations of the present invention, preemption is based on priorities given by users, deadlines, confidence of transfer time estimates, or other appropriate criteria. Error recovery module 380 assures that the operations controlled by transfer module 300 can be returned to a consistent state if an unanticipated event occurs.
Several of the above-described modules in transfer module 300 are optional in different applications.
In alternate embodiments, the above-described transfer modules can have many different configurations. Also note that roles of the nodes operating as receiver 210, intermediary 230, and sender 220 can change—requiring their respective transfer modules to adapt their operation for supporting the roles of sender, receiver, and intermediary. For example, in one data transfer a specific computing device acts as intermediary 230 while in another data transfer the same device acts as sender 220. In
If the requested data is not stored locally (step 402), transfer module 300 determines whether the data request can be serviced externally by receiving a data transfer from another node in network 10 (step 404). If the request can be serviced, admission control module 310 accepts the user's data request (step 406). Since the data is not stored locally (step 410), the node containing transfer module 300 receives the data from an external source (step 414), namely the node in network 10 that indicated it would provide the requested data. In one instance, forward proxy 34 provides the data to transfer module 300 in content transfer client 36. The received data satisfies the data transfer request. Once the data is received, admission control module 310 signals the user that the data is available for use.
If the data request cannot be serviced externally (step 404), admission control module 310 provides the user with a soft rejection (408) in one embodiment. In one implementation, the soft rejection suggests a later deadline, higher priority, or a later submission time for the original request. A suggestion for a later deadline is optionally accompanied by an offer of waiting list status for the original deadline. Transfer module 300 determines whether the suggested alternative(s) in the soft rejection is acceptable (step 412). In one implementation, transfer module 300 queries the user. If the alternative(s) is acceptable, transfer module 300 once again determines whether the request can be externally serviced under the alternative condition(s) (step 404). Otherwise, the scheduling process is complete and the request will not be serviced. Alternate embodiments of the present invention do not provide for soft rejections.
If the receiver has sufficient resources (step 440), routing module 330 identifies the potential data sources for sending the requested data to the receiver (step 442). In one embodiment, routing module 330 maintains a listing of potential data sources. In another embodiment, routing module 330 in content transfer client 36 queries topology server 120 to obtain the identity of forward proxy 34. Scheduling module 320 selects an identified data source (step 444) and sends the data source an external scheduling request for the requested data (step 446). In one implementation, the external scheduling request identifies the desired data and a deadline for receiving the data. In further implementations, the scheduling request also defines a required bandwidth schedule that must be satisfied by the data source when transmitting the data.
The data source replies to the scheduling request with an acceptance or a denial. If the scheduling request is accepted, scheduling module 320 reserves bandwidth in the receiver for receiving the data (step 450) and informs admission control module 310 that the data request is serviceable. In the case of a virtual node, transfer module 300 reserves bandwidth (step 450) by instructing the associated non-member node to reserve the bandwidth. In alternate virtual node embodiments, the non-member node cannot be instructed to reserve bandwidth.
If the scheduling request is denied, scheduling module 320 determines whether requests have not yet been sent to any of the potential data sources identified by routing module 330 (step452). If there are remaining data sources, scheduling module 320 selects a new data source (step 444) and sends the new data source an external scheduling request (step 446). Otherwise, scheduling module 320 informs admission control module 310 that the request is not serviceable.
Transfer module 300 determines whether sufficient transmission resources exist for servicing the request (step 472). In one embodiment, scheduling module 300 in the data source determines whether sufficient bandwidth exists for transmitting the requested data (step 472). If the transmission resources are not sufficient, scheduling module 312 denies the scheduling request (step 480). In embodiments using soft rejections, scheduling module 320 also suggests alternative schedule criteria that could make the request serviceable, such as a later deadline.
If the transmission resources are sufficient (step 472) transfer module 300 reserves bandwidth at the data source for transmitting the requested data to the receiver (step 474). Virtual nodes reserve bandwidth by issuing an instruction to an associated non-member node. In some embodiments, bandwidth is not reserved, because the non-member node does not receive instructions from the virtual node.
Transfer module 300 in the data source determines whether the requested data is stored locally (step 476). If the data is stored locally, transfer module 300 informs the receiver that the scheduling request has been accepted (step 482) and transfers the data to the receiver at the desired time (step 490).
If the requested data is not stored locally (step 476), scheduling module 320 in the data source determines whether the data can be obtained from another node (step 478). If the data cannot be obtained, the scheduling request is denied (step 480). Otherwise, transfer module 300 in the data source informs the receiver that the scheduling request is accepted. Since the data is not store locally (step 484), the data source receives the data from another node (step 486) and transfers the data to the receiver at the desired time (step 490).
Preemption module 502 is employed in embodiments of the invention that support multiple levels of priority for data requests. More details regarding preemption based on priority levels is provided below.
The difference arising in
Transfer module 300 in node B determines whether multiple nodes are calling for the delivery of the same data from node B (step 520,
If node B is attempting to satisfy multiple requests for the same data (step 520), scheduling module 310 in node B generates a composite bandwidth schedule (step 522). After the composite bandwidth schedule is generated, transfer module 300 moves to step 440 and carries on the process as described in
The composite bandwidth schedule identifies the bandwidth demands a sender or intermediary must meet when providing data to node B, so that node B can service multiple requests for the same data. Further embodiments of the present invention are not limited to only servicing two requests. The principles for servicing two requests for the same data can be extended to any number of requests for the same data.
In one embodiment, node B issues a scheduling request for the composite bandwidth schedule before issuing any individual scheduling requests for the node C and node D bandwidth schedules. In an alternate embodiment, node B generates a composite bandwidth schedule after a scheduling request has been issued for servicing an individual bandwidth schedule for node C or node D. In this case, transfer module 300 instructs the recipient of the individual bandwidth scheduling request that the request has been cancelled. Alternatively, transfer module 300 receives a response to the individual bandwidth scheduling request and instructs the responding node to free the allocated bandwidth. In yet another embodiment, the composite bandwidth is generated at a data source (sender or intermediary) in response to receiving multiple scheduling requests for the same data.
Data transfers can be scheduled as either “store-and-forward” or “flow through” transfers.
Bandwidth schedule r(t) 532 shows a store-and-forward response to the scheduling request associated with bandwidth schedule s(t) 530. In store-and-forward bandwidth schedule 532, all data is delivered to the receiver prior to the beginning of schedule 530. This allows the node that issued the scheduling request with schedule 530 to receive and store all of the data before forwarding it to another entity. In this embodiment, the scheduling request could alternatively identify a single point in time when all data must be received.
Bandwidth schedule r(t) 534 shows a flow through response to the scheduling request associated with bandwidth schedule s(t) 530. In flow through bandwidth schedule 534, all data is delivered to the receiver prior to the completion of schedule 530. Flow through schedule r(t) 534 must always provide a cumulative amount of data greater than or equal to the cumulative amount called for by schedule s(t) 530. This allows the node that issued the scheduling request with schedule s(t) 530 to begin forwarding data to another entity before the node receives all of the data. Greater details regarding the generation of flow through bandwidth schedule r(t) 534 are presented below with reference to
The process in
Wherein:
This relationship allows the composite bandwidth schedule cb(t) to correspond to the latest possible data delivery schedule that satisfies both c(t) 536 and d(t) 538.
At some points in time, C(t) may be larger than D(t). At other points in time, D(t) may be larger than C(t). In some instances, D(t) and C(t) may be equal. Scheduling module 320 determines whether there is a data demand crossover within the selected interval (step 560,
When a data demand crossover does not occur within a selected interval, scheduling module 320 sets the composite bandwidth schedule to a single value for the entire interval (step 566). If C(t) is larger than D(t) throughout the interval, scheduling module 320 sets the single composite bandwidth value equal to the bandwidth value of c(t) for the interval. If D(t) is larger than C(t) throughout the interval, scheduling module 320 sets the composite bandwidth value equal to the bandwidth value of d(t) for the interval. If C(t) and D(t) are equal throughout the interval, scheduling module 320 sets the composite bandwidth value to the bandwidth value of d(t) or c(t)—they will be equal under this condition.
When a data demand crossover does occur within a selected interval, scheduling module 320 identifies the time in the interval when the crossover point of C(t) and D(t) occurs (step 562).
In one embodiment, scheduling module 320 identifies the time of the crossover point as follows:
Q=INT[(c—oldint−d—oldint)/(d(x)−c(x))]; and
RM=(c—oldint−d—oldint)−Q*(d(x)−c(x))
Wherein:
Scheduling module 320 employs the crossover point to set one or more values for the composite bandwidth schedule in the selected interval (step 564).
If the interval is not a single unit (step 582), scheduling module 320 sets two values for the composite bandwidth schedule within the selected interval (step 590). In one embodiment, these values are set as follows:
If the integer portion of the crossover does not occur at the starting point of the interval (step 580), scheduling module 320 determines whether the integer portion of the crossover occurs at the end point of the selected interval—meaning Q>0 and Q+1=w (step 584). If this is the case, scheduling module 320 sets two values for the composite bandwidth schedule within the interval (step 588). In one embodiment, these values are set as follows:
If the integer portion of the crossover is not an end point (step 584), scheduling module 320 sets three values for the composite bandwidth schedule in the selected interval (step 600). In one embodiment, these values are set as follows:
By applying the above-described operations, the data demanded by the composite bandwidth schedule during the selected interval equals the total data required for servicing the individual bandwidth schedules, c(t) and d(t). In one embodiment, this results in the data demanded by the composite bandwidth schedule from the beginning of time through the selected interval to equal the largest cumulative amount of data specified by one of the individual bandwidth schedules through the selected interval. In mathematical terms, for the case where a crossover exists between C(t) and D(t) within the selected interval and D(t) is larger than C(t) at the end of the interval:
Q=INT[(80−72)/(5−1)]=2
RM=(80−72)−2*(5−1)=0
For 0<=t<2: cb(t)=1;
For 2<=t<3: cb(t)=5−0=5; and
For 3<=t<5: cb(t)=5.
Composite bandwidth schedule 574 in
Q=INT[(80−72)/(5−2)]=2
RM=(80−72)−2*(5−2)=2
For 0<=t<2: cb(t)=2;
For 2<=t<3: cb(t)=5−2=3; and
For 3<=t<5: cb(t)=5.
Scheduling module 320 in the data source considers bandwidth schedule s(t) and constraints on the ability of the data source to provide data to the requesting node. One example of such a constraint is limited availability of transmission bandwidth. In one implementation, the constraints can be expressed as a constraint bandwidth schedule cn(t). In this embodiment, bandwidth schedules are generated as step functions. In alternate embodiments, bandwidth schedules can have different formats.
Scheduling module 320 selects an interval of time where bandwidth schedules s(t) and cn(t) have constant values (step 630). In one embodiment, scheduling module 320 begins selecting intervals from the time at the end of scheduling request bandwidth schedule s(t)—referred to herein as s_end. The selected interval begins at time x and extends for all time before time x+w—meaning the selected interval is expressed as x <=t<x+w. In one implementation, scheduling module 320 determines the values for send bandwidth schedule r(t) in the time period x+w<=t<s_end before selecting the interval x<=t<x+w.
Scheduling module 320 sets one or more values for the send bandwidth schedule r(t) in the selected interval (step 632). Scheduling module 300 determines whether any intervals remain unselected (step 634). In one implementation, intervals remain unselected as long the requirements of s(t) have not yet been satisfied and the constraint bandwidth schedule is non-zero for some time not yet selected.
If any intervals remain unselected, scheduling module 320 selects a new interval (step 630) and determines one or more send bandwidth values for the interval (step 632). Otherwise, scheduling module 320 determines whether the send bandwidth schedule meets the requirements of the scheduling request (step 636). In one example, constraint bandwidth schedule cn(t) may prevent the send bandwidth schedule r(t) from satisfying scheduling request bandwidth schedule s(t). If the scheduling request requirements are met (step 636), sufficient bandwidth exists and scheduling module 320 reserves transmission bandwidth (step 474,
For the selected interval, scheduling module 320 initially sets send bandwidth schedule r(t) equal to the constraint bandwidth schedule cn(t) (step 640). Scheduling module 320 then determines whether the value for constraint bandwidth schedule cn(t) is less than or equal to scheduling request bandwidth schedule s(t) within the selected interval (step 641). If so, send bandwidth schedule r(t) remains set to the value of constraint bandwidth schedule cn(t) in the selected interval. Otherwise, scheduling module 320 determines whether a crossover occurs in the selected interval (642).
A crossover may occur within the selected interval between the values R(t) and S(t), as described below:
(representing the accumulated data specified by send bandwidth schedule r(t) as initially set, in a range spanning the beginning of the selected interval through spend); and
(representing the accumulated data specified by scheduling request bandwidth schedule s(t) in a range spanning the beginning of the selected interval through s_end).
A crossover occurs when the lines defined by R(t) and S(t) cross. When a crossover does not occur within the selected interval, scheduling module 320 sets send bandwidth schedule r(t) to the value of constraint bandwidth schedule cn(t) for the entire interval (step 648).
When a crossover does occur within a selected interval, scheduling module 320 identifies the time in the interval when the crossover point occurs (step 644).
In one embodiment, scheduling module 300 identifies the time of the crossover point as follows:
Q=INT[(s—oldint−r—oldint)/(cn(x)−s(x))]; and
RM=(s—oldint−r—oldint)−Q*(cn(x)−s(x))
Wherein:
Scheduling module 320 employs the crossover point to set one or more final values for send bandwidth schedule r(t) in the selected interval (step 646,
If the interval is not a single unit (step 662), scheduling module 320 sets two values for send bandwidth schedule r(t) within the selected interval (step 668). In one embodiment, these values are set as follows:
If the integer portion of the crossover does not occurs at the end point of the interval (step 660), scheduling module 320 determines whether the integer portion of the crossover occurs at the start point of the selected interval—meaning Q>0 and Q+1=w (step 664). If this is the case, scheduling module 320 sets two values for send bandwidth schedule r(t) within the selected interval (step 670). In one embodiment, these values are set as follows:
If the integer portion of the crossover is not a start point (step 664), scheduling module 320 sets three values for send bandwidth schedule r(t) in the selected interval (step 670). In one embodiment, these values are set as follows:
By applying the above-described operations, send bandwidth schedule r(t) provides data that satisfies scheduling request bandwidth schedule s(t) as late as possible. In one embodiment, where cn(t)>s(t) for a selected interval, the above-described operations result in the cumulative amount of data specified by r(t) from s_end through the start of the selected interval (x) to equal the cumulative amount of data specified by s(t) from s_end through the start of the selected interval (x).
Q=INT[(80−72)/(5−1)]=2
RM=(80−72)−2*(5−1)=0
For 0<=t<2: r(t)=1;
For 2<=t<3: r(t)=1+0=1; and
For 3<=t<5: r(t)=5.
Send bandwidth schedule 654 in
Q=INT[(80−72)/(5−2)]=2
RM=(80−72)−2*(5−2)=2
For 0<=t<2: r(t)=2;
For 2<=t<3: r(t)=2+2=4; and
For 3<=t<5: r(t)=5.
Some embodiments of the present invention employ forward and reverse proxies. A forward proxy is recognized by a node that desires data from a data source as a preferable alternate source for the data. If the node has a forward proxy for desired data, the node first attempts to retrieve the data from the forward proxy. A reverse proxy is identified by a data source in response to a scheduling request as an alternate source for requested data. After receiving the reverse proxy, the requesting node attempts to retrieve the requested data from the reverse proxy instead of the original data source. A node maintains a redirection table that correlates forward and reverse proxies to data sources, effectively converting reverse proxies into forward proxies for later use. Using the redirection table avoids the need to receive the same reverse proxy multiple times from a data source. In alternate embodiments of the system in
In order to handle proxies, the process in
Priority module 370 (
If preemption of a lower priority transfer will not allow a request to be serviced (step 720), the request is finally rejected (step 724). Otherwise, transfer module 300 preempts a previously scheduled transfer so the current request can be serviced (step 722). In one embodiment, preemption module 502 (
Transfer module 300 determines whether the preemption causes a previously accepted request to miss a deadline (step 726). Fox example, the preemption may cause a preempted data transfer to fall outside a specified window of time. If so, transfer module 300 notifies the data recipient of the delay (step 728). In either case, transfer module 300 accepts the higher priority data transfer request (step 406) and proceeds as described above with reference to
In further embodiments, transfer module 300 instructs receiver scheduling module 320 to poll source nodes of accepted transfers to update their status. Source node scheduling module 320 replies with an OK message (no change in status), a DELAYED message (transfer delayed by some time), or a CANCELED message.
If the assigned priority of the current request is not higher than any of the scheduled transfers (step 740), preemption is not available. Otherwise, priority module 370 determines whether the current request was rejected because all transmit bandwidth at the source node was already allocated (step 742). If so, preemption module 502 preempts one or more previously accepted transfers from the source node (step 746). If not, priority module 370 determines whether the current request was rejected because there was no room for padding (step 744). If so, preemption module 502 borrows resources from other transfers at the time of execution in order to meet the deadline. If not, preemption module 502 employs expensive bandwidth that is available to requests with the priority level of the current request (step 750). In some instances, the available bandwidth may still be insufficient.
For a time slice of length TS, execution module 330 apportions B bytes to transfer T (step 770), where B is the integral of the bandwidth schedule from CTT to CTT+TS. After detecting the end of time slice TS (step 772), execution module 340 determines the number of bytes actually transferred, namely B′ (step 774). Execution module 340 then updates CTT to a new value, namely CTT′ (step 776), where the integral from CTT to CTT′ is B′.
At the end of time slice TS, execution module 340 determines whether the B′ amount of data actually transferred is less than the scheduled B amount of data (step 778). If so, execution module 340 updates a carry forward value CF to a new value CF′, where CF′=CF+B−B′. Otherwise, CF is not updated. The carry forward value keeps track of how many scheduled bytes have not been transferred.
Any bandwidth not apportioned to other scheduled transfers can be used to reduce the carry forward. Execution module 340 also keeps track of which scheduled transfers have been started or aborted. Transfers may not start as scheduled either because space is not available at a receiver or because the data is not available at a sender. Bandwidth planned for use in other transfers that have not started or been aborted is also available for apportionment to reduce the carry forward.
As seen from
Execution module 340 is responsible for transferring data at the scheduled rates. Given a set of accepted requests and a time interval, execution module 340 selects the data and data rates to employ during the time interval. In one embodiment, execution module 340 uses methods as disclosed in the co-pending application entitled “System and Method for Controlling Data Transfer Rates on a Network.”
The operation of execution module 340 is responsive to the operation of scheduling module 320. For example, if scheduling module 320 constructs explicit schedules, execution module 340 attempts to carry out the scheduled data transfers as close as possible to the schedules. Alternatively, execution module 340 performs data transfers as early as possible, including ahead of schedule. If scheduling module 320 uses feasibility test module 502 to accept data transfer request, execution module 340 uses the results of those tests to prioritize the accepted requests.
As shown in
Execution module 340 on each sender apportions the available transmit bandwidth among all of these competing transfers. In some implementations, each sender attempts to send the amount of data for each transfer determined by this apportionment. Similarly, execution module 340 on each receiver may apportion the available receive bandwidth among all the competing transfers. In some implementations, receivers control data transfer rates. In these implementations, the desired data transfer rates are set based on the amount of data apportioned to each receiver by execution module 340 and the length of the time slice TS.
In other implementations, both a sender and receiver have some control over the transfer. In these implementations, the sender attempts to send the amount of data apportioned to each transfer by its execution module 340. The actual amount of data that can be sent, however, may be restricted either by rate control at a receiver or by explicit messages from the receiver giving an upper bound on how much data a receiver will accept from each transfer.
Execution module 340 uses a dynamic request protocol to execute data transfers ahead of schedule. One embodiment of the dynamic request protocol has the following four message types:
DREQ(id, start, rlimit, Dt) is a message from a receiver to a sender calling for the sender to deliver as much as possible of a scheduled transfer identified by id. The DREQ specifies for the delivery to be between times start and start+Dt at a rate less than or equal to rlimit. The receiver reserves rlimit bandwidth during the time interval from start to start+Dt for use by this DREQ. The product of the reserved bandwidth, rlimit, and the time interval, Dt, must be greater than or equal to a minimum data size BLOCK. The value of start is optionally restricted to values between the current time and a fixed amount of time in the future. The DREQ expires if the receiver does not get a data or message response from the sender by time start+Dt.
DGR(id, rlimit) is a message from a sender to a receiver to acknowledge a DREQ message. DGR notifies the receiver that the sender intends to transfer the requested data at a rate that is less than or equal to rlimit. The value of rlimit used in the DGR command must be less than or equal to the limit of the corresponding DREQ.
DEND_RCV(id, size) is a message from a receiver to a sender to inform the sender to stop sending data requested by a DREQ message with the same id. DEND also indicates that the receiver has received size bytes.
DEND_XMIT(id, size, Dt) is a message from a sender to a receiver to signal that the sender has stopped sending data requested by a DREQ message with the same id, and that size bytes have been sent. The message also instructs the receiver not to make another DREQ request to the sender until Dt time has passed. In one implementation, the message DEND_XMIT(id, 0, Dt) is used as a negative acknowledgment of a DREQ.
A transfer in progress and initiated by a DREQ message cannot be preempted by another DREQ message in the middle of a transmission of the minimum data size BLOCK. Resource reservations for data transfers are canceled when the scheduled data transfers are completed prior to their scheduled transfer time. The reservation cancellation is done each time the transfer of a BLOCK of data is completed.
If a receiver has excess receive bandwidth available, the receiver can send a DREQ message to a sender associated with a scheduled transfer that is not in progress. Transfers not in progress and with the earliest start time are given the highest priority. In systems that include time varying cost functions for bandwidth, the highest priority transfer not in progress is optionally the one for which moving bandwidth consumption from the scheduled time to the present will provide the greatest cost savings. The receiver does not send a DREQ message unless it has space available to hold the result of the DREQ message until its expected use (i.e. the deadline of the scheduled transfer).
If a sender has transmit bandwidth available, and has received several DREQ messages requesting data transfer bandwidth, the highest priority DREQ message corresponds to the scheduled transfer that has the earliest start time. The priority of DREQ messages for transfers to intermediate local storages is optionally higher than direct transfers. Completing these transfers early will enable the completion of other data transfers from an intermediary in response to DREQ messages. While sending the first BLOCK of data for some DREQ, the sender updates its transmit schedule and then re-computes the priorities of all pending DREQ'S. Similarly, a receiver can update its receive schedule and re-compute the priorities of all scheduled transfers not in progress.
In one embodiment of the present invention, transfer module 300 accounts for transmission rate variations when reserving resources. Slack module 350 (
In one embodiment slack module 350 reserves a fixed percentage of all bandwidth resources (e.g. 20%). In an alternative embodiment, slack module 350 reserves a larger fraction of bandwidth resources at times when transfers have historically run behind schedule (e.g., between 2 and 5 PM on weekdays). The reserved fraction of bandwidth is optionally spread uniformly throughout each hour, or alternatively concentrated in small time intervals (e.g., 1 minute out of each 5 minute time period).
In one implementation, transfer module 300 further guards against transmission rate variations by padding bandwidth reserved for data transfers. Padding module 360 (
In one embodiment of padding module 360, P is set as follows:
P=MAX[MIN—PAD, PAD_FRACTION*ST]
Wherein:
In one implementation MIN_PAD is 15 minutes, and PAD_FRACTION is 0.25. In alternative embodiments, MIN_PAD and PAD_FRACTION are varied as functions of time of day, sender-receiver pairs, or historical data. For example, when a scheduled transfer spans a 2 PM-5 PM interval, MIN_PAD may be increased by 30 minutes.
In another embodiment, P is set as follows:
P=ABS—PAD+FRAC—PAD_TIME
Wherein:
In this embodiment, available bandwidth is taken into account when FRAC_PAD_TIME is computed from B.
In further embodiments, transfer module 300 employs error recovery module 380 (
In one implementation, data is stored in each node to facilitate restarting data transfers. Examples of this data include data regarding requests accepted by scheduling module 320, resource allocation, the state of each transfer in progress, waiting lists 508 (if these are supported), and any state required to describe routing policies (e.g., proxy lists).
Error recovery module 380 maintains a persistent state in an incremental manner. For example, data stored by error recovery module 380 is updated each time one of the following events occurs: (1) a new request is accepted; (2) an old request is preempted or; (3) a DREQ transfers data of size BLOCK. The persistent state data is reduced at regular intervals by eliminating all requests and DREQs for transfers that have already been completed or have deadlines in the past.
In one embodiment, the persistent state for each sender includes the following: (1) a description of the allocated transmit bandwidth for each accepted request and (2) a summary of each transmission completed in response to a DREQ. The persistent state for each receiver includes the following: (1) a description of the allocated receive bandwidth and allocated space for each accepted request and (2) a summary of each data transfer completed in response to a DREQ.
Although many of the embodiments discussed above describe a distributed system, a centrally controlled system is within the scope of the invention. In one embodiment, a central control node, such as a server, includes transfer module 300. In the central control node, transfer module 300 evaluates each request for data transfers between nodes in communication network 10. Transfer module 300 in the central control node also manages the execution of scheduled data transfers and dynamic requests.
Transfer module 300 in the central control node periodically interrogates (polls) each node to ascertain the node's resources, such as bandwidth and storage space. Transfer module 300 then uses this information to determine whether a data transfer request should be accepted or denied. In this embodiment, transfer module 300 in the central control node includes software required to schedule and execute data transfers. This allows the amount of software needed at the other nodes in communications network 10 to be smaller than in fully distributed embodiments. In another embodiment, multiple central control devices are implemented in communications network 10.
The system of
Portable storage medium drive 962 operates in conjunction with a portable non-volatile storage medium, such as a floppy disk, to input and output data and code to and from the computer system of
User input device(s) 960 provide a portion of a user interface. User input device(s) 960 may include an alpha-numeric keypad for inputting alpha-numeric and other information, or a pointing device, such as a mouse, a trackball, stylus, or cursor direction keys. In order to display textual and graphical information, the computer system of
The components contained in the computer system of
The foregoing detailed description of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The described embodiments were chosen in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto.
This Application is related to the following Applications: U.S. patent application Ser. No. 09/853,816, entitled “System and Method for Controlling Data Transfer Rates on a Network,” filed May 11, 2001; U.S. patent application Ser. No. 09/935,016, entitled “System and Method for Scheduling and Executing Data Transfers Over a Network,” filed Aug. 21, 2001; U.S. patent application Ser. No. 09/852,464, entitled “System and Method for Automated and Optimized File Transfers Among Devices in a Network,” filed May 9, 2001; U.S. patent application Ser. No. 10/356,709, entitled “Scheduling Data Transfers For Multiple Use Request,” Attorney Docket No. RADI-01000US0, filed Jan. 31, 2003; and U.S. patent application Ser. No. 10/356,714, entitled “Scheduling Data Transfers Using Virtual Nodes,” Attorney Docket No. RADI-01001US0, filed Jan. 31, 2003. Each of these related Applications is incorporated herein by reference.