Often data backups are stored remotely from a storage being backed up. For example, a central remote network server performs and manages backup of clients to a remote networked storage. However, when data is backed up to a remote storage, the data to be backed is sent via a network that is often bandwidth constrained. Additionally, when data is restored from a remote storage, the data to be used to restore data is also sent via the bandwidth constrained network. This may lead to long backup and restore times. Therefore, there exists a need for a way to more efficiently perform and restore from a backup.
Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
Improving performance of a backup system is disclosed. In some embodiments, contents of a plurality of backups that share a common characteristic are profiled. For example, backup data from a plurality of clients that share the same operating system are divided into chunks and analyzed to determine a commonality within the plurality of backup data. A portion of the plurality of backups is selected as a base backup reference data to be distributed. For example, chunks that have been identified as frequently included in the plurality of backups are selected to be included in a set of chunks to be utilized as a backup reference set. If data to be backed up or restored includes data portions (e.g., chunks) that are also included in the backup reference set, an identifier of the data portion(s) may be communicated over a network rather than the actual contents of the data portion(s) because both parties are storing the backup reference set. A first copy of the base backup reference data is stored in a storage of a backup server. For example, the base backup reference is stored in a storage in close network proximity to the backup server facilitating backup and restore of a backup source. A second copy of the base backup reference is stored at a storage of a backup source that shares the common characteristic. The backup source is located remotely for the backup server. For example, a client that is utilizing the same operating system as the backup sources of the plurality of backups stores the base backup reference data for use during backups and restores.
In some embodiments, remote backup application/server 102 is located remotely (e.g., connected via a network) from a backup source to be backed up and/or restored. For example, when data is to be backed from client 106, backup application 102 remotely initiates a backup process to store the backup data to: storage included in backup server 102, a storage physically connected to server 102, and/or a remote data storage accessible via network connection (e.g., network storage located close in network proximity to backup server 102). In another example, when data is to be restored to client 106, backup application 102 initiates a backup process using stored backup data from: storage included in backup server 102, storage physically connected to server 102, and/or to a remote data storage accessible via network connection (e.g., network storage located close in network proximity to backup server 102). In various embodiments, the data to be backed up and/or restored data is stored in one or more of the following: a storage included in a client (e.g., included in client 106/108), a storage attached to the client, a storage included in edge server 104, a storage attached to edge server 104, and/or a storage connected to the client via a network connection.
Because backup data is communicated between remote backup application/server 102 and a client (e.g., client 106/108) via a network, constraints of the network (e.g., bandwidth, latency, reliability, etc.) may hinder performance of a backup or backup restoration from a remote backup source. In some embodiments, backup application/server 102 profiles already stored backup content of a particular group to identify portions of the backups that appear frequently. For example, backups that are of backup sources that share a common characteristic (e.g., same operating system, same user, same network topology position, same associated application, same function, same device, etc.) are identified and chunked (e.g., divide backup data into consistently comparable chunks). The chunking may be also performed as a consequence of performing deduplication to locate data chunks that are duplicated in backup data. In some embodiments, by tallying a number of times a particular data portion (e.g., chunk) appears across the backups, popular data portions are identified for inclusion in a base backup reference data. If data to be backed up or restored includes a data portion (e.g., chunk) that is also included in the backup reference set, an identifier of the data portion(s) may be communicated over a network without their actual contents to save bandwidth and processing resources. In order for both communication parties to understand the data portion identifiers, both ends of the communication parties store the base backup reference data.
In the example shown in
Any of the components shown in
At 202, backups that share a common characteristic are received. In some embodiments, the backups include one or more backup instances of one or more backup sources. For example, backup data from a server, a user device, a virtual machine, a database and any system, computer or storage are received via a network to be stored. The backups may include a data image, a full system backup data, an incremental backup data, and/or a copy of a portion of data storage. In some embodiments, the common characteristic includes one or more of the following: an operating system type, an operating system provider, an operating system version, a user association, a user category, a device type, a user group, a data size, a storage type, an application, an application type, a network group, a geographical location, a virtualization characteristic, and any other characteristic associated with data, a device, a network and/or storage. For example, within a repository of received backups, backups of backup sources that are utilizing the same operating system type and version are located.
At 204, the backups that share the common characteristic are divided into data chunks. In some embodiments, dividing a backup into data chunks includes dividing data stored in the backup into data portions. For example, data included in a backup is divided (e.g., division is reproducible because data is divided along identified marker within data) into consistently comparable chunks. In some embodiments, the chunking may be also performed as a consequence of performing deduplication to locate data chunks that are duplicated in the backup. In some embodiments, each backup has been already divided into data chunks when the backup data is received. For example, client 106 of
At 206, the data chunks are sorted. In some embodiments, sorting the data chunks includes identifying the number of times a particular same chunk has been utilized. For example, a backup includes data portions that are repeated within the backup and a number of times each data chunk has been utilized in a single backup of the backups that share the common characteristic is determined. In another example, a number of times each same data chunk has been utilized across the backups that share the common characteristic is determined. In some embodiments, sorting the data chunks includes determining an identifier of the data chunks. For example, a hash of each of the data chunks is determined to allow comparison and indexing of the data chunks.
At 208, a portion of the data chunks are selected based at least in part on the sorting and the selected chunks are indexed. In some embodiments, selecting the data chunks includes selecting data chunks that will be included in a base backup reference chunk set for backups that share the common characteristic. For example, the base backup reference chunk set is known to both a sender and a receiver of backup data and for a data chunk that is in the base backup reference chunk set, an identifier of the data chunk can be sent without contents of the data chunk. The selected data chunks may be stored at a central backup server and its backup sources (e.g., clients) to enable efficient backup and restore. In some embodiments, selecting the data chunks includes selecting data chunks that are frequently accessed. For example, a result of the sorting in 206 indicates the number of times a data chunk has been included in one or more backups with the same common characteristic and data chunks with the highest utilization numbers are selected.
In some embodiments, selecting the data chunks includes selecting a certain percentage (e.g., percentage predetermined or dynamically determined) of data chunks that have been most frequently utilized in the backups that share the common characteristic. In some embodiments, selecting the data chunks includes selecting the most number of data chunks that is less than a maximum total data size (e.g., size predetermined or dynamically determined) and have been most frequently utilized in the backups that share the common characteristic. In some embodiments, selecting the data chunks includes selecting smaller total data size of (1) a certain percentage of data chunks that have been most frequently utilized in the backups that share the common characteristic or (2) the most number of data chunks that is less than a maximum total data size and have been most frequently utilized in the backups that share the common characteristic. In some embodiments, selecting the data chunks includes selecting for each backup in the backups that share the common characteristic a group of data chunks that have been frequently accessed and selecting data chunks that are common across all the groups of data chunks of the backups.
In some embodiments, indexing the selected data chunks includes determining an identifier for each of the selected data chunks. For example, a hash function is used to hash each selected data chunk. In some embodiments, indexing the selected data chunks includes storing the selected data chunks in a hash table.
At 210, a copy of the selected data chunks is stored at storage of a backup server. For example, the copy of the selected data chunk is stored at storage of backup server 102 of
At 212, a copy of the selected data chunks is stored at storage of one or more backup sources. For example, the copy of the selected data chunk is stored at storage of client 106 of
At 214, a copy of the selected data chunks is included in data that is to be utilized to initialize a new client that shares the common characteristic. For example, the copy of the selected data chunk is included in storage of client 106 of
At least a portion of the process of
At 302, data to be backed up is identified. For example, backup is performed periodically, dynamically, and/or at a predetermined interval (e.g., pursuant to a backup policy) and backup data that is to be provided to a remote backup server for backup is identified. In some embodiments, determining data to be backed up includes identifying data that has been modified since a previous backup.
At 304, the identified data to be backed up is divided into a group of data chunks. In some embodiments, dividing the identified data into data chunks includes dividing the identified data into data portions. For example, data included in the identified data is reliably/reproducibly divided (e.g., division reproducible because data is divided along identified marker within data) into consistently comparable chunks. In some embodiments, the chunking may be also performed as a consequence of performing deduplication.
At 306, a next data chunk is selected in the resulting group of data chunks. For example, each data chunk is individually processed and each data chunk from the group of data chunks is selected for processing.
At 308, it is determined whether the selected data chunk is included in a base backup reference chunk set. In some embodiments, the base backup reference chunk set has been received as a result of 212 and/or 214 of
If at 308 it is determined that the selected data chunk is included in the base backup reference chunk set, at 310 a copy of the selected data chunk is not provided to a remote backup storage. For example, an identifier (e.g., hash value) of the selected data chunk is provided via a network instead of the content of the selected data chunk to identify the data chunk to be backed up. In some embodiments, the recipient of the identifier of the selected data chunk may obtain contents of the selected data chunk using the received identifier to locate contents of the selected data chunk in the recipient's copy of the base backup reference chunk set. In some embodiments, the recipient of the identifier of the selected data chunk backs up an identifier of the selected backup rather than contents of the selected backup.
If at 308 it is determined that the selected data chunk is not included in the base backup reference chunk set, at 312 a copy of the selected data chunk is provided to the remote backup storage. For example, a copy of the selected data chunk is provided via a network to be stored at the backup storage.
At 314, it is determined whether any additional unprocessed data chunk exists in the group of data chunks of the identified data. If at 314 it is determined that an additional unprocessed data chunk exists, the process returns to 306 where the next data chunk to be processed is selected. If at 314 it is determined that an additional unprocessed data chunk does not exist, a backup is completed and the process ends.
At 402, data to be restored is identified. For example, a backup source (e.g., client) is to be restored using backed up data (e.g., restored from data backed up using at least a portion of the process of
At 404, data chunks of the identified data to be restored are identified. In some embodiments, a list that identifies data chunks included in the identified data to be restored is obtained. For example, a backup data source of the identified data to be restored has been chunked (e.g., using at least a portion of the process of
At 406, a next data chunk is selected in the data chunks of the identified data to be restored. For example, each data chunk is individually processed and each data chunk is selected for processing.
At 408, it is determined whether the selected data chunk is included in a base backup reference chunk set. In some embodiments, the base backup reference chunk set has been received as a result of 210 of
If at 408 it is determined that the selected data chunk is included in the base backup reference chunk set, at 410 a copy of the selected data chunk is not provided to a remote client to be restored. For example, an identifier (e.g., hash value) of the selected data chunk is provided via a network instead of the content of the selected data chunk to identify the data chunk to be utilized to restore data. In some embodiments, the recipient of the identifier of the selected data chunk may obtain contents of the selected data chunk using the received identifier and the recipient's copy of the base backup reference chunk set to utilize in the data restore.
If at 408 it is determined that the selected data chunk is not included in the base backup reference chunk set, at 412 a copy of the selected data chunk is provided to the remote client to be restored. For example, a copy of the selected data chunk is provided via a network to be utilized to restore data.
At 414, it is determined whether any additional unprocessed data chunk exists in the data chunks of the identified data to be restored. If at 414 it is determined that an additional unprocessed data chunk exists, the process returns to 406 where the next data chunk to be processed is selected. If at 414 it is determined that an additional unprocessed data chunk does not exist, the data to be utilized to restore the identified data has been all identified and/or provided and the process ends.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.
This application is a continuation of co-pending U.S. patent application Ser. No. 14/293,977 entitled CACHING OF BACKUP CHUNKS filed Jun. 2, 2014 which is incorporated herein by reference for all purposes.
Number | Name | Date | Kind |
---|---|---|---|
6128627 | Mattis | Oct 2000 | A |
6353878 | Dunham | Mar 2002 | B1 |
6904449 | Quinones | Jun 2005 | B1 |
7499958 | Ionescu | Mar 2009 | B1 |
8135676 | Poojary | Mar 2012 | B1 |
8504529 | Zheng | Aug 2013 | B1 |
8584145 | Havemose | Nov 2013 | B1 |
8943282 | Armangau | Jan 2015 | B1 |
9189341 | Natanzon | Nov 2015 | B1 |
20010034812 | Ignatius | Oct 2001 | A1 |
20040250033 | Prahlad | Dec 2004 | A1 |
20040267836 | Armangau | Dec 2004 | A1 |
20050015663 | Armangau | Jan 2005 | A1 |
20050065985 | Tummala | Mar 2005 | A1 |
20050182888 | Murotani | Aug 2005 | A1 |
20070073791 | Bruce | Mar 2007 | A1 |
20070101186 | Chen | May 2007 | A1 |
20070185936 | Derk | Aug 2007 | A1 |
20080091895 | Chen | Apr 2008 | A1 |
20080228850 | Samuels | Sep 2008 | A1 |
20080228933 | Plamondon | Sep 2008 | A1 |
20090113167 | Camble | Apr 2009 | A1 |
20100332456 | Prahlad | Dec 2010 | A1 |
20120005165 | Tsai | Jan 2012 | A1 |
20130232120 | Aronovich | Sep 2013 | A1 |
20130339298 | Muller | Dec 2013 | A1 |
20140052692 | Zhang | Feb 2014 | A1 |
20140250079 | Gardner | Sep 2014 | A1 |
20140250283 | Boullery | Sep 2014 | A1 |
20140258236 | Vijayan | Sep 2014 | A1 |
20150186519 | Popov | Jul 2015 | A1 |
20150242291 | Chang | Aug 2015 | A1 |
20160041906 | Mukherjee | Feb 2016 | A1 |
20170075768 | Oberhofer | Mar 2017 | A1 |
20170206145 | Panchanathan | Jul 2017 | A1 |
Number | Date | Country | |
---|---|---|---|
20170206145 A1 | Jul 2017 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 14293977 | Jun 2014 | US |
Child | 15475436 | US |