Patent Grant
11487430
The present invention relates in general to computing systems, and more particularly to, various embodiments for reducing data using a plurality of compression operations in a computing storage environment using a computing processor.
Data storage libraries provide a means of storing large amounts of data for host computer systems. Automated data storage libraries are known for providing cost effective storage and retrieval of large quantities of data, typically from data storage cartridges stored in storage slots of the library. The libraries typically comprise large numbers of data storage media that are stored in storage shelves, numbers of data storage drives to read and write data with respect to the data storage media, one or more robot accessors to access the data storage media to move the data storage media between the storage shelves and data storage drives, and control units to control the operation of the library and control the flow of data and information between the library and host computer systems. The data storage cartridges are typically extracted from the storage slots, placed in the storage slots, and transported within the library by one or more accessors.
Various embodiments are provided for reducing data using a plurality of compression operations in a computing storage environment by a processor. A speed of data writing to a virtual tape device and an availability of one or more processor devices for the virtual tape device may be monitored. One or more requests may be received for writing data to the virtual tape device. Data to be written to the virtual tape device, corresponding to a selected number of the one or more requests for writing the data, may be compressed according to both the speed of data writing to the virtual tape device and the availability of one or more processor devices for the virtual tape device. The compressed data may be stored in the virtual tape device in record units. After writing the data to the virtual tape device is completed, non-compressed data may be compressed in the virtual tape device at a subsequent period of time (e.g., future time period).
In order that the advantages of the invention will be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings. Understanding that these drawings depict embodiments of the invention and are not therefore to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:
Virtual tapes are typically emulated by a virtual tape library. A virtual tape library comprises a computing system executing a tape virtualization software. This tape virtualization software emulates virtual tape drives, virtual tape libraries and virtual tapes. A virtual tape library may be connected via an interface and a network—such as a Storage Area Network (SAN) or Local Area Network (LAN)—to the storage management application. The storage management application “sees” the virtual tape devices and virtual tapes just as real tape devices. When the storage management application writes a virtual tape, this virtual tape is stored on a disk system also comprised in the virtual tape library. Each virtual tape in a virtual tape library may have a unique serial number which is also called VOLSER. The VOLSER allows the unique identification of a virtual tape in a virtual tape library.
A virtual tape server (VTS) may virtualize a tape volume, and the tape volume on the VTS handled by a host application may be present in the VTS as a logical volume (also referred to as LVOL, a logical tape volume, or a virtual tape volume). The logical volume may be present in a disk device, under the control of the VTS, called a cache serving as a virtual storage region or in a physical tape library also under the control of the VTS. The logical volume present in the cache may be transferred (hereinafter referred to as “migrate” or “migration”) to a physical tape volume (hereinafter referred to as a physical volume) in the physical tape library if the LVOL is not referred to for a long time or in accordance with a storage management policy.
In one aspect, virtual tape servers (“VTS”), such as the IBM® TS7700, may include a disk cache subsystem, which may be installed with a variety of different physical disk drive media (DDM). This encompasses different physical drive types, capacities, manufacturers, rotational speeds, etc.
Additionally, the VTS may read data from and/or write data to a storage device such as, for example, a hard disk drive (HDD), which may be in a format of a virtual tape device. The virtual tape device may emulate a physical tape. The virtual tape device (such as a virtual tape device of the IBM TS7700) may have, for example, a format as illustrated in
In one aspect, has used herein, the virtual tape device (which may be a virtual tape system “VTS”) may control a virtual tape device by using processor devices (central processing unit “CPU”) resources of the VTS. A host application may write a virtual tape file by using virtual tape device that the VTS provides.
It is important for the virtual tape apparatus that the capacity of a storage device such as, for example, an HDD to which the virtual tape apparatus is written be used efficiently. Thus, in one embodiment, data may be compressed in real time when the data is written into a virtual tape apparatus to reduce the bandwidth necessary for writing to the storage device (e.g., a virtual tape device/apparatus) and the capacity of the storage necessary for saving. When data is compressed, for example, the virtual tape device of a VTS such as, for example, the IBM® TS7700 may have a format as depicted in
Since data is compressed in record units, data can be decompressed and read in record units. That is, data may be compressed record-by-record, so the data can be decompressed and read in record-by-record. When data is compressed by software in real time, the speed of compression processing may be a bottleneck of the data writing speed. Furthermore, since a higher compression algorithm tends to use more processing devices (e.g., central processing unit “CPU” resources), particularly when a large volume of data is written in parallel, compression processing by an application may negatively affect performance of the entire system. In one aspect, dedicated hardware may be used to perform real-time compression without affecting the data writing speed or CPU resources of the entire system, which may create computing inefficiency since the cost is higher as compared to an operation that provides compression by application (e.g., compression by software). For this reason, real-time data compression by application can also preferably reduce the bandwidth necessary for writing data and the capacity of the storage necessary for saving the data without having effects such as, for example, a reduction in the data writing speed or lack of CPU resources on the system.
Accordingly, various embodiments are provided herein for reducing data using a plurality of compression operations in a computing storage environment by a processor. A speed of data writing to a virtual tape device and an availability of one or more processor devices for the virtual tape device may be monitored. One or more requests may be received for writing data to the virtual tape device. Data to be written to the virtual tape device, corresponding to a selected number of the one or more requests for writing the data, may be compressed according to both the speed of data writing to the virtual tape device and the availability of one or more processor devices for the virtual tape device. The compressed data may be stored in the virtual tape device in record units. After writing the data to the virtual tape device is completed, non-compressed data may be compressed in the virtual tape device at a subsequent period of time (e.g., future time period).
In one aspect, when data is written to the virtual tape device, the data may be “compressed by software” in real time for a selected number of records (e.g., every few records) depending on the speed of writing data to a storage device (e.g., the virtual tape device from a host perspective or a disk cache of the VTS) and the availability of CPU resources. It is thereby possible to reduce the bandwidth used for writing to the storage device and the amount of data saved without having negative effects such as, for example, a reduction in the data writing speed and the lack of CPU resources on the system. Furthermore, records in the virtual tape device, which have not been compressed during the data writing, may be compressed after writing the data to the virtual tape device is completed and when the amount of Input/Outputs (“I/O”) to and/or from the storage device is less than a defined amount, a threshold, or small size, that is, and when there are sufficient CPU resources. In one aspect, a “small” size may be a size below a defined threshold. For example, if a user has priority to writing new data from a host, the threshold may be relatively low (e.g., more bandwidth should be used for writing new data from the host). If a user has priority to reducing data size on a storage device of VTS (disk cache), the threshold may be relatively high (e.g., more bandwidth may be used for reading already existing non-compressed data and writing the data compressed subsequently). In one aspect, the threshold may be an amount of read/write rate that is lower than 20% of a maximum read/write rate of the system.
Each of the records in the virtual tape device may be compressed (e.g., eventually compressed) and only those records, having the same capacity as that in the case where all the records are compressed at the time of writing, may be saved in the storage device. Since data in the virtual tape device is read and/or written in record units, corresponding compression/decompression of data may also be performed in record units. Therefore, even when compressed and non-compressed records coexist in the virtual tape device, there are no challenges or problems accessing a virtual tape device in which compressed and non-compressed records coexist. During read operations of the data, only the necessary records need to be decompressed (as compared with each of the records), and when records are subsequently compressed, only non-compressed records may be compressed.
When data is written to virtual tape devices, the present invention compresses the data every few records by software in real time depending on how fast write speed is and how much CPU resource is idle. The present invention prevents real time software compression from having a negative influence (e.g., a degradation of write speed or lack of CPU resource to the system) and may also reduce bandwidth used for writing data to the storage device and amount of data written to the storage device to a certain degree. In addition, when there are only a few I/Os (e.g., less than 10 I/Os) and one or more CPUs are available (e.g., CPUs that are idle) after data has been written to virtual tape devices, mechanisms of the present invention may compress non-compressed records which were skipped and not compressed when these non-compressed records were originally written to the virtual tape devices. This results in all records in the virtual tape devices to become compressed and the amount of data stored in the virtual tape devices is similar to the scenario when all records written to the virtual tape devices are compressed in real time. Data in a virtual tape device is read and/or written in record-by-record so data may also be compressed and/or decompressed in record-by-record. Therefore, any challenges are eliminated relating to reading data from a virtual tape device that include compressed records and non-compressed records that co-exist in a virtual tape device. Thus, the present invention enables the decompression and reading of a record from a virtual tape device and enables the compression of a non-compressed record in the virtual tape device at a subsequent period of time.
As a preliminary matter,
The tape library 102 may include a library manager 110, one or more data drive devices, which may be tape cartridges 112 (shown as 112A-E), an accessor 114, and a plurality of mountable media 116. In one embodiment, the mountable media 116 includes tape cartridges, magnetic disks, optical disks, CDs, DVDs, other devices that can store data and be mounted to a drive unit, and the like. The library manager 110, which includes at least one computing processor, may be interconnected with and may control the actions of the tape cartridges 112 and the accessor 114. The library manager 110 may include one or more hard disk drives (not shown) for memory storage, as well as, a control panel or keyboard (not shown) to provide user input. The control panel may be a computer in communication with the library manager 110 so that a user can control the operating parameters of the automated tape library 102 independently of one or more hosts 106.
In
The interconnections between the library manager 110, the tape cartridges 112, and the accessor 114 are shown as dashed lines to indicate that the library manager 110 transmits and receives control signals, rather than data to be stored or retrieved, to the tape cartridges 112 and/or the accessor 114. Data for storage or retrieval may instead be transmitted directly between the VTS 104 and the tape cartridges 112 via a network 118, which may be a storage area network (SAN), local area network (LAN), wide area network (WAN), or another suitable type of network, including the Internet or a direct connection between the VTS 104 and the tape cartridges 112 via a point to point or multi-drop bus connection, for example, a Small Computer Storage Interface (SCSI). Alternatively, control signals for tape drives 112 can be transmitted and received through connections between the VTS 104 and the library manager 110 and the VTS 104 and the tape drives 112 via network 118.
The accessor 114 may be a robotic arm or another mechanical device configured to transport a selected mountable media 116 between a storage bin and tape cartridges 112. The accessor 114 typically includes a gripper and a bar code scanner, or a, similar read system, mounted on the gripper. The bar code scanner is used to read a volume serial number (VOLSER) printed on a cartridge label affixed to the tape cartridges 112. In alternative embodiments, the tape cartridges 112 may be replaced by optical disk drives or other magnetic drives. Similarly, the mountable media 116 and the tape drives 113 may include magnetic media, optical media, or any other removable media corresponding to the type of drive employed. A control console 120 may be connected to the library manager 110. The control console 120 may be a computer in communication with the library manager 110 so that a user can control the operating parameters of the tape library 102 independently of the host 106.
As shown, the VTS 104 may include a plurality of virtual tape drives 200, a file system manager 202, an automated storage manager 206, a queue 208, and at least one direct access storage device (DASD) cache 210. The DASD cache 210 may take the form of one or more virtual tape drives that contain data in the form of logical, or virtual, volume and may temporarily store data from the host 106 on virtual or logical volumes in the form of files. A write command from the host 106 is processed by the ITS 104, through a virtual tape drive 200 into the DASD cache 210, prior to transferring the updated logical volume from the DASD cache 210 to the mountable media or physical volume 116 (
The file system manager 202 manages and coordinates data storage in the DASD cache 210. The automated storage manager 206 controls the interface communications between the file system manager 202 and the tape cartridges 112. The automated storage manager 206 also controls communications between the VTS 104 and the library manager 110. In one embodiment, the host 106 may request a particular logical volume. The automated storage manager 206 determines whether the logical volume is in the DASD cache 210. If it is not, the automated storage manager 206 requests a recall for it from the physical volume or mountable media 116. The automated storage manager 206 may also contain a queue 208 for temporarily placing additional recall requests to be processed. Thus, the automated storage manager 206 is an apparatus for recalling logical volumes from mountable media 116 by means of the tape cartridges 112A, B, C, D, and E (
The library manager 110 manages the virtual and physical volumes as well as the constructs. More specifically, the library manager 110 includes the command processor 225 that receives control commands from the virtual tape drives 200 and the automated storage manager 206. The command processor 225 passes instructions about the management of the virtual and physical volumes to the volume manager 235. The volume manager 235 stores information about the virtual and physical volumes on a database 230 of the library manager 110. In addition, depending on the instructions received, the volume manager 235 sends instructions (e.g., via a network 118) to the tape cartridges 112 and/or the accessor 114 to load or “mount” the cartridges or other mountable media 116 on which copies of the virtual volumes are to be made or retrieved. Mounting of multiple cartridges 112 may be generally simultaneous or in a certain order, depending on the configuration of the accessor 114 and the tape cartridges 112.
The library manager 110 also has a construct manager 240 that receives user instructions from the control console 120 regarding the volume management actions to be followed for a given construct name. The volume management actions are stored and retrieved by the construct manager 240 on a database 230 of the library manager 110. For certain control commands received by the command processor 225, the command processor 225 instructs the construct manager 240 to provide the volume management actions for a specific virtual volume. The command processor 225 then passes the returned volume management actions for a specific virtual volume to the automated storage manager 206.
Having previously described
That is,
The functionality 600 may start in block 602. A determination may be performed to determine “X” number of records is equal to at least 1, as in block 604. In block 606, a sleep operation may be performed at time interval “T.” A current writing speed W and availability I of CPU resources from the system may be acquired, as in block 608. A determination operation may be performed to determine whether the current writing speed W is greater than the acceptable writing speed “Wmin,” as in block 610. If no, the functionality 600 may move to block 618. If yes, a determination operation may be performed to determine whether the availability “I” of CPU resources is greater than the user acceptable availability of CPU resources “Imin,” as in block 612. If no, the functionality 600 may move to block 618. If yes, a determination operation may be performed to determine whether X number of records is greater than 1, as in block 614. If no, the functionality may return to block 606.
If yes, X number of records may be decreased (“−−”), as in block 616. In block 618, the X number of records may be increased (“++”). That is, as pertaining to blocks 616 and 618, data may be compressed every X records meaning that the smaller the X value is, the more frequently a compression operation is performed. The smallest X value may be represented as 1 (compress every record). A compression operation may use CPU resources and require a certain amount of time. Thus, as more frequent compression operations are performed the more negative is the effect to the writing speed and room of CPU resources (e.g., speed W and availability “I” of CPU resources values decrease). Wmin and Imin values mean the amount of negative effect by compression is allowed by the user during the writing of data operation from the host. The smaller the Wmin and Imin are, the more negative effect the user allows. When the writing speed W is allowed to decrease (e.g., a “W>Wmin” condition is achieved) and the use of more CPU resources are allowed (e.g., a “I>Imin” condition is achieved), the frequency of compression operation increases (e.g., X number of records may be decreased “X−−” of step 616). When the user is not allowed to use more CPU resources and the writing speed W is not allowed to decrease, the frequency of compression operation decreases (e.g., X number of records may be increased “X++” of step 618). The functionality 600 may return to block 606 following blocks 616 and 618.
Upon performing the mechanisms of the illustrated embodiments, compressed and non-compressed records may coexist in the virtual tape device where data is written as shown in
Additionally, records in a virtual tape device, which have not been compressed during data writing, may be compressed after data writing to the virtual tape device is completed and when the amount of I/O to and/or from the storage device is small (e.g., less than a defined threshold or value) and there are sufficient CPU resources (e.g., at least one processor device). Thus, each of the records in the virtual tape device may eventually be compressed (e.g., compressing records in the virtual tape device at different time intervals) and only the records having the same capacity such as, for example, the situation where all the records are compressed during writing are saved in the storage device. However, the virtual tape device may be one file. Therefore, even when records are compressed and overwritten at a later period of time, the compressed records may be saved in the form depicted in
For example,
In
Accordingly, the present invention may reduce the bandwidth used for writing data and reduce the required or necessary capacity of the storage device by compressing data to a certain degree (e.g., compressing only selected data) in real time during data writing without suffering the negative effects that may impact the performance of the system. Furthermore, post-processing of data in a record, after data writing to the virtual tape device, allows capacity used by the storage device to eventually save the data to be equivalent to that when all the records in the virtual tape device are compressed.
Turning now to
In one aspect, in conjunction with and/or as part of at least one block of
The operations of 1000 may detect non-compressed data that is stored in a record of the virtual tape device, compress the detected non-compressed data according to both the speed of data writing to the virtual tape device and the availability of one or more processor devices for the virtual tape device, and/or create a hole in a record by compressing the detected non-compressed data and converting the virtual tape device to a sparse file.
The operations of 1000 may compress non-compressed data upon completing the writing the data to the virtual tape device, upon determining an amount of Input/Output (I/O) operations to and from the virtual tape device is below a threshold value, according to the availability of one or more processor devices for the virtual tape device, or a combination thereof, and/or compress data corresponding to the selected number of the one or more requests for each N records, wherein N is a positive integer or a selected value.
The operations of 1000 may receive one or more requests for reading data from the virtual tape device; determine whether the data is compressed and stored in records corresponding to the one or more requests for reading data; decompress the compressed data in the records for the data that is determined to be compressed and stored; and/or send the decompressed data to a requester.
The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowcharts and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowcharts and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowcharts and/or block diagram block or blocks.
The flowcharts and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5761536 | Franaszek | Jun 1998 | A |
| 8054879 | Fallon et al. | Nov 2011 | B2 |
| 8806062 | Vertongen | Aug 2014 | B1 |
| 8836548 | Chandra | Sep 2014 | B1 |
| 9582207 | McCloskey | Feb 2017 | B1 |
| 20150046398 | Camble | Feb 2015 | A1 |
| 20150242309 | Talagala | Aug 2015 | A1 |
| 20170123704 | Sharma | May 2017 | A1 |
| 20170199707 | Varghese | Jul 2017 | A1 |
| 20180074723 | Delaney | Mar 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 2006090412 | Aug 2006 | WO |
| Entry |
|---|
| https://whatis.techtarget.com/definition/streaming-media taken by archive.org on Dec. 24, 2017 that describes the term “stream” (Year: 2017). |
| An article “video Streaming: Concepts, Algorithms, and Systems by John G Apostolopoulos”, Sep. 18, 2002 that clarifies that audio video streams are characterized by compression at the source , before being sent over the network. (Year: 2002). |
| Fischer, “NetApp offers virtual tape library with compression,” Computerworld, Oct. 17, 2006 (3 pages). |
| Number | Date | Country | |
|---|---|---|---|
| 20190286333 A1 | Sep 2019 | US |
