The present disclosure generally relates to systems, methods, and devices for compressing images.
Image compression reduces the cost of storage and/or transmission of digital images.
So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be had by reference to aspects of some illustrative implementations, some of which are shown in the accompanying drawings.
In accordance with common practice the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method or device. Finally, like reference numerals may be used to denote like features throughout the specification and figures.
Various implementations disclosed herein include devices, systems, and methods for compressing an image. In various implementations, the method is performed by a device including one or more processors and non-transitory memory. The method includes obtaining an uncompressed image. The method includes compressing the uncompressed image to generate a compressed image by mapping a substantially circular portion of the uncompressed image to a substantially square portion of the compressed image, wherein a central region of the substantially circular portion of the uncompressed image is compressed more than a remainder of the substantially circular portion of the uncompressed image.
In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors. The one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes: one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.
In various implementations, an image captured with an image sensor having a fisheye lens includes information only in a substantially circular region of the image. Such images may be referred to as circular images. Further, in various implementations, the radial resolution of circular images drops off as a function of distance from the center of the image.
Image compression can reduce the cost of storage and/or transmission of such images. In various implementations, image compression can be lossless or lossy. Lossy image compression seeks to substantially reduce the size of the image while minimizing loss of visual fidelity. To that end, in various implementations, a substantially circular portion of an uncompressed image is mapped to a substantially square portion of an uncompressed image such that more pixels of the compressed image encode information. Further, a central region with high radial resolution is compressed more (e.g., downsampled more) than other regions with lower radial resolution. Lowering the resolution of the central region with high radial resolution introduces less visual difference than lowering the resolution of the other regions with lower resolution.
In various implementations, to compress the uncompressed image 100, the image 100 is uniformly downsampled.
In various implementations, to compress the uncompressed image 100 and reduce the number of pixels having pixel values that do not encode information, a circle-to-square mapping is used to map the substantially circular region 101 of the uncompressed image 100 to a substantially square region of a compressed image. Various circle-to-square mappings may be used including radial stretching, Shirley-Chiu mapping, Fernandez-Guasti's squircle mapping, elliptical arc mapping, or conformal mapping.
Assuming an image is M×N, the normalized coordinates of a pixel in the image at a point (m, n) are (2m/M−1, −2n/N−1). Thus, the image spans from −1 to 1 both horizontally and vertically. Further, the radius of a pixel is the distance from the center of the image in the normalized coordinate system to the location of the pixel, or
The normalized radius of a pixel in the uncompressed image 100 is the radius of the pixel divided by the length of a line segment from the center of the substantially circular region 101 to its edge passing through the pixel. If the substantially circular region 101 is perfectly circular, the length of this line segment is invariably 1 and the normalized radius is equal to the radius. However, in various implementations, the substantially circular region is truncated (e.g., in
To map a circle to a square using uniform radial stretching, the radius of each point in the circle is stretched (or scaled), based on its angle, in proportion to the distance at that angle from the center of the square to its edge. Thus, the normalized radius of a point in the compressed image is the same as the normalized radius of the point in the uncompressed image 100.
For example, where u and v are the normalized coordinates of the circle, x and y are the normalized coordinates of the square, and r=√{square root over (u2+v2)},
In various implementations, the radial resolution of an image captured using an image sensor having a fisheye lens is highest at the center of the image and decreases as the distance from the center increases.
Thus, the center of the uncompressed image 100 encodes much more information than the outer region. Compressing the uncompressed image 100 using uniform radial stretching decreases the radial resolution uniformly.
Thus, information at the outer region of the uncompressed image 100, already scant as compared to the center, is discarded in the uniformly stretched square compressed image. Further, in various implementations, a display may be limited as to the radial resolution that can be displayed. For example, in various implementations, a display can display up to 40 PPD. Thus, the information at the center of the uniformly stretched square compressed image 220 having a PPD greater than 45 PPD is undisplayable.
Thus, in various implementations, a nonuniform circle-to-square mapping is used in which a central region 102 of the substantially circular region 101 is compressed more than an outer region 103 including the remainder of the substantially circular region 101.
For example, where u and v are the normalized coordinates of the circle, x and y are the normalized coordinates of the square, r=√{square root over (u2+v2)}, and ƒ(r) is a nonuniform stretching function,
In various implementations, the nonuniform stretching function is convex.
Through nonuniform stretching, the radial resolution of the central region can be capped (e.g., to a maximum display resolution) while maintaining the radial resolution of the outer region.
Because of the radial stretching, there are more pixels (and more information) along the diagonals than the equator and meridian in the nonuniformly stretched square compressed image 230. Accordingly, in various implementations, the substantially circular region 101 of the uncompressed image is rotated −45 degrees (or 45 degrees, 135 degrees, or −135 degrees) or before the mapping is applied to maintain as much information along the equator and meridian of the uncompressed image 100. In various implementations, the rotation is integrated into the mapping to form a rotated nonuniform circle-to-square mapping.
Thus, point A in the uncompressed image 100, having a radius of 0, is represented by point A in the rotated nonuniformly stretched square compressed image 240 having a radius of 0. Point B in the uncompressed image 100, having a radius of approximately 0.5 and an angle of 0 degrees, is represented by point B in the rotated nonuniformly stretched square compressed image 240, having a radius of approximately (√2)/4 and an angle of −45 degrees. Point C in the uncompressed image 100, having a radius of approximately 1 and an angle of 0 degrees, is represented by point C in the rotated nonuniformly stretched square compressed image 240, having a radius of approximately √2 and an angle of −45 degrees. Point D in the uncompressed image 100, having a radius of approximately 1 and an angle of 90 degrees, is represented by point D in the rotated nonuniformly stretched square compressed image 240, having a radius of approximately √2 and an angle of 45 degrees. Point E in the uncompressed image 100, having a radius of approximately 1 and an angle of 45 degrees, is represented by point E in the rotated nonuniformly stretched square compressed image 240, having a radius of approximately 1 and an angle of 0 degrees.
In various implementations, a compressed image is provided to a player which decompresses the compressed image and displays the decompressed image. In various implementations, with the compressed image, the player is also provided a decompression mapping which used by the player to decompress the compressed image. In various implementations, the decompression making is provided to the player as a set of vertices on the unit sphere and corresponding pixel locations, forming a triangular mesh. Thus, in various implementations, the decompression mapping is a piecewise affine mapping, as an affine map is defined by three point pair correspondences. Accordingly, in various implementations, the compression mapping is also a piecewise affine map.
The circle-to-square mappings described above are neither affine nor piecewise affine. Accordingly, in various implementations, a piecewise affine mapping is defined in which each of a plurality of vertices in the uncompressed image 101 are mapped to corresponding vertices in the compressed image using one of the circle-to-square mappings described above. In various implementations, the vertices in the uncompressed image 101 is a substantially regular polygon mesh, such as the icosphere. Further, pixels of the uncompressed image 101 within a triangle defined by three vertices are mapped to corresponding pixel locations in the compressed image using the affine mapping defined by the three vertices. The corresponding vertices in the compressed image are provided to the player as part of the decompression mapping.
The method 500 begins, in block 510, with the device obtaining an uncompressed image. In various implementations, obtaining the uncompressed image includes capturing, using an image sensor having a fisheye lens, an image of a physical environment. In various implementations, the uncompressed image is a rectangular matrix of pixels having respective pixel values. In various implementations, the uncompressed image includes a substantially circular portion including pixels having pixel values that encode information and the remainder of the pixels have pixel values that do not encode information. Although
The method 500 continues, in block 520, with the device compressing the uncompressed image to generate a compressed image by mapping a substantially circular portion of the uncompressed image to a substantially square portion of the compressed image. The compressed image is a matrix of pixels having respective pixel values. In various implementations, the compressed image is a square matrix of pixels. In various implementations, the compressed image includes fewer pixels than the uncompressed image. Although
In various implementations, a central region of the substantially circular portion of the uncompressed image is compressed more than a remainder of the substantially circular portion of the uncompressed image. For example, in
In various implementations, mapping the substantially circular portion of the uncompressed image to a substantially square portion of the compressed image includes nonuniform radial stretching. Thus, in various implementations, a normalized radius of a particular pixel in the central region of the uncompressed image is greater than a normalized radius of a corresponding pixel in the compressed image. Similarly, in various implementations, a normalized radius of a particular pixel in the remainder of the substantially circular portion of the image is less than a normalized radius of a corresponding pixel in the compressed image.
In various implementations, the radial resolution of the uncompressed image decreases as a function of eccentricity. Thus, in various implementations, the radial resolution of the uncompressed image decreases from a maximum at the center of the uncompressed image as the normalized radius increases. In various implementations, each point in the compressed image has a radial resolution less than the maximum radial resolution of the uncompressed image. In various implementations, the maximum radial resolution of the compressed image is less than or equal to a display threshold, e.g., the maximum radial resolution displayable by a particular display. In various implementations, the radial resolution of the compressed image is substantially constant in a central region of the compressed image corresponding the central region of the substantially circular portion of the uncompressed image.
In various implementations, the horizontal radial resolution of the compressed image differs from the vertical radial resolution of the compressed image. For example, in various implementations, the vertical radial resolution (in the central region of the compressed image) is approximately 38 PPD and the horizontal radial resolution (in the central region of the compressed image) is approximately 36 PPD.
In various implementations, mapping the substantially circular portion of the uncompressed image to a substantially square portion of the compressed image includes mapping a plurality of vertices of the uncompressed image to corresponding vertices of the compressed image. Further, in various implementations, pixels of the uncompressed image within a triangle defined by three vertices are mapped to corresponding pixel locations in the compressed image using the affine mapping defined by the three vertices. Thus, in various implementations, mapping the substantially circular portion of the uncompressed image to a substantially square portion of the compressed image includes performing a piecewise affine mapping.
In various implementations, mapping the substantially circular portion of the uncompressed image to a substantially square portion of the compressed image includes additional mappings. For example, in various implementations, the uncompressed image is rotated prior to compression to increase the sampling along the equator and prime meridian. In various implementations, the rotation is incorporated into the mapping. Thus, in various implementations, in the compressed image, the substantially circular portion of the uncompressed image is rotated approximately 45 degrees, 135 degrees, 225 degrees, or 315 degrees.
In various implementations, the mapping includes image rectification (e.g., stereo image rectification). In various implementations, a camera includes two or more image sensors which generate a set of two or more images. Due to camera misalignment, lens distortion, or other factors, the images may be vertically offset. Image rectification projects the images onto a common image plane in such a way that the corresponding points in the images have the same row coordinates. Image rectification is a projection (or mapping) that corrects the images to appear as if the two or more image sensors are parallel. It may be particularly beneficial to perform image rectification before compression to increase the efficiency of the encoding stage with an encoding that uses interframe similarity, e.g., block matching, motion vector prediction and bundling, etc.
In various implementations, the mapping includes lens distortion correction, perspective correction, or any other projection or mapping.
While various aspects of implementations within the scope of the appended claims are described above, it should be apparent that the various features of implementations described above may be embodied in a wide variety of forms and that any specific structure and/or function described above is merely illustrative. Based on the present disclosure one skilled in the art should appreciate that an aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method may be practiced using any number of the aspects set forth herein. In addition, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to or other than one or more of the aspects set forth herein.
It will also be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first node could be termed a second node, and, similarly, a second node could be termed a first node, which changing the meaning of the description, so long as all occurrences of the “first node” are renamed consistently and all occurrences of the “second node” are renamed consistently. The first node and the second node are both nodes, but they are not the same node.
The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the claims. As used in the description of the implementations and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As used herein, the term “if” may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” may be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.
This application claims priority to U.S. Provisional Patent App. No. 63/403,943, filed on Sep. 6, 2022, which is hereby incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
63403943 | Sep 2022 | US |