Patent Application
20220138917
In general, the current invention relates to apparatus, system and methods for image distortion correction, and specifically adapted for improved scanning/imaging an object profile in an air-water interface (AWI) zone, or any such interfaces between two media of different refractive indices including air and glass, oil and water, among others.
Current technologies for both above-water and underwater scanning have developed to the level where high-density point cloud scans can reach accuracies much finer than the required 1.0 mm. Commercially available above-water laser scanners can reach accuracies of 0.025 mm, while underwater counterparts can reach 0.1 mm. However, through-water scanning remains difficult even at extremely shallow water depths no matter the type of scanner used, which makes creating accurate, high-density point clouds of the Air-Water Interface (AWI) area difficult in even the most optimal conditions.
The challenge of through-water scanning is a physics problem: reflection and refraction. When electromagnetic radiation crosses a smooth interface into a dielectric medium that has a higher refractive index (nt>ni), two phenomenon can occur, that is reflection and refraction. The angle of reflection, Or, equals the angle of incidence, θi, where each is defined with respect to the surface normal. The angle of refraction, θt, (t for transmitted) is described by Snell's law (of Refraction): ni sin θi=nt sin θt.
It is a known phenomenon that when light traveling one transparent medium encounters a boundary with a second transparent medium (e.g., air and glass), a portion of the light is reflected and a portion is transmitted into the second medium. As the transmitted light moves into the second medium, it changes its direction of travel; that is, it is refracted. The law of refraction, Snell's law above, describes the relationship between the angle of incidence (θ1) and the angle of refraction (θ2), measured with respect to the normal (“perpendicular line”) to the surface; in mathematical terms: N1 sin θ1=n2 sin θ2, where n1 and n2 are the index of refraction of the first and second media, respectively. The index of refraction for any medium is a dimensionless constant equal to the ratio of the speed of light in a vacuum to its speed in that medium.
The amount of bending of a light ray as it crosses a boundary between two media is dictated by the difference in the two indices of refraction. When light passes into a denser medium, the ray is bent toward the normal. Conversely, light emerging obliquely from a denser medium is bent away from the normal. In the special case where the incident beam is perpendicular to the boundary (that is, equal to the normal), there is no change in the direction of the light as it enters the second medium.
When scanning an interface between two media, since the electromagnetic wave such as laser, or even light will have to traverse two media, refraction will occur and thus cause a distortion of the scan/image result. Further, there are likely further distortions caused by another characteristic of liquid medium, which is waves, and such waves only escalate challenge of imaging at an air-water interface or such media, because the waves cause refraction of light in many different directions. From ripples on a pond to deep ocean swells, sound waves, and light, all waves share some basic characteristics. Broadly speaking, a wave is a disturbance that propagates through space. Most waves move through a supporting medium, with the disturbance being a physical displacement of the medium. The time dependence of the displacement at any single point in space is often an oscillation about some equilibrium position. For example, a sound wave travels through the medium of air, and the disturbance is a small collective displacement of air molecules individual molecules oscillate back and forth as the wave passes.
Unlike particles, which have well-defined positions and trajectories, waves are not localized in space. Rather, waves fill regions of space, and their evolution in time are not described by simple trajectories. This is essentially problematic since the distortion of imaging through an interface characterized by such waves would be defined by complex geometry rather than simple mathematical relationships. It is also important to consider one defining characteristic of all waves, which is superposition, which describes the behaviour of overlapping waves. The superposition principle states that when two or more waves overlap in space, the resultant disturbance is equal to the algebraic sum of the individual disturbances. It is thus important for any solution to factor in for these possibilities in order to provide a workable design for through-water scanning.
On the other hand, machines can be taught to interpret images the same way human brains do and to analyze those images much more thoroughly than we can. For example, when applied to image processing, artificial intelligence (AI) can power face recognition and authentication functionality for ensuring security in public places, detecting and recognizing objects and patterns in images and videos, image correction applications, and so on.
Image enhancement is the process of improving the picture quality without any information loss so that the results are more suitable for display (desired resolution, color, and style), or prepare images for further analysis in various computer vision applications, including object detection, image classification, scene understanding, and much more. Image enhancement usually consists of several transformations like image denoising, deblurring, up-scaling, contrast enhancement, lighting up low-light pictures, removing optical distortion, etc. Image post-processing has always been an essential part of the whole photography process, and it is required to address common photographic flaws, and is done by image enhancement algorithms.
Deep learning (DL), on the other hand, is a relatively new field of machine learning (ML), and it can be effectively applied to image processing. Different types of neural networks can be utilized for solving different image enhancement tasks, for example, successful denoising, producing high-resolution images from low-resolution images by training super-resolution, and much more.
As with all inventions that are based off the necessity to improve prior art, the current invention has identified a gap in the prior art, in that there is not a reliable system and method for accurate scanning/imaging an object profile in an air-water interface (AWI).
This disclosure presents an apparatus, system and methods for image distortion correction for accurate scanning/imaging an object profile in an air-water interface (AWI).
The following summary is an explanation of some of the general inventive steps for the system, method, architecture and tools in the description. This summary is not an extensive overview of the invention and does not intend to limit the scope beyond what is described and claimed as a summary.
The present invention, in some embodiments thereof, relates to apparatus, system and methods for image distortion correction when scanning/imaging an air-water interface (AWI), or any such interfaces between two media including air and glass, among others. According to one embodiment, the apparatus comprises of a means of scanning a mean water level, and two scanners, wherein is set slightly above the water surface, and one positioned just below, with the scanners having an intersecting view of the AWI. A suitably trained machine learning algorithm recognizes key features from both the above-water and underwater scans, determines distortion from the AWI, make a correction of the distortion and automatically stitch the distortion-corrected scans together. According to another embodiment, the resulting single complete, accurate, and high-density point cloud of all surface profiles in, around, and below the AWI area.
The novel features believed to be characteristic of the illustrative embodiments are set forth in the appended claims. The illustrative embodiments, however, as well as a preferred mode of use, further objectives and descriptions thereof, will best be understood by reference to the following detailed description of one or more illustrative embodiments of the present disclosure when read in conjunction with the accompanying drawings, wherein:
Hereinafter, the preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. The terminologies or words used in the description and the claims of the present invention should not be interpreted as being limited merely to their common and dictionary meanings. On the contrary, they should be interpreted based on the meanings and concepts of the invention in keeping with the scope of the invention based on the principle that the inventor(s) can appropriately define the terms in order to describe the invention in the best way.
It is to be understood that the form of the invention shown and described herein is to be taken as a preferred embodiment of the present invention, so it does not express the technical spirit and scope of this invention. Accordingly, it should be understood that various changes and modifications may be made to the invention without departing from the spirit and scope thereof.
In this disclosure, the terms imaging apparatus and scanner may be used interchangeably, and will generally be directed at any such equipment capable of using electromagnetic, sonar or optical means to obtain the surface profile of an object.
Further in this disclosure, the terms scans, images or point clouds may be used interchangeably and will generally be directed at any such obtained surface profiles of an object by means of an imaging apparatus or scanner.
Still, the term AWI will be used to mean an air-water interface, however this is representative of all types of interfaces between media of different refraction index.
The term MWL may be used in this disclosure to mean the mean water level at an interface between air and water, however this is representative of all types of interfaces between media of different refraction index.
In the first embodiment according to
Further, and according to the same
According to one embodiment, the arrangement works like so, the imaging apparatus 3 and 4 will take a plurality of images, both above and below the interface and transmit them to the compute resource 7 via the transmission means 6. The compute resource 7 comprises a suitable computer program product on its memory adapted to determine the location of the interface at any point in time for images taken at the same time from above and below. The computer program then stitches together the images into a single surface profile image/data point cloud/etc. based on the surface profile for interface location. It is anticipated that the imaging apparatus is comprised of photography equipment, sonar equipment, optical, ultrasound equipment, radiography equipment, or any such equipment capable of imaging or scanning a surface using electromagnetic means.
In an embodiment exemplified by the accompanying
The primary scanning units 3 and 4 are of the current invention preferable to be integrated platforms with all components housed within a single portable IP68 waterproof chassis. With a powerful wireless communication means and waterproofed internal components, the platform 1 is anticipated to be a cordless device. The components selected will have underwater applications in mind and be optimized for use in hydraulic laboratory conditions. However, it is also anticipated that a corded application would also work well in solving the problem.
It is anticipated that the applicable range of height adjustable mechanisms encompasses three lift systems: electric, manual and brackets. Generally, and in accordance with the current invention, the adjustable mechanisms, or re-configurable or programmable mechanisms, are mechanisms in which one or more of their parameters are made adjustable. The height adjustment of the platform allows the changing of the field of view of the imaging apparatus, or focus on a different position of the artifact at the air-water interface. It is further anticipated that the types of adjustment mechanisms may include linear, tilt and rotary adjustments among others.
In the embodiment according to the
On the other hand, the imaging apparatus 4 is always below the AWI, where a complimentary secondary scanner 9 with a field of view 90 is provided. In a similar fashion, the imaging apparatus 4 has a field of view 40 in that the field of view intersects the AWI from below at the zone 20 of the artifact 12 located at the air-water interface, whereby the apparatus is capable of imaging or scanning the artifact in the zone 20. For the secondary scanner 9, the field of view 90 in the second medium (water) intersects with the field of view 40 of the second primary scanner 4, wherein the two fields of view intersect at 400. Again, the objective of the intersection is to make it possible to stitch a continuous image from the air-water interface at 20, and across the artifact to get a more complete view of the artifact 12 in the second medium.
In a preferred embodiment, it is expected that the platform 1 able to support scanning from as far as 2.0 meters of distance from the artifact 12, thus ensuring that the scans can be non-intrusive no matter the operational environment. The platform and imaging/scanning apparatus should be light enough to be portable (preferably about 10 kg such that when suspended in water, a person will be able to hold for an extended period. However, even heavier or lighter equipment would still work as well) giving the option of being either human operated or drone/ROV-mounted, depending on individual use cases. For the primary imaging apparatus, so long as scanning is continuous, it would be possible to detach or rotate individual scanners flexibly in-situ. The two or more secondary scanner units 8 and 9 may be deployed for scanning alternative fields of view, increasing the field coverage, uniformity, and scan speed, as well as flexible scanning around complex objects, thus removing any shadow zones. It is anticipated that the additional scanning units may be built around a single portable scanner which will be either human-operated or drone/ROV-mounted, depending on the particular needs of each use case. The scan time required to achieve the required outcomes, ignoring post-processing, is expected to be well within the guidelines set, but reduced scan speeds will improve the uniformity of the scan.
For both secondary scanners 8 and 9, the objective of the secondary imaging is to stitch the images/scans of portions secondary to the zone 20 of the artifact since the primary scanners may probably not fully cover the entire artifact 12. It is to be understood further that in the arrangement demonstrated by the
The scan resolution of any scanner may be selected during the calibration phase. In the case of drone/ROV-mounted operation, the scanning speed and routes may be selected prior to deployment and be modified as required. Wireless transmitters will send initial point cloud scans to an operator using the compute resource 7 in real-time, and in the case of human-operation, both a miniature real-time display and a warning indicator may be mounted on the scanning unit which will inform the operator if the program believes the area requires additional scanning. This real-time processing and post-processing computer program may be developed for any operating system.
One possible problem of the arrangement is the occurrence of an object blocking the intersecting fields of view of the primary scanner unit, occurring primarily at very shallow depths in the AWI area, due to the interference of the water surface limiting the possible fields of view. A possible solution to this problem is for the above-water scanner 3 of the primary scanning unit to be able to scan through the water surface to a limited depth. In conjunction with a high-resolution optical scanner, the unit should determine the exact water surface fluctuations, allowing a suitable computer program to reconstruct through-water surface profiles to a limited depth with great accuracy.
Using a suitably trained machine learning algorithm, the suitable computer program is able to identify the high and low points in the water surface heights. By preferably utilizing a neural network such as a Deep Neural Network (DNN), the program will be trained to recognize key features from both the above-water and underwater scans, allowing optimal measurements based on surface properties and automatically stitch together the scans in post-processing. The high-power primary scanners will produce high-resolution scans of the surface profiles of solid objects, porous objects, and topography in and surrounding the AWI area. The result will be a single complete, accurate, and high-density point cloud of all surface profiles in, around, and below the AWI area.
In the embodiment according to
During an imaging operation, when light traveling in one transparent medium encounters a boundary with a second transparent medium (e.g., air and glass), a portion of the light is reflected and a portion is transmitted into the second medium. As the transmitted light moves into the second medium, it changes its direction of travel; that is, it is refracted. The law of refraction, Snell's law above, describes the relationship between the angle of incidence (θi) and the angle of refraction (θt), measured with respect to the normal (“perpendicular line”) to the surface, in mathematical terms: n1 sin θi=nt sin θt, where ni and nt are the index of refraction of the first and second media, respectively. The index of refraction for any medium is a dimensionless constant equal to the ratio of the speed of light in a vacuum to its speed in that medium. A ray of light 60 in the first medium with an incidence angle θi is transmitted in the second medium as 61, with an angle of refraction θt.
In the second
In the
In the current technical application of scanning/imaging an artifact through an interface of two media such as air and water, since the electromagnetic wave such as laser, or even light will have to traverse two media, refraction will occur and thus cause a distortion of the scan/image result. This is also likely to cause distortion of sonar waves. Further, there are likely more distortions caused by waves in the liquid medium, and such waves only escalate challenge of imaging at an air-water interface or such media, because the waves cause refraction of light in many different directions.
The subsequent embodiment as in
For the avoidance of doubt, the training of a machine learning algorithm will require a plurality of distorted images 102, labeled data 100, since that is the only way to generate a useful prediction model capable of distortion correction. Also, for each of the plurality of distorted images 102, an interference pattern that includes waves may also contribute to the distortion or noise, and as such an interference or surface disturbance scan may also be performed by the imaging apparatus at the air-water interface (or any such interface between two media), wherein such a profile may provide useful patterns. The output of a training process is a trained machine learning engine capable of noise/distortion correction from distorted images.
A suitable computer program provided in the compute resource 7 of
In the
In the proceeding embodiment according to the
On the other hand, a residual block shall be comprised of a Rectified Linear Unit (ReLU), which is a non-linear activation function that performs on multi-layer neural networks. The Rectified Linear Unit, or ReLU, is not a separate component of the convolutional neural networks' process. It's a supplementary step to the convolution operation. Further contained therein is a Convolutional Neural Network (Cony/CNN), which is a Deep Learning algorithm which can take in an input image, assign importance (learnable weights and biases) to various aspects/objects in the image and be able to differentiate one from the other. Also important is the Batch normalization (BN), which is a layer that allows every layer of the network to do learning more independently. It is used to normalize the output of the previous layers. The activations scale the input layer in normalization.
The prediction 104 is then passed on to a discriminator 105, comprised of a plurality of deep neural network blocks (DNN) such as 1051 and 1052, which attempt to distinguish predicted images from the ground truth. More precisely, once an image 104 is predicted as in 112, it is passed to the discriminator as in 113, to determine a loss function 1050, which could be caused by either a generator or discriminator. The loss function may then be passed back to the generator to retrain the engine until an appropriate accuracy is achieved. In the context of the current invention, a generator is a function that behaves like an iterator. An iterator loops (iterates) through elements of an object, like items in a list or keys in a dictionary. On the other hand, a discriminator in a neural network is simply a classifier. It tries to distinguish real data from the data created by the generator. It could use any network architecture appropriate to the type of data it's classifying.
The current invention uses the machine learning engine to understand the variances in water surface height (ripples, currents) in order to get around the interference caused by the properties of the water surface for scans in the AWI area. It is anticipated that interference problem would give an expected water surface height variance of over 1.0-2.0 cm, while the solution of the invention would have a measurement error preferably within 1.0 mm or less with the correction factors from the trained prediction water surface model, which is preferably generated by processing the data from an RGB camera through our 3D water surface ripple model. The difference in scale between the water surface height variance and a measurement resolution should allow for more than enough data from the AWI area to be recorded by both scanners. Additionally, since the platform 1 of
In the embodiment according to
Next, in the step 72 is the receiving of a plurality distorting interference patterns, each corresponding to a distorted image received from above or below the air-water interface, and wherein the interference may be a measured value or determined by a suitable computer program in the compute resource 7. Further, in the step 73 is the receiving of a plurality of corresponding control images of a surface without a distortion for training and validation, the control images constituting the ground truth and made up of labeled data. Next in 74 is to attempt to predict clean images that do not have the distortion from the plurality of distorted images, the corresponding scans of the air-water interface. Next is the using of the control images to determine the accuracy of the prediction and retrain until the desired accuracy is achieved in a step 75. The final step 76 of the training is to output a prediction model capable of accurately correcting the distortion caused by the air-water-interface and interference.
It is anticipated that the trained algorithm could be any such machine learning algorithm capable of distortion correction including neural networks, support vector machines, among others, and for purposes of this disclosure, it may be referred to simply as machine learning algorithm.
In the exemplary embodiment according to
Subsequently in the step 84 is the determination of the effect of interference of the air-water interface on light, and thus scan distortion on the stitched image in readiness for the distortion correction. Next in 85 is to make an algorithmic correction of the distortion using a suitably trained machine learning algorithm, whereby the distorted image is passed through the trained prediction model, which used a suitable algorithm to predict a clear image. Finally in the step 86 is to output a corrected scan without the distortion caused by the air-water-interface and interference. In further processing, a predicted image without distortions caused by refraction and interference at an air water interface may be coupled to secondary scanners. The secondary scanners can take additional images of the zone and these images can be stitched to the predicted.
It is anticipated that the imaging apparatus may comprise of a means to stitch images taken by the primary scanners.
It is also anticipated that the primary imaging apparatus may comprise a means to measure the dimensions of an artifact zone under imaging.
It is anticipated that the secondary imaging apparatus may comprise a means to measure the dimensions of an artifact zone under imaging.
Although a preferred embodiment of the present invention has been described for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.
The invention is applicable in the imaging industry, and specifically in the improving imaging through different media such as across air and water, among others.
| Number | Date | Country | |
|---|---|---|---|
| 63106915 | Oct 2020 | US |
