Patent Application
20240293206
The present disclosure relates to a system and method for generating a three-dimensional (3D) representation of an object. In particular, the present disclosure relates to a system and method for determining a solution to the correspondence problem associated with imaging using a 3D scanning device.
In scanning systems employing triangulation, a central task is to solve the so-called correspondence problem. The correspondence problem may refer to the task of determining image features in images within a set of images and associating each image feature with a projector ray, wherein said images captures the same three-dimensional (3D) scene or object. The images can be taken from a different point of view, at different times, or with objects in a scene in general motion relative to the camera(s).
The correspondence problem can occur in a stereo situation when two images of the same object are acquired, or it can be generalized to an N-view correspondence problem. In the latter case, the images may come from either N different cameras photographing at the same time or from one camera which is moving relative to the object/scene. The problem is made more difficult when the objects in the scene are in motion relative to the camera(s).
In order to generate a three-dimensional digital representation of a scanned object, such as a person's teeth, the correspondence problem generally need to be solved, at least when using a triangulation-based scanning device to acquire the images of the object. In general, it is desired that the 3D digital representation is generated in real-time, or at least in what is perceived as real-time to the user, e.g. the dentist. Therefore, the 3D representation is typically generated simultaneously with the acquisition of images, such that the user can immediately view the generated 3D representation while scanning a patient. Consequently, it is desired that the correspondence problem is solved in a fast, reliable, and accurate manner since it enables a fast generation of the 3D representation.
Thus, it is of interest to develop improved systems and methods for generating a digital 3D representation of a 3D object. In particular, it is desired to find improved methods for solving the correspondence problem and related issues within 3D scanning systems.
While techniques for solving the correspondence problem when projecting a time-varying light pattern exist, the problem becomes more difficult when using a static light pattern. Additional challenges occur when projecting a pattern with a high density of pattern features. Such patterns are generally desired since a high-density pattern improves the resolution of the generated digital representation. However, utilizing a high-density pattern makes the correspondence problem even more difficult to solve, since there is a greater degree of ambiguity. It is an aim of the present disclosure to provide a more robust and accurate way of solving the correspondence problem associated with 3D imaging of an object, in particular in order to obtain a more accurate 3D representation of the scanned object.
The present disclosure addresses the above-mentioned challenges by providing a scanning system for generating a three-dimensional (3D) representation of an object, the scanning system comprising:
The present disclosure further relates to a computer-implemented method for generating a three-dimensional (3D) representation of a surface of an object, the method comprising the steps of:
The present disclosure further relates to a computer-implemented method for determining a solution to a correspondence problem associated with a triangulation-based 3D scanning device, the method comprising the steps of:
The present disclosure further relates to a data processing system, such as the scanning system disclosed herein, comprising one or more processors configured to perform the steps of the method disclosed herein.
The present disclosure further relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method disclosed herein. The present disclosure further relates to a computer-readable data carrier having stored thereon said computer program product.
The present disclosure further relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method disclosed herein.
The disclosed system and method provide a more efficient and reliable framework for solving the correspondence problem associated with 3D scanning systems, and consequently a more accurate three-dimensional (3D) representation of the scanned object can be generated. In particular, the disclosed system and method provides an improvement over existing methods, e.g. due to the ability to resolve the ambiguity encountered when solving the correspondence problem, whereby a more accurate 3D representation is obtained.
The three-dimensional (3D) object may be a dental object. Examples of dental objects include any one or more of: tooth/teeth, gingiva, implant(s), dental restoration(s), dental prostheses, edentulous ridge(s), and/or combinations thereof. Alternatively, the dental object may be a gypsum model or a plastic model representing a subject's teeth. As an example, the three-dimensional (3D) object may comprise teeth and/or gingiva of a subject. The dental object may only be a part of the subject's teeth and/or oral cavity, since the entire set of teeth of the subject is not necessarily scanned during a scanning session. A scanning session may be understood herein as a period of time during which data (such as images) of the 3D object is obtained.
The scanning device disclosed herein may be an intraoral scanning device for acquiring images within an intraoral cavity of a subject. The scanning device may be a handheld scanning device, i.e. a device configured to be held with a human hand. The scanning device may employ any suitable scanning principle such as triangulation-based scanning, stereo vision, structure from motion, confocal scanning, or other scanning principles.
In preferred embodiments, the scanning device employs a triangulation-based scanning principle. As an example, a projector unit and one or more camera units may be utilized to determine points in 3D space based on triangulation. As another example, the scanning device comprises a projector unit and two or more camera units, wherein the camera units are configured to image the scanned object from separate views, i.e. from different directions. In particular, the camera units may be configured to acquire a set of images, wherein a correspondence problem is solved within said set of images based on triangulation. The images within a set of images are preferably acquired simultaneously, i.e. at the same moment in time. The correspondence problem generally refers to the problem of ascertaining which parts of one image correspond to which parts of another image. Specifically, the projector unit may be configured to project a plurality of projector rays, which are projected onto a surface of the dental object. Solving the correspondence problem may include the steps of determining image features in the images within a set of images, and further associate said image features with a specific projector ray. Subsequently, the depth of each projector ray may be computed, whereby a 3D representation of the scanned object may be generated. In some embodiments, the correspondence problem is solved jointly for groups of projector rays, as opposed to e.g. solving the correspondence problem projector ray by projector ray.
The scanning device may comprise one or more scan units, wherein each scan unit comprises a projector unit and one or more camera units. As an example, the scanning device may comprise one scan unit comprising one projector unit and at least two camera units. As another example, the scanning device may comprise one scan unit comprising one projector unit and four camera units. In yet another example, the scanning device may comprise at least two scan units, wherein each scan unit comprises a projector unit and two or more camera units. In yet another example, the scanning device may comprise at least two scan units, wherein each scan unit comprises a projector unit and four camera units.
The scanning device may further comprise a reflecting element arranged in combination with a given scan unit. The reflecting element is preferably configured to reflect light from the projector unit of the scan unit and/or from the surface of the dental object and onto the image sensor(s) of each camera unit of the scan unit associated with the reflecting element. In preferred embodiments, the scanning device comprises or constitutes an elongated probe, which defines a longitudinal axis of the scanning device.
A projector unit may be understood herein as a device configured for projecting light onto a surface, such as the surface of a three-dimensional object. In preferred embodiments, the projector unit is configured to project a pattern of light onto the surface of a dental object. Preferably, the projector unit is configured to project a pattern of light such that the pattern of light is in focus at a predefined focus distance measured along a projector optical axis. In some embodiments, the projector unit is configured to project the pattern of light such that the pattern of light is defocused at the opening of the probe of the scanning device and/or at a surface of an optical window in said probe.
The projector unit may comprise Digital Light Processing (DLP) projectors using a micro mirror array for generating a time varying pattern, or a diffractive optical element (DOF), or front-lit reflective mask projectors, or micro-LED projectors, or Liquid crystal on silicon (LCoS) projectors or back-lit mask projectors, wherein a light source is placed behind a mask having a spatial pattern, whereby the light projected on the surface of the dental object is patterned. The pattern may be dynamic, i.e. such that the pattern changes over time, or the pattern may be static in time, i.e. such that the pattern remains the same over time. The projector unit may comprise one or more collimation lenses for collimating the light from the light source. The collimation lens(es) may be placed between the light source and the mask. The projector unit may further comprise one or more focus lenses, or lens elements, configured for focusing the light at a predefined focus distance.
In some embodiments, the projector unit of the scanning device comprises at least one light source and a pattern generating element for defining a pattern of light. The pattern generating element is preferably configured for generating a light pattern to be projected on a surface of a dental object. As an example, the pattern generating element may be a mask having a spatial pattern. Hence, the projector unit may comprise a mask configured to define a pattern of light. The mask may be placed between the light source of the projector unit and the one or more focus lenses, such that light transmitted through the mask is patterned into a light pattern. As an example, the mask may define a polygonal pattern comprising a plurality of polygons, such as a checkerboard pattern. The projector unit may further comprise one or more lenses such as collimation lenses or projection lenses. In other embodiments, the pattern generating element is based on diffraction and/or refraction to generate the light pattern, such as a pattern comprising an array of discrete unconnected dots.
The projector unit may be configured to generate a predefined static pattern, which may be projected onto a surface of an object. An advantage of using a static pattern is that it enables the ability to capture all image data simultaneously, thus preventing warping due to movement. Another advantage is that a static pattern makes it possible to acquire only one image per camera unit in the set of images, thereby reducing power consumption e.g. of the light source.
Alternatively, the projector unit may be configured to generate a dynamic pattern, which changes in time. The projector unit may be associated with its own projector plane, which is determined by the projector optics. As an example, if the projector unit is a back-lit mask projector, the projector plane may be understood as the plane wherein the mask is contained. The projector plane comprises a plurality of pattern features of the projected pattern. Preferably, the camera units and projector unit are arranged such that the image sensors and the projector plane, e.g. defined by the mask, are in the same plane.
The projector unit may define a projector optical axis. An optical axis may be understood as a line along which there is some degree of rotational symmetry in an optical system such as a camera lens or a projector unit. In some embodiments, the projector optical axis of the projector unit is substantially parallel with the longitudinal axis of the scanning device. In other embodiments, the projector optical axis of the scan unit defines an angle, such as at least 45° or at least 75°, with the longitudinal axis of the scanning device. In other embodiments, the projector optical axis of the projector unit is substantially orthogonal to the longitudinal axis of the scanning device.
The projector unit may comprise one or more light sources. The projector unit may be configured to project a pattern of light defined by a plurality of projector rays when the light source(s) are on/active. The light source(s) may be configured to generate light of a single wavelength or a combination of wavelengths (mono- or polychromatic). The combination of wavelengths may be produced by a light source configured to produce light comprising different wavelengths (such as white light).
In some embodiments, each projector unit comprises a light source for generating white light. Alternatively, the projector unit may comprise multiple light sources such as LEDs individually producing light of different wavelengths (such as red, green, and blue) that may be combined to form light comprising different wavelengths. Thus, the light produced by the light source(s) may be defined by a wavelength defining a specific color, or a range of different wavelengths defining a combination of colors such as white light. In some embodiments, the light source is a diode, such as a white light diode, or a laser diode.
In some embodiments, the scanning device comprises a light source configured for exciting fluorescent material to obtain fluorescence data from the dental object such as from teeth. Such a light source may be configured to produce a narrow range of wavelengths. In other embodiments, the scanning device comprises an infrared light source, which is configured to generate wavelengths in the infrared range, such as between 700 nm and 1.5 μm. In some embodiments, the scanning device comprises one or more light sources selected from the group of: Infrared (IR) light source, near-infrared (NIR) light source, blue light source, violet light source, ultraviolet (UV) light source, and/or combinations thereof. In some embodiments, the scanning device comprises a first light source forming part of the projector unit, and one or more second light sources, e.g. IR-LED(s) and/or UV-LED(s), located in a distal part of the scanning device, such as in the tip of the scanning device.
The projector unit may be configured to project a pattern of light defined by a plurality of projector rays when a light source of the projector unit is turned on. The terms ‘pattern of light’, ‘spatial pattern’, and ‘pattern’ are used herein interchangeably. The pattern may be generated using a pattern generating element, e.g. located in the projector unit. The pattern generating element may be a mask, such as a transparency or transmission mask, having a spatial pattern. In other embodiments, the pattern generating element is configured to utilize diffraction and/or refraction to generate a light pattern.
The spatial pattern may be a polygonal pattern comprising a plurality of polygons. The polygons may be selected from the group of: triangles, rectangles, squares, pentagons, hexagons, and/or combinations thereof. Other polygons can also be envisioned. In general, the polygons are composed of edges and corners. In some embodiments, the polygons are repeated in the pattern in a predefined manner. As an example, the pattern may comprise a plurality of repeating units, wherein each repeating unit comprises a predefined number of polygons, wherein the repeating units are repeated throughout the pattern. Alternatively, the pattern may comprise a predefined arrangement comprising any of stripes, squares, dots, triangles, rectangles, and/or combinations thereof. In some embodiments, the pattern is non-coded, such that no part of the pattern is unique.
In some embodiments, the generated pattern of light is a polygonal pattern, such as a checkerboard pattern comprising a plurality of checkers. Similar to a common checkerboard, the checkers in the pattern may have alternating dark and bright areas corresponding to areas of low light intensity (dark) and areas of high(er) light intensity (bright). In some embodiments the pattern of light is a checkerboard pattern comprising alternating squares of dark and bright. In some embodiments, the light pattern comprises a distribution of discrete unconnected spots of light.
The pattern preferably comprises a plurality of pattern features. The pattern may be a high-density pattern, which may be understood as a pattern comprising more than 3000 pattern features. However, the presently disclosed system and method is not limited to high-density patterns, since it works equally well for patterns of lower density. In some embodiments, the pattern comprises at least 1000 pattern features, or at least 3000 pattern features, or at least 10000 pattern features. When projecting a pattern comprising pattern features onto a surface of the 3D object, the acquired images of the object will similarly comprise a plurality of image features corresponding to the pattern features. A pattern/image feature may be understood as an individual well-defined location in the pattern/image. Examples of image/pattern features include corners, edges, vertices, points, transitions, dots, stripes, etc. In some embodiments, the image/pattern features comprise the corners of checkers in a checkerboard pattern. In other embodiments, the image/pattern features comprise corners in a polygon pattern such as a triangular pattern.
A camera unit may be understood herein as a device for capturing an image of an object. Each camera unit may comprise an image sensor for generating an image based on incoming light e.g. received from an illuminated 3D object. As an example, the image sensor may be an electronic image sensor such as a charge-coupled device (CCD) or an active-pixel sensor (CMOS sensor). In some embodiments, the image sensor is a global shutter sensor configured to expose the entire image area (all pixels) simultaneously and generate an image in a single point in time. The image sensor may have an image frame rate of at least 30 frames per second, such as at least 60 frames per second, or even at least 90 frames per second.
The image sensor(s) may comprise an array of pixels, wherein each pixel is associated with a corresponding camera ray. Similarly, each image, within the set of images, may be composed of an array of pixels, wherein each pixel comprises a pixel color defined by one or more color channels. The array of pixels may be a two-dimensional (2D) array. In some embodiments, the image sensor is a CMOS sensor comprising an analog-to-digital converter (ADC) for each column of pixels, making conversion time significantly faster and allowing each camera unit to benefit from greater speed. Each image sensor may define an image plane, which may be understood as the plane that contains the object's projected image. Each image obtained by the image sensor(s) may comprise a plurality of image features, wherein each image feature originates from a pattern feature of the projected pattern. In some embodiments, one or more of the camera units comprise a light field camera. Preferably, each camera unit defines a camera optical axis. The camera units may further comprise one or more focus lenses for focusing light.
In some embodiments, the image sensor is a monochrome image sensor, wherein each pixel is associated with a single color channel, e.g. is a grayscale color channel, wherein the value of each pixel represents only an amount of light. In other embodiments, the image sensor is a color image sensor or an image sensor comprising a color filter array on the array of pixels. As an example, the color filter array may be a Bayer filter employing an arrangement of four color filters: Red (R), Green (G), Green (G), and Blue (B). The Bayer filter may also be referred to as a RGGB filter. When utilizing the image sensor data, color pixels may be combined to monochrome pixels of 2×2 color pixels for 3D depth reconstruction. In this case, the resolution of the 3D depth reconstruction is only half the resolution of the image sensor in each direction. When obtaining texture (color) images the full native resolution is preferably utilized (with color filtered pixels).
In accordance with some embodiments, the projector optical axis and the camera optical axis of at least one camera unit define a camera-projector angle of approximately 5 to 15 degrees, preferably 5 to 10 degrees, even more preferably 8 to 10 degrees. In some embodiments, the camera units are defocused at the opening of the probe of the scanning device and/or at the surface of an optical window in said probe. In some embodiments of the scanning device, the camera units and projector unit of a given scan unit are focused at the same distance. In some embodiments, each camera unit has a field of view of 50-115 degrees, such as 65-100 degrees, or 80-90 degrees.
A scan unit may be understood herein as a unit comprising at least one projector unit and one or more camera units. In some embodiments, each scan unit comprises at least two camera units having at least partly overlapping fields of view along different camera optical axes. Preferably, each scan unit comprises at least four camera units having at least partly overlapping fields of view along different camera optical axes.
A reflecting element may be understood herein as an element configured to change the direction of light rays incident on the surface of said reflecting element or being transmitted through said reflecting element, e.g. in case of a prism. In particular, the reflecting element is preferably configured to change the direction of a center beam of the projected light from a projector unit from a direction substantially parallel to the longitudinal axis of the scanning device to a direction substantially orthogonal to said longitudinal axis. In some embodiments, a surface normal of each reflecting element defines an angle with respect to the projector optical axis of approximately 40-50 degrees, preferably approximately 45 degrees.
As an example, the reflecting element may be selected from the group of: mirrors, prisms, and/or combinations thereof. In some embodiments, the reflecting element is configured to reflect light from the projector unit of the scan unit and/or reflect light from the surface of the object being scanned and onto the image sensors of the scan unit. In some embodiments, the scanning device comprises a mirror as the reflecting element. In other embodiments, the scanning device comprises a prism as the reflecting element. Some embodiments feature a combination of mirror(s) and prism(s). In case of a prism as the reflecting element, the prism is preferably configured to change the direction of a center beam of the projected pattern of light from substantially parallel to the longitudinal axis of the scanning device to a direction having an angle of at least 45 degrees with respect to said longitudinal axis. Even more preferably, the prism is configured to change the direction of a center beam of the projected pattern of light from substantially parallel to the longitudinal axis of the scanning device to a direction having an angle of approximately 90 degrees with respect to said longitudinal axis.
In some embodiments, the scanning device comprises a scan unit, wherein the scanning device further comprises a reflecting element positioned on the projector optical axis of the projector unit of said scan unit. In other embodiments, the scanning device comprises at least two scan units, wherein the scanning device further comprises a reflecting element arranged in combination with each scan unit. The reflecting element is then configured to reflect light projected from the projector unit of said scan unit. The reflecting element is preferably further arranged to reflect light from the object being scanned and onto the image sensor(s) of each camera unit of the scan unit. In some embodiments, the reflecting element of each scan unit is positioned on the projector optical axis. The projector optical axis may in some embodiments coincide with the longitudinal axis of the scanning device.
In accordance with some embodiments, the scanning device comprises one or more processors selected from the group of: central processing units (CPU), accelerators (offload engines), general-purpose microprocessors, graphics processing units (GPU), neural processing units (NPU), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), dedicated logic circuitry, dedicated artificial intelligence processor units, or combinations thereof.
The scanning device may further comprise computer memory for storing instructions, which when executed, causes the processor(s) to carry out the step of determining image features in the set of images. The computer memory may further store instructions, which when executed, causes the processor(s) to carry out the step of generating a digital representation of a three-dimensional (3D) object. In general, the processor(s) may be configured to perform any of the disclosed computer-implemented methods herein, either fully or in part, e.g. such that some processor(s) perform some method steps and other processors perform other method steps, when executed.
The processor(s) may both be located on the scanning device. Alternatively, the processor(s), or a sub-set of the processor(s), may be located on a computer system as described herein. As an example, a first processor may be configured to determine image features in the images, and subsequently provide data related to the determined image features to a second processor. The data may comprise image feature coordinates as well as other attributes such as a camera index or other predefined properties of the image features.
The processor(s) may be configured to generate the digital representation of the 3D object, e.g. in the form of a point cloud. The scanning device may be further configured to provide the digital representation to a computer system for rendering the representation. The computer system may further process the digital representation, e.g. by stitching the point clouds received from the scanning device and/or by fitting one or more surfaces to the stitched point clouds. This further processing by the computer system may also be referred to herein as reconstruction. The output of the reconstruction is a digital 3D model of the scanned object. The digital 3D model may be rendered and displayed on a display, e.g. connected to the computer system.
The scanning device may comprise a module for transmitting data, such as images or point clouds, to one or more external devices, such as to a computer system. The module may be a wireless module configured to wirelessly transfer data from the scanning device to the computer system. The wireless module may be configured to perform various functions required for the scanning device to wirelessly communicate with a computer network. The wireless module may utilize one or more of the IEEE 802.11 Wi-Fi protocols/integrated TCP/IP protocol stack that allows the scanning device to access the computer network. It may also use other network The wireless module may include a system-on-chip having different types of inbuilt network connectivity technologies. These may include commonly used wireless protocols such as Bluetooth, ZigBee, Wi-Fi, WiGig (also known as 60 GHz Wi-Fi), etc. The scanning device may further (or alternatively) be configured to transmit data using a wired connection, such as an ethernet cable or a USB cable. In some embodiments, the scanning device comprises a wireless module configured to wirelessly transfer data from the scanning device to the computer system.
A computer system may be understood as an electronic processing device for carrying out sequences of arithmetic or logical operations. In the present context, a computer system refers to one or more devices comprising at least one processor, such as a central processing unit (CPU), along with some type of computer memory. Examples of computer systems falling within this definition include desktop computers, laptop computers, computer clusters, servers, cloud computers, quantum computers, mobile devices such as smartphones and tablet computers, and/or combinations thereof.
The computer system may comprise hardware such as one or more central processing units (CPU), graphics processing units (GPU), and computer memory such as random-access memory (RAM) or read-only memory (ROM). The computer system may comprise a CPU, which is configured to read and execute instructions stored in the computer memory e.g. in the form of random-access memory. The computer memory is configured to store instructions for execution by the CPU and data used by those instructions. As an example, the memory may store instructions, which when executed by the CPU, cause the computer system to perform, wholly or partly, any of the computer-implemented methods disclosed herein. The computer system may further comprise a graphics processing unit (GPU). The GPU may be configured to perform a variety of tasks such as video decoding and encoding, rendering of the digital representation, and other image processing tasks.
The computer system may further comprise non-volatile storage in the form of a hard disc drive. The computer system preferably further comprises an I/O interface configured to connect peripheral devices used in connection with the computer system. More particularly, a display may be connected and configured to display output from the computer system. The display may for example display a 2D rendering of the generated digital 3D representation. Input devices may also be connected to the I/O interface. Examples of such input devices include a keyboard and a mouse, which allow user interaction with the computer system. A network interface may further be part of the computer system in order to allow it to be connected to an appropriate computer network so as to receive and transmit data (e.g. scan data, sub-scans, and/or images) from and to other computing devices. The CPU, volatile memory, hard disc drive, I/O interface, and network interface, may be connected together by a bus.
The computer system is preferably configured for receiving data from the scanning device, either directly from the scanning device or via a computer network such as a wireless network. The data may comprise images, processed images, sub-scans, point clouds, sets of data points, or other types of data. The data may be transmitted/received using a wireless connection, a wired connection, and/or combinations thereof. In some embodiments, the computer system is configured for generating a digital representation of a three-dimensional (3D) object as described herein. In some embodiments, the computer system is configured for receiving data, such as sub-scans or point clouds, from the scanning device and then subsequently perform the steps of reconstruction and rendering a digital representation of a three-dimensional (3D) object. Rendering may be understood as the process of generating one or more images from three-dimensional data. The computer system may comprise computer memory for storing a computer program, said computer program comprising computer-executable instructions, which when executed, causes the computer system to carry out the method of generating a digital representation of a three-dimensional (3D) object.
The method disclosed herein may comprise the step of acquiring or obtaining a set of images. A set of images may be understood herein as comprising two or more two-dimensional (2D) images. The 2D images may be digital images, such as digital color images. The set of images may be acquired by the scanning system disclosed herein, such as by an intraoral scanning device. The intraoral scanning device may comprise multiple camera units, such as two or more camera units, or four or more camera units. Each camera unit may comprise an image sensor having an array of pixels. Each pixel may be associated with a corresponding camera ray in three-dimensional (3D) space originating from said pixel. Generally, a point located along a given camera ray corresponding to a given pixel will be imaged on the image sensor on said pixel.
Each 2D image may be composed of an array of pixels, such as a two-dimensional array of pixels, corresponding to the array of pixels on the image sensor(s). Each pixel may be associated with a camera ray as described above. Each pixel may comprise a pixel color defined by one or more color channels. An example of a single color channel is a grayscale color channel, wherein the value of each pixel represents only an amount of light, thereby carrying intensity information. Thus, the images within the set of images may be grayscale images. Another example of a single color channel is red, green, or blue. Thus, the presently disclosed method is not limited to using pixel color information, but may instead utilize pixel intensity information.
Alternatively, each pixel may comprise more than one color channel, such as three color channels, e.g. a red, green, and blue channel (RGB). In that case, the images within the set of images may be RGB images. The RGB images may be 24-bit, such that each channel has 8 bits, for red, green, and blue. In other words, the image may be composed of three images (one for each channel), wherein each image can store discrete pixels with brightness intensities between 0 and 255. The RGB images may alternatively be 48-bit, corresponding to a very high color-depth, wherein each channel has 16 bits per pixel color, that is 16-bit red, green, and blue for each pixel. In some embodiments, each sensor pixel comprises a color filter so that it can only capture light of a certain wavelength or narrow range of wavelengths, e.g. such that a given pixel can only capture either red, green, or blue light. The arrangement of color filters over the sensor pixels may constitute a color filter array, such as a Bayer filter. In some cases, a color is assigned to a given image pixel by averaging the color of neighboring sensor pixels.
In some embodiments, each pixel in the array of pixels on the image sensor comprises a pixel color defined by a single color channel, such as red, green, or blue. In such embodiments, the method may comprise the step of acquiring three consecutive sets of images, wherein a first set of images are obtained with e.g. red pixels, a second set of images are obtained with green pixels, and a third set of images are obtained with blue pixels. In that case, a given image does not comprise three color channels; instead the three color channels may be provided by three different sets of images. Thus, color in the image pixels may be inferred from the three consecutive sets of images obtained by the scanning device.
In some embodiments, the set of images are acquired using an intraoral scanning device, such as a triangulation-based intraoral scanning device employing multiple camera units or image sensors. As an example, the set of images may be acquired using an intraoral scanning device comprising two or more camera units, such as four or more camera units. In some embodiments, the set of images comprises four images, wherein images within the set of images are acquired by four different camera units. In other words, each camera unit or image sensor may contribute with one image to the set of images, such that the number of images within the set of images corresponds to the number of camera units or image sensors used to acquire said images. In some embodiments, the images are acquired simultaneously, i.e. at the same moment in time.
Each image within the set of images may comprise a number of image features, such as a plurality of image features. In some embodiments, the disclosed method comprises the step of determining one or more image features within the set of images. The image features may be associated with pattern features which are part of the pattern projected on the surface of the object. In general, the images may be associated with a correspondence problem, which is desired to solve in order to determine corresponding image features between the images. Thus, a purpose of the presently disclosed system and method is to solve said correspondence problem, whereby a 3D representation of the scanned object may be generated.
The scanning system may be configured to continuously acquire sets of images with a predetermined framerate during a scanning session, wherein one or more objects, such as the teeth of a patient, is scanned using an intraoral scanning device. Preferably, the computer-implemented method disclosed herein is executed for each set of images acquired, such that the correspondence problem associated with said set of images, is solved. In preferred embodiments, said correspondence problem is solved before the acquisition of a subsequent set of images. As an example, if the image sets are acquired with a framerate of 20 image sets per second, then the scanning system may be configured to solve the correspondence problem in a time of less than 1/20 of a second.
The scanning system may comprise an intraoral scanning device comprising one or more projector units configured to project a pattern on a surface of the object. The projected pattern may comprise a plurality of pattern features. In general, the projected pattern is generated by a plurality of projector rays. Each pattern feature may be associated with a corresponding projector ray in three-dimensional (3D) space. In preferred embodiments, the method comprises the step of determining image features within the set of images. The method may further comprise the step of projecting camera rays in 3D space, each camera ray corresponding to a given image feature.
The method may comprise the step of determining potential points in three-dimensional (3D) space that form one or more candidate solutions to the correspondence problem. The potential points may be determined based on triangulation. Specifically, potential points may be determined based on intersections of camera rays with a given projector ray, each camera ray originating from a given image feature in an image. Thus, camera rays may be projected from all the image features in each image. In general, images within the set of images may comprise a number of corresponding image features, whose correspondence is not yet known. As an example, in the case of four images obtained by four separate camera units, a potential point in 3D space may be identified based on four camera rays intersecting a projector ray. The intersections of camera rays and a given projector ray may not form a perfect intersection; rather the camera rays may intersect a projector ray within some tolerance. Bases on said intersections, one or more potential 3D points may be determined for each projector ray. The potential points may be assigned a depth corresponding to the distance measured along a given projector ray originating from the pattern generating element. For each projector ray, there is only one point in 3D space that forms the true solution to the correspondence problem. However, this solution may not be known at this point. Rather, the method may identify one or more potential points for each projector ray, wherein the true solution may be determined among said potential points.
The potential points in 3D may be connected to form one or more meshes or three-dimensional surfaces. Said surfaces form a collection of plausible solutions, or candidate solutions, to the correspondence problem. An example of such candidate solutions are illustrated in
To achieve this, the method may further comprise the step of assigning one or more parameters to each potential point. The parameters may include a color, ci, and a likelihood, αi, that the potential point is part of the solution to the correspondence problem. The likelihood may be expressed as a number between 0 and 1, wherein said number is a measure of how probable/likely it is that the given potential point forms part of the true solution. It should be noted that the step does not necessarily include assigning specific values of the parameters. The parameters may be understood as mathematical variables. However, the method may comprise the step of providing an initial value for the colors and/or likelihoods.
The method may further comprise the step of generating a computed color for each camera ray corresponding to a pixel, wherein the computed color is based on the assigned colors and likelihoods. Thus, each pixel may similarly be associated with a computed color and a pixel color, wherein the latter is considered the true color of the pixel as given by the one or more color channels in the image. In some embodiments, the computed color of a given camera ray, r, is defined by =α0c0+α1c1(1−α0)+α2c2(1−α0)(1−α1)+( . . . ). This equation may be written in the more generalized and concise form given below:
The method may further comprise the step of determining the numeric value of the color and likelihood of the potential points by minimizing a cost function based on the difference between the computed colors and the pixel colors. As previously mentioned, each pixel in the array of pixels (in the image or on the image sensor) may have a pixel color defined by one or more color channels. Thus, the true color of a given pixel, or camera ray corresponding to that pixel, is known. Instead of utilizing pixel colors, pixel intensities may be utilized, e.g. as defined by a single color channel. By utilizing the expression for the computed color of the camera rays and the true color (e.g. the pixel color or pixel intensity) of the camera rays, the numeric value of the color and likelihood of each potential point may be determined by solving an optimization problem, wherein a cost function is minimized. In some embodiments, the cost function is defined by the equation below:
Accordingly, the disclosed method may comprise the step of solving an optimization problem wherein the cost function is minimized in order to determine the numeric value of the color and likelihood of the potential points. The optimization problem may be solved using gradient-based methods. Subsequently, the true solution to the correspondence problem may be found among the potential points based on their likelihood. Specifically, potential points having a high numeric value of the likelihood are considered likely to be part of the solution. Thus, the method may comprise the step of selecting 3D points among the potential points, wherein said selection is based on the likelihood of the potential points, e.g. based on likelihoods exceeding a predetermined threshold value. Alternatively, for each projector ray, the potential point having the highest likelihood may be selected as a 3D point forming part of the true solution. Finally, the method may comprise the step of generating a three-dimensional (3D) representation based on the determined/selected 3D points. The collection of 3D points determined to form part of the true solution may simply collectively form the three-dimensional (3D) representation, e.g. in the form of a point cloud.
The three-dimensional (3D) representation may be a point cloud comprising a plurality of points in 3D space. In some embodiments, the 3D representation is a 3D polygon mesh. The 3D representation based on a single set of images may also be referred to herein as a sub-scan. The 3D representation may only represent a part of the surface of the scanned object. During a scanning session, e.g. using an intraoral scanning device, the scanning device may be moved across the object, typically with a variety of positions and angles, whereby a larger part of the object may be scanned. Thus, during a scanning session the scanning device may be configured to continuously acquire sets of images and generate sub-scans based on said sets of images. The sub-scans may then subsequently be stitched together to form a larger digital 3D model of the scanned object. The disclosed method herein relates in particular to the generation of a sub-scan based on a set of images, wherein the sub-scan constitutes a 3D representation of at least a part of the scanned object.
If programmable logic is used, such logic may execute on a commercially available processing platform configured by executable software code to become a specific purpose computer or a special purpose device (e.g., programmable logic array, application-specific integrated circuit, etc.). A person having ordinary skill in the art may appreciate that embodiments of the disclosed subject matter can be practiced with various computer system configurations, including multi-core multiprocessor systems, minicomputers, mainframe computers, computers linked or clustered with distributed functions, as well as pervasive or miniature computers that may be embedded into virtually any device. For instance, at least one processor device and a memory may be used to implement the above described embodiments.
A processor unit or device as discussed herein may be a single processor, a plurality of processors, or combinations thereof. Processor devices may have one or more processor “cores.” The terms “computer program medium,” “non-transitory computer readable medium,” and “computer usable medium” as discussed herein are used to generally refer to tangible media such as a removable storage unit 518, a removable storage unit 522, and a hard disk installed in hard disk drive 512.
Various embodiments of the present disclosure are described in terms of this example computer system 500. After reading this description, it will become apparent to a person skilled in the relevant art how to implement the present disclosure using other computer systems and/or computer architectures. Although operations may be described as a sequential process, some of the operations may in fact be performed in parallel, concurrently, and/or in a distributed environment, and with program code stored locally or remotely for access by single or multi-processor machines. In addition, in some embodiments the order of operations may be rearranged without departing from the spirit of the disclosed subject matter.
Processor device 504 may be a special purpose or a general purpose processor device specifically configured to perform the functions discussed herein. The processor device 504 may be connected to a communications infrastructure 506, such as a bus, message queue, network, multi-core message-passing scheme, etc. The network may be any network suitable for performing the functions as disclosed herein and may include a local area network (LAN), a wide area network (WAN), a wireless network (e.g., WiFi), a mobile communication network, a satellite network, the Internet, fiber optic, coaxial cable, infrared, radio frequency (RF), or any combination thereof. Other suitable network types and configurations will be apparent to persons having skill in the relevant art. The computer system 500 may also include a main memory 508 (e.g., random access memory, read-only memory, etc.), and may also include a secondary memory 410. The secondary memory 510 may include the hard disk drive 512 and a removable storage drive 514, such as a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, etc.
The removable storage drive 514 may read from and/or write to the removable storage unit 518 in a well-known manner. The removable storage unit 518 may include a removable storage media that may be read by and written to by the removable storage drive 514. For example, if the removable storage drive 514 is a floppy disk drive or universal serial bus port, the removable storage unit 518 may be a floppy disk or portable flash drive, respectively. In one embodiment, the removable storage unit 518 may be non-transitory computer readable recording media.
In some embodiments, the secondary memory 510 may include alternative means for allowing computer programs or other instructions to be loaded into the computer system 500, for example, the removable storage unit 522 and an interface 520. Examples of such means may include a program cartridge and cartridge interface (e.g., as found in video game systems), a removable memory chip (e.g., EEPROM, PROM, etc.) and associated socket, and other removable storage units 522 and interfaces 520 as will be apparent to persons having skill in the relevant art.
Data stored in the computer system 500 (e.g., in the main memory 508 and/or the secondary memory 510) may be stored on any type of suitable computer readable media, such as optical storage (e.g., a compact disc, digital versatile disc, Blu-ray disc, etc.) or magnetic tape storage (e.g., a hard disk drive). The data may be configured in any type of suitable database configuration, such as a relational database, a structured query language (SQL) database, a distributed database, an object database, etc. Suitable configurations and storage types will be apparent to persons having skill in the relevant art.
The computer system 500 may also include a communications interface 524. The communications interface 524 may be configured to allow software and data to be transferred between the computer system 500 and external devices. Exemplary communications interfaces 524 may include a modem, a network interface (e.g., an Ethernet card), a communications port, a PCMCIA slot and card, etc. Software and data transferred via the communications interface 524 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals as will be apparent to persons having skill in the relevant art. The signals may travel via a communications path 526, which may be configured to carry the signals and may be implemented using wire, cable, fiber optics, a phone line, a cellular phone link, a radio frequency link, etc.
The computer system 500 may further include a display interface 502. The display interface 502 may be configured to allow data to be transferred between the computer system 500 and external display 530. Exemplary display interfaces 502 may include high-definition multimedia interface (HDMI), digital visual interface (DVI), video graphics array (VGA), etc. The display 530 may be any suitable type of display for displaying data transmitted via the display interface 502 of the computer system 500, for example a cathode ray tube (CRT) display, liquid crystal display (LCD), light-emitting diode (LED) display, capacitive touch display, thin-film transistor (TFT) display, etc.
Computer program medium and computer usable medium may refer to memories, such as the main memory 508 and secondary memory 510, which may be memory semiconductors (e.g., DRAMs, etc.). These computer program products may be means for providing software to the computer system 500. Computer programs (e.g., computer control logic) may be stored in the main memory 508 and/or the secondary memory 510. Computer programs may also be received via the communications interface 524. Such computer programs, when executed, may enable computer system 500 to implement the present methods as discussed herein. In particular, the computer programs, when executed, may enable processor device 504 to implement the processes and methods illustrated by
The processor device 504 may comprise one or more modules or engines configured to perform the functions of the computer system 500. Each of the modules or engines may be implemented using hardware and, in some instances, may also utilize software, such as corresponding to program code and/or programs stored in the main memory 508 or secondary memory 510. In such instances, program code may be compiled by the processor device 504 (e.g., by a compiling module or engine) prior to execution by the hardware of the computer system 500. For example, the program code may be source code written in a programming language that is translated into a lower level language, such as assembly language or machine code, for execution by the processor device 504 and/or any additional hardware components of the computer system 500. The process of compiling may include the use of lexical analysis, preprocessing, parsing, semantic analysis, syntax-directed translation, code generation, code optimization, and any other techniques that may be suitable for translation of program code into a lower level language suitable for controlling the computer system 500 to perform the functions disclosed herein. It will be apparent to persons having skill in the relevant art that such processes result in the computer system 500 being a specially configured computer system 500 uniquely programmed to perform the functions discussed above.
Although some embodiments have been described and shown in detail, the disclosure is not restricted to such details, but may also be embodied in other ways within the scope of the subject matter defined in the following claims. In particular, it is to be understood that other embodiments may be utilized, and structural and functional modifications may be made without departing from the scope of the present disclosure. Furthermore, the skilled person would find it apparent that unless an embodiment is specifically presented only as an alternative, different disclosed embodiments may be combined to achieve a specific implementation and such specific implementation is within the scope of the disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| PA202370115 | Mar 2023 | DK | national |
