Patent Grant
12136208
Specialized dental laboratories typically use computer-aided design (CAD) and computer-aided manufacturing (CAM) milling systems to manufacture dental prostheses based on patient-specific instructions provided by dentists. In a typical work flow, the dental laboratories receive information about a patient's oral situation from a dentist. Using this information, the dental laboratory designs a dental prosthesis on the CAD system and manufactures the prosthesis on the CAM system with a mill or other fabrication system. To use the CAD/CAM system, a digital model of the patient's dentition is required as an input to the process.
Although digitizing a physical dental impression can provide a digital dental model for a CAD/CAM system, digital dental impressions can contain extraneous data such as remnants of an impression tray or other regions not useful for dental processing which can interfere with viewing useful information.
A computer-implemented method of processing a digital jaw scan includes: receiving a digital model comprising a digital jaw; performing segmentation on at least a portion of the digital jaw to provide one or more digital surface segments; determining a digital tooth center for at least one digital tooth in the digital jaw; determining a digital tooth area around the digital tooth center; deleting one or more surface segments not intersecting with the digital tooth area; and smoothing a mesh boundary.
A system of processing a digital jaw scan includes: a processor; and a computer-readable storage medium comprising instructions executable by the processor to perform steps including: receiving a digital model comprising a digital jaw; performing segmentation on at least a portion of the digital jaw to provide one or more digital surface segments; determining a digital tooth center for at least one digital tooth in the digital jaw; determining a digital tooth area around the digital tooth center; deleting one or more surface segments not intersecting with the digital tooth area; and smoothing a mesh boundary.
A non-transitory computer readable medium storing executable computer program instructions for processing a digital jaw scan, the computer program instructions including instructions for: receiving a digital model comprising a digital jaw; performing segmentation on at least a portion of the digital jaw to provide one or more digital surface segments; determining a digital tooth center for at least one digital tooth in the digital jaw; determining a digital tooth area around the digital tooth center; deleting one or more surface segments not intersecting with the digital tooth area; and smoothing a mesh boundary.
For purposes of this description, certain aspects, advantages, and novel features of the embodiments of this disclosure are described herein. The disclosed methods, apparatus, and systems should not be construed as being limiting in any way. Instead, the present disclosure is directed toward all novel and nonobvious features and aspects of the various disclosed embodiments, alone and in various combinations and sub-combinations with one another. The methods, apparatus, and systems are not limited to any specific aspect or feature or combination thereof, nor do the disclosed embodiments require that any one or more specific advantages be present or problems be solved.
Although the operations of some of the disclosed embodiments are described in a particular, sequential order for convenient presentation, it should be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed methods can be used in conjunction with other methods. Additionally, the description sometimes uses terms like “provide” or “achieve” to describe the disclosed methods. The actual operations that correspond to these terms may vary depending on the particular implementation and are readily discernible by one of ordinary skill in the art.
As used in this application and in the claims, the singular forms “a,” “an,” and “the” include the plural forms unless the context clearly dictates otherwise. Additionally, the term “includes” means “comprises.” Further, the terms “coupled” and “associated” generally mean electrically, electromagnetically, and/or physically (e.g., mechanically or chemically) coupled or linked and does not exclude the presence of intermediate elements between the coupled or associated items absent specific contrary language.
In some examples, values, procedures, or apparatus may be referred to as “lowest,” “best,” “minimum,” or the like. It will be appreciated that such descriptions are intended to indicate that a selection among many alternatives can be made, and such selections need not be better, smaller, or otherwise preferable to other selections.
In the following description, certain terms may be used such as “up,” “down,” “upper,” “lower,” “horizontal,” “vertical,” “left,” “right,” and the like. These terms are used, where applicable, to provide some clarity of description when dealing with relative relationships. But, these terms are not intended to imply absolute relationships, positions, and/or orientations. For example, with respect to an object, an “upper” surface can become a “lower” surface simply by turning the object over. Nevertheless, it is still the same object.
Some embodiments can include a computer-implemented method of processing a digital jaw model. Some embodiments can include receiving a digital model that can include a digital jaw, for example.
In some embodiments, the digital jaw model can be generated by scanning a physical impression using any scanning technique known in the art including, but not limited to, for example, optical scanning, CT scanning, etc. or by intraoral scanning of the patient's mouth (dentition). A conventional scanner typically captures the shape of the physical impression/patient's dentition in 3 dimensions during a scan and digitizes the shape into a 3 dimensional digital model. The digital jaw model can each include multiple interconnected polygons in a topology that corresponds to the shape of the physical impression/patient's dentition, for example. In some embodiments, the polygons can include two or more digital triangles. In some embodiments, the scanning process can produce STL, PLY, or CTM files, for example that can be suitable for use with a dental design software, such as FastDesign™ dental design software provided by Glidewell Laboratories of Newport Beach, Calif. One example of CT scanning is described in U.S. Patent Application No. US20180132982A1 to Nikolskiy et al., which is hereby incorporated in its entirety by reference.
A computed tomography (CT) scanner uses x-rays to make a detailed image of an object. A plurality of such images are then combined to form a 3D model of the object. A schematic diagram of an example of a CT scanning system 140 is shown in
An example of a suitable scanning system 140 includes a Nikon Model XTH 255 CT Scanner (Metrology) which is commercially available from Nikon Corporation. The example scanning system includes a 225 kV microfocus x-ray source with a 3 μm focal spot size to provide high performance image acquisition and volume processing. The processor 150 may include a storage medium that is configured with instructions to manage the data collected by the scanning system. A particular scanning system is described for illustrative purposes; any type/brand of CT scanning system can be utilized.
During operation of the scanning system 140, the object 146 is located between the x-ray source 142 and the x-ray detector 148. A series of images of the object 146 are collected by the processor 150 as the object 146 is rotated in place between the source 142 and the detector 146. An example of a single radiograph 160 is shown in
The plurality of radiographs 160 of the object 146 are generated by and stored within a storage medium contained within the processor 150 of the scanning system 140, where they may be used by software contained within the processor to perform additional operations. For example, in an embodiment, the plurality of radiographs 160 can undergo tomographic reconstruction in order to generate a 3D virtual image 170 (see
In some embodiments, the digital jaw model can also be generated by intraoral scanning of the patient's dentition, for example. In some embodiments, each electronic image is obtained by a direct intraoral scan of the patient's teeth. This will typically take place, for example, in a dental office or clinic and be performed by a dentist or dental technician. In other embodiments, each electronic image is obtained indirectly by scanning an impression of the patient's teeth, by scanning a physical model of the patient's teeth, or by other methods known to those skilled in the art. This will typically take place, for example, in a dental laboratory and be performed by a laboratory technician. Accordingly, the methods described herein are suitable and applicable for use in chair side, dental laboratory, or other environments.
A conventional scanner typically captures the shape of the physical impression/patient's dentition in 3 dimensions during a scan and digitizes the shape into a 3 dimensional digital model. The 3 dimensional digital model can include multiple interconnected polygons in a topology that corresponds to the shape of the physical impression/patient's dentition, for example, for a responding jaw. In some embodiments, the polygons can include two or more digital triangles. In some embodiments, the scanning process can produce STL, PLY, or CTM files, for example that can be suitable for use with a dental design software, such as FastDesign™ dental design software provided by Glidewell Laboratories of Newport Beach, Calif.
In some embodiments, processing the digital jaw scan can include performing segmentation on at least a portion of the digital jaw to provide one or more digital surface segments. In some embodiments, performing segmentation can include determining one or more geometric properties of boundaries between the one or more digital surface segments. In some embodiments, the geometric boundary can include a convex boundary in the case of a physical dental impression, for example. In some embodiments, the convex boundary can include the most convex portion of the digital surface. In the case of a direct scan of teeth such as with an optical scanner such as an intra oral scanner, for example, the boundary can be the most concave boundary between the one or more digital surface segments.
In some embodiments, performing segmentation can include performing curvature-based segmentation. In some embodiments, curvature-based segmentation can include curvature determination of digital surface regions in the digital model. In some embodiments, the computer-implemented method can receive a digital model and determine curvatures of digital surface regions. The computer-implemented method can determine curvature of digital surface regions using any technique. In some embodiments, curvature determination can be performed by the computer-implemented method automatically.
In some embodiments, the digital surface regions include triangles. The curvature of a triangle can be determined by taking an average of the curvature of the triangle's edges, or an average of the curvature of the triangle's vertices.
In some embodiments, the computer-implemented method can determine the curvature of the triangle by taking an average of the curvature of its edges.
Alternatively, in some embodiments, the computer-implemented method can determine the curvature of the triangle by taking an average of the curvature of the triangle's vertices. For example, in some embodiments, the computer-implemented method can determine curvature at each vertex P by selecting a neighborhood of vertices (size N) around P, optionally using connection information to decrease the search space. The computer implemented method can fit a quadric patch F(x,y,z)=0 onto the neighborhood of points. The computer implemented method can determine a projection P0 of P onto the patch, such that F(P0)=0. The computer-implemented method can determine the curvature properties of F at P0 and assign the curvature properties to P.
In some embodiments, the computer-implemented method can, for example, use quadric form ax2+by2+cz2+2exy+2fyz+2gzx+2lx+2my+2nz+d=0 since each datum (x,y,z) will not lie perfectly on the surface of F. The computer-implemented method can determine the coefficients of the patch surface (a, b, c, e, f, g, l, m, n, d), from a 10×10 real symmetric eigenproblem of the form A=DTD, where Di is the N×10 design matrix, each row of which is built up by [xi2 yi2 zi2 xiyi yizi xizi xi yi zi 1], where i=1, . . . , N. The matrix can have 10 real eigenvalues and 10 corresponding eigenvectors. The coefficients of the eigenvector corresponding to the smallest eigenvalue λ1 are the coefficients a, b, c, e, f, g, l, m, n, d of the quadric surface that best approximates the point cloud locally around P. The computer-implemented method uses a, b, c, e, g, l, m, n to determine values E, F, G, L, M, N by letting F(x,y,z)=ax2+by2+cz2+exy+fyz+gxz+lx+my+nz+d=0, an implicit quadric surface in R3, so that first order partial derivatives are Fx=2ax+ey+gz+l, Fy=2by+ex+fz+m, and Fz=2cz++gx+n. The coefficients E, F, G are determined as E=1+Fx2/Fz2, F=FxFy/Fz2, and G=1+Fy2/Fz2. Since second order partial derivatives are Fxx=2a, Fyy=2b, Fzz=2c, Fxy=Fyx=e, Fyz=Fzy=f, and Fxz=Fzx=g and the magnitude of the gradient is |∇F|=√{square root over (Fx2+Fy2+Fz2)}, then coefficients L, M, N of the Second Fundamental Form are:
The computer-implemented method then determines matrices A and B from E, F, G, L, M, N as:
and determines principle curvatures k1 and k2 as the eigenvalues of the matrix B−1*A.
The computer-implemented method can apply a selected scalar function to the principal curvatures k1 and k2 to determine the selected curvature function (“SCF”). For example, for principle curvatures k1 and k2, the computer-implemented method can determine Gaussian curvature (K) as K=k1 k2 or mean curvature (H) as H=½(k1+k2).
The radius of either method of determining curvature can be up to and including 60 digital vertices on average in the neighborhood of the vertex being evaluated, and can be a user selectable value. A selection of a smaller number of points and smaller radius can lead to faster computations, while selecting a larger number of points and larger radius can provide a more precise curvature estimation. The computer-implemented method can be repeated for all vertices of the digital surface mesh, for example.
In some embodiments, the computer-implemented method can segment the entire digital dental impression surface into one or more digital segments. In some embodiments, the computer-implemented method can segment the digital dental impression surface in three dimensions (3D) using curvature based segmentation. This can include, for example, watershed segmentation. Segmentation can be performed by the computer-implemented method automatically in some embodiments.
In some embodiments, the digital dental impression surface can include one or more triangles that connect at edges and vertices to form the digital surface mesh. In some embodiments, the computer-implemented method determines the curvature of every triangle in the digital surface mesh. The computer-implemented method can determine the curvature of each particular triangle by either determining the average curvature of the particular triangle's vertices or the average curvature of the particular triangle's edges as described previously.
In one embodiment, the computer-implemented method can determine the curvature of a particular triangle by determining a curvature at each of the edge of the particular triangle and calculating an average of the edge curvatures as discussed earlier of the present disclosure.
In some embodiments, the computer-implemented method can assign a user-selectable positive or negative sign to each triangle's curvature. For example, the sign of convex and convex regions can be set arbitrarily. In some embodiments, the computer-implemented method can assign convex regions a positive sign.
After determining each particular triangle's curvature, the computer-implemented method can segment triangles based on 3D curvature-based segmentation. In some embodiments, performing segmentation can include performing watershed segmentation to generate one or more initial digital surface segments. For example, in some embodiments, the computer-implemented method can determine the curvature for each triangle. The curvature of each triangle can, in some embodiments, be stored in a lookup table. The computer implemented-method can start with a triangle with a minimum curvature as a particular triangle being evaluated. The computer-implemented method can look up the curvatures of triangles in the neighborhood of the particular triangle being evaluated from the look up table, for example. In some embodiments, the computer-implemented method can determine neighboring triangle curvatures from the look-up table. Any neighboring triangles with curvatures greater than the particular triangle being evaluated can be added to a segment to which the particular triangle being evaluated belongs. Any neighboring triangles with curvatures less than the curvature of the particular triangle are not added to the particular triangle's segment. The computer-implemented method can then select a neighborhood triangle as the next particular triangle to be evaluated and repeats the process for every triangle.
The computer-implemented method next can compare the curvature of neighboring triangle 2404 with the curvature of the particular triangle 2402, for example. If, for example, the curvature of neighboring triangle 2408 is greater than the minimum curvature (i.e. the curvature of 2402), then the triangle 2408 is merged with the segment 2411 containing triangle 2402. As illustrated in
If a neighborhood triangle has a lower curvature than the particular triangle 2402 in question, then the neighborhood triangle is not merged with the segment containing the particular triangle 2402 by the computer-implemented method. For example, if neighboring triangle 2404 has a lower curvature than the triangle 2402, then 2404 is not merged with the segment 2412 to which particular triangle 2402 belongs.
After processing a first particular triangle, the computer-implemented method changes to a new particular triangle which can be a neighboring triangle of the first particular triangle. The computer-implemented method can repeat determining segmentation with the new particular triangle being evaluated and segment the entire digital surface.
After performing segmentation of triangles, the digital surface mesh can contain a large number of small segments as illustrated in
In some embodiments, the computer-implemented method can merge small segments into larger ones based on geometric attributes such as their average curvature, average size, area, perimeter, perimeter to area ratio, and/or other geometric factors. In some embodiments, the computer-implemented method can merge the one or more initial digital surface segments into one or more merged digital surface segments.
In some embodiments, merging can include determining a merge metric for each pair of adjacent initial digital surface segments based on a perimeter of the after-merged segment and an average mean curvature on a boundary between the pair of adjacent initial digital surface segments. For example, a merge metric can be determined as follows in some embodiments:
m=−p*c
where m is the merge metric, p is the perimeter of the proposed after-merged segment, and c is the average mean curvature on a boundary between the two segments to be merged. In some embodiments, this is merge metric used in the case of digital models of dental impressions. In some embodiments, in the case of digital models of directly scanned dentition, the merge priority is m=p*c.
In some embodiments, merging can include merging adjacent initial digital surface segments based on the merge metric. In some embodiments, merging can include merging adjacent initial digital surface segments starting from the greatest merge metric value and then updating the merge metrics after the merge occurs. In some embodiments, merging terminates when the largest merge metric falls below a user-configurable merge metric threshold. In some embodiments, the user-configurable merge metric threshold is negative 500. In some embodiments, smaller initial digital surface segments can be prioritized during merging. In some embodiments, smaller segments are merged first due to the p term, as are segments with small curvature in between due to the c term. The result of merging can be, for example, a few large segments with high convex curvature between the segments in the case of digital models of dental impressions, and high concave curvature between the segments in the case of digital models of directly scanned dentition.
Merging can be performed automatically by the computer-implemented method in some embodiments.
In some embodiments, the computer-implemented method determines a merge-priority for every two neighboring segments. The computer-implemented method can determine merge-priority of two neighboring segments based on their attributes. If two segments can merge based on their attributes, then in some embodiments the computer-implemented method determines priority based on geometric factors. For example, the computer-implemented method can determine priority based on −p*c as discussed earlier in the case of digital models of dental impressions, and based on p*c as discussed earlier in the case of digital models of directly scanned dentition such as intraoral scanners, for example.
In some embodiments, the computer-implemented method can store priorities in a priority-queue. The computer-implemented method can extract the highest priority from the queue, merge the corresponding two segments, and update the priorities between newly formed segments and their neighbors in the queue. The computer-implemented method can repeat this process until no two segments can be merged any more.
Some embodiments of the computer-implemented method can include determining a digital tooth center for at least one digital tooth in the digital jaw. In some embodiments, determining the digital tooth center can include determining a bounding region for one or more digital teeth in the digital jaw. In some embodiments, the computer-implemented method can determine a bounding region by using a trained neural network. In some embodiments, the bounding region can be a bounding box, for example.
Neural networks are computational models that are part of machine learning. A neural network typically includes nodes organized in layers. The layers of nodes can include, for example, an input layer, one or more hidden layers, and an output layer. A neural network with more than one hidden layer—typically many more hidden layers—is a deep neural network (“DNN”). Information from one layer can be processed and provided to a next layer.
In some embodiments, the DNN can be a convolutional neural network (“CNN”), which is a network that uses convolution in place of the general matrix multiplication in at least one of the hidden layers of the deep neural network. A convolution layer can calculate its output values by applying a kernel function to a subset of values of a previous layer. The computer-implemented method can train the CNN by adjusting weights of the kernel function based on the training data. The same kernel function can be used to calculate each value in a particular convolution layer. One advantage to using a CNN can include learning fewer weights during training. Another advantage of using a CNN can be detecting edge features, for example.
The CNN can also include one or more pooling layers such as first pooling layer 212. First pooling layer can apply a filter such as pooling filter 214, to the first convoluted image 206. Any type of filter can be used. For example, the filter can be a max filter (outputting the maximum value of the pixels over which the filter is applied) or an average filter (outputting the average value of the pixels over which the filter is applied). The one or more pooling layer(s) can down sample and reduce the size of the input matrix. For example, first pooling layer 212 can reduce/down sample first convoluted image 206 by applying first pooling filter 214 to provide first pooled image 216. The first pooled image 216 can include one or more feature channels 217. The CNN can optionally apply one or more additional convolution layers (and activation functions) and pooling layers. For example, the CNN can apply a second convolution layer 218 and optionally an activation function to output a second convoluted image 220 that can include one or more feature channels 219. A second pooling layer 222 can apply a pooling filter to the second convoluted image 220 to generate a second pooled image 224 that can include one or more feature channels. The CNN can include one or more convolution layers (and activation functions) and one or more corresponding pooling layers. The output of the CNN can be optionally sent to a fully connected layer, which can be part of one or more fully connected layers 230. The one or more fully connected layers can provide an output prediction such as output prediction 224. In some embodiments, the output prediction 224 can include labels of teeth and surrounding tissue, for example.
In some embodiments, the neural network can receive a 2D depth map of the 3D digital model and provide a bounding region around one or more digital teeth in the 2D depth map. In some embodiments, determining a digital tooth area is based off of a depth map. In some embodiments, the depth map is generated a depth map based on an occlusion direction. In some embodiments, the computer-implemented method can generate the 2D depth map. In some embodiments, the computer-implemented method can determine an occlusion direction. In some embodiments, the bounding region can be a bounding box, for example.
Some embodiments of the computer-implemented method can include generating a 2D image from the 3D digital model. In some embodiments, the 2D image can be a 2D depth map. The 2D depth map can include a 2D image that contains in each pixel a distance from an orthographic camera to an object along a line passing through the pixel. The object can be, for example, a digital jaw model surface, in some embodiments, for example. In some embodiments, an input can include, for example, an object such as a 3D digital model of patient's dentition (“digital model”), such as a jaw, and a camera orientation. In some embodiments, the camera orientation can be determined based on an occlusion direction. The occlusal direction is a normal to an occlusal plane and the occlusal plane can be determined for the digital model using any technique known in the art. For example, one technique is described in AN AUTOMATIC AND ROBUST ALGORITHM OF REESTABLISHMENT OF DIGITAL DENTAL OCCLUSION, by Yu-Bing Chang, James J. Xia, Jaime Gateno, Zixiang Xiong, Fellow, IEEE, Xiaobo Zhou, and Stephen T. C. Wong in IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 29, NO. 9, September 2010, the entirety of which is incorporated by reference herein. Alternatively, in some embodiments, the occlusal direction can be specified by a user using an input device such as a mouse or touch screen to manipulate the digital model on a display, for example, as described herein. In some embodiments, the occlusal direction can be determined, for example, using the Occlusion Axis techniques described in PROCESSING DIGITAL DENTAL IMPRESSION U.S. patent application Ser. No. 16/451,968, of Nikolskiy et al., the entirety of which is incorporated by reference herein.
The 2D depth map can be generated using any technique known in the art, including, for example z-buffer or ray tracing. For example, in some embodiments, the computer-implemented method can initialize the depth of each pixel (j, k) to a maximum length and a pixel color to a background color, for example. The computer-implemented method can for each pixel in a polygon's projection onto a digital surface such as a 3D digital model determine a depth, z of the polygon at (x, y) corresponding to pixel (j, k). If z<depth of pixel (j, k), then set the depth of the pixel to the depth, z. “Z” can refer to a convention that the central axis of view of a camera is in the direction of the camera's z-axis, and not necessarily to the absolute z axis of a scene. In some embodiments, the computer-implemented method can also set a pixel color to something other than a background color for example. In some embodiments, the polygon can be a digital triangle, for example. In some embodiments, the depth in the map can be per pixel.
In some embodiments, the neural network can be trained by providing a 2D depth map training dataset that can include one or more 2D depth maps of at least a portion of a digital dental arch having one or more digital teeth, with each digital tooth marked with a marked digital tooth bounding region, such as a rectangular shaped boundary, for example. Other shapes for the marked digital tooth bounding region can also be used. To generate the training dataset, each digital tooth in each 2D depth map in the training dataset can be marked by a digital tooth bounding region.
CNNs can be structured and used in different ways. In some embodiments, the neural network can include a YOLO neural network. For example, details of an example of a You Only Look Once (“YOLO”) network are described in You Only Look Once: Unified, Real-Time Object Detection, by Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, submitted 8 Jun. 2015, last revised 9 May 2016, v5, the entirety of which is hereby incorporated by reference. Additional details of a YOLO network can be found in YOLO9000: Better, Faster, Stronger, by Joseph Redmon, Ali Farhadi, University of Washington, Allen Institute for AI, published 25 Dec. 2016, arXiv, the entirety of which is hereby incorporated by reference. Details of an example of a YOLO network are also described in YOLOv3: An Incremental Improvement, by Joseph Redmon and Ali Farhadi, University of Washington, published 2018, ArXiv, the entirety of which is hereby incorporated by reference. A trained YOLO network can receive, for example, a 2D digital model of patient's dentition and output the digital model with a digital bounding box as the digital tooth bounding region around each digital tooth.
A YOLOv3 network (hereinafter, “YOLO network” or “YOLO”) as described can include one or more convolutional networks that predict multiple bounding boxes and class probability for each bounding box. In some embodiments, the YOLO network can divide an input image into a S×S grid. Each of grid cells can predict B bounding boxes and can determine confidence scores for the bounding boxes. The confidence scores can indicate the model's confidence that the bounding box contains an object as well as the accuracy of the predicted box. Confidence can be expressed as Pr(Object)*IOUpredtruth, where IOU is intersection over union.
In some embodiments, YOLO can use dimension clusters as anchor boxes to predict bounding boxes. For example, YOLO can predict four coordinates for a bounding box: tx, ty, tw, th. If a cell is offset from the left top corner of the image by (Cx,Cy) and a prior bounding box has width pw and a height ph, the predictions can correspond to:
bx=σ(tx)+cx
by=σ(ty)+cy
bw=pwet
bh=phet
where box center coordinates relative to the filter application location are predicted using a sigmoid function (providing σ). In some embodiments, YOLO can predict each bounding box's objectness score by performing logistic regression. The result can be 1 if the prior bounding box overlaps a ground truth object by more than any other prior bounding box. A prior bounding box that is not best but that overlaps a ground truth object by more than a threshold such as 0.5 can be ignored. Other threshold values can be used and can be set in a user configurable file, for example. A prior bounding box not assigned to a ground truth object incurs a loss for objectness, but not coordinate or class prediction. In some embodiments, each box can predict classes within the bounding box by utilizing multilabel classification. For example, independent logistic classifiers can be used. Binary cross-entropy loss for class predictions can be used in training. YOLO can make predictions across scales. For example, YOLO can predict boxes at three different scales. Features can be extracted from the scales. Several convolutional layers can be added to the base feature extractor. The final convolutional layer can predict a 3D tensor encoding bounding box, objectness, and class predictions. The tensor can be N×N×[(number of boxes at each scale)*(4+1+(number of class predictions))]. For example, the number of boxes at each scale can be 3, and the class prediction number can be 80 class predictions. YOLO can obtain a feature map from two layers previous and up-sample the feature map. For example, YOLO can up-sample the feature map by 2×. Another previous feature map can be concatenated with the up-sampled feature map to provide a combined feature map. One or more convolutional layers can be added to process the combined feature map and provide a second tensor of twice the size. The same design can be performed a final time to predict boxes for the final scale. K-means clustering can be used to determine prior bounding box values. For example, 9 clusters and 3 scales can be used and the clusters divided evenly across the scales.
In some embodiments, YOLO can perform feature extraction using one or more convolution layers. One or more of the convolution layers can optionally include residual operations.
3 × 3/2
3 × 3/2
3 × 3/2
3 × 3/2
3 × 3/2
Layer 310 can be performed 2×, Layer 314 can be performed 8×, layer 318 can be performed 8×, and layer 322 can be performed 4×, bringing the total number of convolutions for the entire network to 53 convolutions. The avgpool can be global. Other arrangements and variations are also contemplated in the YOLO network. In some embodiments, a trained YOLO network can receive an image and provide bounding regions around each feature in the image. The features can be defined during training. YOLO training can include minimizing loss functions. The loss function only penalizes classification errors when an object is in the particular grid cell. The loss function penalizes bounding box coordinate errors if a particular predictor is responsible for the ground truth box. For example, if the particular predictor has the highest IOU of all predictors in the particular grid cell.
In some embodiments, the computer-implemented method can train a YOLO network with one or more 2D depth maps, each with marked digital tooth bounding regions shaped as rectangles or boxes. In some embodiments, the training dataset can include 10,000 2D depth map images, for example. Other suitable numbers of 2D depth map images can be used as the training dataset in some embodiments, for example.
After training, in some embodiments, the 2D depth map trained neural network can receive one or more unmarked 2D depth maps each having a digital dental arch and provide a digital tooth bounding region for each digital tooth in at least a portion of each digital dental arch. In some embodiments, the computer-implemented method can use the trained neural network to roughly define a digital tooth bounding region around each digital tooth, for example. Each digital tooth bounding region can provide a rough approximation of the position of each tooth when viewed from an occlusal direction.
In some embodiments, the 2D depth map trained neural network is a 2D depth map trained convolutional neural network as described previously. In some embodiments, the 2D depth map trained CNN can be a 2D depth map trained YOLO network as described previously. The trained 2D depth map YOLO network can receive a 2D depth map and can provide a digital tooth bounding region for each digital tooth in at least a portion of the 2D depth map. The computer-implemented method can label all pixels bounded by a digital tooth bounding region with a unique label in some embodiments for example. The digital tooth bounding regions provided by a trained 2D depth map YOLO network can be digital tooth bounding boxes, for example. Thus, in some embodiments, the computer-implemented method can receive a 2D depth map and, using one or more 2D depth map trained neural networks, label one or more regions of the 2D depth map to provide the labeled 2D depth map. The trained neural network can be a YOLO network. In some embodiments, the computer-implemented method can receive a 2D depth map and, using one or more 2D depth map trained neural networks, label one or more regions of the 2D depth map to provide the labeled 2D depth map.
Although certain values and arrangements are discussed for one or more features in the one or more neural networks, the values are provided as examples only. Other suitable values, arrangements, and variations are contemplated and can be used.
In some embodiments, the computer-implemented method can train and use any CNN to receive a 2D depth map and determine a digital tooth bounding region for each digital tooth in the 2D depth map. For example, other CNNs such as RetinaNet, Feature Pyramid Network (“FPN”), Fast Region-based Convolutional Network (“FRCN”), Region-based Fully Convolutional Network (“R-FCN”), or any other type of CNN known in the art can be trained and used as described in the present disclosure in place of the YOLO network.
An example of RetinaNet can be found in “Focal Loss for Dense Object Detection”, Lin, Tsung-Yi & Goyal, Priyal & Girshick, Ross & He, Kaiming & Dollar, Piotr, (2018), IEEE Transactions on Pattern Analysis and Machine Intelligence, PP. 1-1, the entirety of which is hereby incorporated by reference. An example of a Feature Pyramid Network can be found in “Feature Pyramid Networks for Object Detection,” T. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan and S. Belongie, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 936-944, the entirety of which is hereby incorporated by reference. An example of a Fast Region-based Convolutional Network can be found in “Fast R-CNN,” R. Girshick, 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 2015, pp. 1440-1448, the entirety of which is hereby incorporated by reference. An example of a Region-based Fully Convolutional Network can be found in “R-FCN: Object Detection via Region-based Fully Convolutional Networks”, Jifeng Dai, Yi Li, Kaiming He, Jian Sun, 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, the entirety of which is hereby incorporated by reference.
In some embodiments, the computer-implemented method can determine a digital tooth center for each digital tooth from the bounding boxes. In some embodiments, the computer-implemented method can determine a bounding box center as the digital tooth center.
Some embodiments of the computer-implemented method can include determining a digital tooth area around the digital tooth center. In some embodiments, determining the digital tooth area can include constructing a proximate region comprising one or more points within a center region distance from the digital tooth center along the digital surface. In some embodiments, the center region distance is a digital surface distance that includes a tooth corresponding to the digital tooth center. In some embodiments, the computer-implemented method can determine a digital tooth associated with each digital tooth center based on a user-configurable center region distance from the digital tooth center. In some embodiments, the center region distance can be 3 mm, for example. However, other suitable values for the center region distance can also be used to determine the digital tooth corresponding to a particular digital tooth center. For example,
Some embodiments can include deleting one or more surface segments not intersecting with the digital tooth area. Some embodiments can include deleting all merged segments located greater than the center region distance from the digital tooth center.
The computer-implemented method determines a first proximate region by generating first region paths 8606 and 8608 from the initial digital tooth center 8604 and extending along the digital surface until reaching the first region endpoints 8610 and 8612, respectively. In this example, the first region endpoints 8610 and 8612 are located at a cutoff value of a cutoff distance from the initial digital tooth center 8604. In some embodiments, this can be, for example, 3 mm. Other suitable values can be used as well.
The computer-implemented method determines a second proximate region by generating second region paths 8614 and 8616 from the first region endpoints 8610 and 8612 and extending along the digital surface until reaching the second region endpoints 8618 and 8620, respectively. In this example, the second region endpoints 8618 and 8620 are located at a cutoff value corresponding to lowest points on the digital surface with respect to the occlusion axis. The computer-implemented method can delete all regions outside of the tooth and gum region from the digital dental impression by, for example, retaining only the first and second digital surface proximate region which in some embodiments include only teeth and gum regions. The computer-implemented method in some embodiments thereby deletes or removes extraneous regions, retaining the teeth and gums.
In some cases, after segmentation, the digital jaw can include a digital boundary region that can be uneven and rough in appearance.
In some embodiments, smoothing can include performing morphological opening on at least a portion of the segmented digital jaw model. In some embodiments, morphological opening can include performing erosion and dilation at a boundary region. For example, in some embodiments, the computer-implemented method can virtually erode the triangles near the boundary to a user configurable erosion distance and then virtually dilate them back a user configurable dilation distance. In some embodiments, the erosion distance and the dilation distance can be the same. In some embodiments, the erosion distance and the dilation distance can be, for example, 3 mm. Other suitable erosion distances and dilation distances can be used. In some embodiments, the computer-implemented method can delete the triangles not reached by the dilation.
In some embodiments, the computer-implemented method can perform erosion starting at one or more mesh boundary vertices. In some embodiments, erosion can include probing the boundary region using a pre-defined shape as a structuring element. In some embodiments, the computer-implemented method can virtually construct the structuring element as a union of all disks around boundary vertices and subtract the union of all disks from the surface region. In some embodiments, a surface disk can be a user-configurable radius around a central point, containing all other points with distance along the surface within the radius. In some embodiments, the radius can be, for example, 3 mm. Other suitable radius values are also contemplated and can be used. In some embodiments, the computer-implemented method can perform erosion on a boundary region, A, using a structuring element, B as follows:
A⊖B={z∈E|Bz⊆A}
where Bz is a translation of B by vector z. In some embodiments, erosion of boundary region A by structured element B can be points reached by the center of B as B moves within A. Other techniques to perform erosion can also be used.
In some embodiments, the computer-implemented method can perform dilation by constructing the union of all disks around boundary vertices after subtraction and add the union to the boundary surface region. In some embodiments, the computer-implemented method can perform dilation as follows:
A⊕B∪b∈BAb
where A is the boundary region and B is the structured element. In some embodiments, B has a center as its origin, so that dilation of A by B is all points covered by B when the center of B moves within A. Other techniques to perform dilation can also be used.
In some embodiments, the computer-implemented method can virtually construct a union of all disks using Dijkstra's shortest path searching algorithm with multiple start vertices. In some embodiments, the multiple start vertices can be one or more boundary vertices, for example.
In some embodiments, the computer-implemented method can perform Dijkstra's algorithm as follows:
1. Set one or more initial digital surface points. In some embodiments, the one or more initial digital surface points can be boundary vertices.
2. Mark all digital surface points as unvisited.
3. Assign every digital surface point—a tentative distance value. The tentative distance value for the initial digital surface point is assigned to zero and the tentative distance value for all other digital surface points on the one side is assigned to infinity or the highest possible value or larger than the sum of all edge lengths, for example. Set one of initial digital surface points as the current digital surface point.
4. For the current digital surface point, consider all unvisited neighboring digital surface points on the one side and determine their calculated tentative distances (e.g. edge length between the current digital surface point and the particular unvisited neighboring digital surface point) through the current digital surface point. In some embodiments, the calculated tentative distance can determine an edge length between current digital surface point and the particular unvisited neighboring digital surface point. In some embodiments, edge length is a Euclidean length.
5. Compare the newly calculated tentative distance to the current assigned value and assign the smaller one. For example, if the current digital surface point A is marked with a distance of 6, and the edge connecting it with a neighboring digital surface point B has length 2, then the distance to B through A will be 6+2=8. If B was previously marked with a distance greater than 8 then change it to 8. Otherwise, keep the current value.
6. After considering all of the unvisited neighbors of the current digital surface point, mark the current digital surface point as visited and remove it from the unvisited set. A visited digital surface point will never be checked again.
7. If the destination digital surface point has been marked visited (when planning a route between two specific digital surface points) or if the smallest tentative distance among the digital surface points in the unvisited set is infinity, or the highest possible value, or larger than the sum of all edge lengths for example (when planning a complete traversal; occurs when there is no connection between the initial digital surface point and remaining unvisited digital surface points), then stop. The algorithm has finished. In some embodiments, the algorithm also stops when the smallest tentative distance becomes larger than a user-configurable max distance. In some embodiments, the max distance can be, for example, 3 mm. Other suitable max distance values are contemplated and can be used.
Otherwise, select the unvisited digital surface point that is marked with the smallest tentative distance, set it as the new “current digital surface point”, and go back to step 4.
In some embodiments, smoothing the mesh boundary can include performing Laplacian smoothing on the segmented digital jaw model. In some embodiments, the computer-implemented method can perform Laplacian smoothing by moving each of one or more boundary vertices to the middle between its two respective neighboring vertices. In some embodiments, Laplacian smoothing can be performed by the computer-implemented method as follows:
where N represents the number of vertices adjacent to node i,
Laplacian smoothing of one or more boundary vertices can continue for a user-configurable number of iterations. In some embodiments, the number of Laplacian smoothing iterations can be 5, for example. Other suitable number of Laplacian smoothing iterations can also be used in some embodiments. In an alternative embodiment, the computer-implemented method can determine a shift of each mesh vertex during smoothing and stop iterations if no vertex has moved more than a user-configurable threshold iteration distance. For example, in some embodiments, the threshold iteration distance can be 0.01 mm. Other suitable threshold iteration distance values can be used.
In some embodiments, the computer-implemented method can train one or more neural networks. In some embodiments, the computer-implemented method can implement one or more neural networks. In some embodiments, the computer-implemented method can implement one or more features in the present disclosure.
The computer-implemented method can include one or more other features in various combinations. In some embodiments, segmentation can include curvature-based segmentation. In some embodiments, performing segmentation can include determining one or more geometric properties of boundaries between the one or more digital surface segments. In some embodiments, the geometric boundary can include a convex boundary. In some embodiments, further comprising merging one or more initial digital surface segments into one or more merged digital surface segments based on a merge metric. In some embodiments, the merge metric is based on a perimeter of the after-merged segment and an average mean curvature on a boundary between the pair of adjacent initial digital surface segments. In some embodiments, determining the digital tooth center can include determining a bounding region for one or more digital teeth in the digital jaw. In some embodiments, the digital tooth center is determined by using a trained neural network.
Some embodiments include a processing system for processing a digital jaw scan that can include a processor, a computer-readable storage medium including instructions executable by the processor to perform steps including: receiving a digital model comprising a digital jaw; performing segmentation on at least a portion of the digital jaw to provide one or more digital surface segments; determining a digital tooth center for at least one digital tooth in the digital jaw; determining a digital tooth area around the digital tooth center; deleting one or more surface segments not intersecting with the digital tooth area; and smoothing a mesh boundary.
One or more advantages of one or more features in the present disclosure can include, for example, automatic clean-up of jaw scans without requiring a user to manually select regions or perform manual steps. One or more advantages of one or more features can include, for example, a more accurate digital dental model reflecting a patient's dentition. One or more advantages of one or more features can include, for example, improved speed in cleaning up a digital surface of a digital model compared to, for example, manual processing performed by a user. One or more advantages of one or more features can include, for example, not requiring precise tooth segmentation, thereby decreasing processing time and using less processing resources. One or more advantages of one or more features can include, for example, removing at least a portion of non-relevant information from a digital jaw model.
In some embodiments, processing a digital jaw scan as disclosed in the present disclosure can be initiated by a user, for example. In some embodiments, processing a digital jaw scan can include one or more of the features described in the present disclosure. In some embodiments, processing a digital jaw scan can be performed by a user using an input device while viewing the digital model on a display, for example. In some embodiments, the computer-implemented method can allow the input device to manipulate the digital model displayed on the display. For example, in some embodiments, the computer-implemented method can rotate, zoom, move, and/or otherwise manipulate the digital model in any way as is known in the art. In some embodiments, processing a digital jaw scan can be performed by a user using the input device. In some embodiments, processing a digital jaw scan can be initiated, for example, using techniques known in the art, such as a user selecting another graphical user interface element such as a graphical button.
One or more of the features disclosed herein can be performed and/or attained automatically, without manual or user intervention. One or more of the features disclosed herein can be performed by a computer-implemented method. The features—including but not limited to any methods and systems—disclosed may be implemented in computing systems. For example, the computing environment 14042 used to perform these functions can be any of a variety of computing devices (e.g., desktop computer, laptop computer, server computer, tablet computer, gaming system, mobile device, programmable automation controller, video card, etc.) that can be incorporated into a computing system comprising one or more computing devices. In some embodiments, the computing system may be a cloud-based computing system.
For example, a computing environment 14042 may include one or more processing units 14030 and memory 14032. The processing units execute computer-executable instructions. A processing unit 14030 can be a central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In some embodiments, the one or more processing units 14030 can execute multiple computer-executable instructions in parallel, for example. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example, a representative computing environment may include a central processing unit as well as a graphics processing unit or co-processing unit. The tangible memory 14032 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two, accessible by the processing unit(s). The memory stores software implementing one or more innovations described herein, in the form of computer-executable instructions suitable for execution by the processing unit(s).
A computing system may have additional features. For example, in some embodiments, the computing environment includes storage 14034, one or more input devices 14036, one or more output devices 14038, and one or more communication connections 14037. An interconnection mechanism such as a bus, controller, or network, interconnects the components of the computing environment. Typically, operating system software provides an operating environment for other software executing in the computing environment, and coordinates activities of the components of the computing environment.
The tangible storage 14034 may be removable or non-removable and includes magnetic or optical media such as magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory way and can be accessed within the computing environment. The storage 14034 stores instructions for the software implementing one or more innovations described herein.
The input device(s) may be, for example: a touch input device, such as a keyboard, mouse, pen, or trackball; a voice input device; a scanning device; any of various sensors; another device that provides input to the computing environment; or combinations thereof. For video encoding, the input device(s) may be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into the computing environment. The output device(s) may be a display, printer, speaker, CD-writer, or another device that provides output from the computing environment.
The communication connection(s) enable communication over a communication medium to another computing entity. The communication medium conveys information, such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can use an electrical, optical, RF, or other carrier.
Any of the disclosed methods can be implemented as computer-executable instructions stored on one or more computer-readable storage media 14034 (e.g., one or more optical media discs, volatile memory components (such as DRAM or SRAM), or nonvolatile memory components (such as flash memory or hard drives)) and executed on a computer (e.g., any commercially available computer, including smart phones, other mobile devices that include computing hardware, or programmable automation controllers) (e.g., the computer-executable instructions cause one or more processors of a computer system to perform the method). The term computer-readable storage media does not include communication connections, such as signals and carrier waves. Any of the computer-executable instructions for implementing the disclosed techniques as well as any data created and used during implementation of the disclosed embodiments can be stored on one or more computer-readable storage media 14034. The computer-executable instructions can be part of, for example, a dedicated software application or a software application that is accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide-area network, a local-area network, a client-server network (such as a cloud computing network), or other such network) using one or more network computers.
For clarity, only certain selected aspects of the software-based implementations are described. Other details that are well known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any specific computer language or program. For instance, the disclosed technology can be implemented by software written in C++, Java, Perl, Python, JavaScript, Adobe Flash, or any other suitable programming language. Likewise, the disclosed technology is not limited to any particular computer or type of hardware. Certain details of suitable computers and hardware are well known and need not be set forth in detail in this disclosure.
It should also be well understood that any functionality described herein can be performed, at least in part, by one or more hardware logic components, instead of software. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
Furthermore, any of the software-based embodiments (comprising, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed through a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, software applications, cable (including fiber optic cable), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.
In view of the many possible embodiments to which the principles of the disclosure may be applied, it should be recognized that the illustrated embodiments are only examples and should not be taken as limiting the scope of the disclosure.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6140950 | Oprescu | Oct 2000 | A |
| 6409504 | Jones et al. | Jun 2002 | B1 |
| 6701016 | Jojic et al. | Mar 2004 | B1 |
| 7084868 | Farag et al. | Aug 2006 | B2 |
| 7187790 | Sabol et al. | Mar 2007 | B2 |
| 7490085 | Walker et al. | Feb 2009 | B2 |
| 7702492 | Marshall | Apr 2010 | B2 |
| 7708557 | Rubbert | May 2010 | B2 |
| 7860300 | Siltanen et al. | Dec 2010 | B2 |
| 8068580 | Lang et al. | Nov 2011 | B2 |
| 8126726 | Matov et al. | Feb 2012 | B2 |
| 8454362 | Rubbert | Jun 2013 | B2 |
| 8602780 | Rubbert | Dec 2013 | B2 |
| 9417700 | El Dokor et al. | Aug 2016 | B2 |
| 9474582 | Musuvathy et al. | Oct 2016 | B2 |
| 9818186 | Kim et al. | Nov 2017 | B2 |
| 9865176 | Tran | Jan 2018 | B2 |
| 10278584 | Glinec et al. | May 2019 | B2 |
| 10438363 | Kim et al. | Oct 2019 | B2 |
| 10537463 | Kopelman | Jan 2020 | B2 |
| 10542944 | Petersilka | Jan 2020 | B2 |
| 10548690 | Wen | Feb 2020 | B2 |
| 10755409 | Salah et al. | Aug 2020 | B2 |
| 10912530 | Mandelkern et al. | Feb 2021 | B2 |
| 10925691 | Yancey et al. | Feb 2021 | B2 |
| 10932890 | Sant et al. | Mar 2021 | B1 |
| 10984529 | Carter et al. | Apr 2021 | B2 |
| 10997727 | Xue et al. | May 2021 | B2 |
| 11000349 | Willers et al. | May 2021 | B2 |
| 11007040 | Azernikov et al. | May 2021 | B2 |
| 11026766 | Chekh et al. | Jun 2021 | B2 |
| 11049248 | Salah et al. | Jun 2021 | B2 |
| 11051912 | Martz et al. | Jul 2021 | B2 |
| 11051914 | Kopelman et al. | Jul 2021 | B2 |
| 11109945 | Salah et al. | Aug 2021 | B2 |
| 11116466 | Mandelkern et al. | Sep 2021 | B2 |
| 11154267 | Mandelkern et al. | Oct 2021 | B2 |
| 11164045 | Paik et al. | Nov 2021 | B2 |
| 11189021 | Shah et al. | Nov 2021 | B2 |
| 11191508 | Mandelkern et al. | Dec 2021 | B2 |
| 11194312 | Cramer et al. | Dec 2021 | B2 |
| 11232573 | Brown et al. | Jan 2022 | B2 |
| 11238586 | Minchenkov et al. | Feb 2022 | B2 |
| 11250580 | Meyer et al. | Feb 2022 | B2 |
| 11270523 | Long et al. | Mar 2022 | B2 |
| 11278375 | Wang et al. | Mar 2022 | B2 |
| 11291404 | Schnabel et al. | Apr 2022 | B2 |
| 11291532 | Azernikov et al. | Apr 2022 | B2 |
| 11399792 | Mandelkern et al. | Aug 2022 | B2 |
| 11403813 | Katzman et al. | Aug 2022 | B2 |
| 11423697 | Wucher et al. | Aug 2022 | B1 |
| 11517272 | Unklesbay et al. | Dec 2022 | B2 |
| 11534272 | Li et al. | Dec 2022 | B2 |
| 11534275 | Wey | Dec 2022 | B2 |
| 11568656 | Mohan et al. | Jan 2023 | B2 |
| 11589955 | Medvinskaya et al. | Feb 2023 | B2 |
| 11589957 | Carrier, Jr. et al. | Feb 2023 | B2 |
| 11622843 | Nikolskiy et al. | Apr 2023 | B2 |
| 11672629 | Chekh et al. | Jun 2023 | B2 |
| 11676701 | Carter et al. | Jun 2023 | B2 |
| 11694418 | Long et al. | Jul 2023 | B2 |
| 11735306 | Wirjadi et al. | Aug 2023 | B2 |
| 11887209 | Traff et al. | Jan 2024 | B2 |
| 11894114 | Paik et al. | Feb 2024 | B2 |
| 11900538 | Katzman et al. | Feb 2024 | B2 |
| 11918437 | Salah et al. | Mar 2024 | B2 |
| 20020015934 | Rubbert et al. | Feb 2002 | A1 |
| 20020028418 | Farag et al. | Mar 2002 | A1 |
| 20040122702 | Sabol et al. | Jun 2004 | A1 |
| 20040122703 | Walker et al. | Jun 2004 | A1 |
| 20040122719 | Sabol et al. | Jun 2004 | A1 |
| 20050192835 | Kuo et al. | Sep 2005 | A1 |
| 20070047794 | Lang et al. | Mar 2007 | A1 |
| 20070183633 | Hoffmann | Aug 2007 | A1 |
| 20090019552 | McLaughlin et al. | Jan 2009 | A1 |
| 20090024416 | McLaughlin et al. | Jan 2009 | A1 |
| 20100309301 | Thiel et al. | Dec 2010 | A1 |
| 20130022252 | Chen et al. | Jan 2013 | A1 |
| 20130022255 | Chen et al. | Jan 2013 | A1 |
| 20130244208 | Rubbert | Sep 2013 | A1 |
| 20140003695 | Dean et al. | Jan 2014 | A1 |
| 20150056576 | Nikolskiy et al. | Feb 2015 | A1 |
| 20160199215 | Kopelman | Jul 2016 | A1 |
| 20170008168 | Weng et al. | Jan 2017 | A1 |
| 20170045950 | El Dokor et al. | Feb 2017 | A1 |
| 20170100213 | Kuo | Apr 2017 | A1 |
| 20170281110 | Mandelkern et al. | Oct 2017 | A1 |
| 20170340418 | Raanan | Nov 2017 | A1 |
| 20180008213 | Rubbert | Jan 2018 | A1 |
| 20180028294 | Azernikov et al. | Feb 2018 | A1 |
| 20180121601 | Hahm et al. | May 2018 | A1 |
| 20180132982 | Nikolskiy et al. | May 2018 | A1 |
| 20180144466 | Hsieh et al. | May 2018 | A1 |
| 20180303581 | Martz et al. | Oct 2018 | A1 |
| 20180360567 | Xue et al. | Dec 2018 | A1 |
| 20180365089 | Okanohara et al. | Dec 2018 | A1 |
| 20190083209 | Lior et al. | Mar 2019 | A1 |
| 20190150724 | Elazar et al. | May 2019 | A1 |
| 20190197358 | Madani et al. | Jun 2019 | A1 |
| 20190282344 | Azernikov et al. | Sep 2019 | A1 |
| 20190318479 | Ajri et al. | Oct 2019 | A1 |
| 20200000562 | Wey | Jan 2020 | A1 |
| 20200013145 | Ratner et al. | Jan 2020 | A1 |
| 20200022783 | Cramer et al. | Jan 2020 | A1 |
| 20200022790 | Fisker | Jan 2020 | A1 |
| 20200027252 | Ye et al. | Jan 2020 | A1 |
| 20200066391 | Sachdeva et al. | Feb 2020 | A1 |
| 20200125069 | Sirovskiy et al. | Apr 2020 | A1 |
| 20200175681 | Ezhov et al. | Jun 2020 | A1 |
| 20200268495 | Ryakhovsky et al. | Aug 2020 | A1 |
| 20200273248 | Jorgensen et al. | Aug 2020 | A1 |
| 20200293878 | Bhaskar et al. | Sep 2020 | A1 |
| 20200297458 | Roschin et al. | Sep 2020 | A1 |
| 20200311934 | Cherkas et al. | Oct 2020 | A1 |
| 20200383752 | Willers et al. | Dec 2020 | A1 |
| 20200402272 | Xu et al. | Dec 2020 | A1 |
| 20200405456 | Nikolskiy et al. | Dec 2020 | A1 |
| 20200405464 | Nikolskiy et al. | Dec 2020 | A1 |
| 20200411201 | Kearney et al. | Dec 2020 | A1 |
| 20210005017 | Hollenbeck | Jan 2021 | A1 |
| 20210012884 | Kim et al. | Jan 2021 | A1 |
| 20210034813 | Wu et al. | Feb 2021 | A1 |
| 20210059796 | Weiss et al. | Mar 2021 | A1 |
| 20210073998 | Brown et al. | Mar 2021 | A1 |
| 20210082184 | Claessen | Mar 2021 | A1 |
| 20210085238 | Schnabel et al. | Mar 2021 | A1 |
| 20210090272 | Meyer et al. | Mar 2021 | A1 |
| 20210093421 | Michaeli et al. | Apr 2021 | A1 |
| 20210100642 | Weiss et al. | Apr 2021 | A1 |
| 20210100643 | Weiss et al. | Apr 2021 | A1 |
| 20210106409 | Ojelund et al. | Apr 2021 | A1 |
| 20210110605 | Haslam et al. | Apr 2021 | A1 |
| 20210153986 | Wirjadi et al. | May 2021 | A1 |
| 20210158607 | Katzman et al. | May 2021 | A1 |
| 20210174604 | Long et al. | Jun 2021 | A1 |
| 20210196434 | Cramer et al. | Jul 2021 | A1 |
| 20210216822 | Paik et al. | Jul 2021 | A1 |
| 20210236249 | Nikolskiy et al. | Aug 2021 | A1 |
| 20210251729 | Schneider et al. | Aug 2021 | A1 |
| 20210255600 | Faust | Aug 2021 | A1 |
| 20210264611 | Xue et al. | Aug 2021 | A1 |
| 20210315667 | Raslambekov | Oct 2021 | A1 |
| 20210321872 | Saphier et al. | Oct 2021 | A1 |
| 20210338400 | Murata et al. | Nov 2021 | A1 |
| 20210353154 | Saphier et al. | Nov 2021 | A1 |
| 20210393375 | Chekh et al. | Dec 2021 | A1 |
| 20220000583 | Parpara et al. | Jan 2022 | A1 |
| 20220008175 | Ojelund et al. | Jan 2022 | A1 |
| 20220028162 | Katzman et al. | Jan 2022 | A1 |
| 20220130045 | Kopelman et al. | Apr 2022 | A1 |
| 20220137592 | Cramer et al. | May 2022 | A1 |
| 20220148173 | Minchenkov et al. | May 2022 | A1 |
| 20220165388 | Chernov et al. | May 2022 | A1 |
| 20220189130 | Long et al. | Jun 2022 | A1 |
| 20220215531 | Azernikov et al. | Jul 2022 | A1 |
| 20220215928 | Tabak et al. | Jul 2022 | A1 |
| 20220257342 | Kuo | Aug 2022 | A1 |
| 20220262007 | Cramer et al. | Aug 2022 | A1 |
| 20220296344 | Lee et al. | Sep 2022 | A1 |
| 20220304782 | Derzapf et al. | Sep 2022 | A1 |
| 20220327774 | Katzman et al. | Oct 2022 | A1 |
| 20220350936 | Azernikov et al. | Nov 2022 | A1 |
| 20220369916 | Carrier, Jr. et al. | Nov 2022 | A1 |
| 20220378548 | Salah et al. | Dec 2022 | A1 |
| 20230048895 | Wucher et al. | Feb 2023 | A1 |
| 20230057664 | Steger | Feb 2023 | A1 |
| 20230066220 | Coulombe et al. | Mar 2023 | A1 |
| 20230075514 | Wiedemann et al. | Mar 2023 | A1 |
| 20230153476 | Fabbri et al. | May 2023 | A1 |
| 20230410495 | Xue et al. | Dec 2023 | A1 |
| 20240028782 | Selberis et al. | Jan 2024 | A1 |
| 20240029380 | Azernikov et al. | Jan 2024 | A1 |
| Number | Date | Country |
|---|---|---|
| 3146240 | Jan 2021 | CA |
| 102017203475 | Sep 2018 | DE |
| 1368986 | Dec 2011 | EP |
| 3671531 | Dec 2018 | EP |
| 3767521 | Jul 2019 | EP |
| 3797732 | Sep 2019 | EP |
| 101718868 | Mar 2017 | KR |
| 102334519 | Dec 2021 | KR |
| 2012061652 | May 2012 | WO |
| 2016044465 | Mar 2016 | WO |
| 2018069736 | Apr 2018 | WO |
| 2018118200 | Jun 2018 | WO |
| 2018158411 | Sep 2018 | WO |
| 2018195554 | Oct 2018 | WO |
| 2019217764 | Apr 2019 | WO |
| 2019204520 | Oct 2019 | WO |
| 2019209820 | Oct 2019 | WO |
| 2020048960 | Mar 2020 | WO |
| 2020173912 | Mar 2020 | WO |
| 2020089406 | May 2020 | WO |
| 2020161245 | Aug 2020 | WO |
| 2020182920 | Sep 2020 | WO |
| 2021046147 | Mar 2021 | WO |
| 2021046434 | Mar 2021 | WO |
| 2021058643 | Apr 2021 | WO |
| 2021110938 | Jun 2021 | WO |
| 2021113501 | Jun 2021 | WO |
| 2021155230 | Aug 2021 | WO |
| 2021210723 | Oct 2021 | WO |
| 2021210966 | Oct 2021 | WO |
| 2021214212 | Oct 2021 | WO |
| 2022131418 | Jun 2022 | WO |
| 2022131419 | Jun 2022 | WO |
| Entry |
|---|
| US 11,182,981 B2, 11/2021, Long et al. (withdrawn) |
| Xu et al. “3D Tooth Segmentation and Labeling Using Deep Convolutional Neural Networks.” IEEE Transactions on Visualization and Computer Graphics, vol. 25, No. 7, Jul. 2019, pp. 2336-2348 (Year: 2019). |
| Mairaj et al. “Teeth Segmentation and Feature Extraction for Odontological Biometrics.” Sixth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, Oct. 15, 2010, pp. 323-328 (Year: 2010). |
| Liu, Rong. “Spectral Mesh Segmentation”, Thesis, Simon Frasier University, 2009. 149 pages (Year: 2009). |
| International Application No. PCT/US22/22587, International Search Report and Written Opinion, mailed on Jun. 22, 2022, in 14 pages. |
| Raydent Cad, AI Based Dental CAD Solution brochure, in 7 pages. (date unavailable). |
| PCT/US2022/022587, Transmittal of Copy of International Preliminary Report, mailed on Oct. 12, 2023 7 pages. |
| Mark Pauly, Mesh Smoothing, Ecole Polytechnique Federale de Lausanne, in 43 pages.(date unavailable). |
| Hu et al., Enhancing Dental CBCT Image by Semi-coupled Generative Adversarial Networks, Journal of Dental Research, DOI:10.1177/ToBeAssigned, in 8 pages, (2016). |
| Groueix, et al. A Papier-Mache Approach to Learning 3D Surface Generation, Computer Vision Foundation, pp. 216-224, (2018). |
| Qi et al, PointNet: Deep Learning on Point Sets for 3D Classifications and Segmentation, Computer Vision Foundation, pp. 652-660, (2017). |
| Qi et al, PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, Computer Vision Foundation, in 14 pages, (2017). |
| H. Wang et al., Multiclass CBCT Image Segmentation for Orthodontics with Deep Learning, Journal of Dental Research, 2021, vol. 100(9), 943-949, International & American Associates for dental Research 2021. |
| Yin et al., Algorithm of drawing isoline based on improved Delaunay triangle net, Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and technology, 978-1-5090-6161-7/17/#31.00, 2017, IEEE, pp. 1022-1026. |
| Cowley et al., The Effect of Gingival-Margin Design on the Retention of Thermoformed Aligners, Nov. 2012 JCO, Inc. vol. XLVI, No. 11, pp. 697-702. |
| Hanchao Li et al., Pyramid Attention Network for Semantic Segmentation, Nov. 25, 2018, in 13 pages. |
| Jeremy Zhang, UNet—Line by Line Explanation, Published in Towards Data Science Oct. 17, 2019, in 20 pages. |
| Joseph Redmon et al., You Only Look Once: Unified, Real-Time Object Detection, University of Washington, Allen Institute for AI, May 9, 2016, in 10 pages, http://pjreddie.com/yolo/. |
| Joseph Redmon et al., YOLO9000: Better, Faster, Stronger, University of Washington, Dec. 25, 2016, in 9 pages, http://pjreddie.com/yolo9000/. |
| Joseph Redmon et al., YOLOv3: An Incremental Improvement, University of Washington, Apr. 8, 2018, in 6 pages. |
| Olaf Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, Computer Science Department and BIOSS Centre for Biological Signalling Studies, University of Freiburg, Germany. May 18, 2015 8 pages. |
| Pablo Ruiz, ResNets, Harvard University, Aug. 2018, in 8 pages. |
| Yu-Bing Chang et al., An Automatic and Robust Algorithm of Reestablishment of Digital dental Occlusion, HHS Public Access, IEEE Trans Med Imaging. Sep. 2010; 29(9) 1652-1663,doi: 10.1109/TMI.2010.2049526, in 36 pages. |
| YOLO Real Time Objection Detection, downloaded from https://pjreddie.com/darknet/yolo/ on Jun. 10, 2020, in 10 pages. |
| Shunping Ji, 3D Convolutional Neural Networks for Crop Classification with Multi-Temporal Remote Sensing Images, Remote Sensing, 2018, 10, 75; doi: 10.3390/rs10010075, www.MDPI.com/journal/remotesensing, pp. 1-17. |
| Imagoworks web page, https://imagoworks.ai/#technology, copyright 2021 downloaded on Feb. 18, 2022, pp. 1-8. |
| International application No. PCT/US21/15820, Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority or the Declaration, date of mailing Jun. 9, 2021, in 17 pages. |
| Youngjun Kevin Kim, Imagoworks B2B Collaboration Proposal, Sep. 2021, in 32 pages. |
| Chen et al., A Convolutional Neural Networks for Teeth Margin Detection on 3-Dimensional Dental Meshes, Mesh CNN for Teeth Margin Detection, in 11 pages, Jul. 7, 2021. |
| Detect Edit Margins—Exocad Wiki, Release from 2022, downloaded Sep. 1, 2023, 5 pages. |
| International Application No. PCT/US2017/043943, International Search Report and Written Opinion, mailed Oct. 12, 2017, in 11 pages. |
| International Application No. PCT/US2023/070692, International Search Report and Written Opinion, mailed Dec. 13, 2023, in 16 pages. |
| Hang-Nga Mai et al., Reliability of automatic finish line detection for tooth preparation in dental computer-aided software, J.Prosthodont Res. 2023:67(1): 138-143. |
| Raydent CAD, AI Based Dental CAD Solution brochure, in 7 pages. |
| Stefan Raith et al., Artificial Neural Networks as a Powerful Numerical Tool to Classify Specific Features of a Tooth Based on 3D Scan Data, Elsevier, Computers in Biology and Medicine 80 (2017) 65-76, in 12 pages. |
| International Application No. PCT/US2022/022587, International Search Report and Written Opinion, mailed Jun. 22, 2022, in 14 pages. |
| PCT/US2022/022587, Transmittal of International Preliminary Report, mailed on Oct. 12, 2023, 7 pages. |
| Jifeng Dai et al., R-FCN: Object Detection via Region-based Fully Convolutional Networks, 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona Spain, in 9 pages. |
| Toshiaki Kondo et al., Tooth Segmentation of Dental Study Models Using Range Images, Research Gate, IEEE Transactions on Medical Imaging, Apr. 2004, vol. 23, No. 3, Mar. 2004, in 14 pages. |
| Ross Girshick, Fast R-CNN, in 9 pages, (Dec. 2015). |
| Shuai Yang et al., 3D Intelligent Scissors for Dental Mesh Segmentation, Hindawi, Computational and Mathematical Methods in Medicine, vol. 2020, Article ID 1394231, 12 pages. |
| Shuai Yang et al., Interactive Tooth Segmentation Method of Dental Model based on Geodesic, Research Gate, Conference paper Jan. 2017, Advances in Engineering Research, vol. 138, in 6 pages. |
| Tsung-Yi Lin et al., Feature Pyramid Networks for Object Detection, arXiv: 1612.03144v2 [cs.CV] Apr. 19, 2017, in 10 pages. |
| Tsung-Yi Lin et al., Focal Loss for Dense Object Detection, arXiv: 1708.02002v2 [cs.CV] Feb. 7, 2018, in 10 pages. |
| Andrew Nealen et al., Laplacian Mesh Optimization, in 9 pages,(Nov. 2006). |
| Mark Pauly, Mesh Smoothing, Ecole Polytechnique Federale de Lausanne, in 43 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20220318989 A1 | Oct 2022 | US |
