The invention relates generally to the field of surface shape imaging and more particularly relates to apparatus and methods for display of 3D surface features indexed to a 2D panorama image.
Structured light imaging is one familiar technique that has been successfully applied for surface characterization. In structured light imaging, a pattern of illumination is projected toward the surface of an object from a given angle. The pattern can use parallel lines of light or more complex periodic features, such as sinusoidal lines, dots, or repeated symbols, and the like. The light pattern can be generated in a number of ways, such as using a mask, an arrangement of slits, interferometric methods, or a spatial light modulator, such as a Digital Light Processor from Texas Instruments Inc., Dallas, Tex. or similar digital micromirror device. Multiple patterns of light may be used to provide a type of encoding that helps to increase robustness of pattern detection, particularly in the presence of noise. Light reflected or scattered from the surface is then viewed from another angle as a contour image, taking advantage of triangulation in order to analyze surface information based on the appearance of contour lines or other patterned illumination.
Structured light imaging has been used effectively for surface contour imaging of solid, highly opaque objects and has been used for imaging the surface contours for some portions of the human body and for obtaining detailed data about skin structure. Structured light imaging methods have also been applied to the problem of dental imaging, helping to provide detailed surface information about teeth and other intraoral features. Intraoral structured light imaging is now becoming a valuable tool for the dental practitioner, who can obtain this information by scanning the patient's teeth using an inexpensive, compact intraoral scanner, such as the Model CS3500 Intraoral Scanner from Carestream Dental, Atlanta, Ga.
Contour imaging uses patterned or structured light to obtain surface contour information for structures of various types. In structured light projection imaging, a pattern of lines or other shapes is projected toward the surface of an object from a given direction. The projected pattern from the surface is then viewed from another direction as a contour image, taking advantage of triangulation in order to analyze surface information based on the appearance of contour lines. Phase shifting, in which the projected pattern is incrementally spatially shifted for obtaining images that provide additional measurements at the new locations, is typically applied as part of structured light projection imaging, used in order to complete the contour mapping of the surface and to increase overall resolution in the contour image.
The advent of less expensive video imaging devices and advancement of more efficient contour image processing algorithms now make it possible to acquire structured light images without the need to fix the scanner in position for individually imaging each tooth. With upcoming intraoral imaging systems, it can be possible to acquire contour image data by moving the scanner/camera head over the teeth, allowing the moving camera to acquire a large number of image views that can be algorithmically fitted together and used to for forming the contour image.
Although textured 3D contour imaging of the teeth can provide a significant amount of useful information on tooth condition as well as overall structure and appearance, there can be practical problems that hamper the effective use of 3D display capabilities in some cases. In order to observe details from the 3D tooth model that is constructed, the practitioner may need to alter the 3D view position as well as making pan, zoom, and scale adjustments. Providing a suitable close-up view of the portion of the dental arch that is of interest can be particularly difficult during a procedure, since this typically involves some fairly complex interaction with the 3D mesh that has been generated and displayed on the system monitor. Even with interface tools such as a positioning glove or wand, direct manipulation of the 3D image model can be challenging, especially when the practitioner is actively working on a particular tooth, for example.
Other difficulties relate to the need to edit the 3D model. The capability to edit the 3D surface is a useful feature in a number of applications, including dental restoration and prosthesis implant planning. Among editing functions executed by the practitioner are cutting, trimming, marking, and tagging, for example. It can be cumbersome to manipulate the full 3D mesh for this purpose, such as by rotating, surface displacement, and other operations.
Thus, it can be seen that there would be significant value in a method for imaging system interaction that allows the practitioner to specify 3D content for display and editing in an intuitive manner, without requiring manipulation of the full 3D model on a display screen.
An object of the present invention is to advance the art of image content manipulation for 3-D surface image presentation. Embodiments of the present disclosure provide operator interface utilities and methods that can help to streamline and simplify the procedure for identifying, editing, and presenting a portion of a full 3D surface image model to the viewer.
These aspects are given only by way of illustrative example, and such objects may be exemplary of one or more embodiments of the invention. Other desirable objectives and advantages inherently achieved by the disclosed invention may occur or become apparent to those skilled in the art. The invention is defined by the appended claims.
According to one aspect of the invention, there is provided a method for display of a region of interest of a 3D tooth surface model, the method comprising:
The foregoing and other objects, features, and advantages of the invention will be apparent from the following more particular description of the embodiments of the invention, as illustrated in the accompanying drawings.
The elements of the drawings are not necessarily to scale relative to each other. Some exaggeration may be necessary in order to emphasize basic structural relationships or principles of operation. Some conventional components that would be needed for implementation of the described embodiments, such as support components used for providing power, for packaging, for interconnection to transmit signal content between components, and for mounting and protecting system optics, for example, are not shown in the drawings in order to simplify description.
The following is a detailed description of the preferred embodiments, reference being made to the drawings in which the same reference numerals identify the same elements of structure in each of the several figures.
Where they are used in the context of the present disclosure, the terms “first”, “second”, and so on, do not necessarily denote any ordinal, sequential, or priority relation, but are simply used as labels to more clearly distinguish one step, element, or set of elements from another and are not intended to impose numerical requirements on their objects, unless specified otherwise.
As used herein, the term “energizable” relates to a device or set of components that perform an indicated function upon receiving power and, optionally, upon receiving an enabling signal.
In the context of the present disclosure, the terms “structured light illumination”, “fringe pattern”, or “patterned illumination” are used to describe the type of illumination that is used for structured light projection imaging or “contour” imaging that characterizes tooth shape. The structured light pattern itself can include, as patterned light features, one or more lines, circles, curves, or other geometric shapes that are distributed over the area that is illuminated and that have a predetermined spatial and temporal frequency. One exemplary type of structured light pattern that is widely used for contour imaging is a pattern of evenly spaced lines of light projected onto the surface of interest.
In the context of the present disclosure, the term “structured light image” refers to the image that is captured during projection of the light pattern or “fringe pattern” that is used for characterizing the tooth contour. “Contour image” and “contour image data” refer to the processed image data that are generated and updated from structured light images.
In the context of the present disclosure, the term “optics” is used generally to refer to lenses and other refractive, diffractive, and reflective components used for shaping and orienting a light beam.
In the context of the present disclosure, the terms “viewer”, “operator”, “editor”, and “user” are considered to be equivalent and refer to the viewing practitioner, technician, or other person who may operate a camera or scanner and may also view and manipulate an image, such as a dental image, on a display monitor. An “operator instruction” or “viewer instruction” is obtained from explicit commands entered by the viewer, such as by clicking a button on the camera or by using a computer mouse or by touch screen or keyboard entry.
The term “set”, as used herein, refers to a non-empty set, as the concept of a collection of one or more elements or members of a set is widely understood in elementary mathematics. The term “subset”, unless otherwise explicitly stated, is used herein to refer to a non-empty proper subset, that is, to a subset of the larger set, having one or more members. For a set S, a subset may comprise the complete set S. A “proper subset” of set S, however, is strictly contained in set S and excludes at least one member of set S.
In the context of the present disclosure, the phrase “in signal communication” indicates that two or more devices and/or components are capable of communicating with each other via signals that travel over some type of signal path. Signal communication may be wired or wireless. The signals may be communication, power, data, or energy signals. The signal paths may include physical, electrical, magnetic, electromagnetic, optical, wired, and/or wireless connections between the first device and/or component and second device and/or component. The signal paths may also include additional devices and/or components between the first device and/or component and second device and/or component.
In the context of the present disclosure, a reflectance image is a conventional 2D image of a subject obtained with the subject illuminated by a field of light. A reflectance image can be monochrome or polychromatic. A polychromatic reflectance image can be obtained using a monochrome sensor with illumination fields of different colors, that is, of different wavelength bands, provided in rapid sequence.
In the context of the present disclosure, the terms “camera” and “scanner” are used interchangeably, as the description relates to structured light images successively projected and captured by a camera device operating in a single-image mode or in a continuous acquisition or video mode.
In structured light imaging, a pattern of lines or other shapes is projected from illumination array 10 toward the surface of an object from a given angle. The projected pattern from the illuminated surface position is then viewed from another angle as a contour image, taking advantage of triangulation in order to analyze surface information based on the appearance of contour lines. Phase shifting, in which the projected pattern is incrementally shifted spatially for obtaining additional measurements at the new locations, is typically applied as part of structured light imaging, used in order to complete the contour mapping of the surface and to increase overall resolution in the contour image.
The schematic diagram of
By projecting and capturing images that show structured light patterns that duplicate the arrangement shown in
A synchronous succession of multiple structured light patterns can be projected and analyzed together for a number of reasons, including to increase the density of lines for additional reconstructed points and to detect and/or correct incompatible line sequences. Use of multiple structured light patterns is described in commonly assigned U.S. Patent Application Publications No. US2013/0120532 and No. US2013/0120533, both entitled “3D INTRAORAL MEASUREMENTS USING OPTICAL MULTILINE METHOD” and incorporated herein in their entirety.
By knowing the instantaneous position of the camera and the instantaneous position of the line of light within a object-relative coordinate system when the image was acquired, a computer and software can use triangulation methods to compute the coordinates of numerous illuminated surface points relative to a plane. As the plane is moved to intersect eventually with some or all of the surface of the object, the coordinates of an increasing number of points are accumulated. As a result of this image acquisition, a point cloud of vertex points or vertices can be identified and used to represent the extent of a surface within a volume. The points in the point cloud then represent actual, measured points on the three dimensional surface of an object. A mesh can then be constructed, connecting points on the point cloud as vertices that define individual congruent polygonal faces (typically triangular faces) that characterize the surface shape. The full 3D image model can then be formed by combining the surface contour information provided by the mesh with polychromatic image content obtained from a camera, such as camera 24 that is housed with camera 24 in the embodiment described with reference to
As noted previously in the background section, the practitioner can find it difficult to manipulate, view, and edit the full 3D model that is displayed, particularly when carrying out a procedure on a tooth or supporting structure. An embodiment of the present disclosure addresses the need for operator interface tools that simplify viewer procedure for manipulating the display and editing the model, and help to identify and positionally orient a relevant portion of the 3D view needed by the practitioner.
The logic flow diagram of
The final surface of the tooth model is denoted Mo. Tooth model Mo is a mesh structure that has J facets F={F1, F2, . . . FJ} and I vertices V={V1, V2, . . . VI}. Each facet is defined by N vertices, Fk={Vk1, Vk2, . . . , Vkn}. Any vertex, Vs, contains 3D geometric coordinates, {Xs, Ys, Zs} and 2d texture coordinates {Vx_s, Vy_s} in a global texture map, Mt.
Mesh-parameterization sets up a parameter representation for Mo, in a 2D parameter-space, Po, if Mo is always locally homeomorphic to a disc. For each facet Fi in F of Mo, there is a corresponding 2d facet Tk corresponding to Fi. The full parameter space, Po, would include the facets set T={T1, T2, . . . , TL} and the vertices set U={U1, U2, . . . , Uo}. Any vertex Up in set U has 2D coordinate {Qx_p, Qy_p}. A result of mesh parameterization of a 3D surface is shown in
Continuing with the
After mesh-parameterization, an index image as a panorama of a textured surface of teeth is generated by the following sequence:
Still following
The exemplary display screen shown in
The 3D surface image view is initially rendered at a view angle that corresponds to the average view angle of the selected area of the ROI. Thus, for example, the average normal vector direction for each facet in the selected ROI is determined and used as the initial view angle for 3D rendering. Once the 3D rendered version of tooth surface 156 is displayed, a manipulable control, on-screen guide 170, displays, indicating the averaged normal vector and view angle with two crossed curves that can be rotated to left and right and up-down for changing the view angle.
The logic flow diagram of
Using an operator interface such as that shown with reference to
It should be noted that the 3D volume image content can be from sources other than structured light images, including reflectance images of other types that can be used to generate a point cloud that is used for 3D mesh construction.
Consistent with an embodiment of the present invention, a computer executes a program with stored instructions that perform on image data accessed from an electronic memory. As can be appreciated by those skilled in the image processing arts, a computer program of an embodiment of the present invention can be utilized by a suitable, general-purpose computer system, such as a personal computer or workstation, as well as by a microprocessor or other dedicated processor or programmable logic device. However, many other types of computer systems can be used to execute the computer program of the present invention, including networked processors. The computer program for performing the method of the present invention may be stored in a computer readable storage medium. This medium may comprise, for example; magnetic storage media such as a magnetic disk (such as a hard drive) or magnetic tape or other portable type of magnetic disk; optical storage media such as an optical disc, optical tape, or machine readable bar code; solid state electronic storage devices such as random access memory (RAM), or read only memory (ROM); or any other physical device or medium employed to store a computer program. The computer program for performing the method of the present invention may also be stored on computer readable storage medium that is connected to the image processor by way of the internet or other communication medium. Those skilled in the art will readily recognize that the equivalent of such a computer program product may also be constructed in hardware.
It will be understood that the computer program product of the present invention may make use of various image manipulation algorithms and processes that are well known. It will be further understood that the computer program product embodiment of the present invention may embody algorithms and processes not specifically shown or described herein that are useful for implementation. Such algorithms and processes may include conventional utilities that are within the ordinary skill of the image processing arts. Additional aspects of such algorithms and systems, and hardware and/or software for producing and otherwise processing the images or co-operating with the computer program product of the present invention, are not specifically shown or described herein and may be selected from such algorithms, systems, hardware, components and elements known in the art.
In the context of the present disclosure, the act of “recording” images means storing image data in some type of memory circuit in order to use this image data for subsequent processing. The recorded image data itself may be stored more permanently or discarded once it is no longer needed for further processing.
It should be noted that the term “memory”, equivalent to “computer-accessible memory” in the context of the present disclosure, can refer to any type of temporary or more enduring data storage workspace used for storing and operating upon image data and accessible to a computer system. The memory could be non-volatile, using, for example, a long-term storage medium such as magnetic or optical storage. Alternately, the memory could be of a more volatile nature, using an electronic circuit, such as random-access memory (RAM) that is used as a temporary buffer or workspace by a microprocessor or other control logic processor device. Display data, for example, is typically stored in a temporary storage buffer that is directly associated with a display device and is periodically refreshed as needed in order to provide displayed data. This temporary storage buffer can also be considered to be a memory, as the term is used in the present disclosure. Memory is also used as the data workspace for executing and storing intermediate and final results of calculations and other processing. Computer-accessible memory can be volatile, non-volatile, or a hybrid combination of volatile and non-volatile types. Computer-accessible memory of various types is provided on different components throughout the system for storing, processing, transferring, and displaying data, and for other functions.
In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim.
While the invention has been illustrated with respect to one or more implementations, alterations and/or modifications can be made to the illustrated examples without departing from the spirit and scope of the appended claims. In addition, while a particular feature of the invention can have been disclosed with respect to one of several implementations, such feature can be combined with one or more other features of the other implementations as can be desired and advantageous for any given or particular function. The term “at least one of” is used to mean one or more of the listed items can be selected. The term “about” indicates that the value listed can be somewhat altered, as long as the alteration does not result in nonconformance of the process or structure to the illustrated embodiment. Finally, “exemplary” indicates the description is used as an example, rather than implying that it is an ideal.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims, and all changes that come within the meaning and range of equivalents thereof are intended to be embraced therein.
This application claims the benefit of U.S. Provisional application U.S. Ser. No. 62/238,760, provisionally filed on Oct. 8, 2015, entitled “OPERATOR INTERFACE FOR 3D SURFACE DISPLAY USING 2D INDEX IMAGE”, in the names of Yingqian Wu, Yu Zhou, Menghui Guan and Qinran Chen, which is incorporated herein by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
62238760 | Oct 2015 | US |