Patent Application
20230355360
The present disclosure relates to a system and computer-implemented method for providing feedback during scanning of a dental object. The disclosure further relates to a 3D scanner system for providing said feedback and an intraoral scanning device.
Three-dimensional (3D) representation is relevant in many fields such as medicine or civil engineering. In dentistry, a 3D digital representation of a dental object represents a clear advantage to help dentists give the best treatment to a patient. Indeed, digital dentistry gives the opportunity of expanding dentist expertise with new treatment options and provides possibilities for more cost-effective ways of taking care of their patients.
Intraoral scanners are devices for capturing a digital impression of a patient's dentition within dentistry. In a typical scanning scenario using a handheld intraoral scanning device, a digital 3D representation is reconstructed in real-time and simultaneously rendered to a display to provide visual guidance and user feedback to the user via the display. This implies that the user needs to focus on a display to see whether he/she is progressing in their scanning as the 3D representation is constructed in real-time on the display providing visual feedback on the progress and data coverages.
This method is not ideal as it shifts the focus of the user away from the patient by drawing attention to the display. State of the art intraoral scanning devices are additionally becoming more than a digital impression tool as diagnostic capabilities are added to the scanners to capture more information than surface information. Some of this additional information, like infrared images or fluorescence images is typically captured in real-time, but not displayed in real-time. Hence, the dentist does not have any feedback on data quality or data coverage of the diagnostic data. Hence, there is a desire to provide another type of feedback which provides the user with information on data coverage or data quality. Furthermore, it is desired to develop feedback outside of the graphical user interface such that the user can focus more on the patient.
As disclosed herein, the above-mentioned challenges are addressed by providing a computer-implemented method for providing feedback during scanning of a dental object, the method comprising the steps of:
The feedback signal may correlate to the added information, which can be determined in a step of the computer-implemented method, as previously described.
As described previously, users may in many cases have to focus on the digital 3D representation displayed through a GUI while scanning, instead of focusing their time on the patient and the scanner. This leads to a situation where the dentist is maneuvering the scanner around in the oral cavity of the patient while looking at a screen. This can generate longer time to scan dental objects as the maneuvering of the scanner may lead over areas which have already been scanned. Additionally, the dentist tends to move the scanner much slower than needed, which both increases scan time and at the same time, increases the amount of scan data acquired during scanning, which may result in a prolonged processing time and may even slow down the performance of the processor responsible for constructing the 3D digital representation from the scan data. By introducing this dynamic feedback signal provided by the computer-implement method, it is possible to assist the user to avoid wasting time by scanning the same area for too long, avoiding overloading the processors with scan data and simultaneously enables the user to focus more on the dental object to be scanned and more generally on the patient.
The computer-implemented method provides a feedback signal which is configured to provide guidance to the user about the information received while scanning a dental object. Accordingly, this could for example be information that a new area is being scanned. However, on the contrary it could also be information that nothing new is being scanned so that the user can move the scanner to scan another area. Because the feedback signal is correlated to the added information to the digital 3D representation, the user can get an indication on how much information is added to the digital 3D representation while scanning a dental object.
In one aspect, a 3D scanner system for scanning a dental object comprises an intraoral scanning device, comprising at least one optical sensor for obtaining scan data of the dental object and one or more processing unit(s) configured to continuously performing the steps of:
In some embodiments, the 3D scanner system comprises:
In some embodiments, the 3D scanner system comprises:
In some embodiments, the 3D scanner system comprises:
The aforementioned embodiments may be combined such that both an auditory feedback signal and a visual feedback signal is provided to the user during a scanning session.
Preferably, the steps of the computer-implemented method as disclosed herein are performed continuously and/or in real-time. The 3D scanner system for scanning a dental object may further comprise at least one output device for providing an output to a user during scanning, where the output is based on the feedback signal provided by the computer-implemented method.
In yet another aspect, a handheld intraoral scanning device, comprising a scanner housing configured to be handled with one hand, wherein the scanner housing comprises:
The handheld intraoral scanning device is configured to receive at least one feedback signal provided by the computer-implemented method and to provide the output to the user from the one or more output device(s) based on the at least one feedback signal.
The feedback signal as discussed in the aspects above, and as generally discussed in the disclosure, is typically output by at least one output device, wherein the at least one output device may be an audio device and/or a haptic device and/or a light-emitting device. In some embodiments, the 3D scanner system comprises a haptic device configured for providing a haptic feedback signal. Preferably, the haptic device is located in the intraoral scanning device. In some embodiments, at least a property of the haptic feedback signal is altered based on the added information and/or the determined overlap. As an example, a vibration level of the haptic device may be increased when the determined overlap is below a predefined lower threshold, and the vibration level may be decreased when the determined overlap is above a predefined upper threshold. An advantage of an intraoral scanning device comprising a haptic device is that the user can receive a non-visible feedback, e.g. by feeling a vibration in the scanning device, if the scanning can be improved. Thus, the user can correct the way he uses the scanning device immediately based on the feedback. In some embodiments, several different types of feedback are provided. As an example, the scanning device may comprise a light-emitting device for providing a visual feedback correlated to the added data and/or the determined overlap, and the scanning device may further comprise a haptic device as previously mentioned. Additionally, in some embodiments the scanning device comprises a speaker for outputting an audible feedback signal as described further herein. In other embodiments, the 3D scanner system comprises an external speaker, i.e. a speaker which is not located on the intraoral scanning device.
In a preferred embodiment, the computer-implemented method for providing feedback during scanning of a dental object, comprises the steps of:
Preferably, the steps of the computer-implemented method as disclosed herein are performed continuously and/or in real-time. An advantage of updating the digital 3D representation and providing a feedback signal in real-time is that the user, e.g. the dentist, can immediately react according to the feedback signal. In other words, if the user for example scans too slowly, the user is preferably notified by the feedback signal in real-time, such that the user can correct his action(s) and e.g. scan faster. If the user holds the scanning device extensively over the same area of the dental object, i.e. such that an excessive overlap between the 3D representation and new scan patches occurs, the user is advantageously notified by the feedback signal as disclosed herein. Preferably, the feedback signal correlates with the amount of overlap between the 3D representation and new scan data/new scan patches, such that the user is encouraged to move the scanning device and scan new areas of the dental object. Another example is if the registration of new scan patches is lost, i.e. if the new scan data/scan patch cannot be registered/added to the digital 3D representation, then the user should preferably be notified in real-time by the feedback signal, e.g. either by a visual feedback, an audible feedback, or a haptic feedback.
The disclosure further relates to an 3D scanner system for scanning a dental object, wherein the scanner system comprises: an intraoral scanning device; and one or more processing unit(s) configured to carry out the computer-implemented method as disclosed herein.
In an embodiment, a handheld 3D scanner is a device which may produce scan data. Scan data may comprise surface information in the form of point clouds, a sequence of images, or one or more scan patches. Each point in the cloud may be a result of a distance measurement in the view direction of the scanner. The coordinates of the point may be relative to a coordinate system fixed to the scanner, which can denote a local frame.
The scanning device may be a focus scanner which uses moving lenses to sweep a narrow focus band through the scanners field of view, which may then be referred to as the scan volume. The scanning device is operated by projecting a pattern and translating a focus plane along an optical axis of the scanning device and capturing a plurality of 2D images at different focus plane positions such that each series of captured 2D images corresponding to each focus plane forms a stack of 2D images. The acquired 2D images are also referred to herein as raw 2D images, wherein raw in this context means that the images have not been subject to image processing. The focus plane position is shifted along the optical axis of the scanning system, such that 2D images captured at a number of focus plane positions along the optical axis form said stack of 2D images for a given view of the object, i.e. for a given arrangement of the scanning system relative to the object. Once a physical object is in focus it's distance to the scanner (in the view direction) can be computed from knowledge of the position of the focus band and the geometry of the scanner.
The intraoral scanning device may be based on other scanning principle such as triangulation-based scanning, confocal scanning, focus scanning, ultrasound scanning, x-ray scanning, stereo vision, structure from motion, optical coherent tomography (OCT), or any other scanning principle
A scanning session may be understood herein as a period of time during which data (such as image data, depth data, 3D data, scan patches, color data, or combinations thereof) of a three-dimensional dental object is acquired using the 3D scanner system. During a scanning session the scanner, and hence the local frames, move relative to a fixed global coordinate frame of the scene which may be a dental object such as the dentition of a patient. The result is a sequence of scan patches recorded at discrete times. The scanner may additionally use external tracking, such as a motion or a position sensor to achieve a priori knowledge of the pose of the scanner as part of the scan data. The frame rate of the intraoral scanning device may be high relative to the movement of physical scanning device such that consecutive scan patches have substantial overlap.
Hence, the amount of overlap typically depends on the frame rate of the scanner and the movement. Typically, a registration algorithm is used to first align and/or register the incoming scan data/scan patches to the current 3D representation, then fuse the new scan patch with the 3D representation and finally render the fused 3D representation on a screen before the next scan patch arrives from the scanner. Accordingly, the step of updating the digital 3D representation preferably comprises registering the new scan data/scan patch to the digital 3D representation. The computer-implemented method may further comprise a step of fusing the new scan data/scan patch to the digital 3D representation. Registration/registering new scan data/scan patch should be understood as determining the location of said scan patch in the (current) digital 3D representation, whereas fusion/fusing the scan data/scan patch should be understood as making said scan data/scan patch a part of the digital 3D representation. For the registration of scan patches a variant of the Iterated Closest Point (ICP) algorithm may be used.
In one embodiment, scan data may be understood as any type of data recorded by the intraoral scanning device. Scan data may typically comprise surface information such as a point cloud or a scan patch as described above providing information about the morphology and structure of the dental object. However, scan data may contain additional data associated with the dental object. Such information may be reflected color texture information, representing the color of the dental object from the direction observed by the scanner. Other information could be a fluorescence texture response recorded from the dental object when exposed to a wavelength capable of exciting a fluorescence response from the dental material. Infrared or near infrared light are examples of wavelength ranges which may provide information related to the internal structure of the dental object such as the enamel/dentin junction, cracks or caries lesions inside the tooth. Scan data may additionally comprise processed information based on the information received directly from the scanner, such information could be shade values associated with the dental object, diagnostic scoring information or data quality check information.
A digital 3D representation should be understood as a point cloud, a point cloud with additional attributes like point normals, point colors, a triangulated point cloud (a triangle mesh), a polygonal mesh, a volumetric representation/voxel model, parametric representations, e.g. a spline representation. The digital 3D representation may additionally include a virtual representation of an object in three dimensions in which internal regions (structures, etc.) are arranged within the volume in three physical dimensions in proportion and relative to the other internal and surface features of the object which is being modelled. For example, a volumetric representation of a tooth may include the outer surface as well as internal structures within the tooth (beneath the tooth surface) proportionately arranged relative to the tooth, so that a section through the volumetric model would substantially correspond to a section through the tooth, showing position and size of internal structures.
A feedback signal should be understood herein as a signal provided to an output device for providing feedback (information) to a user. Alternatively, a feedback signal may simply refer to the actual feedback provided to the user during a scanning session. The feedback may relate to a variety of information related to scanning the dental object. Examples include whether scan data is acquired, the rate of which new scan data is acquired, whether scan patches are correctly registered, if registration of scan patches are unsuccessful, the progress of the scanning, whether the dental object is completely scanned, etc. Other information given as feedback may be envisioned without departing from the scope of the present disclosure.
In preferred embodiments, the feedback signal and/or the feedback is correlated to the rate of which new scan data is acquired or to the amount of added data. The added data may be determined by comparing scan data associated with a first and second 3D representation or by comparing second scan data to the first 3D representation or by comparing first and second scan data. Accordingly, the user should be able to (qualitatively) tell from the feedback signal/feedback the rate (high/low) of new data being acquired or the amount (high/low) of added data. Therefore, preferably the feedback is altered dynamically, preferably in real-time, during a scanning session, wherein the alteration/change reflects the change in the amount of added data and/or the change in the rate of new data acquired. New data should be understood herein as data covering new areas/surfaces of the dental object, which does not form part of the existing 3D representation. In other words, the presently disclosed system and method(s) provide a framework for providing dynamic feedback to the user during a scanning session, wherein the feedback is adjusted/altered based on the rate of which new scan data is acquired or based on the amount of new scan data/added data.
In preferred embodiments, the 3D scanner system comprises an output device for providing feedback as described herein. As an example, the output device may be configured to receive a feedback signal, which is modulated based on e.g. the rate of which new scan data is acquired. The output device may then further be configured to provide feedback to the user, e.g. in the form of audible feedback, tactile feedback, or visible feedback. Examples of suitable output devices include speaker(s), vibrating unit(s), and light source(s). Some embodiments of the 3D scanner system comprise multiple output devices. In some embodiments, the intraoral scanning device comprises a vibrating unit and a light source, such as an LED ring around the circumference of the scanning device. As an example, the light source may be configured to increase/decrease in brightness depending on the rate of which new scan data is acquired or depending on whether scan patches are correctly registered.
In preferred embodiments, at least one type of feedback is provided as a sound/audible signal. In preferred embodiments, the 3D scanner system comprises a speaker for providing the sound. In some embodiments, the speaker is external to the intraoral scanning device. In other embodiments, the intraoral scanning device comprises a speaker. In preferred embodiments, the sound comprises a plurality of sound pulses separated by a temporal distance (e.g. a time delay), i.e. the time between two pulses.
In some embodiments, the feedback signal is an auditory signal comprising a plurality of sound pulses separated by a time delay, wherein the time delay is changed based on the determined overlap. As an example, the time delay may be increased when the determined overlap is below a predefined lower threshold, and the time delay may be decreased when the determined overlap is above a predefined upper threshold. As another example, the time delay may be selected from a predefined first range when the determined overlap is above a predefined upper threshold, and the time delay may further be selected from a predefined second range when the determined overlap is below a predefined lower threshold. Preferably, the low end of the predefined second range is above the low end of the predefined first range. In some embodiments, the predefined first range covers the range 10 ms to 150 ms, such as 20 ms to 130 ms, such as 30 ms to 110 ms, and the predefined second range covers the range 50 ms to 250 ms, such as 60 ms to 200 ms, such as 70 ms to 150 ms. In some embodiments, the time delay is randomly generated within said first and second ranges. In experiments performed by the inventors, said first and second ranges were found particularly useful in providing feedback to the user, which could be intuitively understood by the user. One reason is that the sound of the auditory signal is familiar to vacuum cleaning over a plurality of smaller objects, e.g. sand or small stones, which creates a sound that depends on how fast the vacuum cleaner is moved across the objects. The ‘nudging effect’ is thus closely correlated with the choice of time delay between the sound pulses in the auditory signal.
The plurality of sound pulses may be experienced by the user as a ticking or clicking sound. As an example of how the feedback (here sound) may be correlated to the rate of which new scan data is acquired or correlated to the amount of new scan data/added data, the temporal distance between the sound pulses may be adjusted, preferably continuously and/or in real-time during the scanning session, such that the distance shortens when the amount of new scan data/added data is high. A high amount of new scan data may be understood as scan data (e.g. provided as a scan patch) having a small overlap with the existing 3D representation, i.e. scan data that covers at least some area which has not yet been imaged during said scanning session. Similarly, the sound may be adjusted by lengthening the temporal distance between the sound pulses, if the amount of new scan data/added data is low, i.e. if the new scan data has a high overlap with the existing 3D representation.
Obtaining current scan data of the dental object (101) is the first step of the computer-implemented method (100) and can be performed with a 3D scanner system, as shown in
The new scan data may generate an updated digital 3D representation by adding the new scan data to the digital 3D representation. Added information to the digital 3D representation can then be determined by comparing the new scan data to at least one of the digital 3D representation(s) and/or by comparing the updated digital 3D representation to the digital 3D representation(s). Based on the added information determined in the previous step, a feedback signal can be provided. In some embodiments, the step of determining added information comprises comparing the new scan data to the digital 3D representation, wherein said comparison comprises determining an overlap between the new scan data and the digital 3D representation.
The digital 3D representation (402) can be a three-dimensional representation of the object to be scanned. This object may be a dental object such as the patient's dentition. The digital 3D representation (402) can be generated based on the scan data. The scan data may be based on a processed group of images acquired by at least one image sensor, which can be comprised in an intraoral scanning device and which may be further comprised in the 3D scanner system (400).
The feedback signal is preferably correlated with the added information. In preferred embodiments, at least a property of the feedback signal is altered based on the determined overlap. In some embodiments, the property of the feedback signal is selected from the group of: frequency, amplitude, pitch, time delay between pulses of the signal, and/or combinations thereof. In other embodiments, the property of the feedback signal is selected from the group of: brightness level, vibration level, color of emitted light, and/or combinations thereof.
In particular, the feedback signal may be modulated depending on the added information. As previously discussed, providing a feedback signal while scanning facilitates scanning and creates an improved scanning experience. Additionally, the data quality may be improved. Time wasted by scanning the same area for too long is reduced and it can also help keeping the user's focus more on the dental object to be scanned and more generally on the patient while simultaneously avoiding overloading the processors.
A dentist will typically be scanning the patient continuously as the scanner is moved to cover the dental object. Accordingly, in order to provide a further improved scanning experience, one or more of the step(s) of the computer-implemented method (100) may in one embodiment be continuously performed during scanning of the dental object.
The added information is a parameter/value/representation which describes new relevant information added to an existing 3D representation. Thus, the information representing the added information may vary depending on the desired information obtained and the form in which the added information is stored. E.g. the added information may be understood as any added information provided by the scan data. Examples may be added surface data or an added texture data etc. Added data may be evaluated as a difference between new scan data and old scan data. Old scan data should be understood as scan data, which forms part of the 3D representation. Differences may be evaluated as an Euclidean distance in color space such as RGB or LAB colors. Added volumetric data, such as additional information related to the internal structures of the dental object. Added data as additional information related to surface properties like roughness or scattering properties or a combination hereof. In other words, added information should be understood as a difference or discrepancy in one or more characteristics between new scan data and existing scan data.
In one embodiment, the intraoral scanning device may acquire scan data as a combination of different modalities, such as reflected light information to obtain surface information and infra-light information to obtain information data of the internal structure of the dental object, such as enamel-dentin junction, cracks or caries. In such a scenario, some scan data information may not be visible to the user during the real time acquisition via the display, hence the scanning device and associated method disclosed herein is particular advantageous as feedback is provided to the user during scanning such that high quality data can be obtained. Internal structures of the dental object may be imaged with IR, near-IR light or any appropriate wavelength which make the dentition appear semi-transparent. For example, deep red light in the range 600 nm to 750 nm, or near infrared light in the range of 700 nm to 1090 nm (e.g., 850 nm) may be used. Other wavelengths and ranges of wavelengths may be used, including wavelengths shorter than the visible spectrum.
In another embodiment, the intraoral scanning device may acquire scan data comprising surface information, while the computer-implemented method is performed completely without a display to render the 3D representation real-time. In such a scenario, no visual guidance or feedback is provided by the display thus it is particular advantageous to provide dynamic feedback to the user, such that the user may be guided to complete scanning a high-quality 3D digital representation of a dental object without a monitor in short time and without overloading the processor with a too high number repetitive surface information data.
Furthermore, the added information may be obtained in many different forms, e.g. in one embodiment it may be a matrix representing different parameters, such as the mentioned geometry, color, and/or diagnostic information and in other embodiments the added information may be represented by a single numerical value. This numerical value may additionally be normalized such that a number between 0-1 will represent the amount of added information. In one example, the added information may result from the comparison between one or more digital 3D representation(s).
More specifically, the added information may result from the comparison of a new scan data with at least one of the existing digital 3D representation(s). The comparison may include any difference between the new scan data and the at least one of the existing digital 3D representation(s) such as an expansion of an existing surface information of the at least one of the existing digital 3D representation(s). As described in
In some embodiments of the computer-implemented method, the step of determining added information comprises the steps of:
Furthermore, in such embodiments, wherein a difference in surface area is determined, the feedback signal may be an auditory signal comprising a plurality of sound pulses separated by a time delay, wherein the time delay is changed based on the determined difference in surface area. Furthermore, the time delay may be increased when the determined difference in surface area is below a predefined lower threshold, and wherein the time delay is decreased when the determined difference in surface area is above a predefined upper threshold. The time delay may be selected from a predefined first range when the determined difference in surface area is above a predefined upper threshold, and the time delay may be selected from a predefined second range when the determined difference in surface area is below a predefined lower threshold. Preferably, the low end of the predefined second range is above the low end of the predefined first range, wherein the predefined first range covers the range 10 ms to 150 ms, such as 30 ms to 110 ms, and wherein the predefined second range covers the range 50 ms to 250 ms, such as 70 ms to 150 ms. Thus, the processing unit(s) of the 3D scanner system may be configured for determining a surface area of the digital 3D representation and the updated digital 3D representation, and the difference in surface area between said 3D representations may further be determined. In some embodiments, the feedback signal is correlated with the difference in surface area, such that the signal is correlated with how much the surface area of the digital three-dimensional has expanded after the step of updating the digital 3D representation.
In one embodiment, the added information may be evaluated based on surface information from the scan data. At least the new surface information obtained via the new scan data may be added to the digital 3D representation by registering the new surface information to existing surface information of the digital 3D representation to find correspondences such that the new surface information can be registered to the correct location on the digital 3D representation. Upon registration of a new scan data patch, the digital 3D representation may be updated by applying some information from the new scan data to the digital 3D representation.
In one example, the digital 3D representation may be represented as volumetric representation, which may contain voxels. Such a representation in known as a “Signed Distance Field” representation. Each voxel in the volumetric representation may be assigned at least one weight as a measure of data quality which can be used to define a surface, represent a color or any other property associated with the dental object being scanned. The weight of a voxel may be updated when a registered scan data patch provide data associated with that particular voxel, this process is generally referred to as “fusion” in the field of model reconstruction. Scan data received from the scanning device may additionally contain one or more weights associated with the scan data quality. A scan data weight may be based on surface information quality determined from the factors associated with depth map generation additionally a scan data weight may be based on texture quality information determined from factors associated with color acquisition like, angle between scanner and surface information, sensor saturation etc. There may be a scan data weight associated with each individual data element.
When a registered scan data patch is added to the 3D digital representation, the individual weights from the scan data may be added to the corresponding weights in each voxel of the digital 3D representation. Added information may thus be evaluated based on changes in individual or grouped voxels of the digital 3D representation when a new scan data patch is added. If the weight of one or more voxels of the digital 3D representation changes significantly, added data may be determined. By evaluating the change of individual weights one numerical value may be generated representing the extent of added information. Consequently, the feedback signal may be adjusted based on the determined added data or based on the numerical value.
In another example, a new scan data may be registered to its corresponding position to at least one previous scan data. By using the surface information associated with the individual scan data, a center of mass may be computed for each scan data patch. When a new scan data patch is registered to any previous scan data patches, the distances between the center-of-mass for the closest previous scan patches may be used to evaluate the added information as a measure of overlap between the scan patches. Threshold values may be defined, such that a distance between the center-of-mass of the new scan data to the closest corresponding precious scan data below a minimum threshold is equal to complete overlap and if a distance above a maximum threshold value equals maximum expansion of the surface information.
The possible difference between at least one of the existing digital 3D representation(s) with the new scan data of the dental object may be described by an update of data in an existing surface of the at least one of the digital 3D representation(s). This update of data can be a difference in color, fluorescence and/or texture of the dental object being scanned.
The scanning area (204) of the handheld intraoral scanning device (206) may also cover a part of the scanned area (201). As can be seen in
The partially scanned area (203) defines an area where some scan data has been previously scanned, but where some additional scan data are either missing, insufficient or with low quality, such as surface, color, texture information, shade measurement, and/or diagnostic data. The scanning scenario described in
Accordingly, at least two types of added information may be determined, one for the information received from non-scanned area (202) and another for the partially scanned area (203). The added information may in another embodiment be considered as the cumulative of the two, or in yet a further embodiment as an average.
Similarly, the feedback signal may be provided by using both types of added information or the added information with the highest content of relevant added information determined when comparing the new scan data, e.g. data received from either non-scanned area (202) and/or from the partially scanned area (203), with the digital 3D representation, such as the tooth scan (201).
In some embodiments, the added information is characterized by a numerical value, for example as discussed briefly with respect to
The feedback signal may be provided as a digital signal or an analogue signal. An analogue signal may be any continuous signal representing some other quantity to another quantity. For instance, in an analogue audio signal, the instantaneous signal voltage varies continuously with the pressure of the sound waves. In the case of the present disclosure, an analogue signal can be implemented as the feedback signal by continuously providing a feedback which may correlate to the added information. Preferably, a digital signal can be implemented as the feedback signal. The digital signal may be adapted to be sent to at least one output device. The at least one output device may be an audio device and/or a haptic device and/or a light-emitting device, or any combination of these three device types.
For instance, an audio device can provide an audible signal output while a haptic feedback may deliver a haptic output, e.g. rumbling, based on the feedback signal, to the user. The feedback signal may be configured differently depending on the type of feedback provided. For examples, a pattern of pulses may be used for haptic, audio and visual feedback, however, the amplitude and frequency may vary. As an example, the temporal distance between the pulses may be adjusted such that the frequency is increased or decreased, wherein the adjustment is correlated to the rate of new scan data acquired. In particular, the temporal distance may be decreased when the rate is high, corresponding to imaging new areas of the dental object, wherein the scan data has a relatively small overlap with the existing 3D representation. Similarly, the temporal distance may be increased when the rate is low, corresponding to imaging areas of the dental object, which largely already form part of the 3D representation, wherein the scan data has a relatively high overlap with the existing 3D representation.
In some embodiments multiple feedback signals may be generated, e.g. a first feedback signal adapted to be sent to an audio device, a second feedback signal adapted to be sent to a light-emitting device and/or a third feedback signal adapted to be sent to a haptic output device. Accordingly, the user may be prompted in different ways on where to scan.
Accordingly, one embodiment of a digital signal being adapted/configured to be sent to an audio device is disclosed, where the digital signal may be modulated based on the added information.
In another embodiment, the digital signal can be defined by different pitches that may correlate to the added information. The pitch is a perceptual property of sounds that allows their ordering on a frequency-related scale, or more commonly, pitch is the quality that makes it possible to judge sounds as “higher” or “lower” in the sense associated with musical melodies. In order to be audible, a generated sound may have its frequency comprised between 20 Hz to 20 kHz. In this specific example, the successive pulses can have different pitches, which would create different audible sounds depending on the added information. In other terms, the pitch may be correlated to the added information. In this context, the added information would be characterized by “lower” or “higher” audible sounds, depending on the added information. It may be preferred that “lower” audible sounds can be heard when the added information is relatively low, while “higher” audible sounds can be heard when the added information is relatively high.
The dental object may be an object that can be included in a mouth. More generally, the dental object can be for instance a jaw, more specifically gingiva (gum), floor of mouth, a tooth (200), a group of teeth, an implant or a restoration. The dental object may be an intraoral dental object of a patient, said dental object comprising e.g. teeth and/or gingiva of the patient. Such a dental object may further comprise other objects and/or materials inside the patient's oral cavity, for example implants or dental restorations. The dental object may only be a part of the patient's teeth and/or oral cavity, since the entire set of teeth of the patient is not necessarily scanned during a given scanning session. Examples of dental objects include one or more of: tooth/teeth, dental arch(es), implant(s), dental restoration(s), dental prostheses, edentulous ridge(s), and combinations thereof. Alternatively, the dental object may be a gypsum/plaster model representing a patient's teeth.
The scan data comprises information relating to the three-dimensional dental object. The scan data may comprise any of: 2D images, 3D point clouds, depth data, texture data, diagnostic data, intensity data, color data, 2D IR data and/or combinations thereof. As an example, the scan data may comprise one or more point clouds, wherein each point cloud comprises a set of 3D points describing the three-dimensional dental object. As another example, the scan data may comprise images, each image comprising image data e.g. described by image coordinates and a timestamp (x, y, t), wherein depth information can be inferred from the timestamp. The image sensor(s) of the scanning device may acquire a plurality of raw 2D images of the dental object in response to illuminating said object using the one or more light projectors. The plurality of raw 2D images may also be referred to herein as a stack of 2D images. The 2D images may subsequently be provided as input to the processor, which processes the 2D images to generate scan data. The processing of the 2D images may comprise the step of determining which part of each of the 2D images are in focus in order to deduce/generate depth information from the images. The depth information may be used to generate 3D point clouds comprising a set of 3D points in space, e.g., described by cartesian coordinates (x, y, z). The 3D point clouds may be generated by the processor or by another processing unit. Each 2D/3D point may furthermore comprise a timestamp that indicates when the 2D/3D point was recorded, i.e., from which image in the stack of 2D images the point originates. The timestamp is correlated with the z-coordinate of the 3D points, i.e., the z-coordinate may be inferred from the timestamp. Accordingly, the output of the processor is the scan data, and the scan data may comprise image data and/or depth data, e.g. described by image coordinates and a timestamp (x, y, t) or alternatively described as (x, y, z). The scanning device may be configured to transmit other types of data included in the scan data. Examples of additional data include 3D information, texture information, diagnostic modalities such as infra-red (IR) images for detecting internal structures of the dental object, fluorescence images, reflectance color images, x-ray images, and/or combinations thereof.
The scan data may comprise at least one two-dimensional (2D) image and/or a scan patch. The scan data may comprise any of 2D images, 3D point clouds, depth data, texture data, intensity data, colour data, and/or combinations thereof. As an example, the scan data may comprise one or more point clouds, wherein each point cloud comprises a set of 3D points describing the three-dimensional dental object. A scan patch can be defined as a combination of at least one distance and at least one 2D image. The scan patch can be generated via a scan procedure. A scan procedure may be performed with an optical sensor (205), placed at a given distance from the object to be scanned. Therefore, a combination of a distance from the optical sensor (205) to the object as well as at least one 2D image recorded at an instant may be comprised in the scan patch. The scan procedure can be done with a scanning device, such as an intraoral scanning device, comprising the optical sensor (205), which is operated by projecting a pattern and translating a focus plane along an optical axis of the scanning device and capturing a plurality of 2D images at different focus plane positions such that each series of captured 2D images corresponding to each focus plane forms a stack of 2D images.
More specifically, the focus plane position may be preferably shifted along the optical axis of the scanning system, such that 2D images captured at a number of focus plane positions along the optical axis form said stack of 2D images for a given view of the object, i.e. for a given arrangement of the scanning system relative to the object. This stack of 2D images may also be referred as a sub-scan.
These steps can be continuously performed as previously discussed. As shown in
The 3D scanner system may also comprise a scanning device or any other type of devices that can comprise at least one optical sensor (205).
The 3D scanner system (400) may be a dental scanning system that may include an intraoral scanning device such as the TRIOS series scanners from 3Shape A/S or a laboratory-based scanner such as the E-series scanners from 3Shape A/S.
The intraoral scanning device may comprise one or more light projectors configured to generate an illumination pattern to be projected on a three-dimensional dental object during a scanning session. The light projector(s) preferably comprises a light source, a mask having a spatial pattern, and one or more lenses such as collimation lenses or projection lenses. The light source may be configured to generate light of a single wavelength or a combination of wavelengths (mono- or polychromatic). The combination of wavelengths may be produced by using a light source configured to produce light (such as white light) comprising different wavelengths. Alternatively, the light projector(s) may comprise multiple light sources such as LEDs individually producing light of different wavelengths (such as red, green, and blue) that may be combined to form light comprising the different wavelengths. Thus, the light produced by the light source may be defined by a wavelength defining a specific color, or a range of different wavelengths defining a combination of colors such as white light. In an embodiment, the scanning device comprises a light source configured for exciting fluorescent material of the teeth to obtain fluorescence data from the dental object. Such a light source may be configured to produce a narrow range of wavelengths. In another embodiment, the light from the light source is infrared (IR) light, which is capable of penetrating dental tissue. The light projector(s) may be DLP projectors using a micro mirror array for generating a time varying pattern, or a diffractive optical element (DOF), or back-lit mask projectors, wherein the light source is placed behind a mask having a spatial pattern, whereby the light projected on the surface of the dental object is patterned. The back-lit mask projector may comprise a collimation lens for collimating the light from the light source, said collimation lens being placed between the light source and the mask. The mask may have a checkerboard pattern, such that the generated illumination pattern is a checkerboard pattern. Alternatively, the mask may feature other patterns such as lines or dots, etc.
The scanning device preferably further comprises optical components for directing the light from the light source to the surface of the dental object. The specific arrangement of the optical components depends on whether the scanning device is a focus scanning apparatus, a scanning device using triangulation principle, confocal scanning, depth of defocus, light field, stereoscopic, deep learning based 3D measurements or any other type of scanning device.
In some embodiments, the intraoral scanning device is based on a focus scanning principle, such as depth-from-focus. In such embodiments, the scanner comprises an optical element, such as a focus lens, which is configured to move back and forth during scanning to change the focus of the scanner, whereby the depth can be estimated based on a focus measure. A focus scanner is further described in EP 2 442 720 B1 by the same applicant, which is hereby incorporated herein by reference. In some embodiments, the intraoral scanning device is based on a depth-from-defocus scanning principle, wherein an optical property, such as an aperture, is changed between the acquisition of two images, whereby depth can be estimated by determining the degree of defocus between the two images.
In some embodiments, the intraoral scanning device is based on triangulation, wherein at least one image sensor and a projector unit are positioned such that they form a triangle with respect to a point on the scanned surface. As an example, a projector and an image sensor may be utilized to determine points in 3D space based on triangulation. Alternatively, the intraoral scanning device may comprise two or more image sensors viewing the scene or scanned object from two different directions, wherein the image sensors are configured to acquire a set of images, wherein a correspondence problem is solved based on triangulation. The correspondence problem generally refers to the problem of ascertaining which parts of one image correspond to which parts of another image. Specifically, the projector unit may be configured to project a plurality of projector rays, which are projected onto a surface of the dental object. In particular, solving the correspondence problem may include the steps of determining image features in the images within a set of images, and further associate said image features with a specific projector ray. Subsequently, the depth of each projector ray may be computed, whereby a 3D representation of the scanned object may be generated.
The light reflected from the dental object in response to the illumination of the dental object is directed, using optical components of the scanning device, towards the image sensor(s). The image sensor(s) are configured to generate a plurality of images based on the incoming light received from the illuminated dental object. The image sensor may be a high-speed image sensor such as an image sensor configured for acquiring images with exposures of less than 1/1000 second or frame rates in excess of 250 frames pr. second (fps). As an example, the image sensor may be a rolling shutter (CCD) or global shutter sensor (CMOS). The image sensor(s) may be a monochrome sensor including a color filter array such as a Bayer filter and/or additional filters that may be configured to substantially remove one or more color components from the reflected light and retain only the other non-removed components prior to conversion of the reflected light into an electrical signal. For example, such additional filters may be used to remove a certain part of a white light spectrum, such as a blue component, and retain only red and green components from a signal generated in response to exciting fluorescent material of the teeth.
The 3D scanner system preferably further comprises a processor configured to generate scan data (intraoral scan data) by processing the two-dimensional (2D) images acquired by the scanning device. The processor may be part of the scanning device. As an example, the processor may comprise a Field-programmable gate array (FPGA) and/or an Advanced RISC Machines (ARM) processor located on the scanning device
The one or more processing unit(s) may be comprised in the 3D scanner system, wherein the one or more processing unit(s) may be either comprised in the desktop or laptop computer (401) or in the scanning device or any other type of devices that can comprise at least one optical sensor (205). More generally, the scanning device or any other type of devices may comprise a scanner housing which can comprise the one or more processing unit(s).
The 3D scanner system (400) may further comprise at least one output device for providing an output to a user during scanning. The at least one output device can be an audio device and/or a haptic device and/or a light-emitting device. The audio device can be a speaker, which may be wireless or wire-based connected to the intraoral scanning device. The power of the speaker may depend on a specific user need. The speaker may be either comprised in the housing of the intraoral scanning device (400), or can also be an external device. The haptic device is based on the haptic technology, which refers to any technology that can create an experience of touch by applying forces, vibrations, or motions to a user. The haptic device can be any device that may be able to implement a haptic effect to a user. The implementation of a haptic technology can be divided in multiple categories such as vibration, force feedback, air vortex rings or ultrasound. One potential implementation of a feedback signal in the intraoral scanning device can be a vibration effect which may correlate to the added information. The vibration effect may be modulated depending on the added information. The light-emitting device can be any device that can emit light. The light-emitting device may be a light-emitting diode. A light-emitting diode may have the advantage of having a light weight, a good efficiency and an option of emitting different colours in a relatively small area. The light-emitting device can be comprised in the intraoral scanning device, but it may also be a peripheral (auxiliary) device, which may have to be connected to the 3D scanner system in order to get the output based on the feedback signal from the intraoral scanning device. The light-emitting device can be comprised in the intraoral scanning device, wherein the light-emitting device may be comprised in the housing of a handheld intraoral scanning device (206).
A handheld intraoral scanning device (206) may comprise a scanner housing and can be configured to be handled with one hand. The scanner housing may comprise:
The handheld intraoral scanning device (206) may be configured for receiving at least one feedback signal that may be provided by the computer-implemented method (100) previously described. The handheld intraoral scanning device (206) may provide an output to an user from the one or more output device(s) based on the at least one feedback signal.
The one or more processing unit(s) may process the scan data obtained from an optical sensor (205). The one or more processing unit(s) can also output the scan data to a peripheral device, which can then process the scan data, outside from the handheld intraoral scanning device (206). The one or more processing unit(s) may also pre-process the scan data before outputting a pre-processed scan data to a peripheral device. The pre-processed scan data can be any step from a step following the acquisition of the scan data of a dental object until a step of delivering an information to an external screen which can be visualized by a user. More generally, the one or more processing unit(s) being comprised in the handheld intraoral scanning device (206) can perform one, more or all of the steps of the computer-implemented method (100).
The one or more output device(s) of the handheld intraoral scanning device (206) can be at least one audio device and/or at least one light-emitting device and/or at least one haptic device. The one or more output device(s) may be located in the housing of the handheld intraoral scanning device (206) but may also be located on the housing, such as it becomes visible to the user, especially in the potential case of the one or more output device(s) being at least one light-emitting device.
In one embodiment, the handheld intraoral scanning device (206) may comprise at least one audio device, the at least one audio device being preferably a speaker. The handheld intraoral scanning device (206) can further comprise a wireless module, which may be able to communicate with at least one audio device being potentially external of the handheld intraoral scanning device (206) housing. The handheld intraoral scanning device (206) may also comprise a jack plug or any other plugs that can allow the handheld intraoral scanning device (206) to be connected to an audio device. The at least one audio device can be configured to output the feedback signal.
In another embodiment, the handheld intraoral scanning device (206) can comprise the at least one light-emitting device. The at least one light-emitting device can be a device comprising at least one LED device. The light-emitting device, such as the at least one LED device, may be configured to output different colours and/or different brightness. This would potentially allow the user to get different information based on the feedback signal. As an example, the brightness of the light-emitting device can be correlated the added information, which means that the brightness may be correlated to the feedback signal. In that specific configuration, the feedback signal can be adapted/pre-processed by either the handheld intraoral scanning device (206) or in the housing of the handheld intraoral scanning device (206), in order to be adapted for being outputted to the light-emitting device.
The light-emitting device may be at least one LED ring (403), as shown in
In some embodiments, the scanning device comprises a beam splitter. A beam splitter may be understood as an optical device configured to split a beam of light in two beams. The beam splitter may be made from two triangular glass prisms which are attached together at their base. The face where the two triangular glass prisms are attached may form an interface, where incident light may be transmitted and/or reflected. The interface may be coated with a coating. The coating may be configured to allow light within a certain range of wavelengths of a certain polarization to pass. Thus, the beam splitter may be configured to accept light of a certain range of wavelengths, e.g. being unpolarized, and then output/transmit light of the same wavelengths of linear polarization.
The beam splitter may be configured such that the polarization of two exiting beams from the beam splitter are orthogonal to each other, e.g. one beam of horizontally linearly polarized light and another beam of vertically linearly polarized light. Thus, the beam splitter may be a polarizing beam splitter. The beam splitter may be configured to receive an incident beam of light and output two beams of light, wherein at least one of said beams is linearly polarized. Thus, the beam splitter may be configured to receive unpolarized light and output linearly polarized light of two different orientations. The beam splitter may be a beam splitter cube comprising six square faces. The inside of the beam splitter cube may comprise an interface where the two triangular glass prisms are attached together. Preferably, the beam splitter is configured to transmit linearly polarized light of a first orientation, e.g. vertical, and further configured to reflect linearly polarized light of a second orientation, e.g. horizontal. The light projected towards the distal end of the scanning device may be linearly polarized of a first orientation, whereas light returning from the object throughout the optical system may be linearly polarized of a second orientation, wherein the beam splitter is configured to reflect the light of the second orientation onto the image sensor(s) of the scanning device.
Techniques consistent with the present disclosure provide, among other features, systems and methods for generating a digital three-dimensional representation of a dental object during scanning with an intraoral scanning device and providing feedback during scanning of said dental object. While various exemplary embodiments of the disclosed system and method have been described above it should be understood that they have been presented for purposes of example only, not limitations. It is not exhaustive and does not limit the disclosure to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practicing of the disclosure, without departing from the breadth or scope. Although operations can be described as a sequential process, some of the operations can in fact be performed in parallel, concurrently, and/or in a distributed environment, and with program code stored locally or remotely for access by single or multi-processor machines. In addition, in some embodiments the order of operations can be rearranged without departing from the spirit of the disclosed subject matter. It will be appreciated by those skilled in the art that the present disclosure can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The presently disclosed embodiments are therefore considered in all respects to be illustrative and not restrictive. The scope of the disclosure is indicated by the appended claims rather than the foregoing description, and all changes that come within the meaning, range, and equivalence thereof are intended to be embraced therein.
| Number | Date | Country | Kind |
|---|---|---|---|
| 22171602.0 | May 2022 | EP | regional |
