Patent Application
20210132223
The present invention is in the field of ultrasound imaging. More particularly, the present invention relates to a method of ultrasound imaging employing novel beamforming procedures, as well as to an apparatus employing such method.
Ultrasound imaging is a well-established diagnostic modality that continuously undergoes tremendous development. Most of the basic research on ultrasound imaging in the early loth century was conducted with single transducer elements capable of converting sound signals into electrical signals and vice versa, where the transducer elements are typically formed by piezoelectric elements. Today's ultrasound transducers however usually consist of an array of transducer elements, typically within the range of 64 to 256 transducer elements. The array may be a 1D array for conventional imaging or a 2D array for so-called volumetric imaging.
Beamforming is employed in signal processing for both transmission and reception of acoustic waves to improve the directivity as well as the sensitivity of resulting data. In general terms, “beamforming” can be referred to as a technique to control electronic parameterization and signal transformation for the generation of ultrasound signals, also referred to as “transmit beamforming” herein, as well as to the processing of received and detected ultrasound signals to allow for a desired quality of the target signal (“receive beamforming”). Herein, the “quality” may refer to characteristics such as spatial resolution of distinguishable objects, signal-to-noise ratio, contrast or similar measures. An important aim of beamforming procedures is that by controlling the joint emission and reception of the array of transducer elements, the properties of the transmitted acoustic or electromagnetic waves can be optimized for specific locations in space such that the quality is increased.
A commonly used beamforming strategy is the so-called delay-and-sum beamforming technique, as e.g. described in Thomenius, ICE.: Evolution of ultrasound beamformers. In: IEEE Ultrasonics Symposium. Volume 2. IEEE (1996) 1615-1622. In delay-and-sum beamforming, in the transmit mode, a plurality of adjacent transducer elements are activated for generating a beam, where the activation of the transducer elements located on or close to the beam axis is delayed with regard to the activation of transducer elements further away from the beam axis. As a result of the transmit delays, the effective wavefront emitted into the tissue is optimized for focusing effects at a desired point in space, due to the compensation of varying times of arrival of the emitted pulses from each transducer element at the focal point. After receiving reflected signals from the tissue, a similar procedure can be applied in receive beamforming, where individual signals received by each transducer element are once again delayed and summed up for all receiving elements.
Delay-and-sum beamforming is currently the most common implementation for ultrasound beamforming. The main advantage of delay-and-sum beamforming is its relatively low computational complexity while providing reasonable image quality. In fact, due to the low computational complexity, both transmit and receive beamforming can be carried out in real time, i.e. while taking for example 50 ultrasound images per second.
In the art, there have been tremendous efforts to improve the image quality over images formed using delay-and-sum beamforming. For this purpose, advanced, data-dependent beamforming strategies have been proposed. An important example of advanced, data-dependent beamforming strategies is the so-called delay-multiply-and-sum method, as described e.g. in Matrone, G., Savoia, A. S., Galiano, G., Magenes, G.: The delay multiply and sum beamforming algorithm in ultrasound B-mode medical imaging. IEEE Transactions on Medical Imaging 34(4) (2015) 940-949. A further high-performance beamforming strategy is the Maximum Likelihood beamforming or Minimum Variance (MV) beamforming, as disclosed e.g. in Li, J., Stoica, P.: Robust adaptive beamforming. Volume 88. John Wiley & Sons (2005). Compared to the traditional data-independent delay-and-sum beamforming technique, these approaches yield a narrower main lobe and reduced side lobes of the received beams, improving both image resolution and contrast. However, such alternative approaches are hindered by a substantially higher computational cost, dramatically decreasing their real-time applicability.
In areas other than ultrasound imaging, learning approaches applied to multi-channel problems conceptually related to beamforming have been researched. For example, learning methods have been considered for significantly improving resulting signal quality in speech recognition. The conceptual relationship of these works as understood by the inventors relies on the fact that these works process complex, multi-channel audio data to a core signal, and the inventors found that insights gained from research applied to speech recognition are therefore also applicable to ultrasound imaging. Moreover, advantages of an end-to-end approach using convolutional neural networks (CNNs) on raw time-domain wave data, over frequency-domain, have first been demonstrated in the field of acoustic modeling, see e.g. Hoshen, Y., Weiss, R. J., Wilson, K. W.: Speech acoustic modeling from raw multichannel waveforms. In: IEEE International Conference on Acoustics, Speech and Signal Processing. (2015) 4624-4628. This approach was further explored in the context of an end-to-end speech recognition framework, as described in Sainath, T. N., Weiss, R. J., Wilson, K. W., Li, B., Narayanan, A., Variani, E., Bacchiani, M., Shafran, I., Senior, A., Chin, K., Misra, A., Kim, C.: Multichannel signal processing with deep neural networks for automatic speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 25(5) (2017) 965-979. CNN-supported beamforming techniques have been proposed by Xiao, X., Watanabe, S., Erdogan, H., Lu, L., Hershey, J., Seltzer, M. L., Chen, G., Zhang, Y., Mandel, M., Yu, D.: Deep beamforming networks for multi-channel speech recognition. In: IEEE ICASSP. (2016) 5745-5749, to estimate the optimal beamforming parameter settings.
In Yoon, Y. H., Ye, J. C.: Deep learning for accelerated ultrasound imaging. arXiv preprint arXiv:1710.10006 (2017), CNNs have been applied to sparse raw channel data within a compressed-sensing framework, in order to fill in missing signals. However, in these studies, learning was applied for pre-beamforming tasks, such as parameter-optimization and signal reconstruction, but the beamforming per se was performed using a conventional implementation.
In the art, there remains a need for ultrasound imaging with high image quality without sacrificing real-time imaging capabilities.
The problem underlying the invention is to provide a method and an apparatus for ultrasound imaging of an object which allows for improved ultrasound image quality at high processing speeds.
This problem is solved by a method according to claim 1 and by an apparatus according to claim 15. Preferable embodiments are defined in the dependent claims.
According to one aspect of the invention, a method of ultrasound imaging of an object is provided, using an ultrasound transducer which comprises an array of transducer elements capable of converting sound signals into electrical signals and vice versa. The method comprises the following steps:
Herein, the formulation according to which the reflected signals “resemble a set of two-dimensional ultrasound data” may be understood to mean that the signals can be represented by such set of two-dimensional ultrasound data. Herein, the term “two-dimensional ultrasound data” shall be understood to be at least two-dimensional, i.e. it could be of precisely two dimensions or of higher dimensions. Moreover, the “scan object” could for example be a scan line, which represents sound reflection at various depths along a line extending from said transducer, and in particular from a given transducer element of the transducer, into the object subjected to imaging. However, other types of scan objects are likewise possible, including plane wave imaging, where a scan object represents the sound reflection at various depths, spanning the volume that was originally subject to the emitted sound wave.
Moreover, the term “first/second subset of transducer elements” may refer to any selection of transducer elements within the array, including all transducer elements within the array, and is in particular not limited e.g. to subsets of directly adjacent transducer elements or the like.
The inventors could confirm that by employing a machine learning based receive beamforming model for mapping said two-dimensional ultrasound data to said scan object, ultrasound images of very high quality can be obtained in real time, once the beamforming model has been properly trained using machine learning. Indeed, it is seen that the quality of the ultrasound images is mainly limited by the quality of the training data, and hence the beamforming that was applied in the training data. Accordingly, the receive beamforming model can be trained using image data obtained with sophisticated, computationally expensive beamforming methods that would be prohibitive for real-time ultrasound imaging, and similar image quality can be obtained with a fully trained model. In other words, all of the explicit formulations of the sophisticated beamforming processes, such as “delay-multiply-and-sum”or “Minimum Variance” beamforming, which are responsible for high computational time, are suppressed in the receive beamforming model of the invention and replaced by an implicit inferring step. The inferring step is completely independent of the mimicked sophisticated beamforming method in terms of its computational speed, and can require as little as 0.03 seconds per image in a current embodiment of the invention.
In a preferred embodiment, said machine learning based receive beamforming model employs one of a deep convolutional neural network or a recurrent neural network. The inventors have found out that both, recurrent neural networks and deep convolutional neural networks are particularly suitable for creating a receive beamforming model for the purposes of the invention. Indeed, both, recurrent neural networks and deep convolutional networks have the ability to learn custom spatial and temporal filters of an arbitrary dimension over the ultrasound data in order to create a mapping to beamformed scan data. Furthermore, representations of ultrasound data which display inferred contextual and diagnostic information may also be generated. One example of such “diagnostic information” is a spatial distribution of speed of sound, as is explained in more detail below. The use of recurrent and convolutional neural network based filters allows for the steps to be concurrently performed in an optimal manner in a learning context.
In a preferred embodiment, said receive beamforming model is an end-to-end beamforming model receiving said two-dimensional ultrasound data as an input and directly converting it into said scan object.
In an alternative embodiment, said receive beamforming model receives said two-dimensional ultrasound data and maps it onto a set of delay values and weight values for use in a delay-and-sum receive beamforming algorithm. Indeed, the performance of ordinary delay-and-sum receive beamforming can be considerably improved if the delay values and weight values are properly chosen. In particular, in conventional delay-and-sum beamforming, the delays are calculated based on the assumption of a constant speed of sound in human tissue, which is generally assumed to be 1540 m/s. However, upon a closer look, it is found that depending on the exact nature of the imaged tissue, this value can indeed vary as much as +/−7%, and as a result of this, the delay values neglecting these variations are not precise, which the inventors believe to be the main source of errors in conventional delay-and-sum beamforming. However, using a machine learning based beamforming model, more appropriate delay values and also improved weight values can be generated from said two-dimensional ultrasound data and then fed into a per se known delay-and-sum receive beamforming algorithm. Note that as before, said “two-dimensional ultrasound data” means that this data is “at least two-dimensional”. Accordingly, a step of “receiving said two-dimensional ultrasound data and mapping it onto a set of delay values and weight values” does not e.g. rule out a scenario in which spatial similarity of neighbouring scan objects is used by employing 3D convolution filters, for example by filtering over a 3D collection of 3D scan objects to gain greater confidence about the generated image.
Note that in both variants, the end-to-end beamforming model or the beamforming model mapping two-dimensional ultrasound data into a set of delay values and weight values for use in a delay-and-sum receive beamforming algorithm have an implicit or encoded “understanding” of the speed of sound within the object, since they allow for compensating the effects of a deviation of the true, local speed of sound from an assumed, constant global speed of sound. Moreover, in a preferred embodiment, said receive beamforming model is further configured to determine a spatial distribution of speed of sound within the object. In other words, in this variant, the beamforming does not only make implicit use of information regarding the spatially dependent speed of sound, but determines it explicitly. In a preferred embodiment, the method further comprises indicating speed of sound related information in the visual image. This speed of sound related information allows for augmenting the ultrasound image with further useful information. Namely, different tissue types, such as fat, blood, muscle, vascular structures, myocardial tissue, et cetera have specific, slightly different and well characterized speed of sound. Therefore, generating an image in which a precise speed of sound information is known for each pixel enables immediate and automatic identification of different types of tissues in the entire image. In preferred embodiments, the machine learning based beamforming model carries out tissue clustering and boundary classification, which again can be well implemented in machine learning, in particular, employing deep convolutional neural networks or recurrent neural networks.
To the knowledge of the inventors, automatic speed of sound retrieval for specific regions in the tissue does not exist yet in ultrasound imaging. However, this task can be achieved with preferred embodiments of the machine learning based beamforming models of the invention. For this purpose, tissues can be insonified from different angles and the variance between receive signals originating from the same point in space can be minimized. This means that additional information may be used to produce an image. The same location may be imaged from several angles of incidence. The gray-level value of every voxel in space may be determined in an optimal fashion, not by simply averaging the collection of acquired planes, but by first determining the optimal delay and weight coefficients for each channel in such way that the variance is minimized.
In a preferred embodiment, one or both of said first and second subsets of transducer elements corresponds to a number of transducer elements within a predefined aperture region centered at a given transducer element.
In preferred embodiments, said first and second subsets of transducer elements may overlap with each other, wherein preferably at least 50%, more preferably at least 75% of the transducer elements in one of said first and second subsets is also part of the other one of said first and second subsets, wherein said first and second subsets are most preferably identical.
In some embodiments, the first subset of transducer elements may be larger than the second subset of transducer elements. In some embodiments, the same first subset may be combined with different second subsets. For example, the first subset may even correspond to the entire array of transducer elements, i.e. a “full aperture”, while different second subsets are used as receive channels for receive beamforming.
As was mentioned above, the “scan object” may be a scan line, representing sound reflection at various depths along a line extending from said transducer, in particular from a given transducer element of said transducer, into the object subjected to imaging.
In a preferred embodiment, said step of constructing a visual image from said plurality of scan objects comprises one or more of a demodulation, in particular a demodulation via envelope detection, a logarithmic compression and a scan conversion/re-interpolation. Herein, scan conversion can for example involve a process of taking linear data and interpolating the same into a different coordinate system. This could for example involve interpolating scan lines that can be acquired on a fan to a rectangular grid, or the like.
In a preferred embodiment, said machine learning based receive beamforming model has been trained, or is obtainable by training using training data obtained with different conventional receive beamforming methods, and in particular with one or more of delay-and-sum beamforming, delay-multiply-and-sum and minimum variance beamforming.
In particular, said machine learning based receive beamforming model may have been trained, or obtainable by training, in a procedure, in which
This embodiment accounts for the fact that the performance of a machine learning based receive beamforming model is dependent on and hence limited by the quality of the underlying training data. This means that when training the machine learning based receive beamforming model using data generated with a single conventional receive beamforming method, it will eventually mimic its behavior both, with regard to its strengths, but also with regard to its deficiencies. According to this embodiment, it is suggested to carry out a plurality of beamforming procedures on the same two-dimensional ultrasound data but using different conventional receive beamforming methods, which then leads to a corresponding number of different scan objects. Based for example on an automatically retrieved quality metrics, such as ultrasound confidence maps, image statistics or the like, a resultant scan object can then be selected or derived from said plurality of different scan objects, which exhibits the best quality. The training is then carried out based on this resultant scan object, which means that the thus obtained machine learning based receive beamforming model combines the advantages and strengths of various different conventional beamforming methods in one model.
In a particularly preferred embodiment, in step A), the activation of said first subset of transducer elements, and in particular a relative weight and/or a relative delay of their activation, is controlled using a transmit beamforming procedure employing a machine learning based transmit beamforming model that has been trained, or is obtainable by training in combination with said machine learning based receive beamforming model, and that receives, as at least part of its input, said two-dimensional ultrasound data or said scan objects.
According to this embodiment, the machine learning is hence not only employed on the receive side, but also for transmit beamforming. This allows for a yet further significant increase in imaging quality. In particular, using machine learning, the transmit beamforming model can implicitly learn from the two-dimensional ultrasound data or scan objects information related to the location dependent speed of sound and use this implicit knowledge in an optimum transmit beamforming leading to optimal image quality, in combination with the receive beamforming with which the transmit beamforming has been trained.
In an alternative embodiment, the activation of said first subset of transducer elements, and in particular a relative weight and/or a relative delay of their activation is controlled using information regarding a spatial distribution of speed of sound within the object determined by means of said receive beamforming model. This variant exploits the fact that along with or as part of the machine learning based receive beamforming, information regarding the spatial distribution of speed of sound in the object to be imaged can be obtained. Once this information is available, it can be exploited in transmit beamforming, where the delays of the activation of individual transducer elements within this first set of transducer elements can be precisely adjusted such that the ultrasound pulses generated by the individual transducer elements arrive simultaneously at the envisaged focal point.
A further aspect of the invention relates to an apparatus for ultrasound imaging of an object. The apparatus comprises ultrasound transducer which comprises an array of transducer elements capable of converting sound signals into electrical signals and vice versa. The apparatus further comprises a control unit, wherein said control unit is configured for controlling the apparatus to carry out the following steps:
In a preferred embodiment, said machine learning based receive beamforming model employs one of a deep convolutional neural network or a recurrent neural network.
In a preferred embodiment, said receive beamforming model is an end-to-end beamforming model receiving said two-dimensional ultrasound data as an input and directly converting it into said scan object.
In an alternative embodiment, said receive beamforming model is configured to receive said two-dimensional ultrasound data and to map it onto a set of delay values and weight values for use in a delay-and-sum receive beamforming algorithm.
The receive beamforming model may further be configured to determine a spatial distribution of speed of sound within the object, wherein the control unit may further be configured for indicating speed of sound related information in the visual image.
In a preferred embodiment, one or both of said first and second subsets of transducer elements corresponds to a number of transducer elements within a predefined aperture region centered at a given transducer element. Preferably, said first and second subsets of transducer elements overlap with each other, wherein preferably at least 50%, more preferably at least 75% of the transducer elements in one of said first and second subsets is also part of the other one of said first and second subsets. In particular, said first and second subsets may be identical.
In some embodiments, the first subset of transducer elements is larger than the second subset of transducer elements. Herein, a same first subset is preferably combined with different second subsets. For example, the first subset may correspond to the entire array of transducer elements, while different second subsets may be used as receive channels for receive beamforming.
In a preferred embodiment, said scan object is a scan line, representing sound reflection at various depths along a line extending from said transducer, in particular from a given transducer element of said transducer, into the object subjected to imaging.
In a preferred embodiment, said step of constructing a visual image from said plurality of scan objects comprises one or more of a demodulation, in particular a demodulation via envelope detection, a logarithmic compression and a scan conversion/re-interpolation.
In various embodiments, said machine learning based receive beamforming model has been trained, or is obtainable by training using training data obtained with different conventional receive beamforming methods, and in particular with one or more of delay-and-sum beamforming, delay-multiply-and-sum and minimum variance beamforming.
In a preferred embodiment, said machine learning based receive beamforming model has been trained in a procedure or is obtainable by training in a procedure, in which
In a preferred embodiment, said control unit is configured to control in step A) the activation of said first subset of transducer elements, and in particular a relative weight and/or a relative delay of their activation,
For the purposes of promoting an understanding of the principles of the invention, reference will now be made to a preferred embodiment illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended, such alterations and further modifications in the illustrated apparatus and such further applications of the principles of the invention as illustrated therein being contemplated as would normally occur now or in the future to one skilled in the art to which the invention relates.
Further schematically shown in
For generating the individual activation pulses 20 for individual transducer elements 12, a carrier signal 22 is combined with a shape signal 24 such as to generate a main pulse 26. The individual activation pulses 20 are derived from the main pulse 26. As is seen in the schematic representation of
The thus generated ultrasound pulse will be reflected at the tissue and reflected signals are detected in a time resolved manner by means of a second subset of said transducer elements 12, which may, but need not be the same as the first subset. Herein, the timing information of the detected signal is associated with information regarding the depth in which the detected signal was reflected within the object subjected to imaging. Accordingly, the reflected signals associated with the second subset of transducer elements resemble essentially a set of two-dimensional ultrasound data, of which one dimension represents the various transducer elements of said second subset and the other dimension represents depth information. The two-dimensional ultrasound data are then converted into a scan object. In the embodiment shown, the scan object is a scan line, associated with the beam axis 16, representing sound reflection at various depths along said beam axis 16. The conversion of the two-dimensional ultrasound data into the scan object is referred to as receive beamforming herein.
The beamforming workflow according to an embodiment of the invention is shown in more detail with reference to
During transmission, an ultrasound pulse of a given frequency is propagated in the tissues through electrical excitation of nA aperture elements centered around the element e. The direction and focal depth of the propagated wave is controlled through transmit beamforming as explained above. After propagation and reflection in the tissue, the received signals impinging on the transducer 10 constitute the raw channel data, of size nA×d, which is an example of the “two-dimensional ultrasound data” referred to above.
During receiving, a receive beamforming operation is carried out for converting the two-dimensional ultrasound data into a scan object. The receive beamforming procedure effectively accounts for differences in distance of individual transducer elements 12 from a given site of sound reflection within the object. This way, a 1D scan line, or radio frequency signal (RF, size 1×yRF) is obtained. Finally, to create a visually interpretable B-mode (i.e. “brightness mode”) image, the RF signal undergoes demodulation, for example via envelope detection, a log compression, to improve the dynamic range, and a scan conversion, which may comprise a re-interpolation from xRF×yRF to xB×yB.
While the receive beamforming procedure accounts for differences in distance of individual transducer elements 12 from the reflection site, in the end-to-end beamforming version of the invention, this is not carried out by introducing explicit delays to the received signals or the like. Instead, the receive beamforming procedure employs a machine learning based receive beamforming model for mapping the two-dimensional ultrasound data to the scan line. Since this process combines beamforming with deep learning, the process is also referred to as “deepforming” herein, and the model is referred to as a “deepformer”. The deepformer may be trained to map the receiving, demodulation, and log compression steps of the ultrasound image formation pipeline as shown in
The machine learning based receive beamforming models may employ one of a convolutional neural network or a recurrent neural network. As was pointed out above, the inventors have noticed that both, recurrent neural networks and deep convolutional neural networks are particularly suitable for creating a receive beamforming model for the purposes of the invention. One of the reasons for this is that recurrent neural networks and deep convolutional networks have the ability to learn custom spatial and temporal filters of an arbitrary dimension over the ultrasound data in order to create a mapping to beamformed scan data. Furthermore, representations of ultrasound data which display inferred contextual and diagnostic information may also be generated. The use of recurrent and convolutional neural network based filters allows for the steps to be concurrently performed in an optimal manner in a learning context. In one embodiment, the neural network employed for the end-to-end deepformer is a deep convolutional neural network with residual skip connections and 4 bottlenecks. The network used in a currently operative version uses 50 total layers. In
The blocks each describe computations performed on their inputs as indicated by the directed connections (the “arrows”). If an operation is labeled as “/2”, the operation is applied to every second pixel, resulting in an output of half the size.
The “Cony” blocks apply filters of the specified size (e.g. 3×3 pixels), “Pool” and “Avg pool” accumulate their input in a small spatial window, thereby performing smoothing. Blocks called “Res” are so called residual blocks, their structure is shown in
The input “ultrasound data” as depicted in
As was indicated above, the performance of receive deepforming is limited by the quality of the underlying training data. When training the deepformer using data generated with a single conventional receive beamforming method, the deepformer will eventually mimic its behavior both, with regard to its strengths, but also with regard to its deficiencies. In a sophisticated implementation of deepforming as an end-to-end receive deepformer, for obtaining optimum training data, the output of multiple advanced conventional beamforming can be associated with a quality metric that can mathematically dictate the properties of an output image, for example color map, contrast etc. This way, an optimized scanline can be constructed based on any given two-dimensional ultrasound input data, using the strengths of various existing advanced beamforming methods, and this optimized scanline, together with the two-dimensional ultrasound data, can be used for training the deepformer.
A conceptual dataflow diagram is shown in
As was explained above, one underlying assumption for all currently practiced beamforming approaches is a constant and predefined speed of sound in human tissue, amounting to 1540 m/s in most medical systems. This assumption is not only used for transmit beamforming, but also in all prior art receive beamforming methods known to the inventors. By modelling the receive beamforming and wave propagation into tissue as a neural network, a specific speed of sound value can be employed as a characteristic value to automatically optimize receive delays in the deepformer. The idea is that a given parameterization of the speed of sound for a received set of signals will for example minimize the variance and thus maximize image quality.
While in the embodiment shown in
In an alternative embodiment, the activation of the first subset of transducer elements, and in particular a relative weight and/or a relative delay of their activation is controlled using information regarding the spatial distribution of speed of sound within the object determined by means of receive beamforming, for example by means of a variant of the speed of sound beamformer described above in a variant that allows for providing explicit speed of sound information.
Next, results obtained with end-to-end beamforming of the type shown in
The network was implemented in Python and Tensorflow v1.5. Training was performed using the Adam optimizer (cf. Kingma, Diederik P., and Jimmy Ba. “Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980 (2014)) with an initial learning rate of 0.001 and the mean squared error (MSE) loss function. The loss was computed between the output of the network and a fully processed scan line of the same dimension. Training was completed with early-stopping regularization. To enforce the consistency of the scan line data along the axial dimension, the input to the network was the raw data for a full scan-line, i.e. a 2D structure spanned by the depth and the aperture. To cope with the variance in the raw data caused by attenuation and time gain compensation, both input and output data were normalized depth-wise. This operation was conducted by calculating the means and standard deviations at equal depths of all the processed scan lines in the training set, followed by normalization to zero mean and unit standard deviation.
Two networks of the same architecture were trained on beamformed scan-line data, processed using conventional delay-and-sum beamforming (DS in the following) and conventional minimum variance beamforming (MV), respectively. Experimental inference was performed on a NVIDIA TITAN Xp GPU (Pascal architecture) with an average inference time of 0.9364±0.0055 s per image.
Data acquisition ultrasound imaging was performed using a publicly available beamforming software pipeline (SUPRA) with a cQuest Cicada scanner (Cephasonics, CA, USA) equipped with a 128 elements linear transducer (CPLA12875, 7 MHz). The imaging depth was set to 40 mm, pulse frequency of 7 MHz with a single focus at 20 mm, and the dynamic range was 50 dB. Ground truth B-mode images were systematically generated using the previously collected raw ultrasound data via SUPRA's DS and MV beamformers. The dimension of the resulting data was as follows: raw data, (d×nE×nA)=(2077×256×64); RF signal, (xRF×yRF)=(256×2000); B-mode image, (xB×yB)=(753×800), corresponding to 37.5×40.0 mm2.
In vivo ultrasound images were acquired on 22 healthy participants (age: 32±13 years, 14 males). All scans were performed for both sides (left, right) and both orientations (longitudinal, transverse), for a duration of at least 10 s (yielding a minimum of 19 frames per scan), while a sweep was conducted to introduce variability in the imaging plane. Human acquisitions were performed free-hand, and participant data was randomly assigned to one of three sets:
To investigate the influence of the ultrasound scanner configuration on the deepformer performances, three additional images were acquired on the CIRS phantom using different parameter settings: config 1, focus depth of 30 mm instead of 20 mm; config 2, central frequency of 5 MHz instead of 7 MHz; config 3, dynamic range of 80 instead of 50. A mechanical probe holder was used to fix the transducer in a position.
Evaluation was performed first with a focus on the suitability of deepforming for B-mode image generation. In this view, quantitative results of image similarity between fully beamformed and deepformed B-mode images are indicated for the Structural Similarity Index Measure (SSIM) in
Shown in the below table is a comparison of the deepformed (DF) and the ground truth (GT) images, for Delay and Sum (DS) as well as Minimum Variance (MV), in the validation (v) and testing (t) datasets, using the Structural Similarity Index Measure (SSIM) and the Mean Square Error (MSE).
It can also be observed that while the overall similarity is lower for DF-MV vs GT-MV (MV) and DF-DS vs GT-DS (DS), they still show widely similar behavior across the test set, with overall similarities of 0.497±0.248 and 0.564±0.222 for MV and DS, respectively. This is also confirmed when comparing the results of the two beamforming methods qualitatively to their respective deepforming counterparts, as indicated in
It is important to note here that the validation set represents unseen data only, and that it also includes anatomies which were not even contained in the training set, thus indicating that deepforming effectively achieves a generalization of the beamforming transformation from raw to pre-scan-converted data.
Another interesting aspect of the proposed model can be seen for the results of the LEGO phantom (
While deepforming clearly expresses artifacts unrelated to the side-lobes, the PSFs show the ability to provide a generalized mapping of beamforming methods. The evaluation of different imaging settings, config 0, config 1, and config 2 as described above, for acquisitions on a general purpose ultrasound phantom is illustrated in
The appearance of deepformed images is determined by an objective function minimized during training. The most common objectives that are often selected for reconstruction tasks are the l1 and l2 or Mean Square Error (MSE) loss functions. More advanced reconstruction losses include the Structural Similarity Index (SSIM) and the Multi-Scale Structural Similarity Index (MS-SSIM) as described in Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4): 600-612, 2004. MS-SSIM is a multi-scale version of the SSIM, calculated in a pyramid of M levels, and the corresponding loss function can be formulated as:
Hybrids of the aforementioned losses are commonly deployed, and a recent example is the composite loss:
1+MS-SSIM=αMS-SSIM+(1−α)
1,
where α was set to 0.84 (H. Zhao, O. Gallo, I. Frosio, and J. Kautz. Loss functions for neural networks for image processing, 2015. arXiv: 1511.08861). Experimentation with existing loss formulations led to the identification of the need for an improved objective function tailored to ultrasound applications. MS-SSIM with its property of preserving the contrast in high-frequency regions was found to be an advantageous ingredient. Even though
1 preserves colors and luminance, it does not produce the same contrast as
MS-SSIM, which is why a loss based on the Peak Signal-to-Noise-Ratio (PSNR) may be preferably employed. PSNR is commonly used to measure the quality of reconstruction of lossy compression codecs, for example in applications of image compression. A corresponding loss function may be defined as:
where PSNRmax is the maximum possible pixel value of the image. Since it has been observed that PSNR does not always correlate well with humans' perception of image quality, in preferred embodiments a higher weight is assigned to the MS-SSIM during training, and a Total Variation (TV) regularization term is incorporated to increase homogeneity in the reconstructions. TV simultaneously preserves the edges and smoothens the noise in flat image regions, even at low signal-to-noise ratios, as was shown by D. Strong and T. Chan. Edge-preserving and scale-dependent properties of total variation regularization. Inverse Problems, 19(6):S165, 2003). For a 2D image y it can be computed as:
The final proposed objective function for preferred embodiments of the deepformer is:
DF=αMS-SSIM+(1−α)
PSNR+
TV
where an α of 0.75 was selected as a preferable value after extensive experiments.
Besides beamforming, the inventors have noticed that anatomies can be classified effectively from raw ultrasound signals. Classifying anatomies has many important applications in medical imaging tasks, such as segmentation of images or assisting robotic navigation. While it would in principle be possible to subject the constructed visual image obtained by the beamformer to some classification algorithm, the present inventors have noticed that better results can be achieved if the anatomy classification is carried out based on so-called latent data in one of the aforementioned bottleneck layers, which are also referred to as the “latent space”. The latent space contains a compressed representation of the image. The bottleneck or latent space is much more compact in size (64×2077) than the input raw signals and the ground truth (2077×256×64).
In some practical implementations, the classification was carried out using the ResNet18 algorithm as known from K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, IEEE CVPR, 2016, which was trained with cross-entropy loss. More precisely, the algorithm was trained using Stochastic Gradient Descent with learning rate initialized to 0.001 and momentum of 0.9. Simply put, cross-entropy indicates the distance between what the model believes the output distribution should be and what the original distribution really is. Gradient descent is best used when the parameters cannot be calculated analytically (e.g. using linear algebra) and must be searched for by an optimization algorithm. A learning rate parameter (alpha) must be specified that controls how much the coefficients can change on each update. However, Gradient descent can be slow to run on very large datasets. In situations of large amounts of data, one can use a variation of gradient descent called stochastic gradient descent. In this variation, the gradient descent procedure described above is run but the update to the coefficients is performed for each training instance, rather than at the end of the batch of instances. A momentum term helps accelerate the optimization process and prevents the algorithm from getting stuck in a local, suboptimal solutions.
The anatomy classification was carried out for various target anatomies, which included common carotid artery, thyroid gland, bicep muscle fibers and forearm, with scans collected from both sides (left, right), thereby providing a total of eight anatomy classes. For comparison purposes, the anatomy classification was not only carried out based on the bottleneck feature, but also based on the Minimum Variance B-mode image and the full deepformer reconstruction. The performance is summarized in the below table:
As is seen therein, the classification from the bottleneck feature achieves the highest accuracy and F−1 Score, with a significant margin of 0.07. This demonstrates that the encoding learned in the low-dimensional bottleneck or latent space of the deepformer can capture relevant information from the raw data better than if the learning was based on the reconstructed images, and thus can be used for various auxiliary tasks. The B-mode images reconstructed by the deepformer achieve a 0.04 accuracy improvement over the minimum variance ground truth images. This is expected, since the speckle noise in the ground truth induces ambiguity to the network and high frequencies that can “distract” the network's attention from the features that are crucial for the identification of each anatomy.
In some embodiments, the deepformer can acquire and increase robustness against variation in the sensitivity or even failure of individual transducer elements (such as individual piezo elements) by means of suitable training. In these embodiments, the raw data is “augmented” by artificially “deactivating” or “attenuating” individual transducer elements in the raw data, by simply omitting or suppressing corresponding channels in the raw data by appropriately scaling the signals or their spectra. This augmentation can be implemented as a random or pseudorandom process during the training phase. However, the B-mode image used for training is an image corresponding to fully operative transducer elements. This way, the deepformer learns to establish correct B-mode images (as would be obtained with fully functional transducer elements) from raw data, where individual transducer elements may have failed or may be attenuated. The inventors have noticed that there is sufficient “overlap” in the data associated with individual transducers such that the deepformer—after being trained in the above-mentioned manner—can reconstruct images in which the failure of the individual transducer elements is corrected for. Such pre-training models with activating a deactivating certain elements can in turn be used to create more tailored or even optimal transmission sequences.
In preferred embodiments, one or both of the receive beamforming model and transmit beamforming model is trained in a Generative Adversarial Network (GAN) scenario. For general details of GANs, reference is made to Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27. 2014. In particular, the GAN approach allows for end-to-end denoising of US images directly from the raw signal received by the transducer. For this task the inventors devised a Generative Adversarial Network (GAN) with tailored architectural choices and objective functions that can successfully reconstruct de-speckled B-mode US images, as will be described next.
Conditional GANs
GANs consist of two different networks, a Generator Gθ
θG min θD max EY˜p(Y)[log Dθ
Y is the target image and z is random generated noise. For a cGAN, instead of using z as input to the Generator, in the present embodiment it is conditioned to start with labeled data x→Gθ
Adversarial Network: Generator
The generator employed in this embodiment is depicted in
Adversarial Network: Discriminator
Since in this embodiment the images are of size (256, 1600), the discriminator is constructed to have a receptive field of 255 pixels and make its decision based on the whole width of the image. A Fully Convolutional Neural Network with 7 layers is used, where each layer has a 3×3 convolution of stride 2 followed by Batch Normalization and LeakyReLu (α=0.2). Then for the classification layer, two fully connected layers of size 256 and 1 are used, and the output is followed by a sigmoid activation. The discriminator is likewise depicted in
Content Loss
For the challenging task of end-to-end US denoising combinations of loss functions were deployed that are able to reconstruct B-mode images with all the significant structures, without maintaining the speckle noise. Pixel-wise Mean Squared Error (MSE) loss is defined as:
Herein, x is the raw US data and Y is the target image with size (W, H). Optimization on only MSE loss converges into an averaged image. While this is good for de-noising, is also desired to avoid blurring out important textures.
As shown by Ding Liu, Bihan Wen, Jianbo Jiao, Xianming Liu, Zhangyang Wang, and Thomas S. Huang. Connecting image denoising and high-level vision tasks via deep learning. CoRR.), using pre-trained high-level vision task networks is beneficial for denoising, since it contributes towards maintaining the high-frequency textures. This property is crucial to the preservation of high quality features, significant for applications like automatic segmentation. The VGG19 is utilized to extract two feature layers and minimize the VGG Loss as follows:
ϕ corresponds to the feature map obtained by the chosen VGG19 layers (2nd and 5th) before the max-pooling step, with size (Hϕ; Wϕ;).
Adversarial Loss
In addition to maintaining high quality content, an adversarial loss was deployed to keep the reconstructed images as close as possible to realistic US images. The generative adversarial loss is defined as:
where it is tried to maximize the probability of the discriminator to classify the generated image Gθ
L=L
Adv
+γL
MSE
+βL
VGG (5)
(see Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CaRR.)
Although preferred exemplary embodiments are shown and specified in detail in the drawings and the preceding specification, these should be viewed as purely exemplary and not as limiting the invention. It is noted in this regard that only the preferred exemplary embodiments are shown and specified, and all variations and modifications should be protected that presently or in the future lie within the scope of protection of the invention as defined in the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 18177593.3 | Jun 2018 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2019/065552 | 6/13/2019 | WO | 00 |
