Patent Grant
9871960
The following description relates generally to slide scanners, and in particular to desktop slide scanners for network-based pathology.
In order to diagnose a disease it is often necessary to examine tissue samples at high magnification. By locating and identifying anomalous features in a tissue sample, a pathologist can make a diagnosis, help the patient's physician select appropriate treatment and provide information on the efficacy of previous treatments. Pathologists are therefore critical to the diagnosis and treatment of many diseases.
In general, pathologists often work at locations geographically distant from the hospital or clinic at which a tissue sample is taken. In the past it was necessary to physically transport a tissue sample from the location of the patient to the pathologist, for example by express mail or courier. A pathologist would then prepare a slide specimen from the tissue sample and examine it under a microscope. However, physically transporting the tissue sample to the pathology laboratory may be time consuming, particularly if the patient is in a rural or remote area. Furthermore, if the tissue sample crosses a border, it must be inspected by customs officials. Finally, in many areas such as third world countries there simply are not many pathologists, thereby making it necessary for pathologists to spend an inordinate amount of time travelling to different facilities. For patients who require immediate diagnosis, this is a serious drawback.
The advent of digital pathology helped to alleviate this problem. In digital pathology, a high resolution digital scan of a slide is taken and this image is electronically transmitted to the pathologist for analysis. A physician or technician can prepare slides from tissue samples and create high resolution scans for off-site analysis by the pathologist. Furthermore, high volume slide scanners may scan dozens of slides per scanning operation. Thus, dozens of different slide specimens from one or more nearby medical facilities may be sent to a single location with a high volume slide scanner where digital images or “virtual slides” are created. These virtual slides are then electronically transmitted to appropriate pathologists over a computer network such as the internet.
Thus, digital pathology and high volume slide scanners have helped streamline pathological analysis by creating a hub to which all physical slides in a region may be sent. The high volume scanner at the hub is then used to electronically distribute virtual slides to pathologists anywhere in the world almost instantly. In other words, it is no longer necessary to send individual slides to pathologists in a number of specialized fields. Instead, all slides may be sent to the location of the high volume slide scanner, which is typically relatively near the medical facility where the tissue sample was taken compared to the location of the appropriate pathologist
However, although digital pathology with high volume slide scanners is an improvement over older pathology methods, it is not without drawbacks. First, existing high volume slide scanners are very large and expensive, often costing several hundred thousand dollars. This cost may be prohibitive, particularly in less wealthy countries and/or rural areas. Additionally, high volume scanners are generally loaded with slides only once a day. If an anomaly is found in a particular slide, it cannot be immediately removed or rescanned at a higher resolution for more detailed analysis. Further, high volume scanners are typically allotted to physicians only as workload allows so a physician may have to wait one or more days before it is possible to scan a new slide.
Finally, another drawback to conventional digital pathology is that very high demands are placed on the computer acquiring the image from the scanner. The acquired images are typically several gigabytes in size, and thus a powerful computer is required to quickly process, manipulate and analyze the images. These computers are generally very expensive, making the combination of a high volume slide scanner and acquisition computer cost prohibitive for many facilities.
In summary, high volume scanners are helpful for streamlining digital pathology and handling a large number of slides at once, but there remains a need for a smaller, more affordable, more flexible and more responsive digital pathology system.
The embodiments of a desktop slide scanner for cloud-based pathology disclosed below satisfy these and other needs. The following simplified summary is provided in order to provide a basic understanding of some aspects of the claimed subject matter. This summary is not an extensive overview, and is not intended to identify key/critical elements or to delineate the scope of the claimed subject matter. Its purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
In one aspect of the disclosed embodiments, a method for processing, saving and viewing a digital image of a microscope slide includes providing an acquisition computer connected to a network server in communication with a computer network. At least one microscope slide is inserted into a digital slide scanner connected to the acquisition computer. The acquisition computer generates a pre-scan of the microscope slide at a pre-scan magnification and a pre-scan resolution. The pre-scan is formed from a plurality of sequentially scanned image tiles acquired by the acquisition computer. The acquisition computer uploads each of the plurality of image tiles to the network server as each image tile is scanned. The network server analyzes the image tiles while the pre-scan is being generated to identify an area of interest in the pre-scan. The acquisition computer generates a local scan of the area of interest at a second magnification higher than the pre-scan magnification. The local scan is formed from a plurality of local image tiles acquired by the acquisition computer. The acquisition computer uploads each of the plurality of local image tiles to the network server while the local scan is being generated. Each local image tile is viewable by a client computer in communication with the computer network while the plurality of local image tiles is being uploaded. The network server assembles a raw final image of the local scan from a mosaic of the plurality of local image tiles. The raw final image is then saved on the network server independent of whether the raw final image is saved on the acquisition computer.
In some embodiments, the above method may also include sharpening each local image tile to create a plurality of sharpened local image tiles while the plurality of local image tiles is being uploaded. The network server assembles a sharpened final image from the plurality of sharpened local image tiles while the plurality of local image tiles is being uploaded. The client computer has the ability to select between the raw final image and the sharpened final image for immediate viewing without saving the raw final image and/or the sharpened final image locally and without transferring the entire raw final image and/or the entire sharpened final image after the acquisition computer generates the local scan.
In other embodiments, a method for remotely analyzing a digital image of a microscope slide includes providing an acquisition computer connected to a network server in communication with a computer network. At least one microscope slide is inserted into a digital slide scanner connected to the acquisition computer. The digital slide scanner includes a microscope and a microscope stage, and a pre-scan of a microscope slide on the microscope stage is generated at a first magnification. A client computer remote from the acquisition computer and connected to the computer network is also provided. A pathologist remotely views the pre-scan on the client computer to identify areas of interest in the microscope slide. The digital slide scanner generates a live stream of the areas of interest of the pre-scan at a second magnification greater than the first magnification. The pathologist then remotely instructs the digital slide scanner to move the microscope stage to analyze different portions of the areas of interest in real time.
The pathologist may use the client computer to remotely focus the microscope on regions of varying depth in the areas of interest in real time. Additionally, the desktop slide scanner may also include an inker that marks the microscope slide in the areas of interest identified by the pathologist.
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings. These aspects are indicative, however, of but a few of the various ways in which the principles of the claimed subject matter may be employed and the claimed subject matter is intended to include all such aspects and their equivalents. Other advantages and novel features may become apparent from the following detailed description when considered in conjunction with the drawings.
In one aspect of the disclosed embodiments, a method for processing, saving and viewing a digital image of a microscope slide includes inserting a microscope slide into a digital slide scanner connected to an acquisition computer. A pre-scan formed from a plurality of image tiles uploaded to a network server while the pre-scan is being generated. The network server analyzes the image tiles in real-time to identify an area of interest. The acquisition computer generates a high magnification local scan of the area of interest. The local scan is formed from a plurality of local image tiles that are uploaded to the network server while the local scan is being generated. Each local image tile is viewable by a client computer in communication with the computer network while the plurality of local image tiles is being uploaded. A raw final image is then saved on the network server independent of whether it is saved on the acquisition computer.
The presently disclosed embodiments facilitate fast and accurate pathological analysis of tissue samples taken from a patient. A physician and/or technician obtains a tissue sample and prepares a microscope slide from the tissue sample in a conventional manner. The slide is then brought to a facility with a digital slide scanner as described below.
The digital slide scanner of the presently disclosed embodiments is much smaller than conventional high volume slide scanners, and ideally is a desktop slide scanner. Unlike high volume slide scanners, the desktop slide scanner only scans a handful of slides at a time. For example, the desktop slide scanner may include a cartridge capable of accommodating between one and ten slides depending on slide size, although cartridges capable of holding more than ten slides are also contemplated. Once the prepared slides are placed in the cartridge, the cartridge is inserted into the desktop slide scanner which can then be activated to prepare a pre-scan of the slides.
The desktop slide scanner is connected to a local acquisition computer which is connected to a network server in communication with a computer network. Throughout this disclosure, it is to be understood that any connection or communication between any two computers or devices may be physical or wireless. Once the cartridge is inserted into the desktop slide scanner, a user may instruct the desktop slide scanner to prepare a pre-scan of a slide in the cartridge. The pre-scan is at a relatively low magnification to ensure that the entire tissue sample in the slide falls within the borders of the scan area. The pre-scan is formed from a mosaic of image tiles which are sequentially uploaded to the network server as each image tile is scanned. The network server stitches together each image tile as it is received until the entire pre-scan image is formed.
It should be noted that by uploading the image tiles forming the pre-scan to the network server in real-time, it is possible to generate a complete pre-scan image without ever saving the pre-scan image locally on the acquisition computer. In other words, the acquisition computer may act as an intermediary for acquiring the image data and uploading it to the network server without ever storing, processing or analyzing the image data. The network server, on the other hand, handles all of the computationally intensive operations on the image data. This network-based structure enables conservation of computational resources by centralizing the most computationally intensive operations on the network server, thereby enabling the acquisition computer and client computer to be less powerful than would be necessary if they were required to locally store and process the image data.
As the network server receives each of the image tiles and stitches them together to form the pre-scan image, it is analyzing the image tiles in real time to automatically identify a local area of interest one the slide. By identifying areas of high contrast in the pre-scan image the network server determines the edges of the tissue sample on the slide. The network server may also be programmed to automatically identify features or anomalies within the tissue sample. Alternatively, a user of a client computer connected to the network server via a computer network may manually identify a local area of interest on the slide. The user may choose either the automatically or manually identified local area of interest for further analysis at higher magnification. Additionally, the client computer may include software for automatically identifying anomalies or other areas of interest in the tissue sample.
Once the local area of interest is identified either automatically or manually, the network server sends instructions to the acquisition computer to commence scanning the local area of interest at a higher magnification. The desktop slide scanner scans the local area of interest by sequentially scanning a mosaic of local image tiles. The acquisition computer uploads each local image tile to the network server as each local image tile is scanned. The network server stitches each local image tile together as the image tiles are being uploaded. Once all local image tiles have been uploaded and stitched together, a raw final image of the local area of interest is produced.
In some embodiments, because the local image tiles are uploaded to the network server in real-time as they are scanned, a user of a client computer connected to the network server via a computer network is able to view the local image tiles in real-time. In other words, a user of the client computer, for example a pathologist, can view the raw final image as it is stitched together piece by piece even though the local image tiles are not saved on the client computer. This enables the user to more quickly identify anomalies in the tissue sample because it is not necessary to wait until the entire raw final image is scanned before the individual local image tiles can be viewed. However, in other embodiments the local image tiles are not viewable by a user of the client computer until all local image tiles are uploaded to the network server and mapped with coordinates. In such embodiments, the lag time between beginning the upload of image tiles to the network server and viewing/analyzing the final image on the client computer may be approximately 3-5 minutes depending on image size and bandwidth.
The network server may optionally automatically sharpen each of the local image tiles as they are uploaded to form a sharpened final image. The user of the client computer is then able to view both the raw final image and the sharpened final image even though neither final image is stored locally on the client computer. By providing images that are automatically sharpened using known image processing methods, identification of anomalies in tissue samples may be facilitated in some cases.
Additionally, the client computer may be equipped with software for analyzing and annotating the images stored on the network. For example, the software may enable a user to annotate features of the image for more detailed analysis. The user may move an onscreen pointer to different areas of interest in an image of a sample and digitally mark those areas and optionally enter notes detailing why the area should be analyzed in more depth. In this way the client software allows the user to generate a list of areas of interest that can be accessed by the acquisition computer over the computer network. In some embodiments, these areas of interest may also be physically marked on the slide in the desktop scanner using a built-in ink ejector that places a small drop of ink on each identified area of interest.
When directed to do so by the user of the client computer, the acquisition computer can then direct the desktop slide scanner to automatically scan each area of interest identified in the list at a higher magnification so that the user can then analyze those areas in greater detail. By only scanning the areas of interest at a higher magnification and not scanning the entire slide at higher magnification, computational resources are conserved and the total scan time is greatly shortened. Of course, it is also possible to scan the entire slide at the higher magnification if the user of the client computer wishes to analyze the entire sample in greater detail.
A sample architecture for a network-based pathology system with desktop slide scanner will now be described with reference to
Network server 24 may include software for analyzing, editing and modifying data received from acquisition computer 22. For example, network server 24 may include software for analyzing image data to identify areas of interest in an image and software for sharpening an image, improving image contrast or otherwise modifying an image to facilitate image analysis by a user of client computer 26. Computer network 30 may include storage devices for storing large data files uploaded to computer network 30 from network server 24. Computer network 30 may also include additional computers for processing data uploaded by network server 24 to computer network 30 in order to distribute the computational power required for data analysis and processing.
Pathology system 10 also includes slide cartridge 40 which holds one or more sample slides 42. Each sample slide 42 includes a tissue sample 44 which is placed on each slide 42 by a technician, physician or pathologist. Slide cartridge 40 is inserted into desktop slide scanner 20 which includes a microscope and digital imaging device for magnifying and digitizing images of tissue samples 44 on sample slides 42.
A first method of using pathology system 10 will now be described with reference to
The acquisition computer then directs the desktop slide scanner to scan identified areas of interest in the tissue sample at a second magnification higher than the first pre-scan magnification (126). Each local area of interest is divided into a grid so that the local scan of the area of interest comprises a plurality of local image tiles that are generated and uploaded to the network server sequentially (130). The network server stitches the local image tiles together as they are uploaded (134). A user of the client computer is able to view the local image tiles in real time as the raw final image of the local area of interest is being stitched together and uploaded by the network server (138). A raw final image comprising a mosaic of the assembled local image tiles is generated and saved on the network server independent of whether the raw final image is saved on the acquisition computer (142).
Another method of using pathology system 10 will now be described with reference to
It should be noted that the phrase “in real time” as used above means that instructions from the client computer to the acquisition computer or desktop slide scanner are carried out immediately as they are made by the client computer and received by the remote device. The phrase “in real time” also means that image data uploaded to the network server by the acquisition computer is viewable on the client computer as soon as it is received. For example, the user of the client computer may instruct the microscope stage to move to different areas and the corresponding changing images are immediately viewable on the client computer. In other words, a series of different still images are not downloaded by the client computer; rather, the client computer receives a stream of image data that instantly changes as the microscope stage is moved in response to instructions sent by the user of the client computer.
What has been described above includes examples of one or more embodiments. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the aforementioned embodiments, but one of ordinary skill in the art may recognize that many further combinations and permutations of various embodiments are possible. Accordingly, the described embodiments are intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.
This application is a continuation of Ser. No. 14/234,013, filed May. 19, 2014, which is a U.S. National Phase of PCT/US2012/047527, filed Jul. 20, 2012, which claims the benefit of U.S. Provisional Application No. 61/509,946, filed Jul. 20, 2011, all of which are incorporated herein by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4700298 | Palcic et al. | Oct 1987 | A |
| 5018029 | Ekhoff et al. | May 1991 | A |
| 5216500 | Krummey et al. | Jun 1993 | A |
| 5216596 | Weinstein | Jun 1993 | A |
| 5297034 | Weinstein | Mar 1994 | A |
| 6101265 | Bacus et al. | Aug 2000 | A |
| 6208374 | Clinch | Mar 2001 | B1 |
| 6272235 | Bacus et al. | Aug 2001 | B1 |
| 6452625 | Kapitza | Sep 2002 | B1 |
| 6606413 | Zeineh | Aug 2003 | B1 |
| 6674881 | Bacus et al. | Jan 2004 | B2 |
| 6711283 | Soenksen | Mar 2004 | B1 |
| 6905300 | Russum | Jun 2005 | B1 |
| 7028075 | Morris | Apr 2006 | B2 |
| 7035478 | Crandall et al. | Apr 2006 | B2 |
| 7116437 | Weinstein et al. | Oct 2006 | B2 |
| 7116440 | Eichhorn et al. | Oct 2006 | B2 |
| 7149332 | Bacus et al. | Dec 2006 | B2 |
| 7215467 | Nakagawa | May 2007 | B2 |
| 7224839 | Zeineh | May 2007 | B2 |
| 7257268 | Eichhorn et al. | Aug 2007 | B2 |
| 7292251 | Gu et al. | Nov 2007 | B1 |
| 7319540 | Tipirneni | Jan 2008 | B2 |
| 7391894 | Zeineh | Jun 2008 | B2 |
| 7432486 | Tanemura et al. | Oct 2008 | B2 |
| 7502519 | Eichhorn et al. | Mar 2009 | B2 |
| 7518652 | Olson et al. | Apr 2009 | B2 |
| 7542596 | Bacus et al. | Jun 2009 | B2 |
| 7602524 | Eichhorn et al. | Oct 2009 | B2 |
| 7646495 | Olson et al. | Jan 2010 | B2 |
| 7668362 | Olson et al. | Feb 2010 | B2 |
| 7738688 | Eichhorn et al. | Jun 2010 | B2 |
| 7755841 | Christenson et al. | Jul 2010 | B2 |
| 7801352 | Uchiyama et al. | Sep 2010 | B2 |
| 7826649 | Crandall et al. | Nov 2010 | B2 |
| 7844125 | Eichhorn et al. | Nov 2010 | B2 |
| 7856131 | Bacus et al. | Dec 2010 | B2 |
| 7893988 | Olson et al. | Feb 2011 | B2 |
| 7916916 | Zeineh | Mar 2011 | B2 |
| 7941275 | Gholap et al. | May 2011 | B2 |
| 7949168 | Crandall et al. | May 2011 | B2 |
| 7979212 | Gholap et al. | Jul 2011 | B2 |
| 8010555 | Eichhorn | Aug 2011 | B2 |
| 8023714 | Soenksen | Sep 2011 | B2 |
| 8086077 | Eichhorn | Dec 2011 | B2 |
| 8094902 | Crandall et al. | Jan 2012 | B2 |
| 8103082 | Olson et al. | Jan 2012 | B2 |
| 8189897 | Leidenbach | May 2012 | B2 |
| 8325150 | Reeves et al. | Dec 2012 | B1 |
| 8456522 | Olson et al. | Jun 2013 | B2 |
| 8515683 | Gholap et al. | Aug 2013 | B2 |
| 8781261 | Eichhorn | Jul 2014 | B2 |
| 8805791 | Eichhorn | Aug 2014 | B2 |
| 8996570 | Stratman et al. | Mar 2015 | B2 |
| 20020061127 | Bacus et al. | May 2002 | A1 |
| 20030123717 | Bacus | Jul 2003 | A1 |
| 20040136582 | Bacus | Jul 2004 | A1 |
| 20060104499 | Zahniser et al. | May 2006 | A1 |
| 20060159367 | Zeineh | Jul 2006 | A1 |
| 20070103739 | Anderson, Jr. et al. | May 2007 | A1 |
| 20090028414 | Crandall et al. | Jan 2009 | A1 |
| 20090238435 | Shields | Sep 2009 | A1 |
| 20100103253 | Sieckmann et al. | Apr 2010 | A1 |
| 20100194681 | Halushka | Aug 2010 | A1 |
| 20100315502 | Tafas | Dec 2010 | A1 |
| 20110217238 | Borrebaeck et al. | Sep 2011 | A1 |
| 20110311123 | Gholap et al. | Dec 2011 | A1 |
| 20120011151 | Eichhorn | Jan 2012 | A1 |
| 20120072452 | Stratman et al. | Mar 2012 | A1 |
| 20120099769 | Eichhorn | Apr 2012 | A1 |
| 20130182922 | Kil | Jul 2013 | A1 |
| 20140193839 | Cunningham | Jul 2014 | A1 |
| 20140333959 | Casas | Nov 2014 | A1 |
| 20170078555 | Casas | Mar 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| WO-9930264 | Jun 1999 | WO |
| WO-2008028944 | Mar 2008 | WO |
| WO-2013013117 | Jan 2013 | WO |
| WO-2016069794 | May 2016 | WO |
| Entry |
|---|
| U.S. Appl. No. 14/853,763 Office Action dated May 16, 2016. |
| Parvin et al. DeepView: A Channel for Distributed Microscopy and Informatics. Supercomputing, ACM/IEEE 1999 Conference, Nov. 13-18, 1999 (15 pgs.). |
| PCT/US2015/057890 International Search Report and Written Opinion dated Feb. 5, 2016. |
| U.S. Appl. No. 14/853,763 Office Action dated Dec. 3, 2015. |
| Battmann et al. Telemedicine: Application Telephathology-Remote miscroscopy for intraoperative diagnoses on frozen sections. Telemedicine pp. 1127-1130 (2000). |
| Co-pending U.S. Appl. No. 14/853,763, filed Sep. 14, 2015. |
| Co-pending U.S. Appl. No. 14/925,905, filed Oct. 28, 2015. |
| PCT/US2012/047527 International Preliminary Report on Patentability dated Jan. 21, 2014. |
| PCT/US2012/047527 International Search Report dated Oct. 1, 2012. |
| U.S. Appl. No. 14/234,013 office Action dated Jan. 30, 2015. |
| U.S. Appl. No. 15/351,889 Office Action dated Mar. 24, 2017. |
| Number | Date | Country | |
|---|---|---|---|
| 20150379328 A1 | Dec 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61509946 | Jul 2011 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14234013 | US | |
| Child | 14848235 | US |
