Patent Application
20230018469
The technology disclosed relates to apparatus and corresponding methods for the automated analysis of an image or recognition of a pattern. Included herein are systems that transform an image for the purpose of (a) enhancing its visual quality prior to recognition, (b) locating and registering the image relative to a sensor or stored prototype, or reducing the amount of image data by discarding irrelevant data, and (c) measuring significant characteristics of the image. In particular, the technology disclosed relates to removing spatial crosstalk from sensor pixels using equalization-based image processing techniques.
The following are incorporated by reference for all purposes as if fully set forth herein:
U.S. Nonprovisional patent application Ser. No. 17/308,035, titled “EQUALIZATION-BASED IMAGE PROCESSING AND SPATIAL CROSSTALK ATTENUATOR,” filed May 4, 2021 (Attorney Docket No. ILLM 1032-2/IP-1991-US);
U.S. Provisional Patent Application No. 63/106,256, titled “SYSTEMS AND METHODS FOR PER-CLUSTER INTENSITY CORRECTION AND BASE CALLING,” filed on Oct. 27, 2020;
U.S. Nonprovisional patent application Ser. No. 15/909,437, titled “OPTICAL DISTORTION CORRECTION FOR IMAGED SAMPLES,” filed on Mar. 1, 2018;
U.S. Nonprovisional patent application Ser. No. 14/530,299, titled “IMAGE ANALYSIS USEFUL FOR PATTERNED OBJECTS,” filed on Oct. 31, 2014;
U.S. Nonprovisional patent application Ser. No. 15/153,953, titled “METHODS AND SYSTEMS FOR ANALYZING IMAGE DATA,” filed on Dec. 3, 2014;
U.S. Nonprovisional patent application Ser. No. 15/863,241, titled “PHASING CORRECTION,” filed on Jan. 5, 2018;
U.S. Nonprovisional patent application Ser. No. 14/020,570, titled “CENTROID MARKERS FOR IMAGE ANALYSIS OF HIGH DENSITY CLUSTERS IN COMPLEX POLYNUCLEOTIDE SEQUENCING,” filed on Sep. 6, 2013;
U.S. Nonprovisional patent application Ser. No. 12/565,341, titled “METHOD AND SYSTEM FOR DETERMINING THE ACCURACY OF DNA BASE IDENTIFICATIONS,” filed on Sep. 23, 2009;
U.S. Nonprovisional patent application Ser. No. 12/295,337, titled “SYSTEMS AND DEVICES FOR SEQUENCE BY SYNTHESIS ANALYSIS,” filed on Mar. 30, 2007;
U.S. Nonprovisional patent application Ser. No. 12/020,739, titled “IMAGE DATA EFFICIENT GENETIC SEQUENCING METHOD AND SYSTEM,” filed on Jan. 28, 2008;
U.S. Nonprovisional patent application Ser. No. 13/833,619, titled “BIOSENSORS FOR BIOLOGICAL OR CHEMICAL ANALYSIS AND SYSTEMS AND METHODS FOR SAME,” filed on Mar. 15, 2013, (Attorney Docket No. IP-0626-US);
U.S. Nonprovisional patent application Ser. No. 15/175,489, titled “BIOSENSORS FOR BIOLOGICAL OR CHEMICAL ANALYSIS AND METHODS OF MANUFACTURING THE SAME,” filed on Jun. 7, 2016, (Attorney Docket No. IP-0689-US);
U.S. Nonprovisional patent application Ser. No. 13/882,088, titled “MICRODEVICES AND BIOSENSOR CARTRIDGES FOR BIOLOGICAL OR CHEMICAL ANALYSIS AND SYSTEMS AND METHODS FOR THE SAME,” filed on Apr. 26, 2013, (Attorney Docket No. IP-0462-US);
U.S. Nonprovisional patent application Ser. No. 13/624,200, titled “METHODS AND COMPOSITIONS FOR NUCLEIC ACID SEQUENCING,” filed on Sep. 21, 2012, (Attorney Docket No. IP-0538-US);
U.S. Nonprovisional patent application Ser. No. 13/006,206, titled “DATA PROCESSING SYSTEM AND METHODS,” filed on Jan. 13, 2011;
U.S. Nonprovisional patent application Ser. No. 15/936,365, titled “DETECTION APPARATUS HAVING A MICROFLUOROMETER, A FLUIDIC SYSTEM, AND A FLOW CELL LATCH CLAMP MODULE,” filed on Mar. 26, 2018;
U.S. Nonprovisional patent application Ser. No. 16/567,224, titled “FLOW CELLS AND METHODS RELATED TO SAME,” filed on Sep. 11, 2019;
U.S. Nonprovisional patent application Ser. No. 16/439,635, titled “DEVICE FOR LUMINESCENT IMAGING,” filed on Jun. 12, 2019;
U.S. Nonprovisional patent application Ser. No. 15/594,413, titled “INTEGRATED OPTOELECTRONIC READ HEAD AND FLUIDIC CARTRIDGE USEFUL FOR NUCLEIC ACID SEQUENCING,” filed on May 12, 2017;
U.S. Nonprovisional patent application Ser. No. 16/351,193, titled “ILLUMINATION FOR FLUORESCENCE IMAGING USING OBJECTIVE LENS,” filed on Mar. 12, 2019;
U.S. Nonprovisional patent application Ser. No. 12/638,770, titled “DYNAMIC AUTOFOCUS METHOD AND SYSTEM FOR ASSAY IMAGER,” filed on Dec. 15, 2009;
U.S. Nonprovisional patent application Ser. No. 13/783,043, titled “KINETIC EXCLUSION AMPLIFICATION OF NUCLEIC ACID LIBRARIES,” filed on Mar. 1, 2013; and
U.S. Nonprovisional patent application Ser. No. 16/826,168, titled “ARTIFICIAL INTELLIGENCE-BASED SEQUENCING,” filed 21 Mar. 2020 (Attorney Docket No. ILLM 1008-20/IP-1752-PRV).
The subject matter discussed in this section should not be assumed to be prior art merely as a result of its mention in this section. Similarly, a problem mentioned in this section or associated with the subject matter provided as background should not be assumed to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which in and of themselves can also correspond to implementations of the claimed technology.
Base calling accuracy is crucial for high-throughput sequencing and downstream analysis such as read mapping and genome assembly. This disclosure relates to optimizing image data to accurately base call clusters during a sequencing run. One challenge with the optimization of image data is variation in intensity profiles (or intensity distributions) of clusters in a cluster population being base called. This is particularly detrimental for multi-cycle imaging of substrates (e.g., flow cells) having a large number (e.g., thousands, millions, billions, etc.) of clusters, as it makes the scale of variation unmanageable, thereby causing a drop in data throughput and an increase in error rate.
Intensity profiles of millions of clusters on a flow cell can vary between respective clusters or between subpopulations of clusters. There are many potential reasons for this variation. It may result from differences in cluster brightness, caused by fragment length distribution in the cluster population or unwanted light emissions from adjacent clusters (spatial crosstalk). It may result from phase error, which occurs when a molecule in a cluster does not incorporate a nucleotide in some sequencing cycle and lags behind other molecules, or when a molecule incorporates more than one nucleotide in a single sequencing cycle. It may result from fading, i.e., an exponential decay in signal intensity of clusters as a function of sequencing cycle number due to excessive washing and laser exposure as the sequencing run progresses. It may result from underdeveloped cluster colonies, i.e., small cluster sizes that produce empty or partially filled wells on a patterned flow cell. It may result from overlapping cluster colonies caused by unexclusive amplification. It may result from under-illumination or uneven-illumination, for example, due to clusters being located on edges of a flow cell. It may result from impurities (e.g., bubbles) on a flow cell that obfuscate emitted signal. It may result from polyclonal clusters, i.e., when multiple clusters are deposited in the same well. It may result from different types of distortion in the image induced by the geometry of the optical lens. Such distortions may include, for example, magnification distortion, skew distortion, translation distortion, and nonlinear distortions such as barrel distortion and pincushion distortion.
This variation can be corrected in a coarse way by training an intensity corrector for the entire cluster population. Different than this would be to train respective intensity correctors for respective subpopulations of clusters, where the subpopulations are segmented in a way that minimizes sequencing errors and maximizes base calling accuracy within the bounds of available compute. This disclosure relates to the latter more granular approach. More details follow.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
In the drawings, like reference characters generally refer to like parts throughout the different views. Also, the drawings are not necessarily to scale, with an emphasis instead generally being placed upon illustrating the principles of the technology disclosed. In the following description, various implementations of the technology disclosed are described with reference to the following drawings, in which.
The following discussion is presented to enable any person skilled in the art to make and use the technology disclosed and is provided in the context of a particular application and its requirements. Various modifications to the disclosed implementations will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations and applications without departing from the spirit and scope of the technology disclosed. Thus, the technology disclosed is not intended to be limited to the implementations shown but is to be accorded the widest scope consistent with the principles and features disclosed herein.
We first describe signal profilers and then the disclosed specialist signal profilers.
As used herein, a “signal profiler” maximizes the signal-to-noise ratio of a signal that is disturbed by noise. A signal profiler can be a value or function that is applied to data to modify the data in a desired way. For example, the data can be modified to increase its accuracy, relevance, or applicability with regard to a particular situation. The signal profiler can be applied to the data by any of a variety of mathematical manipulations including, but not limited to addition, subtraction, division, multiplication, or a combination thereof. The signal profiler can be a mathematical formula, logic function, computer implemented algorithm, or the like. The data can be image data, electrical data, or a combination thereof
In one implementation, the signal profiler is an equalizer (e.g., a spatial equalizer). The equalizer can be trained (e.g., using least square estimation, adaptive equalization algorithm) to maximize the signal-to-noise ratio of cluster intensity data in sequencing images. In some implementations, the equalizer is a lookup table (LUT) bank with a plurality of LUTs with subpixel resolution, also referred to as “equalizer filters” or “convolution kernels.” In one implementation, the number of LUTs in the equalizer depends on the number of subpixels into which pixels of the sequencing images can be divided. For example, if the pixels are divisible into n by n subpixels (e.g., 5×5 subpixels), then the equalizer generates n2 LUTs (e.g., 25 LUTs).
In one implementation of training the equalizer, data from the sequencing images is binned by well subpixel location. For example, for a 5×5 LUT, approximately 1/25th of the wells have a center that is in bin (1,1) (e.g., the upper left corner of a sensor pixel), 1/25th of the wells are in bin (1,2), and so on. In one implementation, the equalizer coefficients for each bin are determined using least squares estimation on the subset of data from the wells corresponding to the respective bins. This way the resulting estimated equalizer coefficients are different for each bin.
Each LUT/equalizer filter/convolution kernel has a plurality of coefficients that are learned from the training. In one implementation, the number of coefficients in a LUT corresponds to the number of pixels that are used for base calling a cluster. For example, if a local grid of pixels (image or pixel patch) that is used to base call a cluster is of size p×p (e.g., 9×9 pixel patch), then each LUT has p2 coefficients (e.g., 81 coefficients).
In one implementation, the training produces equalizer coefficients that are configured to mix/combine intensity values of pixels that depict intensity emissions from a target cluster being base called and intensity emissions from one or more adjacent clusters in a manner that maximizes the signal-to-noise ratio. The signal maximized in the signal-to-noise ratio is the intensity emissions from the target cluster, and the noise minimized in the signal-to-noise ratio is the intensity emissions from the adjacent clusters, i.e., spatial crosstalk, plus some random noise (e.g., to account for background intensity emissions). The equalizer coefficients are used as weights and the mixing/combining includes executing element-wise multiplication between the equalizer coefficients and the intensity values of the pixels to calculate a weighted sum of the intensity values of the pixels, i.e., a convolution operation. Furthermore, in cases the image data spans across multiple color channels, a set of equalizer coefficients is generated for each color channel (e.g., one channel, three channels, four channels, etc.).
In
Then, a base calling logic 2322 uses the expectation maximization (EM) algorithm discussed above to predict a base call 2324 based on the weighted sums 2316, 2318. In
In
In some implementations, the above steps are executed for a plurality of clusters. For example, in the case of three clusters depicted in
Additional details about the equalizer can be found in U.S. Nonprovisional patent application Ser. No.: 17/308,035, titled “EQUALIZATION-BASED IMAGE PROCESSING AND SPATIAL CROSSTALK ATTENUATOR,” filed May 4, 2021 (Attorney Docket No. ILLM 1032-2/IP-1991-US), which is incorporated by reference as if fully set forth herein.
The discussion now turns to the specialist signal profilers.
A global intensity correction applied at a pan-flow cell level or a pan-sequencing run level fails to consider a variety of noises in the image data. For example, non-linear distortion and noise can be induced by the shape of the optical lens that captures the image data. In addition, the imaged flow cell can also introduce distortion in the well pattern due to the manufacturing process, e.g., a 3D bathtub effect introduced by bonding or movement of the wells due to non-rigidity of the substrate. Finally, the tilt of the flow cell within the holder is not accounted for by the global intensity correction.
As used herein, a “specialist signal profiler” is a signal profiler that is configured to/trained to maximize the signal-to-noise ratio of a particular category/type/configuration/characteristic/class/bin of data. We disclose a variety of specialist signal profilers. For example, a “surface-specific specialist signal profiler” is configured to/trained to maximize the signal-to-noise ratio of sequencing data of clusters located on a particular surface or a particular surface-type/category/class (e.g., top surfaces or bottom surfaces or surfaces 1 to N of a flow cell). Similarly, a “lane-specific specialist signal profiler” is configured to/trained to maximize the signal-to-noise ratio of sequencing data of clusters located on a particular lane or a particular lane-type/category/class (e.g., central lanes or peripheral lanes or lanes 1 to N of a flow cell). Also, a “tile-specific specialist signal profiler” is configured to/trained to maximize the signal-to-noise ratio of sequencing data of clusters located on a particular tile or a particular tile-type/category/class (e.g., central tiles or peripheral tiles or tiles 1 to N of a flow cell). Also, a “sub-tile-specific specialist signal profiler” is configured to/trained to maximize the signal-to-noise ratio of sequencing data of clusters located on a particular sub-tile or a particular sub-tile-type/category/class (e.g., central sub-tiles or peripheral sub-tiles or sub-tiles 1 to N of a flow cell). More examples and details of the disclosed specialist signal profilers follow.
In some implementations, a single signal profiler can comprise a plurality of specialist coefficient sets, such that each specialist coefficient set is configured to/trained to maximize the signal-to-noise ratio of a particular category/type/configuration/characteristic/class/bin of data. In some implementations, the single signal profiler can comprise a variety of specialist coefficient sets. For example, a “surface-specific specialist coefficient set” is configured to/trained to maximize the signal-to-noise ratio of sequencing data of clusters located on a particular surface or a particular surface-type/category/class (e.g., top surfaces or bottom surfaces or surfaces 1 to N of a flow cell). Similarly, a “lane-specific specialist coefficient set” is configured to/trained to maximize the signal-to-noise ratio of sequencing data of clusters located on a particular lane or a particular lane-type/category/class (e.g., central lanes or peripheral lanes or lanes 1 to N of a flow cell). Also, a “tile-specific specialist coefficient set” is configured to/trained to maximize the signal-to-noise ratio of sequencing data of clusters located on a particular tile or a particular tile-type/category/class (e.g., central tiles or peripheral tiles or tiles 1 to N of a flow cell). Also, a “sub-tile-specific specialist coefficient set” is configured to/trained to maximize the signal-to-noise ratio of sequencing data of clusters located on a particular sub-tile or a particular sub-tile-type/category/class (e.g., central sub-tiles or peripheral sub-tiles or sub-tiles 1 to N of a flow cell). More examples and details of the disclosed specialist coefficient sets follow.
The disclosed specialist signal profilers are applicable to clusters located on both patterned and unpatterned surfaces of a flow cell. With unpatterned surfaces, the clusters are randomly distributed on the flow cell. The randomly distributed clusters and data therefor (e.g., images) can be binned spatially, temporally, signal-wise, or by any combination thereof. Accordingly, the specialist signal profilers can be configured and trained for different configurations of the differently binned randomly distributed clusters. With patterned surfaces, the clusters are located on patterned wells with fixed locations. The patterned wells and the constituent clusters can be binned spatially, temporally, signal-wise, or by any combination thereof. Accordingly, the specialist signal profilers can be configured and trained for different configurations of the differently binned patterned clusters.
The disclosed specialist signal profilers are configuration-specific signal profilers that are trained to maximize the signal-to-noise ratio of image data generated for different configurations of a sequencing run. These configurations can be spatial configurations relating to different regions on a flow cell, temporal configurations relating to different sequencing/imaging cycles of the sequencing run, signal distribution configurations relating to different distributions/patterns of signal profiles observed/encoded in the imaged data, or a combination thereof. Before describing various implementations of the systems and methods disclosed herein in greater detail, it is useful to describe an example environment with which the technology disclosed herein can be implemented.
As can be seen in the example of
Fluid delivery module or device 100 directs the flow of reagents (e.g., fluorescently labeled nucleotides, buffers, enzymes, cleavage reagents, etc.) to (and through) sample container 110 and waste valve 120. Sample container 110 can include one or more substrates upon which the samples are provided. For example, in the case of a system to analyze a large number of different nucleic acid sequences, sample container 110 can include one or more substrates on which nucleic acids to be sequenced are bound, attached, or associated. In various implementations, the substrate can include any inert substrate or matrix to which nucleic acids can be attached, such as for example glass surfaces, plastic surfaces, latex, dextran, polystyrene surfaces, polypropylene surfaces, polyacrylamide gels, gold surfaces, and silicon wafers. In some applications, the substrate is within a channel or other area at a plurality of locations formed in a matrix or array across the sample container 110.
In some implementations, the sample container 110 can include a biological sample that is imaged using one or more fluorescent dyes. For example, in a particular implementation, the sample container 110 can be implemented as a patterned flow cell including a translucent cover plate, a substrate, and a liquid sandwiched therebetween, and a biological sample can be located at an inside surface of the translucent cover plate or an inside surface of the substrate. The flow cell can include a large number (e.g., thousands, millions, or billions) of wells or regions that are patterned into a defined array (e.g., a hexagonal array, rectangular array, etc.) into the substrate. Each region can form a cluster (e.g., a monoclonal cluster) of a biological sample such as DNA, RNA, or another genomic material which can be sequenced, for example, using sequencing by synthesis. The flow cell can be further divided into a number of spaced apart lanes (e.g., eight lanes), each lane including a hexagonal array of clusters. Example flow cells that can be used in implementations disclosed herein are described in U.S. Pat. No. 8,778,848.
The system also comprises temperature station actuator 130 and heater/cooler 135 that can optionally regulate the temperature of conditions of the fluids within the sample container 110. Camera system 140 can be included to monitor and track the sequencing of sample container 110. Camera system 140 can be implemented, for example, as a charge-coupled device (CCD) camera (e.g., a time delay integration (TDI) CCD camera), which can interact with various filters within filter switching assembly 145, objective lens 142, and focusing laser/focusing laser assembly 150. Camera system 140 is not limited to a CCD camera and other cameras and image sensor technologies can be used. In particular implementations, the camera sensor can have a pixel size between about 5 and about 15 μm.
Output data from the sensors of camera system 140 can be communicated to a real time analysis module (not shown) that can be implemented as a software application that analyzes the image data (e.g., image quality scoring), reports or displays the characteristics of the laser beam (e.g., focus, shape, intensity, power, brightness, position) to a graphical user interface (GUI), and, as further described below, dynamically corrects intensity noise in the image data.
Light source 160 (e.g., an excitation laser within an assembly optionally comprising multiple lasers) or other light source can be included to illuminate fluorescent sequencing reactions within the samples via illumination through a fiber optic interface (which can optionally comprise one or more re-imaging lenses, a fiber optic mounting, etc.). Low watt lamp 165, focusing laser 150, and reverse dichroic 185 are also presented in the example shown. In some implementations focusing laser 150 can be turned off during imaging. In other implementations, an alternative focus configuration can include a second focusing camera (not shown), which can be a quadrant detector, a Position Sensitive Detector (PSD), or similar detector to measure the location of the scattered beam reflected from the surface concurrent with data collection.
Although illustrated as a backlit device, other examples can include a light from a laser or other light source that is directed through the objective lens 142 onto the samples on sample container 110. Sample container 110 can be ultimately mounted on a sample stage 170 to provide movement and alignment of the sample container 110 relative to the objective lens 142. The sample stage can have one or more actuators to allow it to move in any of three dimensions. For example, in terms of the Cartesian coordinate system, actuators can be provided to allow the stage to move in the X, Y, and Z directions relative to the objective lens. This can allow one or more sample locations on sample container 110 to be positioned in optical alignment with objective lens 142.
A focus (z-axis) component 175 is shown in this example as being included to control positioning of the optical components relative to the sample container 110 in the focus direction (typically referred to as the z axis, or z direction). Focus component 175 can include one or more actuators physically coupled to the optical stage or the sample stage, or both, to move sample container 110 on sample stage 170 relative to the optical components (e.g., the objective lens 142) to provide proper focusing for the imaging operation. For example, the actuator can be physically coupled to the respective stage such as, for example, by mechanical, magnetic, fluidic, or other attachment or contact directly or indirectly to or with the stage. The one or more actuators can be configured to move the stage in the z-direction while maintaining the sample stage in the same plane (e.g., maintaining a level or horizontal attitude, perpendicular to the optical axis). The one or more actuators can also be configured to tilt the stage. This can be done, for example, so that sample container 110 can be leveled dynamically to account for any slope in its surfaces.
Focusing of the system generally refers to aligning the focal plane of the objective lens with the sample to be imaged at the chosen sample location. However, focusing can also refer to adjustments to the system to obtain a desired characteristic for a representation of the sample such as, for example, a desired level of sharpness or contrast for an image of a test sample. Because the usable depth of field of the focal plane of the objective lens can be small (sometimes on the order of 1 μm or less), focus component 175 closely follows the surface being imaged. Because the sample container is not perfectly flat as fixtured in the instrument, focus component 175 can be set up to follow this profile while moving along in the scanning direction (herein referred to as the y-axis).
The light emanating from a test sample at a sample location being imaged can be directed to one or more detectors of camera system 140. An aperture can be included and positioned to allow only light emanating from the focus area to pass to the detector. The aperture can be included to improve image quality by filtering out components of the light that emanate from areas that are outside of the focus area. Emission filters can be included in filter switching assembly 145, which can be selected to record a determined emission wavelength and to cut out any stray laser light.
Although not illustrated, a controller can be provided to control the operation of the scanning system. The controller can be implemented to control aspects of system operation such as, for example, focusing, stage movement, and imaging operations. In various implementations, the controller can be implemented using hardware, algorithms (e.g., machine executable instructions), or a combination of the foregoing. For example, in some implementations the controller can include one or more CPUs or processors with associated memory. As another example, the controller can comprise hardware or other circuitry to control the operation, such as a computer processor and a non-transitory computer readable medium with machine-readable instructions stored thereon. For example, this circuitry can include one or more of the following: field programmable gate array (FPGA), application specific integrated circuit (ASIC), programmable logic device (PLD), complex programmable logic device (CPLD), a programmable logic array (PLA), programmable array logic (PAL) or other similar processing device or circuitry. As yet another example, the controller can comprise a combination of this circuitry with one or more processors.
In some implementations, system 200 can be used for the sequencing of nucleic acids. Applicable techniques include those where nucleic acids are attached at fixed locations in an array (e.g., the wells of a flow cell) and the array is imaged repeatedly. In such implementations, system 200 can obtain images in two different color channels, which can be used to distinguish a particular nucleotide base type from another. More particularly, system 200 can implement a process referred to as “base calling,” which generally refers to a process of a determining a base call (e.g., adenine (A), cytosine (C), guanine (G), or thymine (T)) for a given spot location of an image at an imaging cycle. During two-channel base calling, image data extracted from two images can be used to determine the presence of one of four base types by encoding base identity as a combination of the intensities of the two images. For a given spot or location in each of the two images, base identity can be determined based on whether the combination of signal identities is [on, on], [on, off], [off, on], or [off, off].
Referring again to imaging system 200, the system includes a line generation module (LGM) 210 with two light sources, 211 and 212, disposed therein. Light sources 211 and 212 can be coherent light sources such as laser diodes which output laser beams. Light source 211 can emit light in a first wavelength (e.g., a red color wavelength), and light source 212 can emit light in a second wavelength (e.g., a green color wavelength). The light beams output from laser sources 211 and 212 can be directed through a beam shaping lens or lenses 213. In some implementations, a single light shaping lens can be used to shape the light beams output from both light sources. In other implementations, a separate beam shaping lens can be used for each light beam. In some examples, the beam shaping lens is a Powell lens, such that the light beams are shaped into line patterns. The beam shaping lenses of LGM 210 or other optical components imaging system 200 be configured to shape the light emitted by light sources 211 and 212 into a line patterns (e.g., by using one or more Powel lenses, or other beam shaping lenses, diffractive or scattering components).
LGM 210 can further include mirror 214 and semi-reflective mirror 215 configured to direct the light beams through a single interface port to an emission optics module (EOM) 230. The light beams can pass through a shutter element 216. EOM 230 can include objective 235 and a z-stage 236 which moves objective 235 longitudinally closer to or further away from a target 250. For example, target 250 can include a liquid layer 252 and a translucent cover plate 251, and a biological sample can be located at an inside surface of the translucent cover plate as well an inside surface of the substrate layer located below the liquid layer. The z-stage can then move the objective as to focus the light beams onto either inside surface of the flow cell (e.g., focused on the biological sample). The biological sample can be DNA, RNA, proteins, or other biological materials responsive to optical sequencing as known in the art.
EOM 230 can include semi-reflective mirror 233 to reflect a focus tracking light beam emitted from a focus tracking module (FTM) 240 onto target 250, and then to reflect light returned from target 250 back into FTM 240. FTM 240 can include a focus tracking optical sensor to detect characteristics of the returned focus tracking light beam and generate a feedback signal to optimize focus of objective 235 on target 250.
EOM 230 can also include semi-reflective mirror 234 to direct light through objective 235, while allowing light returned from target 250 to pass through. In some implementations, EOM 230 can include a tube lens 232. Light transmitted through tube lens 232 can pass through filter element 231 and into camera module (CAM) 220. CAM 220 can include one or more optical sensors 221 to detect light emitted from the biological sample in response to the incident light beams (e.g., fluorescence in response to red and green light received from light sources 211 and 212).
Output data from the sensors of CAM 220 can be communicated to a real time analysis module 225. Real time analysis module, in various implementations, executes computer readable instructions for analyzing the image data (e.g., image quality scoring, base calling, etc.), reporting or displaying the characteristics of the beam (e.g., focus, shape, intensity, power, brightness, position) to a graphical user interface (GUI), etc. These operations can be performed in real-time during imaging cycles to minimize downstream analysis time and provide real time feedback and troubleshooting during an imaging run. In implementations, real time analysis module can be a computing device (e.g., computing device 1000) that is communicatively coupled to and controls imaging system 200. In implementations further described below, real time analysis module 225 can additionally execute computer readable instructions for maximizing the signal-to-noise ratio of the output image data received from CAM 220.
Sequencing produces m sequencing images per sequencing cycle for corresponding m image channels. That is, each of the sequencing images has one or more image (or intensity) channels (analogous to the red, green, blue (RGB) channels of a color image). In one implementation, each image channel corresponds to one of a plurality of filter wavelength bands. In another implementation, each image channel corresponds to one of a plurality of imaging events at a sequencing cycle. In yet another implementation, each image channel corresponds to a combination of illumination with a specific laser and imaging through a specific optical filter. The image patches are tiled (or accessed) from each of the m image channels for a particular sequencing cycle. In different implementations such as 4-, 2-, and 1-channel chemistries, m is 4 or 2. In other implementations, m is 1, 3, or greater than 4. In other implementations, the images can be in blue and violet color channels instead of or in addition to the red and green channels.
Regarding the image data generated during the sequencing run 300, imaging cycle of the flow cell is performed such that the image data can be collected for the entire flow cell by scanning the flow cell area (e.g., using a line scanner), with one or more coherent sources of light. By way of example, the imaging system 200 can use LGM 210 in coordination with the optics of the system to line scan the flow cell with light having wavelengths within the red color spectrum and to line scan the sample with light having wavelengths within the green color spectrum. In response to line scanning, fluorescent dyes situated at the different clusters of the flow cell fluoresce and the resultant light can be collected by the objective lens 235 and directed to an image sensor of CAM 220 to detect the florescence. For example, fluorescence of each cluster can be detected by a few pixels of CAM 220. Image data output from CAM 220 can then be communicated to the real time analysis module 225 for image noise correction and base calling.
The population of clusters is spatially distributed across the flow cell. The flow cell, and therefore the population of clusters and the image data, is partitionable into different spatial configurations defined by different regions of the flow cell. So, for example, if the flow cell is partitionable into three rectangular regions, this would result in three spatial configurations of the flow cell, three sub-populations or classes of the clusters, three subsets of the image data, and three specialist signal profilers. At the next granular scale, each of the three rectangular regions of the flow cell can be further divided into three squares, resulting in a total of nine squares. This would in turn result in nine spatial configurations of the flow cell, nine sub-populations or classes of the clusters, nine subsets of the image data, and nine specialist signal profilers.
Regarding the segmentation of the image data, the image data is divided into a plurality of image data subsets corresponding to a respective region of the flow cell. In various implementations, the size of the image data subsets can be determined using the placement of fiducial markers or fiducials in the field of view of the imaging system 200, in the flow cell, or on the flow cell. The image data subsets can be divided such that the pixels of each image data subset have a predetermined number of fiducials (e.g., at least three fiducials, four fiducials, six fiducials, eight fiducials, etc.). For example, the total number of pixels of the image data subset may be predetermined based on predetermined pixel distances between the boundaries of the image data subset and the fiducials.
Furthermore, intensity data in each subset of image data can span across multiple color channels (e.g., red, blue, green, and/or violet image channels). Accordingly, each specialist signal profiler has a respective set of coefficients trained to maximize the signal-to-noise ratio of intensity data in a corresponding color channel.
During inference, new/unseen/wild image data is segmented on the same basis as the spatial configurations defined and used to generate the respective/separate/different/independent specialist signal profilers at training. The respective/separate/different/independent specialist signal profilers are applied to segmented new image data subsets in a way that the application of a particular specialist signal profiler is confined only to a corresponding new image data subset.
The application of the specialist signal profilers 1 to N to the image data subsets 1 to N generates signal-to-noise ratio maximized image data subsets 1 to N, additional details about which can be found in U.S. Nonprovisional patent application Ser. No.: 17/308,035, titled “EQUALIZATION-BASED IMAGE PROCESSING AND SPATIAL CROSSTALK ATTENUATOR,” filed May 4, 2021 (Attorney Docket No. ILLM 1032-2/IP-1991-US). Other corrections for channel, spatial, or phasing noise can be applied in advance to the unprocessed image data, or subsequently to the signal-to-noise ratio maximized image data subsets 1 to N.
The output of the specialist signal profilers 1 to N, i.e., the signal-to-noise ratio maximized image data subsets 1 to N, are provided as input to a base caller 332 to produce base calls 1 to N for the population of clusters. Base calling may be performed by fitting a mathematical model to the intensity data. Suitable mathematical models that can be used include, for example, a k-means clustering algorithm, a k-means-like clustering algorithm, expectation maximization clustering algorithm, a histogram based method, and the like. Four Gaussian distributions may be fit to the set of two-channel intensity data such that one distribution is applied for each of the four nucleotides represented in the data set. In one particular implementation, an expectation maximization (EM) algorithm may be applied. As a result of the EM algorithm, for each X, Y value (referring to each of the two channel intensities respectively) a value can be generated which represents the likelihood that a certain X, Y intensity value belongs to one of four Gaussian distributions to which the data is fitted. Where four bases give four separate distributions, each X, Y intensity value will also have four associated likelihood values, one for each of the four bases. The maximum of the four likelihood values indicates the base call. For example, if a cluster is “off” in both channels, the base call is G. If the cluster is “off” in one channel and “on” in another channel the base call is either C or T (depending on which channel is on), and if the cluster is “on” in both channels the base call is A.
In some implementations, the flow cell 400 can be a multi-plane sample comprising multiple planes of clusters that are sampled during an imaging run. Examples of the multiple planes of clusters include a top surface 402 and a bottom surface 412. Accordingly, in one implementation, the flow cell 400 can have two spatial configurations corresponding to the top and bottom surfaces 402 and 412, which in turn result in two sub-populations or classes of the clusters, two subsets of the image data, and two specialist signal profilers.
The first specialist signal profiler 604a is configured to maximize the signal-to-noise ratio of intensity data in the first image data subset 632 to generate a signal-to-noise ratio maximized version 634 of the first image data subset 632. The second specialist signal profiler 604b is configured to maximize the signal-to-noise ratio of intensity data in the second image data subset 642 to generate a signal-to-noise ratio maximized version 644 of the second image data subset 642. The base caller 332 processes the signal-to-noise ratio maximized versions 634, 644 and generates base calls 638, 648.
As used herein, the term “maximized version” refers to data produced as output by a specialist signal profiler. The maximized version of an input is an output produced by a specialist signal profiler in response to processing a corresponding input. The maximized version of the input has a signal-to-noise ratio (SNR, S/R) that is greater than the corresponding input processed by the specialist signal profiler. For example, the maximized version of the input (i.e., the output) produced by the specialist signal profiler is corrected by the specialist signal profiler to have less spatial crosstalk and background noise compared to the corresponding input. Similarly, the maximized version of the input (i.e., the output) produced by the specialist signal profiler is corrected by the specialist signal profiler to have less phasing and pre-phasing noise compared to the corresponding input.
In one implementation, the top surface 402 can be divided or partitioned into a plurality of lanes 508a, 508b, . . . , 508l. In the example illustrated in
The first specialist signal profiler 714a is configured to maximize the signal-to-noise ratio of intensity data in the first image data subset 902 to generate a signal-to-noise ratio maximized version 904 of the first image data subset 902. The second specialist signal profiler 714b is configured to maximize the signal-to-noise ratio of intensity data in the second image data subset 912 to generate a signal-to-noise ratio maximized version 914 of the second image data subset 912, and so on (continuing to signal-to-noise ratio maximized version 924 of the 1th image data subset 922). The base caller 332 processes the signal-to-noise ratio maximized versions 904, 914, . . . , 924 and generates base calls 908, 918, . . . , 928.
In some implementations, the lanes can be grouped into lane groups 502a, 502b, and 502c. Examples of the lane groups include top peripheral lanes, central lanes, and bottom peripheral lanes. Accordingly, in one implementation, the flow cell 400 can have six spatial configurations corresponding to three lane groups of the top surface 402 and three lane groups of the bottom surface 412, which in turn result in six sub-populations or classes of the clusters, six subsets of the image data, and six specialist signal profilers.
The first specialist signal profiler 704a is configured to maximize the signal-to-noise ratio of intensity data in the first image data subset 802 to generate a signal-to-noise ratio maximized version 804 of the first image data subset 802. The second specialist signal profiler 704b is configured to maximize the signal-to-noise ratio of intensity data in the second image data subset 812 to generate a signal-to-noise ratio maximized version 814 of the second image data subset 812. The third specialist signal profiler 704c is configured to maximize the signal-to-noise ratio of intensity data in the third image data subset 822 to generate a signal-to-noise ratio maximized version 824 of the third image data subset 822. The base caller 332 processes the signal-to-noise ratio maximized versions 804, 814, 824 and generates base calls 808, 818, 828.
In some implementations, each lane comprises one or more columns/swathes 506a, 506b of tiles, as shown in
The first specialist signal profiler 724a is configured to maximize the signal-to-noise ratio of intensity data in the first image data subset 1002 to generate a signal-to-noise ratio maximized version 1004 of the first image data subset 1002. The second specialist signal profiler 724b is configured to maximize the signal-to-noise ratio of intensity data in the second image data subset 1012 to generate a signal-to-noise ratio maximized version 1014 of the second image data subset 1012. The base caller 332 processes the signal-to-noise ratio maximized versions 1004, 1014, 1024 and generates base calls 1008, 1018.
Each swath comprises a plurality of tiles 512a, 512b, 512t, as shown in
The first specialist signal profiler 734a is configured to maximize the signal-to-noise ratio of intensity data in the first image data subset 1102 to generate a signal-to-noise ratio maximized version 1104 of the first image data subset 1102. The second specialist signal profiler 734b is configured to maximize the signal-to-noise ratio of intensity data in the second image data subset 1112 to generate a signal-to-noise ratio maximized version 1114 of the second image data subset 1112, and so on (continuing to signal-to-noise ratio maximized version 1124 of the tth image data subset 1122). The base caller 332 processes the signal-to-noise ratio maximized versions 1104, 1114, . . . , 1124 and generates base calls 1108, 1118, . . . , 1128.
Each tile can be divided into a plurality of sub-tiles 518a, 518b, . . . , 518s, as shown in
The first specialist signal profiler 744a is configured to maximize the signal-to-noise ratio of intensity data in the first image data subset 1202 to generate a signal-to-noise ratio maximized version 1204 of the first image data subset 1202. The second specialist signal profiler 744b is configured to maximize the signal-to-noise ratio of intensity data in the second image data subset 1212 to generate a signal-to-noise ratio maximized version 1214 of the second image data subset 1212. The third specialist signal profiler 744c is configured to maximize the signal-to-noise ratio of intensity data in the third image data subset 1222 to generate a signal-to-noise ratio maximized version 1224 of the third image data subset 1222. The fourth specialist signal profiler 744d is configured to maximize the signal-to-noise ratio of intensity data in the fourth image data subset 1232 to generate a signal-to-noise ratio maximized version 1234 of the fourth image data subset 1232, and so on (continuing to signal-to-noise ratio maximized version 1244 of the sth image data subset 1242). The base caller 332 processes the signal-to-noise ratio maximized versions 1204, 1214, 1224, 1234, . . . , 1244 and generates base calls 1208, 1218, 1228, 1238, . . . , 1248.
The first specialist signal profiler 1422 is configured to maximize the signal-to-noise ratio of intensity data of the clusters located on the sub-tile 518a and generated during the sequencing cycles 1 to N1. The second specialist signal profiler 1424 is configured to maximize the signal-to-noise ratio of intensity data of the clusters located on the sub-tile 518a and generated during the sequencing cycles N1+1 to N2. The third specialist signal profiler 1428 is configured to maximize the signal-to-noise ratio of intensity data of the clusters located on the sub-tile 518a and generated during the sequencing cycles N2+1 to N.
The fourth specialist signal profiler 1432 is configured to maximize the signal-to-noise ratio of intensity data of the clusters located on the sub-tile 518b and generated during the sequencing cycles 1 to N1. The fifth specialist signal profiler 1434 is configured to maximize the signal-to-noise ratio of intensity data of the clusters located on the sub-tile 518b and generated during the sequencing cycles N1+1 to N2. The sixth specialist signal profiler 1438 is configured to maximize the signal-to-noise ratio of intensity data of the clusters located on the sub-tile 518b and generated during the sequencing cycles N2+1 to N.
Segmentation logic 1622 segments the training data 1612 based on one or more configurations selected from different spatial configurations, temporal configurations, signal distribution configurations, or any combination thereof. What results is segmented training data 1632 with configuration-specific training data subsets 1 to N. For example, the training data 1612 can include K images of a tile from K imaging cycles of a completed/already-executed sequencing run, with each tile image having multiple color channels.
Offline training logic 1642 trains respective/separate/different/independent specialist signal profilers 1 to N on the respective configuration-specific training data subsets 1 to N. What results is trained specialist signal profilers 1 to N. Returning to the example in
Inference data 1618 is generated during an inference stage 1608. The inference data 1618 includes the sequenced data from the ongoing sequencing run (e.g., the first i cycles of the ongoing sequencing run).
The segmentation logic 1622 segments the inference data 1618 based on the same one or more configurations that were used to segment the training data 1612 during the training stage 1602. What results is segmented inference data 1638 with configuration-specific inference data subsets 1 to N. For example, the inference data 1618 can include K images of a tile from K imaging cycles of the ongoing sequencing run, with each tile image having multiple color channels. Returning to the example in
Runtime logic 1648 applies the respective trained specialist signal profilers 1 to N 1658 on the respective configuration-specific inference data subsets 1 to N. Returning to the example in
Segmentation logic 1622 segments the inference data 1812 based on one or more configurations selected from different spatial configurations, temporal configurations, signal distribution configurations, or any combination thereof. What results is segmented inference data 1832 with configuration-specific training data subsets 1 to N. For example, the inference data 1812 can include N1 images of a tile from the N1 earlier sequencing cycles of the ongoing sequencing run, with each tile image having multiple color channels.
Offline training logic 1842 trains respective/separate/different/independent specialist signal profilers 1 to N on the respective configuration-specific training data subsets 1 to N. What results is trained specialist signal profilers 1 to N. Returning to the example in
Inference data 1818 is also generated during the inference stage 1802. The inference data 1818 includes the sequenced data from the later sequencing cycles (e.g., cycles N1+1 to N2) of the ongoing sequencing run.
The segmentation logic 1622 segments the inference data 1818 based on the same one or more configurations that were used to segment the inference data 1812 during the earlier sequencing cycles (e.g., cycles 1 to N1) of the ongoing sequencing run. What results is segmented inference data 1838 with configuration-specific inference data subsets 1 to N. For example, the inference data 1818 can include N2 images of a tile from the N2 later sequencing cycles of the ongoing sequencing run, with each tile image having multiple color channels. Returning to the example in
Runtime logic 1648 applies the respective trained specialist signal profilers 1 to N 1858 on the respective configuration-specific inference data subsets 1 to N. Returning to the example in
In some implementations, the training process is iteratively repeated where the respective trained specialist signal profilers 1 to N 1858 are retrained/further trained on the segmented inference data 1838 from the later sequencing cycles (e.g., cycles N1+1 to N2) of the ongoing sequencing run, and applied on segmented inference data from even later sequencing cycles (e.g., cycles N2+1 to N3) of the ongoing sequencing run.
A control logic (not shown) can iterate, at each successive sequencing cycle of the ongoing sequencing run, or at successive subseries of sequencing cycles of the ongoing sequencing run (e.g., every ten or twenty sequencing cycles)—(i) the segmentation of a current batch of image data based on one or more configurations selected from different spatial configurations, temporal configurations, signal distribution configurations, or any combination thereof, (ii) the retraining of the respective trained specialist signal profilers 1 to N on the segmented current batch of image data, (iii) segmentation of a next batch of image data on the same basis as the current batch of image data, and (iv) the application of the respective retrained specialist signal profilers 1 to N to the segmented next batch of image data.
As used herein, the phrase “similar signal distribution” refers to those signal distributions that share substantially overlapping signal patterns. For example, two signal patters of similar shape (e.g., trapezoids) but of different shape sizes (e.g., one bigger trapezoid and one smaller trapezoid) can be considered to have similar signal distributions. Similarly, two signal patterns with a common centroid or centroids within a range of, for example, one to five units in each dimension, can be considered to have similar signal distributions.
In
As used herein, the phrase “different temporal stages of an ongoing sequencing run” refers to different sequencing cycles or different sequencing cycle groups of the ongoing sequencing run. For example, if the sequencing run has 150 sequencing cycles, then each successive sequencing cycle can be considered a different temporal stage, or groups of sequencing cycles like cycles 1-20, cycles 20-40, cycles 40-70, and so on can be considered different temporal stages.
At operation 2113, a template image is produced that identifies locations of clusters on a tile using sequencing images from some number of initial sequencing cycles called template cycles. The template image is used as a reference for subsequent registration and intensity extraction steps. The template image is generated by detecting and merging bright spots in each sequencing image of the template cycles, which in turn involves sharpening a sequencing image 2114 (e.g., using the Laplacian convolution), determining an “on” threshold by a spatially segregated Otsu approach, and subsequent five-pixel local maximum detection with subpixel location interpolation. The phrase “on threshold” can refer to an intensity value that exceeds a preset value, for example, 200 or 320, such that the intensity value is detected to be more than a background intensity value or a noise intensity value.
The processing pipeline then registers a current sequencing image against the template image. This is achieved by using image correlation to align the current sequencing image to the template image on a sub-region, or by using non-linear transformations (e.g., a full six-parameter linear affine transformation).
At operation 2115, the processing pipeline applies a non-linear distortion to each spot to account for, for example, optical distortions caused by the geometry of the optical lens. The non-linear distortion can be applied as channel-dependent coefficients of third order polynomials.
At operation 2116, the processing pipeline segments the tile images into sub-tile images based on one or more configurations selected from different spatial configurations, temporal configurations, signal distribution configurations, or any combination thereof.
At operation 2118, intensity is extracted from the segmented sub-tile images using the corresponding specialist signal profilers 1 to N.
At operation 2123, the sub-tile intensities are spatially normalized, for example, by making the 90th percentiles of their extracted intensities are equal.
At operation 2124, the sub-tile intensities are compressed.
At operation 2125, the processing pipeline applies empirical phasing correction to compensate noise in the image data caused by phasing and pre-phasing errors.
At operation 2125, the processing pipeline spatially normalizes the extracted signal intensities to account for variation in illumination across the sampled imaged. For example, intensity values can be normalized such that a 5th and 95th percentiles have values of 0 and 1, respectively. The normalized signal intensities for the image (e.g., normalized intensities for each channel) can be used to calculate mean chastity for the plurality of spots in the image.
At operation 2133, the processing pipeline scale the intensities on a cluster-by-cluster basis to account for variation in brightness of clusters.
At operation 2134, the processing pipeline uses the expectation maximization (EM) algorithm to produce base calls, as discussed above.
At operation 2135, the processing pipeline assigns quality scores to the called bases using quality tables (Q-Tables) 2152.
At operation 2136, the processing pipeline aligns the called bases to a reference genome (e.g., of the PhiX bacteria) to compute a mismatch rate.
The processing pipeline generates certain outputs, such as base call and quality score 2128, inter-operation (InterOp) files 2138 (binary reporting files for sequencing analysis view), and logs 2148 (e.g., error logs, general event logs, processing event logs, warning event logs).
Other examples of configurations covered by this disclosure include segmenting sequencing data and training corresponding specialist signal profilers by library type, sample type, indexing type (first index read v/s second index read), read type (forward read v/s reverse read), physical properties of the sample, noise type (e.g., bubble), and reagent type.
In one implementation, specialist signal profilers 3718 are communicably linked to the storage subsystem 3710 and the user interface input devices 3738.
User interface input devices 3738 can include a keyboard; pointing devices such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touch screen incorporated into the display; audio input devices such as voice recognition systems and microphones; and other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computer system 3700.
User interface output devices 3776 can include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem can include an LED display, a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem can also provide a non-visual display such as audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computer system 3700 to the user or to another machine or computer system.
Storage subsystem 3710 stores programming and data constructs that provide the functionality of some or all of the modules and methods described herein. These software modules are generally executed by processors 3778.
Processors 3778 can be graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and/or coarse-grained reconfigurable architectures (CGRAs). Processors 3778 can be hosted by a deep learning cloud platform such as Google Cloud Platform™, Xilinx™, and Cirrascale™. Examples of processors 3778 include Google's Tensor Processing Unit (TPU)™, rackmount solutions like GX4 Rackmount Series™, GX37 Rackmount Series™, NVIDIA DGX-1™, Microsoft' Stratix V FPGA™, Graphcore's Intelligent Processor Unit (IPU)™, Qualcomm's Zeroth Platform™ with Snapdragon processors™, NVIDIA's Volta™, NVIDIA's DRIVE PX™, NVIDIA's JETSON TX1/TX2 MODULE™, Intel's Nirvana™, Movidius VPU™, Fujitsu DPI™, ARM's DynamiclQ™, IBM TrueNorth™, Lambda GPU Server with Testa V100s™, and others.
Memory subsystem 3722 used in the storage subsystem 3710 can include a number of memories including a main random access memory (RAM) 3732 for storage of instructions and data during program execution and a read only memory (ROM) 3734 in which fixed instructions are stored. A file storage subsystem 3736 can provide persistent storage for program and data files, and can include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations can be stored by file storage subsystem 3736 in the storage subsystem 3710, or in other machines accessible by the processor.
Bus subsystem 3755 provides a mechanism for letting the various components and subsystems of computer system 3700 communicate with each other as intended. Although bus subsystem 3755 is shown schematically as a single bus, alternative implementations of the bus subsystem can use multiple busses.
Computer system 3700 itself can be of varying types including a personal computer, a portable computer, a workstation, a computer terminal, a network computer, a television, a mainframe, a server farm, a widely-distributed set of loosely networked computers, or any other data processing system or user device. Due to the ever-changing nature of computers and networks, the description of computer system 3700 depicted in
The following discussion focuses on a neural network-based base caller described herein that can be used in conjunction with the specialist signal profilers. First, the input to the neural network-based base caller is described, in accordance with one implementation. Then, examples of the structure and form of the neural network-based base caller are provided. Finally, the output of the neural network-based base caller is described, in accordance with one implementation.
A data flow logic provides the sequencing images to the neural network-based base caller for base calling. The neural network-based base caller accesses the sequencing images on a patch-by-patch basis (or a tile-by-tile basis). Each of the patches is a sub-grid (or sub-array) of pixelated units in the grid of pixelated units that forms the sequencing images. The patches have dimensions q×r of the sub-grid of pixelated units, where q (width) and r (height) are any numbers ranging from 1 and 10000 (e.g., 3×3, 5×5, 7×7, 10×10, 15×15, 25×25, 64×64, 78×78, 115×115). In some implementations, q and r are the same. In other implementations, q and r are different. In some implementations, the patches extracted from a sequencing image are of the same size. In other implementations, the patches are of different sizes. In some implementations, the patches can have overlapping pixelated units (e.g., on the edges).
Sequencing produces m sequencing images per sequencing cycle for corresponding m image channels. That is, each of the sequencing images has one or more image (or intensity) channels (analogous to the red, green, blue (RGB) channels of a color image). In one implementation, each image channel corresponds to one of a plurality of filter wavelength bands. In another implementation, each image channel corresponds to one of a plurality of imaging events at a sequencing cycle. In yet another implementation, each image channel corresponds to a combination of illumination with a specific laser and imaging through a specific optical filter. The image patches are tiled (or accessed) from each of the m image channels for a particular sequencing cycle. In different implementations such as 4-, 2-, and 1-channel chemistries, m is 4 or 2. In other implementations, m is 1, 3, or greater than 4. In other implementations, the images can be in blue and violet color channels instead of or in addition to the red and green channels.
Consider, for example, that a sequencing run is implemented using two different image channels: a blue channel and a green channel. Then, at each sequencing cycle, the sequencing run produces a blue image and a green image. This way, for a series of k sequencing cycles of the sequencing run, a sequence of k pairs of blue and green images is produced as output and stored as the sequencing images. Accordingly, a sequence of k pairs of blue and green image patches is generated for the patch-level processing by the neural network-based base caller.
The input image data to the neural network-based base caller for a single iteration of base calling (or a single instance of forward pass or a single forward traversal) comprises data for a sliding window of multiple sequencing cycles. The sliding window can include, for example, a current sequencing cycle, one or more preceding sequencing cycles, and one or more successive sequencing cycles.
In one implementation, the input image data comprises data for three sequencing cycles, such that data for a current (time t) sequencing cycle to be base called is accompanied with (i) data for a left flanking/context/previous/preceding/prior (time t−1) sequencing cycle and (ii) data for a right flanking/context/next/successive/subsequent (time t+1) sequencing cycle.
In another implementation, the input image data comprises data for five sequencing cycles, such that data for a current (time t) sequencing cycle to be base called is accompanied with (i) data for a first left flanking/context/previous/preceding/prior (time t−1) sequencing cycle, (ii) data for a second left flanking/context/previous/preceding/prior (time t−2) sequencing cycle, (iii) data for a first right flanking/context/next/successive/subsequent (time t+1), and (iv) data for a second right flanking/context/next/successive/subsequent (time t+2) sequencing cycle.
In yet another implementation, the input image data comprises data for seven sequencing cycles, such that data for a current (time t) sequencing cycle to be base called is accompanied with (i) data for a first left flanking/context/previous/preceding/prior (time t−1) sequencing cycle, (ii) data for a second left flanking/context/previous/preceding/prior (time t−2) sequencing cycle, (iii) data for a third left flanking/context/previous/preceding/prior (time t−3) sequencing cycle, (iv) data for a first right flanking/context/next/successive/subsequent (time t+1), (v) data for a second right flanking/context/next/successive/subsequent (time t+2) sequencing cycle, and (vi) data for a third right flanking/context/next/successive/subsequent (time t+3) sequencing cycle. In other implementations, the input image data comprises data for a single sequencing cycle. In yet other implementations, the input image data comprises data for 10, 15, 20, 30, 58, 75, 92, 130, 168, 175, 209, 225, 230, 275, 318, 325, 330, 525, or 625 sequencing cycles.
The neural network-based base caller processes the image patches through its convolution layers and produces an alternative representation, according to one implementation. The alternative representation is then used by an output layer (e.g., a softmax layer) for generating a base call for either just the current (time t) sequencing cycle or each of the sequencing cycles, i.e., the current (time t) sequencing cycle, the first and second preceding (time t−1, time t−2) sequencing cycles, and the first and second succeeding (time t+1, time t+2) sequencing cycles. The resulting base calls form the sequencing reads.
In one implementation, the neural network-based base caller outputs a base call for a single target cluster for a particular sequencing cycle. In another implementation, the neural network-based base caller outputs a base call for each target cluster in a plurality of target clusters for the particular sequencing cycle. In yet another implementation, the neural network-based base caller outputs a base call for each target cluster in a plurality of target clusters for each sequencing cycle in a plurality of sequencing cycles, thereby producing a base call sequence for each target cluster.
In one implementation, the neural network-based base caller is a multilayer perceptron (MLP). In another implementation, the neural network-based base caller is a feedforward neural network. In yet another implementation, the neural network-based base caller is a fully-connected neural network. In a further implementation, the neural network-based base caller is a fully convolution neural network. In yet further implementation, the neural network-based base caller is a semantic segmentation neural network. In yet another further implementation, the neural network-based base caller is a generative adversarial network (GAN).
In one implementation, the neural network-based base caller is a convolution neural network (CNN) with a plurality of convolution layers. In another implementation, the neural network-based base caller is a recurrent neural network (RNN) such as a long short-term memory network (LSTM), bi-directional LSTM (Bi-LSTM), or a gated recurrent unit (GRU). In yet another implementation, the neural network-based base caller includes both a CNN and an
RNN.
In yet other implementations, the neural network-based base caller can use 1D convolutions, 2D convolutions, 3D convolutions, 4D convolutions, 5D convolutions, dilated or atrous convolutions, transpose convolutions, depthwise separable convolutions, pointwise convolutions, 1×1 convolutions, group convolutions, flattened convolutions, spatial and cross-channel convolutions, shuffled grouped convolutions, spatial separable convolutions, and deconvolutions. The neural network-based base caller can use one or more loss functions such as logistic regression/log loss, multi-class cross-entropy/softmax loss, binary cross-entropy loss, mean-squared error loss, L1 loss, L2 loss, smooth L1 loss, and Huber loss. The neural network-based base caller can use any parallelism, efficiency, and compression schemes such TFRecords, compressed encoding (e.g., PNG), sharding, parallel calls for map transformation, batching, prefetching, model parallelism, data parallelism, and synchronous/asynchronous stochastic gradient descent (SGD). The neural network-based base caller can include upsampling layers, downsampling layers, recurrent connections, gates and gated memory units (like an LSTM or GRU), residual blocks, residual connections, highway connections, skip connections, peephole connections, activation functions (e.g., non-linear transformation functions like rectifying linear unit (ReLU), leaky ReLU, exponential liner unit (ELU), sigmoid and hyperbolic tangent (tanh)), batch normalization layers, regularization layers, dropout, pooling layers (e.g., max or average pooling), global average pooling layers, and attention mechanisms.
The neural network-based base caller is trained using backpropagation-based gradient update techniques. Example gradient descent techniques that can be used for training the neural network-based base caller include stochastic gradient descent, batch gradient descent, and mini-batch gradient descent. Some examples of gradient descent optimization algorithms that can be used to train the neural network-based base caller are Momentum, Nesterov accelerated gradient, Adagrad, Adadelta, RMSprop, Adam, AdaMax, Nadam, and AMSGrad.
In one implementation, the neural network-based base caller uses a specialized architecture to segregate processing of data for different sequencing cycles. The motivation for using the specialized architecture is described first. As discussed above, the neural network-based base caller processes image patches for a current sequencing cycle, one or more preceding sequencing cycles, and one or more successive sequencing cycles. Data for additional sequencing cycles provides sequence-specific context. The neural network-based base caller learns the sequence-specific context during training and base calls them. Furthermore, data for pre and post sequencing cycles provides second order contribution of pre-phasing and phasing signals to the current sequencing cycle.
However, images captured at different sequencing cycles and in different image channels are misaligned and have residual registration error with respect to each other. To account for this misalignment, the specialized architecture comprises spatial convolution layers that do not mix information between sequencing cycles and only mix information within a sequencing cycle.
Spatial convolution layers (or spatial logic) use so-called “segregated convolutions” that operationalize the segregation by independently processing data for each of a plurality of sequencing cycles through a “dedicated, non-shared” sequence of convolutions. The segregated convolutions convolve over data and resulting feature maps of only a given sequencing cycle, i.e., intra-cycle, without convolving over data and resulting feature maps of any other sequencing cycle.
Consider, for example, that the input image data comprises (i) current image patch for a current (time t) sequencing cycle to be base called, (ii) previous image patch for a previous (time t−1) sequencing cycle, and (iii) next image patch for a next (time t+1) sequencing cycle. The specialized architecture then initiates three separate convolution pipelines, namely, a current convolution pipeline, a previous convolution pipeline, and a next convolution pipeline. The current data processing pipeline receives as input the current image patch for the current (time t) sequencing cycle and independently processes it through a plurality of spatial convolution layers to produce a so-called “current spatially convolved representation” as the output of a final spatial convolution layer. The previous convolution pipeline receives as input the previous image patch for the previous (time t−1) sequencing cycle and independently processes it through the plurality of spatial convolution layers to produce a so-called “previous spatially convolved representation” as the output of the final spatial convolution layer. The next convolution pipeline receives as input the next image patch for the next (time t+1) sequencing cycle and independently processes it through the plurality of spatial convolution layers to produce a so-called “next spatially convolved representation” as the output of the final spatial convolution layer.
In some implementations, the current, previous, and next convolution pipelines are executed in parallel. In some implementations, the spatial convolution layers are part of a spatial convolution network (or subnetwork) within the specialized architecture.
The neural network-based base caller further comprises temporal convolution layers (or temporal logic) that mix information between sequencing cycles, i.e., inter-cycles. The temporal convolution layers receive their inputs from the spatial convolution network and operate on the spatially convolved representations produced by the final spatial convolution layer for the respective data processing pipelines.
The inter-cycle operability freedom of the temporal convolution layers emanates from the fact that the misalignment property, which exists in the image data fed as input to the spatial convolution network, is purged out from the spatially convolved representations by the stack, or cascade, of segregated convolutions performed by the sequence of spatial convolution layers.
Temporal convolution layers use so-called “combinatory convolutions” that groupwise convolve over input channels in successive inputs on a sliding window basis. In one implementation, the successive inputs are successive outputs produced by a previous spatial convolution layer or a previous temporal convolution layer.
In some implementations, the temporal convolution layers are part of a temporal convolution network (or subnetwork) within the specialized architecture. The temporal convolution network receives its inputs from the spatial convolution network. In one implementation, a first temporal convolution layer of the temporal convolution network groupwise combines the spatially convolved representations between the sequencing cycles. In another implementation, subsequent temporal convolution layers of the temporal convolution network combine successive outputs of previous temporal convolution layers. The output of the final temporal convolution layer is fed to an output layer that produces an output. The output is used to base call one or more clusters at one or more sequencing cycles.
Additional details about the neural network-based base caller can be found in U.S. Provisional Patent Application No. 62/821,766, titled “ARTIFICIAL INTELLIGENCE-BASED SEQUENCING,” (Attorney Docket No. ILLM 1008-9/IP-1752-PRV), filed on Mar. 21, 2019, which is incorporated herein by reference.
This application claims the benefit of U.S. Provisional Patent Application No. 63/223,408, titled, “SPECIALIST SIGNAL PROFILERS FOR BASE CALLING,” filed Jul. 19, 2021 (Attorney Docket No. ILLM 1041-1/IP-2063-PRV). The provisional application is hereby incorporated by reference for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 63223408 | Jul 2021 | US |
