Patent Grant
9109861
The entire contents of a paper copy of the “Sequence Listing” in computer readable form (.txt) that is identical to the PatentIn 3.5 computer software generated sequence listing containing the file named 408230_SequenceListing_ST25.txt, which is 2 kilobytes in size and was created on Jul. 20, 2010, is herein incorporated by reference in its entirety.
Not applicable.
Genomics, the study of the genomes of organisms, is an active area of biotechnology that has been used to identify targets for pharmaceutical compounds and to gain insight into disease mechanisms. As the field of genomics has matured, automated nucleotide sequencing techniques and advanced assembly algorithms have resulted in the complete sequencing of the genomes of hundreds of organisms, including the human genome. Recently developed automated sequencing devices make use of automated and massively parallel sequencing techniques to produce gigabases of data in a single machine run. These automated sequencing devices have greatly reduced the time and cost of sequencing the genomes of individual humans, making personalized medicine increasingly feasible. Personalized medicine makes use of an individual's sequenced genome to determine the individual's susceptibility to diseases and to determine the individual's responsiveness to drug regimes.
Typically, automated sequencing technologies cleave the nucleotide sample to be processed into a set of smaller nucleotide strands, which are then sequenced, yielding a read sequence set containing all of the reads from a sample. Each read of the read sequence set is the nucleotide sequence of one of the smaller nucleotide strands, as determined by the automated sequencing device. The read sequence set is then assembled into a complete genome by piecing together the reads into a continuous nucleotide sequence by aligning overlapping portions of the reads in the set.
A challenging computational task related to assembling the read sequence data into a genome is the alignment of the sequences. The sequences in the read sequence data set may either be compared to other sequences in the data set (de novo assembly) and arranged so that sequences with the same series of nucleotide bases are overlapped, or the sequences may be assembled by aligning the sequences in the read sequence data set to a data set of an existing template genome, such as a data set in a publicly available genetic database (reference-guided assembly). Regardless of the methods used, the assembly of the sequences into a genome entails the pairwise comparison of the sequences of one data set to the sequences from another data set.
During the pairwise comparison process, all of the nucleotide sequences of one data set are compared individually to all of the nucleotide sequences of another data set. When both of the data sets are small and can be completely stored into the volatile memory of a computing device, it is reasonably efficient to pass nucleotide sequences of the data sets from storage on a nonvolatile memory device, such as a hard disk, to the volatile memory of the computing device to make the pairwise comparisons.
However, automated sequencing devices may potentially generate enormous data sets containing large numbers of nucleotide sequences that typically are stored in nonvolatile memory during the genome assembly process. Conventional gene assembly systems process entire data sets in a single operation, requiring careful management of the transfer of nucleotide sequence data from nonvolatile memory to volatile memory for processing. However, despite the measures taken to minimize processing times, conventional gene assembly systems typically have long processing times. For example, typical existing techniques organize the pairwise comparison of sequences in a raster-type scan, which requires a great deal of computational overhead for larger data sets. Advanced computational systems with large volumes of volatile memory operating at maximum capacity and a large time frame are typically required to assemble such large data sets.
In one embodiment, a layout assembly system automatically assembles a genomic nucleotide sequence. The layout assembly system includes memory that contains a read sequence set, a comparing sequence set, and one or more modules configured to execute on at least one processor.
The read set includes a plurality of read entries. Each read entry includes a read sequence and a read sequence index assigned to identify the read sequence. The read sequence includes a first sequence of nucleotides.
The comparing sequence set includes one or more comparing entries. Each comparing entry includes a comparing sequence and a comparing sequence index assigned to identify the comparing sequence. The comparing sequence includes a second sequence of nucleotides.
The one or more modules include a mer generating and sorting system, a hit determination system, a hit table sorting system, a hit table condensing system, and a post processing system.
The mer generating and sorting system includes a sequence preprocessing system, a cmer generation system, and a mer sorting system. The sequence preprocessing system divides one or more read mers from each read sequence, assigns to each read mer the read sequence index corresponding to the read sequence from which the read mer was divided, assigns a read position index to each read mer identifying a number of nucleotides from a location within the read sequence from which the read mer was divided and assigns a read strand index to each read mer identifying each read mer as a forward sequence or a reverse sequence. Each read mer includes at least one of a first sequence of nucleotides. The sequence preprocessing system also generates a plurality of read mer table entries that each include one read mer, the read sequence index assigned to the one read mer, the read position index assigned to the one read mer, and the read strand index assigned to the one read mer. The sequence preprocessing system generates a read mer table that includes each read mer table entry.
The sequence preprocessing system also divides one or more comparing mers from each comparing sequence, assigns to each comparing mer the comparing sequence index corresponding to the comparing sequence from which the comparing mer was divided, assigns a comparing position index to each comparing mer identifying a number of nucleotides from a location within the comparing sequence from which the comparing mer was divided and assigns a comparing strand index to each comparing mer identifying each comparing mer as a forward sequence or a reverse sequence. Each comparing mer includes at least one of the second sequences of nucleotides. The sequence preprocessing system also generates a plurality of comparing mer table entries that each includes one comparing mer, the comparing sequence index assigned to the one comparing mer, the comparing position index assigned to the one comparing mer, and the comparing strand index assigned to the one comparing mer. The sequence preprocessing system generates a comparing mer table that includes each comparing mer table entry.
The cmer generation system replaces each read mer that includes a pyrimidine central nucleotide with a first reverse complement. The pyrimidine central nucleotide is a nucleotide located at the same distance from a first nucleotide of the read mer and a last nucleotide of the read mer. The pyrimidine central nucleotide may be cytosine, thymine, or uracil. The first reverse complement is a third sequence of nucleotides. Each third sequence nucleotide is the complement of a first corresponding nucleotide of the first nucleotide sequence arranged in reverse order. Similarly, the cmer generation system replaces each comparing mer comprising the pyrimidine central nucleotide with a second reverse complement. The second reverse complement is a fourth sequence of nucleotides in which each fourth sequence nucleotide is the complement of a second corresponding nucleotide of the second nucleotide sequence arranged in reverse order.
The mer sorting system sorts the one or more read mer table entries by the read mer, and sorts the one or more comparing mer table entries by the comparing mer.
The hit determination system identifies each match between the comparing mers and the read mers and generates a hit table entry for each match. Each hit table entry includes the comparing sequence index corresponding to a matching comparing mer, the read sequence index corresponding to a matching read mer, an orientation index assigned to identify an orientation of the matching comparing mer and the matching read mer as a same orientation or an opposite orientation, and a frameshift assigned to identify a third number of nucleotides to shift the read sequence corresponding to the matching read mer along the comparing sequence to align the matching read mer to the matching comparing mer. The hit determination system also generates a hit table that contains each hit table entry.
The hit table sorting system sorts the hit table entries by the comparing sequence index, sorts the hit table entries having a same comparing sequence index by the read sequence index, sorts the hit table entries having the same comparing sequence index and the same read sequence index by the orientation index, and sorts the hit table entries having the same comparing sequence index, the same read sequence index, and the same orientation index by the frameshift.
The hit table condensing system determines a score for each match between one hit table entry and one or more other hit table entries. The score is a total count of each match. The hit table condensing system creates a condensed hit table entry for each match that includes the one hit table entry and the score, and generates a condensed hit table that contains each condensed hit table entry.
The post processing system generates an output file that includes the condensed hit table.
Systems and methods of the present invention may be used to match the nucleotide sequences of a first sequence set (sequence set 1) to each other, or to match the nucleotide sequences of the first sequence set to the nucleotide sequences of a second sequence set (sequence set 2), which may be used as a template. A sequence set has one or more sequences, and each sequence contains one or more nucleotides. A sequence set optionally may include information identifying the sequences.
The sequences in each sequence set are divided into mers, where each mer may be a subset of a sequence in one of the sequence sets. As the mers are formed, two or more indices are assigned to each of the mers to identify the particular nucleotide sequence from which the mer originated, as well as the location of the mer's sequence of nucleotides within the larger nucleotide sequence. The mers from each sequence set are assembled into a respective mer table along with their associated indices. In one aspect, the mers from each sequence set may be assembled into one mer table, along with their associated indices.
Because the nucleotide sequences of the sequence sets may originate from double-stranded DNA samples, some of the mers may have originated from forward-directed strands and other mers may have originated from reverse-directed strands. The mers in each mer table are processed to generate strand-independent mer tables that remove the effects of the direction of the sequences from subsequent processing.
In one aspect, two strand-independent mer tables are stored in nonvolatile memory. Each of the mer tables is sorted to arrange the mers in ascending or descending order, resulting in the mer table entries with the same mer nucleotide sequence being grouped together in the first sorted mer table and the second sorted mer table. In an aspect, the mers of both tables are the same length to facilitate a pairwise comparison of the mers from the first sorted mer table (table 1) and the mers from the second sorted mer table (table 2).
In another aspect, one strand-independent mer table containing the mers from two or more sequence sets is stored in nonvolatile memory. The mer table is sorted to arrange the mers in ascending or descending order. Each group of mer table entries having the same mer are sorted to arrange each group into two or more subgroups, where each subgroup contains the mer table entries corresponding to one sequence set from which the mers originated.
The mers from each sorted mer table are compared pairwise. Each match between the mers is recorded as an entry in a hit table that contains indices identifying the sequences from which each matched mer originated, the location of each mer within its originating sequence, and a frameshift index that specifies a number of nucleotides to shift the first sequence along the second sequence to align the segments of the sequences from which the matched mers were derived.
During processing, the mer table entries from the two sorted mer tables may be transferred from nonvolatile memory to volatile memory for the pairwise matching in one aspect. Because of the sorted arrangement of the mer table entries, small subsets of the sorted mer tables can be loaded into volatile memory from nonvolatile memory to compare the nucleotide sequences of the mers. In one aspect, a first group of mer table entries from a first mer table having the same mer sequence is transferred from nonvolatile memory to volatile memory for processing. Similarly, only a first group of mer table entries from a comparing table having the same mer sequence is transferred from nonvolatile memory to volatile memory for pairwise matching. When the next groups of mers from the sorted mer tables are obtained for subsequent pairwise matching, each of the mer tables are read starting at the entry following the last entries used in the previous pairwise comparison. In this aspect, the entries of the sorted mer tables are read a single time in one direction, minimizing the processing time spent locating and transferring the mer table entries of the sorted mer tables from nonvolatile memory to volatile memory. In addition, only those table entries required for the comparison are transferred from nonvolatile memory to volatile memory for pairwise comparison, which reduces the amount of volatile memory used for the pairwise comparison process.
Reading and processing groups of mer table entries during the pairwise comparison process greatly reduces the processing time relative to existing assembly methods that require one of the sequence sets to be loaded in its entirety into volatile memory for processing. Because only those mers that actually match are read from the nonvolatile memory into volatile memory for pairwise comparison and matching, the amount of volatile memory required to assemble a nucleotide sequence set using the layout assembly system is greatly reduced relative to the amount of volatile memory required by existing assembly methods. In addition, because the assembly methods of the layout assembly system are not limited by volatile memory capacity, the maximum size of the mer tables that may be processed by the layout assembly system is limited only by nonvolatile memory capacity, which is much larger than volatile memory capacity.
Depending on the type of sequence data and the desired outcome of the assembly process, the entries of the hit table may be analyzed using different processes. For example, the sequences in a first sequence set may be derived from a deoxyribonucleic acid (DNA) sample taken from a patient, and the sequences in a second sequence set may be a set of comparing genes related to a disease state. In this example, the entries of the hit table may be assessed to determine the degree of matching between the patient's sequences and the sequences of the comparing genes to diagnose the patient. In another example, the sequences in the first sequence set may be derived from a messenger ribonucleic acid (mRNA) sample, and the sequences in the second sequence set may be derived from a set of RNA sequences associated with the expression of a particular gene in an organism. In this example, the matches in the hit table may be used to determine the degree of expression of one or more genes in an organism.
Layout Assembly System Overview
The layout generation system 104 processes the one or more sequence sets, each having one or more sequences, identifies matches between one or more of the sequences from one or more of the sequence sets and one or more of the sequences from one or more of the other sequence sets, and generates a hit table that is a summary of the matches. In some instances, the matches may be used to assemble the sequences of one or more of the sequence sets into a longer continuous nucleotide sequence. In some instances, the results of the processing are written to an output file 116 in the output system 108. For example, the arrangement of sequences in a sequence set may be written to the output file 116. The layout generation system 104 may generate information to the user interface 110, including the status of the layout assembly process, results such as the output file 116, or queries for system input.
The layout generation system 104 includes one or more processors 112 and volatile and/or nonvolatile memory and can be embodied by or in one or more distributed or integrated components or systems. The layout generation system 104 may include computer readable media (CRM) 114 on which one or more algorithms, software, modules, data, computer readable instructions, and/or firmware is loaded and/or operates and/or which operates on the one or more processors 112 to implement the systems and methods identified herein. The computer readable media 114 may include volatile media, nonvolatile media, removable media, non-removable media, and/or other media or mediums that can be accessed by a general purpose or special purpose computing device. For example, computer readable media 114 may include computer storage media and communication media, including computer readable media. Computer storage media further may include volatile, nonvolatile, removable, and/or non-removable media implemented in a method or technology for storage of information, such as computer readable instructions, data structures, program modules, and/or other data. Communication media may, for example, embody computer readable instructions, data structures, program modules, algorithms, and/or other data, including as or in a modulated data signal. The communication media may include an information delivery system. The communication media may include wired and/or wireless connections and technologies and may be used to transmit and/or receive wired or wireless communications. Combinations and/or sub-combinations of the systems, components, modules, and methods and processes described herein may be made.
The input system 106 includes one or more devices or systems used to generate or transfer an electronic version of one or more sequences or sequence sets and/or other inputs and data to the layout generation system 104. The input system 106 may include, for example, a hard disk that stores input files that are read by or transferred to the layout generation system 104. The input system 106 also may include a storage system that stores electronic data, such as electronic data files. The input system 106 also may be one or more processing systems and/or communication systems that transmit and/or receive electronic files and/or other electronic document information or data through wireless and/or wire line communication systems, and/or other data to the layout generation system 104. The input system 106 further may include one or more processors, a computer, volatile and/or nonvolatile memory, computer readable media, a mouse, a trackball, touch pad, or other pointer, a key board, another data entry device or system, another input device or system, a user interface for entering data or instructions, and/or a combination of the foregoing. The input system 106 may be embodied by or in or operate using one or more processors or processing systems, one or more distributed or integrated systems, and/or computer readable media. The input system 106 is optional for some embodiments.
Examples of nonvolatile memory include nonvolatile read-only memory, flash memory, and magnetic and/or optical storage devices, such as hard disks, floppy disks, magnetic tape, and optical discs. Other examples of nonvolatile memory exist. Examples of volatile memory include random access memory, such as SRAM and DRAM. Other examples of volatile memory exist.
The output system 108 includes one or more systems or devices that receive, display, and/or store data. The output system 108 may include a communication system that communicates data with another system or component. The output system 108 may be a storage system that temporarily and/or permanently stores data, such as input files, intermediate data tables generated by the layout generation system, output files, and/or other data. The output system 108 also may include a computer, one or more processors, one or more processing systems, or one or more processes that further process data. The output system 108 may otherwise include a monitor or other display device, a computer, a printer, another data output device, volatile and/or nonvolatile memory, other output devices, computer readable media, a user interface for displaying data, and/or a combination of the foregoing. The output system 108 may receive and/or transmit data through a wireless and/or wire line communication system. The output system 108 may be embodied by or in or operate using one or more processors or processing systems, one or more distributed or integrated systems, and/or computer readable media. The output system 108 is optional for some embodiments.
The user interface 110 receives one or more inputs from a user and/or generates data for display to a user. The user interface 110 includes one or more systems or devices that receive, transmit, display, and/or store data of the layout assembly system 102. The user interface 110 may include a monitor or other display device, a keyboard, mouse, or other input device, one or more processors, a computer, a printer, and/or a combination of the foregoing.
Layout Assembly System
The layout generation system 104A matches one of the sequence sets 202-206 to one or more other sequence sets. Each of the sequence sets 202-206 includes one or more nucleotide sequences and, optionally, information identifying the sequences. Each of the nucleotide sequences contains one or more nucleotides. In one example, two sequence sets 202-204 are processed. In another example, one sequence set 202 is processed. In another example, N sequence sets 202-206 are processed, where N is a number greater than 2. In another example, sequence set 1 202 is a read sequence set containing one or more read sequences. A read sequence may be a nucleotide sequence resulting from the analysis of a fragment of genetic material by an automated sequencing device. Examples of genetic material include deoxyribonucleic acid (DNA), messenger ribonucleic acid (mRNA), ribosomal ribonucleic acid (rRNA), and transfer ribonucleic acid (tRNA). Other examples exist. In this example, sequence set 2 204 is a comparing sequence set, which is a sequence set that is compared to the read sequence set to identify the one or more matching sequences. In this example, sequence set 1 202 is assembled using sequence set 2 204 as a template.
In one aspect, one or more sequence sets 204 and/or 206 are used as a template to assemble the sequences of a first sequence set 202. In one aspect, the template sequences are previously analyzed and assembled nucleotide sequences. Examples of previously analyzed and assembled nucleotide sequences include genes of prokaryotes and eukaryotes, prokaryote and eukaryote genomes, and sequences listed in databases, such as the Human Genome Project database, the Human Gene Mutation Database, the Single Nucleotide Polymorphism database, the Haplotype Map, the Watson 454 genome database, and others. In another aspect, the template sequence set is a copy of the first sequence set 202, such as when the first sequence set is a novel sequence set with no existing template. In one aspect, the sequence set ranges in size from about 50 reads per set to about 100 million reads per set. In another aspect, the total size of a sequence set ranges from about 103 nucleotides to about 1011 nucleotides per sequence set.
The layout generation system 104A includes modules to process the sequence sets 202-206. In the example of
The mer generating and sorting system 208 divides each sequence of each sequence set 202-206 into one or more mers and generates a mer table entry for each mer. Each mer table entry includes one or more indices identifying the sequence from which the mer's nucleotides were divided and the location on the sequence from which the mer's nucleotides were divided. The mer generating and sorting system 208 generates a mer table for each sequence set 202-206 processed during a comparison and matching operation, which may include 1, 2, or N sequence sets. Alternately, the mer generating and sorting system 208 generates one mer table that includes mer table entries for each of the sequence sets processed during the comparison and matching operation. Each mer table is sorted by the mer generating and sorting system 208 by the nucleotide sequence to arrange its entries in mer nucleotide sequence order resulting in the sorted mer tables 210-214. The number of sorted mer tables generated by the mer generating and sorting system 208 corresponds to the number of mer tables and the number of sequence sets processed by the mer generating and sorting system. In one example, sequence sets 1 and 2 202-204 are processed and sorted mer tables 1 and 2 210-212 are generated by the mer generating and sorting system. In another example, a single sorted mer table is generated from a single mer table.
The hit determination system 216 pairwise compares the entries of one sorted mer table 210 to the entries of one or more other sorted mer tables 212-214. The hit determination system 216 identifies matches between mers and generates the entries of a hit table 218. A match between one mer and another mer occurs when each of the nucleotides within the one mer occur in the same order as each of the nucleotides within the other mer. In one example, the hit determination system 216 pairwise compares the mers from a sorted mer table 1 210 to the mers of a sorted mer table 2 212. In one example, the sorted mer table 1 210 is a sorted read mer table, and the sorted mer table 2 212 is a sorted comparing mer table.
Each of the hit table entries contain one or more indices, including a sequence set index 1, a sequence set index 2, a sequence index 1, a sequence index 2, a position index 1, a position index 2, a strand index 1, a strand index 2, an orientation index, and a frameshift. The sequence set index 1 and sequence index 2 identify the originating sequence sets of the matched mers. The sequence index 1 and sequence index 2 identify the originating sequences of each matched mer. The position index 1 and position index 2 identify locations in the originating sequences of a selected nucleotide in the matched mers. The strand index 1 and strand index 2 identify whether the matched mers correspond to forward originating sequences or reverse originating sequences. The orientation index specifies whether the matched mers were oriented in the same direction or in the opposite direction relative to each other. The frameshift identifies a number of nucleotides that the originating sequence of one matched mer must be shifted down the originating sequence of the other matched mer to align the nucleotides of both mers. The originating sequence is the sequence from which the mer was divided. Similarly, the originating sequence set is the sequence set containing the originating sequence.
In one aspect, if there are any mers in the sorted mer tables 210-214 that do not match a corresponding mer in one or more other sorted mer tables, the hit table determination system 216 places the mer table entries corresponding to the non-matched mers into orphan mer tables 220-224. For example, mer table entries corresponding to non-matched mers from sorted mer table 1 210 are placed in orphan mer table 1 220, and mer table entries corresponding to non-matched mers from sorted mer table 2 212 are placed in an orphan mer table 2 222. In another aspect, one orphan mer table holds mer table entries for all non-matched mers from all mer tables processed by the hit determination system 216.
Overly repeated matches between mers from the sorted mer tables 210-214 and 212 may not be informative to the nucleotide assembly process in one aspect. A group of mer matches may be identified as overly repeated if the number of mer matches in the group exceeds a user-defined repeated mer threshold. In this aspect, mer table entries for these overly repeated mer matches are placed into repeat mer tables 226-230. For example, mer table entries from overly repeated mer matches from sorted mer table 1 210 are placed into repeat mer table 1 226, and mer table entries from overly repeated mer matches from mer table 2 212 are placed into repeat mer table 2 228. Alternately, one repeat mer table holds mer table entries for all overly repeated mer matches.
The hit table sorting system 232 sorts the hit table entries of the hit table 218 to group together entries that include mers derived from the same sequence index 2 (second sequence index group) in the sequence set 2 204. The hit table sorting system 232 further sorts the hit table entries so that the hit table entries that include mers derived from the same sequence index 1 (first sequence index group) in sequence set 1 202 are placed together within each second sequence index group. The hit table sorting system 232 further sorts the hit table entries so that the hit table entries that include the same orientation index (orientation index group) are placed together within each first sequence index group. The hit table sorting system 232 further sorts the hit table entries so that the hit table entries that include the same frameshift (frameshift group) are placed together within each orientation index group. The hit table sorting system 232 further sorts the hit table entries so that the hit table entries that include the same position index 2 (position index group) are placed together within each frameshift group. After sorting the hit table entries of the hit table 218, the hit table sorting system 232 transfers the hit table entries into a sorted hit table 234.
The hit table condensing system 236 compares the entries of the sorted hit table 234 to identify repeated hit table entries, which are hit table entries having at least one other hit table entry in the same position index 2 group. The hit table condensing system 236 generates a condensed hit table entry for each set of repeated hit table entries that includes one instance of the repeated hit table entry and an index containing the number of repeated entries. The hit table condensing system 236 transfers all of the condensed hit table entries into a condensed hit table 238. In one aspect, the hit table condensing system 236 is modified and exists in a different form from the layout generation system 104. In another aspect, the hit table condensing system 236 is excluded from the layout generation system 104, and the post processing system 240 instead processes a sorted hit table 234, as explained below.
The layout generation system 104 also includes a post processing system 240 that processes the entries of the condensed hit table 238. In one aspect, the entries of the condensed hit table 238 are processed to assemble a sequence set 202 into a longer nucleotide sequence. Examples of longer nucleotide sequences include a gene, a chromosome, and a genome. Other examples exist. In another aspect, the entries of the condensed hit table 238 are processed to determine the expression of a particular gene product. The post processing system 240 may process the entries of the condensed hit table 238 using one or more post processing algorithms. The particular post processing algorithm used is selected based on the type of data contained within one or more sequence sets 202-206 being processed, as well as the desired information to be obtained from the layout assembly system 102A. In one example, if the sequence set 1 202 is a DNA sample from a patient, and the sequence set 2 204 is a set of nucleotide sequences from a previously assembled human genome, the desired result may be to generate an assembled genome for the patient by determining the most likely arrangement of the nucleotide sequences of the sequence set 1 onto the nucleotide sequences of sequence set 2 204.
Sequence Set
A sequence set is one or more sequence entries, where each sequence entry includes a sequence with one or more nucleotides and information identifying the sequence.
The sequence index 308 is a number assigned to each sequence 310 to uniquely identify each sequence 310. For example, as the sequence set 202A is divided into mers and further processed, the sequence index 308 may be included with the processed data to specify the location in the sequence set 202A from which the data was derived.
The sequence 310 is a string of one or more nucleotides. For example, the sequence 310 may be a read sequence or a sequence from a genetic database. In one example, the sequence 310 is encoded as a series of letters, for example ATGC, corresponding to the nucleotide bases of the sequence. In another example, the nucleotide bases are encoded as binary numbers using symmetric bit coding or symmetric fitch coding. In symmetric bit coding, A is encoded as 11, G is encoded as 10, C is encoded as 01, and T is encoded as 00. In symmetric fitch coding, A is encoded as 1000, G is encoded as 0100, C is encoded as 0010, and T is encoded as 0001. Other examples exist.
The sequence set index 312 is an integer assigned to each entry to identify the sequence set from which the sequence 310 originated. In an aspect, the sequence set index is included with data resulting from the processing of one or more sequences from a sequence set 202 to identify the sequence set 202 from which the data were derived.
All or a portion of the sequence entries 302-306 are alphanumeric data, integers, hexadecimal numbers, binary numbers, or another data encoding convention. Other examples exist.
Other forms and formats of the sequence sets 202-206 may be used, and the order of the indices 308-312 may be changed.
Mer Generating and Sorting System
The sequence processing system 402 reads in or otherwise receives each of the sequence entries of the sequence set 202B and divides each sequence into the mers. The sequence processing system 402 assigns one or more indices to each mer to identify the location within the sequence and sequence set 202B from which the mer was divided. The sequence processing system 402 generates a mer table entry for each mer, and each mer table entry includes the mer, the sequence index, a position index, a strand index, and the sequence set index. The sequence index and the sequence set index are the indices previously assigned to the sequence and sequence set from which the mer was divided. The position index is a number identifying a location in the originating sequence of a selected nucleotide in the mer. The strand index is a number that specifies whether the mer is derived from a forward originating sequence or a reverse originating sequence. In one aspect, a strand index of “1” indicates that the mer was derived from a forward nucleotide sequence, and a strand index of “2” indicates that the mer was derived from a reverse nucleotide sequence. Other numbers, flags, or indicators may be used. The sequence processing system 402 assembles each of the mer table entries into the mer table 404.
The amer/cmer generation system 406 removes the potentially confounding effect of the direction of the nucleotide strand from which the mers were derived. In one aspect, the nucleotide sequence of each mer is used to generate the reverse complement nucleotide sequence. The mer is then compared to its corresponding reverse complement, and one or the other or neither is retained according to one or more rules (identified below). In one aspect, the same rule is applied to all mer tables consistently. The mer or its reverse complement is then reassigned to the location indices to form a strand-independent mer table entry. The amer/cmer generation system assembles the one or more amer/cmer table entries derived from the entries of the mer table 404 into a strand-independent mer table 408.
The mer sorting system 410 sorts the entries of the strand-independent mer table by the mer nucleotide sequences to generate a sorted mer table 210A.
Sequence Preprocessing System
As discussed above, a mer is a shorter nucleotide sequence divided out of a longer sequence. The length of each mer is less than or equal to the length of the shortest sequence length among all of the sequences of all of the sequence sets 202-206 to be processed. In addition, the length of the mers may be based on other factors, including the estimated frequency of sequencing errors, the estimated frequency of single nucleotide polymorphisms (SNPs) in the sequences, and the desired processing time to assemble the sequence set into at least one longer nucleotide sequence. Other factors may exist. Shorter mers increase the likelihood of capturing regions of the sequences that do not contain sequencing errors or SNPs, but shorter mers also increase the total number of mers to be processed by the layout assembly system, resulting in a longer processing time. In addition, shorter mers may have a higher chance of introducing random matches that are not indicative of any homology. Thus, these factors are balanced by the layout assembly system. In one example, a mer is between about 2 and about 100 nucleotides long. In another example, a mer is about 19, 20, or 21 nucleotides long.
The sequence preprocessing system 402A reads in the first K nucleotides of a sequence from a sequence set, such as sequence set 1 202, into the nucleotide data register at step 502 and copies the nucleotide data to the mer at step 504. K is a number that specifies the length of the mers.
As shown at steps 506-512, the sequence preprocessing system 402A assigns the sequence index, a position index, a strand index, and a sequence set index to the mer. An entry for the mer table 404 is generated at step 514. The mer table entry includes the mer, the assigned sequence index, assigned position index, assigned strand index, and the assigned sequence set index. In one aspect, the sequence preprocessing system 402A assigns an initial value of 1 for the strand index, corresponding to a forward sequence. In this aspect, the final strand index is determined in the amer/cmer generation system 406.
The sequence preprocessing system 402A then determines whether less than K+1 nucleotides remain in the sequence at step 516. If there are at least K+1 nucleotides remaining in the sequence, the system deletes the first nucleotide from the nucleotide data register at step 518, then reads the next nucleotide from the sequence and appends the nucleotide to the end of the current nucleotide data at step 520. The nucleotide data assembled in step 520 is copied into the next mer table entry at step 504. The sequence preprocessing system 502A repeats steps 518-520 and 504-514 if K+1 or more nucleotides remain in the sequence at step 516. If less than K+1 nucleotides remain in the sequence, then too few nucleotides remain to form any additional mers, and the system advances to the next sequence in the set at step 522.
In one aspect, the sequence preprocessing system 402A creates mer table entries for all mers resulting from the mer generation process. In another aspect, the sequence preprocessing system 502A creates mer table entries for only a subset of mers that meet a specified inclusion criterion. In this aspect, the selection criterion is selected to reduce the total number of mers processed by the hit determination system 216 without significantly impacting the accuracy of the layout generation process.
In general, mers may include a subset of contiguous nucleotides divided out of a longer sequence, a subset of non-contiguous nucleotides divided out of a longer sequence based on a different pattern or rule, or a combination of contiguous and non-contiguous nucleotides. For example, a mer may be formed by assembling every other nucleotide in a sequence, every third nucleotide in a sequence, or every sequence following a palindromic sequence. Other examples exist.
In an exemplary aspect, mers may be formed by selecting a subset of contiguous nucleotides and replacing any instances of repeating contiguous nucleotides with a single nucleotide that matches the repeating group. For example, a sequence that includes nucleotide C followed by string of ten repeats of the nucleotide A, followed by the nucleotide G would be replaced by the nucleotide sequence CAG. Mers that are formed in this manner are defined as “flowmers” in the context of the application. That is, a flowmer is a subset of a sequence of nucleotides in which repeated nucleotides are removed, eliminated, or otherwise not included, leaving a single instance of the repeated nucleotides. Methods of generating flowmers from nucleotide sequences and assembling the sequences into flowmer tables are discussed in detail below.
An alternative sequence preprocessing system 402B for generating flowmers from a nucleotide sequence and for creating flowmer table entries is shown in
The first nucleotide of the sequence is read into the last (Kth) position in the flowmer register at step 524, and the Kth position of the coordinate register is set to a value of 1 at step 526. Each nucleotide in the flowmer register is then shifted over one position at step 528 by copying the Kth position nucleotide into the (K−1) position, the second position into the first position, and so on. Similarly, the coordinate register positions are shifted over one position at step 530 in a similar manner by copying the Kth position coordinate into the (K−1) position coordinate and so on.
The next nucleotide in the sequence is then read into the Kth position in the flowmer register at step 532. The position counter, tracking the nucleotide's position within the sequence is incremented by one at step 534, and this position counter is copied into the Kth coordinate register at step 536. The Kth nucleotide in the flowmer register is compared to the K−1 nucleotide in the register at step 538. If the two nucleotides are the same, then the next nucleotide is read at step 532, and the process repeats until a Kth nucleotide is read that is different from the K+1 nucleotide at step 538.
When a non-repeating nucleotide is detected at step 538, the flowmer registers are checked to see if all registers are full at step 540. If the registers are not yet full at step 540, then more nucleotides need to be read into the register before a full-length mer can be generated. Steps 528-538 are repeated until the flowmer registers are full at step 540, indicating that a complete flowmer sequence has been identified.
A sequence index is assigned to the flowmer sequence at step 542, and a position index is determined by subtracting 1 from the coordinate of coordinate register position 2 at step 544. A strand index and a sequence set index are assigned to the flowmer sequence at steps 546 and 548 respectively. A flowmer length is determined by subtracting the coordinate register in position K from position 1 at step 550, and a flowmer table entry is generated at step 552. If there are any remaining nucleotides to be read from the sequence at step 554, the registers of the flowmer and flowmer position registers are shifted in steps 528 and 530 respectively to start the process of generating the next flowmer. This process repeats until no more nucleotides remain to be read in the sequence.
Once the end of a sequence is detected at step 554, if there are additional sequences at step 556, the next sequence is read at step 558 and the process repeats for each sequence in the sequence set. If no more sequences remain at step 556, the system ends processing at step 560.
The flowmerlength is a quantity that specifies the length of the subset of the sequence from which the flowmer was generated by removing repeating nucleotides. Because the flowmerlength includes the nucleotides that were removed, the number of nucleotides in the flowmer may be less than the flowmerlength. In addition, although each flowmer has the same specified length of K nucleotides, the flowmerlength may vary due to the variable number of repeating nucleotides that may be removed in the process of generating flowmers. In one example, if a flowmer that contains 15 nucleotides is generated by removing 40 repeating nucleotides from a sequence, the flowmerlength would be 55 nucleotides (15+40). In the special case in which a sequence from which the flowmer was generated has no repeating nucleotides, the flowmerlength would be equal to the number of nucleotides in the flowmer.
Except where otherwise specified below, the flowmers may be processed using essentially the same processing methods and assembly techniques as those used for mers containing subsets of contiguous nucleotides from the sequence. In general, within any of the discussions below, the terms “mer” and “flowmer” may be used interchangeably, unless specifically stated otherwise.
Mer Thinning and Mer Minimization
The efficiency of the layout generation system 104 may be lowered when relatively large volumes of sequencing data are analyzed. Each nucleotide in the data is effectively considered in K different mers and C different hits due to the overlapping nature of mers, where K is the number of bases in a mer and C is the average coverage, defined in this context as the total number of nucleotides in all read sequences divided by the total number of nucleotides within the comparing sequence. However, if some reduction in the number of mers, and the corresponding matches between mers is allowed, some mers may be dropped in a systematic manner, resulting in a considerable gain in efficiency. Ideally, the vast majority of dropped mers may well be redundant due to the overlapping nature of mers.
In one aspect, mer thinning may be used to reduce the quantity of mers included as entries in a mer table 404 by simply skipping mers and only retaining every Nth mer only in the mer table entries. For a given value of N, where N is an integer specifying the mer thinning interval, an N-fold reduction in the number of mers to be included in the mer table entries is achieved. In one aspect, n is selected to be less than the length of any sequence in a sequence set. If N is less than the mer length K, then every nucleotide in each sequence of a sequence set is still included at least once in the reduced set of mers.
Mer thinning generally is not performed on both the comparing sequence set and the read sequence set, because the criterion of retaining every Nth mer generated from a sequence is a purely spatial selection criterion, in which the mers to be retained are selected based on their spatial position within a sequence. For example, retaining every Nth mer of each sequence in a sequence set imposes a very regular spatial pattern of mer retention on the sequences of the sequence set. However, the spatial positions between the read and comparing sequences may not be easily reconciled. In one aspect, the full set of comparing mers may be retained and the full set of read mers may be reduced using the mer thinning method, because typically the full set of read mers is much larger than the full set of comparing mers. The thinning interval n is selected to provide a subset of mers which incorporate the majority of nucleotides in all of the read sequences as completely as possible. Although mer thinning may render an inconsistent number of hits of different read sequences, each read sequence is internally consistent and therefore may be reasonably mapped on its own.
In another aspect, the quantity of mers included as entries in the mer tables may be reduced by a method of mer minimization. This method checks each mer against an internal pattern and eliminates those mers which do not match the pattern. In this aspect, the pattern used to select mers for inclusion in the mer tables may be a non-biased pattern. The non-biased pattern results in an approximately random selection of mers with respect to all sequences in the sequence set overall, and a sufficient number of mers per individual sequence to ensure the inclusion of the majority of nucleotides within each individual sequence during the layout assembly process. Non-limiting examples of a non-biased pattern include the selection of mers immediately following or containing an “A” nucleotide, a “C” nucleotide, a “T” nucleotide, or a “G” nucleotide; the selection of mers immediately following or containing a specified nucleotide sequence ranging between 2 and about 6 nucleotides in length; and the selection of mers immediately following a palindrome mer, in which the mer sequence is the same as the reverse complement sequence.
In another aspect, a non-biased pattern of bases may be derived from the analysis of a variety of genomes. A candidate pattern of mer minimization may be evaluated by assessing the amount of reduction in total mers using the candidate pattern. An approximately 4n-fold reduction of the number of mers may be achieved using a non-biased pattern, where ‘n’ is the number of bases included in the pattern.
Mer minimization methods may be applied to read mers, comparing mers, and to both read and comparing mers with minimal loss in the accuracy of the layout generation process, because the non-spatial mer selection criterion results in a consistent but random pattern of mers that is independent of the sequence from which the mers are derived. In yet another aspect, the non-biased pattern of bases used in the mer minimization method is selected to produce a sufficiently high frequency of occurrence of mers relative to the mer length and read length, resulting in a sufficient amount of mer data to accurately and quickly perform the mapping of read sequences to the comparing sequence.
In still another aspect, the number of mers in a mer table generated by the sequence preprocessing system 402A from a sequence set may be reduced by selecting one or more mertags, which are a subset of the set of all mers originating from a sequence set, based on a mertag selection criterion. Reducing the entries of a mer table reduces the length of time to assemble a sequence set by reducing the number of pairwise comparisons made during the mer matching process, reduces the number of hits in the hit table to be sorted and condensed, and reduces the number of condensed hit table entries to be processed by the post processing system. In this aspect, the mer table is processed to eliminate all mer table entries that do not include one of the mertags. In an embodiment, about 2 or 3 mertags are present in each sequence of a sequence set, the mertags are randomly distributed throughout the sequences of the sequence set, and the mertags are repeated between about 2 and 500 times within all of the sequences of a sequence set. Though, more than 3 mertags may exist.
The mertags may be selected by choosing every nth mer in a mer table, where n may range between 1 and about 100. In another aspect, the mertags may be selected by choosing every mer immediately following or containing an “A” nucleotide, a “C” nucleotide, a “T” nucleotide, or a “G” nucleotide in a mer table. In another aspect, the mertags may be selected by choosing every mer immediately following or containing a specified nucleotide sequence ranging between 2 and about 6 nucleotides in length. In another aspect, the mertags may be selected by choosing every mer immediately following a palindrome mer, in which the mer sequence is the same as the reverse complement sequence.
A list of potential mertags may be selected by choosing 3 mers at random from each sequence of the sequence set. The mertags are then selected by choosing all mers in the mer table that match an entry from the list of potential mertags. In addition, after all mertags have been selected, the separation of the mertags on each sequence of the sequence set (a mertag gap) may be determined.
If a mertag gap is detected, such as when the separation of two consecutive mertags on a sequence exceeds a threshold mertag separation, then additional mertags may be selected to fill in each mertag gap. In this aspect, the threshold mertag separation is a distance between the end of a mertag and the beginning of the next consecutive mertag on a sequence. For example, the threshold mertag separation may range between about one mer length and about 10 mer lengths. Also in this aspect, the mertags selected to fill in the mertag gap on all of the sequences of the sequence set may be selected by inspection by a user or selected by applying another rule, such as any one of the mertag selection rules described above. Once the additional mertags have been selected to fill in all mertag gaps detected in all sequences of the sequence set, all mer table entries from the complete mer table with mers matching the additional mertags may be added to the mertag table. Other examples exist.
In one aspect, the mertags are generated during the mer generation process. In another aspect, the mertags are selected from the mer tables generated by the sequence preprocessing system 502A.
Mer Table
Other forms and formats of the mer table 404A or the flowmer table 404A may be used, and the order of the indices may be changed. Other index types, numbers, or designations may be used.
Mer Generation Example
In an aspect, this convention of trimming any leading and trailing repeated nucleotides during flowmer generation is followed in order to simplify the subsequent processing of the flowmers, in particular, in aligning sequences of opposite direction (one sequence forward-directed and one sequence reverse-directed) based on matching flowmers, as will be discussed further below. In another aspect, the repeating leading nucleotides and repeated trailing nucleotides may be both retained in the process of generating a flowmer from a sequence. For example, if this convention was used for flowmer 704A, the first nucleotide would be the first repeated A from position 1, and the last nucleotide would be the last G of the repeating group from position 20. As a result, the flowmerlength would be 20, rather than 13 as shown in
The second flowmer 706A includes all but the first nucleotide of flowmer 704A. In addition, the second flowmer 706A includes a C from position 21. The leading C is from a repeated group. Therefore, the C from position 7 is used in this flowmer. This change in position is because the position of the leading nucleotide is calculated by subtracting 1 from the position of the second nucleotide in the flowmer, as described in step 544 of
The third flowmer 708A includes all of the nucleotides of the second flowmer 706A except that the G from position 22 is appended onto the end of the flowmer. The fourth flowmer 710A includes all of the nucleotides of the previous flowmer 708A, with the first T of the repeated group from position 23 appended to the end. This process of removing the leading nucleotide and adding the next nucleotide of the sequence 702A to the end of the previous flowmer is repeated to generate the subsequent flowmers until the final nucleotide in the sequence 702A has been added. Each sequence from each of the sequence sets 202-206 is divided into flowmers using the same process.
AMER/CMER Generation System
The sequences that are analyzed by the layout assembly system 102 may be read sequences resulting from the analysis of a genetic sample by an automated sequencing device. In some cases, such as the sequencing of double-stranded DNA, the read sequences may result from the analysis of either a forward DNA strand or a reverse DNA strand. However, the read sequences do not usually identify the direction of the DNA strand from which it was read. Because a sequence set may be a mixture of forward and reverse sequences, the sequences may be further processed to remove the effect of the direction of the sequences. The amer/cmer generation system 406 reads in the mers from the mer table 404, processes the mers, and generates a strand-independent mer table 408.
The effect of strand direction is removed from each mer by generating a reverse complement sequence for the mer. The reverse complement sequence is formed by assembling all of the complementary nucleotides corresponding to each nucleotide in the mer, and reversing the order of the complementary nucleotides. The mer sequence and the complement sequence are compared, and either the mer or its reverse complement is used to generate the strand-independent mer table entry based on a mer sequence selection rule. In one example of a mer selection rule, the mer or reverse complement that is the lowest alphabetic sequence is used except if the mer is alphabetically equal to its reverse complement (a palindrome), in which case the mer is excluded from the mer table. In another example of a mer selection rule, the mer or reverse complement that is the highest alphabetic sequence is used except if the mer is alphabetically equal to its reverse complement (a palindrome), in which case the mer is excluded from the mer table. In one embodiment, mers of an odd-number sequence length are used to avoid palindromes. Other examples exist. By applying this corrective process, mers resulting from forward sequences and mers resulting from reverse sequences are compared on a consistent basis.
In another aspect, the amer or cmer generation system may also be used to process flowmers in order to generate a table of strand-independent flowmer tables.
Read AMER Generation System
In an aspect, the comparison of the mer nucleotide may be performed on the basis of the letters ATCG used to represent the nucleotide bases, and the mer is determined to be less than the reverse complement if the mer sequence alphabetically precedes the reverse complement sequence. In another aspect, the reverse complement may be substituted for the mer when the mer is alphabetically after the reverse complement.
In another aspect, the nucleotides may be encoded using numbers, such as the symmetric bit coding, in which A is encoded as 11, G is encoded as 10, C is encoded as 01 and T is encoded as 00. In this aspect, the mer is determined to be less than the reverse complement when the binary representation of the mer is determined to be less than the binary representation of the reverse complement. Other representation of the nucleotide sequences of the mer and its reverse complement may be used, and other rules for comparing the sequences may be used, so long as any rules for comparing the sequences are applied consistently to all nucleotides within all sequence sets processed by the AMER generation system 406A.
Read CMER Generation System
In an aspect, the length of the mer is an odd number, so that the center nucleotide of the mer is a discrete single nucleotide. In another aspect, in the case of a mer with an even number of nucleotides, a pair of central nucleotides immediately adjacent to the center of the nucleotide strand is assessed using a mer loading rule to determine whether to load the mer or the reverse complement of the mer into the cmer register. In one example of a mer loading rule, the pair of central nucleotides is compared to the corresponding reverse complement of the pair of central nucleotides. The mer is loaded into the cmer if the pair of central nucleotides alphabetically follows the corresponding reverse complement of the pair of central nucleotides. For example, the central nucleotides GT alphabetically follow the reverse complement of the central nucleotides AC. The reverse complement is loaded into the cmer if the reverse complement of the central nucleotide pair alphabetically follows the pair of central nucleotides. If the pair of central nucleotides is the same as their reverse complement (a palindrome), then the next pair of nucleotides out from the center of the mer sequence is compared to its corresponding reverse complement. The mer is loaded into the cmer if the next pair of nucleotides is alphabetically higher than the corresponding reverse complement of the next pair of nucleotides. The reverse complement is loaded into the cmer if the reverse complement of the next nucleotide pair alphabetically follows the next nucleotide pair. If the next pair of nucleotides is a palindrome, then each successive pair of nucleotides out from the center of the mer sequence is compared until either the mer or reverse complement is loaded into the cmer, or the entire mer sequence is found to be a palindrome, in which case the mer is eliminated from the mer table. In another embodiment, the mer table entries corresponding to palindrome mers are moved to a palindrome mer table. Other rules may be used.
Amer Generation Example
For example, the mer entry 612A (AAAAGGGGG) precedes its reverse complement 1012 (CCCCCTTTT), so mer entry 612A appears as the sixth entry in the strand-independent mer table 408.
Strand-Independent Mer Table
Other forms and formats of the strand-independent mer table 408A may be used, and the order of the indices may be changed. Strand-independent flowmer tables may be generated using the methods described above in another aspect. Other index types, numbers, or designations may be used.
Mer Sorting System/Sorted Mer Table
The mer sorting system 410 sorts the entries of the strand-independent mer table 408B by the nucleotide sequences of the mers and generates a sorted mer table 210.
Other forms and formats of the strand-independent mer table 408A may be used, and the order of the indices may be changed. In one example, one strand-independent mer table may include entries for each of the sequence sets 202-206. Other index types, numbers, or designations may be used.
As a result of the sorting, the table entries containing mers 1102A having the same nucleotide sequence are grouped together in each of the sorted mer tables 404. This ordered arrangement of the sorted mer table entries facilitates the subsequent pairwise comparison of table entries of the two or more mer tables 404 by the hit determination system 216.
Hit Determination System
Mer Matching/Filtering System
If the current mer2 is different than the previous mer2, a comparison of the group of mer2 nucleotide sequences to each mer1 from sorted mer table 1 210 is performed. At step 1408, a mer1 is read from the sorted mer table 210. Mer2 is compared to mer1 at step 1410.
If the current mer1 is less than the current mer2, then any mer1 that matches the current mer2 is located after the current mer1. In this case, the mer table entry containing the current mer 1 is transferred to the orphan mer table 1 220 at step 1412, and the next entry of sorted mer table 1 210 is read.
If the current mer1 matches mer2, a hit table entry is recorded for the matching comparing mer2/read mer1 pair. For example, if there are six mer2's in the current mer2 group that match the current mer1, a total of six hit table entries are recorded corresponding to the matches of the current mer1 to each individual mer2 in the current group. In addition, the number of mer1 nucleotide sequences that produce matches with the mer2 sequences is tracked by incrementing a mer1 count (read mer count) at step 1416 each time a match occurs.
If mer1 is greater than mer2, then any matching mer1 has been identified previously. In this case, the system 1302A checks the mer1 count at step 1418. If the mer1 count is 1 or less, no matches were determined for the current mer2 group, and the mer table entries containing the comparing mer2 sequence in the current comparing mer 2 group are written to the orphan mer table 2 222.
If the mer1 count is greater than one, then at least one match was detected for the current mer2 group. In this case, the total number of matches between the current mer2 group and the mer1's is determined by multiplying the current mer 2 count by the current mer 1 count at step 1424. If the product exceeds a repeated match threshold, then the current group of matching mer2 and mer1 table entries saved at step 1416 are transferred to a repeat mer table 2 228 and a repeat mer table 1 226, respectively.
In one aspect, the repeated match threshold is a value equal to the product of the total number of sequences in sequence set 1 times the total number of sequences in sequence set 2 204. In another aspect, the repeated mer threshold may range between about 10 and about 100,000,000 mer sequence comparisons (hits). The repeated match threshold is configurable in some aspects.
If the total number of matches in the current group is less than the repeated match threshold for each data set, then the group of mer table entries saved in step 1416 is passed to the frameshift calculation system at step 1428.
In yet another aspect, repeated mers may be identified by counting the number of mers in each of the sorted mer tables 210-212 that have identical mer nucleotide sequences. If the number of repeated mers in a mer table exceeds the repeated mer threshold, then the sorted mer table entries having the repeated mer sequence are transferred to one of the corresponding repeat mer tables 226-230. The counting of mers may be performed by the mer matching/filtering system 1302, the mer sorting system 410, in a separate mer counting system, or in other systems of the layout assembly system 102.
In yet another aspect, one sorted mer table contains entries for each of the sequence sets 202-206. In this aspect, those entries in a group of matching mers having a sequence set index 312 corresponding to one sequence set are added to the mer2 group, and those entries having a sequence index 213 corresponding to another sequence set are compared to the entries in the mer2 group.
The value of the repeated mer threshold may depend on the expected frequency of repeated nucleotide sequences in the sequence sets and the length of the mers. In another aspect, the layout assembly system 702 uses a larger repeated mer threshold for a sequence set in which the expected frequency of repeated nucleotide sequences is high, and a smaller repeated mer threshold when the expected frequency is low. In another aspect, the repeated mer threshold ranges between about 2 mer repeats and about 1000 mer repeats. In yet another aspect, the repeated mer threshold is about 20 mer repeats.
Example of Mer Matching
The sorted mer table entry 1512 of the second sorted mer table 210′ matches the sorted mer table entries 1204 and 1206 of the first sorted mer table 210′. So, two hit table entries are generated by the mer matching/filtering system 1302. Two hit table entries are also generated for the matched sorted mer table entries 1208/1514 and 1210/1516. Because entry 1220 matches entry 1520 and entry 1522, two additional hit table entries are generated by the mer matching/filtering system 1302. Once all of the matches have been determined by the mer matching/filtering system 1302, the hit table entries are passed to the frameshift calculation system 1304.
Frameshift Calculation System
For each of the matches, the strand index 1, alternately referred to as the read strand index, from the mer1 table entry is compared to the strand index 2, alternatively referred to as the comparing strand index, from the mer2 table entry at step 1604.
If the strand indices have the same value, then mer2 and mer1 are oriented in the same direction. In this case, the frameshift is determined by calculating the difference between position index 2 and position index 1 at step 1608. In one example, position index 1 may be a read mer position index, and position index 2 may be a comparing mer position index. An orientation index of 1 is assigned to a hit table entry corresponding to the match to specify that mer2 and mer1 in the match were aligned in the same direction at step 1610. The hit table entry is passed to the hit table generation system 1306A at step 1612.
If strand index 1 is not equal to strand index 2, then mer2 and mer1 are oriented in opposite directions. In this case, the frameshift is determined by obtaining the sum of position index 2, position index 1, and the mer length−1 at step 1614. An orientation index value of 2 is assigned to the mer2/mer1 pair to specify that the mer sequences are oppositely directed at step 1616, and a hit table entry is passed to the hit table generation system at step 1618.
In another aspect, the methods of hit determination described above may be used for flowmer tables without change. However, due to the nature of flowmers, in which only a single instance of a string of repeating nucleotides in the sequence is retained during the process of forming the flowmers, a modification of the method of determining frameshifts for flowmer matches is used.
Hit Table Generation System
Hit Table
The hit table 218 is a summary of all matches between the mers from sorted mer table 1 210 and the mers from sorted mer table 2 212. An exemplary hit table is shown in
Hit Table Sorting System
The entries of the hit table 218 are sorted by sequence index 2 at step 1902, which arranges the hit table entries into groups (sequence index 2 groups) having the same sequence index 2. Within each sequence index 2 group, the hit table entries are further sorted by sequence index 1 at step 1904, forming groups of hit tables having same sequence index 1 (sequence index 1 group). Within each sequence 1 group, the hit table entries are further sorted by orientation index at step 1906, forming groups of hit table entries (orientation index groups) having the same orientation index value. Within each orientation index group, the hit table entries are further sorted by frameshift at step 1908, forming groups of hit table entries (frameshift groups) having the same frameshift value. Within each frameshift group, the hit table entries are further sorted by position index 2 at step 1910, forming groups of hit table entries (position index groups) having the same value of position index.
The resulting sorted hit table 234 contains a series of hit table entries that are arranged into smaller groups containing the hit table entries corresponding to all matches between mers derived from one sequence of sequence set 2, and mers derived from one or more sequences of sequence set 1. Within each of these groups, the entries corresponding to matches between mers derived from one sequence 2, such as one comparison sequence, and one sequence 1, such as one read sequence, are grouped together. Within each group of entries corresponding to matches between one sequence 2, such as the comparing sequence, and one sequence 1, such as the read sequence, the entries are further arranged into groups of matches between mers oriented in the same direction and mers with opposite orientation. The sorted hit table 234A is passed to the hit table condensing system 240.
Different sorting algorithms may be used to implement the sorting functions of the hit table sorting system 232A. Examples of sorting algorithms include bubble sort, cocktail sort, comb sort, gnome sort, selection sort, insertion sort, shell sort, binary tree sort, library sort, merge sort, in-place merge sort, heapsort, smoothsort, quicksort, introsort, patience sorting, strand sort, pigeonhole sort, bucket sort, counting sort, LSD Radix sort, MSD Radix sort, and spreadsort. Other examples exist.
Sorted Hit Table
An exemplary sorted hit table 234A is shown in
Hit Table Condensing System
The hit table condensing system 236 processes the sorted hit table 234 to identify and combine any multiple hits between each sequence of sequence set 1 202 (read sequence) and each sequence of sequence set 2 204 (comparing sequence). For each hit identified as having duplicates, the number of duplicates is recorded as a score, defined in this context as the number of duplicate hits. The score is an approximation of the number of bases from a given sequence from sequence set 1 202 that forms a hit at a particular location in sequence set 2.
If the score is not equal to zero at step 2106, various indices from the current sorted hit table entry are compared to the corresponding indices from the previous sorted hit table entry. The current sequence index 1 2014 is compared to the previous sequence index 1 at step 2110. If the sequence index 1 values match, the current sequence index 2 2016 is compared to the previous sequence index 2 at step 2112. If the sequence index 2 values match, the current frameshift 2030 and the previous frameshift are compared at step 2114.
In step 2114, the absolute value of the difference between the current frameshift and the previous frameshift are compared to a frameshift tolerance. The frameshift tolerance, in this context, is the acceptable variation of frameshift due to error sources, such as SNPs or sequencing errors. In one aspect, the frameshift tolerance, expressed as a number of nucleotide bases, may be any value from zero up to about 10 nucleotides. If the incremental change between the current frameshift and the previous frameshift falls within the frameshift tolerance at step 2114, the position index 1 2022 is evaluated at step 2116.
As described previously, position index 2 2024 specifies the position of the first nucleotide of the sequence used to generate a mer2 relative to the first nucleotide of a comparing sequence from sequence set 2 204 when the mer2 is aligned with the mer1. If the current and previous mer2s are located too far apart on the sequence from sequence set 1 2014, a gap may exist between the two mers that may confound the layout assembly process. If the incremental difference between the current and previous position index 2 falls within a gap threshold, then the current hit table entry is added to the current condensed hit table entry group by deleting the current hit table entry at step 2118 and increasing the score for the current hit table entry by 1 at step 2108. A new comparison is then started by reading a new current hit table entry at step 2104.
In an aspect, the gap threshold may vary from 0 to about 10% of the length of the sequence from sequence set 2. In another aspect, the gap threshold may vary from 0 to about 1000 nucleotides. In an exemplary aspect, the gap threshold may be 10 nucleotides.
If the current hit table entry does not match the previous hit table at any one of steps 2110, 2112, 2114, or 2116, the current hit table entry is set as the first hit table entry of a new group. The current score is assigned to the previous sorted hit table entry at step 2120, and an entry is written to the condensed hit table that includes the previous sorted hit table entry as well as the score at step 2122. After resetting the score to a value of 0 at step 2102, the next hit table entry is read from the sorted hit table 234 at step 2104.
In one aspect, the resulting condensed hit table 238 contains at least one condensed hit table entry that includes all of the indices of the sorted hit table entry, plus a score that specifies how many times the hit table entry was repeated in the sorted hit table 234A. The number of repeats of a hit table entry may be used as an indication of the strength or confidence in the nucleotide match specified in the hit table entry. In another aspect, each condensed hit table entry may contain other descriptive statistics such as a coverage score, defined as the total number of nucleotides from sequence set 1 that were matched to generate the condensed hit table entry. The condensed hit table 238A is passed to the post processing system 240 for further processing and analysis.
Condensed Hit Table
Post Processing System
The templated assembly system 2304 processes the condensed hit table entries to produce an initial assembly of a first sequence set 202 using a second sequence set 204 as a template. In this system, the first sequence set 202 may be a set of reads sequenced from a DNA sample, and the second sequence set 204 may be a set of chromosome sequences from a genetic database. Other examples exist.
The de novo assembly system processes the condensed hit table entries to produce an initial assembly of a first sequence set 202 using a copy of the first sequence set 202 as the template for assembly. In this system, the first sequence set 202 may be a set of reads sequenced from an organism for which no previous sequence sets exist. In this case, the first sequence set 202 is assembled using a copy of the first sequence set 202 as the template. In one aspect, the second sequence set 204 is the copy of the first sequence set 202. Other examples exist.
The quantitative sequence assembly system 2308 processes the condensed hit table entries to produce a summary of the number of matches between a second sequence set 204 and a first sequence set 202 in which the summary is organized by location in the second sequence set 204. In this system, the first sequence set 202 may be a set of sequenced mRNA or DNA fragments from a patient, and the second sequence set 204 may be a set of gene sequences associated with a genetic product of interest. For example, a genetic product of interest may be a gene that is upregulated in response to a disease state in a patient. Other examples exist.
The pairwise genome alignment system 2310 processes the sorted hit table entries to produce an initial alignment of an organism's genome using an existing sequenced genome as a template. In this system, the first sequence set 202 may be a set of reads from the entire genome of an organism, and the second sequence set 204 may be a sequenced genome from a genetic database. Because of the long sequence length of the genome sequences, the incidence of random sequence errors due to errors in obtaining the reads from the genome, mutations, and other error sources may be higher than for shorter sequence lengths. The pairwise genome alignment accounts for these errors by skipping past regions that appear to harbor potential mutations or other errors that might otherwise confound the assembly process. The pairwise genome alignment system 2310 processes the sorted hit table 234, not the condensed hit table 238. In one embodiment of the pairwise genome alignment system 2310, the hit table condensing system 236 and the condensed hit table 238 are not used or are not present.
Templated Assembly System
In one aspect, the coverage of each condensed hit table entry is evaluated and all entries not meeting a minimum coverage threshold may be removed prior to processing the condensed hit table 238 by the templated assembly system 2304A. In one aspect, any condensed hit table entries with a score falling under a minimum threshold score are deleted. In another aspect, any condensed hit table entries with a coverage score that is less than a minimum coverage threshold are deleted prior to processing by the templated assembly system 2304A. In this aspect, the minimum coverage threshold may be specified as about twenty nucleotide bases or about fifty percent of the length of the read sequence from sequence set 1 202, whichever is greater. Because each read sequence may have a different length, the minimum coverage threshold may be calculated for each read sequence. Once the condensed hit table entries having insufficient coverage have been removed, processing by the templated assembly system 2304A proceeds.
The first entry of the condensed hit table is read at step 2406 and is written to the output file at step 2408. The next condensed hit table entry is read at step 2410, and the current sequence index 1 is compared to the previous sequence index 1 of the previous condensed hit table entry at step 2412. If the current sequence index is the same as the previous sequence index 1, the current condensed hit table entry is from the same group as the previous condensed hit table entry, and therefore has a lower score. In this case, a new condensed hit table entry is read at step 2410.
If the current sequence index 1 is not the same as the previous sequence index 1 at step 2412, then the current condensed hit table entry is the first of a new group having the same sequence index 1, and which also has the highest score of the group. So, the current condensed hit table entry is written to the output file at step 2414.
The resulting output file is a list of all of the condensed hit file entries having the highest score among all of the other entries having the same sequence index. This list represents an initial assembly of sequence set 1, since each entry of the output file specifies the alignment of each sequence in sequence set 1 that resulted in the highest number of mer matches between sequence set 1 and sequence set 2.
De Novo Assembly System
In one aspect, the de novo assembly system 2306 processes a condensed hit table resulting from the matching of a first sequence set with a second sequence set, in which the second sequence set is a copy of the first sequence set. The de novo assembly system is used when assembling a sequence set for which no template exists.
The output file generated by the de novo assembly system 2306A is the concentrated hit table with the hits of sequences against their own copies removed. This output file provides a summary of the matches between each strand of sequence set 1 against all of the other strands of sequence set 1.
In another aspect, a cluster-based de novo assembly system 4102, shown in
The cluster-based de novo assembly system 4102 achieves reduced processing times relative to the previously described de novo assembly system 2306 (see
An exemplary cluster-based de novo assembly system 4102 is illustrated in
An exemplary cluster generation system 4106A, illustrated in
The minimum mer threshold and maximum mer threshold may be specified based on the estimated coverage of the genome by the sequenced reads. In this context, the estimated coverage is calculated by dividing the total number of nucleotides in all sequenced reads in the read set by the estimated number of nucleotides in the genome. In one aspect, the minimum mer threshold may be set at about 20% of the estimated genome coverage of the sequenced read set. In another embodiment, the maximum mer threshold may be set at about 200% of the estimated genome coverage.
If the number of mers within the set of identical mers falls between the minimum mer threshold and the maximum mer threshold at step 4204, the set of identical mers is further processed. If the set of identical mers is the first set processed, none of the mers have been assigned to a cluster, and the group is formed into a cluster by the new cluster formation system 4210. If the group of identical mer sequences is not the first group to be processed, the group of identical mer sequences is processed further using the cluster affinity assessment system 4212 to determine whether the group of mers should be merged with an existing group, formed into a new cluster, or whether existing clusters should be merged based on the degree of overlap by any previously unassigned sequences within the group of identical mers.
An exemplary new cluster formation system 4210A is illustrated in
A cluster is formed by specifying a cluster ID for the new cluster at step 4302, setting the parent cluster ID to same value as the cluster ID at step 4304, and writing a cluster table entry that includes the cluster ID and the parent cluster ID at step 4306. Each cluster table entry, which documents a single cluster, is typically of the form (cluster ID, parent cluster ID). The cluster ID documents the original cluster to which each mer group was assigned, and the parent cluster ID documents any clusters with which the original cluster was merged during the processing of subsequent mer groups. For example, if a mer group is the fifth cluster group to be processed by the system and did not merge with any existing clusters at the time the mer group was processed, then a cluster table entry of (5,5) would be appended to the cluster table. As another example, if the fifth cluster was later merged with the 3rd cluster, the parent cluster ID of this cluster's table entry would be changed from 5 to 3 to indicate that the 5th cluster was merged with the 3rd cluster, resulting in a revised cluster table entry of (5,3). By one convention, the cluster that was formed earliest (i.e. the cluster that has the lowest cluster ID) is the parent cluster with which the later-formed cluster is merged. Other conventions may also be used in other aspects. In other embodiments, the cluster table entries may include additional statistical information about the clusters such as the estimated number of read sequences in the cluster and the number of mers that the sequences of the cluster have in common.
In addition to the cluster table entries, a microread table containing the read sequence and the cluster to which the read sequence is assigned is generated to further document the clusters. Referring back to
An exemplary cluster affinity assessment system 4212A is illustrated in
Because the existing clusters may not be arranged in order of cluster size, some ambiguity exists in comparing the degree of overlap of the mer group processed by the cluster merge assignment system 4410A with the first and second existing clusters. To simplify the analysis process, in one aspect NCLUSTER_1 and NCLUSTER_2 are assigned such that NCLUSTER_1>NCLUSTER_2 at step 4502. Using this convention, the mer group processed by the cluster merge assignment system 4410A has more sequences in common with the first cluster than with the second cluster.
At step 4504, NCLUSTER_1 is compared to a cluster threshold. The cluster threshold is a minimum number of sequences in a mer group under consideration that must be previously assigned to an existing cluster to support adding any unassigned sequences in the mer group to that existing cluster. In one aspect, the cluster threshold may be less than about one half of the coverage. In another aspect, the cluster threshold may range from about 2 to about 4. In yet another aspect, the cluster threshold may be about 2. If NCLUSTER_1 is less than the cluster threshold, than none of the previous clusters possess sufficient sequence overlap to support any merging of sequences, and the unassigned sequences in the mer set are processed by the new cluster formation system 4506 to form a new cluster if there are sufficient numbers of unassigned sequences in the mer group.
If NCLUSTER_1 is greater than the cluster threshold at step 4504, NCLUSTER_2 is compared to the cluster threshold at step 4508. If NCLUSTER_2 is less than the cluster threshold, then the unassigned sequences in the mer group are added to cluster 1. The cluster ID of all unassigned sequences is set to CLUSTER ID_1 (the index of cluster 1) at step 4510, and a microread table entry containing (SEQUENCE ID, CLUSTER ID) is created in the microread table for each unassigned sequence in the mer group at step 4512. Because no new cluster is formed, no new entries to the cluster table are created.
If NCLUSTER_2 is greater than the cluster threshold, then both the first and the second clusters have sufficient numbers of assigned sequences in the mer set to support merging the unassigned sequences, and therefore the merging of the first and second clusters is supported. The unassigned sequences are assigned the CLUSTER ID_1 corresponding to the first cluster at step 4514, and the first and second clusters are merged by the cluster-cluster merge system 4516.
An exemplary cluster-cluster merge system 4524A is illustrated in
Once all mer groups from the sorted mer table 210E have been processed in the manner described above, the cluster table will be analyzed to determine whether each of the clusters is a candidate for contig creation. In this context, a contig is a set of overlapping DNA fragments used in a sequence assembly process to assemble the DNA sequence of a larger DNA strand that may include a gene, a chromosome, or a genome.
If a cluster chain satisfies the criteria to qualify as a potentially useful contig, the read sequences that constitute the cluster chain will be gathered together and submitted to a DNA Assembler device. In one aspect, a cluster chain is qualified as a contig if the cluster chain includes at least about 100 read sequences. In another aspect, a cluster chain is qualified as a contig if the cluster includes between about 100 and about 1,000,000 read sequences. In yet another aspect, the qualification of a cluster chain as a contig is determined based on an estimated contig length to be created by the cluster chain, defined as (number of read sequences in cluster chain * average length of a read sequence)/(coverage). In this aspect, if the estimated contig length is greater than about 2000 bases, the cluster chain is qualified for inclusion as a contig. Alternatively in this aspect, if the estimated contig length is from about 2000 bases to about 1,000,000 bases, the cluster chain is qualified for inclusion as a contig.
Quantitative Sequence Assembly System
The quantitative sequence assembly system 2308A includes an initial match count system 2602. The initial match count system 2602 sorts the condensed hit table by sequence index 1, and then sorts each group of entries having the same sequence 1 by sequence 2.
For each group of condensed hits for a particular sequence from sequence set index 1, the initial match count system 2602 eliminates all condensed hit table entries in the group that have scores less than a minimum threshold score and retains only the entry or entries having the maximum score of the group by eliminating all hit table entries having less than the maximum score within the group. If a single condensed hit table entry remains after the elimination process described above, a match counter assigned to the particular sequence from sequence set 2 is increased in value. If two or more entries remain after the elimination process described above, the condensed hit table entries are written to a repeat assignments table for subsequent analysis by the fractional count dividend system 2604.
The fractional count dividend system 2604 determines the contribution of the other entries of the condensed hit table that did not meet the criterion described above. The score of each sequence of the second sequence set is increased by an amount that is proportional to the frequency of matches between each of the sequences of the second sequence set and the first sequence set.
The number of entries remaining in the group is assessed at step 2712. If more than one entry remains in the group, then the remaining entries are written to a repeat assignments table at step 2714. If only one entry remains in the group, a score of 1 is added to a match counter assigned to the sequence index 2 indicated by the remaining entry at step 2716. The match counter is a number that tracks the frequency of matches between the sequences of sequence set 1 and each of the sequences of sequence set 2. One match counter is assigned to each sequence of sequence set 2, and the match counters are stored in an assignment count table. The initial match count system 2602A advances to the next sequence index 1 at step 2718 after step 2716 or if no entries remain in the group at step 2712.
In the course of processing the condensed hit table, the initial match count system generates two tables: an assignment count table and a repeat assignments table. The assignment count table contains a list of match counters, and each match counter specifies the number of times of the sequences from sequence set 1 produced mer matches with each of the sequences from sequence set 2. Each match counter in the assignment count table corresponds to one of the sequences from sequence set 2. The repeat assignments table stores the condensed hit table entries that specified matches in which a sequence from sequence set 1 matched two or more sequences from sequence set 2 with equal frequency.
The output file of the quantitative sequence assembly system is the assignment count table that contains an entry for each sequence index 2 of sequence set 2. Each assignment count table entry includes a sequence index 2 and the match count determined by the quantitative sequence assembly system 2308A for that sequence. The match count for each sequence index is used as a signal to quantify the coverage or expression of each sequence of sequence set 2. In one aspect, the match count for a given sequence is a raw signal for the coverage or expression of the sequence. In other aspects, the match count may be used to determine other normalized signal quantities that are well-known in the art.
In one aspect, the hit count for each target sequence in sequence set 2 may be used to express a signal as reads mapped per kilobase of target sequence (RPK), as shown in equation 1 below:
RPK=match_count/(target_sequence_length/1,000) (1)
In one aspect, the hit count may be used to express a signal as reads mapped per million (RPM), as shown in equation 2 below:
RPM=match_count/(n_reads/1,000,000) (2)
In another aspect, the hit count may be used to express a signal as reads mapped per kilobase of target sequence, per million total mapped reads (RPKM), as shown in equation 3 below:
RPKM=match_count/(target_sequence_length/1000)(n_reads/1,000,000) (3)
Pairwise Genome Alignment System
If the current frameshift falls within the maximum frameshift variation at step 2910, the current position index 2 is compared with the previous position index 2 at step 2916. If the current frameshift does not fall within the maximum frameshift variation at step 2910, the current score is compared to a score threshold at step 2912. For example, the maximum frameshift variation is the amount of allowable variation in the frameshifts of two consecutive hit table entries due to minor variations in the sequences being compared. In an embodiment, the maximum frameshift variation may range between 0 and about 10% of the length of the sequences being compared. In another embodiment, the maximum frameshift variation may range between 0 and about 1000 nucleotides.
If the current position index 2 falls within the gap threshold at step 2916, then the score, which specifies the number of duplicate hit table entries, is increased by one at step 2918, and the next sorted hit table entry is read at step 2906. If the current position index 2 falls outside the gap threshold at step 2916, then the current score is compared to a score threshold at step 2912.
The gap threshold is a maximum allowable separation distance between two consecutive hits within a sequence from sequence set 2 in one condensed hit table entry. In one embodiment, the gap threshold may vary from 0 to about 10% of the length of the sequence from sequence set 2. In another embodiment, the gap threshold may vary from 0 to about 1000 nucleotides.
The score threshold specifies a minimum number of matches that a sequence from sequence set 1 must have with a sequence from sequence set 2 to be included as a condensed hit table entry. In one embodiment, the score threshold may range between 0 and about 100,000 matches.
If the current score is greater than a score threshold at step 2912, then the current hit table entry and current score are used to generate a condensed hit table entry, which is written to the output file at step 2920. The condensed hit table entry includes all of the indices of the current hit table entry, plus the score, which specifies the number of duplicate entries for the hit table entry. After the condensed hit table entry is written to the output file, the score is reset to zero at step 2914. If the current score is less than the score threshold, then the score is reset to zero at step 2914, no condensed hit table entry is written to the output file for the current hit table entry, and the next hit table entry is read from the resorted hit table at step 2906.
The output file generated by the pairwise genome alignment system 2310A is a condensed hit table with a small amount of tolerance introduced to the assessment of matching frameshift and a minimum score requirement for the inclusion of all condensed hit table entries. The condensed hit table specifies the regions of homology between the genome contained in the first sequence set and the genome contained in the second sequence set.
LUT-Based Layout Generation System
Various aspects of the layout generation system 104 described previously utilize the comparison of lists of mers and associated information that are resident on nonvolatile media such as a computer hard disk to match the nucleotide sequences of one or more sequence sets. To reduce processing time, the lists of mers may be sorted in order to organize the mers such that small subsets of the mer lists may be systematically compared and matched using relatively modest processor capacity.
In another aspect, a list of mers may be compared to the entries of a lookup table (LUT), which may be stored on volatile memory such as processor memory. The storage of mers in a LUT may be especially advantageous for storing comparing mers generated using a comparing sequence set for repeated use in a templated sequence assembly system, quantitative sequence assembly system, or pairwise genome alignment system. The use of a LUT allows nearly constant time access to information about any comparing mer stored in the LUT, without need to access or manipulate any of the comparing sequence sets or comparing mer tables. In one aspect, the LUT is substantially similar to the LUTs described in U.S. Pat. No. 6,223,128, which is hereby incorporated by reference in its entirety.
A particular advantage of the inclusion of a LUT to provide comparing mer sequences and associated location data is that the LUT-based layout generation system is capable of processing a subset of the total mer1 s generated from the sequence set 1 202C. In this particular aspect, the LUT-based layout generation system 3002 may process the sequences from the sequence set 1 202C one sequence at a time. Because the time taken to access matching comparing mer2s is essentially independent of the order in which the mer1s are presented for matching, there is no need to assemble or sort the mer1s generated from the sequence set 1 202C to minimize processing time.
The LUT-based layout generation system 102B reads a single sequence from the sequence set 1 202C at step 3004 into a single-sequence mer generating system 3002 that is substantially similar to the mer generating and sorting system 208 previously described in
The LUT-based layout generation system 3002 further includes a LUT-based hit determination system 3006 that matches each mer1 in the strand-independent mer1 table 3004 to the corresponding mer2s provided by the stored mer2 LUT 3012. Any matches between the mer1s and the mer2s are recorded as one or more entries in a temporary hit table 3014, which is essentially the same as the hit table 218 described previously in
The temporary hit table 3014 is processed using the methods previously described and illustrated in
Once all entries of the temporary hit table 3014 have been sorted, the entries may be deleted. Similarly, once all entries of the temporary sorted hit table 3016 have been condensed, the entries may be deleted. Because all intermediate table entries are deleted once they have been processed, the LUT-based layout generation system 3002 incurs relatively less computational overhead such as nonvolatile storage space and volatile processor memory capacity compared to the disk-based assembly method. Once all of the sequences of the sequence set 1 202C have been processed into mer1s, matched and recorded into the condensed hit table 238B, a post processing system 240B further processes the data using methods substantially similar to those describes previously in
Other aspects of the invention may process two or more sequences from the sequence set 1 202C at a time. In yet another aspect, all sequences from the sequence set 1 202C may be processed at the same time by the LUT-based layout generation system 3002. In still another aspect, the entries of the temporary tables used to store intermediate data during the processing of the sequence set 1 202C data may be deleted after the processing of a single sequence, after the processing of at least two or more sequences, or after the processing of all sequences from the sequence set 1 202C by the LUT-based layout generation system 3002. Non-limiting examples of temporary tables that may be deleted after the completion of processing include the strand-independent mer1 table 3004, the sequence set 1 202C, the temporary hit table 3014, the temporary sorted hit table 3016, and combinations thereof. A more detailed description of the systems and methods used by the LUT-based hit determination system 3006 is provided below.
Stored Mer2 LUT
In one aspect, the stored mer2 LUT 3012 may be arranged in a hash table format in which a hash key is used to index the hash table entries. The hash key may be encoded in the form of a prefix that includes the first kh bases of each mer sequence in the LUT where kh is typically less than the total number of bases in a mer nucleotide sequence. In one aspect, kh may range from about 8 to about 12 bases. The hash key may be encoded using a 2-bit representation of each base in the prefix combined into a single integer value.
During the creation of the hash table, a hash table chain or bucket is created for each possible hash key, and zero or more actual mers are stored in the hash table chain. All bases of the mer nucleotide sequences are stored along with associated data within the chain. Non-limiting examples of associated data includes a count of the number of occurrences of each mer in a sequence set, a sequence identifier for each mer occurrence, a position of the mer within a sequence for each mer occurrence, and a strand direction (i.e. forward-directed or reverse-directed) for each mer occurrence. Initially, the hash table chain may be a linear list of the mers kept in sorted order by their suffixes. However, to preserve the constant-time lookups of the hash table and efficiency of building the hash table for very large chains, large lists may be replaced by a nested tree of recursive mer prefix hashes as the number of table entries grows.
Typical uses of the stored mer2 LUT 3012 are to store a count of the number of occurrences of each mer present in a set of sequences or to store the count with additional positional information, such as the sequence identifier, position, and strand of each occurrence of the mers.
When the positional information for each mer2 is stored in the entries of the stored mer2 LUT 3012, sequence mapping may be performed directly from the LUT 3012. Any mer1 may be mapped to the genome by directly looking up the mer1 sequences in the stored mer2 LUT 3012le. The combinations of a mer1 sequence and the possible matched positions on sequence set 2 (comparing sequence set) from the hits for that mer1. This information is essentially the same as the information contained in the entries of a hit table 218 sorted by sequence index 1 (see
In rare cases where K, the number of nucleotides in the mer nucleotide sequence is relatively small, a more direct approach to hash table structure may be used. Instead of using a mer prefix as a hash, the entire mer may be used as the initial lookup. This hash table structure represents a special case conceptually in which Kh=K. In this case, a simple array of these hashes may be used in the hash table structure, allowing significantly lower search times to access the mer2 data.
In one aspect, the mer length used for the entries of the LUT may range from about 5 to about 33 bases. In an exemplary aspect, the mer length used for each entry of the LUT may be at least about 15 or 17 nucleotides for quantitative sequencing assembly applications. In another exemplary aspect, the mer length used for each entry of the LUT may from about 15 bases to about 25 bases for positional applications such as templated assembly and templated genome alignment.
Single-Sequence Mer Generating System
In an aspect, the single-sequence mer generating system 3006 generates a set of mer1s from a single sequence from a sequence table 1 202, assigns positional information to document the position of each mer within the sequence, removes the effect of sequence strand direction, and records the mer nucleotide sequence and positional information in a strand-independent mer1 table 3106.
The single-sequence preprocessing system 3102A, illustrated in
The sequence is checked to determine whether a sufficient number of nucleotides remain in the sequence to generate another mer1 at step 3216. If another mer1 can be generated from the sequence, the first nucleotide is deleted from the current nucleotide data register at step 3218, the next nucleotide in the sequence is read and added to the end of the nucleotide data register at step 3220, and the process of copying the nucleotide data into another mer1 and generating another mer table entry is repeated. If no other mer1s may be generated from the sequence at step 3216, the mer1 table entries are transferred to the amer/cmer generation system 406C at step 3222. The amer/cmer generation system 406C removes the effects of strand direction using the methods described in
LUT-Based Hit Determination System
The LUT-based hit determination system 3010A, illustrated in
If the mer1 read at step 3302 was determined to be the last mer entry from the strand-independent mer1 table 3008 at step 3310, the temporary hit table is processed by the hit table sorting system, as shown in
An exemplary LUT-based mer matching system 3304A is illustrated in
The single-sequence frameshift calculation system 3306, illustrated in
Hybrid Layout Generation System Overview
A hybrid layout generation system 3600, described in detail in
The hybrid layout generation system 3600 makes use of a mer2 table that is stored in volatile memory, in a similar manner to the LUT-based layout generation system 3002. However, rather than include all mers generated from the sequence set 2 204, the in-memory mer2 LUT 3610 includes only a subset of the entire set of mer2s based on a variety of selection criteria explained in detail below.
The mer2s to be included in the in-memory mer2 LUT 3610 are a subset of the entire mer2 set that is predicted to be most frequently matched by the mer1s generated from the read sequence set 1. Because the mer2 entries in the in-memory mer2 LUT 3610 are the most likely to be frequently accessed during the mer-matching processes, the volatile memory resources of the system are allocated in a manner designed to reduce the electromechanical load on the nonvolatile memory devices, such as computer disks, by reducing the number of matches to be performed using disk-based approaches. But because the disk-based approach is used to determine the hits between less frequently matched mer1s and mer2s, the volatile memory demand is reduced relative to a memory-intensive exclusively LUT-based approach. The remaining mer2 entries not selected for inclusion in the in-memory mer2 LUT 3610 are stored in a disk-based mer2 table 3608 in non-volatile memory.
During the processing of mer1 data by the hybrid layout generation system 3600, each mer1 may be quickly checked against the in-memory mer2 LUT 3610. If the matching mer2 is not found in the LUT, the mer1 table entry may be placed in a disk-based mer1 table 3612 and matched with any corresponding mer table entries from a disk-based mer2 table 3608 using the disk-based hit determination system 3614. A detailed description of the hybrid layout system 3600 is given below.
Hybrid Layout Generation System
A block diagram of an exemplary hybrid layout system 3600 is shown in
The mer generating and sorting system 208B is essentially identical to the mer generating and sorting system 208 previously described in
In this aspect, the sorted mer table 1 210C is used to populate a seed table 3604 using a seed table generation system 3602. The seed table includes a list of seed table entries containing only a mer nucleotide sequence, and a mer count index specifying the number of times the mer nucleotide occurs within the sequence set 1 202E. The seed table is used by the in-memory LUT generation system 3606 to select the mer2 entries from the sorted mer table 2 212A for inclusion in the in-memory mer2 LUT 3610. In this aspect, the seed table contains all mer1 (read) nucleotide sequences generated from the sequences within the sequence set 1 202E. As a result, the in-memory LUT 3610 may contain only those mer2 (comparing) nucleotide sequences that have matching mer1 sequences. The remaining sorted mer table 2 entries may be either transferred to a disk-based mer2 table 3608 or deleted. In this aspect, all mer2 entries that have matching mer1 entries are included in the in-memory mer2 LUT, without regard for the number of matches for each mer2. In other aspects, some of which are described below, a subset of the mer2 table entries are selected based on other criteria that predict the number of matches with mer1 table entries based on an analysis of the sorted mer table 1 210C, sorted mer table 2 212A, or a combination of both sorted mer tables.
The in-memory mer2 LUT 3610 is interrogated by the LUT-based hit determination system 3006B to identify mer2 table entries that match mer1 table entries from the sorted mer table 1 210C and generate entries for a hit table 218C. The LUT-based hit determination system 3006B is essentially identical to the LUT-based hit determination system described previously in
The resulting hit table 218C is then processed using previously described systems and techniques. In brief, a hit table sorting system 232C arranges the hit table entries such that entries are grouped by mer2 sequence, and by mer1 within each identical mer2 group, in addition to other positional and strand orientation groupings described previously in
Exemplary In-Memory LUT Generation Systems
The subset of table entries from the sorted mer table 2 212A to be included in the in-memory mer2 LUT table may be selected in order to optimize the tradeoff between the faster processing speed versus the increased demand for volatile memory inherent in the various aspects of the LUT-based method. The advantages of in-memory data processing may be best realized for repeated operations on a relatively limited amount of data. Thus, the selection of mer2 table entries that possess the highest number of matches with the mer1 table entries within sorted mer table 1 would result in more highly repeated operations (i.e. locating the mer2 LUT entries that match each mer1 entry) for a more limited amount of data. The identification of mer2 table entries that are likely to be frequently matched to corresponding mer1 table entries may be achieved using one of at least several methods described below.
In one aspect, all mers from the sorted mer table 2 212A are included in the in-memory mer2 LUT 3610. In another aspect, the entries of the sorted mer table 1 210C are analyzed to determine the nucleotide sequences and frequency of occurrence of all mer1 entries in the sorted mer table 1 210C. An illustration of one aspect using this method is illustrated in
The next mer1 sequence from the mer table 1 404B is read at step 3708 and compared to the mer1 nucleotide sequences in the seed table entries at step 3710. If the next mer1 nucleotide sequence matches a mer1 nucleotide sequence within the entries of the seed table 3604A, then the mercount of the matching seed table entry is increased by one at step 3716. If the next mer1 nucleotide sequence does not match any mer1 sequence within the seed table 3604A, then a new mercount index is set to an initial value of 1 at step 3712, and an entry to the seed table 3604 is appended to the other entries at step 3714.
At step 3718, if the end of the mer table 1 entries is not detected, the next mer1 sequence is read from the mer table 1 404B. Otherwise, the seed table 3604 is used by the in-memory LUT generation system 3606A to create the in-memory mer2 LUT used by the LUT-based hit determination system 3006B.
An exemplary in-memory LUT generation system 3606A is illustrated in
If the mer2 does not match a mer 1 in the seed table 3604, then the next mer2 is formed from the sequence set 2 204A at step 3806 and is similarly assessed at step 3804. If the mer2 does match a mer1 from the seed table 3604, then the mercount associated with the mer1 from the seed table 3604 is compared with a LUT inclusion threshold at step 3808.
The LUT inclusion threshold is a value corresponding to the minimum number of repeated occurrences of a mer1 nucleotide sequence that a mer2 nucleotide sequence must match in order to be included in the in-memory mer2 LUT 3610. As discussed previously, mer2 sequences that have a high number of predicted matches with the mer1 sequences take the best advantage of the fast processing speed of the LUT-based methods. However, if too many mer2 sequences are included in the in-memory mer2 LUT 3610, the volatile memory may become nearly full, leaving little excess capacity for processing data in the LUT-based hit determination system 3006B. In an aspect, the LUT inclusion threshold is set to a value selected so that no more than about 50% of the available volatile memory capacity of the system is used for storage of the in-memory mer2 LUT 3610.
At step 3808, if the frequency of occurrence of the mer1 sequence that is matched to the mer2 sequence under consideration is greater than the LUT inclusion threshold, an entry for this mer2 is appended to the in-memory mer2 LUT 3610 at step 3810, and the next mer2 is formed from the sequence set 2 204A at step 3814. If the frequency of occurrences of the matching mer1 sequence falls below the LUT inclusion threshold at step 3808, the mer2 is used to create a table entry on the disk-based mer2 table 3608.
In another aspect, the mer2 nucleotide sequences may be analyzed to estimate the most frequently matched mer sequences, in order to determine a criterion for including a particular mer2 sequence in the in-memory mer2 LUT 3610. In this aspect, it is assumed that a mer2 sequence that has a high frequency of occurrence would predict a relatively higher number of matches with mer1 sequences compared to less frequently occurring mer2 sequences. An exemplary in-memory LUT generation system 3606B that makes use of an analysis of the mer2 nucleotide sequences is illustrated in
The number of mer1s having identical nucleotide sequences or the number of mer2s having identical nucleotide sequences were used as criteria for the inclusion of a particular mer2 sequence in the in-memory mer2 LUT 3610 produced by the two exemplary in-memory LUT generation systems 3606A and 3606B, respectively. In a third aspect, the number of both the mer1s and the mer2s having an identical mer nucleotide sequence may be used as a criterion for including the mer sequence in the in-memory mer2 LUT 3610. In particular the product of the number of mer1s (N) and the number of mer2s (M) having identical nucleotide sequences is used as a criterion for inclusion in the in-memory mer2 LUT 3610. The product N×M is the actual number of hits that result from the matches of the mer1s and mer2s having that particular nucleotide sequence.
An exemplary in-memory LUT generation system 3602C is illustrated in
In the various aspects of in-memory LUT generation systems 3602A, 3602B, and 3602C, the frequency of occurrence of repeated mer1 sequences, repeated mer2 sequences, or the product of the two frequencies of occurrence were used to estimate the amount of volatile memory to be allocated to the storage of the in-memory mer2 LUT 3610. The particular choice of in-memory LUT generation system 3602 may depend on at least several factors including the availability of mer1 and mer2 sequence data for analysis, the depth of coverage or overall length of the sequenced reads or comparing sequence, and the characteristics of the processor 112 or CRM 114 included in the layout generation system 104, such as the amount of available volatile memory capacity. In another aspect, the frequency of occurrence of mer1s and mer2s having the same mer nucleotide sequence may be used to monitor the volatile memory usage of an automated hybrid layout generation system 3600.
Automated Hybrid Layout Generation System
In one aspect, a hybrid layout generation system may be implemented in an automated manner that may be executed without need for user intervention or adjustment. In order to provide sufficient information to allocate mer matching activity between the LUT-based hit determination system 3006B and the disk-based hit determination system 3614, a combination of the three in-memory LUT generation systems 3602A, 3602B and 3602C may be utilized. Initially a seed table 3604 may be generated using a method similar to that illustrated in
An initial in-memory mer2 LUT 3610 may be generated using a method similar to that described for the in-memory LUT generation system 3606A illustrated in
The remaining mer2 entries in the disk-based mer2 table 3608 may then be scanned iteratively to add additional mer2 entries to the in-memory mer2 LUT using the methods described in any one of the in-memory LUT generation systems 3602A, 3602B, and 3602C or any combination thereof. Within each of the in-memory LUT generation systems 3602A, 3602B, and 3602C, the LUT inclusion threshold value is iteratively decreased in order to allow more mer2 entries to be transferred into the in-memory mer2 LUT 3610. As before, the usage of volatile memory is monitored through the process of adding additional mer2 sequences to the in-memory mer2 LUT 3610 and the process may be halted or the LUT inclusion thresholds may be iteratively increased if the volatile memory used to store the in-memory mer2 LUT 3610 exceeds a threshold storage limit such as 50% of the available volatile storage capacity.
As the volatile memory allocated to the storage of the in-memory mer2 LUT 3610 nears the allocated memory storage limit, the entries of the seed table 3604 having the lowest mercounts may be incrementally discarded. The elimination of low mercount entries in the seed table results in the inclusion of only the mer2 sequences matching the most frequently occurring mer1s in the remainder of the in-memory mer2 LUT 3610 and frees up additional volatile memory for the creation of additional entries for the in-memory mer2 LUT 3610. If all mer2 sequences above a LUT inclusion threshold have been included in the in-memory mer2 LUT 3610, the seed table 3604 may be discarded altogether, and the selection of subsequent mer2s for inclusion in the in-memory mer2 LUT 3610 may be performed using a method similar to that described in the in-memory LUT generation system 3606B, in which the frequency of the mer2 selection is based on the frequency of each mer2 in the comparing sequence. Using this process, the final in-memory mer2 LUT 3610 can grow as large as desired without interference from the seed table.
In one aspect, mer2 entries may be placed in both the disk-based mer2 table 3608 and the in-memory mer2 LUT 3610. Since the mer1 nucleotide sequences are matched against the in-memory mer2 LUT 3610 first during analysis, mer2 sequences which are included in both locations may still be excluded from the disk-based mer2 table 3608, and the corresponding hits may be generated directly using the LUT-based hit determination system 3006B.
In another aspect, the methods of the automated hybrid layout generation system may still be utilized even if the seed table was generated only partially. Because all mer2 table entries not included in the in-memory mer2 LUT 3610 are written to the disk-based mer2 table 3608, no mer2 sequences are excluded from consideration due to having no seed count. In this aspect, the in-memory LUT generation system 3602B that makes use of the frequency of occurrence of mer2 gains higher importance. The entire sorted mer table 2 212A must be placed on disk for use in the LUT-based hit determination system 3006B and for mer2 generation. The hit generation processes remain unchanged in this aspect even if the seed table was only partially generated.
The following examples illustrate various aspects of the invention.
To assess the impact of reducing the number of read mers using various degrees of mer thinning, the following experiments were conducted. The Illumina 36-base sequenced read set and a ˜200 base 454 sequenced read set generated from isolates of the Escherichia coli K-12 strain, MG1655, were processed along with the MG1655 comparing sequence set using a disk-based layout generation system similar to the system illustrated in
The layout generation system was run using mer lengths of 32, 25, and 21 bases. In addition, mer minimization was performed on the read mers by skipping every 2nd, 4th, 8th, and 16th mer generated by the system. For each data set, the layout generation system was run with no mer thinning as a control case.
The impact of mer thinning of the read mers on the performance of the layout generation system was assessed by calculating the percentage of the total read sequences that were included in the layout file produced by the layout generation system. This measure of performance assessed how many reads were excluded from the layout file due to various levels of thinning. In addition, the overall processing time was measured for different levels of read mer thinning.
In general, at least two matching mers from a given read sequence were required to include that read sequence in the layout file. Reads matching in more than one location on the comparing sequence were added to each location. A summary of the effect of mer thinning on layout generation system performance is summarized in Table 1 below:
For the layouts assembled using the 36-base Illumina reads, using mers with a length of 32 nucleotides resulted in the inclusion of a relatively low percentage of reads in the layout file even before thinning the mers. Because there must be at least two 32-mers that match exactly out of only three mers per read, only about 15% of the read sequences were included in the layout file with no mer thinning. When every fourth or eighth mers were used in the process, only one mer remained per read, rendering the mers from all read sequences insufficient for inclusion in the layout file. When the mer size was lowered to 25 nucleotides, nearly 90% of the reads were included in the layout file prior to mer thinning. There were minor reductions when the read mers were thinned by using every fourth or eighth mer. A significant drop in the number of read sequences included in the layout file occurred when only every eighth mer was considered as there were only two mers used per read, both of which must match the comparing mer exactly for inclusion in the layout file.
When the 72 base Illumina data set was processed, the majority of reads were present in the assembly with shorter mer sizes, resulting in more qualifying reads. Overall, fewer read sequences were included in the layout files compared to the 36 base Illumina data set, largely because the Illumina 75 base data set was prepared from an E. coli strain that was more distantly related to the strain used to prepare the comparing sequences. Significantly, only a minor reduction in the number of reads in the layout was experienced when the data was thinned to every fourth mer. Moreover, there was a nearly linear reduction in the processing time as the data was increasingly thinned. When a fourth of the mer data was used, the layout file was processed in about a fourth of the time.
The much longer 200 base sequences of the 454 data set were even more tolerant of thinning. Nearly all of the read sequences were included in the layout file regardless of the mer size or the level of mer skipping. The processing time showed a near linear improvement with respect to the amount of mer data included, in a manner similar to the 72 base Illumina data set.
To assess the sensitivity of the effects of mer thinning on the genome of a different species, a similar analysis was performed on the ˜7.4× deep James Watson data set available from 454 using the NCBI Build 36 human reference sequence as a comparing sequence.
For the Watson human genome data set, the percentage of reads included in the layout file appeared to be improved by mer thinning. When all or most of the read mers were used, the percentage of read sequences in the layout was greater than 100%, indicating that a significant number of read sequences had been placed in more than one location, presumably due to the numerous repeated elements in the human genome. When the read mers were thinned by using every sixteenth mer, the percentage of reads included in the layout was close to 100%. The processing time again showed a linear reduction with respect to the amount of thinning down to approximately an eighth of the data, in a manner similar to the E. coli results.
Those skilled in the art will appreciate that variations from the specific embodiments disclosed above are contemplated by the invention. The invention should not be restricted to the above embodiments, but should be measured by the following claims.
This application claims priority to U.S. Provisional Application Ser. No. 61/158,730 filed on Mar. 9, 2009, which is hereby incorporated herein by reference in its entirety.
This work may have been partially supported by a Phase I grant from the National Institutes of Health grant number R43 GM082117-01 and a Phase II grant from the National Institutes of Health grant number R44 GM082117-02A1. The U.S. government may have certain rights in this invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6223128 | Allex et al. | Apr 2001 | B1 |
| 8209130 | Kennedy et al. | Jun 2012 | B1 |
| Entry |
|---|
| Velvet Manual (Version 1.1, Aug. 29, 2008, pp. 1-22). |
| Kent et al. (Genome Res., 2001, 11:1541-1548). |
| Mullikin et al., “The Phusion Assembler”, Genome Research, 2003, pp. 81-90, vol. 13. |
| Pennisi, “Semiconductors Inspire New Sequence Technologies”, Science, 2010, pp. 1190, vol. 327, No. 5970. |
| Batzoglou, et al., “ARACHNE: A Whole-Genome Shotgun Assembler”, Genome Research 12:177-189 (2002). |
| Bonfield, et al., “A New DNA Sequence Assembly Program”, Nucleic Acid Res. 23(24):4992-4999 (1995). |
| Klein, et al., “LOCAS—A Low Coverage Assembly Tool for Resequencing Projects”, PLoS ONE 6(8): e23455. (2011). |
| Langmead et al., “Ultrafast and memory-efficient alignment of short DNA sequences to the human genome”, Genome Biology vol. 10, Issue 3, Article R25 (2009). |
| Li et al., “Fast and accurate short read alignment with Burrows-Wheeler transform”, doi:10.1093/bioinformatics/btp324, vol. 25 No. 14, pp. 1754-1760 (2009). |
| Li et al. “SOAP2: an improved ultrafast tool for short read alignment”, Bioinformatics 25(15):1966-1967 (2009). |
| MacCallum, et al. “ALLPATHS 2: small genomes assembled accurately and with high continuity from short paired reads”, Genome Biology 10: R103 (2009). |
| Simpson, et al., “ABySS: A parallel assembler for short read sequence data”, Genome Research 19:1117-1123 (2009). |
| Sutton et al., “TIGR Assembler: A New Tool for Assembling Large Shotgun Sequencing Projects”, Genomic Science & Tech. 1(1):9-19 (1995). |
| Tritt, et al., “An Integrated Pipeline for de Novo Assembly of Microbial Genomes”, PLoS ONE 7(9): e42304. (2012). |
| Venter et al., “A New Strategy for Genome Sequencing”, Nature 381:364-365 (1996). |
| Weber, J. and E.W. Meyers, “Human Whole-Genome Shotgun Sequencing”, Genome Research 7:401-409 (1997). |
| Zerbino, D.R. and E. Birney, “Velvet: Algorithms for de novo short read assembly using de Bruijn graphs”, Genome Research 18:821-829 (2008). |
| Number | Date | Country | |
|---|---|---|---|
| 61158730 | Mar 2009 | US |
