METHOD AND SYSTEM FOR THE TRANSMISSION OF BIOINFORMATICS DATA

Abstract

Method and system for the transmission of genomic data. The transmission of genomic data is realized by employing the multiplexing of a structured compressed genomic dataset in a stream of genomic data partitioned into randomly accessible access units.

Description

TECHNICAL FIELD

The present application provides new methods for efficient storing, access, transmission and multiplexing of bioinformatic data and in particular genomic sequencing data.

BACKGROUND

An appropriate representation of genome sequencing data is fundamental to enable efficient processing, storage and transmission of genomic data to render possible and facilitate analysis applications such as genome variants calling and all analysis performed, with various purposes, by processing the sequencing data and metadata. Today, genome sequencing information are generated by High Throughput Sequencing (HTS) machines in the form of sequences of nucleotides (a. k. a. bases) represented by strings of letters from a defined vocabulary.

These sequencing machines do not read out an entire genomes or genes, but they produce short random fragments of nucleotide sequences known as sequence reads.

A quality score is associated to each nucleotide in a sequence read. Such number represents the confidence level given by the machine to the read of a specific nucleotide at a specific location in the nucleotide sequence.

This raw sequencing data generated by NGS machines are commonly stored in FASTQ files (see also FIG. 1).

The smallest vocabulary to represent sequences of nucleotides obtained by a sequencing process is composed by five symbols: {A, C, G, T, N} representing the 4 types of nucleotides present in DNA namely Adenine, Cytosine, Guanine, and Thymine plus the symbol N to indicate that the sequencing machine was not able to call any base with a sufficient level of confidence, so the type of base in such position remains undetermined in the reading process. In RNA Thymine is replaced by Uracil (U). The nucleotides sequences produced by sequencing machines are called “reads”. In case of paired reads the term “template” is used to designate the original sequence from which the read pair has been extracted. Sequence reads can be composed by a number of nucleotides in a range from a few dozen up to several thousand. Some technologies produce sequence reads in pairs where each read can come from one of the two DNA strands.

In the genome sequencing field the term “coverage” is used to express the level of redundancy of the sequence data with respect to a reference genome. For example, to reach a coverage of 30× on a human genome (3.2 billion bases long) a sequencing machine shall produce a total of about 30×3.2 billion bases so that in average each position in the reference is “covered” 30 times.

STATE OF THE ART SOLUTIONS

The most used genome information representations of sequencing data are based on FASTQ and SAM file formats which are commonly made available in zipped form to reduce the original size. The traditional file formats, respectively FASTQ and SAM for non-aligned and aligned sequencing data, are constituted by plain text characters and are thus compressed by using general purpose approaches such as LZ (from Lempel and Ziv) schemes (the well-known zip, gzip etc). When general purpose compressors such as gzip are used, the result of the compression is usually a single blob of binary data. The information in such monolithic form results quite difficult to archive, transfer and elaborate particularly in the case of high throughput sequencing when the volumes of data are extremely large.

After sequencing, each stage of a genomic information processing pipeline produces data represented by a completely new data structure (file format) despite the fact that in reality only a small fraction of the generated data is new with respect to the previous stage.

FIG. 1 shows the main stages of a typical genomic information processing pipeline with the indication of the associatedfile format representation.

Commonly used solutions presents several drawbacks: data archival is inefficient for the fact that a different file format is used at each stage of the genomic information processing pipelines which implies the multiple replication of data, with the consequent rapid increase of the required storage space. This is inefficient and unnecessary and it is also becoming not sustainable for the increase of the data volume generated by HTS machines. This has in fact consequences in terms of available storage space and generated costs, and it is also hindering the benefits of genomic analysis in healthcare from reaching a larger portion of the population. The impact of the IT costs generated by the exponential growth of sequence data to be stored and analysed is currently one of the main challenges the scientific community and that the healthcare industry have to face (see Scott D. Kahn “On the future of genomic data”—Science 331, 728 (2011) and Pavlichin, D. S., Weissman, T., and G. Yona. 2013. “The human genome contracts again” Bioinformatics 29(17): 2199-2202). At the same time several are the initiatives attempting to scale genome sequencing from a few selected individuals to large populations (see Josh P. Roberts “Million Veterans Sequenced”—Nature Biotechnology 31, 470 (2013))

The transfer of genomic data is slow and inefficient because the currently used data formats are organized into monolithic files of up to several hundred Gigabytes of size which need to be entirely transferred at the receiving end in order to be processed. This implies that the analysis of a small segment of the data requires the transfer of the entire file with significant costs in terms of consumed bandwidth and waiting time. Often online transfer is prohibitive for the large volumes of the data to be transferred, and the transport of the data is performed by physically moving storage media such as hard disk drives or storage servers from one location to another.

These limitations occurring when employing state of the art approaches are overcome by the present invention. Processing the data is slow and inefficient for to the fact that the information is not structured in such a way that the portions of the different classes of data and metadata required by commonly used analysis applications cannot be retrieved without the need of accessing the data in its totality. This fact implies that common analysis pipelines can require to run for days or weeks wasting precious and costly processing resources because of the need, at each stage of accessing, of parsing and filtering large volumes of data even if the portions of data relevant for the specific analysis purpose is much smaller.

These limitations are preventing health care professionals from timely obtaining genomic analysis reports and promptly reacting to diseases outbreaks. The present invention provides a solution to this need.

There is another technical limitation that is overcome by the present invention.

In fact the invention aims at providing an appropriate genomic sequencing data and metadata representation by organizing and partitioning the data so that the compression of data and metadata is maximized and several functionality such as selective access and support for incremental updates are efficiently enabled.

A key aspect of the invention is a specific definition of classes of data and metadata to be represented by an appropriate source model, coded (i.e. compressed) separately by being structured in specific layers. The most important achievements of this invention with respect to existing state of the art methods consist in:

• • the increase of compression performance due to the reduction of the information source entropy constituted by providing an efficient model for each class of data or metadata;
  • the possibility of performing selective accesses to portions of the compressed data and metadata for any further processing purpose;
  • the possibility to incrementally (without the need of re-encoding) update and add encoded data and metadata with new sequencing data and/or metadata and/or new analysis results;
  • the possibility of efficiently process data as soon as they are produced by the sequencing machine or alignment tools without the need to wait the end of the sequencing or alignment process.

The present application discloses a method and system addressing the problem of efficient manipulation, storage and transmission of very large amounts of genomic sequencing data, by employing a structured access units approach combined with multiplexing techniques.

The present application overcomes all the limitations of the prior art approaches related to the functionality of genomic data accessibility, efficient processing of data subsets, transmission and streaming functionality combined with an efficient compression.

Today the most used representation format for genomic data is the Sequence Alignment Mapping (SAM) textual format and its binary correspondent BAM. SAM files are human readable ASCII text files whereas BAM adopts a block based variant of gzip. BAM files can be indexed to enable a limited modality of random access. This is supported by the creation of a separate index file.

The BAM format is characterized by poor compression performance for the following reasons:

• • 1. It focus on compressing the inefficient and redundant SAM file format rather than on extracting the actual genomic information conveyed by SAM files and using appropriate models for compressing it.
  • 2. It employs a general purpose text compression algorithm such as gzip rather than exploiting the specific nature of each data source (the genomic information itself).
  • 3. It lacks any concept related to data classification that would enable a selective access to specific classes of genomic data.

A more sophisticated approach to genomic data compression that is less commonly used, but more efficient than BAM is CRAM (CRAM specification: https://samtools.github.io/hts-specs/CRAMv3.pdf). CRAM provides a more efficient compression for the adoption of differential encoding with respect to an existing reference (it partially exploits the data source redundance), but it still lacks features such as incremental updates, support for streaming and selective access to specific classes of compressed data.

CRAM relies on the concept of the CRAM record. Each CRAM record encodes a single mapped or unmapped reads by encoding all the elements necessary to reconstruct it.

The main differences of the present invention with respect to the CRAM approach are:

• • 1. For CRAM, data indexing is out of the scope of the specification (see section 12 of CRAM specification v 3.0) and it is implemented as a separate file. In the present invention the data indexing is integrated with the encoding process and indexes are embedded in the encoded bit stream.
  • 2. In CRAM all core data blocks can contain any type of mapped reads (perfectly matching reads, reads with substitutions only, reads with indels). In the present invention there is no notion of classification and grouping of reads in classes according to the result of a mapping with respect to a reference sequence.
  • 3. In the described invention there is no notion of record encapsulating each read because the data needed to reconstruct each read is scattered among several data containers called “layers”. This enables a more efficient access to set of reads with specific biological characteristics (e.g. reads with substitutions, but without indels, or perfectly mapped reads) without the need of decoding each (block of) read(s) to inspect its features.
  • 4. In a CRAM record each type of data is denoted by a specific flag. Differently from CRAM in the present invention there is no notion of flag denoting data because this is intrinsically defined by the “layer” the data belongs to. This implies a largely reduced number of symbols to be used and a consequent reduction of the information source entropy which results into a more efficient compression. This is due to the fact that the use of different “layers” enables the encoder to reuse the same symbol across each layer with different meanings. In CRAM each flag must always have the same meaning as there is no notion of contexts and each CRAM record can contain any type of data.
  • 5. In CRAM substitutions, insertions and deletions are expressed according to different syntaxes, whereas the present invention uses a single alphabet and encoding for substitutions, insertions and deletions. This makes the encoding and decoding process simpler and produces a lower entropy source model which coding yields bitstreams characterized by higher compression performance.

Genomic compression algorithms used in the state of the art can be classified into these categories:

• • Transform-based
    • LZ-based
    • Read reordering
  • Assembly-based
  • Statistical modeling

The first two categories share the disadvantage of not exploiting the specific characteristics of the data source (genomic sequence reads) and process the genomic data as string of text to be compressed without taking into account the specific properties of such kind of information (e.g. redundancy among reads, reference to an existing sample). Two of the most advanced toolkits for genomic data compression, namely CRAM and Goby (“Compression of structured high-throughput sequencing data”, F. Campagne, K. C. Dorff, N. Chambwe, J. T. Robinson, J. P. Mesirov, T. D. Wu), make a poor use of arithmetic coding as they implicitly model data as independent and identically distributed by a Geometric distribution. Goby is slightly more sophisticated since it converts all the fields to a list of integers and each list is encoded independently using arithmetic coding without using any context. In the most efficient mode of operation, Goby is able to perform some inter-list modeling over the integer lists to improve compression. These prior art solutions yield poor compression ratios and data structures that are difficult if not impossible to selectively access and manipulate once compressed. Downstream analysis stages can result to be inefficient and very slow due to the necessity of handling large and rigid data structures even to perform simple operation or to access selected regions of the genomic dataset.

A simplified vision of the relation among the file formats used in genome processing pipelines is depicted in FIG. 1. In this diagram file inclusion does not imply the existence of a nested file structure, but it only represents the type and amount of information that can be encoded for each format (i.e. SAM contains all information in FASTQ, but organized in a different file structure). CRAM contains the same genomic information as SAM/BAM, but it has more flexibility in the type of compression that can be used, therefore it is represented as a superset of SAM/BAM.

The use of multiple file formats for the storage of genomic information is highly inefficient and costly. Having different file formats at different stages of the genomic information life cycle implies a linear growth of utilized storage space even if the incremental information is minimal. Further disadvantages of prior art solutions are listed below.

• • 1. Accessing, analysing or adding annotations (metadata) to raw data stored in compressed FastQ files or any combination thereof requires the decompression and recompression of the entire file with extensive usage of computational resources and time.
  • 2. Retrieving specific subsets of information such as read mapping position, read variant position and type, indels position and types, or any other metadata and annotation contained in aligned data stored in BAM files requires to access the whole data volume associated to each read. Selective access to a single class of metadata is not possible with prior art solutions.
  • 3. Prior art file formats require that the whole file is received at the end user before processing can start. For example the alignment of reads could start before the sequencing process has been completed relying on an appropriate data representation. Sequencing, alignment and analysis could proceed and run in parallel.
  • 4. Prior art solution do not support structuring and are not able of distinguishing genomic data obtained by different sequencing processes according to their specific generation semantic (e.g. sequencing obtained at different time of the life of the same individual). The same limitation occurs for sequencing obtained by different types of biological samples of the same individual.
  • 5. The encryption of entire or selected portions of the data is not supported by prior art solutions. For example the encryption of:
    • a. selected DNA regions
    • b. only those sequences containing variants
    • c. chimeric sequences only
    • d. unmapped sequences only
    • e. specific metadata (e.g. origin of the sequenced sample, identity of sequenced individual, type of sample)
  • 6. The transcoding from sequencing data aligned to a given reference (i.e. a SAM/BAM file) to a new reference requires to process the entire data volume even if the new reference differs only by a single nucleotide position from the previous reference.

There is therefore a need of an appropriate Genomic Information Storage Layer (Genomic File Format) that enables efficient compression, supports the selective access in the compressed domain, supports the incremental addition of heterogeneous metadata in the compressed domain at all levels of the different stages of the genomic data processing.

The present invention provides a solution to the limitations of the state of the art by employing the method, devices and computer programs as claimed in the accompanying set of claims.

LIST OF FIGURES

FIG. 1 shows the main steps of a typical genomic pipeline and the related file formats.

FIG. 2 shows the mutual relationship among the most used genomic file formats

FIG. 3 shows how genomic sequence reads are assembled in an entire or partial genome via de-novo assembly or reference based alignment.

FIG. 4 shows how reads mapping positions on the reference sequence are calculated.

FIG. 5 shows how reads pairing distances are calculated.

FIG. 6 shows how pairing errors are calculated.

FIG. 7 shows how the pairing distance is encoded when a read mate pair is mapped on a different chromosome.

FIG. 8 shows how sequence reads can come from the first or second DNA strand of a genome.

FIG. 9 shows how a read mapped on strand 2 has a corresponding reverse complemented read on strand 1.

FIG. 10 shows the four possible combinations of reads composing a reads pair and the respective encoding in the rcomp layer.

FIG. 11 shows how N mismatches are encoded in a nmis layer.

FIG. 12 shows an example of substitutions in a mapped read pair.

FIG. 13 shows how substitutions positions can be calculated either as absolute or differential values.

FIG. 14 shows how symbols encoding substitutions without IUPAC codes are calculated.

FIG. 15 shows how substitution types are encoded in the snpt layer.

FIG. 16 shows how symbols encoding substitutions with IUPAC codes are calculated.

FIG. 17 shows an alternative source model for substitution where only positions are encoded, but one layer per substitution type is used.

FIG. 18 shows how to encode substitutions, inserts and deletions in a reads pair of class I when IUPAC codes are not used.

FIG. 19 shows how to encode substitutions, inserts and deletions in a reads pair of class I when IUPAC codes are used.

FIG. 20 shows the structure of the header of the genomic information data structure.

FIG. 21 shows how the Master Index Table contains the positions on the reference sequences of the first read in each Access Unit.

FIG. 22 shows an example of partial MIT showing the mapping positions of the first read in each pos AU of class P.

FIG. 23 shows how the Local Index Table in the layer header is a vector of pointers to the AUs in the payload.

FIG. 24 shows an example of Local Index Table.

FIG. 25 shows the functional relation between Master Index Table and Local Index Tables

FIG. 26 shows how Access Units are composed by blocks of data belonging to several layers. Layers are composed by Blocks subdivided in Packets.

FIG. 27 shows how a Genomic Access Unit of type 1 (containing positional, pairing, reverse complement and read length information) is packetized and encapsulated in a Genomic Data Multiplex.

FIG. 28 shows how Access Units are composed by a header and multiplexed blocks belonging to one or more layers of homogeneous data. Each block can be composed by one or more packets containing the actual descriptors of the genomic information.

FIG. 29 shows the structure of Access Units of type 0 which do not need to refer to any information coming from other access units to be accessed or decoded and accessed.

FIG. 30 shows the structure of Access Units of type 1.

FIG. 31 shows the structure of Access Units of type 2 which contain data that refer to an access unit of type 1. These are the positions of N in the encoded reads.

FIG. 32 shows the structure of Access Units of type 3 which contain data that refer to an access unit of type 1. These are the positions and types of mismatches in the encoded reads.

FIG. 33 shows the structure of Access Units of type 4 which contain data that refer to an access unit of type 1. These are the positions and types of mismatches in the encoded reads.

FIG. 34 shows the first five type of Access Units.

FIG. 35 shows that Access Units of type 1 refer to Access Units of type 0 to be decoded.

FIG. 36 shows that Access Units of type 2 refer to Access Units of type 0 and 1 to be decoded.

FIG. 37 shows that Access Units of type 3 refer to Access Units of type 0 and 1 to be decoded.

FIG. 38 shows that Access Units of type 4 refer to Access Units of type 0 and 1 to be decoded.

FIG. 39 shows the Access Units required to decode sequence reads with mismatches mapped on the second segment of the reference sequence (AU 0-2).

FIG. 40 shows how raw genomic sequence data that becomes available can be incrementally added to pre-encoded genomic data.

FIG. 41 shows how a data structure based on Access Units enables genomic data analysis to start before the sequencing process is completed.

FIG. 42 shows how new analysis performed on existing data can imply that reads are moved from AUs of type 4 to one of type 3.

FIG. 43 shows how newly generated analysis data are encapsulated in a new AU of type 6 and a corresponding index is created in the MIT.

FIG. 44 shows how to transcode data due to the publication of a new reference sequence (genome).

FIG. 45 shows how reads mapped to a new genomic region with better quality (e.g. no indels) are moved from AU of type 4 to AU of type 3

FIG. 46 shows how, in case new mapping location is found, (e.g. with less mismatches) the related reads can be moved from one AU to another of the same type.

FIG. 47 shows how selective encryption can be applied on Access Units of Type 4 only as they contain the sensible information to be protected.

FIG. 48 shows the data encapsulation in a genomic multiplex where one or more genomic datasets 482-483 contain Genomic streams 484 and streams of Genomic Datasets Lists 481, Genomic Dataset Mapping Tables 485, and Reference Identifiers Mapping Tables 487. Each genomic stream is composed by a Header 488 and Access Units 486. Access Units encapsulate Blocks 489 which are composed by Packets 4810.

FIG. 49 shows how raw genomic sequence data or aligned genomic data are processed to be encapsulated in a Genomic Multiplex. The alignment, re-alignment, assembly stages can be necessary to prepare the data for encoding. The generated layers are encapsulated in Access Units and multiplexed by the Genomic Multiplexer

FIG. 50 shows how a genomic demultiplexer (501) extracts Access Units layers from the Genomic Multiplex, one decoder per AU type (502) extrac is the genomic descriptors which are then decoded (503) into various genomic formats such as for example FASTQ and SAM/BAM

DETAILED DESCRIPTION

The present invention describes a multiplexing file format and the relevant access units to be used to store, transport, access and process genomic or proteomic information in the form of sequences of symbols representing molecules.

These molecules include, for example, nucleotides, amino acids and proteins. One of the most important pieces of information represented as sequence of symbols are the data generated by high-throughput genome sequencing devices.

The genome of any living organism is usually represented as a string of symbols expressing the chain of nucleic acids (bases) characterizing that organism. Current state of the art genome sequencing technology is able to produce only a fragmented representation of the genome in the form of several (up to billions) strings of nucleic acids associated to metadata (identifiers, level of accuracy etc.). Such strings are usually called “sequence reads” or “reads”.

The typical steps of the genomic information life cycle comprise Sequence reads extraction, Mapping and Alignment, Variant detection, Variant annotation and Functional and Structural Analysis (see FIG. 1).

Sequence reads extraction is the process—performed by either a human operator or a machine—of representation of fragments of genetic information in the form of sequences of symbols representing the molecules composing a biological sample. In the case of nucleic acids such molecules are called “nucleotides”.

The sequences of symbols produced by the extraction are commonly referred to as “reads”. This information is usually encoded in prior art as FASTA files including a textual header and a sequence of symbols representing the sequenced molecules.

When the biological sample is sequenced to extract DNA of a living organism the alphabet is composed by the symbols (A,C,G,T,N).

When the biological sample is sequenced to extract RNA of a living organism the alphabet is composed by the symbols (A,C,G,U,N).

In case the IUPAC extended set of symbols, so called “ambiguity codes” are also generated by the sequencing machine, the alphabet used for the symbols composing the reads are (A, C, G, T, U, W, S, M, K, R, Y, B, D, H, V, N or -).

When the IUPAC ambiguity codes are not used, a sequence of quality score can be associated to each sequence read. In such case prior art solutions encode the resulting information as a FASTQ file. Sequencing devices can introduce errors in the sequence reads such as:

• • 1. identification of a wrong symbol (i.e. representing a different nucleic acid) to represent the nucleic acid actually present in the sequenced sample; this is usually called “substitution error” (mismatch);
  • 2. insertion in one sequence read of additional symbols that do not refer to any actually present nucleic acid; this is usually called “insertion error”;
  • 3. deletion from one sequence read of symbols that represent nucleic acids that are actually present in the sequenced sample; this is usually called “deletion error”;
  • 4. recombination of one or more fragments into a single fragment which does not reflect the reality of the originating sequence.

The term “coverage” is used in literature to quantify the extent to which a reference genome or part thereof can be covered by the available sequence reads. Coverage is said to be:

• • partial (less than 1×) when some parts of the reference genome are not mapped by any available sequence read
  • single (1×) when all nucleotides of the reference genome are mapped by one and only one symbol present in the sequence reads
  • multiple (2×, 3×, N×) when each nucleotide of the reference genome is mapped multiple times.

Sequence alignment refers to the process of arranging sequence reads by finding regions of similarity that may be a consequence of functional, structural, or evolutionary relationships among the sequences. When the alignment is performed with reference to a pre-existing nucleotides sequence referred to as “reference genome”, the process is called “mapping”. Sequence alignment can also be performed without a pre-existing sequence (i.e. reference genome) in such cases the process is known in prior art as “de novo” alignment. Prior art solutions store this information in SAM, BAM or CRAM files. The concept of aligning sequences to reconstruct a partial or complete genome is depicted in FIG. 3.

Variant detection (a.k.a. variant calling) is the process of translating the aligned output of genome sequencing machines, (sequence reads generated by NGS devices and aligned), to a summary of the unique characteristics of the organism being sequenced that cannot be found in other pre-existing sequence or can be found in a few pre-existing sequences only. These characteristics are called “variants” because they are expressed as differences between the genome of the organism under study and a reference genome. Prior art solutions store this information in a specific file format called VCF file.

Variant annotation is the process of assigning functional information to the genomic variants identified by the process of variant calling. This implies the classification of variants according to their relationship to coding sequences in the genome and according to their impact on the coding sequence and the gene product. This is in prior art usually stored in a MAF file.

The process of analysis of DNA (variant, CNV=copy number variation, methylation etc,) strands to define their relationship with genes (and proteins) functions and structure is called functional or structural analysis. Several different solutions exist in the prior art for the storage of this data.

Genomic File Format

The invention disclosed in this document consists in the definition of a compressed data structure for representing, processing manipulating and transmitting genome sequencing data that differs from prior art solutions for at least the following aspects:

• • It does not rely on any prior art representation formats of genomic information (i.e. FASTQ, SAM).
  • It implements a new original classification of the genomic data and metadata according to their specific characteristics. Sequence reads are mapped to a reference sequence and grouped in distinct classes according to the results of the alignment process. This results in data classes with lower information entropy that can be more efficiently encoded applying different specific compression algorithms.
  • It defines syntax elements and the related encoding/decoding process conveying the sequence reads and the alignment information into a representation which is more efficient to be processed for downstream analysis applications.

Classifying the reads according to the result of mapping and coding them using descriptors to be stored in layers (position layer, mate distance layer, mismatch type layer etc, etc, . . . ) present the following advantages:

• • A reduction of the information entropy when the different syntax elements are modelled by a specific source model.
  • A more efficient access to data that are already organized in groups/layers that have a specific meaning for the downstream analysis stages and that can be accesses separately and independently.
  • The presence of a modular data structure that can be updated incrementally by accessing only the required information without the need of decoding the whole data content.
  • The genomic information produced by sequencing machines is intrinsically highly redundant due to the nature of the information itself and to the need of mitigating the errors intrinsic in the sequencing process. This implies that the relevant genetic information which needs to be identified and analyzed (the variations with respect to a reference) is only a small fraction of the produced data. Prior art genomic data representation formats are not conceived to “isolate” the meaningful information at a given analysis stage from the rest of the information so as to make it promptly available to the analysis applications.
  • The solution brought by the disclosed invention is to represent genomic data in such a way that any relevant portion of data is readily available to the analysis applications without the need of accessing and decompressing the entirety of data and the redundancy of the data is efficiently reduced by efficient compression to minimize the required storage space and transmission bandwidth.

The key elements of the invention are:

• • 1. The specification of a file format that “contains” structured and selectively accessible data elements (Access Units (AU) in compressed form. Such approach can be seen as the opposite of prior art approaches, SAM and BAM for instance, in which data are structured in non-compressed form and then the entire file is compressed. A first clear advantage of the approach is to be able to efficiently and naturally provide various forms of structured selective access to the data elements in the compressed domain which is impossible or extremely awkward in prior art approaches.
  • 2. The structuring of the genomic information into specific “layers” of homogeneous data and metadata presents the considerable advantage of enabling the definition of different models of the information sources characterized by low entropy. Such models not only can differ from layer to layer, but can also differ inside each layer when the compressed data within layers are partitioned into Data Blocks included into Access Units. This structuring enables the use of the most appropriate compression for each class of data or metadata and portion of them with significant gains in coding efficiency versus prior art approaches.
  • 3. The information is structured in Access Units (AU) so that any relevant subset of data used by genomic analysis applications is efficiently and selectively accessible by means of appropriate interfaces. These features enable faster access to data and yield a more efficient processing.
  • 4. The definition of a Master Index Table and Local Index Tables enabling selective access to the information carried by the layers of encoded (i.e. compressed) data without the need to decode the entire volume of compressed data.
  • 5. The possibility of performing realignment of already aligned and compressed genomic data when it needs to be re-aligned versus newly published reference genomes by performing an efficient transcoding of selected data portions in the compressed domain. The frequent release of new reference genomes currently requires resource consuming and time for the transcoding processes to re-align already compressed and stored genomic data with respect to the newly published references because all data volume need to be processed.

The method described in this document aims at exploiting the available a-priori knowledge on genomic data to define an alphabet of syntax elements with reduced entropy. In genomics the available knowledge is represented by an existing genomic sequence usually—but not necessarily—of the same species as the one to be processed. As an example, human genomes of different individuals differ only of a fraction of 1%. On the other hand that small amount of data contain enough information to enable early diagnosis, personalized medicine, customized drugs synthesis etc. This invention aims at defining a genomic information representation format where the relevant information is efficiently accessible and transportable and the weight of the redundant information is reduced.

The technical features used in the present invention are:

• • 1. Decomposition of the genomic information into “layers” of homogeneous metadata in order to reduce the information entropy as much as possible;
  • 2. Definition of a Master Index Table and Local Index Tables to enable selective access to the layers of encoded information without the need to decode the entire coded information;
  • 3. Adoption of different source models and entropy coders to code the syntax elements belonging to different layers defined at point 1;
  • 4. Correspondence among dependent layers to enable selective access to the data without the need to decode all the layers if not necessary;
  • 5. Differential encoding with respect to one or more adaptive reference sequences that can be modified to reduce entropy. After a first reference based encoding, the recorded mismatches can be used to “adapt/modify” the reference sequences in order to further reduce the information entropy. This is a process that can be performed iteratively as long as the reduction of information entropy is meaningful.

In order to solve all the aforementioned problems of the prior art (in terms of efficient access to random positions in the file, efficient transmission and storing, efficient compression) the present application re-orders and packs together the data that are more homogeneous and or semantically significant for the easiness of processing.

The present invention also adopts a data structure based on the concept of Access Unit and the multiplexing of the relevant data.

Genomic data are structured and encoded into different access units. Hereafter follows a description of the genomic data that are contained into different access units.

Genomic Data Classification

The sequence reads generated by sequencing machines are classified by the disclosed invention into 5 different “Classes” according to the results of the alignment with respect to one or more reference sequences or genomes.

When aligning a DNA sequence of nucleotides with respect to a reference sequence five are the possible results:

• • 1. A region in the reference sequence is found to match the sequence read without any error (perfect mapping). Such sequence of nucleotides will be referenced to as “perfectly matching read” or denoted as “Class P”
  • 2. A region in the reference sequence is found to match the sequence read with a number of mismatches constituted by a number of positions in which the sequencing machine was not able to call any base (or nucleotide). Such mismatches are denoted by an “N”. Such sequences will be referenced to as “N mismatching reads” or “Class N”.
  • 3. A region in the reference sequence is found to match the sequence read with a number of mismatches constituted by a number of positions in which the sequencing machine was not able to call any base (or nucleotide) OR a different base than the one reported in the reference sequence has been called. Such type of mismatch is called Single Nucleotide Variation (SNV) or Single Nucleotide Polymorphism (SNP). The sequence will be referenced to as “M mismatching reads” or “Class M”.
  • 4. A fourth class is constituted by sequencing reads presenting a mismatch type that includes the same mismatches of class M plus the presence of insertions or deletions (a.k.a. indels). Insertions are represented by a sequence of one or more nucleotides not present in the reference, but present in the read sequence. In literature when the inserted sequence is at the edges of the sequence it is referred to as “soft clipped” (i.e. the nucleotides are not matching the reference but are kept in the aligned reads contrarily to “hard clipped” nucleotides which are discarded). Keeping or discarding nucleotides is typically a user's decisions implemented as a configuration of the aligning tool. Deletion are “holes” (missing nucleotides) in the aligned read with respect to the reference. Such sequences will be referenced to as “I mismatching reads” or “Class I”.
  • 5. A fifth class includes all reads that do now find any valid mapping on the reference sequence according to the specified alignment constraints. Such sequences are said to be Unmapped and belonging to “Class U”

Unmapped reads can be assembled into a single sequence using de-novo assembly algorithms. Once the new sequence has been created unmapped reads can be further mapped with respect to it and be classified in one of the 4 classes P, N, M and I.

The data structure of said genomic data requires the storage of global parameters and metadata to be used by the decoding engine. These data are structured in a main header described in the table below.

TABLE 1
Main Header structure
Element	Type	Description
Unique ID	Byte array	Unique identifier for the encoded
		content
Version	Byte array	Major + Minor version of the encoding
		algorithm
Header Size	Integer	Size in bytes of the entire encoded
		content
Reads Length	Integer	Size of reads in case of constant reads
		length. A special value (e.g. 0) is reserved
		for variable reads length
Ref count	Integer	Number of reference sequences used
Access Units counters	Byte array	Total Number of encoded Access Units
	(e.g. integers)	per reference sequence
Ref ids	Byte array	Unique identifiers for reference
		sequences
Master index table	Byte array	This is a multidimensional array
Alignment positions of first read in each	(e.g. integers)	supporting random access to Access
block (Access Unit).		Units.
I.e. smaller position of the first read on the
reference genome per each block of the 4
classes
1 per pos class (4) per reference

Once the classification of reads is completed with the definition of the Classes, further processing consists in defining a set of distinct syntax elements which represent the remaining information enabling the reconstruction of the DNA read sequence when represented as being mapped on a given reference sequence. A DNA segment referred to a given reference sequence can be fully expressed by:

• • The starting position on the reference sequence pos (292).
  • A flag signaling if the read has to be considered as a reverse complement versus the reference rcomp (293).
  • A distance to the mate pair in case of paired reads pair (294).
  • The value of the read length (295) in case of the sequencing technology produces variable length reads. In case of constant reads length the read length associated to each reads can obviously be omitted and can be stored in the main file header.
  • For each mismatch:
    • Mismatch position nmis (300) for class N, snpp (311) for class M, and indp (321) for class I)
    • Mismatch type (not present in class N, snpt (312) in class M, indt (322) in class I)
  • Flags (296) indicating specific characteristics of the sequence read such as:
    • template having multiple segments in sequencing
    • each segment properly aligned according to the aligner
    • unmapped segment
    • next segment in the template unmapped
    • signalization of first or last segment
    • quality control failure
    • PCR or optical duplicate
    • secondary alignment
    • supplementary alignment
  • Soft clipped nucleotides string (323) when present for class I

This classification creates groups of descriptors (syntax elements) that can be used to univocally represent genome sequence reads. The table below summarizes the syntax elements needed for each class of aligned reads.

TABLE 2
Defined layers per class of data.
	P	N	M	I
	pos	X	X	X	X
	pair	X	X	X	X
	rcomp	X	X	X	X
	flags	X	X	X	X
	rlen	X	X	X	X
	nmis		X
	snpp			X
	snpt			X
	indp				X
	indt				X
	indc				X

Reads belonging to class P are characterized and can be perfectly reconstructed by only a position, a reverse complement information and an offset between mates in case they have been obtained by a sequencing technology yielding mated pairs, some flags and a read length.

The next section details how these descriptors are defined.

Position Descriptors Layer

In each Access Unit, only the mapping position of the first encoded read is stored in the AU header as absolute position on the reference genome. All the other positions are expressed as a difference with respect to the previous position and are stored in a specific layer. This modeling of the information source, defined by the sequence of read positions, is in general characterized by a reduced entropy particularly for sequencing processes generating high coverage results. Once the absolute position of the first alignment has been stored, all positions of other reads are expressed as difference (distance) with respect to the first one.

For example FIG. 4 shows how after encoding the starting position of the first alignment as position “10000” on the reference sequence, the position of the second read starting at position 10180 is coded as “180”. With high coverage data (>50×) most of the descriptors of the position vector will show very high occurrences of low values such as 0 and 1 and other small integers. FIG. 4 shows how the positions of three read pairs are encoded in a pos Layer.

The same source model is used for the positions of reads belonging to classes N, M, P and I. In order to enable any combination of selective access to the data, the positions of reads belonging to the four classes are encoded in separate layers as depicted in Table I.

Pairing Descriptors Layer

The pairing descriptor is stored in the pair layer. Such layer stores descriptors encoding the information needed to reconstruct the originating reads pairs, when the employed sequencing technology produces reads by pairs. Although at the date of the disclosure of the invention the vast majority of sequencing data is generated by using a technology generating paired reads, it is not the case of all technologies. This is the reason for which the presence of this layer is not necessary to reconstruct all sequencing data information if the sequencing technology of the genomic data considered does not generate paired reads information.

Definitions:

mate pair: read associated to another read in a read pair (e.g. Read 2 is the mate pair of Read 1 in the example of FIG. 4)

pairing distance: number of nucleotide positions on the reference sequence which separate one position in the first read (pairing anchor, e.g. last nucleotide of first read) from one position of the second read (e.g. the first nucleotide of the second read)

most probable pairing distance (MPPD): this is the most probable pairing distance expressed in number of nucleotide positions.

position pairing distance (PPD): the PPD is a way to express a pairing distance in terms of the number of reads separating one read from its respective mate present in a specific position descriptor layer.

most probable position pairing distance (MPPPD): is the most probable number of reads separating one read from its mate pair present in a specific position descriptor layer.

position pairing error (PPE): is defined as the difference between the MPPD or MPPPD and the actual position of the mate.

pairing anchor: position of first read last nucleotide in a pair used as reference to calculate the distance of the mate pair in terms of number of nucleotide positions or number of read positions.

FIG. 5 shows how the pairing distance among read pairs is calculated.

The pair descriptor layer is the vector of pairing errors calculated as number of reads to be skipped to reach the mate pair of the first read of a pair with respect to the defined decoding pairing distance.

FIG. 6 shows an example of how pairing errors are calculated, both as absolute value and as differential vector (characterized by lower entropy for high coverages).

The same descriptors are used for the pairing information of reads belonging to classes N, M, P and I. In order to enable the selective access to the different data classes, the pairing information of reads belonging to the four classes are encoded in different layer as depicted in.

Pairing Information in Case of Reads Mapped on Different References

In the process of mapping sequence reads on a reference sequence it is not uncommon to have the first read in a pair mapped on one reference (e.g. chromosome 1) and the second on a different reference (e.g. chromosome 4). In this case the pairing information described above has to be integrated by additional information related to the reference sequence used to map one of the reads. This is achieved by coding

1. A reserved value (flag) indicating that the pair is mapped on two different sequences (different values indicate if read 1 or read 2 are mapped on the sequence that is not currently encoded)

2. a unique reference identifier referring to the reference identifiers encoded in the main header structure as described in Table 1.

3. a third element containing the mapping information on the reference identified at point 2 and expressed as offset with respect to the last encoded position.

FIG. 7 provides an example of this scenario.

In FIG. 7, since Read 4 is not mapped on the currently encoded reference sequence, the genomic encoder signals this information by crafting additional descriptors in the pair layer. In the example shown in FIG. 7 Read 4 of pair 2 is mapped on reference no. 4 while the currently encoded reference is no. 1. This information is encoded using 3 components:

1) One special reserved value is encoded as pairing distance (in this case 0xffffff)

2) A second descriptor provides a reference ID as listed in the main header (in this case 4)

3) The third element contains the mapping information on the concerned reference (170).

Reverse Complement Descriptor Layer

Each read of the read pairs produced by sequencing technologies can be originated from either genome strands of the sequenced organic sample. However, only one of the two strands is used as reference sequence. FIG. 8 shows how in a reads pair one read (read 1) can come from one strand and the other (read 2) can come from the other.

When the strand 1 is used as reference sequence, read 2 can be encoded as reverse complement of the corresponding fragment on strand 1. This is shown in FIG. 9.

In case of coupled reads, four are the possible combinations of direct and reverse complement mate pairs. This is shown in FIG. 10. The rcomp layer codes the four possible combinations.

The same coding is used for the reverse complement information of reads belonging to classes P, N, M, I. In order to enable enhanced selective access to the data, the reverse complement information of reads belonging to the four classes are coded in different layers as depicted in Table 2.

Mismatches of Class N

Class N includes all reads which show mismatches where an ‘N’ is present instead of a base call. All other bases perfectly match on the reference sequence.

The positions of Ns in read 1 are encoded as

• • absolute position in read 1 OR
  • as differential position with respect to the previous N in the same read (whichever has lowest entropy).

The positions of Ns in read 2 are encoded as

• • absolute position in read 2+read 1 length OR
  • differential position with respect to the previous N (whichever has lowest entropy).

In the nmis layer, the encoding of each reads pair is terminated by a special “separator” “S” symbol. This is shown in FIG. 11.

Encoding Substitutions (Mismatches or SNPs)

A substitution is defined as the presence, in a mapped read, of a different nucleotide with respect to the one that is present in the reference sequence at the same position (see FIG. 12).

Each substitution can be encoded as

• • “position” (snpp layer) and “type” (snpt layer). See FIG. 13, FIG. 14, FIG. 16 and FIG. 15. OR
  • “position” only but using one snpp layer per mismatch type. See FIG. 17

Substitutions Positions

A substitution position is calculated as for the values of the nmis layer, i.e.:

In read 1 substitutions are encoded

• • as absolute position in read 1 OR
  • as differential position with respect to the previous substitution in the same read In read 2 substitutions are encoded

In read 2 substitutions are encoded:

• • as absolute position in read 2+read 1 length OR
  • as differential position with respect to the previous substitution FIG. 13 shows how substitutions positions are encoded in layer snpp. Substitutions positions can be calculated either as absolute or as differential values.

In the snpp layer, the encoding of each reads pair is terminated by a special “separator” symbol.

Substitutions Types Descriptors

For class M (and I as described in the next sections), mismatches are coded by an index (moving from right to left) from the actual symbol present in the reference to the corresponding substitution symbol present in the read {A, C, G, T, N, Z}. For example if the aligned read presents a C instead of a T which is present at the same position in the reference, the mismatch index will be denoted as “4”. The decoding process reads the encoded syntax element, the nucleotide at the given position on the reference and moves from left to right to retrieve the decoded symbol. E.g. a “2” received for a position where a G is present in the reference will be decoded as “N”. FIG. 14 shows all the possible substitutions and the respective encoding symbols when IUPAC ambiguity codes are not used and FIG. 15 provides an example of encoding of substitutions types in the snpt layer.

In case of presence of IUPAC ambiguity codes, the substitution indexes change as shown in FIG. 16.

In case the encoding of substation types described above presents high information entropy, an alternative method of substitution encoding consists in storing only the mismatches positions in separate layers, one per nucleotide, as depicted in FIG. 17.

Coding of Insertions and Deletions

For class I, mismatches and deletions are coded by an indexes (moving from right to left) from the actual symbol present in the reference to the corresponding substitution symbol present in the read: {A, C, G, T, N, Z}. For example if the aligned read presents a C instead of a T present at the same position in the reference, the mismatch index will be “4”. In case the read presents a deletion where a A is present in the reference, the coded symbol will be “5”. The decoding process reads the coded syntax element, the nucleotide at the given position on the reference and moves from left to right to retrieve the decoded symbol. E.g. a “3” received for a position where a G is present in the reference will be decoded as “Z” which indicates the presence of a deletion in the sequence read.

Inserts are coded as 6, 7, 8, 9, 10 respectively for inserted A, C, G, T, N.

In case of adoption of the IUPAC ambiguity codes the substitution mechanism results to be exactly the same however the substitution vector is extended as: S={A, C, G, T, N, Z, M, R, W, S, Y, K, V, H, D, B}.

FIG. 18 and FIG. 19 show examples of how to encode substitutions, inserts and deletions in a reads pair of class I.

The following structures of file format, access units and multiplexing are described referring to the coding elements disclosed here above. However, the access units, the file format and the multiplexing produce the same technical advantage also with other and different algorithms of source modeling and genomic data compression.

File Format: Selective Access to Regions of Genomic Data

Master Index Table

In order to support selective access to specific regions of the aligned data, the data structure described in this document implements an indexing tool called Master Index Table (MIT). This is a multi-dimensional array containing the loci at which specific reads map on the used reference sequences. The values contained in the MIT are the mapping positions of the first read in each pos layer so that non-sequential access to each Access Unit is supported. The MIT contains one section per each class of data (P, N, M and I) and per each reference sequence. The MIT is contained in the Main Header of the encoded data. FIG. 20 shows the generic structure of the Main Header, FIG. 21 shows a generic visual representation of the MIT and FIG. 22 shows an example of MIT for the class P of encoded reads.

The values contained in the MIT depicted in FIG. 22 are used to directly access the region of interest (and the corresponding AU) in the compressed domain.

For example, with reference to FIG. 22, if it is required to access the region comprised between position 150,000 and 250,000 on reference 2, a decoding application would skip to the second reference in the MIT and would look for the two values k1 and k2 so that k1<150,000 and k2>250,000. Where k1 and k2 are 2 indexes read from the MIT. In the example of FIG. 22 this would result in positions 3 and 4 of the second vector of the MIT. These returned values will then be used by the decodingapplication to fetch the positions of the appropriate data from the pos layer Local Index Table as described in the next section.

Together with pointers to the layer containing the data belonging to the four classes of genomic data described above, the MIT can be uses as an index of additional metadata and/or annotations added to the genomic data during its life cycle.

Local Index Table

Each data layer described above is prefixed with a data structure referred to as local header. The local header contains a unique identifier of the layer, a vector of Access Units counters per each reference sequence, a Local Index Table (LIT) and optionally some layer specific metadata. The LIT is a vector of pointers to the physical position of the data belonging to each AU in the layer payload. FIG. 23 depicts the generic layer header and payload where the LIT is used to access specific regions of the encoded data in a non-sequential way.

In the previous example, in order to access region 150,000 to 250,000 of reads aligned on the reference sequence no. 2, the decoding application retrieved positions 3 and 4 from the MIT. These values shall be used by the decoding process to access the 3^rd and 4^th elements of the corresponding section of the LIT. In the example shown in FIG. 24, the Total Access Units counters contained in the layer header are used to skip the LIT indexes related to AUs related to reference 1 (5 in the example). The indexes containing the physical positions of the requested AUs in the encoded stream are therefore calculated as:

position of the data blocks belonging to the requested AU=data blocks belonging to AUs of reference 1 to be skipped+position retrieved using the MIT, i.e.

First block position: 5+3=8

Last block position: 5+4=9

The blocks of data retrieved using the indexing mechanism called Local Index Table, are part of the Access Units requested.

FIG. 26 shows how the data blocks retrieved using the MIT and the LIT compose one or more Access Units.

Access Units

The genomic data classified in data classes and structured in compressed or uncompressed layers are organized into different access units.

Genomic Access Units (AU) are defined as sections of genome data (in a compressed or uncompressed form) that reconstructs nucleotide sequences and/or the relevant metadata, and/or sequence of DNA/RNA (e.g. the virtual reference) and/or annotation data generated by a genome sequencing machine and/or a genomic processing device or analysis application.

An Access Unit is a block of data that can be decoded either independently from other Access Units by using only globally available data (e.g. decoder configuration) or by using information contained in other Access Units. Access Units contain data information related to genomic data in the form of positional information (absolute and/or relative), information related to reverse complement and possibly pairing and additional data. It is possible to identify several types of access units.

Access units are differentiated by:

• • type, characterizing the nature of the genomic data and data sets they carry and the way they can be accessed,
  • order, providing a unique order to access units belonging to the same type.

Access units of any type can be further classified into different “categories”.

Hereafter follows a non-exhaustive list of definition of different types of genomic access units:

• • 1) Access units of type 0 do not need to refer to any information coming from other access units to be accessed or decoded and accessed (see FIG. 29). The entire information carried by the data or data sets they contain can be independently read and processed by a decoding device or processing application.
  • 2) Access units of type 1 contain data that refer to data carried by access units of type 0 (see FIG. 30). Reading or decoding and processing the data contained in access units of type 1 requires having access to one or more access units of type 0.

Access units of this type can contain information of mismatching or dissimilarity or non-correspondence with respect to the information contained in the access unit of type 0.

• • 3) Access units of type 2, 3 and 4 contain data that refer to an access unit of type 1 (see FIG. 31, FIG. 32 and FIG. 33). Reading or decoding and processing the data or data sets contained by access units of type 2, 3, and 4 requires information carried by the data or data sets contained in an access units of type 0 and 1. The difference among types 2, 3, and 4 access units relies in the nature of the information they contain.
  • 4) Access units of type 5 contain metadata (e.g. quality scores) and/or annotation data associated to the data or data sets contained in the access unit of type 1. Access units of type 5 may be classified and labelled in different layers.
  • 5) Access units of type 6 contain data or data sets classified as annotation data. Access units of type 6 may be classified and labelled in layers.
  • 6) Access units of additional types can extend the structure and mechanisms described here. As an example, but not as a limitation, the results of genomic variant calling, structural and functional analysis can be encoded in access units of new types. The data organization in Access Units described herein does not prevent any type of data to be encapsulated in Access Units being the mechanism completely transparent with respect to the nature of encoded data.
  • Access units of this type can contain information of mismatching or dissimilarity or non-correspondence with respect to the information contained in the access unit of type 0.

FIG. 28 shows how Access Units are composed by a header and one or more layers of homogeneous data. Each layer can be composed by one or more blocks. Each block contains several packets and the packets are a structured sequence of the descriptors introduced above to represent e.g. reads positions, pairing information, reverse complement information, mismatches positions and types etc.

Each Access unit can have a different number of packets in each block, but within an Access Unit all blocks have the same number of packets.

Each data packet can be identified by the combination of 3 identifiers X Y Z where:

• • X identifies the access unit it belongs to
  • Y identifies the block it belongs to (i.e. the data type it encapsulates)
  • Z is an identifier expressing the packet order with respect to other packets in the same block

FIG. 28 shows an example of Access Units and packets labelling.

FIG. 34 to FIG. 38 show Access Units of several types, the common syntax to denote them is the following: AU_T_N is an access unit of type T with identifier N which may or may not imply a notion of order according to the Access Unit Type. Identifiers are used to uniquely associate Access Units of one type with those of other types required to completely decode the carried genomic data.

Access units of any type can be classified and labelled in different “categories” according to different sequencing processes. For example, but not as a limitation, classification and labelling can take place when

• • sequencing the same organism at different times (access units contain genomic information with a “temporal” connotation),
  • sequencing organic samples of different nature of the same organisms (e.g. skin, blood, hair for human samples). These are access units with “biological” connotation.

The access units of type 1, 2, 3 and 4 are built according to the result of a matching function applied on genome sequence fragments (a.k.a. reads) with respect to the reference sequence encoded in Access Units of type 0 they refer to.

For example access units (AUs) of type 1 (see FIG. 30) may contain the positions and the reverse complement flags of those reads which result in a perfect match (or maximum possible score corresponding to the selected matching function) when a matching function is applied to specific regions of the reference sequence encoded in AUs of type 0. Together with the data contained in AUs of type 0, such matching function information is sufficient to completely reconstruct all genome sequence reads represented by the data set carried by the access units of type 1.

With reference to the genomic data classification previously described in this document, the Access Units of type 1 described above would contain information related to genomic sequence reads of class P (perfect matches).

In case of variable reads length and paired reads the data contained in AUs of type 1 mentioned in the previous example, have to be integrated with the data representing the information about reads pairing and reads length in order to be able to completely reconstruct the genomic data including the reads pairs association. With respect to the data classification previously introduced in the present document, pair and rlen layers would be encoded in AU of type 1.

The matching functions applied with respect to access units of type 1 to classify the content of AU for the type 2, 3 and 4 can provide results such as:

• • each sequence contained in the AU of type 1 perfectly matches sequences contained in the AU of type 0 in correspondence to the specified position;
  • each sequence contained in the AU of type 2 perfectly matches a sequence contained in the AU of type 0 in correspondence to the specified position, except for the “N” symbols present (base not called by the sequencing device) in the sequence in the AU of type 2;
  • each sequence contained in the AU of type 3 includes variants in the form of substituted symbols (variants) with respect to the sequence contained in the AU of type 0 in correspondence to the specified position;
  • each sequence contained in the AU of type 4 includes variants in the form of substituted symbols (variants), insertions and/or deletions with respect to the sequence contained in the AU of type 0 in correspondence to the specified position.

Access units of type 0 are ordered (e.g. numbered), but they do not need to be stored and/or transmitted in an ordered manner (technical advantage: parallel processing/parallel streaming, multiplexing)

Access units of type 1, 2, 3 and 4 do not need to be ordered and do not need to be stored and/or transmitted in an ordered manner (technical advantage: parallel processing/parallel streaming).

Technical Effects

The technical effect of structuring genomic information in access units as described here is that the genomic data:

1. can be selectively queried in order to access:

• • specific “categories” of data (e.g. with a specific temporal or biological connotation) without having to decompress the entire genomic data or data sets and/or the related metadata.
  • specific regions of the genome for all “categories”, a subset of “categories”, a single “category” (with or without the associated metadata) without the need to decompress other regions of the genome

2. can be incrementally updated with new data that can be available when:

• • new analysis is performed on the genomic data or data sets
  • new genomic data or data sets are generated by sequencing the same organisms (different biological samples, different biological sample of the same type, e.g. blood sample, but acquired at a different time, etc.)

3. can be efficiently transcoded to a new data format in case of

• • new genomic data or data sets to be used as new reference (e.g. new reference genome carried by AU of type 0)
  • update of the encoding format specification

With respect to prior art solutions such as SAM/BAM, the aforementioned technical features address the issues of requiring data filtering to happen at the application level when the entire data has been retrieved and decompressed from the encoded format.

Hereafter follows examples of application scenario where the access unit structure becomes instrumental for a technological advantage.

Selective Access

In particular the disclosed data structure based on Access Units of different types enables to

• • extract only the read information (data or data sets) of the whole sequencing of all “categories” or a subset (i.e. one or more layers) or a single “category” without having to decompress also the associated metadata information (limitation of current state of the art: SAM/BAM that cannot even support distinction between different categories or layers)
  • extract all the reads aligned on specific regions of the assumed reference sequence for all categories, subsets of the categories, a single category (with or without the associated metadata) without the need to decompress also other regions of the genome (limitation of current state of the art: SAM/BAM);

FIG. 39 shows how the access to the genomic information mapped on the second segment of the reference sequence (AU 0-2) with mismatches only requires the decoding of AUs 0-2, 1-2 and 3-2 only. This is an example of selective access according to both a criteria related to a mapping region (i.e. position on the reference sequence) and a criteria related to the matching function applied to the encoded sequence reads with respect to the reference sequence (e.g. mismatches only in this example).

A further technical advantage is that the querying on the data is much more efficient in terms of data accessibility and execution speed because it can be based on accessing and decoding only selected “categories”, specific regions of longer genomic sequences and only specific layers for access units of type 1, 2, 3, 4 that match the criteria of the applied queries and any combination thereof.

The organization of access units of type 1, 2, 3, 4 into layers allow for efficient extraction of nucleotides sequences

• • with specific variations (e.g. mismatches, insertions, deletions) with respect to one or more reference genomes ;
  • that do not map to any of the considered reference genomes;
  • that perfectly map on one or more reference genomes;
  • that map with one or more accuracy levels

Incremental Update

The access units of type 5 and 6 allow for easy insertion of annotations without the need to depacketize/decode/decompress the whole file thereby adding to the efficient handling of the file which is a limitation of prior art approaches. Existing compression solutions may have to access and process a large amount of compressed data before the desired genomic data can be accessed. This will cause inefficient RAM bandwidth utilization and more power consumption also in hardware implementations. Power consumption and memory access issues may be alleviated by using the approach based on Access Units described here.

The data indexing mechanism described in the Master Index Table (see FIG. 21) together with the utilization of Access Unites enables incremental update of the encoded content as described below.

Insertion of Additional Data

New genomic information can be periodically added to existing genomic data for several reasons. For example when:

• • An organism is sequenced at different moments in time;
  • Several different samples of the same individual are sequenced at the same time;
  • New data generated by a sequencing process (streaming).

In the above mentioned situations, structuring data using the Access Units described here and the data structure described in the file format section enables the incremental integration of the newly generated data without the need to re-encode the existing data. The incremental update process can be implemented as follows:

• • 1. The newly generated AUs can simply be concatenated in the file with the pre-existing AUs and
  • 2. the indexing of the newly generated data or data sets are included in the Master Index Table described in the file format section of this document. One index shall position the newly generated AU on the existing reference sequence, other indexes consist in pointers of the newly generated AUs in the physical file to enable direct and selective access to them.

This mechanism is illustrated in FIG. 40 where pre-existing data encoded in 3 AUs of type 1 and 4 AUs per each type from 2 to 4 are updated with 3 AUs per type with encoding data coming for example from a new sequence run for the same individual.

In the specific use case of streaming genomic data and data sets in compressed form, the incremental update of a pre-existing data set may be useful when analysing data as soon as they are generated by a sequencing machine and before the actual sequencing is completed. An encoding engine (compressor) can assemble several AUs in parallel by “clustering” sequence reads that map on the same region of the selected reference sequence. Once the first AU contains a number of reads above a pre-configured threshold/parameter, the AU is ready to be sent to the analysis application. Together with the newly encoded Access Unit, the encoding engine (the compressor) shall make sure that all Access Units the new AU depends on have already been sent to the receiving end or is sent together with it. For example an AU of type 3 will require the appropriate AU of type 0 and type 1 to be present at the receiving end in order to be properly decoded.

By means of the described mechanism, a receiving variant calling application would be able to start calling variants on the AU received before the sequencing process has been completed at the transmitting side. A schematic of this process is depicted in FIG. 41.

New Analysis of Results.

During the genome processing life cycle several iterations of genome analysis can be applied on the same data (e.g. different variant calling using different processing algorithm). The use of AUs as defined in this document and the data structure described in the file format section of this document enable incremental update of existing compressed data with the results of new analysis.

For example, new analysis performed on existing compressed data can produce new data in these cases:

• • 1. A new analysis can modify existing results already associated with the encoded data. This use case is depicted in FIG. 42 and it is implemented by moving entirely or partially the content of one Access Unit from one type to another. In case new AUs need to be created (due to a pre-defined maximum size per AU), the related indexes in the Master Index Table must be created and the related vector is sorted when needed.
  • 2. New data are produced from new analysis and have to be associated to existing encoded data. In this case new AUs of type 5 can be produced and concatenated with the existing vector of AUs of the same type. This and the related update of the Master Index Table are depicted in FIG. 43.

The use cases described above and depicted in FIG. 42 and FIG. 43 are enabled by:

• • 1. The possibility to have direct access only to data with poor mapping quality (e.g. AUs of type 4);
  • 2. The possibility to remap reads to a new genomic region by simply creating a new Access Unit possibly belonging to a new type (e.g. reads included in a Type 4 AU can be remapped to a new region with less (type 2-3) mismatches and included in a newly created AU);
  • 3. The possibility to create AU of type 6 containing only the newly created analysis results and/or related annotations. In this case the newly created AUs only require to contain “pointers” to the existing AUs to which they refer to.

Transcoding

Compressed genomic data can require transcoding, for example, in the following situations:

• • Publication of new reference sequences;
  • Use of a different mapping algorithm (re-mapping).

When genomic data is mapped on an existing public reference genome, the publication of a new version of said reference sequence or the desire to map the data using a different processing algorithm, today requires a process of re-mapping. When remapping compressed data using prior art file formats such as SAM or CRAM the entire compressed data has to be decompressed into its “raw” form to be mapped again with reference to the newly available reference sequence or using a different mapping algorithm. This is true even if the newly published reference is only slightly different from the previous or the different mapping algorithm used produces a mapping that is very close (or identical) to the previous mapping.

The advantage of transcoding genomic data structured using Access Units described here is that:

• • 1. Mapping versus a new reference genome only requires re-encoding (decompressing and compressing) the data of AUs that map on the genome regions that have changes. Additionally the user may select those compressed reads that for any reason might need to be re-mapped even if they originally do not map on the changed region (this may happen if the user believes that the previous mapping is of poor quality). This use case is depicted in FIG. 44.
  • 2. In case the newly published reference genome differs from the previous only in terms of entire regions shifted to different genomic locations (“loci”), the transcoding operation results particularly simple and efficient. In fact in order to move all the reads mapped to the “shifted” region it is sufficient to change only the value of the absolute position contained in the related (set of) AU(s) header. Each AU header contain the absolute position the first read contained in the AU is mapped to on the reference sequence, while all other reads positions are encoded differentially with respect to the first. Therefore, by simply updating the value of the absolute position of the first read, all the reads in the AU are moved accordingly. This mechanism cannot be implemented by state of the art approaches such as CRAM and BAM because genome data positions are encoded in the compressed payload, thus requiring complete decompression and re-compression of all genome data sets.
  • 3. When a different mapping algorithm is used, it is possible to apply it only on a portion of compressed reads that was deemed mapped with poor quality. For example it can be appropriate to apply the new mapping algorithm only on reads which did not perfectly match on the reference genome. With existing formats today it is not possible (or it's only partially possible with some limitations) to extract reads according to their mapping quality (i.e. presence and number of mismatches). If new mapping results are returned by the new mapping tools the related reads can be transcoded from one AU from another of the same type (FIG. 46) or from one AU of one type to an AU of another type (FIG. 45).

Moreover, prior art compression solutions may have to access and process a large amount of compressed data before the desired genomic data can be accessed. This will cause inefficient RAM bandwidth utilization and more power consumption and in hardware implementations. Power consumption and memory access issues may be alleviated by using the approach based on Access Units described here.

A further advantage of the adoption of the genomic access units described here is the facilitation of parallel processing and suitability for hardware implementations. Current solutions such as SAM/BAM and CRAM are conceived for single-threaded software implementation.

Selective Encryption

The approach based on Access Units organized in several types an layers as described in this document enables the implementation of content protection mechanisms otherwise not possible with state of the art monolithic solutions.

A person skilled in the art knows that the majority of genomic information related to an organism's genetic profile relies in the differences (variants) with respect to a known sequence (e.g. a reference genome or a population of genomes). An individual genetic profile to be protected from unauthorized access will therefore be encoded in Access Units of type 3 and 4 as described in this document. The implementation of controlled access to the most sensible genomic information produced by a sequencing and analysis process can therefore be realized by encrypting only the payload of AUs of type 3 and 4 (see FIG. 47 for an example). This will generate significant savings in terms of both processing power and bandwidth since the resources consuming encryption process shall be applied on a subset of data only.

Transport of Genomic Access Units

Genomic Data Multiplex

Genomic Access Units can be transported over a communication network within a Genomic Data Multiplex. A Genomic Data Multiplex is defined as a sequence of packetized genomic data and metadata represented according to the data classification disclosed as part of this invention, transmitted in network environments where errors, such as packet losses, may occur.

The Genomic Data Multiplex is conceived to ease and render more efficient the transport of genomic coded data over different environments (typically network environments) and has the following advantages not present in state of the art solutions:

• • 1. it enables encapsulation of either a stream or a sequence of genomic data (described below) or Genomic File Format generated by an encoding tool into one or more Genomic Data Multiplex, in order to carry it over a network environment, and then recover a valid and identical stream or file format in order to render the transmission and access to information more efficient
  • 2. It enables selective retrieval of encoded genomic data from the encapsulated Genomic Data Streams, for decoding and presentation.
  • 3. It enables multiplexing several Genomic Datasets into a single container of information for transport and it enables de-multiplexing a subset of the carried information into a new Genomic Data Multiplex.
  • 4. It enables the multiplexing of data and metadata produced by different sources (with the consequent separate access) and/or sequencing/analysis processes and transmit the resulting Genomic Data Multiplex over a network environment.
  • 5. It supports identification of errors such as packet losses.
  • 6. It supports proper reorder data which may arrive out of order due to network delays, therefore rendering more efficient the transmission of genomic data when compared with the state of the art solutions

An example of genomic data multiplexing is shown in FIG. 49.

Genomic Dataset

In the context of the present invention a Genomic Dataset is defined as a structured set of Genomic Data including, for example, genomic data of a living organism, one or more sequences and metadata generated by several steps of genomic data processing, or the result of the genomic sequencing of a living organism. One Genomic Data Multiplex may include multiple Genomic Datasets (as in a multi-channel scenario) where each dataset refers to a different organism. The multiplexing mechanism of the several datasets into a single Genomic Data Multiplex is governed by information contained in data structures called Genomic Dataset List (GDL) and Genomic Dataset Mapping Table (GDMT).

Genomic Dataset List

A Genomic Dataset List (GDL) is defined as a data structure listing all Genomic Datasets available in a Genomic Data Multiplex. Each of the listed Genomic Datasets is identified by a unique value called Genomic Dataset ID (GID).

Each Genomic Dataset listed in the GDL is associated to:

• • one Genomic Data Stream carrying one Genomic Dataset Mapping Table (GDMT) and identified by a specific value of Stream ID (genomic_dataset_map_SID);
  • one Genomic Data Stream carrying one Reference ID Mapping Table (RIDMT) and identified by a specific value of Stream ID (reference_id_map_SID).

The GDL is sent as payload of a single Transport Packet at the beginning of a Genomic Data Stream transmission; it can then be periodically re-transmitted in order to enable random access to the Stream.

The syntax of the GDL data structure is provided in the table below with an indication of the data type associated to each syntax element.

Syntax Data type
genomic_dataset_list( ) {
       list_length               bitstring
       multiplex_id              bitstring
       version_number            bitstring
       applicable_section_flag   bit
       list_ID                   bitstring
       for (i = 0; i < N; i++) { N = number of Genomic
       Datasets in this
       Genomic Multiplex
       genomic_dataset_ID        bitstring
       genomic_dataset_map_SID   bitstring
       reference_id_map_SID      bitstring
       }
       CRC_32                    bitstring
}

The syntax elements composing the GDL described above have the following meaning and function.

section_length          bitstring field, specifying the number of bytes composing the section, starting
                        immediately following the section_length field, and including the CRC.
multiplex_id            bitstring field which serves as a label to identify this multiplexed stream from
                        any other multiplex within a network.
version_number          bitstring field indicating the version number of the whole Genomic Dataset List
                        Section. The version number shall be incremented by 1 whenever the definition
                        of the Genomic Dataset Mapping Table changes. Upon reaching the value 127,
                        it wraps around to 0. When the applicable_section_flag is set to ‘1’, then the
                        version_number shall be that of the currently applicable Genomic Dataset List.
                        When the applicable_section_flag is set to ‘0’, then the version_number shall
                        be that of the next applicable Genomic Dataset List.
applicable_section_flag A 1 bit indicator, which when set to ‘1’ indicates that the Genomic Dataset
                        Mapping Table sent is currently applicable. When the bit is set to ‘0’, it indicates
                        that the table sent is not yet applicable and shall be the next table to become
                        valid.
list_ID                 This is a bitstring field identifying the current genomic dataset list.
genomic_dataset_ID      genomic_dataset_ID is a bitstring field which specifies the genomic dataset to
                        which the genomic_dataset_map_SID is applicable. This field shall not take any
                        single value more than once within one version of the Genomic Dataset
                        Mapping Table.
genomic_dataset_map_SID genomic_dataset_map_SID is a bitstring field identifying the Genomic Data
                        Stream carrying the Genomic Dataset Mapping Table (GDMT) associated to this
                        Genomic Dataset. No genomic_dataset_ID shall have more than one
                        genomic_dataset_map_SID associated. The value of the
                        genomic_dataset_map_SID is defined by the user.
reference_id_map_SID    reference_id_map_SID is a bitstring field identifying the Genomic Data Stream
                        carrying the Reference ID Mapping Table (RIDMT) associated to this Genomic
                        Dataset. No genomic_dataset_ID shall have more than one
                        reference_id_map_SID associated. The value of the reference_id_map_SID is
                        defined by the user.
CRC_32                  This is a bitstring field that contains an integrity check value for the entire GDL.
                        One typical algorithm used for this purpose function is the CRC32 algorithm
                        producing a 32 bit value.

Genomic Dataset Mapping Table

The Genomic Dataset Mapping Table (GDMT) is produced and transmitted at the beginning of a streaming process (and possibly periodically re-transmitted, updated or identical in order to enable the update of correspondence points and the relevant dependencies in the streamed data). The GDMT is carried by a single Packet following the Genomic Dataset List and lists the SIDs identifying the Genomic Data Streams composing one Genomic Dataset. The GDMT is the complete collection of all identifiers of Genomic Data Streams (e.g., the genomic sequence, reference genome, metadata, etc) composing one Genomic Dataset carried by a Genomic Multiplex. A genomic dataset mapping table is instrumental in enabling random access to genomic sequences by providing the identifier of the stream of genomic data associated to each genomic dataset.

The syntax of the GDMT data structure is provided in the table below with an indication of the data type associated to each syntax element.

genomic_dataset_mapping_table( ) {
table_length              bitstring
genomic_dataset_ID        bitstring
version_number            bitstring
applicable_section_flag   bit
mapping_table_ID          bitstring
genomic_dataset_ef_length bitstring
for (i=0; i<N; i++) {     N = number of extension
fields associated to this
Genomic Dataset
extension_field( )        data structure
}
for (i = 0;i < M ; i++) { M = number of Genomic
Data Streams associated to
this specific Dataset
data_type                 bitstring
genomic_data_SID          bitstring
gd_component_ef_length    bitstring
for (I = 0; I < K; i++) { K = number of extension
fields associated to each
Genomic Data Stream
extension_field ( )       data structure
}
}
CRC_32                    bitstring
}

The syntax elements composing the GDMT described above have the following meaning and function.

version_number,           These elements have the same meaning as for the GDL
applicable_section_flag
table_length,             bitstring field specifying the number of bytes composing the table, starting
                          after the table_length field, and including the CRC_32 field.
genomic_dataset_ID        bitstring field identifying a Genomic Dataset
mapping_table_ID          bitstring bit field identifying the current Genomic Dataset Mapping Table
genomic_dataset_ef_length bitstring field specifying the number of bytes of the optional
                          extension_field associated with this Genomic Dataset
data_type                 bitstring field specifying the type of genomic data carried by the packets
                          identified by the genomic_data_SID.
genomic_data_SID          bitstring bit field specifying the Stream ID of the packets carrying the
                          encoded genomic data associated with one component of this Genomic
                          Dataset (e.g. read p positions, read p pairing information etc. as defined in
                          this invention)
gd_component_ef_length    bitstring field specifying the number of bytes of the optional
                          extension_field associated with the genomic Stream identified by
                          genomic_data_SID.
CRC_32                    This is a bitstring field that contains an integrity check value for the entire
                          GDMT. One typical algorithm used for this purpose function is the CRC32
                          algorithm producing a 32 bit value.

extension_fields are optional descriptors that might be used to further describe either a Genomic Dataset or one Genomic Dataset component.

Reference ID Mapping Table

The Reference ID Mapping Table (RIDMT) is produced and transmitted at the beginning of a streaming process. The RIDMT is carried by a single Packet following the Genomic Dataset List. The RIDMT specifies a mapping between the numeric identifiers of reference sequences (REFID) contained in the Block header of an access unit and the (typically literal) reference identifiers contained in the main header specified in Table 1.

The RIDMT can be periodically re-transmitted in order to:

• • enable the update of correspondence points and the relevant dependencies in the streamed data,
  • support the integration of new reference sequences added to the pre-existing ones (e.g. synthetic references created by de-novo assembly processes)

The syntax of the RIDMT data structure is provided in the table below with an indication of the data type associated to each syntax element.

Syntax Data type
reference_id_mapping_table( ) {
       table_length                  bitstring
       genomic_dataset_ID            bitstring
       version_number                bitstring
       applicable_section_flag       bit
       reference_id_mapping_table_ID bitstring
       for (i = 0; i < N; i++) {     N = number of reference
       sequences associated with
       the Genomic Dataset
       identified by
       genomic_dataset_ID
       ref_string_length             bitstring
       for (i=0;i<ref_string_length;i++){
       ref_string[i]                 byte
       }
       REFID                         bitstring
       }
       CRC_32                        bitstring
}

The syntax elements composing the RIDMT described above have the following meaning and function.

table_length, genomic_dataset_ID, These elements have the same meaning as for the GDMT
version_number, applicable_section_flag
reference_id_mapping_table_ID    bitstring field identifying the current Reference ID Mapping
                                 Table
ref_string_length                bitstring field specifying the number of characters (bytes)
                                 composing ref_string, excluding the end of string (‘\0’)
                                 character.
ref_string[i]                    byte field encoding each character of the string
                                 representation of a reference sequence (e.g. “chr1” for
                                 chromosome 1). The end of string (‘\0’) character is not
                                 necessary, as it is implicitly inferred from the
                                 ref_string_length field
REFID                            This is a bitstring field uniquely identifying a reference
                                 sequence. This is encoded in the data Block header as
                                 REFID field.
CRC_32                           This is a bitstring field that contains an integrity check value
                                 for the entire RIDMT. One typical algorithm used for this
                                 purpose function is the CRC32 algorithm producing a 32 bit
                                 value.

Genomic Data Stream

A Genomic Data Multiplex contains one or several Genomic Data Streams where each stream can transport

• • data structures containing transport information (e.g. Genomc Dataset List, Genomic Dataset Mapping Table etc.)
  • data belonging to one of the Genomic Data Layers described in this invention.
  • Metadata related to the genomic data
  • Any other data

A Genomic Data Stream containing genomic data is essentially a packetized version of a Genomic Data Layer where each packet is prepended with a header describing the packet content and how it is related to other elements of the Multiplex.

The Genomic Data Stream format described in this document and the File Format defined in this invention are mutually convertible. Whereas a full file format can be reconstructed in full only after all data have been received, in case of streaming a decoding tool can reconstruct and access, and start processing the partial data at any time.

A Genomic Data Stream is composed by several Genomic Data Blocks each containing one or more Genomic Data Packet. Genomic Data Blocks (GDBs) are containers of genomic information composing one genomic AU. GDB can be split into several Genomic Data Packets, according to the communication channel requirements. Genomic access units are composed by one or more Genomic Data Blocks belonging to different Genomic Data Streams.

Genomic Data Packets (GDPs) are transmission units composing one GDB. Packet size is typically set according to the communication channel requirements.

FIG. 27 shows the relationship among Genomic Multiplex, Streams, Access Units, Blocks and Packets when encoding data belonging to the P class as defined in this invention. In this example three Genomic Streams encapsulate information on position, pairing and reverse complement of sequence reads.

Genomic Data Blocks are composed by a header, a payload of compressed data and padding information. The table below provides an example of implementation of a GDB header with a description of each field and a typical data type.

Data type	Description	Data type
Block Start Code	Reserved value used to unambiguously identify the beginning of a	bitstring
Prefix (BSCP)	Genomic Data Block.
Format Identifier	Unambiguously identifies the Genomic Data Layer the block belongs to.	bitstring
(FI)
POS Flag (PSF)	If the POS Flag is set, the block contains the 40 bit POS field at the end	bit
	of the block header and before the optional fields.
Padding Flag	If the Padding Flag is set, the block contains additional padding bytes	bit
(PDF)	after the payload which are not part of the payload.
Block size (BS)	Number of bytes composing the block, including this header and	bitstring
	payload, and excluding padding (total block size will be BS + padding
	size).
Access Unit ID	Unambiguous ID, linearly increasing (not necessarily by 1, even though	bitstring
(AUID)	recommended). Needed to implement proper random access, as
	described in the Master Index Table defined in this invention.
(Optional)	Unambiguous ID, identifying the reference sequence the AU containing	bitstring
Reference ID	this block refers to. This is needed, along with POS field, to have proper
(REFID)	random access, as described in the Master Index Table.
(Optional) POS	Present if PSF is 1. Position on the reference sequence of the first read	bitstring
(POS)	in the block.
(Extra optional	Additional optional fields, presence signaled by BS.	bytestring
fields)
Payload	Block of encoded genomic information (syntax elements as described in	bytestring
	this invention
(Optional)	(Optional, presence signaled by PDF) Fixed bitstring value that can be	bitstring
Padding	inserted in order to meet the channel requirements. If present, the first
	byte indicates how many bytes compose the padding. It is discarded by
	the decoder.

The use of AUID, POS and BS enables the decoder to reconstruct the data indexing mechanisms referred to as Master Index Table (MIT) and Local Index Table (LIT) in this invention. In a data streaming scenario the use of AUID and BS enables the receiving end to dynamically re-create a LIT locally, without the need to send extra-data. The use of AUID, BS and POS will enable to recreate a MIT locally without the need to send additional data. This has the technical advantage to

• • reduce the encoding overhead which might be large if the entire LIT is transmitted;
  • avoid the need of a complete mapping between genomic positions and Access Units which is not normally available in a streaming scenario

A Genomic Data Block can be split into one or more Genomic Data Packets, depending on network layer constraints such as maximum packet size, packet loss rate, etc. A Genomic Data Packet is composed by a header and a payload of encoded or encrypted genomic data as described in the table below.

Data type	Description	Data size
Stream ID (SID)	Unambiguously identifies data type carried by this packet. A Genomic	bitstring
	Dataset Mapping Table is needed at the beginning of the stream in
	order to map Stream IDs to data types. Used also for updating
	correspondence points and relevant dependencies.
Access Unit	Set for the last packet of the access unit. Allows to identify the last	bit
Marker Bit	packet of an AU.
(MB)
Packet Counter	Counter associated to each Stream ID linearly increasing by 1. Needed	bitstring
Number (SN)	to identify gaps/packet losses. Wrap around at 255.
Packet Size	Number of bytes composing the packet, including header, optional	bitstring
(PS)	fields and payload.
Extension Flag	Set if extension fields are present.	bit
(EF)
Extension	Optional fields, presence signaled by PS.	bytestring
Fields
Payload	Block data (entire block or fragment)	bytestring

The Genomic Multiplex can be properly decoded only when at least one Genomic Dataset List, one Genomic Dataset Mapping Table and one Reference ID Mapping Table have been received, allowing to map every packet to a specific Genomic Dataset component.

Multiplex Encoding Process

FIG. 49 shows how before being transformed in the data structures presented in this invention, raw genomic sequence data need to be mapped on one or more reference sequence known a-priori (493). In case a reference sequence is not available a synthetic reference can be built from the raw sequence data (490). This process is known as de-novo assembly. Already aligned data can be re-aligned in order to reduce the information entropy (492). After alignment, a genomic classificator (494) creates the data classes according to a matching function of the sequence reads on one or more reference sequences and separates metadata (432) (e.g. quality values) and annotation data (431) from the genomic sequences. A data parser (495) generates then the Access Units described in this invention and sends them to the Genomic Multiplexer (496) which generates the Genomic Multiplex.

Claims

1. A method for the transmission of genomic data as multiplexed data streams comprising a genomic dataset list data structure for providing a list of all genomic datasets, said genomic datasets comprising genomic data available in the genomic streams;a genomic dataset mapping table data structure for providing the identifier of each stream of said genomic data associated to each genomic dataset;and genomic datasets partitioned into randomly accessible access unitssaid genomic streams comprising encoded aligned reads organized into multiple layers of homogeneous data descriptors to univocally represent genome sequence readswherein in one layer it is stored the mapping position of the first read as absolute position with respect to the reference genome andall the other positions are expressed as a difference with respect to the previous position and are stored in a specific layersaid method further comprisingthe compression of said layers of homogeneous data descriptorsand the transmission of said data streams.

2. The method of claim 1 further comprising a reference ID mapping table for providing the mapping between the numeric identifiers of the references sequences contained in a block header of said access units and the reference identifiers contained in the main header of the stream.

3. The method of claim 2 wherein said genomic dataset are partitioned into access units.

4. The method of claim 3 wherein said access units are partitioned into blocks.

5. The method of claim 4 wherein said blocks are partitioned into packets.

6. The method of claim 1 wherein said genomic dataset list comprises information for identifying the stream associated to each genomic dataset and to be multiplexed in the multiplexed stream.

7. The method of claim 1 wherein said genomic dataset mapping table comprises information for identifying the correspondence points and relevant dependencies among the various multiplexed streams.

8. The method of claim 7 wherein said various multiplexed streams comprise: the genomic sequence, the reference genomic sequence and metadata.

9. The method of claim 1 wherein said genomic dataset mapping table is transmitted in a single packet following the genomics dataset list.

10. The method of claim 9 wherein said genomic dataset mapping table is periodically retransmitted or updated in order to up-date the correspondence points and the relevant dependencies in the streamed data.

11. The method of claim 1 wherein said genomic data list is sent as payload of a single transport packet.

12. The method of claim 11 wherein said genomic data list is periodically retransmitted in order to enable random access to the stream.

13. Apparatus for the transmission of multiplexed genomic data comprising means suitable for carrying out the method of claim 1.

14. Storage device storing genomic data compressed according to the method of claim 1.

15. A computer readable recording medium having recorded there-on a program comprising instruction sets for executing the method of claim 1.

16. The method of claim 1, wherein data are organized as to form a file format.

17. Apparatus for receiving genomic data comprising means for demultiplexing a stream of genomic data, said stream being transmitted according to the method of claim 1.

18. A system for the transmission of multiplexed genomic data comprising an apparatus for transmission and a receiving apparatus as claimed in claims 13.

19. A system for the transmission of multiplexed genomic data comprising an apparatus for transmission and a receiving apparatus as claimed in claim 17.