Systems and Methods for Performing Lossless Source Coding

Abstract

Systems and methods in accordance with various embodiments of the invention perform lossless source coding. In several embodiments, a nested code structure is utilized to perform Random Access Source Coding (RASC), where the number of active encoders is initially unknown. In several embodiments, the decoder can attempt to decode using a number of Slepian-Wolf decoders corresponding to an estimated number of sources. One embodiment includes multiple source encoders configured to receive a start message and transmit a portion of a codeword selected by encoding data from a source until an end of epoch message is received. A source decoder can transmit at least one start message, and receive codeword portions transmitted by the plurality of source encoders. When a decoding rule is satisfied, the source decoder can decode data from multiple source encoders based upon received codeword portions, and cause the broadcast transmitter to transmit an end of epoch message.

Description

FIELD OF THE INVENTION

The present invention relates generally to lossless source coding and more specifically to finite blocklength lossless source coding in applications involving one or more encoder.

BACKGROUND

The information-theoretic limit in any lossless source coding scenario is the set of code sizes or rates at which a desired level of reconstruction error is achievable. Shannon's theory analyzes this fundamental limit by allowing an arbitrarily long encoding blocklength in order to obtain a vanishing error probability.

Finite-blocklength limits, which are of particular interest in delay-sensitive and computationally-constrained coding environments, allow a non-vanishing error probability and enable study of refined asymptotics of the rates achievable withencoding blocklength n. Due to their non-vanishing error probability, the resulting codes are sometimes called “almost-lossless” source codes. The tem “source coding” can be used to refer to this almost-lossless coding paradigm.

In point-to-point source coding, non-asymptotic bounds and asymptotic expansions of the minimum achievable rate have been published. For example, a third-order characterization is known for an optimal code that can provide the minimum achievable rate R*(n, ϵ) at blocklength n and error probability ϵ. For a finite-alphabet, stationary, memoryless source with single-letter distribution P_X, entropy H(X), and varentropy V(X)>0,

$\begin{array}{cc}{R}^{*}\left(n,ϵ\right)\approx H\left(X\right)+\sqrt{\frac{V\left(X\right)}{n}}{Q}^{-1}\left(ϵ\right)-\frac{\log n}{2n},& \left(1\right)\end{array}$

where Q⁻¹(⋅) is the inverse complementary Gaussian distribution function, and any higher-order term is bounded by

$O\left(\frac{1}{n}\right).$

For a multiple access source coding (MASC) also known as a Slepian-Wolf (SW) source coding the fundamental limit is the set of achievable rate tuples known is the rate region. The first-order rate region for stationary, memoryless and general sources are known and Second-order asymptotic expansions of the MASC rate region for stationary, memoryless sources have also been published. However, optimal codes that can approach the fundamental limits identified by Shannon are only known for a small number of special communication scenarios.

SUMMARY OF THE INVENTION

Systems and methods in accordance with various embodiments of the invention perform lossless source coding. A variety of systems and methods are described including systems and methods that are capable of performing source coding in one or more of three scenarios: point-to-point communication, multiple access communication, and random access communication. It can be shown that for point-to-point coding on stationary, memoryless sources, random code design with maximum likelihood decoding can achieve the same coding rate up to the third-order as an optimal code.

SysteMS and methods in accordance with many embodiments of the invention axe capable of performing Multiple Access Source Coding (MASC) utilizing a decoder such as (but not limited to) a decoder formed by a number of stand alone Slepian Wolf decoders to decode k users by choosing the jointly most probable source realizations consistent with the received codewords. Random access source coding (RASC) systems are also introduced and various embodiments are described in which RASC is performed for all possible encoder activity patterns. In several embodiments, a nested structure of the code is utilized to perform RASC, and there is no need for the encoders to know the set of active encoders a priori. In several embodiments, the decoder can attempt to decode using a number of Slepian Wolf decoders corresponding to an estimated number of sources. In the event that the decoder is unable to successfully decode the received information, then the decoder can retry using a different estimate of the number of sources. The third-order-optimal MASC performance can be achievable even when the only information the encoders receive is the acknowledgment that tells them when to stop transmitting.

One embodiment of the random access source coding (RASC) system includes a plurality of source encoders that are each configured to receive data from one of a plurality of sources. When one of the plurality of source encoders receives data from one of the plurality of sources, then the source encoder is configured to: receive a start of epoch message; and transmit a portion of a codeword selected by encoding data from one of the plurality of sources until an end of epoch feedback message is received. In addition, the RASC system includes a receiver comprising a source decoder, where the source decoder is configured to: cause a broadcast transmitter to transmit at least one start of epoch message; receive at least one portion of a codeword transmitted by at least one of the plurality of source encoders; and when a decoding rule is satisfied, decode data from the at least one of the plurality of the source encoders based upon the received at least one portion of a codeword, and cause the broadcast transmitter to transmit an end of epoch message.

In a further elribodiunent, the source decoder is configured to deteimine whether the decoding rule is satisfied at each of a predetermined set of decode times and transmit a feedback message; each of the plurality of source encoders is configured to receive feedback messages at the predetermined set of potential decoding times; and each of the predetermined set of decode times corresponds to one of a plurality of blocklengths of the codeword selected by encoding data from one of the plurality of sources.

In another embodiment, a first of the plurality of blocklengths of the codeword selected by encoding data from one of the plurality of sources forms a first blocklength code; a second of the plurality of blocklengths of the codeword selected by encoding data from one of the plurality of sources forms a second blocklength code; and the first blocklength code forms a prefix for the second blocklength code.

In a still further embodiment, a third of the plurality of blocklengths of the codeword selected by encoding data from one of the plurality of sources forms a third blocklength code; and the second blocklength code forms a prefix for the third blocklength code.

In still another embodiment, each source encoder is configured to select codewords from a codebook.

In a yet another embodiment, the data received from the plurality of sources is dependent.

In yet another embodiment, the broadcast transmitter sends a negative acknowledgement feedback message when the decoding rule is lot satisfied at a predetermined decode time.

In a further embodiment again, each source encoder that is transmitting is configured to transmit an additional portion of the codeword selected by encoding data from one of the plurality of sources in response to receipt of a negative acknowledgement feedback message.

A source encoding system in accordance with another embodiment again includes: a receiver configured to receive a start of epoch message; and a source encoder configured to transmit a portion of a codeword selected by encoding data from a source until an end of epoch feedback message is received.

In a further additional embodiment, each of the plurality of source encoders is configured to receive feedback messages at each of a predetermined set of decode times; and each of the predetermined set of decode times corresponds to one of a plurality of blocklengths of the codeword selected by encoding data from the source.

In another additional embodiment, a first of the plurality of blocklengths of the codeword selected by encoding data from the source forms a first blocklength code; a second of the plurality of blocklengths of the codeword selected by encoding data from the source forms a second blocklength code; and the first blocklength code forms a prefix for the second blocklength code.

In a still yet further embodiment, a third of the plurality of blocklengths of the codeword selected by encoding data from the source forms a third blocklength code; and the second blocklength code forms a prefix for the third blocklength code.

In still yet another embodiment, the source encoder is configured to transmit an additional portion of the codeword selected by encoding data from the source in response to receipt of a negative acknowledgement feedback message.

In a still further embodiment again, the data received from the plurality of sources is dependent.

A receiver system in accordance with still another embodiment again, includes: a source decoder configured using a codebook; a transmitter configured to transmit at least one start of epoch message; and a receiver configured to receive at least one portion of a codeword. In addition, the source decoder is further configured to determine when a decoding rule is satisfied, and when the decoding rule is satisfied: decode at least one message based upon the received at least one portion of a codeword using the codebook; and cause the transmitter to transmit an end of epoch message.

In a still further additional embodiment, the source decoder is configured to determine whether the decoding rule is satisfied at each of a predetermined set of decode times and transmit a feedback message; and each of the predetermined set of decode times corresponds to one of a plurality of blocklengths of the codeword.

In a yet further embodiment again, a first of the plurality of blocklengths of the codeword forms a first blocklength code; a second of the plurality of blocklengths of the codeword forms a second blocklength code; and the first blocklength code forms a prefix for the second blocklength code.

In yet another embodiment again, a third of the plurality of blocklengths of the codeword forms a third blocklength code; and the second blocklength code forms a prefix for the third blocklength code.

In a yet further additional embodiment, the data received from the plurality of sources is dependent.

In yet another additional embodiment, the broadcast transmitter sends a negative acknowledgement feedback message when the decoding rule is not satisfied at a predetermined decode time.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a point-to-point coding system in accordance with an embodiment of the invention.

FIGS. 2A and 2B are plots that illustrate the rate-blocklength trade-offs at ϵ=10⁻¹ (FIG. 2A) and ϵ=10⁻³ (FIG. 2B) for a pair of binary, stationary, memoryless sources.

FIG. 3 is a plot showing the third-order rate region for systems and methods that perform Multiple Access Source Coding (MASC) in accordance with various embodiments of the invention.

FIGS. 4A and 4B are schematic plots of the (n, ϵ)-rate region and the third-order-optimal sum rate when the pair of random variables X₁, X₂ are dependent (FIG. 4A), and when X₁, X₂ are independent (FIG. 4B).

FIG. 5A illustrates a MASC system incorporating feedback and FIG. 5B illustrates a MASC system that utilizes a cooperation facilitator.

FIG. 6 is a flow chart illustrating a process that can be utilized by an encoder to perform source coding within a Random Access Source Coding (RASC) system in accordance with an embodiment of the invention.

FIG. 7 conceptually illustrates an RASC system in accordance with an embodiment of the invention.

FIG. 8 conceptually illustrates a source coding scheme employed by encoders during a single time epoch in an RASC system in accordance with an embodiment of the invention.

FIG. 9 shows the relationship between decoding blocklength, code symbol alphabet sizes, and source coding rate vector.

FIG. 10 shows a number of sensors reading measurements of a common hidden state as an example of an application of RASC in accordance with an embodiment of the invention.

FIGS. 11A and 11B are schematic illustrations of the MASC rate regions for a less redundant source.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for performing source coding using random codes in accordance with various embodiments of the invention are illustrated. In several embodiments, codes are utilized that can be characterized in that they can achieve performance similar to an optimal code in source coding applications involving point-to-point communication between an encoder and a decoder. In a number of embodiments, linear random codes are employed and achieve performance similar to an optimal code in source coding applications involving point-to-point communications. In several embodiments, any of a number of different types of decoders can be utilized to perform decoding including (but not limited to) maximum likelihood decoders and/or threshold decoders.

In point-to-point source coding, the encoder maps a discrete random variable X defined on finite or countably infinite alphabet X into a message from codebook [M]. The decoder reconstructs X from the compressed description. A random code is one in which each code word in the codebook [M] has a finite length and the association between the value of X and a particular code word from codebook [M] is determined randomly. In this way, the encoder need not possess any information regarding the distribution of X to encode the source. Random codes are typically sub-optimal, but random codes utilized in accordance with many embodiments of the invention can be shown to have near-optimal performance when utilized for point-to-point source coding. Furthermore, the set of random codes that have near-optimal performance includes codes that can be utilized by a linear compressor. Therefore, systems and methods in accordance with a number of embodiments utilize one of a set of linear random codes that are capable of achieving near-optimal performance. In certain embodiments, random code are utilized in combination with a maximum likelihood decoder. In a number of embodiments, random code are utilized in combination with a threshold decoder. As can readily be appreciated, the use of a threshold decoder will typically involve a requirement to operate at a higher rate than a system that relies upon use of a maximum likelihood decoder.

Systems and methods in accordance with certain embodiments of the invention perform multiple access source coding (MASC) using random codes. In several embodiments, the MASC process employs limited feedback and/or cooperation feedback. In several embodiments, MASC involves uses of a maximum likelihood decoder that chooses the jointly most probable source realizations consistent with received codewords. Where rate points converge to a non-corner point on the asymptotic sum-rate boundary, systems and methods in accordance with embodiments of the invention can be shown to not compromise the performance in lossless data compression up to the third-order term despite each encoder performing separate encoding. For independent sources, it is shown that there are no non-corner points, and MASC separate encoding incurs a positive penalty in the second-order term relative to joint encoding with a point-to-point code. When two sources have the same marginals, this penalty equals the penalty for using two independent blocklength-n codes rather than a single blocklength-2n point-to-point code for encoding 2n samples.

While a MASC assumes a fixed, known collection of encoders, the set of transmitters communicating with a given access point in applications like sensor networks, the internet of things, and random access communication may be unknown or time-varying. To address these scenarios, a new applicationof source coding is introduced which can be referred to as random access source coding (RASC). RASC extends MASC to scenarios where some encoders are inactive, and the decoder seeks to reliably reconstruct the sources associated with the active encoders assuming that the set of active encoders is unknown a priori.

Systems and methods in accordance with several embodiments of the invention perform RASC using rateless encoders that transmit codewords symbol by symbol until a receiver sends a message telling them to stop. Unlike typical rateless codes, which allow arbitrary decoding times the codes utilized in accordance with many embodiments of the invention possess a small set of decoding times. A single-bit feedback from the decoder to all encoders at each potential decoding time can provide, the encoders with feedback concerning whether or not to continue transmitting.

In a number of embodiments, a single random code is utilized to perform RASC that is characterized in that it is a single code that simultaneously achieves, for every possible set of active encoders, the third-order-optimal MASC performance for the active source set. Since traditional random coding arguments do not guarantee the existence of a single deterministic code that meets multiple independent constraints, prior code designs for multiple-constraint scenarios typically employ a family of codes indexed using common randomness shared by all communicators. Systems and methods in accordance with many embodiments of the invention employ an alternative approach that utilizes a single deterministic code. As can readily be appreciated, for stationary, memoryless, permutation-invariant sources, employing identical encoders at all transmitters can reduce RASC design complexity.

While mach of the discussion that follows relates to source coding, the hypothesis testing techniques described herein can be useful in many applications including (but not limited to) testing components (e.g., finding positive cases in pooled coronavirus tests, defective components in a factory). In the discussion below, hypothesis testing is used as a tool for analyzing the performance of various codes, but decoders in accordance with many embodiments of the invention can apply a type of composite hypothesis test that compares the possible decoder outputs as hypotheses of the form “the encoded message is ‘a,’” against the composite hypothesis “the encoded message is one of ‘b’, ‘c’, . . . , ‘z’.” Accordingly, the hypothesis testing techniques described herein can be utilized in any of a variety of applications including (but not limited to) the applications specifically described herein. Systems and methods for performing source coding in accordance with various embodiments of the invention are discussed further below.

Point-To-Point Random Source Coding

A compression system 100 that employs point-to-point random source coding in accordance with an embodiment of the invention is illustrated in FIG. 1. An (M, ϵ) code for a random variable X with discrete alphabet X can be implemented by an encoder 102 employing an encoding function f: X→[M] and a decoder 104 employing a decoding function g: [M]→X with error probability [g(f(X))≠X]≤ϵ.

In the illustrated embodiment, an (n, M, ϵ) block point-to-point source code is utilized, which is an (M, ϵ) code defined for a random vector Xⁿ with discrete vector alphabet X². The minimum code size M*(n, ϵ) and rate R*(n, ϵ) achievable at blocklength n and error probability ϵ for the code are

$\begin{array}{cc}{M}^{*}\left(n,ϵ\right)\stackrel{\Delta }{=}\min \left\{M:\exists \left(n,M,ϵ\right)\mathrm{code}\right\}& \left(2\right)\\ R^{*}\left(n,ϵ\right)\stackrel{\Delta }{=}\frac{1}{n}\log M^{*}\left(n,ϵ\right).& \left(3\right)\end{array}$

Shannon's source coding theorem describes the fundamental limit on the asymptotic performance for lossless source coding on a stationary, memoryless source, giving

$\begin{array}{cc}\underset{n->\infty }{\lim }{R}^{*}\left(n,ϵ\right)=H\left(X\right),\forall ϵ\in \left(0,1\right).& \left(4\right)\end{array}$

Optimal codes that can approach the fundamental limits identified by Shannon are only known for a small number of special communication scenarios. Consequently, systems and methods in accordance with many embodiments of the invention utilize random codes that have been determined to achieve comparable performance to optimal codes in point-to-point source coding applications. Rates that can be achieved in point-to-point source coding using random codes in accordance with various embodiments of the invention are discussed further below.

Rates Achievable in Point-To-Point Source Coding Using Random Codes

Achievability results that are based on Shannon's random coding argument are important because use of a random code does not require knowledge of the optimal code, which is available only in a few special communication scenarios. The following random coding achievability bound can be obtained by assigning source realizations to codewords independently and uniformly at random. The threshold decoder decodes to x∈X if and only if x is a unique source realization that (i) is compatible with the observed codeword under the given (random) code design, and (ii) has information i(x) below log M−γ.

It can be shown that there exists an (M, ϵ) code for discrete random variable X such that

ϵ≤[(X)>log M−γ]+exp(−γ), ∀γ>0.  (5)

Particularizing (5) to a stationary, memoryless source with single-letter distribution P_X satisfying V(X)>0 and T(X)<∞, choosing log M and γ optimally, and applying the Berry-Esseen inequality gives

$\begin{array}{cc}{R}^{*}\left(n,ϵ\right)\le H\left(X\right)+\sqrt{\frac{V\left(X\right)}{n}}{Q}^{-1}\left(ϵ\right)+\frac{\log n}{2n}+O\left(\frac{1}{n}\right).& \left(6\right)\end{array}$

Since (6) exceeds the rate bounds for optimal codes by

$+\frac{\log n}{n}$

in the third-order term, a question arises as to whether random code design, threshold decoding, or both yield third-order performance penalties. Accordingly, a new random coding bound is derived using a maximum likelihood decoder; this result demonstrates that random coding suffices to achieve the third-order optimal performance for a stationary, memoryless source.

Random code design can be used to derive two new non-asymptotic achievability bounds for point-to-point source coding. These results can be referred to as the dependence testing (DT) bound and the random coding union (RCU) bound.

The Dependence Testing Bound

The DT bound states that, given a discrete random variable X, there exists an (M, ϵ) code with a threshold decoder for which

ϵ≤[exp{−|log M−(X)]|₊}].  (7)

The DT bound provides a bound on the random coding performance of a threshold decoder with threshold log γ as

$\begin{array}{cc}ϵ\le \mathbb{P}\left[i\left(X\right)>\log \gamma \right]+\frac{1}{M}𝕌\left[i\left(X\right)\le \log \gamma \right],& \left(8\right)\end{array}$

where [⋅] denotes a mass with respect to the counting measure U_X on X, which assigns unit weight to each x∈X.

The right-hand side of (8) can be shown to equal

$\frac{M+1}{M}$

times the minimum measure of the error event in a Bayesian binary hypothesis test between P_X with a priori probability

$\frac{M}{M+1}$

and U_X with a priori probability

$\frac{1}{M+1}.$

This error measure can be minimized by a test that compares the log likelihood ratio

$\log \frac{U_X\left(X\right)}{P_X\left(X\right)}$

to the log ratio of a priori probabilities

$\log \frac{M/\left(M+1\right)}{1/\left(M+1\right)},$

giving

H₀:P_X, selected if (X)≤log M

H₁:U_X, selected if (X)>log M.

Taking γ=M minimizes the right-hand side of (8), which implies that the DT bound is the tightest possible bound for random coding with threshold decoding.

Particularizing the DT bound to a stationary, memoryless source with a single-letter distribution P_X satisfying V(X)>0 and T(X)<∞ and invoking the Berry-Esseen inequality, an asymptotic expansion can be obtained

$\begin{array}{cc}{R}^{*}\left(n,ϵ\right)\le H\left(X\right)+\sqrt{\frac{V\left(X\right)}{n}}{Q}^{-1}\left(ϵ\right)+O\left(\frac{1}{n}\right).& \left(9\right)\end{array}$

Unfortunately, (9) is sub-optimal in its third-order term. Thus, random code design with threshold-based decoding fails to achieve the optimal third-order performance. Despite the sub-optimal performance of utilizing random codes with a threshold-based decoder, the use of threshold-based decoding can significantly reduce computational complexity compared to several categories of decoders including (but not limited to maximum likelihood decoders. Furthermore, the result with respect to random codes suggest that many practically imple nentable codes including (but not limited to) linear codes can also achieve acceptable levels of performance and/or performance approaching the performance of optimal codes. Accordingly, systems and methods in accordance with various embodiments of the invention can utilize any decoder (including threshold-based decoders) as appropriate to the requirements of specific applications.

The Random Coding Union Bound

The RCU bound employs random code design and maximum likelihood decoding. The RCU bound states that, given a discrete random variable X, there exists an (M, ϵ) code with a maximum likelihood decoder for which

$\begin{array}{cc}ϵ\le 𝔼\left[\min \left\{1,\frac{1}{M}𝔼\left[\exp \left(\iota \left(\stackrel{\_}{X}\right)\right)1\left\{\iota \left(\stackrel{\_}{X}\right)\le \iota \left(X\right)\right\}❘X\right]\right\}\right],& \left(10\right)\end{array}$

where P_XX(a, b)=P_X(a)P_X(b) for all a, b∈X.

The same RCU bound is obtained by randomizing only over linear encoding maps. Thus, there is no loss in performance when restricting to linear compressors.

The RCU bound can be shown to recover the first three terms of the achievability result for an optimal code. Thus,the sub-optimal third-order terms in (6) and (9) result from the sub-optimal decoder rather than the random encoder design. This is important since optimal codes are not available for scenarios like MASC, which is discussed below.

The RCU bound focuses on a stationary, memoryless source with single-letter distribution P_X satisfying

V(X)>0  (11)

T(X)<∞  (12)

Define constants

$\begin{array}{cc}B\stackrel{\Delta }{=}{C}_0\frac{T\left(X\right)}{{V\left(X\right)}^{3/2}}& \left(13\right)\\ C\stackrel{\Delta }{=}2\left(\frac{\log 2}{\sqrt{2\pi V\left(X\right)}}+2B\left(X\right)\right),& \left(14\right)\end{array}$

where C₀ is the absolute constant in the Berry-Esseen inequality for i.i.d. random variables.

The following demonstrates that third-order-optimal achievability is possible via random coding of a stationary, memoryless source satisfying the conditions in (11) and (12). For all 0<ϵ<1,

$\begin{array}{cc}{R}^{*}\left(n,ϵ\right)\le H\left(X\right)+\sqrt{\frac{V\left(X\right)}{n}}{Q}^{-1}\left(ϵ\right)-\frac{\log n}{2n}+\xi \left(n\right),& \left(15\right)\end{array}$

where

$\xi \left(n\right)=O\left(\frac{1}{n}\right)$

is bounded more precisely as follows.

1) For all

$0<ϵ\le \frac{1}{2}$ $\mathrm{and}$ $n>{\left(\frac{B+C}{ϵ}\right)}^2,$

$\begin{array}{cc}\xi \left(n\right)\le \frac{1}{n}\log C+\frac{1}{n}\frac{B+C}{\varphi \left({\Phi }^{-1}\left(\Phi \left(Q^{-1}\left(ϵ\right)\right)+\frac{B+C}{\sqrt{n}}\right)\right)}.& \left(16\right)\end{array}$

2) For all

$\frac{1}{2}<ϵ\le 1$ $\mathrm{and}$ $n>{\left(\frac{B+C}{ϵ-\frac{1}{2}}\right)}^2,$

$\begin{array}{cc}\xi \left(\eta \right)\le \frac{1}{n}\log C+\frac{1}{n}\frac{B+C}{\varphi \left(Q^{-1}\left(ϵ\right)\right)}.& \left(17\right)\end{array}$

Accordingly, the RCU bound demonstrates that systems and methods in accordance with various embodiments of the invention can achieve third-order optimal performance using random codes. While various systems and methods that employ random codes to perform source coding are described above with reference to the system shown in FIG. 1, encoders and decoders in accordance with many embodiments of the invention can be utilized in MASC applications and/or in the newly described RASC applications detailed below. Systems and methods that can perform MASC and/or RASC in accordance with various embodiments of the invention are discussed further below.

Multiple Access Source Coding

To simplify notation when discussing systems that are utilized tea perform MASC, the following discussion focuses on MASC with two encoders. However, the definitions and results generalize to more than two encoders. Accordingly, it should be readily appreciated that systems and methods in accordance with many embodiments of the invention can utilize two or more sources and two or more decoders (e.g., Slepian Wolf decoders) as appropriate to the requirements of specific applications. In MASC, also known as a Slepian-Wolf source coding, independent encoders compress a pair of random variables (X₁, X₂) with discrete alphabets X₁ and X₂. Encoder i, i∈[2], observes only X_i, which it maps to a codeword in [M_i]; a single decoder jointly decodes the pair of codewords to reconstruct (X₁, X₂). Codes can be defined for abstract random objects and then particularized to random objects that live in an alphabet endowed with a Cartesian product structure.

By way of definition, an (M₁, M₂, ϵ) MASC for random variables (X₁, X₂) with discrete alphabets X₁ and X₂ includes two encoding functions f₁: X₁→[M₁] and f₂: X₂→[M₂] and a decoding function, g: [M₁]×[M₂]→X₁×X₂ with error probability

[g(f₁(X₁),f₂(X₂))≠(X₁,X₂)]≤ϵ.  (18)

In block coding, encoders individually observe X₁ⁿ and X₂ⁿ drawn from distribution P_X1nX2n on X₁ⁿ×X₂ⁿ. An (n, M₁, M₂, ϵ) block MASC is an (M₁, M₂, ϵ) MASC for random vectors (X₁ⁿ, X₂ⁿ) on X₁ⁿ×X₂ⁿ. The code rate for the block MASC R=(R₁, R₂) is given by

$\begin{array}{cc}{R}_1\stackrel{\Delta }{=}\frac{1}{n}\log M_1,R_2\stackrel{\Delta }{=}\frac{1}{n}\log M_2.& \left(19\right)\end{array}$

Rate R=(R₁, R₂) is (n, ϵ)-achievable if there exists an (n, M₁, M₂, ϵ) MASC with

$R_1\le \frac{1}{n}\log M_1$ $\mathrm{and}$ $R_2\le \frac{1}{n}\log M_2.$

The (n, ϵ)-rate region *(n, ϵ) is the closure of the set of (n, ϵ)-achievable rate pairs.

While the above definitions can apply to arbitrary discrete random variables (X_1i, X_2i), i=1, 2, . . . , with transition probability kernels P_{(X1X2)i|(X1X2)i−1}, the asymptotic analysis discussed below focuses on stationary, memoryless sources, where P_{(X1X2)i|(X1X2)i−1}=P_X1X2 for all i=1, 2, . . .

For stationary, memoryless sources with rate R=(R₁, R₂) and distribution P_X1X2, the following can be defined

$\overline{R}\stackrel{\Delta }{=}\left[\begin{array}{c}{R}_1\\ R_2\\ R_1+R_2\end{array}\right],\overline{H}\stackrel{\Delta }{=}\left\{\begin{array}{c}H\left(X_1|X_2\right)\\ H\left(X_2|X_1\right)\\ H\left(X_1,X_2\right)\end{array}\right\}.$

It can be proven if (X₁ⁿ, X₂ⁿ) are stationary and memoryless, then for every ϵ∈(0, 1),

$\begin{array}{cc}\begin{array}{ccc}\underset{n\to \infty }{\lim }\left(n,ϵ\right)=\{\left(R_1,R_2\right):R_1& \ge & H\left(X_1|X_2\right)\\ R_2& \ge & H\left(X_2|X_1\right)\\ R_1+R_2& \ge & H\left(X_1,X_2\right)\},\end{array}& \left(21\right)\end{array}$

(i.e., the strong converse holds). This region can be referred to as the asymptotic MASC rate region. While the above result is presented with respect to stationary and memoryless sources, systems and methods in accordance with various embodiments of the invention can be utilized with any of a variety of different sources as appropriate to the requirements of specific applications.

Development of systems and methods for use in the MASC in accordance with various embodiments of the invention involved development of a MASC RCU bound, extending the RCU bound introduced above to the multiple-encoder case.

Given discrete random variables (X₁, X₂) there exists an (M₁, M₂, ϵ) MASC with

$\begin{array}{cc}ϵ\le 𝔼\left[\min \left\{1,A_1+A_2+A_{12}\right\}\right]\mathrm{where}{A}_1\stackrel{\Delta }{=}\frac{1}{M_1}{𝔼\left[\exp \left(i\left({\stackrel{\_}{X}}_1^{\prime }|X_2\right)\right)1\{\left(i\left({\stackrel{\_}{X}}_1^{\prime }|X_2\right)\le i\left(X_1|X_2\right)\right\}X_1,X_2\right]}^{}{A}_2\stackrel{\Delta }{=}\frac{1}{M_2}𝔼\left[\exp \left(i\left({\stackrel{\_}{X}}_2^{\prime }|X_1\right)\right)1\{\left(i\left({\stackrel{\_}{X}}_2^{\prime }|X_1\right)\le i\left(X_1|X_2\right)\right\}X_1,X_2\right]A_{12}\stackrel{\Delta }{=}\frac{1}{M_1{M}_2}𝔼\left[\exp \left(i\left({\stackrel{\_}{X}}_1^{\prime }|X_2\right)\right)1\{\left(i\left({\stackrel{\_}{X}}_1^{\prime }|X_2\right)\le i\left(X_1,X_2\right)\right\}X_1,X_2\right]P_{X_1{X}_2{\stackrel{\_}{X}}_1{\stackrel{\_}{X}}_2{\stackrel{\_}{X}}_1^{\prime }{\stackrel{\_}{X}}_2^{\prime }}\left(a,b,\stackrel{\_}{a},\stackrel{\_}{b},{\stackrel{\_}{a}}^{\prime },{\stackrel{\_}{b}}^{\prime }\right)=P_{X_1{X}_2}\left(a,b\right)P_{X_1{X}_2}\left(\stackrel{\_}{a},\stackrel{\_}{b}\right)P_{X_1X_2}({\stackrel{\_}{a}}^{\prime }b)P_{X_1X_2}({\stackrel{\_}{b}}^{\prime }a).& \left(22\right)\end{array}$

FIGS. 2A and 2B plots the point-to-point and MASC RCU bounds. Rate-blocklength trade-offs at ϵ=10⁻¹ (FIG. 2A) and ϵ=10⁻³ (FIG. 2B) for a pair of binary, stationary, memoryless sources with joint distribution p_X1X2(0, 0)=½, p_X1X2(0, 1)=p_X1X2(1, 0)=p_X1X2(1, 1)=⅙ are shown. The plot labelled point-to-point (P2P) hypothesis testing (HT) converse coincides with the optimum R*(n, ϵ).

Asymptotics: Third-Order MACS Rate Region

The following third-order asymptotic characterization of the MASC rate region for stationary, memoryless sources closes the

$O\left(\frac{\log n}{n}\right)$

gap between previous published bounds based upon second order characterization of the MASC rate-region for finite-alphabet stationary, memoryless sources in terms of the asymptotic rate region and the entropy dispersion matrix.

In order to develop a third-oreder asymptotic characterization, memoryless sources with single-letter joint distribution P_X1X2 were considered for which

V(X₁,X₂)>0,[V_c(X₁|X₂)]>0,[V(X₂|X₁)]>0,  (27)

T(X₁,X₂)>∞,T(X₁|X₂)>∞,T(X₂|X₁)>∞,  (28)

[T_c²(X₁|X₂)]>∞,[T_c²(X₂|X₁)]>∞.  (29)

When (27) holds, rank (V)≥1 (in fact, the weaker condition V(X₁|X₂)>0, V(X₂|X₁)>0, V(X₁, X₂)>0 suffices). Assumption (28) is also satisfied automatically if the alphabets X₁ and X₂ are finite.

The following set can be defined

$\begin{array}{cc}\left(n,ϵ\right)\stackrel{\Delta }{=}\left\{\stackrel{\_}{R}\in 3^{}:\stackrel{\_}{R}=\stackrel{\_}{H}+\frac{{\mathrm{inv}}_{}\left(V,ϵ\right)}{\sqrt{n}}-\frac{\log n}{2n}1\right\}.& \left(30\right)\end{array}$

where vector H is defined in (20), V is the entropy dispersion matrix for (X₁, X₂), and _inv(V, ϵ) is defined as the multidimensional counterpart of the function Q⁻¹(⋅) as

_inv(V,ϵ){z∈^d:[Z≤z]≤1−ϵ}.  (31)

Note that *(n, ϵ)⊂² but *(n, ϵ)⊂³. The inner and outer bounding sets can be defined as follows

$\begin{array}{cc}{\mathrm{in}}_{*}^{}\left(n,ϵ\right)\stackrel{\Delta }{=}\left\{\stackrel{\_}{R}\in 2^{}:\stackrel{\_}{R}\in \left(n,ϵ\right)+O\left(\frac{1}{n}\right)1\right\}& \left(32\right)\\ {\mathrm{out}}_{*}^{}\left(n,ϵ\right)\stackrel{\Delta }{=}\left\{\stackrel{\_}{R}\in 2^{}:\stackrel{\_}{R}\in \left(n,ϵ\right)-O\left(\frac{1}{n}\right)1\right\}.& \left(33\right)\end{array}$

When a pair of stationary, memoryless sources is utilized with single-letter joint distribution P_X1X2 satisfying (27)-(29), for any 0<ϵ<1, the (n, ϵ)-rate region *(n, ϵ) satisfies

*_in(n,ϵ)⊆*(n,ϵ)⊆*_out(n,ϵ).  (34)

Since the upper and lower bounds agree up to their third-order terms, we call (n, ϵ) the third-order MASC rate region.

FIG. 3 plots the boundaries of (n, ϵ) at different values of n for an example pair of sources. The third-order MASC rate regions (n, ϵ) at ϵ=10⁻³ are shown for stationary, memoryless sources (X₁, X₂) with p_X1,X2(0, 0)=½, p_X1,X2(1, 1)=p_X1,X2(1, 1)=⅙.

For point-to-point source coding, zero varentropy means that the source is uniform; the

$-\frac{\log n}{2n}$

third-order term is absent in that case. While condition (27) limits the third-order MASC rate region considered above to sources with positive varentropies, the case where one or more varentropies are zero is discussed below. Roughly, each zero varentropy yields a zero dispersion, and the absence of a

$-\frac{\log n}{2n}$

third-order term, similar to the point-to-point case. Furthermore, if V(X₁|X₂)>0 but [V_c(X₁|X₂)]=0, the corresponding achievable third order term increases from

$-\frac{\log n}{2n}$

to 0.

In this context, the third-order MASC rate region described above can be generalized to any finite number of encoders. Let ⊂ be a nonempty ordered set with a unique index for each encoder. For any vector ∈, the (−1)-dimensional vector of its partial sums can be defined as

$\begin{array}{cc}{R}_{𝒯}\stackrel{\Delta }{=}\left(\sum _{i\in A}{R}_i,\widehat{𝒯}\in 𝒫\left(𝒯\right)\right).& \left(35\right)\end{array}$

For any distribution defined on and any ∈, (−1)-dimensional vectors can be defined as follows:

()((|),∈())  (36)

[()],  (37)

and (−1)×(−1) entropy dispersion matrix

Cov[()]  (38)

for random vector . The following set can be defined

$\begin{array}{cc}\left(n,ϵ\right)\stackrel{\Delta }{=}{\stackrel{\_}{H}}_{𝒯}+\frac{{\mathrm{inv}}_{}\left(V_{𝒯},ϵ\right)}{\sqrt{n}}-\frac{\log n}{2n}1.& \left(39\right)\end{array}$

Thus, (n, ϵ)⊂ while (n, ϵ)⊂⁻¹. Finally,

$\begin{array}{cc}{\mathrm{in},𝒯}_{*}^{}\left(n,ϵ\right)\stackrel{\Delta }{=}\left\{R_{𝒯}\in {𝒯}^{}:{\stackrel{\_}{R}}_{𝒯}\in {}_{𝒯}\left(n,ϵ\right)+O\left(\frac{1}{n}\right)1\right\}& \left(40\right)\\ {\mathrm{out},𝒯}_{*}^{}\left(n,ϵ\right)\stackrel{\Delta }{=}\left\{R_{𝒯}\in {𝒯}^{}:{\stackrel{\_}{R}}_{𝒯}\in {}_{𝒯}\left(n,ϵ\right)-O\left(\frac{1}{n}\right)1\right\}.& \left(41\right)\end{array}$

If every element of () has a positive variance and a finite third center moment, then for any 0<ϵ<1,

(n,ϵ)⊆(n,ϵ)⊆(n,ϵ).  (42)

Comparison with Point-To-Point Sorce Coding

FIGS. 2A and 2B compare joint (point-to-point) compression of (X₁ⁿ, X₂ⁿ) to the MASC sum-rate at the symmetrical rate point, (R₁=R₂). The gap between the MASC and point-to-point HT converses captures a penalty due to separate encoding. For small n, the third-order Gaussian approximation (without the

$O\left(\frac{1}{n}\right)$

term) is more accurate at ϵ=10⁻¹ than at ϵ=10⁻³ since the

$O\left(\frac{1}{n}\right)$

term blows up as ϵ approaches 0.

It can be shown that optimal MASC incurs no first-order penalty in achievable sum rate when compared to joint coding A quantity known as the local dspersion can be utilized to characterize the second-order speed of convergence to any asymptotic MASC rate point from any direction. For any non-corner point on the diagonal boundary of the asymptotic MASC rate region, the sum rate's second-order coefficient is optimal when approached either vertically or horizontally. Approaching corner points can incur a positive second-order penalty relative to point-to-point coding. The MASC penalty can be bounded by considering the achievable sum rate R₁+R₂ for different choices of R₁ and R₂. The cases where X₁ and X₂ are dependent and X₁ and X₂ are independent can be separately addressed, assuming throughout that (27) and (28) hold.

When X₁ and X₂ are dependent, H(X₁)+H(X₂)>H(X₁, X₂)>H(X₁|X₂)+H(X₂|X₁), and the asymptotic sum-rate boundary contains non-corner and corner points. It can be shown that a MASC incurs no first-, second-, or third-order performance penalty relative to joint coding at non-corner points (i.e., when R₁<H(X₁) and R₂<H(X₂)). In contrast, a MASC suffers a second-order performance penalty at corner points (i.e., when R₁=H(X₁) or R₂=H(X₂)). See FIG. 4A for an illustration. FIGS. 4A and 4B are schematic plots of the (n, ϵ)-rate region and the third-order-optimal sum rate when X₁, X₂ are dependent (FIG. A), X₁, X₂ are independent (FIG. 4B). In FIG. 4A, the boundary of *(n, ϵ) between H(X₁) and H(X₂) (excluding the end points) contains rate points that achieve the optimal point-to-point rate up to the third order, while the end points do not achieve that optimal rate. The value of r* in FIG. 4A is defined in (49). The values of r₁*, r₂* in FIG. 4B are defined as

$\begin{array}{cc}\left(r_1^{*},r_2^{*}\right)=\arg \underset{\underset{\Phi \left(r_1\right)\Phi \left(r_2\right)\ge 1-ϵ}{\left(r_1,r_2\right):}}{\min }\left(\sqrt{V\left(X_1\right)}{r}_1+\sqrt{V\left(X_2\right)}{r}_2\right).& \left(43\right)\end{array}$

Suppose that X₁ and X₂ are dependent.

1. The constants δ₁, δ₂, G>0 and ϵ∈(0, 1) can be fixed. Then there exists some constant n(δ₁, δ₂, G) such that if

$\begin{array}{cc}{R}_1\le H\left(X_1\right)-{\delta }_1& \left(44\right)\\ R_2\le H\left(X_2\right)-{\delta }_2& \left(45\right)\\ R_1+R_2=H\left(X_1,X_2\right)+\sqrt{\frac{V\left(X_1,X_2\right)}{n}}{Q}^{-1}\left(ϵ-\frac{G}{\sqrt{n}}\right)-\frac{\log n}{2n}& \left(46\right)\end{array}$

• • then R=(R₁, R₂)∈(n, ϵ) for all n>n(δ₁, δ₂, G).

2. When ϵ∈(0, 1) is fixed. If

$\begin{array}{cc}{R}_2\ge H\left(X_1❘X_2\right)+\frac{r^{*}}{\sqrt{n}}-\frac{\log n}{2n}+\frac{G}{n}& \left(47\right)\end{array}$

• • for some G>0, then R=(H(X₁), R₂)∈(n, ϵ). Conversely, if R=(H(X₁), R₂)∈(n, ϵ), then

$\begin{array}{cc}{R}_2\ge H\left(X_2|X_1\right)+\frac{r^{*}}{\sqrt{n}}-\frac{\log n}{2n},& \left(48\right)\end{array}$

• • where r* is the solution to equation

Φ(V₂;r,r)=1−ϵ,  (49)

• • and V₂ is the covariance matrix for random vector ((X₂|X₁), ((X₁, X₂)).

For independent sources, the asymptotic sum-rate boundary contains only the single (corner) point (R₁, R₂)=(H(X₁), H(X₂)), and the entropy dispersion matrix

$\left[\begin{array}{cc}V\left(X_1\right)& 0\\ 0& V\left(X_2\right)\end{array}\right].$

is singular.

The next result concerns the third-order-opitimal sum rate

R _sum*(n,ϵ)min{R₁+R₂:R=(R₁,R₂) s.t. R∈*(n,ϵ)}.  (50)

According to the third-order MASC rate region described above, R_sum*(n, ϵ) characterizes the best achievable sum rate in SW source coding up to an

$O\left(\frac{1}{n}\right)$

gap.

For independent sources a unique (r₁*, r₂*) can capture the best MASC second-order sum-rate; the third-order term is achieved at all points on a segment of the rate region boundary. See FIG. 4B. Under assumption (27),

$\begin{array}{cc}\underset{\Phi \left(r_1\right)\Phi \left(r_2\right)\ge 1-ϵ}{\underset{\left(r_1,r_2\right):}{\min }}\left(\sqrt{V\left(X_1\right)}{r}_1+\sqrt{V\left(X_1\right)}{r}_2\right)>\sqrt{V\left(X_1\right)+V\left(X_2\right)}{Q}^{-1}\left(ϵ\right),& \left(51\right)\end{array}$

where V(X₁)+V(X₂)=V(X₁, X₂) for (X₁, X₂) independent. Here (51) follows since its left-hand side solves

$\begin{array}{cc}\underset{\left(a_1,a_2\right)}{\min }\left(a_1+a_2\right)s.t.\Phi \left(\frac{a_1}{\sqrt{V\left(X_1\right)}}\right)\Phi \left(\frac{a_2}{\sqrt{V\left(X_2\right)}}\right)\ge 1-ϵ,& \left(52\right)\end{array}$

and the constraint in (52) requires a₁>√{square root over (V(X₁))}Q⁻¹(ϵ) and a₂>√{square root over (V(X₂))}Q⁻¹(ϵ) , which gives the bound since

√{square root over (V(X₁))}+√{square root over (V(X₂))}>√{square root over (V(X₁)+V(X₂))}.  (53)

Therefore, when X₁ and X₂ are independent, a MASC approach incurs a positive second-order sum-rate penalty relative to joint coding. Closed-form expressions for this penalty are available in special cases. When V(X₁)−V(X₂), r₁*=r₂*=Q⁻¹(1−√{square root over (1−ϵ)}), and the penalty is

$\begin{array}{cc}2\sqrt{\frac{V\left(X_1\right)}{n}}{Q}^{-1}\left(1-\sqrt{1-ϵ}\right)-\sqrt{\frac{2V\left(X_1\right)}{n}}{Q}^{-1}\left(ϵ\right).& \left(54\right)\end{array}$

When and X₁ and X₂ are i.i.d., the penalty equals the penalty fir coding a vector X²ⁿ of 2n i.i.d. outputs from P_X by applying an independent (n, ϵ) (point-to-point) code with error probability 1−√{square root over (1−ϵ)} to each of (X₁, . . . , X_n) and (X_n+1, . . . , X_2n) instead of a single (2n, ϵ) code to vector X²ⁿ.

Limited Feedback and Cooperation

Systems and methods in accordance with various embodiments of the invention that are employed in MASC scenarios can utilize limited feedback. In several embodiments, the decoder broadcasts the same bits of feedback to all encoders. A bit sent at time i is a function of the encoder outputs received in time steps 1, . . . , i−1. (See FIG. 5A.) The impact of feedback can be bounded by studying a MASC with a cooperation facilitator (CF). The CF broadcasts the same -bit function of the sources to both encoders prior to their encoding operations. (See FIG. 5B.) Since the MASC network has no channel noise, feedback from the decoder cannot convey more information than feedback from the CF. As a result, the impact of feedback can be bound by bounding the impact of cooperation.

The CF-MASC and its rate region can be defined as follows. An (L, M₁, M₂, ϵ) CF-MASC for random variables (X₁, X₂) on X₁×X₂ includes a CF function L, two encoding functions f₁ and f₂, and a decoding function g given by

L: X₁×X₂→[L]

f₁: [L]×X₁→[M₁]

f₂: [L]×X₂→[M₂]

g: [M₁]×[M₂]→X₁×X₂,

with error probability

[g(f₁(L(X₁,X₂),X₁),f₂(L(X₁,X₂),X₂))≠(X₁,X₂)]≤ϵ.

An (n, L, M₁, M₂, ϵ) MASC is a CF MASC for random variables (X₁ⁿ, X₂ⁿ) on X₁ⁿ×X₂ⁿ. The code's finite blocklength rates are defined by

$R_1=\frac{1}{n}\log M_1,R_2=\frac{1}{n}\log M_2.$

A rate pair (R₁, R₂) is (n, , ϵ)-CF achievable if there exists an (n, L, M₁, M₂, ϵ) CF-MASC with M₁≤exp(nR₁), M₂≤exp(nR₂), and L≤exp(). The (n, , ϵ)-CF rate region _CF*(n, , ϵ) can be defined as the closure of the set of all (n, , ϵ)-CF achievable rate pairs.

_FB*(n, , ϵ) can be used to denote the feedback-MASC (FB-MASC) rate region, which can be defined as the closure of the set of all (n, ϵ)-achievable rate pairs when the same bits of feedback from the decoder are available to both encoders.

Since the CF sees the source vectors while the decoder sees a coded description of those vectors (using a deterministic code), an -bit CF can implement any function used to determine the decoder's -bit feedback. As a result, any rate point that is achievable by an -bit FB-MASC is also achievable by an -bit CF-MASC. Therefore, for any 0<ϵ<1 and <∞,

_FB*(n,,ϵ)⊆_CF*(n,,ϵ).  (56)

Consider stationary, memoryless sources with single-letter distribution P_X1X2 satisfying (27) and (28). For any 0<ϵ<1 and <∞,

_CF*(n,,ϵ)⊆_out*(n,ϵ).  (57)

Thus _CF*(n, , ϵ) and (n, ϵ) share the same outer bound.

Accordingly, the CF-MASC (and FB-MASC) performance is bounded, so that for any <∞, the third-order rate region for -bit CF-MASCs cannot exceed the corresponding MASC rate region. Hence finite feedback does not enlarge the third-order (n, ϵ) MASC rate region. This result generalizes to scenarios with more than two encoders.

While a variety of systems and methods for performing MASC in a manner that can achieve third-order optimal performance are described above with reference to FIGS. 2A-5B, any of a variety of systems and methods can be implemented in order to perform MASC with or without feedback and/or the use of a CF as appropriate to the requirements of specific applications in accordance with various embodiments of the invention. Furthermore, systems and methods in accordance with a number of embodiments of the invention can be utilized for the newly described RASC applications, which are discussed further below.

Random Access Source Coding

Systems and methods in accordnace with many embodiments are used to perform RASC. RASC is a generalization of MASC for networks where the set of participating encoders is unknown to both the encoders and the decoder a priori.

Definitions and Coding Strategy

In a RASC system, an assumption can be made that there is maximum number of active encoder K<∞. Accordingly, each encoder with a unique source from the set of sources can be indexed by [K]. In addition, each encoder can choose whether to be active or silent. Only sources associated with active encoders are compressed and reconstructed. By assumption, the decision to remain silent is independent of the observed source instance. Given the joint distribution P_X[K] on countable alphabet X_[K], when ordered set ∈([K]) of [K] is active, the marginal on the transmitted sources is

$\begin{array}{cc}{P}_{X_{𝒯}}\left(X_{𝒯}\right)=\sum _{\varkappa \left[K\right]\backslash {𝒯}^{\in {\chi }_{\left[K\right]\backslash 𝒯}}}{P}_{X_{\left[K\right]}}\left(X_{\left[K\right]}\right),{\forall }_{X_{𝒯}}\in {\chi }_{𝒯}.& \left(58\right)\end{array}$

Thus, each encoder's state has no effect on the statistical relationship among sources observed by other encoders.

Systems and methods in accordance with many embodiments of the invention involve the use of a RASC process that organizes communication into epochs. At the beginning of each epoch, each encoder can independently deckle its activity state; that activity state remains unchanged until the end of the epoch. Thus, the active encoder set is fixed in each epoch.

A process 600 that can be implemented by an encoder within an RASC system in accordance with an embodiment of the invention is illustrated in FIG. 6. As illustrated in FIG. 6, each active encoder i∈ observes source output X_i∈X_i. When a determination (602) is made that a source output is available, the encoder independently maps (604) the source data to a codeword that includes a sequence of code symbols from alphabet [Q_i]. When a start of an epoch is detected (606), the || codewords are sent (608) simultaneously to the decoder. Since set is unknown a priori, the encoder behavior cannot vary with . The decoder sees and decides a time , called the decoding blocklength, at which to jointly decode all received partial codewords. The set of potential decoding blocklengths {: ∈([K])} is part of the code design; it is known to all encoders and to the decoder. When the decoder performs the joint decoding, the decoder provides feedback to the encoder indicating (610) that the epoch has ended. The encoding and transmission process 600 repeats as more data becomes available to each encoder and as each new epoch commences.

FIGS. 7 and 8 illustrate an RASC scheme employed in accordance with an embodiment of the invention during one epoch when =[k]. Each encoder i∈ sends a single code symbol per time step. At each time m∈{m′∈: m′<}, the decoder sends a “0” to indicate that it is not yet ready to decode; at time m=, the decoder sends a “1,” ending one epoch and starting the next. The decoder then reconstructs source vector using the first code symbols from each active encoder. To avoid wasting time in an epoch with no active encoders, decoding time m_∅=1 is included in set . The decoder sends at most 2^K bits of feedback, and encoders need only listen for decoder feedback at the times in set .

To formalize the above strategy, K can be fixed to K≥1. In addition, the following vectors can be defined

ϵ _K (,∈([K]))  (59)

m _K (,∈([K])∪{∈})  (60)

with m_∅=1 and m_maxmax{: ∈([K])}.

An (m_K, Q_[K], ϵ_K) RASC for sources X_[K] on source alphabet X_[K] comprises a collection of encoding and decoding functions

$\begin{array}{cc}{f}_i\text{:}{\chi }_i\to {\left[Q_i\right]}^{m_{\max }},i\in \left[K\right],& \left(61\right)\\ g_{𝒯}{\underset{i\in 𝒯}{\text{:}\prod }\left[Q_i\right]}^{m_{𝒯}}\to {\chi }_{𝒯},𝒯\in 𝒫\left(\left[K\right]\right),& \left(62\right)\end{array}$

where f_i is the encoding function for source X_i and is the decoding function for active coder set . For each ∈([K]), source vector is decoded at time with error probability [(f_i(X_i)_[], i∈)≠]≤, where f_i(x_i)_[m] denotes the first m code symbols of f_i(x_i).

An (n, m_K, Q_[K], ϵ_K) RASC is an RASC for an m-block of source outcomes. The parameter n, called the encoding blocklength does not vary with . An (m_K, Q_[K], ϵ_K) RASC behaves, for each , like a ((, ∈), ) MASC with a finite number |{m∈: m≤}| of feedback bits. However, the RASC is one code. Its descriptions are nested so that the previously transmitted symbols are a prefix for the codeword of the next larger blocklength (i.e., for each ∈, if ′<, then f_i(x_i) is a prefix of f_i(x_i)_[]). It simultaneously satisfies the error constraints for all ∈([K]). And, since the code symbol alphabet sizes Q_[K] are fixed, its rate vectors are coupled. See FIG. 9, which shows the relationship between decoding blocklength , code symbol alphabet sizes (Q₁, Q₂), and source coding rate vector (illustrated for ={1, 2}).

The following definitions can be utilized to build a non-asymptotic fundamental limit of RASCs.

A collection of rate vectors is n-valid if ∃ (m_K, Q_[K]) s.t.

$\begin{array}{cc}{R}_{𝒯}=\frac{1}{n}\left(m_{𝒯}\log Q_i,i\in 𝒯\right),\forall 𝒯\in 𝒫\left(\left[K\right]\right).& \left(63\right)\end{array}$

The set _valid(n) is the set of n-valid rate collections. The collection is (n, ϵ_K)-achievable if there exists an (n, m_K, Q_[K], ϵ_K) RASC scheme. The (n, ϵ_K)-rate set * (n, ϵ_K) is the set of (n, ϵ_K)-achievable rate collections.

Asymptotics: Third-Order Performance of the RASC

The performance of an (n, m_K, Q_[K], ϵ_K) RASC for stationary, memoryless sources can be analyzed. Results include both achievability and converse characterizations of the (n, ϵ_K)-rate set * (n, ϵ_K) under the assumption that the single-letter joint source distribution P_X[K] satisfies

[V_c(|)]>0 ∀⊆⊆[K],,≠∅  (64)

T(|)>∞ ∀⊆⊆[K],,≠∅  (65)

[T_c²(|)]>∞ ∀⊆⊆[K],,≠∅  (66)

Constraints (64)-(66) enable the use of Berry-Esseen bounds in subsequent analysis. The resulting characterization is tight up to the third-order term. While the existence of an (n, m_K, Q_[K], ϵ_K) RASC implies the existence of an (n, (, i∈), ) MASC for each ∈([K]), the existence of individual MASCs does not imply the existence of a single RASC that simultaneously satisfies the error probability constraints for all possible configurations of active encoders. Indeed, the ability of systems and methods in accordance with various embodiments of the invention to employ a single RASC that simultaneously performs as well (up to the third-order term) as the optimal MASC for each possible nuinber of encoders (i.e. each ∈([K])) is an important and surprising result.

The inner and outer bounding sets can be defined as follows:

_in*(n,ϵ_K){(R∈_valid(n): ∈(n, ϵ)∀∈([K])}  (67)

_out*(n,ϵ_K){(R∈_valid(n): ∈(n, ϵ)∀∈([K])},  (68)

where (n, ϵ) and (n, ϵ) are the third-order MASC bounding sets for distribution . (See (40) and (41).)

Third-Order RASC Performance

For any K<∞, stationary, memoryless sources specified by a single-letter joint distribution P_X[K] satisfying (64)-(66) can be considered. For any 0<ϵ_K<1,

_in*(n,ϵ_K)⊆*(n,ϵ_K)≣R_out*(n,ϵ_K).  (69)

The achievability of third-order RASC performance provides a sufficient condition for the existence of a single RASC that is simultaneously good for all ∈([K]). To prove this, an achievability result can be derived assuming that the encoders and decoder share the common randomness used to generate a random code. Unfortunately, the existence of a random code ensemble with expected error probability satisfying the error probability constraint for each ∈([K]) does not guarantee the existence of a single deterministic code satisfying those constraints simultaneously. Therefore a different approach can be taken in accordance with many embodiments of the invention, which, unexpectedly, combines a converse bound on error probability and a random coding argument to show achievability.

The following refinement of the random coding argument provides a bound on the probability (with respect to the random code choice) that the error probability of a randomly chosen code exceeds a certain threshold. The code of interest here call be any type of source or channel code.

Given any RASC c, for each ∈([K]) let (c) denote the error probability of code c under active encoder set . Before proceeding with the proof, a random code ensemble is defined in order to calculate its expected error probability.

For any K<∞, consider a source distribution P_X[K] defined on countable alpha-bet X_[K]. There exists a random code ensemble C defined on the set of all RASCs with decoding blocklengths m_K and code alphabets Q_[K] for which the following inequalities hold simultaneously for all ∈([K]):

$\begin{array}{cc}𝔼\left[P_{e,𝒯}\left(C\right)\right]\le 𝔼\left[\min \left\{1,\sum _{\widehat{𝒯}\mathrm{ϵ𝒫}\left(𝒯\right)}\exp \left(-m_{𝒯}·\stackrel{\_}{Q}\left(\widehat{𝒯}\right)\right)A_{\widehat{𝒯}}\right\}\right],\mathrm{where}& \left(70\right)\\ \stackrel{\_}{Q}\left(\widehat{𝒯}\right)\stackrel{\Delta }{=}\sum _{iϵ\widehat{𝒯}}\log Q_i& \left(71\right)\\ A_{\widehat{𝒯}}\stackrel{\Delta }{=}𝔼\left[\exp (i\left({\stackrel{\_}{X}}_{\widehat{𝒯}}X_{𝒯\backslash \widehat{𝒯}})\right)\mathrm{.1}\left\{i\left({\stackrel{\_}{X}}_{\widehat{𝒯}}❘X_{𝒯\backslash \widehat{𝒯}}\right)\le i\left(X_{\widehat{𝒯}}❘X_{𝒯\backslash \widehat{𝒯}}\right)\right\}X_{𝒯}\right]& \left(72\right)\end{array}$

and the expectation in (72) is with respect to the conditional distribution

=  (73)

When parameters (n, Q_[K], ϵ_K) are fixed, increasing yields larger decoding blocklengths . Therefore, the choice of ( to satisfy

$\begin{array}{cc}\sum _{\mathrm{𝒯ϵ𝒫}\left(\left[K\right]\right)}\frac{1}{{\lambda }_{𝒯}+1}<1,& \left(74\right)\end{array}$

controls the RASC performance trade-off across different active encoder sets. This trade-off affects the performance of the RASC in the fourth- or higher-order terms.

RASC for Permutation-Invariant Sources

A permutation-invariant source is defined by the constraint

P _{X [K]} (x_[K])=P_X[K](x_π([K]))  (75)

for all permutations π on [K] and all x_[K]∈X_[K]. For example, given any P_S and P_X|S, the marginal P_X[K] of P_X[K]S=(P_X|S)^KP_S satisfies (75). Such “hidden variable” models have applications in statistics, science, and economics, where latent variables (e.g., the health of the world economy or the state of the atmosphere) influence observabies (e.g., stock prices or climates). FIG. 10 shows an example with K sensors reading measmements of a common hidden state S.

Permutation-invariant source models can be of interest both because of their wide applicability and because they present an opportunity for code simplification through identical encoding, where all encoders employ the same encoding map. For any permutation-invariant source, (58) and (75) imply that X_i=X for all i∈[K] and, for any ∈([K]) with ||=k,

=P_X[k].  (76)

Thus, is permutation-invariant for every and the joint source distribution depends on the number of active encoders but not their identities. Assuming the same error probability ϵ_k is further employed for all ∈([K]) with ||=k, a single decoding blocklength can be fixed for each number k∈[K] of active encoders. Furthermore, identical encoders can be utilized at all transmitters, allowing the accommodation of an arbitrarily large number of encoders without designing a unique encoder for each source.

In analyzing RASC performance with identical encoders on a permutation-invariant source, an assumption can be made in addition to (64) and (65) that no two sources are identical, i.e.,

$\begin{array}{cc}\left[\bigcup _{i,\mathrm{jϵ}\left[K\right],i\ne j}\left\{X_i=X_j\right\}\right]<1.& \left(77\right)\end{array}$

This can be important since using identical encoders on identical sources yields identical descriptions, in which case descriptions from multiple encoders are no better than descriptions from a single encoder. Under these assumptions, third-order RASC performance continues to hold. A number of embodiments of the invention utilize a modified decoder that outputs the most probable source vector ∈ that contains no repeated symbols, treating the case where contains repeated symbols as an error. In several embodiments, the system permits multiple identical sources and the decoder relies upon a system configuration in which multiple identical sources are active with low likelihood. In certain embodiments, a decoder can detect that multiple identical sources are active and can command all except one of the identical sources to cease activity. In several embodiments, a source encoder ceases activity for a specified and/or random period of time. In the asymptotic analysis for stationary, memoryless sources, the probability of this error event is bounded by

$\begin{array}{cc}\mathbb{P}\left[\bigcup _{i,j\in \left[K\right],i\ne j}\left\{X_i^n=X_j^n\right\}\right]\le {\left(\mathbb{P}\left[\bigcup _{i,j\in \left[K\right],i\ne j}\left\{X_i=X_j\right\}\right]\right)}^n,& \left(78\right)\end{array}$

which decays exponentially in n by (77). Therefore, under the assumption in (77), identical encoding does not incur a first- second-, or third-order performance penalty.

MASC Using Sources with Less Redundancy

Referring again to systems and methods in accordance with various embodimennts of the invention that employ MASC, the asymptotic achievability result for the MASC schemes described above requires that all V(X₁, X₂), V(X₁|X₂), and V(X₂|X₁) are strictly positive (as an implication of assumption (27)). Thus, the analysis presented above can break down when any of these varentropies is equal to zero. Such a source can be referred to as being less redundant. Accordingly, the performance systems and methods that perform MASC in accordance with a number of embodiments of the invention can be analyzed when utilized in combination with less redundant sources. Specifically, a pair of stationary, memoryless sources can be analyzed with respect to the following three cases:

1) all three varentropies are equal to zero;

2) exactly two of the varentropies are equal to zero;

3) exactly one of the varentropies is equal to zero.

The assumption can be maintained that the joint distribution P_X1X2 satisfies (28) and (29). For those cases in which V(X₂|X₁)>0 the assumption can be maintained that [V_c(X₂|X₁)]>0. Furthermore, if V(X₂|X₁)>0, the assumption can be maintained that [V_c(X₁|X₂)]>0.

When encoders are required to operate independently in a MASC, it is typically not a simple matter to find optimal codes. In Section A below, characterizations of the (n, ϵ)-rate region are provided with respect to the three general cases listed above. Then, in Section B, the case where P_X1X2(x₁, x₂)>0 for every (x₁, x₂)∈X₁×X₂ is considered; under this condition, the optimal codes can be found and analyzed directly.

Section A: General Characterizations of the (n, ϵ)-Rate Region

The results in the three general cases are listed below.

Case 1): Suppose that V(X₁|X₂)=0, and V(X₂|X₁)=0, and V(X₁, X₂)=0. For any δ₁, δ₂, δ₁₂>0, let

$\begin{array}{cc}\begin{array}{c}{\mathrm{in}}_{\left(1\right)}^{}\left(n,{\delta }_1,{\delta }_2,{\delta }_{12}\right)\stackrel{\Delta }{=}\{\left(R_1,R_2\right)\in {ℝ}^2:\\ R_1\ge H(X_1X_2)+\frac{1}{n}\log \frac{1}{{\delta }_1}\\ R_2\ge H(X_2X_1)+\frac{1}{n}\log \frac{1}{{\delta }_2}\\ R_1+R_2\ge H\left(X_1,X_2\right)+\frac{1}{n}\log \frac{1}{{\delta }_{12}}\}.\end{array}& \left(79\right)\end{array}$

Define

$\begin{array}{cc}{\mathrm{in}}_{\left(1\right)}^{}\left(n,ϵ\right)\stackrel{\Delta }{=}\bigcup _{{\delta }_1,{\delta }_2,{\delta }_{12}>0{\delta }_1+{\delta }_2+{\delta }_{12}=ϵ}{\mathrm{in}}_{\left(1\right)}^{}\left(n,{\delta }_1,{\delta }_2,{\delta }_{12}\right)& \left(80\right)\\ \begin{array}{c}{\mathrm{out}}_{\left(1\right)}^{}\left(n,ϵ\right)\stackrel{\Delta }{=}\{\left(R_1,R_2\right)\in {ℝ}^2:\\ R_1\ge H(X_1X_2)+\frac{1}{n}\log \frac{1}{1-ϵ}\\ R_2\ge H(X_2X_1)+\frac{1}{n}\log \frac{1}{1-ϵ}\\ R_1+R_2\ge H\left(X_1,X_2\right)+\frac{1}{n}\log \frac{1}{1-ϵ}\}.\end{array}& \left(81\right)\end{array}$

When V(X₁|X₂)=0, V(X₂|X₁)=0, and V(X₁, X₂)=0, the (n, ϵ)-rate region *(n, ϵ) satisfies

_in ⁽¹⁾(n,ϵ)⊆*(n,ϵ)⊆_out⁽¹⁾(n,ϵ).  (82)

As in the point-to-point scenario, there are no second-order dispersion terms or

$-\frac{\log n}{2n}$

third-order terms in the characterization of *(n, ϵ) in this case. For any n and ϵ, the achievable region _in⁽¹⁾(n, ϵ) has a curved boundary due to the trade-off in the

$O\left(\frac{1}{n}\right)$

fourth-order terms, while the converse region _out⁽¹⁾(n, ϵ) has three linear boundaries.

Case 2): There are three possible cases where exactly two of the three varentropies are equal to zero. Here, we suppose that V(X₁|X₂)>0 while V(X₂|X₁)=V(X₁, X₂)=0. The other two cases can be analyzed in the same way.

When V(X₁|X₂)>0, V(X₂|X₁)=0, and V(X₁, X₂)=0, the (n, ϵ)-rate region *(n, ϵ) satisfies

_in ⁽²⁾(n,ϵ)⊆*(n,ϵ)⊆_out⁽²⁾(n,ϵ).  (83)

The achievable region _in⁽²⁾(n, ϵ) has a curved boundary due to the trade-off in δ₁, ϵ₂, and δ₁₂. If

$\begin{array}{cc}{\delta }_1=ϵ-\frac{2}{\sqrt{n}},{\delta }_2=\frac{1}{\sqrt{n}},{\delta }_{12}=\frac{1}{\sqrt{n}},& \left(84\right)\end{array}$

then it is apparent that the dispersion corresponding to R₁ is V(X₁|X₂) with a

$-\frac{\log n}{2n}$

third-order term, while the dispersions of R₂ and R₁+R₂ are zero.

Case 3): Similar to Case 2), there are three possible cases where exactly one of the three varentropies is equal to zero. Here, the case where V(X₁|X₂)=0 while V(X₂|X₁)>0 and V(X₁, X₂)>0 is considered. When V(X₁|X₂)=0, V(X₂|X₁)=0, and V(X₁, X₂)>0, the (n, ϵ)-rate region *(n, ϵ) satisfies

_in ⁽³⁾(n,ϵ)⊆*(n,ϵ)⊆_out⁽³⁾(n,ϵ).  (85)

For any n and ϵ the achievable region _in⁽³⁾(n, ϵ) has a curved boundary that is characterized by the trade-off between a separate bound on R₁ and a region in ² that bounds (R₂, R₁+R₂) jointly. The converse region _out⁽³⁾(n, ϵ) is the intersection of a region with a linear boundary that bounds R₁ only and a region with a curved boundary that bounds (R₂, R₁+R₂) jointly. Letting

$\begin{array}{cc}\delta =\frac{1}{\sqrt{n}},& \left(86\right)\end{array}$

then it is apparent that the dispersion corresponding to R₂ and R₁+R₂ is given by V₂ with a

$-\frac{\log n}{2n}$

third-order term, while the dispersion of R₁ is zero.

A less redundant stationary memoryless source has some useful properties. When V(X₁, X₂)=0,

P_X1nX2n(x₁ⁿ,x₂ⁿ)∈{0, exp(−nH(X₁,X₂))},  (87)

for every (x₁ⁿ, x₂ⁿ)∈X₁ⁿ×X₂ⁿ; in other words, (X₁, X₂) is uniformly distributed over its support in X₁×X₂. When V(X₁|X₂)=0,

$\begin{array}{cc}{P}_{X_1^nX_2^n}(x_1^nx_2^n)=\{\begin{array}{c}\exp (-nH\left(X_1X_2)\right),\\ \mathrm{if}{P}_{X_1^n{X}_2^n}\left(x_1^n,x_2^n\right)>0\\ 0,\mathrm{otherwise};\end{array}& \left(88\right)\end{array}$

in other words, X₁ is uniformly distributed over its conditional support for each x₂∈X₂. When V(X₂|X₁)=0, a result analogous to (88) holds. These properties do not reduce the difficulty of characterizing the optimal MASCs in general. As a result, the random coding techniques described above are utilized. For the achievability argument, the MASC bound described above can be invoked.

Section B: Two Special Cases

The analysis above applies to any stationary, memoryless source with single-letter distribution P_X1X2 that satisfies (28). In such a general setting, it is hard to find an optimal code. However, there are some special cases in which the optimal codes for a less redundant source can be characterized.

To enable the following analysis on these special cases, it is assumed that P_X1X2(x₁, x₂)>0 for every (x₁, x₂)∈X₁×X₂. Under this assumption, V(X₁, X₂)=0 if and only if V(X₁|X₂)=V(X₂|X₁)=0. As a result, the three cases discussed in Section A reduce to only two possible scenarios:

1) V(X₁, X₂)=V(X₁|X₂)=V(X₂|X₁)=0;

2) V(X₁, X₂)>0, and either V(X₁|X₂)=0 or V(X₂|X₁)=0.

Note that X₁ and X₂ are independent in both of these scenarios. The results are summarized below.

Special Case 1): Suppose that V(X₁|X₂)=0, V(X₂|X₁)=0, and V(X₁, X₂)=0. If P_X1X2 satisfies P_X1X2(x₁, x₂)>0 for every (x₁, x₂)∈X₁×X₂, then

*(n,ϵ)=_out⁽¹⁾(n,ϵ),  (89)

where _out⁽¹⁾(n, ϵ) is defined in (81).

This scenario is a special example of Case 1) discussed in Section A above. The (n, ϵ)-rate region here coincides with the converse region _out⁽¹⁾(n, ϵ) presented in (81) for general source distributions. See FIGS. 11A and 11B for a comparison among _in⁽¹⁾(n, ϵ), _out⁽¹⁾(n, ϵ), and *(n, ϵ) in this special case. FIGS. 11A and 11B are schematic illustrations of the MASC rate regions for a less redundant source. The drawing in FIG. 11A illustrates the achievable and converse regions in Case 1) (V(X₁, X₂)=V(X₁|X₂)=V(X₂|X₁)=0) and the (n, ϵ)-rate region *(n, ϵ) when P_X1X2 is assumed to have no zeros (Special Case 1)). The drawing in FIG. 11B illustrates the achievable and converse regions in Case 3) (V(X₁|X₂)=0, V(X₁, X₂), V(X₂|X₁)>0) and the (n, ϵ)-rate region *(n, ϵ) when P_X1X2 is assumed to have no zeros (Special Case 2)).

Special Case 2): With V(X₁, X₂)>0, it can be assumed that V(X₁|X₂)=0 and V(X₂|X₁)>0. The other case can be analyzed similarly. For any δ∈[0, ϵ), the following can be defined

$\begin{array}{cc}{\mathrm{in}}_s^{}\left(n,\delta \right)\stackrel{\Delta }{=}\left\{\left(R_1,R_2\right)\in {ℝ}^2:R_1\ge H\left(X_1\right)-\frac{1}{n}\log \frac{1}{1-\delta }{R}_2\ge H\left(X_2\right)+\sqrt{\frac{V\left(X_2\right)}{n}}{Q}^{-1}\left(\frac{ϵ-\delta }{1-\delta }\right)-\frac{\log n}{2n}+{\xi }_{\mathrm{in}}\left(ϵ,\delta ,n\right)\right\}& \left(90\right)\\ {\mathrm{out}}_s^{}\left(n,\delta \right)\stackrel{\Delta }{=}\left\{\left(R_1,R_2\right)\in {ℝ}^2:R_1\ge H\left(X_1\right)-\frac{1}{n}\log \frac{1}{1-\delta }{R}_2\ge H\left(X_2\right)+\sqrt{\frac{V\left(X_2\right)}{n}}{Q}^{-1}\left(\frac{ϵ-\delta }{1-\delta }\right)-\frac{\log n}{2n}-{\xi }_{\mathrm{out}}\left(ϵ,\delta ,n\right)\right\},& \left(91\right)\end{array}$

where the functions ξ_in(ϵ, δ, n) and ξ_out(ϵ, δ, n) are characterized as follows. For any fixed δ,

${\xi }_{\mathrm{out}}\left(ϵ,\delta ,n\right)=O\left(\frac{1}{n}\right)\mathrm{and}{\xi }_{\mathrm{in}}\left(ϵ,\delta ,n\right)=O\left(\frac{1}{n}\right).$

For any fixed n, both ξ_out(ϵ, δ, n) and ξ_in(ϵ, δ, n) blow up as δ approaches ϵ.

Also the following can be defined

$\begin{array}{cc}{\mathrm{in}}_s^{}\left(n,ϵ\right)\stackrel{\Delta }{=}\bigcup _{\delta \in 0,t)}{\mathrm{in}}_s^{}\left(n,\delta \right)& \left(92\right)\\ {\mathrm{out}}_s^{}\left(n,ϵ\right)\stackrel{\Delta }{=}\bigcup _{\delta \in 0,t)}{\mathrm{out}}_s^{}\left(n,\delta \right).& \left(93\right)\end{array}$

Suppose that V(X₁|X₂)=0, V(X₂|X₁)>0, and V(X₁, X₂)>0. If P_X1X2 satisfies P_X1X2(x₁, x₂)>0 for every (x₁, x₂)∈X₁×X₂, then

_in ^s(n,ϵ)⊆*(n,ϵ)⊆_out^s(n,ϵ).  (94)

This scenario is a special example of Case 3) discussed in Section A above. The achievable region is

$\begin{array}{cc}{\mathrm{in}}_{\left(3\right)}^{}\left(n,ϵ\right)=\bigcup _{\delta \in \left(0,ϵ\right)}\left\{\left(R_1,R_2\right)\in {ℝ}^2:R_1\ge H\left(X_1\right)+\frac{1}{n}\log \frac{1}{\delta }{R}_2\ge H\left(X_2\right)+\sqrt{\frac{V\left(X_2\right)}{n}}{Q}^{-1}\left(ϵ-\delta -\frac{C_{\mathrm{in}}}{\sqrt{n}}\right)-\frac{\log n}{2n}+\frac{1}{n}\log \frac{1}{1-\delta }\right\},& \left(95\right)\end{array}$

and the converse region is

$\begin{array}{cc}\begin{array}{c}{\mathrm{out}}_{\left(3\right)}^{}\left(n,ϵ\right)=\{\left(R_1,R_2\right)\in {ℝ}^2:\\ R_1\ge H\left(X_1\right)+\frac{1}{n}\log \frac{1}{1-ϵ}\\ R_2\ge H\left(X_2\right)+\sqrt{\frac{V\left(X_2\right)}{n}}{Q}^{-1}\left(ϵ\right)-\frac{\log n}{2n}-O\left(\frac{1}{n}\right)\}.\end{array}& \left(96\right)\end{array}$

As δ approaches ϵ, the boundary of the (n, ϵ)-rate region given in (92) approaches the line

$R_1=H\left(X_1\right)-\frac{1}{n}\log \frac{1}{1-ϵ},$

which matches the vertical segment of the boundary of the converse region _out⁽³⁾(n, ϵ). See FIG. 11B for a comparison of _in⁽³⁾(n, ϵ), _out⁽³⁾(n, ϵ), and *(n, ϵ) in this case.

Although the present invention has been described in certain specific aspects, many additional modifications and variations would be apparent to those skilled in the art. It is therefore to be understood that the present invention can be practiced otherwise than specifically described including using any of a variety of different decoders and/or sources without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

1. A random access source coding (RASC) system, comprising: a plurality of source encoders that are each configured to receive data from one of a plurality of sources;when one of the plurality of source encoders receives data from one of the plurality of sources, then the source encoder is configured to: receive a start of epoch message; andtransmit a portion of a codeword selected by encoding data from one of the plurality of sources until an end of epoch feedback message is received; anda receiver comprising a source decoder, where the source decoder is configured to: cause a broadcast transmitter to transmit at least one start of epoch message;receive at least one portion of a codeword transmitted by at least one of the plurality of source encoders; andwhen a decoding rule is satisfied, decode data from the at least one of the plurality of the source encoders based upon the received at least one portion of a codeword, and cause the broadcast transmitter to transmit an end of epoch message.

2. The RASC system of claim 1, wherein: the source decoder is configured to determine whether the decoding rule is satisfied at each of a predetermined set of decode times and transmit a feedback message;each of the plurality of source encoders is configured to receive feedback messages at the predetermined set of potential decoding times; andeach of the predetermined set of decode times corresponds to one of a plurality of blocklengths of the codeword selected by encoding data from one of the plurality of sources.

3. The RASC system of claim 2, wherein: a first of the plurality of blocklengths of the codeword selected by encoding data from one of the plurality of sources forms a first blocklength code;a second of the plurality of blocklengths of the codeword selected by encoding data from one of the plurality of sources forms a second blocklength code; andthe first blocklength code forms a prefix for the second blocklength code.

4. The RASC system of claim 3, wherein: a third of the plurality of blocklengths of the codeword selected by encoding data from one of the plurality of sources forms a third blocklength code; andthe second blocklength code forms a prefix for the third blocklength code.

5. The RASC system of claim 1, wherein each source encoder is configured to select codewords from a codebook.

6. The RASC system of claim 1, wherein the data received from the plurality of sources is dependent.

7. The RASC system of claim 1, wherein the broadcast transmitter sends a negative acknowledgement feedback message when the decoding rule is not satisfied at a predetermined decode time.

8. The RASC system of claim 1, wherein each source encoder that is transmitting is configured to transmit an additional portion of the codeword selected by encoding data from one of the plurality of sources in response to receipt of a negative acknowledgement feedback message.

9. A source encoding system, comprising: a receiver configured to receive a start of epoch message; anda source encoder configured to transmit a portion of a codeword selected by encoding data from a source until an end of epoch feedback message is received.

10. The source encoding system of claim 9, wherein: each of the plurality of source encoders is configured to receive feedback messages at each of a predetermined set of decode times; andeach of the predetermined set of decode times corresponds to one of a plurality of blocklengths of the codeword selected by encoding data from the source.

11. The source encoding system of claim 10, wherein: a first of the plurality blocklengths of the codeword selected by encoding data from the source forms a first blocklength code;a second of the plurality of blocklengths of the codeword selected by encoding data from the source forms a second blocklength code; andthe first blocklength code forms a prefix for the second blocklength code.

12. The source encoding system of claim 11, wherein: a third of the plurality of blocklengths of the codeword selected by encoding data from the source forms a third blocklength code; andthe second blocklength code forms a prefix for the third blocklength code.

13. The source encoding system of claim 9, wherein the source encoder is configured to transmit an additional portion of the codeword selected by encoding data from the source in response to receipt of a negative acknowledgement feedback message.

14. The source encoding system of claim 9, wherein the data received from the plurality of sources is dependent.

15. A receiver system, comprising: a source decoder configured using a codebook;a transmitter configured to transmit at least one start of epoch message; anda receiver configured to receive at least one portion of a codeword;wherein the source decoder is further configured to determine when a decoding rule is satisfied, and when the decoding rule is satisfied: decode at least one message based upon the received at least one portion of a codeword using the codebook; andcause the transmitter to transmit an end of epoch message.

16. The receiver system of claim 15, wherein: the source decoder is configured to determine whether the decoding rule is satisfied at each of a predetermined set of decode times and transmit a feedback message; andeach of the predetermined set of decode times corresponds to one of a plurality of blocklengths of the codeword.

17. The receiver system of claim 16, wherein: a first of the plurality of blocklengths of the codeword forms a first blocklength code;a second of the plurality of blocklengths of the codeword forms a second blocklength code; andthe first blocklength code forms a prefix for the second blocklength code.

18. The receiver system of claim 17, wherein: a third of the plurality of blocklengths of the codeword forms a third blocklength code; andthe second blocklength code forms a prefix for the third blocklength code.

19. The source encoding system of claim 17, wherein the data received from the plurality of sources is dependent.

20. The receiver system of claim 15, wherein the broadcast transmitter sends a negative acknowledgement feedback message when the decoding rule is not satisfied at a predetermined decode time.