DIGITAL COMMUNICATION SYSTEM USING REAL-TIME CAPACITY ACHIEVING ENCODER DESIGN FOR CHANNELS WITH MEMORY AND FEEDBACK

Abstract

A method for characterizing a capacity of a channel with memory and feedback comprises defining a channel model corresponding to the channel, wherein: the channel is utilized to transmit information from a source to a destination, and the channel model indicates a dependence of outputs from the channel on past and present channel input symbols and on past channel output symbols. The method further includes determining a representation of the capacity based on the channel model and based on a channel input distribution that achieves the capacity, wherein the representation represents the capacity for a finite number of transmissions over the channel, and wherein the representation includes an optimization, and solving the optimization of the representation of the capacity to determine the capacity of the channel for the finite number of transmissions.

Description

TECHNICAL FIELD

This patent application relates generally to systems and methods for facilitating real-time communication, and more particularly, to determining properties of channel input distributions that characterize capacities of channels with memory, with feedback, with or without transmission cost, and without feedback (in certain cases in which feedback does not increase capacity), to designing encoders to achieve capacities of channels with memory, to characterizing nonanticipative rate distortion functions for sources of information with memory, and to utilize characterized rate distortion functions and channel capacities in Joint Source-Channel Coding (JSCC).

BACKGROUND

Advancements in information technology and networks are transforming the everyday lives of many people with respect to employment, health care, communication, education, environment, etc. In particular, advancements in information technology and networks have spawned the field of Cyber-Physical Systems (CPSs), which field refers to the next generation of engineering systems integrated via advanced technologies and protocols. These engineering systems are capable of performing ubiquitous computing, communication, and control for complex physical systems and can be implemented in energy systems (e.g., the electric power distribution and smart grids), transportation systems (e.g., traffic networks), health care and medical systems, surveillance networks, control systems for underwater and unmanned aerial vehicles, etc. In many of these applications, sub-systems, sensors or observation posts, and controllers or control stations are distributed, often at distinct locations, and communication among sub-systems is limited. Thus, in the context of such systems, there is a demand for real-time communication, decentralized decisions, and the integration of real-time communication and decentralized decisions into complex networks.

In the field of communications, most encoders and decoders for transmitting information over channels (e.g., for transmitting speech signals over wireless communication channels) are designed based on an assumption that the channels do not have memory. That is, most communication systems are configured based on theories, methods, expressions, etc. assuming that channels of the communication system have an conditional output probability distribution that depends only on a current input (i.e., the output of the channels is conditionally independent of previous channel inputs or outputs symbols and the source symbols). However, typical communication channels are not memoryless due to Inter-symbol interference (ISI), correlated channel noise, etc. As a result, most communication systems are configured with components (e.g., encoders) that do not operate optimally when transmitting information over channels (e.g., the components do not achieve the capacity of the channels with memory) and are, in many cases, overly complicated due to the lack of knowledge of capacity achieving properties of the encoders. A characterization of channel capacity and corresponding capacity achieving channel input distributions, which would allow for the design of capacity achieving encoders, is not known for most channels with memory.

Further, the field of communications has developed primarily based on the following model: a message is generated randomly by an information source, the message is encoded by an encoder, the message is transmitted as a signal over a noisy channel, and the transmitted signal is decoded to produce an output as an approximation of the message generated by the source. The fundamental problem of this model is to determine simultaneously what information should be transmitted (source coding) and how the information should be transmitted (channel coding) to achieve performance. Over the years, this fundamental problem has been separated into the two subproblems of source coding and channel coding. The first sub-problem, source coding, is related to efficient representation of information, such as information representing speech, so as to minimize information storage and to characterization of a minimum rate of compressing the information generated by the source of the information (e.g., via the classical Rate Distortion Function (RDF) of the source subject to a fidelity of reconstruction). The second sub-problem, channel coding or “error correction coding,” is related to a correction of errors arising from channel noise, such as flaws in an information/data storage or transmission system, loss of information packets in networks, failures of communications links, etc., and to the characterization of the maximum rate of information transmission, called “channel capacity.”

The general separation of the fundamental problem into source coding and channel coding sub-problems, has divided the community of developers into independent groups developing source codes and channel codes, respectively. Although, extremely useful in some contexts, this idealized separation is limiting future advances in communication technology, in that developers are ignoring practical design criteria, such as computational complexity, delay, and optimal performance. Further, the ideal separation of source coding and channel coding is often violated for point-to-point communications over channels with memory and for network communication systems. On the other hand, the optimal design of simultaneously performing data compression and channel coding is, in those known cases, elegantly simple. However, this optimal design is, in general, hard to find. For example, separation of source and channel coding leads to the design of channel codes which treat all information bits as equally important. However, a scenario in which all information bits are equally important (e.g., in achieving an optimal channel capacity) is rare, and, hence, a separation of source and channel coding can lead to performance degradation.

SUMMARY

In an embodiment, a method for characterizing a capacity of a channel with memory and feedback comprises defining a channel model corresponding to the channel, wherein: the channel is utilized to transmit information from a source to a destination, and the channel model indicates a dependence of outputs from the channel on past and present channel input symbols and on past channel output symbols. The method further includes determining a representation of the capacity based on the channel model and based on a channel input distribution that achieves the capacity, wherein the representation represents the capacity for a finite number of transmissions over the channel, and wherein the representation includes an optimization, and solving, by one or more processors of a specially configured computing device, the optimization of the representation of the capacity to determine the capacity of the channel for the finite number of transmissions and a per unit time limit of the capacity of the channel.

In another embodiment, a system comprises one or more processors and one or more non-transitory memories. The one or more non-transitory memories store computer-readable instructions that specifically configure the system such that, when executed by the one or more processors, the computer-readable instructions cause the system to: receive a channel model corresponding to the channel, wherein: the channel is utilized to transmit information from a source to a destination, and the channel model indicates a dependence of outputs from the channel on past and present channel input symbols and on past channel output symbols. the computer-readable instructions further cause the system to: determine a representation of the capacity based on the channel model and based on a channel input distribution that achieves the capacity, wherein the representation represents the capacity for a finite number of transmissions over the channel, and wherein the representation includes an optimization, and solve the optimization of the representation of the capacity to determine the capacity of the channel for the finite number of transmissions.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an example communication system for encoding and transmitting messages generated by a source;

FIG. 2 illustrates an example process to characterize a capacity of a channel, such as the channel illustrated in FIG. 1;

FIG. 3 is a flow diagram of an example method for characterizing a capacity and optimal channel input distribution with a two-step procedure;

FIG. 4 is a flow diagram of an example method for determining subsets of channel input distributions and a corresponding capacity based on step one of a two-step procedure, such as the two step procedure utilized in the method of FIG. 3;

FIG. 5 is a flow diagram of an example method for narrowing subsets of channel input distributions and determining a corresponding refined capacity based on step two of a two-step procedure, such as the two step procedure utilized in the method of FIG. 3;

FIG. 6 illustrates an example process for designing an information lossless encoder, which encoder may be implemented in the system illustrated in FIG. 1;

FIG. 7 illustrates an example encoding scheme which may be implemented by an information lossless encoder designed according to the process illustrated in FIG. 6;

FIG. 8 is a flow diagram of an example method for designing a capacity achieving and information lossless encoder, which encoder may be implemented in the system illustrated in FIG. 1;

FIG. 9 is a flow diagram of an example method for determining and utilizing a rate distortion function, which function may be utilized to configure the system illustrated in FIG. 1;

FIG. 10A depicts a combination of components in a communication system 1000 that realizes a rate distortion function, such as a rate distortion function determined in the method of FIG. 9;

FIG. 10B depicts an example realization of a nonanticipative rate distortion function for a vector of independent sources with memory;

FIG. 11 depicts another example communication system in which encoders and decoders are designed according to joint source channel coding;

FIG. 12 is a flow diagram of an example method for identifying and utilizing an {encoder, decoder} pair, which pair may be utilized in the system illustrated in FIG. 1;

FIG. 13 illustrates a realization of a finite-time RDF;

FIG. 14 illustrate example{encoder, decoder} pairs that realize an RDF and a capacity of a channel; and

FIG. 15 is a block diagram of an example computing device.

DETAILED DESCRIPTION

The techniques of the present disclosure facilitate a characterization of capacity and capacity achieving channel input distributions for channels with memory, with or without feedback, and with transmission cost. Further, encoders of the present disclosure satisfy necessary and sufficient conditions to achieve the capacity of channels with memory, with or without feedback, and with transmission cost, and methods of the present disclosure include determining whether an optimal transmission, for a given model of channels and transmission cost, is indeed real-time transmission. Encoders of the present disclosure may also be configured that such that the encoders simultaneously compress and encode information for sources with memory and/or with zero-delay, by performing a JSCC design.

To this end, a two-step procedure determines the structural properties of (i) capacity achieving encoders, and (ii) capacity achieving channel input distributions, for general channels with memory and with or without feedback encoding and transmission cost. More specifically, the two-step procedure identifies “multi-letter,” “finite block length” feedback capacity expressions along with the corresponding capacity achieving channel input distributions and encoders. By extension, the procedure identifies per unit time limiting feedback capacity formulas along with the corresponding capacity achieving channel input distributions and encoders.

Further, necessary and sufficient conditions allow for the design of an encoder with or without feedback that achieves the capacity of a channel with memory. These encoders, referred to herein as “information lossless” encoders, define a mapping of information from a source, which mapping generates the information to the output of the encoder. The mapping may be invertible such that no information is lost in the encoding process. For each of a plurality a channel classes, the determined capacity achieving input distributions, mentioned above, allow specific mapping (e.g., encoders) to be defined. These specific mappings adhere to the necessary and sufficient conditions for capacity achieving encoders.

Nonanticipative rate distortion functions (RDFs) of the present disclosure may achieve a zero-delay compression of information from a source. These nonanticipative RDFs may represent an optimal (e.g., capacity achieving) compression that is zero-delay. For example, the nonanticipative RDFs of the present disclosure may represent a scheme for compressing information that only depends on previously communicated symbols (e.g., is causal), not on all communicated symbols. The nonanticipative RDFs may also represent compression of both time-varying, or “nonstationary,” and stationary sources of information.

Still further, methods for designing encoding schemes may utilize Joint Source Channel Coding (JSCC) to generate encoding schemes for simultaneous compression and channel coding operating with zero-delay or in real-time. As opposed to having codes for compression and codes for channel coding of compressed information, encoding schemes of the present disclosure may utilize a single encoding scheme, generated via JSCC design, that provides both compression and channel coding with zero-delay. JSCC methods to design such encoding scheme may utilize characterized capacities, necessary and sufficient conditions of information lossless encoders, and nonanticipative RDFs as discussed above.

A. System Overview

FIG. 1 is a block diagram of an example communication system 100 including at least some components that may be configured according to and/or utilize the methods discussed herein. In the example system 100, a source 102 generates a message (e.g., including any suitable information). An encoder 104 encodes the generated message, and the encoded message is transmitted as a signal over a channel 106. A decoder 108 decodes and the transmitted signal to produce an output.

The source 102 may include one or more stationary or mobile computing or communication devices, in an implementation. For example, the source 102 may include a laptop, desktop, tablet, or other suitable computer generating messages including digital data, such as digital data representing pictures, text, audio, videos, etc. In other examples, the source 102 may be a mobile phone, smartphone, land line phone, or other suitable phone generating messages including signals representative of audio or text messages to one or more other suitable phones or communication devices. Generally, the source 102 may include any number of devices or components of devices generating messages to be encoded by the encoder 104 and transmitted over the channel 106. Further details of an example computing device, which may be implemented as the source 102, are discussed with reference to FIG. 15.

In particular, the example source 102 generates messages including source symbols xⁿ{x₀, x₁, . . . , x_n}, x_jεX_j, where j=0, 1, . . . , n, according to a source distribution P_Xn(dxⁿ). As discussed above, these symbols may represent any suitable information, such as photos, audio, video, etc. generated by or forwarded by the source 102.

The encoder 104 may include one or more devices, circuits, modules, engines, and/or routines communicatively and/or operatively coupled to the source 102. For example, the encoder 104 may be communicatively and/or operatively coupled to the source 102 via a bus of a computing device (e.g., a computing device implemented as the source 102), such as an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus or a Mezzanine bus, or the Peripheral Component Interconnect Express (PCI-E) bus, or the encoder 104 may be communicatively and/or operatively coupled to the source 102 via one or more wires or cables, such as Ethernet cables, ribbon cables, coaxial cables, etc. In any event, the encoder 104 receives messages (e.g., source symbols xⁿ) from the source 102.

The example encoder 104 encodes the received symbols xⁿ from the source 102 into channel input symbols aⁿ{a₀, a₁, . . . , a_n}, a_jεX_j, where j=0, 1, . . . , n. These channel input symbols have “induced” channel input distributions {P_{Ai|Bi−1,Ai−1}(da_i|bⁱ⁻¹,aⁱ⁻¹):i=1, 2, . . . , n} if there is feedback or {P_Ai|Ai−1(da_i|aⁱ⁻¹):i=1, 2, . . . , n} if there is no feedback encoding. Further details of encoders and the encoding of the generated source symbols is discussed with reference to FIGS. 6, 7, and 8.

The channel 106 may include any number of wires, cables, wireless transceivers, etc., such as Ethernet cables, configured to facilitate a transmission of the encoded source symbols from the encoder 104 to the decoder 108. Further details of various types of channels that may be implemented as the channel 106 are discussed in section B.2. entitled “Characterizing Channels.” The example channel 106 is noisy with memory defined by a sequence of conditional distributions, {P_Bi|Bi−1,Ai(db_i|bⁱ⁻¹,aⁱ):i=1, 2, . . . , n}. The channel produces a channel output bⁿ{b₀, b₁, . . . , b_n}, b_jεB_j, where j=0, 1, . . . , n, which channel output is input to the decoder 108.

The decoder 108 may include one or more devices, modules, engines, and/or routines communicatively and/or operatively coupled to one or more computing devices, phones, etc. to receive decoded information transmitted over the channel 106. The decoder 108 receives the channel output, B_j, and decodes the channel output to produce the decoded output Y_j. The decoder 108 may or may not use past channel outputs and decoder outputs to produce Y_j. For example, when the source 102 is a originating phone, the decoder 108 may be communicatively and/or operatively coupled to terminating phone to decode audio signals sent from the originating phone to the terminating phone.

In some implementations, the source 102 and/or encoder 104 may know (e.g., receive transmissions indicative of and/or store information indicative of) all, or at least some outputs (B_j) from the channel 106, before generating, encoding, and/or transmitting a next signal. This functionality may be referred to herein as “feedback.” In other words, a channel or communication system with feedback may be a system in which a source and/or encoder receive indications of, or otherwise “know,” all or at least some previous outputs from the channel before sending a subsequent signal over the channel.

B. Characterization of Capacities and Identification of Capacity Achieving Channel Input Distributions and Capacity Achieving Encoders

The methods described below allow the “finite block length” feedback capacity for channels, such as the channel 106, with memory and with and without transmission cost constraints on the encoders to be characterized, in some implementations. “Feedback capacity” may refer to a capacity of a channel with feedback, as discussed above, and “finite block length” may refer to a feedback capacity defined in terms of a finite number of transmissions or a finite time period such that the feedback capacity can be defined without taking a limit of infinite transmissions. In characterizing the finite block length feedback capacity, operators of communications systems may also identify the corresponding capacity achieving channel input distributions and the capacity achieving encoders. Operators may also utilize the characterization to determine whether feedback increases the finite block length capacity of channels without feedback, to characterize the finite block length capacity without feedback for general channels with memory and with or without transmission cost constraints on the encoders, and to identify the corresponding capacity achieving channel input distributions and capacity achieving encoders. By extension, the methods described below may allow a per unit time limiting version of the finite block length feedback capacity to determine the capacity and capacity achieving channel input distributions.

In some implementations, the characterization of the finite block length feedback capacity may facilitate the design of optimal encoding and decoding schemes that both reduce the complexity of communication systems and operate optimally. Operating optimally may include an optimal operation in terms of the overall number of processing elements (e.g., CPUs) required to process transmissions and/or the number of memory elements and steps required to encoder and decode messages. Such optimal encoding and decoding schemes may require small processing delays and short code lengths in comparison to encoding and decoding schemes designed based on an assumption of a channel without memory or based on a separate treatment of source codes and channel codes.

Architectures and methodologies discussed herein provide encoders and decoders based on channel characteristics, transmission cost requirements, and the characteristics of messages generated by a source (e.g., the source 102). Although, the characterizations, encoders, distributions, etc. discussed below are described, by way of example, with reference to the example communication system 100, which system 100 is a point-to-point communication system, characterizations, encoders, distributions, etc. may be applied to or implemented in systems other than point-to-point communication systems. For example, the characterizations, encoders, distributions, etc. of the present disclosure may be implemented in multi-user and network communication systems by repeating the procedures described below (for each user, for each node of a network, etc.).

B.1. Characterizing Channels

A determination and characterization of the capacity of a channel, such as the channel 106, may first include a characterization or description of the channel. The techniques of the present disclosure characterize a plurality of channel types with memory and with or without feedback allowing the capacity of these channels to be determined along with capacity achieving encoders and distributions, in some implementations. Generally, the characterization of a channel includes: (i) an “alphabet” defining the signals spaces (e.g., inputs and outputs) of a communication system; (ii) a “transmission cost function” defining a dependence of a rate of information transfer over the channel on the amount of energy or, more generally, any cost imposed on symbols transferred over the channel; and (iii) a model for the channel described by, for example, conditional distributions or stochastic discrete-time recursions in linear, nonlinear and state space form.

Channel input and output alphabets may, in some implementations, be complete separable metric spaces such as functions spaces of finite energy signals. These metric spaces may include, by way of example and without limitation, continuous alphabets, countable alphabets, and finite alphabets, such as real-valued R^p-dimensional and/or complex-valued C^p-dimensional alphabets for channel output alphabets and real-valued R^p-dimensional and/or complex-valued C^q-dimensional alphabets for channel input alphabets, finite energy or power signals, and bounded energy signals defined on metric spaces.

Transmission cost functions may include, by way of example, nonlinear functions of past and present channel inputs and past channel outputs or conditional distributions. The transmission cost functions define a cost of transmitting certain symbols over a channel, which cost is generally not the same for all symbols. For example, an energy, or other cost, required to transmit one symbol may differ from an energy required to transmit another symbol. Operators of a communication system may determine transmission cost functions, to utilize in the methods discussed below, by sending certain information (e.g., diagnostic or configuration information) over a channel and measuring an output of the channel.

Similarly, by sending certain information over a channel, operators of a communication system may determine a channel model. The channel model may model the behavior of the channel including a dependence (e.g., non-linear dependence) of transmission on past channel inputs, outputs and noise processes of memory. By way of example, the channel model may be a linear channel models with arbitrary memory, a Gaussian channel model, a state space model, or an arbitrary conditional distribution defined on countable, finite channel input and output alphabets, or continuous alphabet spaces. For waveform signals (e.g., continuous speech signals) transferred over a channel, the channel model may be a non-linear differential equation, and, for quantized signals (e.g., zeros and ones) transferred over a channel, the channel model may be a conditional distribution.

Models of channels with memory and feedback may include, by way of example, the following channel conditional distributions defining certain classes of channels:

Class A.

P _{B i |B i−1 ,A i} (db_i|bⁱ⁻¹,aⁱ)=P_Bi|Bi−1,Ai(db_i|bⁱ⁻¹,a_i), i=0, . . . , n.  1.

P _{B i |B i−1 ,A i} (db_i|bⁱ⁻¹,aⁱ)=P_{Bi|Bi−1,Ai−Li}(db_i|bⁱ⁻¹,a_i−Lⁱ), i=0, . . . , n.  2.

Class B.

P _{B i |B i−1 ,A i} (db_i|bⁱ⁻¹,aⁱ)=P_Bi,Bi−1,Ai(db_i|b_i−1,a_i), i=0, . . . , n.  1.

P _{B i |B i−1 ,A i} (db_i|bⁱ⁻¹,aⁱ)=P_{Bi,Bi−Mi−1,Ai}(db_i|b_i−Mⁱ⁻¹,a_i), i=0, . . . ,n.  2.

P _{B i |B i−1 ,A i} (db_i|bⁱ⁻¹,aⁱ)=P_{Bi,Bi−Mi−1,Ai}(db_i|b_i−Mⁱ⁻¹,a_i), i=0, . . . ,n.  3.

Class C.

P _{B i |B i−1 ,A i} (db_i|bⁱ⁻¹,aⁱ)=P_{Bi,Bi−2i−1,Ai−1i}(db_i|b_i−2ⁱ⁻¹,a_i−1ⁱ), i=0, . . . ,n.  1.

P _{B i |B i−1 ,A i} (db_i|bⁱ⁻¹,aⁱ)=P_{Bi,Bi−Mi−1,Ai−Li}(db_i|b_i−Mⁱ⁻¹,a_i−Lⁱ), i=0, . . . ,n.  2.

where {L,M} are nonnegative finite integers. The above example classes of channel conditional distributions may be induced by nonlinear channel models and linear time-varying Autoregressive models or by linear and nonlinear channel models expressed in state space form.

Classes of transmission cost functions may include, by way of example, the following:

Class A.

γ_i(Tⁿaⁱ,Tⁿbⁱ⁻¹)=γ_i^A.1(a_i,bⁱ⁻¹), i=0, . . . n,  1.

γ_i(Tⁿaⁱ,Tⁿbⁱ⁻¹)=γ_i^A.2(a_i−Nⁱ,bⁱ⁻¹), i=0, . . . n,  2.

Class B.

γ_i(Tⁿaⁱ,Tⁿbⁱ⁻¹)=γ_i^B.1(a_i,b_i−1), i=0, . . . n,  1.

γ_i(Tⁿaⁱ,Tⁿbⁱ⁻¹)=γ_i^B.2(a_i,b_i−Kⁱ⁻¹), i=0, . . . n,  2.

γ_i(Tⁿaⁱ,Tⁿbⁱ⁻¹)=γ_i^B.3(aⁱ,b_i−Kⁱ⁻¹), i=0, . . . n,  3.

Class C.

γ_i(Tⁿaⁱ,Tⁿbⁱ⁻¹)=γ_i^C.1(a_i−1ⁱ,b_i−2ⁱ⁻¹), i=0, . . . n,  1.

γ_i(Tⁿaⁱ,Tⁿbⁱ⁻¹)=γ_i^C.2(a_i−1ⁱ,b_i−Kⁱ⁻¹), i=0, . . . n,  2.

where {N,K} are nonnegative finite integers.

If M=0 in the above example classes of channels and transmission cost functions, then

P _{B i |B i−M i−1 ,Ā i} (db_i|b_i−Mⁱ⁻¹,āⁱ)|_M=0≡P_Bi|Āi(db_i|āⁱ) for Āⁱε{Aⁱ,A_i−Lⁱ}, i=0, 1, . . . ,n.

Hence, for M=L=0,

P _{B i | i−M i−1 ,A i−L i} (db_i|b_i−Mⁱ⁻¹,a_i−Lⁱ)|_M=L=0≡P_Bi|Ai(db_i|a_i), i=0,1, . . . ,n

is a memoryless channel. Similarly, if K=0 then

γ_i^C.2(a_i−Nⁱ,b_i−Kⁱ⁻¹)|_K=0≡γ_i^C.2(a_i−Nⁱ), i=0, . . . ,n.

B.2. Capacity Without Feedback

For further clarification, varous notations and expressions related to capacities without feedback are presented below. The problem of the capacity of channels with memory and without feedback includes a maximization of mutual information between channel input and output sequences over an admissible set of channel input distributions. The mutual information may be:

$I\left(A^n;B^n\right)\stackrel{\Delta }{=}E_{A^n,B^n}\left\{\log \left(\frac{P_{A^n,B^n}\left(·,·\right)}{\left(P_{A^n}(·)\times P_{B^n}(·)\right)}\left(A^n,B^n\right)\right)\right\}.$

The admissible set of channel input distributions may be:

_[0,n] ^noFB {P _{A i |A i−1} (da_i|aⁱ⁻¹):i=0,1, . . . ,n}

and the maximization of mutual information may be expressed as:

$C_{A^{\infty };B^{\infty }}^{\mathrm{noFB}}\underset{n->\infty }{\lim \sup }\frac{1}{n+1}C_{A^n;B^n}^{\mathrm{noFB}},C_{A^n;B^n}^{\mathrm{noFB}}=I\left(A^n;B^n\right)$

where P_An,Bn, (daⁿ, dbⁿ)=P_Bn,|An(dbⁿ|aⁿ)P_An(daⁿ) is referred to herein as “the joint distribution,” P_Bn(dbⁿ)=∫_AnP_Bn|An(dbⁿ|aⁿ)P_An(daⁿ) is referred to herein as “the channel output distribution,” P_Bn|An(dbⁿ,aⁿ)=_i=0ⁿP_Bi|Bi−1,Ai(db_i|bⁱ⁻¹,aⁱ)−a.a.(aⁿ,bⁿ) is referred to herein as “the channel distribution because the encoder does not utilize feedback, and E_An,Bn{•} denotes the expectation over the joint distribution of {Aⁿ,Bⁿ}.

Further, the extremum problems of capacity of channels with memory without feedback, when transmission cost is imposed on channel input distributions may be written, by way of example, as:

$C_{A^{\infty };B^{\infty }}^{\mathrm{noFB}}\left(\kappa \right)\underset{n\to \infty }{\mathrm{limsup}}\frac{1}{n+1}C_{A^n;B^n}^{\mathrm{noFB}},C_{A^n;B^n}^{\mathrm{noFB}}\left(\kappa \right)=I\left(A^n;B^n\right)$ $\mathrm{where}$ ${\left[0,n\right]}_{\mathrm{noFB}}^{}\left(\kappa \right)\left\{P_{A_i|A^{i-1}}\left({\mathrm{da}}_i|a^{i-1}\right),i=0,\dots ,n\text{:}\frac{1}{n+1}E\left\{\sum _{i=0}^n{\gamma }_i\left\{T^nA^i,T^nB^{n-1}\right)\right\}\le \kappa \right),\kappa \in \left[0,\infty \right)$ $\mathrm{and}$ $T^na^i\subseteq \left\{a_0,a_1,\dots ,a_i\right\},T^nb^{i-1}\subseteq \left\{b_0,b_1,\dots ,b_{i-1}\right\},i=0,1,\dots ,n$

In this notation, C_An;Bn^noFB; is referred to herein as the finite block length capacity without feedback, C_A∞;B∞^noFB is referred to herein as the capacity without feedback, C_An;Bn^noFB() is referred to herein as the finite block length capacity without feedback with transmission cost, and C_A∞;B∞^noFB() is referred to herein as the capacity without feedback with transmission cost.

B.3. Capacity with Feedback

For still further clarification, various notations and expressions related to capacities with feedback are presented below. The problem of the capacity of channels with memory and with feedback includes a maximization of directed information from channel input and output sequences over an admissible set of channel input distributions. The directed information may be written as:

$I\left(A^n\to B^n\right)\sum _{i=0}^nI\left(A^i;B_i|B^{i-1}\right)=\sum _{i=0}^nE_{A^i,B^i}\left\{\log \left(\frac{{\mathrm{dP}}_{B_i|B^{i-1},A^i}\left(·|B^{i-1},A^i\right)}{{\mathrm{dP}}_{B_i|B^{i-1}}\left(·|B^{i-1}\right)}\left(B_i\right)\right)\right\},$

and the admissible set of channel input distributions may be expressed as:

_[0,n] {P _{A i |A i−1 ,B i−1} (da_i|aⁱ⁻¹,bⁱ⁻¹):i=0,1, . . . ,n}

The maximization of directed information may be expressed as:

$C_{A^{\infty }\to B^{\infty }}^{\mathrm{FB}}\underset{n\to \infty }{\mathrm{limsup}}\frac{1}{n+1}C_{A^n\to B^n}^{\mathrm{FB}},C_{A^n\to B^n}^{\mathrm{FB}}I\left(A^n\to B^n\right)$

For each i=0, 1, . . . , n, {P_AiBi(daⁱ,dbⁱ),P_Bi|Bi−1(db_i|bⁱ⁻¹)} are referred to herein as the joint and conditional distributions induced by the channel and the channel input distributions, and E_Ai,Bi{•} denotes the expectation over the joint distribution of {Aⁱ,Bⁱ}.

Further, the extremum problems of capacity of channels with memory and with feedback, when transmission cost is imposed on channel input distributions may be written, by way of example, as:

$C_{A^{\infty }\to B^{\infty }}^{\mathrm{FB}}\left(\kappa \right)\underset{n\to \infty }{\mathrm{limsup}}\frac{1}{n+1}C_{A^n\to B^n}^{\mathrm{FB}}\left(\kappa \right),C_{A^n\to B^n}^{\mathrm{FB}}I\left(A^n\to B^n\right)$ $\mathrm{where}$ ${\left[0,n\right]}_{}\left(\kappa \right)\left\{P_{A_i|A^{i-1},B^{i-1}},i=1,\dots ,n\text{:}\frac{1}{n+1}E\left\{\sum _{i=0}^n{\gamma }_i\left(T^nA^i,T^nB^{i-1}\right)\right\}\le \right\}.$

In this notation, C_An→Bn^FB is referred to herein as the finite block length feedback capacity, C_A∞→B∞^FB is referred to herein as the feedback capacity, C_An→Bn^FB() is referred to herein as the finite block length feedback capacity with transmission cost, and C_A∞→B∞^FB() is referred to herein as the feedback capacity with transmission cost.

B.4. Characterization of Capacities and Identification of Optimal Channel Input Distributions Overview

FIG. 2 illustrates an example process 200 for characterizing capacities and identifying optimal channel input distributions. The capacities characterized via the process 200 may include finite block length feedback capacities, feedback capacities, finite block length feedback capacities with transmission cost, feedback capacities with transmission cost, finite block length capacities without feedback, capacities without feedback, finite block length capacities without feedback and with transmission cost, and capacities without feedback and with transmission cost. These types of capacities are further described in sections B.2. and B.3. entitled “Capacity Without Feedback” and “Capacity With Feedback,” respectively. A computing device, such as the computing device illustrated in detail in FIG. 15, may be specially and/or specifically configured (e.g., by one or more algorithms, routines, modules, or engines) to implement at least a portion of the process 200. Further, components of the example communication system 100, such as the encoder 104, may be configured according to the output of the process 200.

In the process 200, a channel model 202 and a transmission cost function 204 are input into a two-step characterization procedure 206. The channel model 202 and transmission cost function 204 may describe one of the plurality of channel classes and transmission cost function classes discussed in section B.1. entitled “Characterizing Channels.” In particular, the channel model 202 may describe the noise of a channel and dependencies amongst channel inputs and channel output via one or more conditional distributions and/or one or more non-linear differential equations. The transmission cost function 204 may define a cost of sending each of a plurality of symbols (e.g., defined in an alphabet) over the channel described by the channel model 202.

The two-step characterization procedure 206 may, based on the channel model 202 and a transmission cost function 204 produce an optimal channel input distribution 208 and a finite block length feedback capacity 210, in the scenario illustrated in FIG. 2. Although the finite block length feedback capacity 210 is illustrated in FIG. 2, the two-step characterization procedure may characterize other capacities, such as C_An→Bn^FB(), C_A∞→B∞^FB(), C_An;Bn^noFB, or C_An;Bn^noFB() depending on the class of channel described by the channel model 202 and the class of the transmission cost function 204. Further details of an implementation of the two-step characterization procedure 206 are discussed with reference to FIGS. 3, 4, and 5. Generally, a first step of the two-step characterization procedure 206 may include utilizing stochastic optimal control to identify subset of channel input distributions, and a second step of the two-step characterization procedure 206 include utilize a variational equality to further narrow the subsets of channel input distributions to the optimal channel input distribution 208.

In some implementations, the process 200 also includes a per unit time conversion 212 of the optimal channel input distribution 208 and the finite block length feedback capacity 210. This per unit time conversion 212 converts the finite block length feedback capacity 210 to a feedback capacity 213, which feedback capacity 214 describes the capacity of the channel as a number of transmissions approaches infinity. The per unit time conversion 212 additionally generates a feedback capacity achieving input distribution 216 corresponding to the feedback capacity 214. Although the feedback capacity 214 is illustrated in FIG. 2, the per unit time conversion 212 may generate other capacities, such as C_A∞→B∞^FB(), C_A∞;B∞^noFB, or C_A∞;B∞^noFB(), depending on the class of channel described by the channel model 202 and the class of the transmission cost function 204. Further details of implementations of the per unit time conversion 212 are discussed below.

B.5. Two-Step Procedure for Characterizing Capacity and Identifying Optimal Input Distributions

As discussed with reference to FIG. 2, the two-step characterization procedure 206 may, based on a channel model and a transmission cost function, produce the optimal channel input distribution 208 and a specific characterization of (e.g., a formula of) a capacity, such as the finite block length feedback capacity 210. In an implementation, step one of the two-step characterization procedure 206 utilizes techniques from stochastic optimal control with relaxed or randomized strategies (e.g., conditional distributions), and step two of the two-step characterization procedure 206 may utilizes a variational equality of directed information, as described further below.

Given a specific channel distribution and a specific transmission cost function from the classes described in section B.1. entitled “Characterizing Channels,” step one of the two-step characterization procedure 206 may include applying stochastic optimal control to show a certain joint process which generates the information structure of the channel input distribution at each time is an extended Markov process. For an example, for the channel input distribution (at every time) describing a channel with memory:

P_{Ai|Ai−1,Bi−1}(da_i|aⁱ⁻¹,bⁱ⁻¹)

the joint process

_i ^P ⊂{a ⁱ⁻¹ ,b ⁱ⁻¹}, for i=0,1, . . . ,n,

is an extended Markov process, with respect to a smaller information structure

_i ^P ⊂{a ⁱ⁻¹ ,b ⁱ⁻¹},

for i=0, 1, . . . , n. Based on this joint process, the optimal channel input distribution corresponding to C_An→Bn^FB and C_An→Bn^FB () is included in specific subsets of input distributions:

_[0,n] ⊂{P _i(da_i|aⁱ⁻¹,bⁱ⁻¹): i=0, . . . ,n}ε_[0,n] and _[0,n]()⊂_[0,n]().

Thus, step one of the two-step characterization procedure 206 narrows all possible channel input distributions to specific subsets of input distributions, where the subsets of input distributions include the optimal channel input distribution.

Further, in some implementations of the step one of the two-step characterization procedure 206, the first step may include a characterization of capacity (e.g., finite block length feedback capacity) corresponding to the determined subsets of input distributions. In particular, step one of the two-step characterization procedure 206 may include generating a formula or other expression representing a capacity corresponding to the determined subsets of input distributions. For the channel input distribution describing a channel with memory (discussed above), the formulas for finite block length feedback capacity and feedback capacity with and without transmission cost are:

${\stackrel{\_}{C}}_{A^{\infty }\to B^{\infty }}^{\mathrm{FB}}=\underset{n\to \infty }{\mathrm{limsup}}\frac{1}{n+1}{\stackrel{\_}{C}}_{A^n\to B^n}^{\mathrm{FB}},{\stackrel{\_}{C}}_{A^n\to B^n}^{\mathrm{FB}}I\left(A^n\to B^n\right)$ ${\stackrel{\_}{C}}_{A^{\infty }\to B^{\infty }}^{\mathrm{FB}}\left(\kappa \right)=\underset{n\to \infty }{\mathrm{limsup}}\frac{1}{n+1}{\stackrel{\_}{C}}_{A^n\to B^n}^{\mathrm{FB}}\left(\kappa \right),{\stackrel{\_}{C}}_{A^n\to B^n}^{\mathrm{FB}}\left(\kappa \right)I\left(A^n\to B^n\right)$

where I(Aⁿ→Bⁿ)=I( π,P) is a specific functional of the channel input distribution πε_[0,n] and the channel conditional distribution Pε{Class A, Class B, Class C} as described in section B.1. entitled “Characterizing Channels.”

Step two of the two-step characterization procedure 206 includes applying a variational equality of directed information to the subsets of input distributions determined in step one. In this manner, step two of the two-step characterization procedure 206 further narrows the determined subsets of input distributions, in some implementations. In particular, for the example above involving a channel with memory, an upper bound is achievable over the determined subsets of input distributions is expressed as:

_[0,n]⊂_[0,n] and _[0,n]()⊂_[0,n]().

Based on such an upper bound or further narrowing of input distributions, step two may further include determining a refined capacity based on the narrowed input distributions. For the example above of a channel with memory, the characterization of finite block length feedback capacity and feedback capacity with and without transmission cost, obtained from step two is:

${\stackrel{°}{C}}_{A^{\infty }\to B^{\infty }}^{\mathrm{FB}}=\underset{n\to \infty }{\mathrm{limsup}}\frac{1}{n+1}{\stackrel{°}{C}}_{A^n\to B^n}^{\mathrm{FB}},{\stackrel{°}{C}}_{A^n\to B^n}^{\mathrm{FB}}I\left(A^n\to B^n\right)$ ${\stackrel{°}{C}}_{A^{\infty }\to B^{\infty }}^{\mathrm{FB}}\left(\kappa \right)=\underset{n\to \infty }{\mathrm{limsup}}\frac{1}{n+1}{\stackrel{°}{C}}_{A^n\to B^n}^{\mathrm{FB}}\left(\kappa \right),{\stackrel{°}{C}}_{A^n\to B^n}^{\mathrm{FB}}\left(\kappa \right)I\left(A^n\to B^n\right)$

where I(Aⁿ→Bⁿ)=I({dot over (π)},P) is a specific functional of the channel input distribution {dot over (π)}ε_[0,n] and the channel conditional distribution Pε{Class A, Class B, Class C} as described in section B.1. entitled “Characterizing Channels.”

FIG. 3 is a flow diagram of an example method 300 for characterizing a capacity and optimal channel input distribution with a two-step procedure, such as the two-step procedure 206. A computing device, such as the computing device illustrated in detail in FIG. 15, may be specially and/or specifically configured to implement at least a portion of the method 300. Further, in some implementations, a suitable combination of a computing device and an operator of a communication system, such as the communication system 100, may implement the method 300.

In the method 300, a computing device or operator receives a channel model and transmission cost function (block 302). The received channel model may include a channel conditional distribution, such as one of the channel conditional distributions Pε{Class A, Class B, Class C} as described in section B.1. entitled “Characterizing Channels,” or the channel model may include other types of functions, such as non-linear differential equations, for channels transmitting continuous signals (e.g., speech signals). Also, the transmission cost function may include one of the transmission cost functions as described in section B.1. entitled “Characterizing Channels.”

The computing device and/or operator then applies step one of a two-step procedure (block 304). In step one, the computing device and/or operator utilizes stochastic optimal control to determine subsets of input distributions and a corresponding capacity formula based on the received channel model and transmission cost function. The determine subsets of input distributions may include the optimal channel input distributions. That is the determined subsets of input distributions may include the channel input distribution that achieves the corresponding capacity of the channel. Further details of step one of the two-step procedure are discussed with reference to FIG. 4.

For certain channel models and transmission cost functions, step one may be sufficient to identify the information structures of channel input distributions and characterize a capacity (e.g., the finite block length feedback capacity and the feedback capacity with and without transmission cost) of the channel. In other cases, step one may only be an intermediate step before step two of the two-step procedure. More specifically, step one may be sufficient for channel conditional distributions which only depend on all previous channel outputs and costs (e.g., not a current and/or previous channel input). As such, the method 300 includes determining if the channel, described by the received channel model, is dependent on more than the previous channel outputs and costs (block 306). If the channel is only dependent on all previous channel outputs and costs, the flow continues to block 310. If the channel is dependent on more than the previous channel outputs and costs (e.g., channel inputs), the flow continues to block 308.

At block 308, the computing device and/or operator applies step two of the two-step procedure. In step two, the computing device and/or operator utilizes a variational equality to further narrow the subsets of input distributions determined at block 304. Further, the computing device and/or operator determines a refined capacity formula, such as a finite block length feedback capacity, based on the further narrowed subsets of input distributions. In some implementations, the further narrowing of the input distributions based on the variational equality includes identifying a single optimal channel input distribution, and, in other implementations, the further narrowing of the input distributions based on the variational equality includes identifying a smaller subset of channel input distribution including fewer distributions than the subsets determined at block 304.

The computing device and/or operator solves the capacity formula (determined at block 304) and/or refined capacity formula (determined at block 308) to determine the capacity of the channel (block 310). The capacity formulas solved at block 308 may include maximizations or other extremum problems, as further discussed above. The computing device and/or operator may utilize various techniques to solve these maximizations or extremum problems including, by way of example and without limitation, dynamic programming and/or stochastic calculus of variations (e.g., the stochastic maximum principle).

FIG. 4 is a flow diagram of an example method 400 for determining subsets of channel input distributions and a corresponding capacity based on step one of the two-step procedure (discussed with reference to FIG. 3). A computing device, such as the computing device illustrated in detail in FIG. 15, may be specially and/or specifically configured to implement at least a portion of the method 400. Further, in some implementations, a suitable combination of a computing device and an operator of a communication system, such as the communication system 100, may implement the method 400.

In the method 400, a computing device and/or operator determines, for a specific channel described by a channel model, a process which generates a structure of the channel input distribution (block 402). The computing device and/or operator may utilize techniques from stochastic optimal control to optimize a “pay-off” (e.g., a finite block length capacity of the specific channel) over all possible processes (e.g., distributions). As discussed above for an example channel with memory, the determination of the process may include demonstrating that a certain joint process which generates the information structure of the channel input distribution at each time is an extended Markov process.

The computing device and/or operator may then determine a smaller set of processes (e.g., distributions) optimizing the capacity (block 404). This smaller set of distributions or processes includes the optimal distribution or process that achieves the finite block length capacity, or “pay-off.” For example, the smaller set of distributions may include only some of a complete set of possible channel input distributions, where the smaller set, or subset, includes the optimal channel input distribution.

The computing device and/or operator also determines a capacity formula based on the determined subset of process or distributions (block 406). As discussed further in sections B.2. and B.3. entitled “Capacity Without Feedback” and “Capacity With Feedback,” respectively, formulas for capacities may be expressed in terms of channel input distributions. Thus, upon determining subsets of channel input distributions at block 404, the computing device and/or operator may generate a formula for capacity, for the specific channel, based on the subsets of channel input distributions.

FIG. 5 is a flow diagram of an example method 400 for narrowing subsets of channel input distributions and determining a corresponding refined capacity based on step two of the two-step procedure (discussed with reference to FIG. 3). A computing device, such as the computing device illustrated in detail in FIG. 15, may be specially and/or specifically configured to implement at least a portion of the method 500. Further, in some implementations, a suitable combination of a computing device and an operator of a communication system, such as the communication system 100, may implement the method 500.

In the method 500, a computing device and/or operator applies a variational equality to subsets of input distributions (block 502). For example, the computing device and/or operator may apply the variational equality of directed information further described in C. D. Charalambous et al., “Directed information on abstract spaces: Properties and variational equalities,” http://arxiv.org/abs/1302.3971, submitted Feb. 16, 2013. Such an application of a variation equality may generate an upper bound over the subsets of input distributions (block 504). That is, the application of the variational equality further narrows subsets of input distributions (e.g., determined according to the example method 400) based on techniques specific to information theory (e.g., directed information). The computing device and/or operator also determines a refined capacity formula based on the further narrowed subsets of input distributions (block 506).

B.6. Example Characterizations of Capacities and Identification of Channel Input Distributions for Channels in Class A

By way of example, example finite block length feedback capacity formulas and input distributions, for example classes of channels, determined according the two-step procedure (described with reference to FIGS. 2, 3, 4, and 5) are presented below. The corresponding feedback capacities with and without transmission cost are limiting versions of the finite block length feedback capacities presented below.

B.6.1. Class A.1.

For an example channel condition distribution,

{P_Bi|Bi−1,Ai(db_i|bⁱ⁻¹,a_i): i=0,1, . . . ,n}

the optimal channel input distribution C_An;Bn^noFB is included in the subset:

_[0,n] ^A.1 {P _{A i |B i−1} (da_i|bⁱ⁻¹): i=0, . . . ,n}⊂_[0,n].

As such, for each i=0, 1, n, the information structures of the maximizing channel input distribution (according to the example two-step procedure described in section B.5. entitiled “Two-Step Procedure for Characterizing Capacity and Identifying Optimal Input Distributions”) is:

_i ^P {b ⁱ⁻¹ }⊂{a ⁱ⁻¹ ,b ⁱ⁻¹}.

The characterization of the finite block length feedback capacity is then

$C_{A^n\to B^n}^{\mathrm{FB},A\mathrm{.1}}\sum _{i=0}^n\int \log \left(\frac{{\mathrm{dP}}_{B_i|B^{i-1},A_i}\left(·|b^{i-1},a_i\right)}{{\mathrm{dP}}_{B_i|B^{i-1}\left(·|b^{i-1}\right)}}\left(b_i\right)\right)P_{B^i,A_i}\left({\mathrm{db}}^i,{\mathrm{da}}_i\right)=\sum _{i=0}^nI\left(A_i;B_i|B^{i-1}\right)$ $\mathrm{where}$ $P_{B^i,A_i}\left({\mathrm{db}}^i,{\mathrm{da}}_i\right)=P_{B_i|B^{i-1},A_i}\left({\mathrm{db}}_i|b^{i-1},a_i\right)\otimes P_{A_i[B^{i-1}}\left({\mathrm{da}}_i|b^{i-1}\right)\otimes P_{B^{i-1}}\left({\mathrm{db}}^{i-1}\right),i=0,1,\dots ,n.\mathrm{and}$ $P_{B_i|B^{i-1}}\left({\mathrm{db}}_i|b^{i-1}\right)=\int P_{B_i|B^{i-1},A_i}\left({\mathrm{db}}_i|b^{i-1},a_i\right)\otimes P_{A_i|B^{i-1}}\left({\mathrm{da}}_i|b^{i-1}\right),i=0,\dots ,1,n.$

If a transmission cost, such as

γ_i^A.1(a_i,bⁱ⁻¹), γ_i^B.1(a_i,b_i−1), γ_i^B.2(a_i,b_i−Kⁱ⁻¹), i=0,1, . . . ,n

is imposed, then the example characterization of the finite block length feedback capacity is:

$C_{A^n\to B^n}^{\mathrm{FB},A\mathrm{.1}}\left(\kappa \right)=\sum _{i=0}^nI\left(A_i;B_i|B^{i-1}\right).$

B.6.1.1. Example Non-Linear and Linear Channel Models

Channel distribution of example class A.1 (as defined in section B.1. entitled “Characterizing Channels”) may include one or both of distributions defined on finite and countable alphabet space and distributions defined on continuous alphabet spaces, which distributions may be induced by the models described below.

Let

${}_i{ℝ}^q,{}_i{ℝ}^p,V_i{ℝ}^r,h_i:{}^{i-1}\times {}_i\times {}_i\mapsto {}_i,i=0,1,\dots ,n$

where {h_i(•,•,•):i=0, 1, . . . } are measurable functions, and {V_i:i=0, 1, . . . } is a random channel noise process with joint distribution:

P_vn(dvⁿ) on (ⁿ).

A recursive expression as follows may define an example nonlinear channel model, for a channel in the example class A.1, with a continuous alphabet:

B _i =h _i(Bⁱ⁻¹,A_i,V_i), i=0, . . . ,n

where transmission in the example model begins at time i=0, and the initial data

B ⁻¹ b _−∞ ⁻¹ ,A ₀ a ₀ ,V ₋₁ v ₋₁

are specified according to the covention utilized in the model. For example, this data may be taken to be the null set of data or any available data prior to transmission time iε{−∞, . . . , −2, −1}.

If the channel noise process {V_i:i=0, 1, . . . } is an independent sequence (e.g., non-necessarily stationary), then the above recursive expression for the example nonlinear channel model may be utilized in step one of the two-step process with the channel probability distribution

$\begin{array}{c}\mathbb{P}\left\{B_i\in A|B^{i-1}=b^{i-1},A^i=a^i\right\}=\mathbb{P}\left\{V_i\text{:}b_i\left(b^{i-1},a_i,V_i\right)\in A\right\},\Gamma \in \left({}_i\right)\\ =Q_i\left(\Gamma |b^{i-1},a_i\right),i=0,1,\dots ,n.\end{array}$

Another recursive expression as follows may define an example linear channel model, for a channel in the example class A.1:

$B_i=-\sum _{j=0}^{i-1}C_{i,j}B_j+A_{i,i}A_i+V_i,i=0,\dots ,n$

where, for each i=0, 1, . . . n, the coefficients {C_i,j, D_i,i:i=0, 1, . . . n, j=0, 1, . . . , i−1} are real-valued matrices with dimensions p by p and p by q, respectively (e.g., with {p,q} being positive integers).

With such a linear model, the channel input distribution is obtained from

$\begin{array}{c}\left\{B_i\in A|B^{i-1}=b^{i-1},A^i=a^i\right\}=\left\{V_i\text{:}h_i\left(b^{i-1},a_i,V_i\right)\in A\right\},\Gamma \in \left({}_i\right)\\ =Q_i\left(\Gamma |b^{i-1},a_i\right),i=0,1,\dots ,n.\end{array}$

and the finite block length feedback capacity is characterized using step one of the two-step procedure, as further described in section B.5. entitled “Two-Step Procedure for Characterizing Capacity and Identifying Optimal Input Distributions.”

B.6.1.2. Example MIMO, ANonGN, and AGN Channel Models

Another example linear channel model, defined by linear dynamics, is:

$B_i=-C_{1,i-1}B_{i-1}+D_{i,1}A_i+V_i,i=0,\dots ,n,\frac{1}{n+1}\sum _{i=0}^nE\left\{{\gamma }^{B\mathrm{.1}}\left(A_i,B_{i-1}\right)\right\}\le \kappa$

An example channel noise is non-Gaussian, independent:

P _{V n} (dvⁿ)=Π_i=0ⁿP_Vi(dv_i),

with a zero mean and covariance matrix:

μ_ViE(V_i)=0,K_ViE(V_iV_i^T), i=0,1, . . . ,n,

and an average transmission cost

$\frac{1}{n+1}E\left\{\left|A_i\right|\right\}\le \kappa$

B.6.1.2.1. Example MIMO, ANonGN Channel with Memory

For an example channel model as described above with memory, a channel input distribution is given by:

{B_i≦b_i|Bⁱ⁻¹=bⁱ⁻¹,Aⁱ=aⁱ}={V_i≦b_i+Σ_j=0ⁱ⁻¹C_i,jb_j−D_i,ja_i}, i=0,1, . . . ,n.

and a characterization, according to step one of the two-step procedure, of the finite block length feedback capacity, with transmission cost, is given by:

$C_{A^n\to B^n}^{\mathrm{FB},\mathrm{ANonGN}-A\mathrm{.1}}\left(\kappa \right)=\underset{\left\{{\pi }_i\left({\mathrm{da}}_i|b^{i-1}\right),i=0,,n\text{:}\frac{1}{n+1}{\Sigma }_{i=0}^nE\left|A_i\right|\le \kappa \right\}}{\sup }\left\{\sum _{i=0}^nH\left(B_i|B^{i-1}\right)\right\}-H\left(V^n\right)$ $\mathrm{with}$ $\left\{B_i\le b_i|B^{i-1}=b^{i-1}\right\}={\int }_{b_i}\left\{V_i\le b_i+\sum _{j=0}^{l-1}C_{i,j}b_j-D_{i,j}a_i\right\}{\pi }_i\left({\mathrm{da}}_i|b^{i-1}\right),i=0,1,\dots ,n.$

The information structure of the example channel input distribution

{π_i(da_i|bⁱ⁻¹)≡P_Ai|Bi−1(a_i|bⁱ⁻¹): i=0,1, . . . }

implies that a measurable function:

$e_i\text{:}{}^{i-1}\times {}_i\to {}_i,{}_i{ℝ}^p,a_i=e_i\left(b^{i-1},u_i\right),i=0,1,\dots n,$

exists, where {U_i: i=0, 1, . . . n} is a p-dimensional random process with distribution {P_Ui(du_i): i=0, 1, . . . n} such that

{U_i:e_i(bⁱ⁻¹,Uⁱ)εda_i}=P_Ai|Bi−1(da_i|bⁱ⁻¹), i=0,1, . . . ,n.

Further, according to the definition of the linear channel model, B_i,

$A_i=e_i\left(B^{i-1},U_i\right),i=0,1,\dots ,n,B_i=\sum _{j=0}^{i-1}C_{i,j}B_j+D_{i,i}e_i\left(B^{i-1},U_i\right)+V_i,i=0,1,\dots ,n,\frac{1}{n+1}\sum _{i=0}^nEe_i\left(B^{i-1},U_i\right)\le \kappa .$

define a class of example admissible functions:

_0,n ^{AN on GN−A.1−IL}(){e_i(bⁱ⁻¹,u_i), i=0, . . . ,n: for a fixed bⁱ⁻¹ the function e_i(bⁱ⁻¹,•)

is one-to-one and onto for i=0, . . . , n,

$\frac{1}{n+1}{\Sigma }_{i=0}^nE\left|e_i\left(B^{i-1},U_i\right)\right|\le \kappa \}$

Another alternative characterization of the finite block length capacity with transmission cost is:

$\begin{array}{c}{C}_{A^n->B^n}^{\mathrm{FB},\mathrm{AN}\mathrm{on}\mathrm{GN}-A\mathrm{.1}-\mathrm{IL}}\left(\kappa \right)=\underset{{\left\{P_{U_i}\right\}}_{i=0}^n,{\left\{e_i\left(·,·\right)\right\}}_{i=0}^n\in {0,n}_{\mathrm{AN}\mathrm{on}\mathrm{GN}-A\mathrm{.1}-\mathrm{IL}}^{}}{\max }\sum _{i=0}^nH^e\left(B_i|B^{i-1}\right)-H\left(V^n\right).\\ \equiv \underset{{\left\{P_{U_i}\right\}}_{i=0}^n,{\left\{e_i\left(·,·\right)\right\}}_{i=0}^n\in {0,n}_{\mathrm{AN}\mathrm{on}\mathrm{GN}-A\mathrm{.1}-\mathrm{IL}}^{}}{\max }\sum _{i=0}^nI^e\left(U_i;B_i|B^{i-1}\right)\end{array}$ $\mathrm{where}$ $\mathbb{P}\left\{B_i\le b_i|B^{i-1}=b^{i-1}\right\}={\int }_{{}_i}\mathbb{P}\left\{V_i\le b_i+\sum _{j=0}^{i-1}C_{i,j}b_j-D_{i,i}e_i\left(b^{i-1},u_i\right)\right\}P_{U_i|B^{i-1}}\left(u_i|b^{i-1}\right),i=0,1,\dots ,n.$

The characterization, or capacity formula, may be solved to find the capacity. For example, a computing device may solve the maximization with dynamic programming or another suitable method as further discussed with reference to FIG. 3.

B.6.1.2.2. Example AGN Channel with Memory

In another example case, the channel noise process is Gaussian:

V _i ˜N(0,K_Vi), i=0,1, . . . ,n,

or approximately Gaussian. By the entropy maximizing property of Gaussian distributions, the finite block length feedback capacity is bounded from above by the inequality

H(Bⁿ)≦H(B^g,n),

where

B ^g,n {B _i ^g :i=0,1, . . . ,n}

is Gaussian distributed. This upper bound may be achieved when

{π_i(da_i|bⁱ⁻¹)≡P_Ai|Bi−1^g(a_i|bⁱ⁻¹):i=0,1, . . . }

is conditional Gaussian and the average transmission cost is satisfied, implying that

{P_Bi|Bi−1(b_i|bⁱ⁻¹)≡P_Bi|Bi−1^g(b_i|bⁱ⁻¹):i=0,1, . . . ,n}

is also conditionally Gaussian.

Similar to the procedure described in section B.6.1.2.1, a measurable function

$e_i\text{:}{}^{i-1}\times {}_i\to {}_i,{}_i\stackrel{\Delta }{=}{ℝ}^p,a_i=e_i\left(b^{i-1},u_i\right),i=0,1,\dots ,n$

exists such that

{U_i:e_i(bⁱ⁻¹,U_i)εda_i}=P_Ai|Bi−1^g(da_i|bⁱ⁻¹), i=0,1, . . . ,n.

Because the channel output is defined by the example linear channel model, B_i,

$B_i^g=-\sum _{j=0}^{i-1}C_{i,j}B_j^g+D_{i,i}e_i\left(B^{g,i-1},U_i\right)+V_i,i=1,\dots ,n,\frac{1}{n+1}\sum _{i=0}^n\mathrm{tr}\left(\mathrm{Cov}\left(A_i\right)\right)\le \kappa .$

Moreover, the corresponding channel input process denoted by

A ^g,n {A _i ^g :i=0,1, . . . ,n}

is Gaussian distributed, satisfying the average transmission cost constraint. Also, {U_i: i=0, 1, . . . n} is Gaussian independent of B^g,i−1 for any i=0, 1, . . . , n, and

$A_i^g=g_i\left(B^{i-1},U_i\right)=\sum _{j=0}^{i-1}{\Gamma }_{i,j}B_j^g+U_i,i=0,1,\dots ,n.B_i^g=\sum _{j=0}^{i-1}\left(D_{i,i}{\Gamma }_{i,j}-C_{i,j}\right)B_j^g+D_{i,i}U_i+V_i,i=1,\dots ,n.$

In terms of

Γ_i[Γ_i,0Γ_i,1 . . . Γ_i,i−1]·K_Bg,i−1E{B^g,i−1(B^g,i−1)T}, i=0,1, . . . ,n.

the average transmission cost may be:

$\sum _{i=0}^nE{A_i^g}_{{ℝ}^k}^2=\sum _{i=0}^n\left({\Gamma }_iK_{B^{g,i-1}}{\Gamma }_i^T+K_{U_i}\right).$

Thus, the finite block length feedback capacity formula, in this example case, is characterized by:

$C_{A^n->B^n}^{\mathrm{FB},\mathrm{AWGNM}-A\mathrm{.1}}\left(\kappa \right)=\underset{\left\{{\left\{{\Gamma }_{i,j},K_{U_i}\right\}}_{i=0,j=0}^{n,n-1};\frac{1}{n+1}{\Sigma }_{i=0}^n\left({\Gamma }_iK_{B^{g,i-1}}{\Gamma }_i^T+K_{U_i}\right)\le \kappa \right\}}{\max }\frac{p}{2}\sum _{i=0}^n\log \frac{D_{i,i}K_{U_i}D_{i,i}^T+K_{V_i}}{K_{V_i}}.$

The covariance matrices {K_B^g,i−1: i=0, 1, . . . n} may be found from B_i^g.

If a process {X_i: i=0, 1, . . . , n} of a source, such as the source 102, intended for transmission over this channel is R^p-valued, Gaussian distributed, and Markov, i.e.,

P _{X i |X i−1} (dx_i|xⁱ⁻¹)=P_Xi|Xi−1(dx_i|x_i−1), i=0,1, . . . ,n},

and the matrices

{Γ_i,j*,K_Ui*:i=0,1, . . . ,n, j=0,1, . . . ,n−1}

are the matrices maximizing the above expression, then the coding scheme which achieves the finite block length feedback capacity, in this case, is:

$A_i^{g,*}=\sum _{j=0}^{i-1}{\Gamma }_{i,j}^{*}B_j^g+{\Delta }_i^{*}\left\{X_i-E\left\{X_i|B^{g,i-1}\right\}\right\}i=0,1,\dots ,n,{\Delta }_i^{*}=K_{U_i}^{*,\frac{1}{2}}{\left\{\mathrm{Cov}\left(X_i-E\left\{X_i|B^{g,i-1}\right\}\right)\right\}}^{-\frac{1}{2}},i=0,1,\dots ,n.$

B.6.2. Class A.2.

For an example channel condition distribution,

{P_{Bi|Bi−1,Ai−Li}:i=0,1, . . . ,n}

the optimal channel input distribution for the finite block length feedback capacity, C_An;Bn^noFB, is included in the subset of channel input distributions:

_[0,n] ^A.2 {P _{A i |A i−L i−1 ,B i−1} (da_i|a_i−Lⁱ⁻¹,bⁱ⁻¹):i=0, . . . ,n}⊂_[0,n]

where

P _{A i |A i−L i−1 ,B i−1} (da_i|a_i−Lⁱ⁻¹,bⁱ⁻¹) for i=0,1, . . . ,L

may be determined from the convention used in the channel model. For example:

P _{A i |A i−L i−1 ,B i−1} (da_i|a_i−Lⁱ⁻¹,bⁱ⁻¹)=P_Ai|Bi−1(da_i|bⁱ⁻¹), i=0,1, . . . ,L.

The characterization of the finite block length feedback capacity, in this case, is:

$C_{A^n->B^n}^{\mathrm{FB},A\mathrm{.2}}\sum _{i=0}^n\int \log \left(\frac{P_{B_i|B^{i-1},A_{i-L}^i}\left(·|b^{i-1},a_{i-L}^i\right)}{P_{B_i|B^{i-1}}\left(·|b^{i-1}\right)}\left(b_i\right)\right)P_{B^i,A_{i-L}^i}\left(b^i,a_{i-L}^i\right)=\sum _{i=0}^nI\left(A_{i-L}^i;B_i|B^{i-1}\right).$

If a transmission cost is imposed corresponding to any

γ_i^A.2(a_i−Nⁱ,bⁱ⁻¹), γ_i^C.2(a_i−Nⁱ,b_i−Kⁱ⁻¹), i=0,1, . . . ,n.

then the characterization of the finite block length feedback capacity with transmission cost, in this example case, is:

$C_{A^n->B^n}^{\mathrm{FB},A\mathrm{.2},L\Lambda N}\left(\kappa \right)=\sum _{i=0}^nI\left(A_{i-L\Lambda N}^i;B_i|B^{i-1}\right),{\left[0,n\right]}_{A\mathrm{.2},L\Lambda N}^{}={\left[0,n\right]}_{A\mathrm{.2}}^{}{|}_{L=L\Lambda N},L\Lambda N\max \left\{L,N\right\}$

B.6.2.1. Example Non-Linear and Linear Channel Models

Similar to the models discussed in section B.6.1.1, channel distribution of example class A.2 (as defined in section B.1. entitled “Characterizing Channels”) may include one or both of distributions defined on finite and countable alphabet space and distributions defined on continuous alphabet spaces, which distributions may be induced by the models described below.

Let

${}_i\stackrel{\Delta }{=}{ℝ}^q,{}_i\stackrel{\Delta }{=}{ℝ}^p,{}_i\stackrel{\Delta }{=}{ℝ}^r,h_i\text{: }{}^{i-1}\times {}_{i-L}^i\times {}_i\mapsto {}_i,i=0,1,\dots ,n$

where {h_i(•,•,•): i=0, 1, . . . } are measurable functions, and {V_i: i=0, 1, . . . } is a random channel noise process with joint distribution:

P _{V n} (dvⁿ) on (ⁿ).

Recursive expressions may define a nonlinear channel model for the example channel class A.2 as follows:

B _i =h _i(Bⁱ⁻¹,A_i−Lⁱ,V_i), i=1, . . . ,n

B ₀ =h ₀(Bⁱ⁻¹,A⁰_−L,V₀),

where transmission in the example model begins at time i=0, and the initial data of a pre-determined convention. If the channel noise process {V_i: i=0, 1, . . . } is an independent sequence (e.g., non-necessarily stationary), then the above nonlinear channel model may be applied in step one of the two-step process with the induced channel probability distribution:

$\mathbb{P}\left\{B_i\in \Gamma |B^{i-1}=b^{i-1},A^i=a^i\right\}=\mathbb{P}\left\{V_i:h_i\left(b^{i-1},a_{i-L}^i,V_i\right)\in \Gamma \right\},\Gamma \in \left({}_i\right)=Q_i\left(\Gamma |b^{i-1},a_{i-L}^i\right),i=0,1,\dots ,n.$

B.6.2.2. Example MIMO AGN Channel with Memory

For a linear version of the example channel class A.2, the channel output may be modeled as:

$B_i=-\sum _{j=0}^{i-1}C_{i,j}B_j+D_{i,i}A_i+D_{i,i-1}A_{i-1}+V_i,i=0,\dots ,n$

The channel noise process, in this example case, is taken to be Gaussian distributed, i.e.,

P _{V n} (dvⁿ)=Π_i=0ⁿP_Vi(dv_i),V_i˜N(0,K_Vi), i=0,1, . . . ,n,

and the average transmission cost is taken to be:

$\frac{1}{n+1}E\left\{{A_i}_{{ℝ}^q}^2\right\}\le \kappa$

This is a generalization of the example channel model discussed in section B.6.1.2.2 entitled “Example AGN Channel with Memory,” and, thus, analogous results (e.g., finite block length capacities) are generated by utilizing similar procedures.

Specifically, the channel input distribution, in this example case, may be given by:

{B_i≦b_i|Bⁱ⁻¹=bⁱ⁻¹,Aⁱ=aⁱ}={V_i≦b_i+Σ_i=0ⁱ⁻¹C_i,jb_j−D_i,ja_i−D_i,1−1a_i−1}, i=0,1, . . . ,n.

The characterization of the finite block length feedback capacity, in this example, (i.e., a formula for the finite block length feedback capacity) is given by:

$C_{A^n->B^n}^{\mathrm{FB},\mathrm{AWGNM}-A\mathrm{.2}\mathrm{.1}}\left(\kappa \right)=\underset{\left\{{\pi }_i\left({\mathrm{da}}_i|a_{i-1},b^{i-1}\right):i=0,\dots ,n\right\}\in }{\sup }\left\{\sum _{i=0}^nH\left(B_i|B^{i-1}\right)\right\}-H\left(V^n\right)$ $\mathrm{with},\mathbb{P}\left\{B_i\le b_i|B^{i-1}=b^{i-1}\right\}={\int }_{{}_i}\mathbb{P}\left\{V_i\le b_i+\sum _{j=0}^{i-1}C_{i,j}b_j-D_{i,i}a_i-D_{i,i-1}a_{i-1}\right\}{\pi }_i\left(a_i|a_{i-1},b^{i-1}\right)P_{A_{i-1}|B^{i-1}}\left(a_{i-1}|b^{i-1}\right),i=0,1,\dots ,n.$

The optimal (e.g., capacity achieving) channel input distribution

{π_i(da_i|a_i−1,bⁱ⁻¹)≡P_Ai|Bi−1^g(a_i|bⁱ⁻¹):i=0,1, . . . ,n}

is conditional Gaussian, in this example, and the average transmission cost is satisfied. Thus,

{P_Bi|Bi−1(b_i|bⁱ⁻¹)≡P_Bi|Bi−1^g(b_i|bⁱ⁻¹):i=0,1, . . . ,n}

is also conditionally Gaussian.

The information structure of the channel input distribution

{π_i(da_i|a_i−1,bⁱ⁻¹)≡P_{Ai|Ai−1,Bi−1}^g(a_i|a_i−1,bⁱ⁻¹):i=0,1, . . . }

implies the following parametrization of the channel and channel input distribution:

$A_i^g=e_i\left(A_{i-1}^g,B^{g,i-1},U_i\right)=\sum _{j=0}^{i-1}{\Gamma }_{i,j}B_j^g+{\Lambda }_{i,i-1}A_{i-1}^g+U_i,i=0,1,\dots ,n,B_i^g=\sum _{j=0}^{i-1}\left(D_{i,i}{\Gamma }_{i,j}-C_{i,j}\right)B_j^g+\left(D_{i,i}{\Lambda }_{i,i-1}+D_{i,i-1}\right)A_{i-1}^g+D_{i,i}U_i+V_i,i=1,\dots ,n.$

Thus, the finite block length feedback capacity may be characterized by using any state space representation of the channel output process. Note, if stationarity is assumed, the above equations are further simplified.

If a process {X_i: i=0, 1, . . . , n} of a source, such as the source 102, intended for transmission over this channel is R^p-valued, Gaussian distributed, and Markov, i.e.,

P _{X i |X i−1} (dx_i|xⁱ⁻¹)=P_Xi|Xi−1(dx_i|x_i−1), i=0,1, . . . ,n},

and the matrices which maximize the parametrization of the channel input distribution are denoted by:

{Γ_i,j*,λ_i,i−1*,K_Ui*:i=0,1, . . . ,n, j=0,1, . . . ,n−1},

then the coding scheme which achieves the finite block length feedback capacity, in this case, is:

$A_i^{g,*}=\sum _{j=0}^{i-1}{\Gamma }_{i,j}^{*}B_j^g+{\Lambda }_{i,i-1}^{*}A_{i-1}^{g,*}+{\Delta }^{*}\left(i\right)\{X_i-E\left\{X_i|A^{g,i-1},B^{g,i-1}\right\}\}$ $i=0,1,\dots ,n,{\Delta }^{*}\left(i\right)=K_{U_i}^{*,\frac{1}{2}}{\left\{\mathrm{Cov}\left(X_i-E\left\{X_i|A^{g,i-1},B^{g,i-1}\right\}\right)\right\}}^{-\frac{1}{2}},i=0,1,\dots ,n.$

Although the above example illustrates a characterization of the finite block length capacity, channel input distributions, and capacity achieving coding schemes for a certain example channel of class A.2, the above discussed procedure may be extended to any continuous alphabet channel of class A.2, which channel is not necessarily driven by Gaussian noise processes.

B.7. Example Characterizations of Capacities and Identification of Channel Input Distributions for Channels in Class B

By way of example, example finite block length feedback capacity formulas and input distributions, for other example classes of channels, determined according the two-step procedure (described with reference to FIGS. 2, 3, 4, and 5) are presented below. The corresponding feedback capacities with and without transmission cost are limiting versions of the finite block length feedback capacities presented below.

B.7.1. Class B.1

For an example channel condition distribution,

{P_Bi|Bi−1,Ai(db_i|b_i−1,a_i):i=0,1, . . . ,n}

referred to as “Unit Memory Channel Output,” the optimal channel input distribution C_An→Bn^FB is included in the subset:

_[0,n] ^B.1 {P _{A i |B i−1} (da_i|b_i−1):i=0,1, . . . ,n}⊂_[0,n]^A.1

This subset implies that the corresponding joint process {(A_i,B_i): i=0, . . . , n} and channel output process {B_i: i=0, . . . ,n} are first-order Markov, i.e.,

P _{A i ,B i |A i−1 ,B i−1} (da_i,db_i|bⁱ⁻¹,aⁱ⁻¹)=P_{Ai,Bi|Ai−1,Bi−1}(da_i,db_i|a_i−1,b_i−1), i=0, . . . ,n,

P _{B i |B i−1} (db_i|b_i−1)=P_Bi|Bi−1(db_i|b_i−1), i=0,1, . . . ,n.

These findings are applicable to any channel input and output alphabets as those described earlier, including countable and continuous alphabet spaces.

The characterization of the finite block length feedback capacity is:

$C_{A^n->B^{n}}^{\mathrm{FB},B\mathrm{.1}}\sum _{i=0}^nI\left(A_i;B_i|B_{i-1}\right).$

This characterization, or capacity formula, is generated by: (i) applying step one of the two-step procedure to determine a candidate set of optimal channel input distributions _[0,n]^A.1 (e.g., because the channel is a special case of channel distributions of class A.1); and (ii) applying step two of the two-step procedure to determine that the optimal channel input distribution is included in the narrower (e.g., including fewer elements) set _[0,n]^B.1.

If a transmission cost is imposed corresponding to

γ_i^B.1(a_i,b_i−1)

then the example characterization of the finite block length feedback capacity with transmission cost is:

$C_{A^n->B^{n}}^{\mathrm{FB},B\mathrm{.1}}\left(\kappa \right)=\sum _{i=0}^nI\left(A_i;B_i|B_{i-1}\right).$

B.7.1.1. Example Non-Linear and Linear Channel Models

Similar to the models discussed in sections B.6.1.1 and B.6.2.1, channel distributions of example class B.1 may include one or both of distributions defined on finite and countable alphabet space and distributions defined on continuous alphabet spaces, which distributions may be induced by the models described below.

A nonlinear model of a channel in the class B.1 with continuous alphabet spaces may include a recursive expression:

$B_i=h_i\left(B_{i-1},A_i,V_i\right),i=0,\dots ,n,\frac{1}{n+1}\sum _{i=0}^nE\left\{{\gamma }^{B\mathrm{.1}}\left(A_i,B_{i-1}\right)\right\}\le \kappa$

The characterization of the finite block length feedback capacity of the channel defined by this model is given by:

$C_{A^n->B^n}^{\mathrm{FB},\mathrm{NCM}-B\mathrm{.1}}\left(\kappa \right)\underset{\left\{P_{A_i|B_{i-1}}\left({\mathrm{da}}_i|b_{i-1}\right):i=0,\dots ,n\right\}\in }{\sup }\sum _{i=0}^nE\left\{\log \left(\frac{P_{B_i|B_{i-1},A_i}\left(B_i|B_{i-1},A_i\right)}{P_{B_i|B_{i-1}}\left(B_i|B_{i-1}\right)}\right)\right\}$ $\mathrm{where}$ $P_{B_i|B_{i-1}}\left(b_i|b_{i-1}\right)={\int }_{{}_i}P_{B_i|B_{i-1},A_i}\left(b_i|b_{i-1},a_i\right){\mathrm{P}}_{A_i|B_{i-1}}\left(a_i|b_{i-1}\right),i=0,1,\dots ,n$ $\mathrm{and}$ ${\left[0,n\right]}_{B\mathrm{.1}}^{}\left(\kappa \right)\left\{P_{A_i|B_{i-1}}\left(a_i|b_{i-1}\right),i=0,\dots ,n:\frac{1}{n+1}\sum _{i=0}^nE\left\{{\gamma }_i^{B\mathrm{.1}}\left(A_i,B_{i-1}\right)\right\}\le \kappa \right\}$

A computing device and/or operator of a communication system may perform the optimization or maximization of C_An→Bn^FB,NCM−B.1() using dynamic programming To illustrate this point, let

C _t:_t−1

denote the “cost-to-go” (corresponding to C_An→Bn^FB,NCM−B.1()) from the time “t” to the terminal time “n” given the value of the output B_t−1=b_t−1. This cost-to-go satisfies the following dynamic programming recursions:

$C_t\left(b_{t-1}\right)=\underset{s\le 0}{\inf }\underset{P_{A_t|B_t}\left({\mathrm{da}}_t|b_{t-1}\right)}{\sup }\left\{\log \left(\frac{P_{B_t|B_{t-1},A_t}\left(b_t|b_{t-1},a_t\right)}{P_{B_t|B_{t-1}}\left(b_t|b_{t-1}\right)}\right)P_{B_t|B_{t-1},A_t}\left(b_t|b_{t-1},a_t\right)\otimes P_{A_t|B_{t-1}}\left(a_t|b_{t-1}\right)+s\left[{\int }_{{}_t}{\gamma }_i^{B\mathrm{.1}}\left(a_t,b_{t-1}\right)P_{A_t|B_{t-1}}\left(a_t|b_{t-1}\right)-\left(n+1\right)\kappa \right]+C_{t-1}\left(b_t\right)P_{B_t|B_{t-1},A_t}\left(b_t|b_{t-1},a_t\right){\mathrm{P}}_{A_t|B_{t-1}}\left(a_t|b_{t-1}\right)\right\},C_n\left(b_{n-1}\right)=\underset{s\le 0}{\inf }\underset{P_{A_n|B_{n-1}}\left({\mathrm{da}}_n|b_{n-1}\right)}{\sup }\left\{\log \left(\frac{P_{B_n|B_{n-1},A_n}\left(b_n|b_{n-1},a_n\right)}{P_{B_n|B_{n-1}}\left(b_n|b_{n-1}\right)}\right)P_{B_n|B_{n-1},A_n}\left(b_n|b_{n-1},a_n\right){\mathrm{P}}_{A_n|B_{n-1}}\left(a_n|b_{n-1}\right)+s\left[{\int }_{{}_n}{\gamma }_n^{B\mathrm{.1}}\left(a_n|b_{n-1}\right)P_{A_n|B_{n-1}}\left(a_n|b_{n-1}\right)-\left(n+1\right)\kappa \right]\right\}.$

The characterization of the finite block length feedback capacity (or the formula for the finite block length feedback capacity) is then expressible as:

C _{A n →B n} ^FB,NCM−B.1()=C₀(b₋₁)P_B−1(db₋₁).

Note, although not discussed in detail here, the above dynamic programmic recursions also apply to channels defined on finite alphabet spaces.

In some implementations, once the optimal channel input distribution is found and the finite block length feedback capacity is found, a computing device may utilize the Blahut-Arimoto algorithm to compute the maximization of the dynamic programming, working backward in time (i.e., sequentially). This utilization may reducer the computational complexity in solving the finite block length capacity, the capacity and the corresponding capacity achieving channel input distribution.

To develop another characterization of the finite block length feedback capacity consider the information structure of the channel input distribution:

{P_Ai|Bi−1(a_i|b_i−1):i=0,1, . . . }

This information structure implies that there exists a measurable function:

$e_i\text{:}{}_{i-1}\times {}_i\to {}_i,{}_i\stackrel{\Delta }{=}{ℝ}^p,a_i=e_i\left(b_{i-1,}u_i\right),i=0,1,\dots ,n,$

where {U_i:i=0, 1, . . . , n} is an r-dimensional random process with distribution:

{P_Ui(du_i):i=0,1, . . . ,n}

such that

{U_i:e_i(b_i−1,U_i)εda_i}=P_Ai|Bi−1(da_i|b_i−1), i=0,1, . . . ,n.

Because the channel output is defined by the model, B

$A_i=e_i\left(B_{i-1},U_i\right),i=0,1,\dots ,n,B_i=h_i\left(B_{i-1},e_i\left(B_{i-1},U_i\right),V_i\right),i=0,1,\dots ,n,\frac{1}{n+1}\sum _{i=0}^nE{e_i\left(B_{i-1},U_i\right)}_{{ℝ}^q}^2\le \kappa .$

Further, a class of admissible functions is:

_[0,n] ^{NCM−B.1−IL}(){e_i(b_i−1,u_i), i=0, . . . ,n: for a fixed b_i−1 the function e_i(b_i−1,•)

is one-to-one and onto A_i for i=0, . . . , n,

$\frac{1}{n+1}\sum _{i=0}^nE{e_i\left(B_{i-1},U_i\right)}_{{ℝ}^q}^2\le \kappa \}.$

The alternative example characterization of the finite block length feedback capacity is:

$C_{A^n->B^n}^{\mathrm{FB},\mathrm{NCM}-B\mathrm{.1}}\left(\kappa \right)=\underset{{\left\{P_{U_i}\right\}}_{i=0}^n,{\left\{e_i\left(·,·\right)\right\}}_{i=0}^n\in {\left[0,n\right]}_{\mathrm{NCM}-B\mathrm{.1}-\mathrm{IL}}^{}}{\max }\sum _{i=0}^nE\left\{\log \left(\frac{P_{B_i|B_{i-1},A_i}\left(B_i|B_{i-1},e_i\left(B_{i-1},U_i\right)\right)}{P_{B_i|B_{i-1}}\left(B_i|B_{i-1}\right)}\right)\right\}\equiv \underset{{\left\{P_{U_i}\right\}}_{i=0}^n,{\left\{e_i\left(·,·\right)\right\}}_{i=0}^n\in {\left[0,n\right]}_{\mathrm{NCM}-B\mathrm{.1}-\mathrm{IL}}^{}}{\max }\sum _{i=0}^nI\left(U_i;B_i|B^{i-1}\right)$ $\mathrm{with}$ $P_{B_i|B_{i-1}}\left(b_i|b_{i-1}\right)={\int }_{{}_i}P_{B_i|B_{i-1},A_i}\left(B_i|B_{i-1},e_i\left(b_{i-1},u_i\right)\right)\otimes P_{U_i|B_{i-1}}\left(u_i|b_{i-1}\right),i=0,1,\dots ,n.$

A computing device or operator of a communication system may solve this example maximization of C_An;Bn^FB,NCM−B.1 () via dynamic programming or the stochastic calculus of variations, for example. Because the optimal channel input distribution is generated by the above described procedure, computations using the dynamic programming equation, or the Blahut-Arimoto Algorithm applied sequentially in time to the dynamic programming, are simplified. Moreover, for stationary versions (e.g., not varying in time), the algorithms further simplify.

A linear model of a channel in the class B.1 may be expressed via a recursive expression:

$B_i=-C_{i,i-1}B_{i-1}+D_{i,i}A_i+V_i,i=0,1,\dots ,n,\frac{1}{n+1}\sum _{i=0}^nE\left\{{\gamma }^{B\mathrm{.1}}\left(A_i,B_{i-1}\right)\right\}\le \kappa$

where {V_i:i=0, 1, . . . } is independently distributed according to

P_Vn(dvⁿ)=Π_i=0ⁿP_Vi(dv_i),

with zero mean and covariance matrix:

μ_ViE(V_i)=0.K_ViE(V_iV_i^T), i=0,1, . . . ,n

For each i=0, 1, . . . , n, the coefficients {C_i,j,D_i,i: i=0, . . . , n, j=0, 1, . . . , i−1} are real-valued matrices with dimensions p by p and p by q, respectively (e.g., {p, q} being positive integers). The channel distribution is given by:

$\mathbb{P}\left\{B_i\le b_i|B^{i-1}=b^{i-1},A^i=a^i\right\}=\mathbb{P}\left\{V_i\le b_i+C_{i,i-1}b_i-D_{i,i}a_i\right\},i=0,1,\dots ,n.I\left(A^n->B^n\right)=\sum _{i=0}^n\left\{H\left(B_i|B^{i-1}\right)-H\left(B_i|B^{i-1},A_i\right)\right\}=\sum _{i=0}^nH\left(B_i|B_{i-1}\right)-H\left(V^n\right)$

and an example characterization of the finite block length feedback capacity is given by:

$C_{A^n->B^n}^{\mathrm{FB},\mathrm{LCM}-B\mathrm{.1}}\left(\kappa \right)=\underset{\left\{P_{A_i|B_{i-1}}\left({\mathrm{da}}_i|b_{i-1}\right),i=0\dots n:\right\}\in {\left[0,n\right]}_{\mathrm{LCM}-B\mathrm{.1}}^{}\left(\kappa \right)}{\sup }\left\{\sum _{i=0}^nH\left(B_i|B_{i-1}\right)\right\}-H\left(V^n\right)$ $\mathrm{where}$ ${\left[0,n\right]}_{\mathrm{LCM}-B\mathrm{.1}}^{}\left(\kappa \right)\left\{P_{A_i|B_{i-1}}\left(a_i|b_{i-1}\right),i=0,\dots ,n:\frac{1}{n+1}\sum _{i=0}^nE\left\{{\gamma }_i^{B\mathrm{.1}}\left(A_i,B_{i-1}\right)\right\}\le \kappa \right\}$ $\mathrm{and}$ $\mathbb{P}\left\{B_i\le b_i|B_{i-1}=b^{i-1}\right\}={\int }_{{}_i}\mathbb{P}\left\{V_i\le b_i+C_{i,i-1}b_i-D_{i,i}a_i\right\}P_{A_i|B_{i-1}}\left(a_i|b_{i-1}\right),i=0,1,\dots ,n.$

The “cost-to-go” satisfies the following example dynamic programming recursions:

$C_t\left(b_{t-1}\right)=\underset{s\le 0}{\inf }\underset{P_{A_t|B_{t-1}}\left({\mathrm{da}}_t|b_{t-1}\right)}{\sup }\left\{-{\int }_{{}_t\times {}_t}\log \left(\frac{P_{B_t|B_{t-1}}\left(b_t|b_{t-1}\right)}{b_t}\right)P_{B_t|B_{t-1},A_t}\left(b_t|b_{t-1},a_t\right)\otimes P_{A_t|B_{t-1}}\left(a_t|b^{t-1}\right)+s\left[{\int }_{{}_t}{\gamma }_i^{B\mathrm{.1}}\left(a_t,b_{t-1}\right)P_{A_t|B_{t-1}}\left(a_t|b_{t-1}\right)-\left(n+1\right)\kappa \right]+C_{t+1}\left(b_t\right)P_{B_t|B_{t-1}}\left(b_t|b_{t-1},a_t\right)\otimes P_{A_t|B_{t-1}}\left(a_t|b_{t-1}\right)\right\},C_n\left(b_{n-1}\right)=\underset{s\le 0}{\inf }\underset{P_{A_n|B_{n-1}}\left({\mathrm{da}}_n|b_{n-1}\right)}{\sup }\left\{-{\int }_{{}_n\times {}_n}\log \left(\frac{P_{B_n|B_{n-1}}\left(b_n|b_{n-1}\right)}{b_n}\right)P_{B_n|B_{n-1},A_n}\left(b_n|b_{n-1},a_n\right)P_{A_n|B_{n-1}}\left(a_n|b^{n-1}\right)+s\left[{\int }_{{}_n}{\gamma }_n^{B\mathrm{.1}}\left(a_n,b_{n-1}\right)P_{A_n|B_{n-1}}\left(a_n|b_{n-1}\right)-\left(n+1\right)\kappa \right]\right\}.$

Further, a computing device and/or operator may generate an alternative characterization of the finite block length feedback capacity based on the information structure of the channel input distribution. For example, the channel input distribution:

{P_Ai|Bi−1(a_i|b_i−1):i=0,1, . . . }

implies that there exists a measurable function:

$e_i\text{:}{}_{i-1}\times {}_i\to {}_i,{}_i\stackrel{\Delta }{=}{ℝ}^p,a_i=e_i\left(b_{i-1,}u_i\right),i=0,1,\dots ,n,$

where {U_i:i=0, 1, . . . , n} is a p-dimensional random process with distribution:

{P_Ui(du_i):i=0,1, . . . ,n}

such that

{U_i:e_i(b_i−1,U_i)εda_i}=P_Ai|Bi−1(da_i|b_i−1), i=0,1, . . . ,n.

Based on the existence of this example measurable function:

$A_i=e_i\left(B_{i-1},U_i\right),i=0,1,\dots ,n,B_i=-C_{i,i-1}B_{i-1}+D_{i,i}e_i\left(B_{i-1},U_i\right)+V_i,i=0,1,\dots ,n,\frac{1}{n+1}\sum _{i=0}^nE{e_i\left(B_{i-1},U_i\right)}_{{ℝ}^q}^2\le \kappa .$

and an example set of admissible functions is:

_0,n ^{LCM−B.1−IL}(){e_i(bⁱ⁻¹,u_i), i=0, . . . ,n: for a fixed b_i−1 the function e_i(bⁱ⁻¹,•)

is one-to-one and onto A_i for i=0, . . . , n,

$\frac{1}{n+1}\sum _{i=0}^nE{e_i\left(B_{i-1},U_i\right)}_{{ℝ}^q}^2\le \kappa \}$

The example alternative characterization of the finite block length feedback capacity is

$\begin{array}{c}{C}_{A^n->B^n}^{\mathrm{FB},\mathrm{LCM}-B\mathrm{.1}}\left(\kappa \right)=\underset{{\left\{P_{U_i}\right\}}_{i=0}^n,{\left\{e_i\left(·,·\right)\right\}}_{i=0}^n\in {0,n}_{\mathrm{LCM}-A\mathrm{.1}-\mathrm{IL}}^{}}{\max }\sum _{i=0}^nH^e\left(B_i|B_{i-1}\right)-H\left(V^n\right).\\ \equiv \underset{{\left\{P_{U_i}\right\}}_{i=0}^n,{\left\{e_i\left(·,·\right)\right\}}_{i=0}^n\in {0,n}_{\mathrm{LCM}-A\mathrm{.1}-\mathrm{IL}}^{}}{\max }\sum _{i=0}^nI\left(U_i;B_i|B_{i-1}\right)\end{array}$ $\mathrm{with}$ $\mathbb{P}\left\{B_i\le b_i|B_{i-1}=b_{i-1}\right\}={\int }_{{}_i}\mathbb{P}\left\{V_i\le b_i+C_{i,i-1}b_i-D_{i,i}e_i\left(b_{i-1},u_i\right)\right\}P_{U_i|B_{i-1}}\left(u_i|b_{i-1}\right),i=0,1,\dots ,n.$

B.7.1.2. Example MIMO AGN Channel with Memory

The following illustrates characterizations of capacities and channel input distributions for a special case of a channel in class B.1 described by a linear channel model. In the special case, a channel noise process is

V _i ˜N(0,K_Vi), i=0,1, . . . ,n,

or approximately Gaussian.

By the entropy maximizing property of Gaussian distributions, the finite block length feedback capacity is bounded from above by the inequality

H(Bⁿ)≦H(B^g,n),

where

B ^g,n {B _i ^g :i=0,1, . . . ,n}

is Gaussian distributed. This upper bound may be achieved when

{P_Ai|Bi−1(da_i|b_i−1)≡P_Ai|Bi−1^g(a_i|b_i−1):i=0,1, . . . ,n}

is conditional Gaussian and the average transmission cost is satisfied, implying that

{P_Bi|Bi−1(b_i|b_i−1)≡P_Bi|Bi−1^g(b_i|b_i−1):i=0,1, . . . ,n}

is also conditionally Gaussian.

Similar to the other procedures described above with reference to linear channel models, a measurable function

$e_i\text{: }{}_{i-1}\times {}_i\to {}_i,{}_i\stackrel{\Delta }{=}{ℝ}^p,a_i=e_i\left(b_{i-1},u_i\right),i=0,1,\dots ,n$

exists such that

{U_i:e_i(b_i−1,U_i)εda_i}=P_Ai|Bi−1(da_i|b_i−1), i=0,1, . . . ,n.

Based on the existence of this measurable function the channel is given by,

$B_i^g=-C_{i,i-1}B_{i-1}^g+D_{i,i}e_i\left(B_{i-1}^g,U_i\right)+V_i,i=1,\dots ,n,\frac{1}{n+1}\sum _{i=0}^n\mathrm{tr}\left(\mathrm{Cov}\left(A_i\right)\right)\le \kappa .$

Because the channel output process is Gaussian distributed and a linear combination of any sequence of random variables is Gaussian distributed if and only if the sequence of random variables is also jointly Gaussian distributed, then the functions

{e_i(•,•):i=0,1, . . . ,n}

are necessarily linear and {U_i:i=0, 1, . . . , n} is necessarily a Gaussian sequence, in this example case. These properties imply that the corresponding channel input process, denoted by

A ^g,n {A _i ^g :i=0,1, . . . ,n}

is Gaussian distributed, satisfying the average tranmission cost constraint. Moreover, U_i is independent of B^g,i−1 for any i=0, 1, . . . , n. Thus,

A _i ^g =e _i(B_i−1^g,U_i)=Γ_i,i−1B_i−1^g+U_i, i=0,1, . . . ,n.

B _i ^g=(D_i,iΓ_i,i−1−C_i,i−1)B_i−1^g+D_i,iU_i+V_i, i=0, . . . ,n.

Also, because the output process is conditionally Gaussian, in this example case, the conditional entropies

H(B_i^g|B_i−1^g=bⁱ⁻¹)

are independent of b_i−1 and

$\sum _{i=0}^nH\left(B_i^g|B_{i-1}^g\right)=H\left(V^n\right)=\frac{p}{2}\sum _{i=0}^n\log \frac{D_{i,i}K_{U_i}D_{i,i}^T+K_{V_i}}{K_{V_i}}.$

Then, defining

Γ _i,i−1 =D _i,iΓ_i,i−1−C_i,i−1, i=0,1, . . . ,n, Γ_0,−1=0.

A computing device or operator of a communication system may express:

A _i ^g=Γ_i,i−1B_i−1^g+U_i, i=0,1, . . . ,n,

B _i ^g= Γ_i,i−1B_i−1^g+D_i,iU_i+V_i, i=1,2, . . . ,n, B₀^g=D_0,0U₀+V₀,

K _{B i−1 g} E{B _i−1 ^g(B_i−1^g)^T}, i=0,1, . . . ,n.

with the following recursion:

K _{B i g} = Γ_i,i−1K_Bi−1gΓ_i,i−1^T+D_i,iK_UiD_i,i^T+K_Vi, K_B0g=D_0,0K_U0D_0,0^T+V₀, i=1, . . . ,n

In this example case, the average transmission cost is:

$\sum _{i=0}^nE{A_i^g}_{{ℝ}^k}^2=\sum _{i=0}^n\mathrm{tr}\left({\Gamma }_{i,i-1}K_{B_{i-1}^g}{\Gamma }_{i,i-1}^T+K_{U_i}\right).$

and the example finite block length feedback capacity is characterized by

$C_{A^n\to B^n}^{\mathrm{FB},\mathrm{AWGNW}-B\mathrm{.1}}\left(\kappa \right)=\underset{\left\{{\left\{{\Gamma }_{i,i-1}K_{U_i}\right)}_{i=0,j=0^{*}}^{n,n-1}·\frac{1}{\left\{n-1\right\}}\sum _{i=0}^n\mathrm{tr}\left({\Gamma }_{i,i-1}K_{B_{i-1}^g}{\Gamma }_{t,i-1}^{\Gamma }+K_{U_i}\right)\le k\right\}\hspace{1em}}{\max }\hspace{1em}\hspace{1em}\hspace{1em}\frac{P}{2}\sum _{i=0}^n\log \frac{D_{i,i}K_{U_i}D_{i,i}^T+K_{V_i}}{K_{V_i}}.$

If a process {X_i:i=0, 1, . . . , n} of a source, such as the source 102, intended for transmission over this channel is R^p-valued, Gaussian distributed, and Markov and the matrices which maximize the finite block length feedback capacity are

{Γ_i,i−1*,K_Ui*:i=0,1, . . . ,n},

then the coding scheme which achieves the finite block length feedback capacity is:

$A_i^{g.*}={\Gamma }_{i.i-1}^{*}B_{i-1}^g+{\Delta }_{i,i-1}^{*}\left\{X_i-E\left\{X_iB^{g,i-1}\right\}\right\}i=0,1,\dots ,n,{\Delta }_{i.i-1}^{*}=K_{U_i}^{*,\frac{1}{2}}{\left\{\mathrm{Cov}\left(X_i-E\left\{X_iB^{g,i-1}\right\}\right)\right\}}^{-\frac{1}{2}},i=0,1,\dots ,n.$

B.7.1. Class B.2

For an example channel condition distribution,

{P_{Bi|Bi−Mi−1,Ai}(db_ib_i−Mⁱ⁻¹,a_i):i=0,1, . . . ,n},

where M is a finite nonnegative integer, the optimal channel input distribution for C_An→Bn^FB is included in the set

${\left[0,n\right]}_{B\mathrm{.2}}^{}\stackrel{\Delta }{=}\left\{P_{A_i|B_{i-M}^{i-1}}\left({\mathrm{da}}_i|b_{i-M}^{i-1}\right)\text{:}i=0,1,\dots ,n\right\}⋐{\left[0,n\right]}_{A\mathrm{.1}}^{}$

This fact implies that the corresponding joint process {(A_i,B₁): i=0, . . . , n} and channel output process {B_i:i=0, . . . , n} are M-order Markov processes.

An example characterization of the finite block length feedback capacity is

$\begin{array}{c}{C}_{A^n\to B^n}^{\mathrm{FB},B\mathrm{.2}}\stackrel{\Delta }{=}\sum _{i=0}^n\int \log \left(\frac{{\mathrm{dP}}_{B_i|B_{i-M}^{i-1},A_i}\left(·|b_{i-M}^{i-1},a_i\right)}{{\mathrm{dP}}_{B_i|B_{i-M}^{i-1}}\left(·|b_{i-M}^{i-1}\right)}\left(b_i\right)\right)P_{B_{i-M}^i,A_i}\left(b_{i-M}^i,a_i\right)\\ =\sum _{i=0}^nI\left(A_i;B_i|B_{i-M}^{i-1}\right)\end{array}$ $P_{B_i|B_{i-M}^{i-1}}\left({\mathrm{db}}_i|b_{i-M}^{i-1}\right)=\hspace{1em}\int P_{B_i|B_{i-M}^{i-1},A_i}\left(b_i|b_{i-M}^{i-1},a_i\right)ØP_{A_i|B_{i-M}^{i-1}}\left(a_i|b_{i-M}^{j-1}\right),i=0,1,\dots ,n.$

Also, if a transmission cost is imposed, then the example characterization may be expressed as

$C_{A^n->B^n}^{\mathrm{FB},B\mathrm{.2},M\bigwedge K}\left(\kappa \right)\sum _{i=0}^n{\int }_{}^{}\log \left(\frac{P_{\mathrm{Bi}B_{i-M}^{i-1},A_i}\left(·b_{i-M}^{i-1},a_i\right)}{P_{\mathrm{Bi}B_{i-M\bigwedge K}^{i-1}}\left(·b_{i-M\bigwedge K}^{i-1}\right)}\left(b_i\right)\right)P_{B_{M\bigwedge K}^i,A_i}\left(b_{i-M\bigwedge K}^i,a_i\right)$ $\mathrm{where}$ ${\stackrel{}{\circ }}_{\left[0,n\right]}^{B\mathrm{.2},M\bigwedge K}\left\{P_{A_i}B_{i-M\bigwedge K}^{i-1}\left(a_ib_{i-M\bigwedge K}^{i-1}\right)\text{:}i=0,1,\dots ,n\right\}$ $P_{B_{i-M\bigwedge K}^i,A_i}\left(b_{i-M\bigwedge K}^i,a_i\right)=P_{B_iB_{i-M}^{i-1},A_i}\left(b_ib_{i-M}^{i-1},a_i\right)\otimes P_{A_iB_{i-M\bigwedge K}^{i-1}}\left(a_ib_{i-M\bigwedge K}^{i-1}\right)\otimes P_{B_{i-M\bigwedge K}^{i-1}}\left(b_{i-M\bigwedge K}^{i-1}\right),i=0,1,\dots ,n,P_{B_iB_{i-M\bigwedge K}^{i-1}}\left(b_ib_{i-M\bigwedge K}^{i-1}\right)=\int P_{B_iB_{i-M}^{i-1},A_i}\left(b_ib_{i-M}^{i-1},a_i\right)\otimes P_{A_iB_{i-M\bigwedge K}^{i-1}}\left(a_ib_{i-M\bigwedge K}^{i-1}\right),i=0,1,\dots ,n.$

B.7.1. Class B.3

For an example channel condition distribution,

{P_{Bi|Bi−Mi−1,Ai}(db_ib_i−Mⁱ⁻¹,a_i):i=0,1, . . . ,n},

where M is a finite nonnegative integer, the optimal channel input distribution for C_An→Bn^FB is included in the set

${\stackrel{°}{}}_{\left[0,n\right]}^{B\mathrm{.3}}\left\{P_{A_i|A^{i-1},B^{i-1}}\left({\mathrm{da}}_i|a^{i-1},b^{i-1}\right)\text{:}i=0,1,\dots ,n\right\}⋐{}_{\left[0,n\right]}$

An example characterization of the finite block length feedback capacity is

$C_{A^n\to B^n}^{\mathrm{FB},B\mathrm{.3}}\sum _{i=0}^n\int \log \left(\frac{{\mathrm{dP}}_{B_i|B_{i-M}^{i-1},A^i}\left(·|b_{i-M}^{i-1},a^i\right)}{{\mathrm{dP}}_{B_i|B^{i-1}\left(·|b^{i-1}\right)}}\left(b_i\right)\right)P_{B^i,A^i}\left({\mathrm{db}}_i,{\mathrm{da}}^i\right).$

If a transmission cost is imposed corresponding to any instantaneous transmission cost function of classes A, B, and C, then the example characterization of the finite block length feedback capacity is given by the above expression for _[0,n]^B.3 using

_[0,n] ^B.3∩_[0,n]().

B.8. Example Characterizations of Capacities and Identification of Channel Input Distributions for Channels in Class C

By way of example, finite block length feedback capacity formulas and input distributions, for still further classes of channels, determined according the two-step procedure (described with reference to FIGS. 2, 3, 4, and 5) are presented below. The corresponding feedback capacities with and without transmission cost are limiting versions of the finite block length feedback capacities presented below.

B.8.1. Class C.1

For an example channel with a channel conditional distribution:

{P_{Bi|Bi−2i−1,Ai−1i}(db_ib_i−2ⁱ⁻¹,a_i−1ⁱ):i=0,1, . . . ,n},

the optimal channel input distribution for C_An→Bn^FB is included in the set

${\stackrel{°}{}}_{\left[0,n\right]}^{C\mathrm{.1}}\left\{P_{A_i|A_{i-1},B_{i-2}^{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-2}^{i-1}\right)\text{:}i=0,1,\dots ,n\right\}$

This inclusion implies that the corresponding joint process {(A_i,B_i):i=0, . . . , n} and channel output process {B_i:i=0, . . . , n} are second-order Markov processes, i.e.,

$\begin{array}{c}{P}_{A_i,B_i|A^{i-1},B^{i-1}}\left({\mathrm{da}}_i,{\mathrm{db}}_i|b^{i-1},a^{i-1}\right)=P_{A_i,B_i|A_{i-2}^{i-1},B_{i-2}^{i-1}}\left({\mathrm{da}}_i,{\mathrm{db}}_i|a_{i-2}^{i-1},b_{i-2}^{i-1}\right),i=0,\dots ,n,\\ P_{B_i|B^{i-1}}\left({\mathrm{db}}_i|b^{i-1}\right)=P_{B_i|B_{i-2}^{i-1}}\left({\mathrm{db}}_i|b_{i-2}^{i-1}\right),i=0,1,\dots ,n.\end{array}$

An example characterization of the finite block length feedback capacity is

$C_{A^n\to B^n}^{\mathrm{FB},C\mathrm{.1}}\frac{1}{n+1}\sum _{i=0}^nI\left(A_{i-1}^i;B_i|B_{i-2}^{i-1}\right).$

If a tranmission cost is imposed, this example characterization may be expressed as:

$C_{A^n\to B^n}^{\mathrm{FB},C\mathrm{.1}\mathrm{.2}\Delta K}\left(\kappa \right)\stackrel{\Delta }{=}\sum _{i=0}^n\int \log \left(\frac{{\mathrm{dP}}_{B_i|B_{i-2}^{i-1},A_{i-1}^i}\left(·|b_{i-2}^{i-1},a_{i-1}^i\right)}{{\mathrm{dP}}_{B_i|B_{i-2\bigwedge K}^{i-1}\left(·|b_{i-2\bigwedge K}^{i-1}\right)}}\left(b_i\right)\right)P_{B_{2\bigwedge K}^iA_{i-1}^i}\left({\mathrm{db}}_{i-2\bigwedge K}^i,{\mathrm{da}}_{i-1}^i\right)$ $\mathrm{where}$ ${\stackrel{°}{}}_{\left[0,n\right]}^{C\mathrm{.1}\mathrm{.2}\Delta K}\left\{P_{A_i|A_{i-1},B_{i-2\bigwedge K}^{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-2\bigwedge K}^{i-1}\right)\text{:}i=0,1,\dots ,n\right\}$ $P_{B_{i-2\bigwedge K}^i,A_i}\left({\mathrm{db}}_{i-2\bigwedge K}^i,{\mathrm{da}}_i,{\mathrm{da}}_{i-1}\right)=P_{B_i|B_{i-2}^{i-1},A_{i-1}^i}\left({\mathrm{db}}_i|b_{i-2}^{i-1},a_{i-1}^i\right)\otimes P_{A_i|A_{i-1},B_{i-2\bigwedge K}^{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-2\bigwedge K}^{i-1}\right)\otimes P_{A_{i-1}|B_{i-2\bigwedge K}^{i-1}}\left(a_{i-1}|b_{i-2\bigwedge K}^{i-1}\right)\otimes P_{B_{i-2\bigwedge K}^{i-1}}\left({\mathrm{db}}_{i-2\bigwedge K}^{i-1}\right),i=0,1,\dots ,n,P_{B_i|B_{i-2\bigwedge K}^{i-1}}\left({\mathrm{db}}_i|b_{i-2\bigwedge K}^{i-1}\right)=\int P_{B_i|B_{i-2}^{i-1},A_{i-1}^i}\left({\mathrm{db}}_i|b_{i-2}^{i-1},a_{i-1}^i\right)\otimes P_{A_i|A_{i-1},B_{i-2\bigwedge K}^{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-2\bigwedge K}^{i-1}\right)\otimes P_{A_{i-1}|B_{i-2\bigwedge K}^{i-1}}\left(a_{i-1}|b_{i-2\bigwedge K}^{i-1}\right),i=0,1,\dots ,n.$

B.8.2. Class C.2

For an example channel condition distribution,

{P_{Bi|Bi−Mi−1,Ai−1i}:i=0,1, . . . ,n},

the optimal channel input distribution for C_An→Bn^FB is included in the set

${\left[0,n\right]}_{C\mathrm{.2}}^{}\left\{P_{A_i|A_{i-L}^{i-1},B_{i-M\bigwedge L}^{i-1}}\left({\mathrm{da}}_i|a_{i-L}^{i-1},b_{i-M\bigwedge L}^{i-1}\right)\text{:}i=0,1,\dots ,n\right\}⋐{\left[0,n\right]}_{A\mathrm{.2}}^{}$

This inclusion implies that the corresponding joint process {(A_i,B_i):i=0, . . . , n} and channel output process {B_i:i=0, . . . , n} are limited-memory Markov processes.

B.8.2.1. Unit Memory Channel Input Output (UMCIO) Example

For a special case of the example channel class C.2, an example channel conditional distribution is

{P_{Bi|Bi−1,Ai−1,Ai}(db_i|b_i−1,a_i,a_i−1):i=0,1, . . . ,n}

and the optimal channel input distribution for C_An→Bn^FB may be included in the set

${\left[0,n\right]}_{\mathrm{UMCIO}}^{}\left\{P_{A_i|A_{i-1},B_{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-1}\right)\text{:}i=0,1,\dots ,n\right\}⋐{\left[0,n\right]}_{C\mathrm{.1}}^{}$

This inclusion implies that the corresponding joint process {(A_i,B_i):i=0, . . . , n} and channel output process {B_i:i=0, . . . , n} are first-order Markov processes.

An example characterization of the finite block length feedback capacity, in this example case is

$C_{A^n\to B^n}^{\mathrm{FB},\mathrm{UMCIO}}\underset{\left\{P_{A_i|A_{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-1}\right)\text{:}i=0,1,\dots ,n\right\}}{\sup }\sum _{i=0}^nI\left(A_{i-1},A_i;B_i|B_{i-1}\right),\mathrm{where}$ $I\left(A_{i-1},A_i;B_i|B_{i-1}\right)=\int \log \left(\frac{{\mathrm{dP}}_{B_i|B_{i-1},A_i,A_{i-1}}\left(·|b_{i-1},a_i,a_{i-1}\right)}{{\mathrm{dP}}_{B_i|B_{i-1}}\left(·|b_{i-1}\right)}\left(b_i\right)\right)P_{B_i|B_{i-},A_i,A_{i-1}}\left({\mathrm{db}}_i|b_{i-1},a_i,a_{i-1}\right)\otimes P_{A_i|A_{i-1},B_{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-1}\right)\otimes P_{A_{i-1},B_{i-1}}\left({\mathrm{da}}_{i-1},{\mathrm{db}}_{i-1}\right),i=0,1,\dots$ $\mathrm{and}$ $P_{B_i|B_{i-1}}\left({\mathrm{db}}_i|b_{i-1}\right)=\int P_{B_i|B_{i-1},A_{i-1},A_i}\left({\mathrm{db}}_i|b_{i-1},a_{i-1},a_i\right)\otimes P_{A_i|A_{i-1},B_{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-1}\right)\otimes P_{A_{i-1}|B_{i-1}}\left(a_{i-1}|b_{i-1}\right),i=0,1,\dots ,n.$

B.8.2.2. Unit Memory Channel Input (UMCI) Example

For another special case of the example channel class C.2, an example channel conditional distribution is

{P_Bi|Ai−1,Ai(db_i|a_i−1,a_i):i=0,1, . . . ,n}

and the optimal channel input distribution for C_An→Bn^FB may be included in the set

_[0,n] ^UMCI {P _{A i |A i−1 ,B i−1} (da_i|a_i−1,b_i−1):i=0,1, . . . ,n}.

This inclusion implies that the corresponding joint process {(A_i,B_i):i=0, . . . , n} and channel output process {B_i:i=0, . . . , n} are first-order Markov processes.

An example characterization of the finite block length feedback capacity, in this example case is

$C_{A^n\to B^n}^{\mathrm{FB},\mathrm{UMCI}}\underset{\left\{P_{A_i|A_{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-1}\right)\text{:}i=0,1,\dots ,n\right\}}{\sup }\frac{1}{n+1}\sum _{i=0}^nI\left(A_{i-1},A_i;B_i|B_{i-1}\right).\mathrm{where}$ $I\left(A_{i-1},A_i;B_i|B_{i-1}\right)=\int \log \left(\frac{{\mathrm{dP}}_{B_i|A_i,A_{i-1}}\left(·|a_i,a_{i-1}\right)}{{\mathrm{dP}}_{B_i|B_{i-1}}\left(·|b_{i-1}\right)}\left(b_{i-1}\right)\right)P_{B_i|A_i,A_{i-1}}\left({\mathrm{db}}_i|a_i,a_{i-1}\right)\otimes P_{B_i|A_i,B_{i-1}}\left({\mathrm{da}}_i|a_i,b_{i-1}\right)\otimes P_{B_i|A_i,B_{i-1}}\left({\mathrm{da}}_i|a_i,{\mathrm{db}}_{i-1}\right),1=0,1,\dots ,n,\mathrm{and}$ $P_{B_i|B_{i-1}}\left({\mathrm{db}}_i|b_{i-1}\right)=\int P_{B_i|A_{i-1},A_i}\left({\mathrm{db}}_i|a_{i-1},a_i\right)\otimes P_{A_i|A_{i-1},B_{i-1}}\left({\mathrm{da}}_i|a_{i-1},b_{i-1}\right)\otimes P_{A_{i-1}|B_{i-1}}\left({\mathrm{da}}_{i-1}|b_{i-1}\right),i=0,1,\dots ,n.$

B.9. Example Characterizations of Capacities and Identification of Channel Input Distributions for MIMO ANonGN Channels with Memory

The two step procedure (described with reference to FIGS. 2, 3, 4, and 5) may also allow computing devices and/or operators of communication systems to characterize generalizations of Additive Gaussian Noise (AGN) channels and nonstationary Multiple-Input Multiple Output (MIMO) Additive Non-Gaussian Noise (ANonGN) channels. These other types of channels may be defined by the example model:

$B_i=\sum _{j=1}^LD_{i,j}A_{i-j}+V_j,i=0,1,\dots ,n,\frac{1}{n+1}E\left\{\sum _{i=0}^n{\gamma }_i\left(A_{i-L}^i,B^{i-1}\right)\right\}\le \kappa$

where {V_i:i=0, 1, . . . , n} is p-dimensional nonstationary non-Gaussian distributed noise, {A_i: i=0, 1, . . . , n} are q-dimensional channel input processes, and a condition “Aⁿ is causally related to Vⁿ” is represented, in the example model, by

$\begin{array}{c}{P}_{A^n,V^n}\left(a^n,v^n\right)={\otimes }_{i=0}^n\left(P_{A_i|A^{i-1},V^{i-1}}\left(a_i|a^{i-1},v^{i-1}\right)\otimes P_{V_i|V^{i-1},A^i}\left(v_i|v^{i-1},a^i\right)\right)\\ ={\otimes }_{i=0}^n\left(P_{A_i|A^{i-1},V^{i-1}}\left(a_i|a^{i-1},v^{i-1}\right)\otimes P_{V_i|V^{i-1},A_{i-L}^i}\left(v_i|v^{i-1},a_{i-L}^i\right)\right).\end{array}$

The channel conditional distribution of the example nonstationary MIMO ANonGN channel is

$\mathbb{P}\left\{B_i\le b_i|B^{i-1},A^i\right\}=\mathbb{P}\left\{V_i\le b_i-\sum _{j=1}^LD_{i,j}A_{i-j}|V^{i-1},A_{i-L}^i\right\}$ $\mathrm{or}$ $\mathbb{P}\left\{B_i\le b_i|B^{i-1}\right\}={\int }_{b_{i-L}^i}\mathbb{P}\left\{V_i\le b_i-\sum _{j=1}^LD_{i,j}a_{i-j}|V^{i-1},a_{i-L}^i\right\}\otimes P_{A_i|A_{i-L}^{i-1},V^{i-1}}\left({\mathrm{da}}_i|a_{i-L}^{i-1},V^{i-1}\right)\otimes P_{A_{i-L}^{i-1}|V^{i-1}}\left({\mathrm{da}}_{i-L}^{i-1}|V^{i-1}\right).$

An example characterization of the finite block length feedback capacity for the MIMO Additive Non-Gaussian Noise channels with memory may be expressed as:

$C_{A^n;B^n}^{\mathrm{FB},\mathrm{ANonGN}}\left(\kappa \right)=\underset{\left\{P_{A_i|A_{i-L}^{i-1},v^{i-1}}\left(a_i|a_{i-L}^{i-1},v^{i-1},\right),i=0,\dots ,n\text{:}E\left\{{\Sigma }_{i=0}^n{\gamma }_i\left(A_{i-L}^iB^{i-1}\right)\right\}\le \kappa \right\}}{\sup }\left\{\sum _{i=0}^nH\left(B_i|B^{i-1}\right)-H\left(V_i|V^{i-1},A_{i-L}^i\right)\right\}$

where the transition probability distribution of the channel output process {B_i:i=0, 1, . . . , n} is given by the above mentioned model.

If the noise process is non-Gaussian with conditional distribution:

{P_Vi|Vi−1,Ai=P_{Vi|Vi−Li−1,Ai−Li}:i=0,1, . . . ,n},

and the instantaneous tranmission cost is:

γ_i(a_i−Lⁱ⁻¹,bⁱ⁻¹)γ_i¹(a_i−Lⁱ,b_i−Lⁱ⁻¹), i=0, . . . ,n.

then another example characterization of the finite block length feedback capacity for a channel in this class of channels is

$C_{W;B^n}^{\mathrm{FB},\mathrm{AnonGN},L}\left(\kappa \right)=\underset{\left\{P_{A_i|A_{i-L}^{i-1},v_{i-L}^{i-1}}\left(a_i|a_{i-L}^{i-1},v_{i-L}^{i-1}\right),i=0,\dots ,n\text{:}E\left\{{\Sigma }_{i=0}^n{\gamma }_i^1\left(A_{i-L}^{i-1},B_{i-L}^{i-1}\right)\right\}\le \kappa \right\}}{\sup }\left\{\sum _{i=0}^nH\left(B_i|B^{i-1}\right)-H\left(V_i|V_{i-L}^{i-1},A_{i-L}^i\right)\right\}$ $\mathrm{where}$ $\mathbb{P}\left\{B_i\le b_i|B^{i-1}\right)={\int }_{{}_i}\mathbb{P}\left\{V_i\le b_i-\sum _{j=1}^LD_{i,j}a_{i-j}|V_{i-L}^{i-1},A_{i-L}^{i-1}\right\}\otimes P_{A_i|A_{i-L}^{i-1},V_{i-L}^{i-1}}\hspace{1em}\left({\mathrm{da}}_i|a_{i-L}^{i-1},V_{i-L}^{i-1}\right)\otimes P_{A_{i-L}^{i-1}|V_{i-L}^{i-1}}\left(a_{i-L}^{i-1}|V_{i-L}^{i-1}\right),i=0,1,\dots ,n.$

In another example, the noise process is Gaussian with conditional distribution:

{P_Vi|Vi−1,Ai=P_{Vi|Vi−1,Ai−Li}:i=0,1, . . . ,n},

and the instantaneous transmission cost function is:

γ_i(a_i−Lⁱ⁻¹,bⁱ⁻¹)γ_i¹(a_i−Lⁱ,b_i−Lⁱ⁻¹), i=0, . . . ,n.

In this case, the optimal channel input distribution is:

P _{A i |A i−L i−1 ,V i−1} *(a_i|a_i−Lⁱ⁻¹,vⁱ⁻¹), i=0, . . . ,n

and it is conditionally Gaussian. A Gaussian process

{A_i^g:i=0,1, . . . ,n}

realizes this distribution, where

$A_i^g=\sum _{j=1}^L{\Gamma }_{i,j}^1A_{i-j}^g+\sum _{j=0}^{i-1}{\Gamma }_{i,j}^2V_j+U_i,i=0,1,\dots ,n.$

That is, at each time i=0, 1, . . . n the Gaussian process is a linear combination of {A_i−L^g,i−1,Vⁱ} and Gaussian random variables.

In yet another example, the noise process is Gaussian, and satisfies

{P_Vi|Vi−1,Ai=P_{Vi|Vi−Li−1,Ai−Li}:i=0,1, . . . ,n},

In this case, the optimal channel input distribution is:

P _{A i |A i−L i−1 ,V i−L i−1} *(a_i|a_i−Lⁱ⁻¹,v_i−Lⁱ⁻¹), i=0, . . . ,n

and it is conditionally Gaussian. A Gaussian process

{A_i^g:i=0,1, . . . ,n}

realizes this distribution, where

$A_i^g=\sum _{j=1}^L{\Gamma }_{i,j}^1A_{i-j}^g+\sum _{j=1}^L{\Gamma }_{i,j}^2V_{i-j}+U_i,i=0,1,\dots ,n$

In still another example, the noise process is scalar Gaussian, Aⁿ is causally related to and defined by:

{P_Vi|Vi−1,Ai=P_Vi|Vi−1:i=0,1, . . . ,n}

and the instantaneous tranmission cost function is)

γ_i(a_i−Lⁱ⁻¹,bⁱ⁻¹)γ(a_i), i=0,1, . . . ,n.

In this case, the Gaussian process

{A_i^g:i=0,1, . . . ,n}

defined by:

$A_i^g=\sum _{j=0}^{i-1}{\Gamma }_{i,j}^2V_j+U_i,i=0,1,\dots ,n.$

is a realization of the optimal channel input distribution. Further, if

{P_Vi|Vi−1=P_{Vi|Vi−Li−1}:i=0,1, . . . ,n}

is stationary, the Gaussian process realization further reduces to:

$A_i^g=\sum _{j=1}^L{\Gamma }_j^2V_{i-j}+U_i,i=0,1,\dots ,n$

B.10. Necessary and Sufficient Conditions for Feedback not to Increase Capacity

The example two-step procedure (described with reference to FIGS. 2, 3, 4, and 5) may allow computing device and/or operators or communication systems to determine necessary and sufficient conditions for feedback encoding to not increase capacity for a channel with memory, such as the channel 106. For such channels, feedback encoding does not increase capacity and the characterization of finite block length capacity with and without feedback is the same. Further, the capacity with and without feedback is the same in these cases. Example characterizations of the finite block length capacity without feedback and the capacity without feedback, developed according to the two-step procedure are described below along with necessary and sufficient conditions for feedback encoding not to increase channel capacity of channels with memory.

If an example channel has memory and an instantaneous transmission cost constraint:

_[0,9]()

then

C _{A n ;B n} ^noFB()≦C_An→Bn^FB(),

and feedback encoding does not provide additional gain compared to encoding without feedback if and only if the following identify holds:

C _{A n →B n} ^FB()=C_An;Bn^noFB()

Further, feedback encoding does not increase capacity without feedback if and only if:

$\underset{n\to \infty }{\mathrm{liminf}}\frac{1}{n+1}C_{A^n\to B^n}^{\mathrm{FB}}\left(\kappa \right)=\underset{n\to \infty }{\mathrm{liminf}}\frac{1}{n+1}C_{A^n;B^n}^{\mathrm{noFB}}\left(\kappa \right)$

where the limits are finite.

Next, further example notation is introduced. Specifically, let

$\begin{array}{c}I\left(A^n;B^n\right)=\sum _{i=0}^nE\left\{\log \left(\frac{{\mathrm{dP}}_{B_i|B^{i-1},A^i}\left(·|B^{i-1},A^i\right)}{{\mathrm{dP}}_{B_i|B^{i-1}}^{\mathrm{npFB}}\left(·|B^{i-1}\right)}\left(B_i\right)\right)\right\}\\ \equiv {}_{A^n;B^n}\left(\left\{P_{A_i|A^{i-1}}^{\mathrm{npFB}},P_{B_i|B^{i-1}.A^i}\text{:}i=0,1,\dots ,n\right\}\right)\end{array}$

where I(Aⁿ;Bⁿ) is a functional of the channel distribution and the channel input distribution without feedback denoted by

{P_Ai|Ai−1^noFB:i=0,1, . . . ,n}ε_[0,n]()

The maximum information structure without feedback, in this example notation, is

{aⁱ⁻¹}, i=0,1, . . . ,n.

Also, let

$\begin{array}{c}I\left(A^n\to B^n\right)=\sum _{i=0}^nE\left\{\log \left(\frac{P_{B_i|B^{i-1},A^i}\left(·|B^{i-1},A^i\right)}{P_{B_i|B^{i-1}}^{\mathrm{FB}}\left(·|B^{i-1}\right)}\left(B_i\right)\right)\right\}\\ \equiv {}_{A^n\to B^n}\left(\left\{P_{A_i|A^{i-1},B^{i-1}}^{\mathrm{FB}},P_{B_i|B^{i-1},A^i}\text{:}i=0,1,\dots ,n\right\}\right)\end{array}$

That is, I(Aⁿ→Bⁿ) is a functional of the channel distribution and the channel input distribution with feedback denoted by

{P_{Ai|Ai−1,Bi−1}^FB:i=0,1, . . . ,n}ε_[0,n]()

The maximum information structure without feedback, in this example notation, is

{aⁱ⁻¹,bⁱ⁻¹}, i=0,1, . . . ,n.

Using this example notation, for a channel with memory and an encoder with a transmission cost constraint, the finite block length capacity without feedback with transmission cost is:

$C_{A^n;B^n}^{\mathrm{npFB}}\left(\kappa \right)=\underset{\left\{P_{A_i;A^{i-1}}^{\mathrm{noFB}}\left({\mathrm{da}}_i|a^{i-1}\right)\text{:}i=0,\dots ,n\right\}\in {\left[0,n\right]}_{}\left(\kappa \right)}{\sup }{}_{A^n\to B^n}\left(\left\{P_{A_i|A^{i-1}{\mathrm{mB}}^{i-1}}^{\mathrm{FB}},P_{B_i|B^{i-1},A^i}\text{:}i=0,1,\dots ,n\right\}\right)$

and, similarly, the finite block length capacity with feedback with transmission cost is:

$C_{A^n\to B^n}^{\mathrm{FB}}\left(\kappa \right)=\underset{\left\{P_{A_i|A^{i-1},B^{i-1}}^{\mathrm{FB}}\left({\mathrm{da}}_i|a^{i-1},b^{i-1}\right)\text{:}i=0,\dots ,n\right\}\in {\left[0,n\right]}_{}\left(\kappa \right)}{\sup }{}_{A^n\to B^n}\left(\left\{P_{A_i|A^{i-1},B^{i-1}}^{\mathrm{FB}},P_{B_i|B^{i-1},A^i}\text{:}i=0,1,\dots ,n\right\}\right)$

Also, define a set satisfying conditional independence as

_[0,n] ^CI {P _{A i |A i−1 ,B i−1} ^FB(da_i|aⁱ⁻¹,bⁱ⁻¹=P_Ai|Ai−1^noFB(da_i|aⁱ⁻¹)−a.a.(aⁱ⁻¹,bⁱ⁻¹):i=0, . . . ,n}

The characterization of finite block length capacity with feedback is equal to the characterization of finite block length capacity without feedback if and only if the corresponding optimal channel input distribution of the former belongs to the above set. Thus, C_An;Bn^noFB() and C_An→Bn^FB () are related by

$C_{A^n;B^n}^{\mathrm{noFB}}\left(\kappa \right)=\underset{\left\{P_{A_i|A^{i-1},B^{i-1}}^{\mathrm{FB}}\left({\mathrm{da}}_i|a^{i-1},b^{i-1}\right)\text{:}i=0,,n\right\}\in {\left[0,n\right]}_{}\left(\kappa \right)\bigcap {\left[0,n\right]}_{\mathrm{CI}}^{}}{\sup }{}_{A^n\to B^n}\left(\left\{P_{A_i|A^{i-1},B^{i-1}}^{\mathrm{FB}},P_{B_i|B^{i-1},A^i}\text{:}i=0,1,\dots ,n\right\}\right)$

For a memoryless channel, this condition holds because the optimal channel input distribution which corresponds to the characterization of finite block length feedback capacity satisfies

P _{A i |A i−L ,B i−1} ^FB(da_i|aⁱ⁻¹,bⁱ⁻¹)=P_Ai(da_i)−a.a.aⁱ⁻¹, i=0, . . . ,n

Also, the optimal channel input distribution which corresponds to the characterization of finite block length capacity without feedback satisfies

P _{A i |A i−1} ^noFB(da_i|aⁱ⁻¹)=P_Ai(da_i)−a.a.aⁱ⁻¹, i=0, . . . ,n

For any example channel in classes A, B, and C and any instantaneous transmission cost function in classes A, B, and C, let

{P(db_i|_i^Q):⊂{aⁱ⁻¹,bⁱ⁻¹}, i=0, . . . ,n}

denote the channel distribution and let

{P^FB,*(da_i|_i^FB):i=0,1, . . . ,n}ε_[0,n]^FB()⊂_[0,n](), _i^FB⊂{aⁱ⁻¹bⁱ⁻¹}, i=0,1, . . . ,n}

denote the channel input distribution corresponding to the characterization of the finite block length feedback capacity C_An→Bn^FB (). With these example definitions, feedback does not increase the finite block length capacity without feedback (i.e., C_An;Bn^noFB()=C_An;Bn^FB()) if and only if there exists a channel input distribution (without feedback)

{P*^,noFB(da_i|_i^noFB):i=0,1, . . . ,n}, _i^noFB⊂{aⁱ⁻¹}, i=0,1, . . . ,n,

which induces the joint distribution

P _{A n ,B n} ^FB,*(daⁿ,dbⁿ)

and the channel output distribution

P _{B n} ^FB,*(dbⁿ)

corresponding to the pair

{P^FB,*(da_i|_i^FB),P(db_i|_i^Q):i=0, . . . ,n}.

C. Designing Information Lossless Encoders to Achieve Characterized Capacities

In addition to characterizing the “finite block length” feedback capacity for channels, such as the channel 106, the techniques of the present disclosure include methods to design encoders that achieve characterized capacities of channels with memory. These capacity achieving encoders are “information lossless encoders,” in that the mapping, implemented by the encoders, of information from a source to encoded symbols are sequentially invertible. Encoders violating this property are not able to achieve the capacity of a given channel.

Further, for each of the example channels (e.g., A, B, and C) discussed herein, a computing device and/or operator of a communication system may generate specific coding schemes or mappings for the “information lossless” encoder based on the characterization of the capacity for the channel and a corresponding optimal channel input distribution. That is, all capacity achieving encoders may be “information lossless” encoders, but, for a given channel, a computing device or operator may generate a specific “information lossless” coding scheme based on a characterization of capacity for that channel. A computing device and/or operator may also define additional conditions of the information lossless encoder for a specific channel based on a characterization of capacity for the specific channel.

The optimal (i.e., “information lossless”) encoding schemes designed according to the method discussed below may reduce the complexity of communication systems and operate optimally. For example, optimal operation may include an optimal operation in terms of the overall number of processing elements (e.g., CPUs) required to process transmissions and/or the number of memory elements and steps required to encoder and decode messages. Such optimal encoding and decoding schemes may require small processing delays and short code lengths in comparison to encoding and decoding schemes designed based on an assumption of a channel without memory or based on a separate treatment of source codes and channel codes.

Although, encoders and corresponding necessary and sufficient conditions discussed below are described, by way of example, with reference to the example communication system 100, which system 100 is a point-to-point communication system, encoders of the present disclosure may be applied to or implemented in systems other than point-to-point communication systems. For example, encoders of the present disclosure may be implemented in multi-user and network communication systems by repeating the procedures described below (for each user, for each node of a network, etc) Implementations of encoders may even be utilized in joint collaborative communication.

C.1. Encoder Design Overview and Methods

FIG. 6 illustrates an example process 600 for designing an information lossless encoder for a particular channel, such as the channel 106. The information lossless encoder 602 designed according to the process 600 may achieve capacities, such as finite block length feedback capacities, feedback capacities, finite block length feedback capacities with transmission cost, feedback capacities with transmission cost, finite block length capacities without feedback, capacities without feedback, finite block length capacities without feedback and with transmission cost, and capacities without feedback and with transmission cost. These types of capacities are further described in sections B.2. and B.3. entitled “Capacity Without Feedback” and “Capacity With Feedback,” respectively. A computing device, such as the computing device illustrated in detail in FIG. 15, may be specially and/or specifically configured (e.g., by one or more algorithms, routines, modules, or engines) to implement at least a portion of the process 600. Further, components of the example communication system 100, such as the encoder 104, may be configured according to the output of the process 600.

In the process 600, an information lossless condition 604, a characterization of capacity 606 (e.g., finite block length feedback capacity), and an optimal channel input distribution 608 are input into an encoder design procedure 610. The characterization of capacity 606 and the optimal channel input distribution 608 may be generated according to an implementation of the two-step procedure (described with reference to FIGS. 2, 3, 4, and 5). The information lossless condition 604 may be defined based on a desired inevitability of the resulting information lossless encoder 602. Specifically, the information lossless condition 604 may be defined, for a give class of channels, in terms of directed information measures, as discussed in further detail below with reference to example classes of channels.

The encoder design procedure 610 may, based on the information lossless condition 604, the characterization of capacity 606, and the optimal channel input distribution 608, generate the information lossless encoder 602. The information lossless encoder 602 may be utilized in an implementation of the system 100 as encoder 104, for example. Also, although not emphasized in the below description, decoders corresponding to the information lossless encoder 602 may also be generated by the encoder design procedure 610 based on the information lossless condition. In this manner, computing devices and/or operators of communication systems may design encoding and decoding processing for a communication system, such as the system 100.

Generally, an encoding process may encode received symbols xⁿ from a source, such as the source 102, into channel input symbols aⁿ{a₀, a₁, . . . , a_n}, a_jεX_j, where j=0, 1, . . . , n, as further discussed with reference to FIG. 1. FIG. 7 illustrates such a procedure. In source information 702, such as sampled speech signals, digital representations of photographs, etc., may be encoded by an encoder 706 to produce encoded symbols 704. As illustrated by the arrows in FIG. 7, this encoder 706 implements an encoding scheme that is invertible. That is, the encoded symbols 704 may be mapped back to the source information 702, and, thus, no information is lost (i.e., the encoder is “information lossless”).

Although FIG. 7 illustrates certain information, symbols, and mapping by way of example, implementations of information lossless encoders may encode any suitable types and formats of data for transmission over a channel other than those illustrated in FIG. 7. Further, encoding schemes may map any number of received symbols from a source to any number of encoded symbols.

FIG. 8 is a flow diagram of an example method 800 for designing a capacity achieving and information lossless encoder. A computing device, such as the computing device illustrated in detail in FIG. 15, may be specially and/or specifically configured to implement at least a portion of the method 800. Further, in some implementations, a suitable combination of a computing device and an operator of a communication system, such as the communication system 100, may implement the method 800.

In the method 800, an information lossless condition is determined for a class of channels (block 802). For example, a computing device or operator may determine an information lossless condition for one or more of the classes of channel described in section B.1. entitled “Characterizing Channels.” The determination of the information lossless condition may include utilizing a general definition of information lossless encoders along with properties of a channel defined in a channel model to generate one or more specific information lossless conditions for a specific class of channels. Example determinations utilizing this procedure are discussed further below for example channels in classes A, B, and C (as defined in described in section B.1).

The method 800 may also include receiving a characterization of channel capacity and a corresponding channel input distribution (block 804). The characterization of channel capacity and corresponding input distribution may correspond to the class of channels associated with the information lossless conditions determined at block 802. The characterization or formula for the capacity may be a characterization or formula for a finite block length feedback capacity, feedback capacity, finite block length feedback capacity with transmission cost, feedback capacity with transmission cost, finite block length capacity without feedback, capacity without feedback, finite block length capacity without feedback and with transmission cost, and capacity without feedback and with transmission cost. Further, the characterization of channel capacity and corresponding channel input distribution may be an output from the two-step procedure (described with reference to FIGS. 2, 3, 4, and 5).

A computer device and/or operator of a communication system may then utilize the information lossless condition(s), characterization of channel capacity, and channel input distribution to determine an encoding scheme (block 806). That is, the computer device and/or operator may design the encoding scheme based on both properties of the channel (e.g., capacity and optimal channel input distribution) and necessary and sufficient conditions for any encoder to achieve the capacity of a channel with memory. Certain of these optimal or capacity achieving encoding schemes for specific classes of channels are discussed in sections B.6.1.2.2., B.6.2.2., and B.7.1.2.

C.2. Example Information Lossless Conditions

For further clarification and by way of example, the section below include necessary and sufficient conditions for any encoder of example classes A and B to be information lossless. Based on these conditions and based on characterizations of capacities and optimal channel input distributions, computing devices and/or operators may design encoders for transmission of information over channels in the example classes A and B. The example corresponding to class A includes an encoder with feedback and the example corresponding to class B includes an encoder without feedback.

C.2.1. Feedback Encoder Corresponding to Example Class a

A feedback encoder corresponding to the example class A (e.g., class A channels) may be referred to herein as

e ⁿε_[0,n]

The information structure entering the example encoder at any time i may be expressed as {aⁱ⁻¹, xⁱ,bⁱ⁻¹}.

By substituting aⁱ⁻¹ recursively into the right side of

a _i =e _i(aⁱ⁻¹,xⁱ,bⁱ⁻¹)

then

a _i =e _i(aⁱ⁻¹,xⁱ,bⁱ⁻¹)≡ē_i(xⁱ,bⁱ⁻¹), i=0, . . . ,n.

Thus, for any feedback encoder of example class A, the information structure of the encoder at each time instant i is:

_i ^e {a ⁱ⁻¹ ,x ⁱ ,b ⁱ⁻¹ }≡{x ⁱ ,b ⁱ⁻¹ }, i=0, . . . ,n,

and this information structure is the most general classical information structure among all possible deterministic nonanticipative encoders with feedback.

Given any feedback encoder of example Class A, an encoder and any source and channel disributions, the information from the source to the channel output is the directed information defined by

$I\left(X^n\to B^n\right)\sum _{i=0}^nI\left(X^i;B_i|B^{i-1}\right)\stackrel{\left(\alpha \right)}{=}\sum _{i=0}^nI\left(X^i,A^i;B_i|B^{i-1}\right)$

Also, given any encoder of Class A, the following chain rule of conditional mutual information holds:

$\begin{array}{c}I\left(X^i;B_i|B^{i-1}\right)=I\left(X^i,A^i;B_i|B^{i-1}\right)\\ =I\left(A_i;B_i|B^{i-1}\right)+I\left(X^i,B_i|B^{i-1},A^i\right),\\ i=0,1,\dots ,n\\ \ge I\left(A^i;B_i|B^{i-1}\right),i=0,1,\dots ,n\end{array}$

where the inequality is due to the nonnegativity of conditional mutual information. In fact, the following stronger version of this expression holds:For any eⁿε_[0,n] then

$\log \left(\frac{P_{B_i|B^{i-1},X^i}}{P_{B_i|B^{i-1}}}\right)=\log \left(\frac{P_{B_i|B^{i-1},A^i}}{P_{B_i|B^{i-1}}}\right)+\log \left(\frac{P_{B_i|B^{i-1},A^i,X^i}}{P_{B_i|B^{i-1},A^i}}\right),i=0,1,\dots ,n.$

A feedback encoder of example class A is “information lossless” with respect to the directed information measures

I(Xⁿ→Bⁿ) and I(Aⁿ→Bⁿ)

if

I(Xⁿ→Bⁿ)=I(Aⁿ→Bⁿ), ∀eⁿε_[0,n]^IL⊂_[0,n].

A sufficient condition for a feedback encoder of example Class A to be “information lossless” according to this definition is based on the following conditional independence:

X ⁱ (Aⁱ,Bⁱ⁻¹)B_i, i=1, . . . ,n.  MC1:

Given any feedback encoder of example class A, if

X ¹ (Aⁱ,Bⁱ⁻¹)B_i then I(Xⁱ,B_i|Bⁱ⁻¹,Aⁱ)=0,

and the following identify holds:

I(Xⁱ;B_i|Bⁱ⁻¹)=I(Aⁱ;B_i|Bⁱ⁻¹), i=0,1, . . . ,n.

Hence, any class of functions which induces MC1 or equivalently, induces the conditional independence on the sequence of channel conditional distributions

P _{B i |B i−1 ,A i ,X i} =P _{B i |B i−1 ,A i} , i=0,1, . . . ,n

is an information lossless class of functions.

Sufficient conditions (and also necessary) for any class of functions to be an information lossless class, for a feedback encoder of example class A channel, may be expressed as follows:

Class A Encoder is Information Lossless if

for fixed b⁻¹ε⁻¹, e₀(Φ,b⁻¹):₀₀ is one-to-one, onto ₀,

and its inverse φ₀e₀⁻¹(•,b⁻¹):₀₀ is measurable,

for fixed (a₀,x₀,b⁻¹,b₀)ε₀×₀×⁻¹×₀, e₁(a₀,x₀,•,b⁻¹,b₀):₁₁

is one-to-one onto ₁, and its inverse φ₁e₁⁻¹(a₀,•,x₀,b⁻¹,b₀):₁₁ is measurable

for any i=2, 3, . . . , n,

for fixed (aⁱ,xⁱ⁻¹,bⁱ⁻¹)εⁱ×ⁱ⁻¹×ⁱ⁻¹×_i, e_i(aⁱ,xⁱ⁻¹,•,bⁱ⁻¹):_ii is one-to-one, onto _i, and its inverse φ_ie_i⁻¹(aⁱ⁻¹,•,xⁱ⁻¹,bⁱ⁻¹):_ii is measurable.

All of the examples of capacity achieving encoders with feedback, discussed in the above sections, satisfy these necessary and sufficient conditions and are, thus, information lossless

This class of Information Lossless encoders also satisfies the following conditional independence:

A ⁱ (Xⁱ,Bⁱ⁻¹)B_i, i=0,1, . . . ,n.  MC2:

Still further, the following stronger identity holds for the information lossless encoders:

$\forall e^n\in {\left[0,n\right]}_{\mathrm{IL}}^{}\mathrm{then}\log \left(\frac{P_{B_i|B^{i-1},X^i}}{P_{B_i|B^{i-1}}}\right)=\log \left(\frac{P_{B_i|B^{i-1},A^i}}{P_{B_i|B^{i-1}}}\right),i=0,1,\dots ,n.$

C.2.2. Encoders without Feedback Corresponding to Example Class B

An encoder without feedback corresponding to the example class B may be referred to herein as

e ⁿε_[0,n]^nfb,

The information structure entering the example encoder without feedback at any time i may be expressed as {aⁱ⁻¹,xⁱ}.

By substituting aⁱ⁻¹ recursively into the right side of

a _i =e _i(aⁱ⁻¹,xⁱ)

then

a _i =e _i(aⁱ⁻¹,xⁱ)≡ē_i(xⁱ), i=0, . . . ,n.

Thus, for any encoder without feedback of example class B, the information structure of the encoder at each time instant i is:

_i ^e,nfb {a ⁱ⁻¹ ,x ⁱ }≡{x ⁱ }, i=0, . . . ,n,

and this information structure is the most general classical information structure among all possible deterministic nonanticipative encoders.

Given any encoder without feedback of example Class B and any source and channel distributions, the information from the source to the channel output is the mutual information defined by

$I\left(X^n;B^n\right)=\sum _{i=0}^nI\left(X^n;B_i|B^{i-1}\right)=\sum _{i=0}^nI\left(X^i;B_i|B^{i-1}\right)\stackrel{\left(\alpha \right)}{=}\sum _{i=0}^nI^e\left(X^i,A^i;B_i|B^{i-1}\right)$

An encoder without feedback of example class B is “information lossless” with respect to the directed information measures

I(Xⁿ;Bⁿ) and I(Aⁿ;Bⁿ)

if

I(Xⁿ;Bⁿ)=I(Aⁿ;Bⁿ), ∀eⁿε_[0,n]^IL.nfb⊂_[0,n]^nfb.

Sufficient conditions (and also necessary) for any class of functions to be an information lossless class, for an encoder without feedback of example class B, may be expressed as follows:

Class B Encoder is information Lossless if

e₀(•):₀ is one-to-one, onto ₀,

and its inverse φ₀e₀⁻¹(•)·₀₀ is measurable

for fixed (a₀,x₀)ε₀×₀, e₁(a₀,x₀,•):₁₁

is one-to-one, onto ₁, and its inverse φ₁e₁⁻¹(a₀,•,x₀):₁₁ is measurable for any i=2, 3, . . . , n,

for fixed (aⁱ,xⁱ⁻¹)εⁱ×ⁱ⁻¹, e_i(aⁱ,xⁱ⁻¹,•):_ii is one-to-one, onto _i, and its inverse φ_ie_i⁻¹(aⁱ⁻¹,•,xⁱ⁻¹):_ii is measurable.

Still further, the following stronger identity holds for the information lossless encoders:

$\forall e^n\in {\left[0,n\right]}_{\mathrm{IL},\mathrm{nfb}}^{}\mathrm{then}\log \left(\frac{P_{B_i|B^{i-1},X^i}}{P_{B_i|B^{i-1}}}\right)=\log \left(\frac{P_{B_i|B^{i-1},A^i}}{P_{B_i|B^{i-1}}}\right),i=0,1,\dots ,n.$

D. Compressing Information with Zero-Delay

The below-discussed compressions of information with zero-delay utilize a “nonanticipative” (e.g., zero-delay) rate distortion function (RDF). This function along with various other relevant quantities are defined and discussed below before specifying a number of example compression schemes. Also, in the following discussion, a “non-stationary” source may refer to a source of information, such as the source 102, that varies in time.

D.1. Nonanticipative RDF of Non-Stationary Sources

Generally, an RDF may define an manner in which data is to be sent over a channel, such as the channel 106. For example, an RDF may define a number of bits per symbol of information that should be sent over a channel. The manner in which data is sent over a particular channel may be optimal or not depending on the particular RDF that defines compression for the particular channel. For example, to achieve a capacity, such as the capacities discussed in section B, a compression of information may utilize a nonanticipative RDF, as discussed below.

A nonanticipative RDF may be defined in terms of a “source distribution,” a “reproduction distribution,” and a “fidelity of reproduction,” in an implementation. The source distribution may be a collection of conditional probability distributions:

{P_Xt|Xt−1(dx_t|x^t−1):t=0,1, . . . ,n}.

The reproduction distribution may also be a collection of conditional probability distributions:

{P_Yt|Yt−1,Xt(dy_t|y^t−1,x^t):t=0,1, . . . ,n}.

Also, to express the nonanticipative RDF, the following family of causal conditional distributions are defined:

{right arrow over (P)} _{Y n |X n} (dyⁿ|xⁿ)_t=0ⁿP_Yi|Yt−1,xt(dy_t|y^t−1,x^t).

Given the source distribution and reproduction distribution, the “joint distribution” is given by:

P _{X n ,Y n} (dxⁿ,dyⁿ)P_Xn(dxⁿ){right arrow over (P)}_Yn|Xn(dxⁿ,dyⁿ),

and the “marginal distributions” are given by:

P _{Y n} (dyⁿ)∫_XnP_Xn(dxⁿ){right arrow over (P)}_Yn|Xn(dyⁿ|xⁿ).

The distortion function of reproducing x_t by y_t, for t=0, 1, . . . , n, may be a measurable function:

d _0,n:_0,n×_0,n[0,∞]d_0,n(xⁿ,yⁿ)=Σ_t=0ⁿρ_t(T^txⁿ,T^tyⁿ),

where,

T ^t x ⁿ ⊂{x ₀ ,x ₁ , . . . ,x _t },T ^t y ⁿ ⊂{y ₀ ,y ₁ , . . . ,y _t }, t=0,1, . . . ,n.

The fidelity set of reproduction conditional distributions is then defined by:

${0,n}_{}\left(D\right)\left\{{Y^nX^n}_{}\left(·x^n\right)\text{:}\frac{1}{n+1}{\int }_{X_{0,n}\times Y_{0,n}}^{}{}_{0,n}\left(x^n,y^n\right)P_{X^n,Y^n}\left(x^n,y^n\right)\le D\right\}$

where D≧0.

The information measure of the nonanticipative RDF may be a special case of directed information defined by:

$I_{P_{X^n}}\left(X^n->Y^n\right)\int \log \left(\frac{{\stackrel{->}{P}}_{Y^nX^n}\left(y^nx^n\right)}{P_{Y^n}\left(y^n\right)}\right)P_{X^n,Y^n}\left(x^n,y^n\right)\equiv {}_{X^n->Y^n}\left(P_{X^n},{\stackrel{->}{P}}_{Y^nX^n}\right).$

The finite time nonanticipative RDF may be defined by:

$R_{0,n}^{\mathrm{na}}\left(D\right){}_{X^n->Y^n}\left(P_{X^n},{\stackrel{->}{P}}_{Y^nX^n}\right).$

and the nonanticipative RDF rate may be defined by:

$R^{\mathrm{na}}\left(D\right)=\underset{n->\infty }{\lim }\frac{1}{n+1}R_{0,n}^{\mathrm{na}}\left(D\right)$

This RDF function may specify that R^na(D) bits/symbol are to be transmitted over a channel, such as the channel 106, such that the distortion does not exceed D. The distortion D may be represented by any suitable distortion measure, such as the Hamming distortion measure, the squared-error distortion measure, etc.D.2. Methods for Compressing Information with Zero-Delay

FIG. 9 is a flow diagram of an example method 900 for compressing information with zero-delay (e.g., for use in real-time communications). A computing device, such as the computing device illustrated in detail in FIG. 15, may be specially and/or specifically configured to implement at least a portion of the method 900. Further, in some implementations, a suitable combination of a computing device, an operator of a communication system, such as the communication system 100, and one or more other suitable components of a communication system (e.g., routers, modems, gateways, etc.) may implement the method 900.

In the method 900, a computing device and/or operator of a communications system determines a nonanticipative rate distortion function (RDF) (block 902). The nonanticipative RDF may be a function, as described above in section D.1, that is not dependent on all transmitted symbols over a channel. Rather, the nonanticipative RDF may be causal in that the nonanticipative RDF only depends on previously transmitted information over a channel. In this manner the determined nonanticipative RDF is zero-delay.

In some implementations, a computer and/or operator of a communications system may determine the nonanticipative RDF according to the definitions in section D.1 and further characterize the nonanticipative RDF according to properties of the channel over which information is to be transmitted. For example, a computer or operator may characterize the nonanticipative RDF according to a properties of an AWGN channel, as further discussed in various examples presented below. In any event, the method 900 includes determining a nonanticipative RDF, where the nonanticipative RDF may be expressed in any suitable form including a general form for any suitable channel or a more specific characterization according to properties of a specific channel.

The method 900 also includes determining a rate of information transfer based on the RDF and an allowed amount of distortion (block 904). In some implementations, the allowed amount of distortion may be a number or expression representing an amount of distortion (e.g., Hamming or squared-error distortion) of information transmitted over one or more channels. The computer or operator of a communications system may determine the allowed amount of distortion based on a desired performance (e.g., efficiency) of transmitting information and/or based on desired qualities of the received information after transmission. For example, for information representing speech, an operator of a communication system may specify (e.g., by configuring a computer to compress information) an allowed amount of distortion such that transmitted speech signals are understandable to a human after being decoded.

The computer and/or operator may provide the allowed amount of distortion to the determined nonanticipative RDF (e.g., as input) to determine the rate of information transfer, which rate corresponds to the allowed amount of distortion. In other words, when the allowed amount of distortion is provided to the nonanticipative RDF, the nonanticipative RDF produces a corresponding rate of information transfer. If information is transferred over a channel at this rate, the information will be distorted (e.g., according to a squared-error distortion) at a level at or below the allowed amount of distortion. The computer and/or operator implementing portions of the method 900 may express the rate as a number of bits per source symbol, a number of bytes per source symbol, or any other suitable amount of data per symbol of source information.

In some implementations, the computer and/or operator may utilize a buffer or range in determining the rate of information transfer. For example, instead of utilizing one allowed amount of distortion, the computer and/or operator may determine a range of information transfer rates (e.g., bits/symbol) based on an allowed range of distortions. In other examples, the computer and/or operator may determine a rate of information transfer based on a proxy amount of distortion, which proxy defines a buffer between an actual allowed amount of distortion and the proxy amount.

Still further, the method 900 includes compressing information from a source according to the determined rate (block 906). The computing device implementing at least this portion of block 900 may apply any number of suitable compression methods to compress information from a source of information, such as the source 102, such that the compressed information results in a rate (e.g., bits/symbol) at or less than the rate determined at block 904. Example compression methods may include, by way of example, A-law algorithms, code-excited linear predictions, linear predictive coding, mu-law algorithms, block truncation coding, fast cosine transform algorithms, set partitioning in hierarchical trees, etc. Alternatively, a computing device may utilize a coding scheme designed according to the JSCC methods discussed herein to simultaneously compress and encode information from a source.

The method 900 still further includes transmitting the compressed information at the determined rate (block 908). Once compressed, components of a communication system, such as the system 100, may transmit the compressed information at the determined rate. In some implementations, this may include further encoding of the compressed information, and, in other implementations utilizing JSCC, the compressed information may already be encoded for optimal transmission over a channel.

D.3. Closed Form Expressions for a Finite-Time Nonanticipative RDF

As discussed in section D.1, a finite-time (e.g., not taking an infinite number of transmissions) expression for the nonanticipative RDF may include an infimum. Below, closed-form expressions are presented for a nonstationary optimal reproduction conditional distribution, which distribution attains the infimum of the finite-time nonanticipative RDF, R_0,n^na(D).

D.3.1. Optimal Reproduction Conditional Distribution for the Nonanticipative RDF

If the infimum of RZ (D) is attained at:

{P_Yt|Yt−1,Xi*(•|y^t−1,x^t), t=0, . . . ,n}

then R_0,n^na (D) satisfies the following backward in time recursive equations:For t=n:

$P_{Y_nY^{n-1},X^n}^{*}\left(y_ny^{n-1},x^n\right)=\frac{{}^{s{\rho }_n\left(T^nx^n,T^ny^n\right)}P_{Y_nY^{n-1}}^{*}\left(y_ny^{n-1}\right)}{{\int }_{{}_n}{}^{s{\rho }_n\left(T^nx^n,T^ny^n\right)}P_{Y_nY^{n-1}}^{*}\left(y_ny^{n-1}\right)}.$

where s<0 is the Lagrange multiplier of the fidelity, andFor t=n−1, n−2, . . . , 0:

$P_{Y_tY^{t-1},X^t}^{*}\left(y_ty^{t-1},x^t\right)=\frac{{}^{s{\rho }_t\left(T^tx^n,T^ty^n\right)-g_{t,n}\left(x^t,y^t\right)}P_{Y_tY^{t-1}}^{*}\left(y_ty^{t-1}\right)}{{\int }_{{}_t}{}^{s{\rho }_t\left(T^tx^n,T^ty^n\right)-g_{t,n}\left(x^t,y^t\right)}P_{Y_tY^{t-1}}^{*}\left(y_ty^{t-1}\right)}$

where g_t,n(x^t,y^t) is given by:

gt,n (x^t, y^t)=−P_XT+1(dx_t+1|X^t)log e^sρt+1(T^t+1xⁿ, T^t+1yⁿ)−g_f+l,n(x^t+l, y^t+1)P^*_Yt+1|Yt(dy_t|y^t))

and the finite time nonanticipative RDF is given by:

$R_{0,n}^{\mathrm{na}}\left(D\right)=\mathrm{sD}\left(n+1\right)-\sum _{t=0}^n{\int }_{X^t\times Y^{t-1}}^{}\{{\int }_{{}_t}g_{t,n}\left(x^t,y^t\right)P_{Y_tY^{t-1},X^t}^{*}\left(y_ty^{t-1},x^t\right)+\log \left({\int }_{{}_t}{}^{s{\rho }_t\left(T^tx^n,T^ty^n\right)-g_{t,n}\left(x^t,y^t\right)}P_{Y_tY^{t-1}}^{*}\left(y_ty^{t-1}\right)\right)\}\otimes P_{X_tX^{t-1}}\left(x_tx^{t-1}\right)\otimes P_{X^{t-1},Y^{t-1}}\left(x^{t-1},y^{t-1}\right).$

If R_0,n^na(D)>0, then s<0, and

$\frac{1}{n+1}{\Sigma }_{t=0}^n\int \rho t\left(T^tx^n,T^ty^n\right)P_{X^tY^t}\left(x^t,y^t\right)=D.$

D.3.2. Information Structures of the Nonanticipative RDF

From the above expressions, given any source distribution, a computing device and/or operator of a communication system may identify a dependence of the optimal nonstationary reproduction distribution on past and present source symbols. However, the above expressions do not immediately yield a dependence on past reproduction symbols, referred to herein as “information structures.” Regarding this dependence, the following observations are presented:

(1) The dependence of P_Yn|Yn−1,Xn*(dy_n|yⁿ⁻¹,xⁿ) on xⁿε_0,n is determined by the dependence of ρ_n(Tⁿxⁿ,Tⁿyⁿ) on xⁿε_0,n as follows:(2) If ρ_n(Tⁿxⁿ,Tⁿyⁿ)={tilde over (ρ)}(x_n,yⁿ) then P_Yn|Yn−1,Xn*(dy_n|yⁿ⁻¹,xⁿ)=P_Yn|Yn−1,Xn*(dy_n|yⁿ⁻¹,x_n), while for t=n−1, n−2, . . . , 0. the dependence of P_Yt|Yt−1,Xt*(dy_t|y^t−1,x^t) on xⁱε_0,i is determined by the dependence of ρ_t(x^t,y^t) on x^tε_0,t and g_t,n(x^t,y^t) on x^tε_0,t.(3) If P_Xt|Xt−1(dx_t|x^t−1)=P_Xt|Xt−1(dx_t|x_t−1) and ρ_t(T^txⁿ,T^tyⁿ)={tilde over (ρ)}(x_t,y_t) then P_Yt|Yt−1,Xt*(dy_t|y^t−1,x^t)=P_Yt|Yt−1,Xt*(dy_t|y^t−1),x_t).(4) If g_t,n(x^t,y^t)=g_t,n(x^t,y^t−1), t=0, . . . , n−1, optimal reproduction distribution (IV.227) reduces to

$P_{Y_tY^{t-1},X^t}^{*}\left(y_ty^{t-1},x^t\right)=\frac{{}^{s{\rho }_t\left(T^tx^n,T^ty^n\right)}P_{Y_tY^{t-1}}^{*}\left(y_ty^{t-1}\right)}{{\int }_{{}_t}{}^{s{\rho }_t\left(T^tx^n,T^ty^n\right)}P_{Y_tY^{t-1}}^{*}\left(y_ty^{t-1}\right)},t=0,1,\dots n.$

D.3.3. Alternative Characterization of the Finite-Time Nonanticipative RDF

To further clarify the dependence of the optimal reproduction distribution on past reproductions, an alternative characterization of R_0,n^na(D) may include a maximization over a certain class of functions. A computing device and/or operator of a communications system may utilize this alternative characterization to derive lower bounds on R_0,n^na(D), which bounds are achievable.

The alternative characterization may be expressed as:

$R_{0,n}^{\mathrm{na}}\left(D\right)=\underset{s\le 0}{\sup }\underset{\left\{{\lambda }_t\in {\Psi }_s^t:t=0,\dots ,n\right\}}{\sup }\left\{\mathrm{sD}\left(n+1\right)-\sum _{=0}^n{\int }_{{}_{0,t}\times {}_{0,t-1}}^{}\left\{{\int }_{{}_t}^{}g_{t,n}\left(x^t,y^t\right)P_{Y_t|Y^{t-1},X^t}^{*}y_t|y^{t-1},x^t\right)+\log \left({\lambda }_t\left(x^t,y^{t-1}\right)\right)\right\}\otimes P_{X_t|X^{t-1}}\left(x|x^{t-1}\right)\otimes P_{X^{t-1},Y^{t-1}}\left(x^{t-1},y^{t-1}\right)$ $\mathrm{where}$ ${\Psi }_s^t\left\{{\lambda }_t\left(x^t,y^{t-1}\right)\ge 0\text{:}{\int }_{{}_{0,t-1}}^{}\left({\int }_{{}_t}^{}{}^{s{\rho }_t\left(T^tx^n,T^ty^n\right)-g_{t,n}\left(x^t,y^t\right)}{\lambda }_t\left(x^t,y^{t-1}\right)P_{X_t|X^{t-1}}\left(x_t|x^{t-1}\right)\right)\otimes P_{X^{t-1}|Y^{t-1}}\left(x^{t-1}|y^{t-1}\right)\le 1\right\}$ $g_{n,n}\left(x^n,y^n\right)=0,g_{t,n}\left(x^t,y^t\right)=-{\int }_{{}_{t+1}}^{}P_{X_{t+1}|X^t}\left(x_{t+1}|x^t\right){\log \left({\lambda }_{t+1}\left(x^{t+1},y^t\right)\right)}^{-1},t=0,1,\dots ,n-1.$

For sε(−∞,0|, a necessary and sufficient condition to achieve the supremum of the above alternative characterization of R_0,n^na(D) is the existence of a probability measure

P _{Y t |Y t−1} *(dy_t|y^t−1)

such that

λ_t(xⁱ, y^t−1)={e^{sρt(xt+1, yt+1)}P^*_Yt|Yt1(dy_t|y^t−1)}⁻¹, t=0,1. . ., n,

and such that,

$c\left(y^t\right){\int }_{{}_{0,t-1}}^{}\left({\int }_{{}_t}^{}{}^{s{\rho }_t\left(T^tx^n,T^ty^n\right)-g_{t,n}\left(x^t,y^t\right)}{\lambda }_t\left(x^t,y^{t-1}\right)\otimes P_{X_t|X^{t-1}}\left(x_t|x^{t-1}\right)\right)\otimes P_{X^{t-1}|Y^{t-1}}\left(x^{t-1}|y^{t-1}\right)=1,t=0,1,\dots ,n.$

The above alternative characterization of R_0,n^na(D) may allow a computing device and/or operator to compute R_0,n^na(D) exactly (e.g., as part of the example method 600) for a given source with memory.

D.4. Example Nonanticipative RDF for a Gaussian Source and Optimal Transmission Over an AWGN Channel

In an example scenario, the following expression describes a p-dimensional nonstationary Gaussian source process:

X _t+1 =A _t X _t +B _t W _t , t=0,1, . . . ,n−1

where

A _tε^p×p,B_tε^p×k, t=0,1, . . . , n−1.

For an Autoregressive Moving Average model with finite tap delays, there may exist a state space representation for some p. For the following example analysis in this scenario, assume:(G1) X₀ε^p is Gaussian N( x₀, λ₀):(G2) {W_t:t=0, . . . , n} is a k-dimensional IID Gaussian N(0,_k×k) sequence, independent of X₀:(G3) The distortion function is single letter defined by d_0,n(xⁿ,yⁿ)Σ_t=0ⁿ∥x_t−y_t∥.

The nonstationary optimal reproduction distribution may be given, for s≦0, by the following recursive equations:

$P_{Y_n|Y^{n-1},X_n}^{*}\left(y_n|y^{n-1},x_n\right)=\frac{{}^{s{y_n-x_n}_{{ℝ}^p}^2}P_{Y_n|Y^{n-1}}^{*}\left(y_n|y^{n-1}\right)}{{\int }_{{}_n}^{}{}^{s{y_n-x_n}_{{ℝ}^p}^2}P_{Y_n|Y^{n-1}}^{*}\left(y_n|y^{n-1}\right)}$ $g_{n,n}\left(x_n,y^n\right)=0$ $P_{Y_t|Y^{t-1},X_t}^{*}\left(y_t|y^{t-1},x_t\right)=\frac{{}^{s{y_t-x_t}_{{ℝ}^p}^2-g_{t,n}\left(x_t,y^t\right)}P_{Y_t|Y^{t-1}}^{*}\left(y_t|y^{t-1}\right)}{{\int }_{{}_t}^{}{}^{s{y_t-x_t}_{{ℝ}^p}^2-g_{t,n}\left(x_t,y^t\right)}P_{Y_t|Y^{t-1}}^{*}\left(y_t|y^{t-1}\right)},t=n-1,n-2,\dots ,0g_{t,n}\left(x_t,y^t\right)=-{\int }_{{}_{t+1}}^{}P_{X_{t+1}|X_t}\left(x_{t+1}|x_t\right)\log \left({\int }_{{}_{t+1}}^{}{}^{s{y_{t+1}-x_{t+1}}_{{ℝ}^p}^2-g_{t+1,n}\left(x_{t+1},y^{t+1}\right)}\otimes P_{Y_{t+1}|Y^t}^{*}\left(y_{t+1}|y^t\right)\right),t=n-1,n-2,\dots ,0.$

Thus, the optimal reproduction distributions may be conditionally Gaussian, and the optimal reproduction distributions may be realized using a general Gaussian channel with memory, modeled by:

Y _t =Ā _t X _t + B _t Y ^t−1 +V _t ^c , t=0, . . . ,n

where

Ā _tε^p×p, B_tε^p×tp, and {V_t^c:t=0, . . . ,n}

are independent sequences of Gaussian vectors:

{N(0:Q_t):t=0, . . . ,n}.

Gaussian error processes may introduce the pre-processing at the encoder and/or decoder, in this example. Let,

{K_t:t=0, . . . ,n}, K_tX_t−{X_t|Y^t−1},

denote the covariance of the pre-processing by:

Λ_t{K_tK_t^tr}, t=0, . . . ,n.

Also, let E_t be a unitary matrix such that:

E _tΛ_tE_t^tr=diag{λ_t,1, . . . λ_t,p}, ΓE_tK_t, t=0, . . . ,n.

Analogously, to obtain the nonanticipative RDF in this example, the processes:

{{tilde over (k)}_t:t=0, . . . ,n}

defined by

{tilde over (K)} _t Y _t − {X _t |Y ^t−1 }≡Y _t −{circumflex over (X)} _t|t−1,{tilde over (Γ)}_t=E_t{tilde over (K)}_t.

are introduced. Using properties of conditional entropy, observations for this example scenario include:

d _0,n(Xⁿ,Yⁿ)=d_0,n(Kⁿ,{tilde over (K)}ⁿ)=Σ_t=0ⁿ∥{tilde over (K)}_t−K_tp=Σ_t=0ⁿ∥{tilde over (Γ)}_t−Γ_tp.

and

R _0,n ^na(D)=R_0,n^{na,Kn,{tilde over (K)}n}(D)=R_0,n^{na,Γn,{tilde over (Γ)}n}(D).

Using these observation, a computing device and/or operator of a communication system may obtain the optimal (e.g., capacity achieving) nonanticipative RDF for the above defined multidimensional Gaussian process. The computing device and/or operator may also identify a “duality” between a multidimensional Gaussian process and a MIMO AWGN channel. These results are described below by way of example.

The R_0,n^na(D) of the example Gaussian source, according to the definitions in section D.1 and the above discussed model of the Gaussian source, is given by:

$R_{0,n}^{\mathrm{na}}\left(D\right)=\frac{1}{2}\sum _{t=0}^n\sum _{i=1}^p\log \left(\frac{{\lambda }_{t,i}}{{\delta }_{t,i}}\right){\Lambda }_t\},{\hat{X}}_{t|t-1}\left\{X_t|Y^{t-1}\right\}$ ${\delta }_{t,i}\{\begin{array}{cc}\xi & \mathrm{if}\xi \le {\lambda }_{t,i}\\ {\lambda }_{t,i}& \mathrm{if}\xi >{\lambda }_{t,i}\end{array},t=0,\dots ,n,i=1,\dots ,p\mathrm{and}\xi \mathrm{is}\mathrm{chosen}\mathrm{such}\mathrm{that}\frac{1}{n+1}\sum _{t=0}^n\sum _{i=1}^p{\delta }_{t,i}=D.$

where the error

X _t − {X _t |Y ^t−1}

is Gaussian, and where A: are given by the following Kalman filter equations:

${\hat{X}}_{t+1|t}=A{\hat{X}}_{t|t-1}+A{{\Lambda }_t\left(E_t^{\mathrm{tr}}H_tE_t\right)}^{\mathrm{tr}}M_t^{-1}\left(Y_t-{\hat{X}}_{t|t-1}\right),{\hat{X}}_{0|-1}=X_0$ ${\Lambda }_t=A{\Lambda }_tA^{\mathrm{tr}}-A{{\Lambda }_t\left(E_t^{\mathrm{tr}}H_tE_t\right)}^{\mathrm{tr}}M_t^{-1}\left(E_t^{\mathrm{tr}}H_tE_t\right){\Lambda }_tA^{\mathrm{tr}}+{\mathrm{BB}}_t^{\mathrm{tr}},{\Lambda }_0={\stackrel{\_}{\Lambda }}_0$ $M_t=E_t^{\mathrm{tr}}H_tE_t{{\Lambda }_t\left(E_t^{\mathrm{tr}}H_tE_t\right)}^{\mathrm{tr}}+E_t^{\mathrm{tr}}t_{}Q_tt_{\mathrm{tr}}^{}E_t$ $H_t\mathrm{diag}\left\{{\eta }_{t,1,\dots ,}{\eta }_{t,p}\right\},{\eta }_{t,i}=1-\frac{{\delta }_{t,i}}{{\lambda }_{t,i}},Q_t\mathrm{Cov}\left(V_t^c\right),\sqrt{H_t{\Delta }_tQ_t^{-1}},{\Delta }_t=\mathrm{diag}\left\{{\delta }_{t,1},\dots ,{\delta }_{t,p}\right\},i=1,\dots ,p,t=0,\dots n.$

Moreover,

σ{Y^t}=σ{{tilde over (K)}^t}=σ{B^t}, t=0, . . . ,n

(i.e., these processes generate the same information).

FIG. 10A depicts a combination of encoder-channel-decoder in a communication system 1000 that realizes R_0,n^na(D). That is, information encoded by an encoder 1002, transmitted over one or more channels 1004, and decoded by a decoder 1006 is distorted according to R_0,n^na(D). A computing device may compress information transmitted over the channels 1004 at a certain rate (e.g., number of bits/symbol), defined by R_0,n^na(D), to limit distortion of the information below a threshold distortion, as further discussed with reference to FIG. 9. In this example, the realization of the optimal nonstationary reproduction distribution is given by:

Y _t =E _t ^tr H _t E _t(X_t−{circumflex over (X)}_t|t−1)+E_t^tr_tV_t^c+X_t|t−1

The encoder 1002 and decoder 1006 of the system 1000 may encode and decode, respectively, with feedback according to:

A _t =E _t ^tr H _t E ^t(X_t−{circumflex over (X)}_t|t−1)

Y _t =E _t ^tr H _t E _t {X _t −{circumflex over (X)} _t|t−1 }+E _t ^tr _t V _t ^c +X _t|t−1

Alternatively, the encoder 1002 and decoder 1006 of the system 1000 may encode and decode, respectively, without feedback according to:

A _t =E _t ^tr H _t E _t(X_t−(X_t))

Y _t =E _t ^tr H _t E _t {X _t−(X_t)}+E_t^tr_tV_t^c+(X_t)

By taking a limit of R_0,n^na(D), a computing device and/or operator may obtain the per unit time nonanticipative RDF R^na(D) from R_0,n^na(D). This R^na(D) may represent the rate distortion function of stationary (e.g., not varying in time) Gaussian sources of information, in this example. Specifically, the R^na(D) is obtained as follows:

$\begin{array}{c}{R}^{\mathrm{na}}\left(D\right)=\frac{1}{2}\underset{n\to \infty }{\lim }\frac{1}{n+1}\sum _{t=0}^n\sum _{i=1}^p\log \left(\frac{{\lambda }_{t,i}}{{\delta }_{t,1}}\right)\\ =\frac{1}{2}\sum _{i=1}^p\log \left(\frac{{\lambda }_{\infty ,i}}{{\delta }_{\infty ,i}}\right).\end{array}$

wherelim_n→∞δ_t,i=δ_∞,i and lim_n→∞λ_t,i=λ_∞,i.

In addition, for a scalar Gaussian stationary source:

$X_{t+1}={\mathrm{\alpha X}}_t+{\sigma }_WW_t,W_t\sim N\left(0;1\right)$ $\begin{array}{c}{R}^{\mathrm{na}}\left(D\right)=\frac{1}{2}\log \left(\frac{{\lambda }_{\infty ,1}}{{\delta }_{\infty ,1}}\right),{\delta }_{\infty ,1}=D,{\lambda }_{\infty ,1}={\alpha }^2D+{\sigma }_W^2,\\ =\frac{1}{2}\log \left({\alpha }^2+\frac{{\sigma }_W^2}{D}\right),{\lambda }_{\infty ,1}\ge {\delta }_{\infty ,1.}\end{array}$

For an independent and identically distributed (IID) scalar Gaussian source:

$R^{\mathrm{na}}\left(D\right)=R\left(D\right)=\frac{1}{2}\log \frac{{\sigma }_X^2}{D},{\sigma }_X\ge D$

Note, for a vector of independent sources with memory, the nonanticipative RDF involves water filing in the spatial dimension. The realization of these sources over an Additive White Gaussian Noise Channel with an average power constraint not exceeding P is shown in FIG. 10B.

Returning to FIG. 10A, the encoder 1002 may implement the compression scheme for vector correlated sources, and the encoder 1002 may achieve the capacity of a communication system where the source is a Gaussian random process and where information is transmitted over additive white Gaussian noise (AWGN) channels. This encoding may also be equivalent to JSCC design using symbol-by-symbol transmission of a vector Gaussian source with memory over a multiple-input multiple output Gaussian memoryless channel. In this capacity achieving case, the encoded symbols may be expressed with respect to the encoder 1002 which either includes feedback (case (1)) or does not include feedback (case (2)). That is, the special cases (1), (2) below follow from the general nonanticipative RDF of correlated Gaussian sources with a square error distortion function.

(1) Capacity Achieving Realization with Feedback.

Consider the realization of the

$R\left(D\right)=R^{\mathrm{na}}\left(D\right)=\frac{1}{2}\log \frac{{\sigma }_X^2}{D}$

(i.e., for IID processes classical RDF=nonantieipative RDF), Let X b a RVN(0;σ_X²) with σ_X≧D. Letting p=1, then from (I.V.256)-(IV.257) we have

${}_t=q_t,1\frac{1}{{\delta }_{t,1}}\left(1-\frac{{\delta }_{t,1}}{{\lambda }_{t,1}}\right),\frac{P_{t,1}}{q_{t,1}}=\frac{{\lambda }_{t,1}}{{\delta }_{t,1}}-1$

which implies

$B_t=\sqrt{\frac{P_{t,1}}{{\lambda }_{t,1}}}K_t+V_t^c,t=0,\dots ,n.$

${}_t=\sqrt{\frac{P_{t,1}}{{\lambda }_{t,1}}},{\lambda }_{t,1}=\mathrm{Var}\left(X-\left(X|B^{i-1}\right)\right),K_t=X-\left(X|B^{i-1}\right),$

Substituting into the encoder the limiting values, δ_∞,1=D, lim_n→∞q_t,1=q_∞,1, lim_n→∞P_t,1=P_∞,1=then for i=0, 1, . . .

$B_t=\sqrt{\frac{P}{{\left\{X-\left(X|B^{i-1}\right)\right\}}^2}}\left\{X-\left(X|B^{i-1}\right)\right\}+V_t^c,t=0,1,\dots ,$

(2) Capacity Achieving Realization without Feedback.

When there is no feedback, all state-ments in (1) hold, λ_∞,1=σ_X², while (X|Bⁱ⁻¹) is replaced by (X_i|σ{})=(X) (i.e, only á priori information is used), and then (IV.264) reduces to

$B_t=\sqrt{\frac{P}{{\left\{X_t-\left(X_t\right)\right\}}^2}}\left\{X_t-\left(X_t\right)\right\}+V_t^c,t=0,1,\dots ,$

E. Joint Source Channel Coding Design

The following description may refer to “Joint Source Channel Coding,” or JSCC, as a coding/decoding scheme or a design of a coding/decoding scheme that does not separate source encoder and decoders and channel encoders and decoders. Such JSCC may produce a coding scheme that both compresses and encodes data for transmission, where the compression occurs with zero-delay and the compression and encoding allow the transmission to achieve a capacity of a channel.

To further clarify this point and by way of example, FIG. 11 depicts another example communication system 1100. A JSCC design of an encoder 1102 may not separate the source coding and channel coding of the encoder 1102. That is, the encoder 1102 may include a single code to perform both source coding (e.g., compression) and channel coding (e.g., encoding of the compressed symbols). Similarly, a JSCC design of a decoder 1104 may not separate the source decoding and channel decoding of the decoder 1104.

E.1. Methods for Joint Source Channel Coding Design

FIG. 12 is a flow diagram of an example method 1200 for designing and utilizing an {encoder, decoder} pair based on JSCC. A computing device, such as the computing device illustrated in detail in FIG. 15, may be specially and/or specifically configured to implement at least a portion of the method 1200. Further, in some implementations, a suitable combination of a computing device, an operator of a communication system, such as the communication system 100, and one or more other suitable components of a communication system (e.g., routers, modems, gateways, etc.) may implement the method 1200.

The example method 900 includes determining a rate distortion function (RDF) of the source (block 1202). The source, for example, may be the source 102, and a computing device and/or operator may determine the RDF to be a nonanticipative RDF corresponding to the source 102, as further discussed in section D entitled “Encoder Design Overview and Methods.” In this manner, the computing device and/or operator may determine an RDF that is zero-delay. In some implementations, the determined RDF may include one or more realizations of an encoder, channel, and decoder representing the RDF, and, in other applications, the determined RDF may include one or more sets of data, algorithms, or instructions that represent the RDF and are stored a non-transitory computer-readable medium.

The example method 900 also includes determining a capacity of a channel over which information, generated by the source, is to be transmitted (block 1204). The determined capacity may include a characterization or formula for a finite block length feedback capacity, feedback capacity, finite block length feedback capacity with transmission cost, feedback capacity with transmission cost, finite block length capacity without feedback, capacity without feedback, finite block length capacity without feedback and with transmission cost, and capacity without feedback and with transmission cost. A computing device and/or operator may determine this characterization or formula according to the method 300, for example.

An {encoder, decoder} pair is then identified (block 1206), where the {encoder, decoder} pair realizes the determined RDF and achieves the determined capacity. The {encoder, decoder} pair may include a single code for the encoder, which compresses and encodes symbols from a source, such as the source 102. The {encoder, decoder} pair may also include a single decoder, which code both decodes and decompresses symbols transmitted over the channel. In other words, the encoder may receive uncompressed symbols from a source and output compressed and encoded symbols for transmission over a channel, and the decoder may receive the transmitted symbols and output uncompressed and decoded symbols.

Identifying the {encoder, decoder} pair that realizes the determined RDF and achieves the determined capacity may include identifying an {encoder, decoder} pair that satisfies one or more conditions, in an implementation. As discussed further below, a computing device and/or operator may utilize the determined RDF and the determined capacity to generate specific conditions that the identified {encoder, decoder} pair must satisfy. Then the computing device and/or operator may identify a specific {encoder, decoder} pair that realizes the determined RDF and achieves the determined capacity.

Returning to FIG. 12, the example method 1200 may also include transmitting information according to the identified {encoder, decoder} pair (block 1208). A computing device and/or operator may configures one or more components of a communication system, such as computing devices, modems, routers, etc., to utilize the identified codes for encoding and decoding in transmitting information. This specific configuration may include specifically programming specialized modules of a computing device or other tangible communications device with the identified codes for encoding and decoding. A computing device and/or operator may then activate transmissions of information according to the identified {encoder, decoder} pair. These transmissions may be optimal or near optimal in the sense that R(D)=C(K) for a range of values (D, K).

E.2. Methodology of JSCC Design for General {Source, Channel} Pairs with Memory

In this section, example methodologies for JSCC design for general {source, channel} pairs with memory and with respect to {distortion function, transmission cost} pairs are developed. A computing device and/or operator may implement these methodologies as part of the example method 900, for example, or in another method in which encoders and/or decoders are designed according to JSCC. The methodologies are for JSCC design with respect to general {distortion function, transmission cost} pairs, where some examples of distortion functions and transmission cost functions are further discussed in sections B and D.

E.2.1. Nonanticipative (Zero-Delay) Code

To facilitate the development of JSCC methodologies, an example definition of nonanticipative code is introduced below. The definition is for any

{source, channel}≡{P_Xn,{P_Bi|Bi−1,Ai:i=0,1, . . . ,n}}

with respect to any{distortion function, transmission cost}≡{d_0,n,c_0,n}

A set of randomized nonanticipative encoders with feedback denoted by

_[0,n] ^fb,E

may be a sequence of conditional distributions:

P _{A i |A i−1 ,B i−1 ,X i} (a_i|aⁱ⁻¹,bⁱ⁻¹,xⁱ), i=0,1, . . . ,n

Also, an example set of randomized feedback encoders embeds nonanticipative deterministic feedback encoders defined by:

${\left[0,n\right]}_{\mathrm{fb}}^{}\left\{e_i\text{:}{}^{i-1}\times {}^{i-1}\times {}^i\mapsto {}_i,a_i=e_i\left(a^{i-1},b^{i-1},x^i\right),i=0,\dots ,n\right\}⋐{\left[0,n\right]}_{\mathrm{fb},E}^{}.$

The example encoders introduced above are nonanticipative in the sense that at each transmission time, i,

P _{A i |A i−1 ,B i−1 ,X i} (a_i|aⁱ⁻¹,bⁱ⁻¹,xⁱ) and e_i(aⁱ⁻¹,bⁱ⁻¹,xⁱ)

do not depend on any future symbols (e.g., symbols to be transmitted at future times). The encoders may only be functions of past and present symbols and past channel inputs and outputs, in an implementation.

A set of randomized and deterministic nonanticipative encoders without feedback may be defined by:

${\left[0,n\right]}_{\mathrm{nfb},E}^{}\left\{P_{A_iA^{i-1},x^i}\text{:}i=0,1,\dots ,n\right\}⋐{\left[0,n\right]}_{\mathrm{fb},E}^{},{\left[0,n\right]}_{\mathrm{nfb}}^{}{\left\{e_i\left(·,·\right)\text{:}a_i=e_i^{\mathrm{nfb}}\left(a^{i-1},x^j\right)\right\}}_{i=0}^n⋐{\left[0,n\right]}_{\mathrm{fb}}^{}.$

Also, a randomized decoder may be a sequence of conditional distributions defined by

${\left[0,n\right]}_D^{}\left\{P_{Y_iY^{i-1},B^i}\left(y_iy^{i-1},b^i\right)\text{:}i=0,1,\dots ,n\right\}$

embedding deterministic decoders denoted by:

${\left[0,n\right]}_D^{}{\{g_i\left(·,·\right)\text{:}y_i=g_i\left(y^{i-1},b^i\right)\}}_{i=0}^n⋐{\left[0,n\right]}_D^{}.$

Given any source, nonanticipative encoder as defined above, and randomized decoder as defined above, a joint probability distribution may be defined as:

P _{X n ,A n ,B n ,Y n} (dxⁿ,daⁿ,daⁿ,dbⁿ,dyⁿ)=_i=0ⁿP_Yi|Yi−1,Bi(dy_i|yⁱ⁻¹,bⁱ)P_Bi|Bi−1,Ai(db_i|bⁱ⁻¹,aⁱ)P_{Ai|Ai−1,Bi−1,Xi}(da_i|aⁱ⁻¹,bⁱ⁻¹,xⁱ)P_Xi|Xi−1(dx_i|xⁱ⁻¹).

Given any

{P_Xn,{P_Bi|Bi−1,Ai:i=0,1, . . . ,n}},

a nonanticipative code for JSCC design of a system,

${\Sigma }_{\mathrm{JSCC}}\left({}^n,{}^n,{}^n,{}^n,P_{X^n},\left\{P_{B_iB^{i-1},A^i}\text{:}i=0,1,\dots ,n\right\},d_{0,n},c_{0,n}\right),$

is a nonanticipative {encoder, decoder} pair

{P_{Ai|Ai−1,Bi−1,Xi},P_Yi|Yi−1,Bi:i=0,1, . . . ,n}

This nonanticipative {encoder, decoder} pair has an excess distortion probability:

{d_0,n(xⁿ,yⁿ)>(n+1)d}≦ε,εε(0,1),d≧0

and transmission cost:

$\frac{1}{n+1}\{c_{0,n}\left(A^n,B^{n-1}\right)\}\le \kappa ,\kappa \ge 0.$

The minimum excess distortion achievable by a nonanticipative code may be: D^o(n,ε,)inf{d:∀(n,d,ε,) nonanticipative code}.

E.2.2. Realization of a Reproduction Distribution of the Finite-Time RDF

The following description defines an example “realization” of the conditional distribution corresponding to the R_0,n^na(D) for a given source. FIG. 13 illustrates such a realization. As described further with reference to FIG. 12, this type of realization may be utilized for JSCC design of an {encoder, decoder} pair.

Given a nonanticipative code:

${\Sigma }_{\mathrm{JSCC}}\left({}^n,{}^n,{}^n,{}^n,P_{X^n},\left\{P_{B_iB^{i-1},A^i}\text{:}i=0,1,\dots ,n\right\},d_{0,n},c_{0,n}\right),$

a realization of the optimal reproduction distribution:

{P_Yi|Yi−1,Xi*(y_i|yⁱ⁻¹,xⁱ):i=0,1, . . . ,n}

corresponding to R_0,n^na(D) may be an {encoder, decoder} pair with the encoder used with or without feedback, such that

{right arrow over (P)} _{Y n |X n} *(yⁿ|xⁿ)=_i=0ⁿP_Yi|Yi−1,Xⁱ(dy_i|yⁱ⁻¹,xⁱ)

In such a case, the R_0,n^na(D) is “realizable,” because the realization operates with average distortion≦D.

In some implementations, given a system:

${\Sigma }_{\mathrm{JSCC}}\left({}^n,{}^n,{}^n,{}^n,P_{X^n},\left\{P_{B_iB^{i-1},A^i}\text{:}i=0,1,\dots ,n\right\},d_{0,n},c_{0,n}\right),$

multiple realization of the optimal reproduction distribution may exist. For example, in systems utilizing Gaussian sources and channels with memory, realization with feedback encoding and without feedback encoding may exist.

To identify the {encoder, decoder} pair that realizes the determined RDF and achieves the determined capacity, as discussed with reference to FIG. 12, a computing device or operator may identify the realization that satisfies

R _0,n ^na(D)=C_An→Bn^FB().

Then, for such a realization, a computing device and/or operator may compute the excess distortion probability is computed and the minimum excess distortion achievable. Because the RDFs and capacities may be determined (e.g., according to the example methods 300 and 600), the {encoder, decoder} pair follows from the above condition.E.3. Example JSCC Design for a Binary Symmetric Markov Source Over a Binary State Symmetric Channel with Memory

Although the following description provides an example design of an {encoder, decoder} pair according to JSCC for a specific type of source and channel, the techniques of the present disclosure may allow {encoder, decoder} pairs to be designed for other types of channels and/or sources. For example, the techniques disclosed herein may be utilized to generate {encoder, decoder} pairs for binary, symmetric, non-symmetric, Gaussian, non-Gaussian, stationary, non-stationary, etc. sources and/or channels. In fact, the realization of FIG. 10A corresponds to a JSCC design.

In the following example, an {encoder, decoder} pair, designed according to JSCC, realizes an RDF for a channel and operates at the capacity of the channel. Specifically, the example includes a binary symmetric Markov source (BSMS(p)) and a binary state symmetric channel (BSSC(α_i,β_i)). The example also utilizes a single letter Hamming definition of distortion and a single letter instantaneous cost function with and without feedback.

The nonanticipative RDF of the BSMS(p) may be developed from the following transition probabilities:

$P_{X_iX_{i-1}}\left(x_ix_{i-1}\right)=\left[\begin{array}{cc}1-p& p\\ p& 1-p\end{array}\right].$

Also, the single letter Hamming distortion criterion between the source symbol and the reproduction (e.g., decoded) symbol may be defined by:

ρ(x_i,y_i)=0 if x_i=y_i,

and 0 otherwise.

According to the definitions of nonanticipative RDF presented in section D, R^na(D) of a BSMS(p) with single letter Hamming distortion is given by:

$R^{\mathrm{na}}\left(D\right)=\{\begin{array}{cc}H\left(p\right)-\mathrm{mH}\left(\alpha \right)-\left(1-m\right)H\left(\beta \right)& \mathrm{if}D\le \frac{1}{2}\\ 0& \mathrm{otherwise}\end{array}$

where D is the average distortion,

$m=1-p-D+2\mathrm{pD},\alpha =\frac{\left(1-p\right)\left(1-D\right)}{1-p-D+2\mathrm{pD}},\beta =\frac{p\left(1-D\right)}{p+D-2\mathrm{pD}}.$

The optimal reproduction distribution may be given by:

$P_{Y_iX_i,Y_{i-1}}^{*}\left(y_ix_i,y_{i-1}\right)=\stackrel{\begin{array}{cccc}0,0& 0,1& 1,0& 1,1\end{array}}{\begin{array}{c}0\\ 1\end{array}\left[\begin{array}{cccc}\alpha & \beta & 1-\beta & 1-\alpha \\ 1-\alpha & 1-\beta & \beta & \alpha \end{array}\right]}.$

To determine the capacity of BSSC(α_i,β₁) with and without feedback, a computing device and or operator may, in this example, consider a special case of the unit memory channel of the same structure as the optimal reproduction distribution:

$P_{B_iA_i,B_{i-1}}\left(b_ia_i,b_{i-1}\right)=\begin{array}{c}0\\ 1\end{array}\stackrel{\begin{array}{cccc}0,0& 0,1& 1,0& 1,1\end{array}}{\left[\begin{array}{cccc}{\alpha }_1& {\beta }_1& 1-{\beta }_1& 1-{\alpha }_1\\ 1-{\alpha }_1& 1-{\beta }_1& {\beta }_1& {\alpha }_1\end{array}\right]}.$

Also, the state of a channel may be:

s _i a _i ⊕b _i−1.

The single letter cost function may be:

$\gamma \left(a_i,b_{i-1}\right)=\{\begin{array}{cc}1& \mathrm{if}a_i=b_{i-1},\left(s_i=0\right)\\ 0& \mathrm{if}a_i\ne b_{i-1},\left(s_i=1\right)\end{array}.$

Also, an average transmission cost is imposed, where the average transmission cost is defined by

$\frac{1}{n+1}\sum _{i=0}^n\left\{\gamma \left(a_i,b_{i-1}\right)\right\}=\kappa ,$

wire =constant.In this example, the average transmission cost at time i may be:

{γ(A_i,B_i−1)}=P_AiBi−1(0,0)+P_Ai,Bi−1(1,1)=P_Si(0).

According to the techniques discussed in section B, the capacity of the BSSC(α₁,β_i) with and without feedback and average transmission cost is equal and may be expressed as:

C _{A ∞ →B ∞} ^FB()=C_A∞;B∞^nfB()=H(λ)(1−))−−H(α₁)−(1−)H(β₁)

where

λ=α₁+(1=β₁).

The capacity achieving channel input distribution without feedback is:

$P_{A_i|A_{i-1}}^{*}\left(a_i|a_{i-1}\right)=\left[\begin{array}{cc}\frac{1-\kappa -\gamma }{1-2\gamma }& \frac{\kappa -\gamma }{1-2\gamma }\\ \frac{\kappa -\gamma }{1-2\gamma }& \frac{1-\kappa -\gamma }{1-2\gamma }\end{array}\right]$

where

γ=α₁+β₁(1−).

and the capacity achieving channel input distribution with feedback is:

$P_{A_i|B_{i-1}}^{*}\left(a_i|b_{i-1}\right)=\left[\begin{array}{cc}\kappa & 1-\kappa \\ 1-\kappa & \kappa \end{array}\right].$

Now that the capacity and RDF are determined for this example, a computing device and/or operator may identify or construct an {encoder, decoder} pair that realizes the determined RDF and achieves the determined capacity. FIG. 14 illustrates example realizations of the {encoder, decoder} pair.

For

=m, α₁=α and β₁=β

the {encoder, decoder} pair to be identified or constructed must satisfy:

C _{A ∞ →B ∞} ^FB()=H(p)−mH(α)−(1−m)H(β)=R^na(D)

For encoding without feedback (as shown in part (a) of FIG. 14), a parameter of the optimal channel input distribution may be equal to the parameter of the conditional distribution of the source:

$\left(\frac{1-\kappa -\gamma }{1-2\gamma }=p\right).$

In this example, the {encoder, decoder} pair satisfying the above condition may be the identity mapping on respective inputs. That is,

a _i =x _i ,y _i =b _i

or uncoded transmission is optimal. This result may imply that no encoding is performed and no decoding is performed. Also, a computing device and/or operator may evaluate the minimum excess distortion achievable. For encoding with feedback (shown in part (b) of FIG. 14), the {encoder, decoder} pair is given by:

a _i =x _i ⊕b _i−1 , y _i =b _i , i=0,1, . . .

Although this case is presented above by way of example, other {encoder, decoder} pairs can be computed using precisely the same methodology (e.g., by invoking the examples of optimal channel input distributions for channels with memory and feedback and transmission cost, and by using the expression of the finite time nonanticipative RDF).

F. Example Computing Device

FIG. 15 illustrates an example computing device 1650, which computing device 1650 may be implemented as the source 102, may implement the encoder 104 or the decoder 108, and/or may implement at least some of the functionality discussed with reference to FIGS. 3, 4, 5, 8, 9, and 12. The computing device 1650 may include one or more central processing units (CPUs) or processing units 1651 (may be called a microcontroller or a microprocessor), a system memory 1652a and 1652b, and a system bus 1654 that couples various system components including the system memory 1652 to the processing units 1651. The system bus 1654 may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus or a Mezzanine bus, and the Peripheral Component Interconnect Express (PCI-E) bus.

The computing device 1650 may include an assortment of computer-readable media. Computer-readable media may be any media that may be accessed by the computing device 1650. By way of example, and not limitation, the computer-readable media may include both volatile and nonvolatile media, removable and non-removable media. Media may also include computer storage media and communication media. The computer-readable media may store information such as computer-readable instructions, program modules, data structures, or other data. Computer-storage media may include non-transitory media, such as a RAM 1652b, a ROM 1652a, EEPROM, optical storage disks, magnetic storage devices, and any other non-transitory medium which may be used to store computer-accessible information.

In an embodiment, the ROM 1652a and/or the RAM 1652b may store instructions that are executable by the processing unit 1651. For example, a basic input/output system (BIOS), containing algorithms to transfer information between components within the computer 1650, may be stored in ROM 1652b. Data or program modules that are immediately accessible or are presently in use by the processing unit 1651 may be stored in RAM 1652a. Data normally stored in RAM 1652a while the computing device 1650 is in operation may include an operating system, application programs, program modules, and program data. In particular, the RAM 1652a may store one or more applications 1660 including one or more routines 1662, 1664, and 1666 implementing the functionality of the example methods 300, 400, 500, 800, 900, and 1200.

The computing device 1650 may also include other storage media such as a hard disk drive that may read from or write to non-removable, non-volatile magnetic media, a magnetic disk drive that reads from or writes to a removable, non-volatile magnetic disk, and an optical disk drive that reads from or writes to a removable, nonvolatile optical disk. Other storage media that may be used includes magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, and solid state ROM. The hard disk drive may be connected to the system bus 1654 through a non-removable memory interface such as interface 1674. A magnetic disk drive and optical disk drive may be connected to the system bus 1654 by a removable memory interface, such as interface 1690.

A user or operator may interact with the computing device 1650 through input devices such as a keyboard or a pointing device (i.e., a mouse). A user input interface 1702 may be coupled to the system bus 1654 to allow the input devices to communicate with the processing unit 1651. A display device such as a monitor 1722 may also be connected to the system bus 1654 via a video interface (not shown).

The computing device 1650 may operate in a networked environment using logical connections to one or more remote computing devices, for example. The remote computing device may be a personal computer (PC), a server, a router, or other common network node. The remote computing device typically includes many or all of the previously-described elements regarding the computing device 1650. Logical connections between the computing device 1650 and one or more remote computing devices may include a wide area network (WAN). A typical WAN is the Internet. When used in a WAN, the computing device 1650 may include a modem or other means for establishing communications over the WAN. The modem may be connected to the system bus 1654 via the network interface 1725, or other mechanism. In a networked environment, program modules depicted relative to the computing device 1650, may be stored in the remote memory storage device. As may be appreciated, other means of establishing a communications link between the computing device 1650 and a remote computing device may be used.

Additional Considerations

Upon reading this disclosure, those of ordinary skill in the art will appreciate still additional alternative structural and functional designs for characterizing channels and capacities, determining optimal input distributions, designing encoders and decoders, etc. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

The particular features, structures, or characteristics of any specific embodiment may be combined in any suitable manner and in any suitable combination with one or more other embodiments, including the use of selected features without corresponding use of other features. In addition, many modifications may be made to adapt a particular application, situation or material to the essential scope and spirit of the present invention. It is to be understood that other variations and modifications of the embodiments of the present invention described and illustrated herein are possible in light of the teachings herein and are to be considered part of the spirit and scope of the present invention. By way of example, and not limitation, the present disclosure contemplates at least the following aspects:

1. A method for characterizing a capacity of a channel with memory and feedback, the method comprising:

defining a channel model corresponding to the channel, wherein:the channel is utilized to transmit information from a source to a destination, andthe channel model indicates a dependence of outputs from the channel on past and present channel input symbols and on past channel output symbols;determining a representation of the capacity based on the channel model and based on a channel input distribution that achieves the capacity, wherein the representation represents the capacity for a finite number of transmissions over the channel, and wherein the representation includes an optimization; andsolving, by one or more processors of a specially configured computing device, the optimization of the representation of the capacity to determine the capacity of the channel for the finite number of transmissions.

2. The method of aspect 1, wherein determining the representation of the capacity includes:

determining, using stochastic optimal control techniques, a subset of distributions which include the channel input distribution that achieves the capacity.

3. The method of aspect 2, wherein the subset of distributions is a first subset of distributions, and wherein determining the representation of the capacity further includes:

determining, based on the first subset of distributions and using a variational equality of conditional mutual information or mutual information, an upper bound to identify a second subset of distributions which includes the channel input distribution that achieves the capacity, wherein the second subset of distributions is smaller than the first subset of distributions.

4. The method of any one of aspects 1 to 3, further comprising:

defining a transmission cost function, wherein the transmission cost function specifies a cost to transmit the information from the source to the destination and indicates a dependence on at least one of the past and present channel input symbols or the past channel output symbols,wherein determining the representation of the capacity includes determining the representation of the capacity based on the channel model and based on the transmission cost function.

5. The method of aspect 4, wherein the subset of distributions is a first subset of distributions, wherein determining the representation of the capacity further includes:

determining, using stochastic optimal control techniques, a subset of distributions which include the channel input distribution that achieves the capacity,determining, based on the channel model and the transmission cost function, if an output of the channel or the transmission cost function is dependent on quantities other than the past channel input symbols and the past channel output symbols,if the output of the channel or the transmission cost function is dependent on quantities other than the past channel input symbols and the past channel output symbols,determining, based on the first subset of distributions and using a variational equality of conditional mutual information, a second subset of distributions which include the optimal channel input distribution, wherein the second subset of distributions is smaller than the first subset of distributions, anddetermining the representation of the capacity based on the second subset of distributions, if the output of the channel or the transmission cost function is not dependent on quantities other than the past channel input symbols and the past channel output symbols, determining the representation of the capacity based on the first subset of distributions

6. The method of any one of aspects 1 to 5, wherein the representation of the capacity is a first representation of a finite-block length capacity, the method further comprising:

determining, based on the first representation of the finite-block length capacity, determining a second representation of the capacity, wherein the second representation is an upper bound on the first representation and represents the capacity for an infinite number of transmissions over the channel per unit time.

7. The method of any one of aspects 1 to 6, wherein the optimization is a maximization, and wherein solving the optimization of the representation of the capacity including solving the maximization.

8. The method of any one of aspects 1 to 7, wherein solving the optimization includes solving the optimization using a dynamic programming algorithm.

9. The method of any one of aspects 1 to 8, wherein solving the optimization includes solving the optimization using a Blahut-Arimoto algorithm sequentially.

10. The method of any one of aspects 1 to 9, wherein solving the optimization includes:

determining a gain or reduction in the capacity for encoding with feedback as compared to encoding without feedback.

11. The method of any one of aspects 1 to 10, further comprising:

designing, by an encoder design procedure, a coding scheme based on the capacity of the channel determined by solving the optimization of the representation of the capacity, wherein the coding scheme utilizes the channel input distribution that achieves the capacity, and wherein the coding scheme satisfies a condition specifying that none of the information is lost in the coding scheme.

12. The method of aspect 11, further comprising:

configuring an encoder coupled to the channel to encode based on the coding scheme that satisfies the condition specifying that none of the information is lost in the coding scheme.

13. A system including:

one or more processors; andone or more non-transitory memories,wherein the one or more non-transitory memories store computer-readable instructions that specifically configure the system such that, when executed by the one or more processors, the computer-readable instructions cause the system to:receive a channel model corresponding to the channel, wherein:the channel is utilized to transmit information from a source to a destination, andthe channel model indicates a dependence of outputs from the channel on past and present channel input symbols and on past channel output symbols;determine a representation of the capacity based on the channel model and based on a channel input distribution that achieves the capacity, wherein the representation represents the capacity for a finite number of transmissions over the channel, and wherein the representation includes an optimization; andsolve the optimization of the representation of the capacity to determine the capacity of the channel for the finite number of transmissions.

14. The system of aspect 13, wherein the computer-readable instructions further cause the system to:

determine a nonanticipative rate distortion function based on a received model of the source and based on a causal reproduction distribution, wherein the second representation and the third representation specify a rate at which symbols from the source should be transmitted to the destination.

15. The system of aspect 14, wherein the computer-readable instructions further cause the system to:

design a coding scheme based on the capacity of the channel determined by solving the optimization of the representation of the capacity and based on the nonanticipative rate distortion function, wherein the coding scheme utilizes the channel input distribution that achieves the capacity, and wherein the coding scheme satisfies a condition specifying that none of the information is lost in the coding scheme,configure an encoder coupled to the channel to encode based on the coding scheme, wherein the coding scheme simultaneously compresses and encodes information transmitted over the channel.

16. The system of aspect 15, wherein designing the coding scheme includes designing the coding scheme by joint source channel coding.

17. The system of any one of aspects 13 to 16, wherein determining the representation of the capacity includes:

determining, using stochastic optimal control techniques, a subset of distributions which include the channel input distribution that achieves the capacity.

18. The system of aspect 17, wherein the subset of distributions is a first subset of distributions, and wherein determining the representation of the capacity further includes:

determining, based on the first subset of distributions and using a variational equality of conditional mutual information or mutual information, an upper bound to identify a second subset of distributions which includes the channel input distribution that achieves the capacity, wherein the second subset of distributions is smaller than the first subset of distributions.

19. The system of any one of aspects 13 to 18, wherein the representation of the capacity is a first representation of a finite-block length capacity, and wherein the computer-readable instructions further cause the system to:

determine, based on the first representation of the finite-block length capacity, determining a second representation of the capacity, wherein the second representation is an upper bound on the first representation and represents the capacity for an infinite number of transmissions over the channel per unit time.

20. The system of any one of aspects 13 to 19, wherein the channel model is a probabilistic map.

21. The system of any one of aspects 13 to 20, wherein the channel model is a function of the past and present channel input symbols.

Claims

1. A method for characterizing a capacity of a channel with memory and feedback, the method comprising: defining a channel model corresponding to the channel, wherein: the channel is utilized to transmit information from a source to a destination, andthe channel model indicates a dependence of outputs from the channel on past and present channel input symbols and on past channel output symbols;determining a representation of the capacity based on the channel model and based on a channel input distribution that achieves the capacity, wherein the representation represents the capacity for a finite number of transmissions over the channel, and wherein the representation includes an optimization; andsolving, by one or more processors of a specially configured computing device, the optimization of the representation of the capacity to determine the capacity of the channel for the finite number of transmissions and a per unit time limit of the capacity of the channel.

2. The method of claim 1, wherein determining the representation of the capacity includes: determining, using stochastic optimal control techniques, a subset of distributions which include the channel input distribution that achieves the capacity.

3. The method of claim 2, wherein the subset of distributions is a first subset of distributions, and wherein determining the representation of the capacity further includes: determining, based on the first subset of distributions and using a variational equality of conditional mutual information or mutual information, an upper bound to identify a second subset of distributions which includes the channel input distribution that achieves the capacity, wherein the second subset of distributions is smaller than the first subset of distributions.

4. The method of claim 1, further comprising: defining a transmission cost function, wherein the transmission cost function specifies a cost to transmit the information from the source to the destination and indicates a dependence on the past and present channel input symbols and the past channel output symbols,wherein determining the representation of the capacity includes determining the representation of the capacity based on the channel model and based on the transmission cost function.

5. The method of claim 4, wherein the subset of distributions is a first subset of distributions, wherein determining the representation of the capacity further includes: determining, using stochastic optimal control techniques, a subset of distributions which include the channel input distribution that achieves the capacity,determining, based on the channel model and the transmission cost function, if an output of the channel or the transmission cost function is dependent on quantities other than the past channel input symbols and the past channel output symbols,if the output of the channel or the transmission cost function is dependent on quantities other than the past channel input symbols and the past channel output symbols, determining, based on the first subset of distributions and using a variational equality of conditional mutual information or mutual information or mutual information, a second subset of distributions which include the optimal channel input distribution, wherein the second subset of distributions is smaller than the first subset of distributions, anddetermining the representation of the capacity based on the second subset of distributions,if the output of the channel or the transmission cost function is not dependent on quantities other than the past channel input symbols and the past channel output symbols, determining the representation of the capacity based on the first subset of distributions

6. The method of claim 1, wherein the representation of the capacity is a first representation of a finite-block length capacity, the method further comprising: determining, based on the first representation of the finite-block length capacity, determining a second representation of the capacity, wherein the second representation is an upper bound on the first representation and represents the capacity for an infinite number of transmissions over the channel per unit time.

7. The method of claim 1, wherein the optimization is a maximization, and wherein solving the optimization of the representation of the capacity including solving the maximization.

8. The method of claim 1, wherein solving the optimization includes solving the optimization using a dynamic programming algorithm.

9. The method of claim 1, wherein solving the optimization includes one of solving the optimization independently using a Blahut-Arimoto algorithm sequentially or solving the optimization using the Blahut-Arimoto algorithm in combination with a dynamic programming algorithm.

10. The method of claim 1, wherein solving the optimization includes: determining a gain in the capacity for encoding with feedback as compared to encoding without feedback.

11. The method of claim 1, further comprising: designing, by an encoder design procedure, a coding scheme based on the capacity of the channel determined by solving the optimization of the representation of the capacity, wherein the coding scheme utilizes the channel input distribution that achieves the capacity, and wherein the coding scheme satisfies a condition specifying that none of the information is lost in the coding scheme.

12. The method of claim 11, further comprising: configuring an encoder coupled to the channel to encode based on the coding scheme that satisfies the condition specifying that none of the information is lost in the coding scheme.

13. A system comprising: one or more processors; andone or more non-transitory memories,wherein the one or more non-transitory memories store computer-readable instructions that specifically configure the system such that, when executed by the one or more processors, the computer-readable instructions cause the system to: receive a channel model corresponding to the channel, wherein: the channel is utilized to transmit information from a source to a destination, andthe channel model indicates a dependence of outputs from the channel on past and present channel input symbols and on past channel output symbols;determine a representation of the capacity based on the channel model and based on a channel input distribution that achieves the capacity, wherein the representation represents the capacity for a finite number of transmissions over the channel, and wherein the representation includes an optimization; andsolve the optimization of the representation of the capacity to determine the capacity of the channel for the finite number of transmissions.

14. The system of claim 13, wherein the computer-readable instructions further cause the system to: determine a nonanticipative rate distortion function based on a received model of the source and based on a causal reproduction distribution, wherein the second representation and the third representation specify a rate at which symbols from the source should be transmitted to the destination.

15. The system of claim 14, wherein the computer-readable instructions further cause the system to: design a coding scheme based on the capacity of the channel determined by solving the optimization of the representation of the capacity and based on the nonanticipative rate distortion function, wherein the coding scheme utilizes the channel input distribution that achieves the capacity, and wherein the coding scheme satisfies a condition specifying that none of the information is lost in the coding scheme,configure an encoder coupled to the channel to encode based on the coding scheme, wherein the coding scheme simultaneously compresses and encodes information transmitted over the channel.

16. The system of claim 15, wherein designing the coding scheme includes designing the coding scheme by joint source channel coding.

17. The system of claim 13, wherein determining the representation of the capacity includes: determining, using stochastic optimal control techniques, a subset of distributions which include the channel input distribution that achieves the capacity.

18. The system of claim 17, wherein the subset of distributions is a first subset of distributions, and wherein determining the representation of the capacity further includes: determining, based on the first subset of distributions and using a variational equality of conditional mutual information or mutual information, an upper bound to identify a second subset of distributions which includes the channel input distribution that achieves the capacity, wherein the second subset of distributions is smaller than the first subset of distributions.

19. The system of claim 13, wherein the representation of the capacity is a first representation of a finite-block length capacity, and wherein the computer-readable instructions further cause the system to: determine, based on the first representation of the finite-block length capacity, determining a second representation of the capacity, wherein the second representation is an upper bound on the first representation and represents the capacity for an infinite number of transmissions over the channel per unit time.

20. The system of claim 13, wherein the channel model is a probabilistic map.

21. The system of claim 13, wherein the channel model is a function of the past and present channel input symbols.