UNCERTAINTY IN ITC
What is uncertainty in information theory and coding ?
In information theory, uncertainty is a measure of the amount of randomness or entropy in a signal or message. It is a way to quantify the amount of information contained in a signal or message. In other words, it is a measure of how much information is needed to specify the state of a system.
Shannon introduced the concept of entropy as a measure of uncertainty in his 1948 paper "A Mathematical Theory of Communication". He defined the entropy of a discrete random variable X as:
H(X) = - ∑ p(x) log2 p(x)
where p(x) is the probability of the variable X taking on the value x.
In coding theory, uncertainty can refer to the amount of noise or errors that are present in a communication channel. The goal of channel coding is to design codes that can correct errors that occur during transmission, and thus reduce the uncertainty of the message received at the receiver.
One way to achieve this is by using Error correcting code. Error correcting codes are used to detect and correct errors that occur during the transmission of a message. They add extra redundancy to the message, which allows the receiver to detect and correct errors that occur during transmission.
The goal of coding theory is to design codes that can efficiently represent and transmit information in the presence of uncertainty. By using the right coding techniques, it is possible to transmit messages over noisy channels with high reliability
What is Information and entropy ?
Information is a concept that refers to the amount of knowledge or data that is contained in a message or signal. It is a way to quantify the amount of meaningful content in a message, and can be thought of as the opposite of noise or randomness.
Entropy, in information theory, is a measure of the amount of uncertainty or randomness in a signal or message. It was introduced by Claude Shannon in 1948 as a way to quantify the amount of information contained in a signal or message.
Shannon defined the entropy of a discrete random variable X as:
H(X) = - ∑ p(x) log2 p(x)
where p(x) is the probability of the variable X taking on the value x.
Entropy can be thought of as a measure of the amount of information needed to specify the state of a system. A message with high entropy contains a lot of randomness and thus requires a lot of information to specify, whereas a message with low entropy is more predictable and requires less information to specify.
In information theory, the entropy of a message is a measure of the amount of information it contains. The greater the entropy, the more information the message contains. Entropy is also closely related to the concept of data compression, which is the process of reducing the size of a file without losing any information. The more random or unpredictable the data, the more difficult it is to compress, thus entropy plays an important role in compression algorithms.
