Information Theory and Coding: Introduction
Information theory is theoretical part of communication developed by American mathematician Shannon.
It deals with mathematical modelling and analysis of a communication system rather than dealing with physical sources (camera, keyboard, microphone etc.) and physical channels (wires, cables, fibre, satellite, radio etc).
Note: Lot of mathematics is involved in manufacturing of things we use in daily-life… like, manufacturing of pen, paper, flash memory, mobiles, antenna design, garment manufacturing ete etc etc
Information theory addresses and answers below fundamental questions of communication theory:
- Data compression
- Data transmission
- Data storage
- Error detection and correction codes
- Ultimate Data rate over a noisy channel (for reliable communication)
Every morning one reads newspaper to receive information. A message is said to convey information, if two key elements are present in it.
- Change in knowledge or meaning
- Uncertainty (unpredictability)
Amount of information contained in a message is inter-related to its probability of occurrence. Information is always about something (occurrence of an event etc.). It may be a true or lie.
Consider the following messages:
- The sun rises in the east
- Gopal will get Nobel prize in physics
- Scattered rain
- Cyclone storm
- Son is born to his wife who is a mother of two daughters
- COVID-19 pandemic disease
By intuition, understand that the above 6 messages carry different information.
| S.No. | Message | Remark |
| 1 | The sun rises in the east | There is little information in this statement because everybody can expect this event. |
| 2 | Gopal will get Nobel prize in physics | This message will contain lot of information. Because it is not frequent. It is a rare event, so lot of information. |
| 3 | Scattered rain | Rare event, so lot of information |
| 4 | Cyclone storm | Rare event, so lot of information |
| 5 | Son is born to his wife who is a mother of two daughters | Rare event, so lot of information |
| 6 | COVID-19 pandemic disease | Rare event, so lot of information |
- Message with lower probability contains higher information content and vice-versa
- Frequent events have less information
- Rare events have more information
- Note that greater the uncertainty (unpredictable), higher the value of information
- Certain events (sure events) carry least information
- Uncertain events (unsure events) carry least information
The messages in the table above have different probabilities of occurrence and hence contain different value of information.
Information theory talks about:
- How to measure the amount of information?
- How to measure the correctness of information?
- What to do if information gets corrupted by errors?
- How much memory does it require to store information?
