# information theory

Tags: computers, math

## papers that measure information flow

### An Information-theoretic Visual Analysis Framework for Convolutional Neural Networks

• Uses CNN’s, but measures entropy directly instead of trying to measure Mutual Information





## Algorithmic Information Theory

### Information-Theoretic Probing with MDL

https://arxiv.org/abs/2003.12298

## Information Theory Course

### Chapter 1

• “The Mathematical Theory of Communication”
• Fundalmental limits of communication
• information is uncertainty -> information is modeled as a random variable
• uncertainty with information source, aka the information source is noisy
• information is digital and can be modeled as bits
• two fundalmental theorems
• source coding theorem: establishes fundalmental limits in data compression
• there is always a minimum size that a file can be compressed
• channel coding theorem: fundalmental limit for reliable communication through a noisy channel
• also called “channel capacity”