عجفت الغور

information theory

Tags: computers, math

papers that measure information flow

An Information-theoretic Visual Analysis Framework for Convolutional Neural Networks

  • Uses CNN’s, but measures entropy directly instead of trying to measure Mutual Information

Information flows of diverse autoencoders

Generalization Bounds for Deep learning

Information Foraging Theory for Programmers

Mutual Information

Algorithmic Information Theory

Information-Theoretic Probing with MDL

https://arxiv.org/abs/2003.12298

Solomonoff Theory of Inductive Inference

Information Bottleneck

Predictive information in RNN’s

Information Bottleneck Theory Based Exploration of Cascade Learning

Information bottleneck thesis:

Information Theory Course

Chapter 1

  • “The Mathematical Theory of Communication”
  • Fundalmental limits of communication
    • information is uncertainty -> information is modeled as a random variable
      • uncertainty with information source, aka the information source is noisy
    • information is digital and can be modeled as bits
  • two fundalmental theorems
    • source coding theorem: establishes fundalmental limits in data compression
      • there is always a minimum size that a file can be compressed
    • channel coding theorem: fundalmental limit for reliable communication through a noisy channel
      • also called “channel capacity”