information theory [a07603af]
1. papers that measure information flow
1. papers that measure information flow
1.1. An Information-theoretic Visual Analysis Framework for Convolutional Neural Networks
1.1. An Information-theoretic Visual Analysis Framework for Convolutional Neural Networks
-
Uses CNN's, but measures entropy directly instead of trying to measure Mutual Information
1.2. Information flows of diverse autoencoders
1.2. Information flows of diverse autoencoders
1.3. Generalization Bounds for Deep learning
1.3. Generalization Bounds for Deep learning
2. Information Foraging Theory for Programmers [79351c57]
2. Information Foraging Theory for Programmers [79351c57]
-
Programmers "seek" out different type of information "diets"
- web.eecs.utk.edu/~azh/blog/informationforaging.html
- These can be modeled in some form
- alexanderell.is/posts/visualizing-code/
2.1. Information foraging for religion
2.1. Information foraging for religion
- Most people go to majlis for information (and need to therefore find religious authorities convincing)
3. Mutual Information [a79368f7]
3. Mutual Information [a79368f7]
- en.wikipedia.org/wiki/Mutual_information
- www.youtube.com/watch?v=U9h1xkNELvY
-
arxiv.org/pdf/1905.06922.pdf <- variational bounds
- ON NETWORK SCIENCE AND MUTUAL INFORMATION FOR EXPLAINING DEEP NEURAL NETWORKS - arxiv.org/pdf/1901.08557.pdf
- mutual information neural estimation - arxiv.org/pdf/1801.04062.pdf
4. Algorithmic Information Theory
4. Algorithmic Information Theory
- en.wikipedia.org/wiki/Algorithmic_information_theory#cite_note-2
- www.cs.auckland.ac.nz/research/groups/CDMTCS/docs/ait.php
- dl.acm.org/doi/10.1145/321892.321894
4.2. Solomonoff Theory of Inductive Inference
4.2. Solomonoff Theory of Inductive Inference
5. Information Bottleneck [6cde88fd]
5. Information Bottleneck [6cde88fd]
- related to Scaling laws for solution compressibility
- www.youtube.com/watch?v=RKvS958AqGY&t=2249s
- www.youtube.com/watch?v=bLqJHjXihK8&t=1482s
5.1. Predictive information in RNN's
5.1. Predictive information in RNN's
5.2. Information Bottleneck Theory Based Exploration of Cascade Learning
5.2. Information Bottleneck Theory Based Exploration of Cascade Learning
5.3. Information bottleneck thesis:
5.3. Information bottleneck thesis:
- dial.uclouvain.be/downloader/downloader.php?pid=thesis%3A33185&datastream=PDF_01&cover=cover-mem
6. Information Theory Course
6. Information Theory Course
6.1. Chapter 1
6.1. Chapter 1
- "The Mathematical Theory of Communication"
-
Fundalmental limits of communication
-
information is uncertainty -> information is modeled as a random variable
- uncertainty with information source, aka the information source is noisy
- information is digital and can be modeled as bits
-
-
two fundalmental theorems
-
source coding theorem: establishes fundalmental limits in data compression
- there is always a minimum size that a file can be compressed
-
channel coding theorem: fundalmental limit for reliable communication through a noisy channel
- also called "channel capacity"
-