Sutton: The Bitter Lesson

Tags: articles, hardware, ml

• http://www.incompleteideas.net/IncIdeas/BitterLesson.html
• Similar to Hooker - The Hardware Lottery
• makes the point that “compute >> clever”
• https://papers.nips.cc/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf

Responses

• https://staff.fnwi.uva.nl/m.welling/wp-content/uploads/Model-versus-Data-AI-1.pdf
• https://rodneybrooks.com/a-better-lesson/

Discussion

• Role of open source
• Rise of intermediate representation
  • abstract interpretation?
• active learning and weak supervision
• malloc and fpga
  • performance constraints of cache misses?
  • cache analysis?
• compilers
  • text/stack/heap
  • isa maps to blocks of code
  • profile guided optimizations to burn in your hot spots
• what other types of machine learning are being foreclosed because we don’t have the hardware?
  • what can we do?
    • can we invest in IR’s?
      • FPGA’s, ML IR, compiler IR’s
    • reconfigurable network in datacenters
• capsul neural networks
• log file system (ssds)
• polynomial error correction -> needs eponetnial speedup