clustering [49986cdc]
1. NVDM
1. NVDM
- varational autoencoder with BOW inputs
- Word order is ignored, only word counts matter
- L1 normalized counts: word probability distribution
2. Transformer
2. Transformer
- pretrained LM, fine tuned on some tasks
-
compute loss with MSE or contrastive
- contrastive learning gives anchor, and a negative one that's dissimilar
2.1. RoBERTa
2.1. RoBERTa
- masked langugage model
2.2. MPNet
2.2. MPNet
- Combines both approaches of permuted language models and masked language models
- Sequence is permuted and last tokens are masked
2.3. MiniLM
2.3. MiniLM
- distillation
- Teacher model teaches a student model
- all-mpnet-base-v2 teacher
- all-miniLM-l6-v2 is 5 times faster
3. Benchmark Dataset
3. Benchmark Dataset
3.1. Multi-News
3.1. Multi-News
- summary dataset that's has custom human written summaries
4. Metrics
4. Metrics
4.1. Accuracy
4.1. Accuracy
- Cosine similarity
- each news story should be closer to its summary than any other summary
- Use AUC to determine how good is the classifier
4.2. Speed
4.2. Speed
-
NVDM
- is NVDM actually fast? Tested on batch sizes
- NVDM is actually not that fast, for small batch sizes is pretty slow, only catches up much later