Zwet and Cator: The Significance Filter, the Winner's Curse, and the Need to Shrink
- Zwet, Erik van, and Eric Cator. “The Significance Filter, the Winner’s Curse and the Need to Shrink.” ArXiv:2009.09440 [Stat], September 20, 2020. http://arxiv.org/abs/2009.09440.
“Signficance filter” means only focusing on statistically significant results, which means that it can lead to an overestimation of the magnitude of the parameter.
Attempts to study this from a bayesian perspective, shows that the
the relativebias of the magnitude is a decreasing function of the power and thatthe usual confidence interval undercovers when the power is less than50%. We conclude that failure to apply the appropriate amount ofshrinkage can lead to misleading inferences.
Basically only focusing on statistically significant data means that the intervals are poorly estimated, since the significance filter is a step function