On shrinkage and selection: ANOVA model

Abstract

This paper considers the estimation of the parameters of an ANOVA model
when sparsity is suspected. Accordingly, we consider the least square estima-
tor (LSE), restricted LSE, preliminary test and Stein-type estimators, together
with three penalty estimators, namely, the ridge estimator, subset selection rules
(hard threshold estimator) and the LASSO (soft threshold estimator). We com-
pare and contrast the L2-risk of all the estimators with the lower bound of L2-
risk of LASSO in a family of diagonal projection scheme which is also the lower
bound of the exact L2-risk of LASSO. The result of this comparison is that nei-
ther LASSO nor the LSE, preliminary test, and Stein-type estimators outperform
each other uniformly. However, when the model is sparse, LASSO outperforms
all estimators except “ridge” estimator since both LASSO and ridge are L2-risk
equivalent under sparsity. We also nd that LASSO and the restricted LSE are
L2-risk equivalent and both outperform all estimators (except ridge) depending
on the dimension of sparsity. Finally, ridge estimator outperforms all estimators
uniformly. Our nding are based on L2-risk of estimators and lower bound of the
risk of LASSO together with tables of efficiency and graphical display of efficiency
and not based on simulation.