Traditional recursive least square (RLS) adaptive filtering is widely used to estimate the impulse responses (IR) of an unknown system. Nevertheless, the RLS estimator shows poor performance when tracking rapidly time-varying systems. In this paper, we propose a multi-layered RLS (m-RLS) estimator to address this concern. The m-RLS estimator is composed of multiple RLS estimators, each of which is employed to estimate and eliminate the misadjustment of the previous layer. It is shown that the mean square error (MSE) of the m-RLS estimate can be minimized by selecting the optimum number of layers. We provide a method to determine the optimum number of layers. A low-complexity implementation of m-RLS is discussed and it is indicated that the complexity order of the proposed estimator can be reduced to O(M), where M is the IR length. In addition, by performing simulations, we show that m-RLS outperforms the classic RLS and the RLS methods with a variable forgetting factor.