The diffusion of nonlinear loads and power electronic devices in power systems deteriorates the signal environment and increases the difficulty of measuring harmonic phasors. Considering accurate harmonic phasor information is necessary to deal with harmonic-related issues, this paper focuses on realizing accurate dynamic harmonic phasor estimation when the signal is contaminated by certain interharmonic tones, i.e. the out-of-band interference (OBI). Specifically, this work introduces the singular value decomposition into the Taylor-Fourier transform algorithm, deriving a general decomposition form for harmonic phasor filters, and yielding a set of adjustable parameters. Then these parameters are configured to minimize the negative influence of OBI on the dynamic harmonic phasor filters. Based on the recommended parameter values, the optimized harmonic estimator is obtained, and then tested under signals distorted by OBI, harmonic, noise, and in worse cases, frequency deviation, and some dynamic conditions. Test results show the proposal achieves good harmonic phasor estimation even under multiple interferences and dynamic scenarios, and has a much weaker dependence on OBI tones compared to conventional approaches.