In order to improve the least mean squares (LMS) adaptation algorithm to accommodate the nonlinear transfer function, and to adjust the coefficients of adaptive filter during the actual implement of bias voltage and signal amplitude, methods are proposed and simulated to develop a nonlinear adaptive filter. The inputs to LMS are replaced by the derivatives of traditional inputs, and the step for each training iteration is adaptively controlled by the difference between target signal and actual signal. The simulation utilizes the implementation of Nyquist pulses optical sampling and works as a digital signal processing pre-compensation to reduce influence of the frequency responses on wires and devices. The simulation result shows promising improvement with the modified adaptation algorithm method in tackling Mach Zehnder modulator's non-monotonic transfer function.