In this study, an algorithm to blind and automatic modulation classification has been proposed. It well benefits combined machine leaning and signal feature extraction to recognize diverse range of modulation in low signal power to noise ratio (SNR). The presented algorithm contains four. First, it advantages spectrum analyzing to branching modulated signal based on regular and irregular spectrum character. Seconds, a nonlinear soft margin support vector (NS SVM) problem is applied to received signal, and its symbols are classified to correct and incorrect (support vectors) symbols. The NS SVM employment leads to discounting in physical layer noise effect on modulated signal. After that, a k-center clustering can find center of each class. finally, in correlation function estimation of scatter diagram is correlated with pre-saved ideal scatter diagram of modulations. The correlation outcome is classification result. For more evaluation, success rate, performance, and complexity in compare to many published methods are provided. The simulation prove that the proposed algorithm can classified the modulated signal in less SNR. For example, it can recognize 4-QAM in SNR=-4.2 dB, and 4-FSK in SNR=2.1 dB with %99 success rate. Moreover, due to using of kernel function in dual problem of NS SVM and feature base function, the proposed algorithm has low complexity and simple implementation in practical issues.