In this study, we propose a novel machine learning based algorithm to improve the performance of beyond 5 generation (B5G) wireless communication system that is assisted by Orthogonal Frequency Division Multiplexing (OFDM) and Non-Orthogonal Multiple Access (NOMA) techniques. The non-linear soft margin support vector machine (SVM) problem is used to provide an automatic modulation classifier (AMC) and a signal power to noise and interference ratio (SINR) estimator. The estimation results of AMC and SINR are used to reassign the modulation type, codding rate, and transmit power through frames of eNode B connections. The AMC success rate versus SINR, total power consuming, and sum capacity are evaluated for OFDM-NOMA assisted 5G system. Results show improvement of success rate compared of some published method. Furthermore, the algorithm directly computes SINR after signal is detected by successive interference cancellation (SIC) and before any signal decoding. Moreover, because of the direct sense of physical channel, the presented algorithm can discount occupied symbols (overhead signaling) for channel quality information (CQI) in network communication signaling. The results also prove that the proposed algorithm reduces the total power consumption and increases the sum capacity through the eNode B connections. Simulation results in compare to other algorithms show more successful AMC, efficient SINR estimator, easier practical implantation, less overhead signaling, less power consumption, and more capacity achievement.