This study proposes an innovative explainable process prediction solution to facilitate the data-driven decision making for process planning in manufacturing. After integrating the top-floor and shop-floor data obtained from various enterprise information systems especially from Manufacturing Execution Systems, a deep neural network was applied to predict the process outcomes. Since we aim to operationalize the delivered predictive insights by embedding them into decision making processes, it is essential to generate the relevant explanations for domain experts. To this end, two local post-hoc explanation approaches, Shapley Values and Individual Conditional Expectation (ICE) plots, are applied which are expected to enhance the decision-making capabilities by enabling experts to examine explanations from different perspectives. After assessing the predictive strength of the adopted deep neural networks with relevant binary classification evaluation measures, a discussion of the generated explanations is provided. Lastly, a brief discussion of ongoing activities in the scope of current emerging application and some aspects of future implementation plan concludes the study.