Skin Segmentation is widely used in biometric applications such as face detection, face recognition, face tracking, and hand gesture recognition. However, several challenges such as nonlinear illumination, equipment effects, personal interferences, ethnicity variations, etc., are involved in detection process that result in the inefficiency of color based methods. Even though many ideas have already been proposed, the problem has not been satisfactorily solved yet. This paper introduces a technique that addresses some limitations of the previous works. The proposed algorithm consists of three main steps including initial seed generation of skin map, Otsu segmentation in color images, and finally a two-stage diffusion. The initial seed of skin pixels is provided based on the idea of ternary image as there are certain pixels in images which are associated to human complexion with very high probability. The Otsu segmentation is performed on several color channels in order to identify homogeneous regions. The result accompanying with the edge map of the image is utilized in two consecutive diffusion steps in order to annex initially unidentified skin pixels to the seed. Both quantitative and qualitative results demonstrate the effectiveness of the proposed system in compare with the state-of-the-art works.