Drowsy driving is a major cause of on-road accidents in the US, which sometimes is fatal to unsuspecting pedestrians. This framework based on Deep Learning proposes an approach to detect the onset of drowsiness in a vehicle operator especially alerting the driver when in the proximity of a pedestrian. Using Convolutional Neural Network (CNN), an approach is proposed to detect drowsiness based on the Viola-Jones algorithm. The pedestrian detector is also based on a deep CNN architecture and is capable to detect multiple pedestrians. In the end, an integration of the output from the two architectures is fed into an Arduino hardware kit to generate warnings for the vehicle operator.