In this work, we implemented a fatigue detection system using machine learning and evaluated its performance. The proposed system is mainly based on the Viola-Jones face detection algorithm and the convolutional neural network (CNN). Viola-Jones object detection framework is mainly focused on the detection of the face and facial features. CNN is extended to DenseNets and made fully convolution to tackle the problem semantic image segmentation. The main idea behind the DenseNets is to capture the dense blocks that perform iterative concatenation of feature maps. The proposed system is implemented on many different video sequences and observed that its average accuracy is 99.18% and the detection rate is 99.71% out of approximately 100 image frames. The system shows high accuracy in segmentation, low error rate, and quick processing of input data distinguishes from the existing similar systems. Finally, if this system is implemented, it can minimize the number of accidents caused by drivers' fatigue.