The growing population in large cities is creating traffic management issues. The metropolis road network management also requires constant monitoring, timely expansion, and modernization. In order to handle road traffic issues, an intelligent traffic management solution is required. Intelligent monitoring of traffic involves the detection and tracking of vehicles on roads and highways. There are various sensors for collecting motion information, such as transport video detectors, microwave radars, infrared sensors, ultrasonic sensors, passive acoustic sensors, and others. In this paper, we present an intelligent video surveillance-based vehicle tracking system. The proposed system uses a combination of the neural network, image-based tracking, and You Only Look Once (YOLOv3) to track vehicles.We train the proposed system with different datasets. Moreover, we use real video sequences of road traffic to test the performance of the proposed system. The evaluation outcomes showed that the proposed system can detect, track, and count the vehicles with acceptable results in changing scenarios.