This project explores the development of a self-driving car system within the video game Grand Theft Auto V using a deep learning approach. Leveraging EfficientNetV2 with LSTM and Transformer encoders, the model was trained on gameplay image sequences and optimized through transfer learning. A GUI developed with Gradio allows users to visualize predictions and model behavior in real time. The model achieved over 70% accuracy on highway scenarios under clear daytime conditions. The system serves as a simulation platform for research, training, and educational purposes, demonstrating the feasibility of using video games to safely and cost-effectively test autonomous driving algorithms.
The self-driving simulation system demonstrates the potential of combining deep learning with gaming platforms for low-risk, low-cost testing of autonomous driving models. With transfer learning and optimized architecture configurations, the system achieves notable performance. The GUI extends accessibility to users lacking high-end hardware for real-time evaluation. However, real-world generalization requires additional sensors, richer datasets, and environmental modeling.
The trained model is constrained by the synthetic nature of the gaming environment and lacks exposure to complex real-world driving conditions. It cannot detect road compliance elements such as traffic lights or weather-based dynamics. Additionally, evaluation accuracy is bounded by the resolution and scope of the collected dataset.
Future extensions include expanding model capabilities to detect traffic signs, handle adverse conditions, and integrate real-time sensor fusion. Enhanced realism via physics-based simulators and integration with multi-modal sensor data will be explored. Deployment to other gaming environments and online collaborative simulations is also envisioned.