Scaled YOLO v4 is the best neural network for object detection on MS COCO dataset
- Pytorch: YOLOv4-CSP, YOLOv4-P5, YOLOv4-P6, YOLOv4-P7 (main repository — use to reproduce results): https://github.com/WongKinYiu/ScaledYOLOv4
- Darknet: YOLOv4-tiny, YOLOv4-CSP, YOLOv4x-MISH: https://github.com/AlexeyAB/darknet
- Structure of YOLOv4-CSP: https://netron.app/?url=https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-csp.cfg
Scaled YOLO v4 is the best neural network for object detection — the most accurate (55.8% AP Microsoft COCO test-dev) among neural network published. In addition, it is the best in terms of the ratio of speed to accuracy in the entire range of accuracy and speed from 15 FPS to 1774 FPS. Now it is the Top1 neural network for object detection.
Scaled YOLO v4 outperforms neural networks in accuracy:
- Google EfficientDet D7x / DetectoRS or SpineNet-190(self-trained on extra-data)
- Amazon Cascade-RCNN ResNest200
- Microsoft RepPoints v2
- Facebook RetinaNet SpineNet-190
We show that YOLO and Cross-Stage-Partial (CSP) Network approaches are the best in terms of both absolute accuracy and accuracy-to-speed ratio.
Even at lower network resolution, Scaled-YOLOv4-P6 (1280x1280) 30 FPS — 54.3% AP is slightly more accurate and 3.7x faster than EfficientDetD7 (1536x1536) 8.2 FPS — 53.7% AP.
Scaled YOLO v4 lies on the Pareto optimality curve — no matter what other neural network you take, there is always such a YOLOv4 network, which is either more accurate at the same speed, or faster with the same accuracy, i.e. YOLOv4 is the best in terms of speed and accuracy.
Scaled YOLOv4 is more accurate and faster than neural networks:
- Google EfficientDet D0-D7x
- Google SpineNet S49s — S143
- Baidu Paddle-Paddle PP YOLO
- And many others…
Scaled YOLO v4 is a series of neural networks built on top of the improved and scaled YOLOv4 network. Our neural network was trained from scratch without using pre-trained weights (Imagenet or any other).
The YOLOv4-tiny neural network speed reaches 1774 FPS on a gaming graphics card GPU RTX 2080Ti when using TensorRT + tkDNN (batch = 4, FP16): https://github.com/ceccocats/tkDNN
Scaled YOLOv4 utilizes massively parallel devices such as GPUs much more efficiently than EfficientDet. For example, GPU V100 (Volta) has performance: 14 TFLops — 112 TFLops-Tensor-Cores https://images.nvidia.com/content/technologies/volta/pdf/tesla-volta-v100-datasheet-letter-fnl-web.pdf
If we test both models on GPU V100 with
- batch = 1
- -hparams = mixed_precision = true
- And without -tensorrt = FP32
- YOLOv4-CSP (640x640) — 47.5% AP — 70 FPS — 120 BFlops (60 FMA) Based on BFlops, it should be 933 FPS = (112,000 / 120), but in fact we get 70 FPS, i.e. 7.5% GPU used = (70/933)
- EfficientDetD3 (896x896) — 47.5% AP — 36 FPS — 50 BFlops (25 FMA) Based on BFlops, it should be 2240 FPS = (112,000 / 50), but in fact we get 36 FPS, i.e. 1.6% GPU used = (36/2240)
Those. efficiency of computing operations on devices with massive parallel computing such as GPUs used in YOLOv4-CSP (7.5 / 1.6) = 4.7x better than the efficiency of operations used in EfficientDetD3.
Usually, neural networks are run on the CPU only in research tasks for easier debugging, and the BFlops characteristic is currently only of academic interest. In real-world tasks, real speed and accuracy are important. The real speed of YOLOv4-P6 is 3.7x faster than EfficientDetD7 on GPU V100. Therefore, devices with massive parallelism GPU / NPU / TPU / DSP with much more optimal speed, price and heat dissipation are almost always used:
- Embedded GPU (Jetson Nano/Nx)
- Mobile-GPU/NPU/DSP (Bionic-NPU/Snapdragon-DSP/Mediatek-APU/Kirin-NPU/Exynos-GPU/…)
- TPU-Edge (Google Coral/Intel Myriad/Mobileye EyeQ5/Tesla-motors TPU 144 TOPS-8bit)
- Cloud GPU (nVidia A100/V100/TitanV)
- Cloud NPU (Google-TPU, Huawei Ascend, Intel Habana, Qualcomm AI 100, …)
Also when using neural networks On Web — usually a GPU is used through the WebGL, WebAssembly or WebGPU libraries, for this case — the size of the model can matter: https://github.com/tensorflow/tfjs#about-this-repo
The use of devices and algorithms with weak parallelism is a dead-end development path, because it is impossible to reduce the lithograph size smaller than the size of a silicon atom to increase the processor frequency:
- The current best size for Semiconductor device fabrication (Lithography) is 5 nanometers.
- The size of the crystal lattice of silicon is 0.5 nanometers.
- The atomic radius of silicon is 0.1 nanometer.
The solution — processors with massive parallelism and more than 10 000 ALUs: single crystal or several crystals on one interposer. Hence, it is imperative to create neural networks that make efficient use of massively parallel computing machines such as GPU, NPU, TPU, DSP.
Improvements in Scaled YOLOv4 over YOLOv4:
- Scaled YOLOv4 used optimal network scaling techniques to get YOLOv4-CSP -> P5 -> P6 -> P7 networks
- Improved network architecture: Backbone is optimized and Neck (PAN) uses Cross-stage-partial (CSP) connections and Mish activation
- Exponential Moving Average (EMA) is used during training — this is a special case of SWA: https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging/
- For each resolution of the network, a separate neural network is trained (in YOLOv4, only one neural network was trained for all resolutions)
- Improved objectness normalizers in [yolo] layers
- Changed activations for Width and Height, which allows faster network training
- The [net] letter_box = 1 parameter (to keep the aspect ratio of the input image) is used for high resolution networks (for all networks except yolov4-tiny.cfg)
There are different Losses in YOLOv3, YOLOv4 and Scaled-YOLOv4:
- for bx and by — this eliminates grid sensitivity in the same way as in YOLOv4, but more aggressively
- for bw and bh — this limits the size of the bounded-box to 4*Anchor_size
In general, Scaled-YOLOv4 has the same AP50, but a higher AP than the original YOLOv4 with the same resolution and approximately the same speed. Then Scaled-YOLOv4 scales up to achieve a higher AP50 and AP at a lower speed.
The Pytorch YOLOv4 implementation predicts better coordinates (higher AP) but detects fewer objects (lower AP50):
- YOLOv4(Darknet) — 608x608— 62 FPS — 43.5% AP — 65.7% AP50
- YOLOv4(Pytorch) — 608x608 — 62 FPS — 45.5% AP — 64.1% AP50
Changes to the network architecture (CSP in the Neck and Mish-activation for all layers) then eliminate flaws of Pytorch implementation, so CSP+Mish improves both AP, AP50 and FPS:
- YOLOv4-CSP — 608x608–75FPS — 47.5% AP — 66.1% AP50
- YOLOv4-CSP — 640x640–70FPS — 47.5% AP — 66.2% AP50
Scaled-YOLOv4 neural network architecture (examples of three networks: P5, P6, P7):
CSP connection is extremely efficient, simple, and can be applied to any neural network. The idea is:
- half of the output signal goes along the main path (generates more semantic information with a large receiving field)
- and the other half of the signal goes bypass (preserves more spatial information with a small perceiving field)
YOLOv4 is used in various fields and tasks:
- Taiwanese government: Traffic control https://www.taiwannews.com.tw/en/news/3957400 and https://youtu.be/IiU6wFmfVnk
- Amazon: Anti-Covid19 Distance-assistant https://github.com/amzn/distance-assistant and Amazon Neurochip / Amazon EC2 Inf1 instances: https://aws.amazon.com/ru/blogs/machine-learning/improving-performance-for-deep-learning-based-object-detection-with-an-aws-neuron-compiled-yolov4-model-on-aws-inferentia/
- BMW Innovation Lab: https://github.com/BMW-InnovationLab
- And in many other tasks….
There are YOLOv4 implementations on various frameworks:
- Pytorch: https://github.com/WongKinYiu/ScaledYOLOv4
- Darknet: https://github.com/AlexeyAB/darknet
- TensorFlow: https://github.com/hunglc007/tensorflow-yolov4-tflite
pip install yolov4 https://pypi.org/project/yolov4/
- OpenCV: https://docs.opencv.org/master/da/d9d/tutorial_dnn_yolo.html
- OpenVINO: https://github.com/TNTWEN/OpenVINO-YOLOV4
- ONNX: https://developer.nvidia.com/blog/announcing-onnx-runtime-for-jetson/
- TensorRT ONNX Scaled-YOLOv4: https://github.com/linghu8812/tensorrt_inference/tree/master/ScaledYOLOv4
- TensorRT + tkDNN: https://github.com/ceccocats/tkDNN
- TensorRT + Deepstream: https://github.com/NVIDIA-AI-IOT/yolov4_deepstream
- Another Pytorch implementations:
- The structure of the network can be viewed using the Netron utility — Visualizer for neural networks: https://github.com/lutzroeder/netron
Also, the YOLOv4 approach can be used in other tasks, for example, when detecting 3D objects:
- Code — Complex-YOLOv4 (5-DOF): https://github.com/maudzung/Complex-YOLOv4-Pytorch
- Code — YOLO3D-YOLOv4 (7-DOF): https://github.com/maudzung/YOLO3D-YOLOv4-PyTorch