Applications of adversarial training part3 (machine learning) | by Monodeep Mukherjee | October 2022

Photo by Kimberly Farmer on Unsplash
  1. Revisiting adapters with contradictory training(arXiv)

Author : Sylvestre-Alvise Rebuffi, Francesco Croce, Sven Gowal

Summary : While adversarial training is typically used as a defense mechanism, recent work shows that it can also act as a regulation mechanism. By co-training a neural network on clean and contradictory inputs, it is possible to improve the classification accuracy on clean and non-conflicting inputs. We demonstrate that, contrary to previous results, it is not necessary to separate batch statistics when co-training on clean and contradictory inputs, and that it suffices to use adapters with few parameters specific to the domain for each type of entry. We establish that using the classification token of a vision transformer (ViT) as an adapter is sufficient to match the classification performance of dual normalization layers, while using far fewer additional parameters. First, we improve the top-1 accuracy of a non-adversarially trained ViT-B16 model by +1.12% on ImageNet (reaching 83.76% top-1 accuracy). Second, and more importantly, we show that training with adapters enables model soups through linear combinations of proper and contradictory tokens. These model soups, which we call adversarial model soups, allow us to compromise between clean and robust accuracy without sacrificing efficiency. Finally, we show that we can easily adapt the resulting models in the face of distribution changes. Our ViT-B16 achieves top-notch accuracies on ImageNet variants that are on average +4.00% better than those achieved with hidden autoencoders.

2.A2: Effective Automated Attacker to Reinforce Adversarial Training (arXiv)

Author : Zhuoer Xu, Guanghui Zhu, Changhua Meng, Shiwen Cui, Zhenzhe Ying, Weiqiang Wang, Ming GU, Yihua Huang

Summary : Based on the significant improvement in model robustness by AT (Adversarial Training), various variants have been proposed to further increase performance. Well-recognized methods have focused on different components of AT (e.g. designing loss functions and exploiting additional unlabeled data). It is generally accepted that stronger perturbations produce more robust models. However, how to efficiently generate stronger disturbances is still missed. In this article, we propose an efficient automated attacker called A2 to boost AT by generating the optimal disturbances on the fly during training. A2 is an automated attacker parameterized to search the attacker space for the best attacker against the defense pattern and examples. Extensive experiments on different datasets demonstrate that A2 generates stronger disturbances with low additional cost and reliably improves the robustness of various AT methods against different attacks.

3.Force Adaptive Adversarial Training (arXiv)

Author : Chaojian Yu, Dawei Zhou, Li Shen, Jun Yu, Bo Han, Mingming Gong, Nannan Wang, Tongliang Liu

Summary : Adversarial training (AT) has been proven to reliably improve network robustness against contradictory data. However, the current AT with a predefined disturbance budget has limitations in training a robust network. First, applying a pre-specified perturbation budget on networks of different model capabilities will produce a divergent degree of robustness mismatch between natural and robust accuracies, which deviates from the desideratum of the robust network. Second, the attack strength of conflicting training data constrained by the predefined perturbation budget fails to scale as the network robustness grows, which leads to robust overfitting and further degrades the contradictory robustness. To overcome these limitations, we propose emph{Strength-Adaptive Adversarial Training} (SAAT). Specifically, the adversary uses an adversary loss constraint to generate adversary training data. Under this constraint, the disturbance budget will be adaptively adjusted based on the training state of conflicting data, which can effectively avoid robust overfitting. In addition, SAAT explicitly limits the attack strength of the training data by adversarial loss, which manipulates the capacity planning of the model during training, and thus can flexibly control the degree of robustness mismatch and adjust the trade-off. between natural precision and robustness. Extensive experiments show that our proposal enhances the robustness of adversarial training

Sherry J. Basler