📝 论文精读

Abstract

• 背景：

• Motivation:

can we get a better accuracy-robustness-efficiency trade-off with tools and architectures other than ResNets?

• Method：

• Result:

semantic attributes的具体示例

Introduction + Background

• TRADES

Experiment

📒 一些地道表达

1. resort to 常用 / 求助于，诉诸于

Currently,the community resorts to deeper and wider models to improve this trade-off, hence decreasing the efficiency and practicality of adversarial training.

1. tailored 定制的， 特定的

We manage to do so by finding a tailored adversarial training recipe –different from the default recipe for standard training– which leads to state-of-the-art results by a significant margin.

1. variations 变体,变种 (模型变体)

2. robustness-accuracy-efficiency trilemma 鲁棒性-准确率-性能 三难问题

3. be leveraged for 被用于

Adversarially trained XCiTs can be leveraged for fine-tuning.

🤗 总结

1. 虽然在读完感觉作者做的工作不是那么创新， 但是确实写作上加大分，读的时候可以明显感觉到我的写作被作者吊打。

2. 虽然作者的工作与我的工作比较相关， 但是可能重点学习的应该是写作和论文中提到的很多相关工作。

3. 此外， 在我的工作中对于data augmentation的使用较为欠缺， 过于低估data augmentation的威力了， 后续可以补充。

4. 在我的工作中， 直接就在MobileViT s, xs, xxs上实验， 然后分析的时候也只是分析参数量对于鲁棒性的影响。
而本文作者采取了一个scale up的分析， 同样对于不同参数来的XCiT模型
从XCiT-N12 到XCiT-S12 再到 XCiT-M12 再到 XCiT-L12， 可以借鉴这种写作方法。

5. 我的工作在evaluation上没有作者那么严谨， 作者为了防止PGD引发梯度掩蔽， 没有在evalution阶段直接采用PGD， 而是采用了一种混合型攻击策略——AutoAttack来进行， 这一部分可以学习改进。

6. 作者做了很多的ablation study， 相比而言， 本文对于这一部分涉及比较少。

7. 最后一部分, 探究XCiT的扰动的Semantic nature。 作者先从现象(visualize adv perturbations) 看出robust的XCiT和non-robust的XCiT区别， 然后从gradient的角度来进行实质性的分析， 让作者采用的adv training recipe更加的可解释。这一部分很值得学习，我们的工作最后一部分在移动端制作了一个轻量化Transformer推断程序， 进行了attention可视化。 相比于这篇文章来说， 只给出了一个验证结果， 没有分析或者较深入的原因解释，效果可能没有本文好。
accumulation of gradient between robust and non-robust XCiT

📎 参考文章

🔧 main paper

• Adversarially Robust Vision Transformer

• XCiT: Cross-Covariance Image Transformers

XCiT: Cross-Covariance Image Transformers

https://proceedings.neurips.cc/paper/2021/hash/a655fbe4b8d7439994aa37ddad80de56-Abstract.html

🌥️ related robustness improvement

• Improving Robustness using Generated Data

• Data Augmentation Can Improve Robustness

• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition

Robustness and Accuracy Could Be Reconcilable by (Proper) Definition

The trade-off between robustness and accuracy has been widely studied in the adversarial literature. Although still controversial, the prevailing view is tha...

https://proceedings.mlr.press/v162/pang22a.html

• Helper-based Adversarial Training: Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off

Helper-based Adversarial Training: Reducing Excessive Margin to...

While adversarial training has become the de facto approach for training robust classifiers, it leads to a drop in accuracy. This has led to prior works postulating that accuracy is inherently at...

https://openreview.net/forum?id=BuD2LmNaU3a

🚩 robust bench

• RobustBench

🚆 vanilla ViT traing recipe

• How to train your vit? data, augmentation, and regularization in vision transformers

• Training data-efficient image transformers & distillation through attention

Training data-efficient image transformers & distillation through attention

Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These high-performing visio...

https://proceedings.mlr.press/v139/touvron21a

🤖 Gradient masking

• Practical black-box attacks against machine learning

🤺 Evaluation of defenses

• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

The field of defense strategies against adversarial attacks has significantly grown over the last years, but progress is hampered as the evaluation of advers...

https://proceedings.mlr.press/v119/croce20b.html

💽 Data Augmentation

• Cutmix: Regularization strategy to train strong classifiers with localizable features

ICCV 2019 Open Access Repository

https://openaccess.thecvf.com/content_ICCV_2019/html/Yun_CutMix_Regularization_Strategy_to_Train_Strong_Classifiers_With_Localizable_Features_ICCV_2019_paper.html

• Randaugment: Practical automated data augmentation with a reduced search space

CVPR 2020 Open Access Repository

https://openaccess.thecvf.com/content_CVPRW_2020/html/w40/Cubuk_Randaugment_Practical_Automated_Data_Augmentation_With_a_Reduced_Search_Space_CVPRW_2020_paper.html

• mixup: Beyond empirical risk minimization

mixup: Beyond Empirical Risk Minimization

Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle...

https://arxiv.org/abs/1710.09412

• Random erasing data augmentation

Random Erasing Data Augmentation

In this paper, we introduce Random Erasing, a new data augmentation method for training the convolutional neural network (CNN). In training, Random Erasing randomly selects a rectangle region in an image and erases its pixels with random values. In this process, training images with various levels of occlusion are generated, which reduces the risk of over-fitting and makes the model robust to occlusion. Random Erasing is parameter learning free, easy to implement, and can be integrated with most of the CNN-based recognition models. Albeit simple, Random Erasing is complementary to commonly used data augmentation techniques such as random cropping and flipping, and yields consistent improvement over strong baselines in image classification, object detection and person re-identification. Code is available at: https://github.com/zhunzhong07/Random-Erasing.

https://ojs.aaai.org/index.php/AAAI/article/view/7000