1. Attention and Self-Attention mechanism:

Intro:

• First attention mechanism (for language) (2015)
  NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE Dzmitry Bahdanau Jacobs, Germany KyungHyun Cho, Yoshua Bengio
  https://arxiv.org/pdf/1409.0473.pdf 
• First application of attention for vision
  http://proceedings.mlr.press/v37/xuc15.pdf 

Papers:

• Self attention- attention is all you need (language, GoogleBrain, NIPS18)
  https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf 
• Self attention for images: https://arxiv.org/pdf/1802.05751.pdf 
• Self attention GAN (Goodfellow): https://arxiv.org/pdf/1805.08318.pdf 

Useful:

• https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html#born-for-translation

2. One/few shot learning

papers:

• https://arxiv.org/pdf/1606.04080.pdf (Deepmind, Matching Networks for One Shot Learning)
• https://arxiv.org/pdf/1806.04734.pdf (Karlinsky, RepMet)
• https://arxiv.org/pdf/1806.04734.pdf (Karlinsky, Delta-encoder)
• https://arxiv.org/pdf/1711.06025.pdf (Learning to Compare)
• https://openreview.net/pdf?id=HkxLXnAcFQ (A closer look at few-shot classification, accepted ICLR19)

3. Semi-supervised learning

papers:

• http://openaccess.thecvf.com/content_ECCV_2018/papers/Safa_Cicek_SaaS_Speed_as_ECCV_2018_paper.pdf (SaaS- Soatto, ECCV18)
• http://openaccess.thecvf.com/content_cvpr_2018/papers/Radosavovic_Data_Distillation_Towards_CVPR_2018_paper.pdf (Towards Omni-Supervised Learning- He, CVPR18)
• https://arxiv.org/pdf/1804.09170.pdf (Realistic Evaluation of Deep Semi-Supervised Learning Algorithms- Goodfellow NIPS18)

4. Advances in transfer learning and multi-task learning

papers:

• Taskonomy (Zamir, Malik) (best paper CVPR18):
  http://openaccess.thecvf.com/content_cvpr_2018/papers/Zamir_Taskonomy_Disentangling_Task_CVPR_2018_paper.pdf 
• Partial Transfer Learning with Selective Adversarial Networks (Michael Jordan, CVPR18):
  http://openaccess.thecvf.com/content_cvpr_2018/papers/Cao_Partial_Transfer_Learning_CVPR_2018_paper.pdf 
• Multi-task learning as multi-objective optimization (Koltun, NIPS18):
  https://papers.nips.cc/paper/7334-multi-task-learning-as-multi-objective-optimization.pdf 

5. Advances in GANs

intro:

• GANs, conditional GANs
• Basic supervised domain transfer- https://arxiv.org/pdf/1611.07004.pdf (Pix2Pix)
• Domain transfer with no explicit supervision- https://arxiv.org/pdf/1703.10593.pdf (CycleGAN)

papers:

• Everybody dance now (Efros):
  https://arxiv.org/pdf/1808.07371.pdf 
• Pix2PixHD (NVIDIA, CVPR18): https://tcwang0509.github.io/pix2pixHD/ 
• https://arxiv.org/pdf/1804.04732.pdf (MUNIT, NVIDIA)
• A Style-Based Generator Architecture for Generative Adversarial Networks (NVIDIA):
  https://arxiv.org/pdf/1812.04948.pdf 
• https://openreview.net/pdf?id=B1xsqj09Fm (BigGAN, accepted ICLR19 oral)
• https://arxiv.org/pdf/1901.08753.pdf Towards a Deeper Understanding of Adversarial Losses
  

6. Advanced algorithms for gradient-based optimization

Intro:

• A Walk with SGD: How SGD Explores Regions of Deep Network Loss? (rejected from ICLR 2019) https://openreview.net/forum?id=B1l6e3RcF7
• Alternatively: Visualizing the Loss Landscape of Neural Nets (NIPS 2018) https://papers.nips.cc/paper/7875-visualizing-the-loss-landscape-of-neural-nets.pdf
• Momentum, ADAgrad, ADAdelta, RMSprop
• https://arxiv.org/pdf/1412.6980.pdf (ADAM)

papers:

• https://openreview.net/pdf?id=ryQu7f-RZ (AMSgrad)
• https://openreview.net/pdf?id=rJTutzbA-  (Insufficiency of momentum schemes for Optimization)
• https://openreview.net/pdf?id=B1YfAfcgl (Entropy-SGD)
• https://arxiv.org/pdf/1802.09568.pdf (Shampoo: Preconditioned Stochastic Tensor Optimization, Yoram Singer)
• Online Learning Rate Adaptation with Hypergradient Descent (ICLR18) https://openreview.net/forum?id=BkrsAzWAb 
• Elimination of All Bad Local Minima in Deep Learning https://arxiv.org/pdf/1901.00279.pdf 

7. Neural architecture search (NAS)

papers:

• https://arxiv.org/pdf/1611.01578.pdf (NAS with RL, ICLR17)
• https://arxiv.org/pdf/1806.09055.pdf (DeepMind&CMU, Differentiable NAS)
• https://arxiv.org/pdf/1711.04528.pdf (Simple NAS for CNNs, ICLR18)
• https://arxiv.org/pdf/1802.03268.pdf (Efficient NAS, ICML18)
• https://arxiv.org/pdf/1707.07012.pdf (Image-net SotA)
• https://arxiv.org/pdf/1812.10666v1.pdf (NAS over a graph)
• https://arxiv.org/pdf/1810.05749.pdf (Urtasun, accepted ICLR19)

useful:

• https://arxiv.org/pdf/1808.05377.pdf 

8. 3D reconstruction and detection

papers:

• http://openaccess.thecvf.com/content_cvpr_2018/papers/Tran_Extreme_3D_Face_CVPR_2018_paper.pdf (Extreme 3D face reconstruction, Hassner, Medioni, CVPR18)
• http://openaccess.thecvf.com/content_ECCV_2018/papers/Martin_Sundermeyer_Implicit_3D_Orientation_ECCV_2018_paper.pdf ( 3D Orientation Learning for 6D Object Detection, ECCV18 best paper)
• http://openaccess.thecvf.com/content_ECCV_2018/papers/Jiajun_Wu_Learning_3D_Shape_ECCV_2018_paper.pdf (Shape priors from single view, Freeman, ECCV18)
• http://openaccess.thecvf.com/content_ECCV_2018/papers/Guandao_Yang_A_Unified_Framework_ECCV_2018_paper.pdf (Single-View 3D Reconstruction with Limited Pose Supervision, ECCV18)
• http://openaccess.thecvf.com/content_ECCV_2018/papers/Yi_Zhou_Single-view_Hair_Reconstruction_ECCV_2018_paper.pdf (Hair reconstruction ECCV18)

Unsupervised Learning of Depth and Ego-Motion from Video (Snavely & Lowe) https://people.eecs.berkeley.edu/~tinghuiz/projects/SfMLearner/

9. Image Enhancement and Image Quality Assessment

papers:

• https://arxiv.org/pdf/1807.02758.pdf (RCAN, SR-SotA, ECCV18)
• http://openaccess.thecvf.com/content_cvpr_2018/papers/Lefkimmiatis_Universal_Denoising_Networks_CVPR_2018_paper.pdf (Universal denoising, CVPR18)
• http://openaccess.thecvf.com/content_cvpr_2018/papers/Zhang_The_Unreasonable_Effectiveness_CVPR_2018_paper.pdf (Deep Features as a Perceptual Metric, Isola, Efros, Shechtman, CVPR18)
• http://openaccess.thecvf.com/content_cvpr_2018/papers/Blau_The_Perception-Distortion_Tradeoff_CVPR_2018_paper.pdf (Perception-Distortion, Blau, Michaeli, CVPR18)

10. Interpretability and Visualizing of deep nets

papers:

• Network Dissection: Quantifying Interpretability of Deep Visual Representations, David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba
• Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values, Julius Adebayo, Justin Gilmer, Ian Goodfellow, Been Kim
• https://arxiv.org/abs/1901.00536
  

11. Normalization

intro (all of these papers are briefly mentioned in the group norm paper):

• https://arxiv.org/abs/1502.03167 (Batch normalization)
• https://arxiv.org/pdf/1607.08022.pdf (InstanceNorm, Ulyanov, Lempitzky, Vedaldi)
• https://arxiv.org/pdf/1607.06450.pdf (LayerNorm)

papers:

• https://arxiv.org/pdf/1803.08494.pdf (GroupNorm, He, ECCV18 honorable mention)
• https://arxiv.org/pdf/1805.11604.pdf (How BN helps optimization? NIPS18 oral)
• https://arxiv.org/pdf/1702.03275.pdf (BatchReNorm)
• https://arxiv.org/pdf/1706.02515.pdf (Self normalizing networks, NIPS17)
• https://arxiv.org/pdf/1705.10941.pdf (Basic spectral Norm)
• https://arxiv.org/pdf/1802.05957.pdf (Spectral Norm for GANs, ICLR18 oral)
• https://openreview.net/pdf?id=HyN-M2Rctm (ModeNorm, accepted ICLR19)

useful:

• http://gradientscience.org/batchnorm/ (Blog from the authors of the NIPS18 oral)

12. Video tracking

papers:

• https://ieeexplore.ieee.org/iel7/34/7029731/06870486.pdf (KCF: High-speed tracking with kernelized correlation filters)
• End-to-end representation learning for Correlation Filter based tracking https://arxiv.org/pdf/1704.06036.pdf
• Fast Online Object Tracking and Segmentation: A Unifying Approach https://arxiv.org/pdf/1812.05050.pdf

• Tracking emerges by colorizing videos http://openaccess.thecvf.com/content_ECCV_2018/papers/Carl_Vondrick_Self-supervised_Tracking_by_ECCV_2018_paper.pdf

• https://arxiv.org/pdf/1705.06368.pdf Re^3:  Real-Time Recurrent Regression Networks for Visual Tracking of Generic Objects

13. Adversarial attacks and defences

papers:

• Towards Deep Learning Models Resistant to Adversarial Attacks: https://arxiv.org/pdf/1706.06083.pdf
• Robustness May Be at Odds with Accuracy: https://arxiv.org/pdf/1805.12152.pdf
• https://arxiv.org/pdf/1805.12177 (Why do deep convolutional networks generalize so poorly to small image transformations?)
• http://papers.nips.cc/paper/8273-with-friends-like-these-who-needs-adversaries.pdf

(With friends like these, who needs adversaries?)

• https://arxiv.org/pdf/1805.12152.pdf  (Robustness may be at odds with accuracy, accepted ICLR19)

14. Understanding properties of CNNs

papers:

• https://openreview.net/pdf?id=BJlxm30cKm (Catastrophic forgetting, Bengio, ICLR19)
• http://papers.nips.cc/paper/6203-understanding-the-effective-receptive-field-in-deep-convolutional-neural-networks.pdf (Effective Receptive Field, Urtasun, NIPS16)
• https://openreview.net/pdf?id=r14EOsCqKX (Deep Learning Heuristics, ICLR19, Socher)
• https://arxiv.org/abs/1902.01996   (Are All Layers Created Equal?, Yoram Singer)

15. Non-local neural networks

papers:

• https://arxiv.org/pdf/1711.07971.pdf (Non-local neural networks, Girshick, He, CVPR18)
• https://arxiv.org/pdf/1806.02919.pdf (Non-local recurrent network for image restoration)
• https://arxiv.org/pdf/1812.03411.pdf (Feature denoising for improving adversarial robustness)

16. Vision and Audio

Papers:

•  https://arxiv.org/pdf/1804.03619 (Looking to listen at the cocktail party, SIG18)
• http://sound-of-pixels.csail.mit.edu/ (The sound of pixels, Torralba, ECCV18)
• http://andrewowens.com/multisensory/ (Self-Supervised, Efros, ECCV18 oral)
• https://deepmind.com/blog/objects-that-sound/ (2 papers, Zisserman, ECCV18)
• http://openaccess.thecvf.com/content_ECCV_2018/papers/Olivia_Wiles_X2Face_A_network_ECCV_2018_paper.pdf (X2Face, Zisserman ECCV18)
• http://www.robots.ox.ac.uk/~vgg/publications/2018/Chung18/chung18.pdf (Learning to lip read words by watching videos)
• https://arxiv.org/pdf/1806.06053 (Deep Lip Reading: a comparison of models and an online application)

17. Non adversarial methods

papers:

• https://arxiv.org/pdf/1812.08985.pdf (GLANN, Hoshen, Malik, 2018)
• https://arxiv.org/pdf/1806.00804.pdf (NAM, Hoshen, Wolf, 2018)
• https://arxiv.org/pdf/1803.04626.pdf (Contextual loss, Mechrez, Zelnik-Manor, 2018)
• https://arxiv.org/pdf/1803.02077.pdf (Contextual loss transformations, Mechrez, Zelnik-Manor, 2018)

18. Self supervision

papers:

• http://graphics.cs.cmu.edu/projects/deepContext/ (Efros, jigsaw, ICCV15)
• https://arxiv.org/pdf/1708.06734.pdf (Learning to count)
• https://arxiv.org/pdf/1808.07507.pdf (Video jigsaw)
• https://arxiv.org/abs/1902.08401 (Neural Conditioner)