Continual Learning Methods

Continual Learning Methods

Classification #1

• Model growing: Increase the model capacity for every new task
  • PNN: Progressive Neural Networks
    • Problems: The model grows linarly with the number of trained tasks, Need to know task labels during test
• Parameter isolation: Explicitly identify important parameters for each task
  • PackNet
• Regularization: Penalize (some) parameter variations
  • EWC: Elastic Weight Consolidation
• Knowledge distillation: Use the model in a previous training state as a teacher
  • LwF: Learning without Forgetting
• Rehearsal: Store old inputs and replay them to the model.
  • GEM: Gradient Episodic Memory
  • A-GEM: Average GEM

Classification #2

• Replay-based
  • Rehearsal
    • iCaRL, SER, TEM
    • Experience Replay (ER; Chaudhry et al., 2019 arXiv)
  • Pseudo rehearsal
    • PR, CCLUGM, LGM
    • Deep Generative Replay (DGR; Shin et al., 2017 NeurIPS)
  • Constrained
    • GEM, A-GEM, GSS
• Regularization-based
  • Prior-focused (parameter regularization)
    • R-EWC, MAS, Riemannian Walk
    • Elastic weight consolidation (EWC; Kirkpatrick et al, 2017 PNAS), Synaptic Intelligence (SI; Zenke et al., 2017 ICML)
  • Data-focused (a.k.a. Knowledge Distillation, Functional regularization)
    • LFL, EBLL, DMC
    • Learning without forgetting (LwF; Li & Hoiem, 2017 TPAMI), Functional Regularization of the Memorable Past (FROMP; Pan et al., 2020 NeurIPS)
• Parameter isolation
  • Fixed network
    • PackNet, PathNet, Piggyback, HAT
  • Dynamic architecture (a.k.a. Model Growing)
    • PNN, Expert Gate, RCL, DAN

Classification #3

• Parameter regularization
  • Elastic weight consolidation (EWC; Kirkpatrick et al, 2017 PNAS)
  • Synaptic Intelligence (SI; Zenke et al., 2017 ICML)
• Functional regularization
  • Learning without forgetting (LwF; Li & Hoiem, 2017 TPAMI)
  • Functional Regularization of the Memorable Past (FROMP; Pan et al., 2020 NeurIPS)
• Replay
  • Deep Generative Replay (DGR; Shin et al., 2017 NeurIPS)
  • Experience Replay (ER; Chaudhry et al., 2019 arXiv)
• Context-specific components
  • Separate Networks (SepN)
  • Context-dependent Gating (XdG; Masse et al., 2018 PNAS)
• Template-based classification
  • Generative Classifier (GenC; van de Ven et al., 2021 CVPR-W)
  • Incremental Classifier and Representation Learning (iCaRL; Rebuffi et al., 2017 CVPR)

Reference

A Continual Learning Survey: Defying Forgetting in Classification Tasks

More …

continual-learning-type