This page contains resources about Artificial Neural Networks.

Subfields and Concepts Edit

  • Feedforward Neural Network
    • Single-Layer Perceptron (i.e. with no hidden layers)
    • Multi-Layer Perceptron (MLP)
    • Radial Basis Function (RBF) Network
    • Extreme Learning Machine (ELM)
    • Convolutional Neural Network (CNN or ConvNet)
  • Recurrent Neural Network (RNN)
    • Hopfield Network
    • Boltzmann Machine
    • Bidirectional RNN
    • Bidirectional associative memory (BAM)
    • Long short-term memory (LSTM)
    • Continuous Time RNN (CTRNN)
    • RNN-RBM
    • Echo State Network (ESN)
  • Stochastic Neural Network (i.e. with stochastic transfer function and units or stochastic weights)
    • Helmholtz Machine
    • Boltzmann Machine
    • Restricted Boltzmann Machine (RBM)
    • Conditional RBM (CRBM)
    • Autoassociative memory
    • Generative Stochastic Network
    • Generative Adversarial Network
    • Stochastic Feedforward Neural Network (with both stochastic and deterministic hidden units)
    • Stochastic Computation Graph
    • Variational Autoencoder (VAE)
  • Kohonen Network / Self-organizing map (SOM) / Self-organising feature map (SOFM)
  • Probabilistic Nerual Network
    • Bayesian Neural Network (i.e. a Gaussian Process with finitely many weights)
      • Probabilistic Backpropagation
      • Bayes by Backprop
    • Bayesian Dark Knowledge (BDK)
    • Natural-Parameter Network (NPN) (i.e. distributions for both the weights and the neurons)
      • Gamma NPN
      • Gaussian NPN
      • Poisson NPN
  • Random Neural Network
  • Autoencoder (used for Dimensionality Reduction)
    • Linear Autoencoder (equivalent to PCA)
    • Stacked Denoising Autoencoder
    • Sparse Autoencoder
    • Contractive Autoencoder
    • Generalized Denoising Autoencoder
    • VAE
  • Deep Neural Network (i.e. more than two hidden layers)
    • Deep Multi-Layer Perceptron
    • Deep Belief Network (DBN)
    • Convolutional Deep Neural Network
    • Long short-term memory (LSTM)
    • Deep Autoencoder
    • Neural Module Network (NMN)
  • Training
    • Automatic Differentiation
      • Backpropagation Algorithm
      • Backpropagation Through Time (for training RNNs)
      • Stochastic Backpropagation
    • Optimization
      • Stochastic Gradient Methods
        • Stochastic Gradient Descent (SGD)
        • SGD with Momentum
      • Simulated Annealing
      • Genetic Algorithms (for training RNNs)
    • Contrastive Divergent (CD) Algorithm (for training RBMs)
      • Persistent CD (PCD)
    • Wake-Sleep Algorithm (for Stochastic ANNs)
    • Generative Stochastic Networks (GSN) for probabilistic models
    • Auto-Encoding Variational Bayes (AEVB) Algorithm
  • Activation Functions / Transfer Functions for deterministic units (must be differentiable)
    • Logistic
    • Rectifier (ReLU)
    • Softmax
    • Swish
  • Cost Functions / Loss Functions / Objective Functions
    • Least-Squares
    • Cross-entropy
    • Relative Entropy / KL Divergence
  • Energy-Based Model (EBM)
    • Free energy (i.e. the contrastive term)
    • Regularization term
    • Loss functionals or Loss functions
      • Energy Loss
      • Generalized Perceptron Loss
      • Generalized Margin Losses
      • Negative Log-Likelihood Loss
  • Improve Generalization (to prevent overfitting)
    • Early stopping
    • Regularization / Weight decay
      • L1-regularization / Laplace prior
      • L2-regularization / Gaussian prior
      • Max norm constraints
    • Dropout
    • Add noise

Online Courses Edit

Video LecturesEdit

Lecture NotesEdit

Books and Book Chapters Edit

  • Bengio, Y., Goodfellow, I. J., & Courville, A. (2016). Deep Learning. MIT Press.
  • Nielsen, M. A. (2015). Neural Networks and Deep Learning. Determination Press.
  • Theodoridis, S. (2015). "Chapter 18: Neural Networks and Deep Learning". Machine Learning: A Bayesian and Optimization Perspective. Academic Press.
  • Barber, D. (2012). "Chapter 26: Distributed Computation". Bayesian Reasoning and Machine Learning. Cambridge University Press.
  • Alpaydin, E. (2010). "Chapter 11: Multilayer Perceptrons". Introduction to Machine Learning. MIT Press.
  • Haykin, S. S., Haykin, S. S., Haykin, S. S., & Haykin, S. S. (2009). Neural Networks and Learning Machines. 3rd Ed. Pearson.
  • Bishop, C. M. (2006). "Chapter 5: Neural Networks". Pattern Recognition and Machine Learning. Springer.
  • MacKay, D. J. (2003). "Chapter 38: Introduction to Neural Networks" Information Theory, Inference and Learning Algorithms. Cambridge University Press.
  • Mandic, D. P., & Chambers, J. (2001). Recurrent neural networks for prediction: learning algorithms, architectures and stability. John Wiley & Sons.
  • Bishop, C. M. (1995). Neural networks for pattern recognition. Oxford University Press.
  • Neal, R. M. (1996). Bayesian learning for neural networks. Springer Science & Business Media.
  • Rojas, R. (1996). Neural networks: a systematic introduction. Springer Science & Business Media. (link)

Scholarly Articles Edit

  • Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2015). Automatic differentiation in machine learning: a survey. arXiv preprint arXiv:1502.05767.
  • Jacobsson, H. (2005). Rule extraction from recurrent neural networks: A taxonomy and review. Neural Computation17(6), 1223-1263.

Tutorials Edit

Software Edit

See Deep Learning Software.

See also Edit

Other Resources Edit