Machine Learning Zoo

Machine Learning Zoo

Function-based Categorization

Supervised Classification

• Support Vector Machines (SVM)
• Discriminant Analysis
• Naive Bayes
• Nearest Neighbors
• Neural Network

Supervised Regression

Linear Models

• Linear Regression
• Logistic Regression
• Ridge Regression
• Lasso Regression

Tree Models

• Decision Trees
• Random Forests
• Gradient Boosting Regression
• Hierarchical Regression
• XGBoost

Ensemble Models

• Ensembles

Neural Networks Models

• Neural Networks

Unsupervised Clustering

• k-Means Clustering
• DBSCAN
• HDBSCAN
• Agglomerative Hierarchical Clustering
• OPTICS
• Gaussian Mixture Models
• Hidden Markov
• Neural Networks

Unsupervised Association Rules

• Apriori
• FP-growth
• FP-Max
• Eclat
• Hypergeometric Network

Unsupervised Dimensionality Reduction

• PCA
• t-distributed Stochastic Neighbor Embedding (t-SNE)
• Uniform Manifold Approximation and Projection (UMAP)
• ICA
• Pairwise Controlled Manifold Approximation (PaCMAP)

Semi-supervised Classification

Semi-supervised Regression

Similarity-based Categorization

Dimensionality Reduction Algorithms

Seek and exploit the inherent structure in the data in an unsupervised manner in order to summarize or describe data using less information.

Tasks: dimensionality reduction

• Principal Component Analysis (PCA)
• Principal Component Regression (PCR)
• Partial Least Squares Regression (PLSR)
• Sammon Mapping
• Multidimensional Scaling (MDS)
• Projection Pursuit
• Linear Discriminant Analysis (LDA)
• Mixture Discriminant Analysis (MDA)
• Quadratic Discriminant Analysis (QDA)
• Flexible Discriminant Analysis (FDA)

Regression Algorithms

Modeling the relationship between variables that is iteratively refined using a measure of error in the predictions made by the model.

Tasks: regression, classification

• Ordinary Least Squares Regression (OLSR)
• Linear Regression
• Logistic Regression
• Stepwise Regression
• Multivariate Adaptive Regression Splines (MARS)
• Locally Estimated Scatterplot Smoothing (LOESS)

Regularization Algorithms

An extension made to another method (typically regression methods) that penalizes models based on their complexity, favoring simpler models that are also better at generalizing.

Tasks: regression

• Ridge Regression
• Least Absolute Shrinkage and Selection Operator (LASSO)
• Elastic Net
• Least-Angle Regression (LARS)

Decision Tree Algorithms

Decision tree methods construct a model of decisions made based on actual values of attributes in the data. Decisions fork in tree structures until a prediction decision is made for a given record.

Tasks: classification, regression

• Classification and Regression Tree (CART)
• Iterative Dichotomiser 3 (ID3)
• C4.5 and C5.0 (different versions of a powerful approach)
• Chi-squared Automatic Interaction Detection (CHAID)
• Decision Stump
• M5
• Conditional Decision Trees

Bayesian Algorithms

Bayesian methods are those that explicitly apply Bayes’ Theorem, an approach to calculate conditional probability based on prior knowledge, and the naive assumption that each feature is independent to each other.

Tasks: regression, classification

• Naive Bayes
• Gaussian Naive Bayes
• Multinomial Naive Bayes
• Averaged One-Dependence Estimators (AODE)
• Bayesian Belief Network (BBN)
• Bayesian Network (BN)

Clustering Algorithms

Concerning with using the inherent structures in the data to best organize the data into groups of maximum commonality.

Tasks: clustering

• k-Means
• k-Medians
• Expectation Maximisation (EM)
• Hierarchical Clustering

Instance Based Algorithms

Decision problem with instances or examples of training data that are deemed important or required to the model by building up a database of example data and compare new data to the database using a similarity measure in order to find the best match and make a prediction.

Tasks: clustering, classification

• k-Nearest Neighbor (kNN)
• Learning Vector Quantization (LVQ)
• Self-Organizing Map (SOM)
• Locally Weighted Learning (LWL)
• Support Vector Machines (SVM)

Association Rule Learning Algorithms

Association rule learning methods extract rules that best explain observed relationships between variables in data.

Tasks: association

• Apriori algorithm
• Eclat algorithm

Artificial Neural Network Algorithms

Models that are inspired by the structure and/or function of biological neural networks.

Tasks: classification, regression

• Perceptron
• Multilayer Perceptrons (MLP)
• Back-Propagation
• Stochastic Gradient Descent
• Hopfield Network
• Radial Basis Function Network (RBFN)

Deep Learning Algorithms

Modern update to Artificial Neural Networks by building much larger and more complex neural networks.

Tasks: classification, regression

• Convolutional Neural Network (CNN)
• Recurrent Neural Networks (RNNs)
• Long Short-Term Memory Networks (LSTMs)
• Stacked Auto-Encoders
• Deep Boltzmann Machine (DBM)
• Deep Belief Networks (DBN)

Ensemble Algorithms

Models composed of multiple weaker models that are independently trained and whose predictions are combined in some way to make the overall prediction.

Tasks: classification, regression

• Boosting. To fit a sequence that is made out of weak learners to the weighted versions of the data, more weight is given to the examples that were misclassified in the earlier rounds.
• Bootstrapped Aggregation (Bagging). Reduce the variance of an estimate is by the average, to average together the multiple estimates.
• Stacking
• AdaBoost
• Weighted Average (Blending)
• Stacked Generalization (Stacking)
• Gradient Boosting Machines (GBM)
• Gradient Boosted Regression Trees (GBRT)
• Random Forest

Feature selection algorithms

Algorithm accuracy evaluation

Performance measures

Optimization algorithms

References

• The Neural Network Zoo - The Asimov Institute
• A Tour of Machine Learning Algorithms
• Data Mining With Python and R documentation

Images