# KNN and ANN

k-Nearest-Neighbors (kNN) is one of the simplest machine-learning algo­rithms. It is a pattern-recognition method for classifying patterns or features

(Duda & Hart, 2000). Classification is based on the similarity of a pattern of current values with respect to training samples in the feature space.

For the purpose of time-series forecasting, the kNN model consists of looking into its history and identifying the timestamp in the past that resembles the “current” conditions most closely. Once the best match—and there may be more than one—is found, forecasting is determined by looking at the time – series values subsequent to it. In essence, the kNN model resembles a lookup table for which previous patterns are used as indicators of sequential behavior.

The first step in developing a kNN model is to construct the database of features that will be used in the comparison with “current” conditions. For a univariate kNN, examples of features used are

• Time-series values

• Averaged time-series values

• Time-series entropy

assuming that the features for time t are assembled in the vector ~p (t) with components pj, and that the features for historical data are assembled in a matrix Ay whose rows correspond to the vector of features for each time in the historical dataset. The match is then the index k that minimizes the mean square error (MSE):

k = arg min

The forecast is obtained from the values of the time series that follow the timestamp corresponding to the index k. For instance:

У knn (t + TH) = y(tk + TH)

If more that one match is found, we can simply obtain the forecasting as the average:

1 n

y knn(t + TH )=~^2y(tk + TH )i

n

1=1

Artificial neural networks (ANNs) (Bishop, 1995) represent models also often used in the forecasting of time series. ANNs are useful tools for problems in classification and regression, are characterized by the ability to correlate highly nonlinear behavior, and have been widely and successfully employed in diverse forecasting problems (Mellit & Pavan, 2010; Marquez & Coimbra, 2011). Extensive reviews of forecasting with ANNs can be found in Zhang et al. (Zhang et al., 1998) and Mellit (Mellit, 2008), with the latter focusing exclusively on solar radiation. In general, neural networks map the input variables to the output by sending signals through elements called neurons. Neurons are arranged in layers, where the first layer receives the input variables, the last produces the output, and the layers in between (referred to as hidden layers) contain the hidden

neurons. A neuron receives the weighted sum of the inputs and produces the output by applying the activation function to the weighted sum. Inputs to a neuron can be from external stimuli or can be output from other neurons.

Once the ANN structure, the number of layers, the number of neurons, the activation functions, and so forth, are established, the ANN undergoes a training process in which the weights that control neuron activation are adjusted so that the minimization of some performance function is achie­ved—typically MSE. Numerical optimization algorithms such as back – propagation, conjugate gradients, quasi-Newton, and Levenberg-Marquardt have been developed to effectively adjust the weights.

The performance of ANN depends strongly on its structure as well as the choice of activation functions, the training method, and the input variables. There are several tools for preprocessing input data to enhance forecasting perform- ance—for example, normalization, principal component analysis (Bishop, 1995), and the gamma test for input selection (Marquez & Coimbra, 2011).

Updated: August 24, 2015 — 8:55 am