Support Vector Regression(SVR)


The Support Vector Regression (SVR) is adopted from support Vector Machine (SVM) for the regression type data to predict the value. While dealing with real number data, the SVM changes its variant as regression.


𝑓(π‘₯) = π’˜π‘‡πœ‘(π‘₯) + 𝑏 ---------------(1)

where

–𝑀 is the weight vector 𝑙 dimension

β€“πœ‘(π‘₯) is a function that maps π‘₯ to the feature space with 𝑙 dimensions

– 𝑏 is biased

β€“πœ€ as the distance between the hyperplane and the two boundary lines

The coefficients 𝑀 and 𝑏 are estimated by minimizing the risk function. Therefore, to maximize the margin 𝛿, a minimum ‖𝑀‖ is needed. Optimization of problem-solving as shown in function (3.2).

π‘šin1/2 ‖𝑀‖² --------------------------(2)

with the condition, 𝑦𝑖 βˆ’ π‘€π‘‡πœ‘(π‘₯𝑖) βˆ’ 𝑏 ≀ πœ€, 𝑓or 𝑖 = 1, … , 𝑙 and π‘€π‘‡πœ‘(π‘₯𝑖) βˆ’ 𝑦𝑖 + 𝑏 ≀ πœ€, 𝑓or 𝑖 = 1, … , 𝑙. Where 𝑦𝑖 is the actual value of 𝑖 period, and πœ‘(π‘₯𝑖) is the estimated value of 𝑖 period.



π‘šin1/2 ‖𝑀‖² + ∁ βˆ‘β„“π‘–=1 (πœ‰π‘– + πœ‰π‘–βˆ—) --------------------------(3.3)

with the condition

𝑦𝑖 βˆ’ π‘€π‘‡πœ‘(π‘₯𝑖) βˆ’ 𝑏 βˆ’ πœ‰π‘– ≀ πœ€, 𝑓or 𝑖 = 1, … , 𝑙

π‘€π‘‡πœ‘(π‘₯𝑖) βˆ’ 𝑦𝑖 + 𝑏- πœ‰π‘–βˆ— ≀ πœ€, 𝑓or 𝑖 = 1, … , 𝑙

πœ‰π‘– , πœ‰π‘–βˆ— β‰₯ 0

The ‖𝑀‖² factor is called regulation. Minimizing ‖𝑀‖² will make a function as thin (flat) as possible so that it can control the functional capacity. All points outside the margin/limit πœ€ will be penalized.


𝐢 > 0 constant determines how much the error deviation is from the tolerable limit πœ€πœ€. The formula above is a Convex Linear Programming NLP Optimization Problem which functions to minimize the quadratic function to be converted into a constraint. This limitation can be solved by using the Lagrange Multiplier function. The process of deriving formulas is very long and complicated. After going through mathematical stages, a new equation is obtained with the function:


𝑓(π‘₯) = βˆ‘li=1 (π‘Žπ‘– βˆ’ π‘Žπ‘– βˆ—). (π‘₯𝑖. π‘₯) + 𝑏 ----(4)


Where xi is the support vector and π‘₯ is the test vector. The above functions can be used to solve linear problems. Whereas for non-linear problems the values of π‘₯𝑖, and π‘₯ are first transformed into a high-dimensional feature space by mapping the vectors π‘₯𝑖 and π‘₯ into the kernel function so that the final function becomes:

𝑓(π‘₯) = βˆ‘li=1 (π‘Žπ‘– βˆ’ π‘Žπ‘– βˆ—). k(π‘₯𝑖. π‘₯) + 𝑏 ----(4)

The function k(π‘₯𝑖. π‘₯) is the Kernel. The table 1 below shows the kernels used in the SVR calculation (Scholkopf & Smola, 2018)