{"id":187,"date":"2020-09-06T23:22:16","date_gmt":"2020-09-06T23:22:16","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/09\/06\/scikit-optimize-for-hyperparameter-tuning-in-machine-learning\/"},"modified":"2020-09-06T23:22:16","modified_gmt":"2020-09-06T23:22:16","slug":"scikit-optimize-for-hyperparameter-tuning-in-machine-learning","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/09\/06\/scikit-optimize-for-hyperparameter-tuning-in-machine-learning\/","title":{"rendered":"Scikit-Optimize for Hyperparameter Tuning in Machine Learning"},"content":{"rendered":"<div id=\"\">\n<p id=\"last-modified-info\">Last Updated on September 7, 2020<\/p>\n<p>Hyperparameter optimization refers to performing a search in order to discover the set of specific model configuration arguments that result in the best performance of the model on a specific dataset.<\/p>\n<p>There are many ways to perform hyperparameter optimization, although modern methods, such as Bayesian Optimization, are fast and effective. The <strong>Scikit-Optimize library<\/strong> is an open-source Python library that provides an implementation of Bayesian Optimization that can be used to tune the hyperparameters of machine learning models from the scikit-Learn Python library.<\/p>\n<p>You can easily use the Scikit-Optimize library to tune the models on your next machine learning project.<\/p>\n<p>In this tutorial, you will discover how to use the Scikit-Optimize library to use Bayesian Optimization for hyperparameter tuning.<\/p>\n<p>After completing this tutorial, you will know:<\/p>\n<ul>\n<li>Scikit-Optimize provides a general toolkit for Bayesian Optimization that can be used for hyperparameter tuning.<\/li>\n<li>How to manually use the Scikit-Optimize library to tune the hyperparameters of a machine learning model.<\/li>\n<li>How to use the built-in BayesSearchCV class to perform model hyperparameter tuning.<\/li>\n<\/ul>\n<p>Let\u2019s get started.<\/p>\n<div id=\"attachment_10461\" class=\"wp-caption aligncenter\">\n<img decoding=\"async\" aria-describedby=\"caption-attachment-10461\" loading=\"lazy\" class=\"size-full wp-image-10461\" src=\"https:\/\/3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com\/wp-content\/uploads\/2020\/06\/Scikit-Optimize-for-Hyperparameter-Tuning-in-Machine-Learning.jpg\" alt=\"Scikit-Optimize for Hyperparameter Tuning in Machine Learning\" width=\"800\" height=\"450\"><\/p>\n<p id=\"caption-attachment-10461\" class=\"wp-caption-text\">Scikit-Optimize for Hyperparameter Tuning in Machine Learning<br \/>Photo by <a href=\"https:\/\/flickr.com\/photos\/dnevill\/44893231465\/\">Dan Nevill<\/a>, some rights reserved.<\/p>\n<\/div>\n<h2>Tutorial Overview<\/h2>\n<p>This tutorial is divided into four parts; they are:<\/p>\n<ol>\n<li>Scikit-Optimize<\/li>\n<li>Machine Learning Dataset and Model<\/li>\n<li>Manually Tune Algorithm Hyperparameters<\/li>\n<li>Automatically Tune Algorithm Hyperparameters<\/li>\n<\/ol>\n<h2>Scikit-Optimize<\/h2>\n<p>Scikit-Optimize, or skopt for short, is an open-source Python library for performing optimization tasks.<\/p>\n<p>It offers efficient optimization algorithms, such as Bayesian Optimization, and can be used to find the minimum or maximum of arbitrary cost functions.<\/p>\n<p>Bayesian Optimization provides a principled technique based on Bayes Theorem to direct a search of a global optimization problem that is efficient and effective. It works by building a probabilistic model of the objective function, called the surrogate function, that is then searched efficiently with an acquisition function before candidate samples are chosen for evaluation on the real objective function.<\/p>\n<p>For more on the topic of Bayesian Optimization, see the tutorial:<\/p>\n<p>Importantly, the library provides support for tuning the hyperparameters of machine learning algorithms offered by the scikit-learn library, so-called hyperparameter optimization. As such, it offers an efficient alternative to less efficient hyperparameter optimization procedures such as grid search and random search.<\/p>\n<p>The scikit-optimize library can be installed using pip, as follows:<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc238c701592524\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nsudo pip install scikit-optimize<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>sudo pip install scikit-optimize<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0001 seconds] --><\/p>\n<p>Once installed, we can import the library and print the version number to confirm the library was installed successfully and can be accessed.<\/p>\n<p>The complete example is listed below.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc2391936576828\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# report scikit-optimize version number<br \/>\nimport skopt<br \/>\nprint(&#8216;skopt %s&#8217; % skopt.__version__)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># report scikit-optimize version number<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">skopt<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;skopt %s&#8217;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">skopt<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">__version__<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0001 seconds] --><\/p>\n<p>Running the example reports the currently installed version number of scikit-optimize.<\/p>\n<p>Your version number should be the same or higher.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>For more installation instructions, see the documentation:<\/p>\n<p>Now that we are familiar with what Scikit-Optimize is and how to install it, let\u2019s explore how we can use it to tune the hyperparameters of a machine learning model.<\/p>\n<h2>Machine Learning Dataset and Model<\/h2>\n<p>First, let\u2019s select a standard dataset and a model to address it.<\/p>\n<p>We will use the ionosphere machine learning dataset. This is a standard machine learning dataset comprising 351 rows of data with three numerical input variables and a target variable with two class values, e.g. binary classification.<\/p>\n<p>Using a test harness of <a href=\"https:\/\/machinelearningmastery.com\/k-fold-cross-validation\/\">repeated stratified 10-fold cross-validation<\/a> with three repeats, a naive model can achieve an accuracy of about 64 percent. A top performing model can achieve accuracy on this same test harness of about 94 percent. This provides the bounds of expected performance on this dataset.<\/p>\n<p>The dataset involves predicting whether measurements of the ionosphere indicate a specific structure or not.<\/p>\n<p>You can learn more about the dataset here:<\/p>\n<p>No need to download the dataset; we will download it automatically as part of our worked examples.<\/p>\n<p>The example below downloads the dataset and summarizes its shape.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc2394168146972\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# summarize the ionosphere dataset<br \/>\nfrom pandas import read_csv<br \/>\n# load dataset<br \/>\nurl = &#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/ionosphere.csv&#8217;<br \/>\ndataframe = read_csv(url, header=None)<br \/>\n# split into input and output elements<br \/>\ndata = dataframe.values<br \/>\nX, y = data[:, :-1], data[:, -1]<br \/>\nprint(X.shape, y.shape)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># summarize the ionosphere dataset<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">pandas <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">read<\/span><span class=\"crayon-sy\">_<\/span>csv<\/p>\n<p><span class=\"crayon-p\"># load dataset<\/span><\/p>\n<p><span class=\"crayon-v\">url<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/ionosphere.csv&#8217;<\/span><\/p>\n<p><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">read_csv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">url<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">header<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into input and output elements<\/span><\/p>\n<p><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">values<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0005 seconds] --><\/p>\n<p>Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 351 rows of data with 34 input variables.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>We can evaluate a <a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.svm.SVC.html\">support vector machine<\/a> (SVM) model on this dataset using repeated stratified cross-validation.<\/p>\n<p>We can report the mean model performance on the dataset averaged over all folds and repeats, which will provide a reference for model hyperparameter tuning performed in later sections.<\/p>\n<p>The complete example is listed below.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc2396258199069\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# evaluate an svm for the ionosphere dataset<br \/>\nfrom numpy import mean<br \/>\nfrom numpy import std<br \/>\nfrom pandas import read_csv<br \/>\nfrom sklearn.model_selection import cross_val_score<br \/>\nfrom sklearn.model_selection import RepeatedStratifiedKFold<br \/>\nfrom sklearn.svm import SVC<br \/>\n# load dataset<br \/>\nurl = &#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/ionosphere.csv&#8217;<br \/>\ndataframe = read_csv(url, header=None)<br \/>\n# split into input and output elements<br \/>\ndata = dataframe.values<br \/>\nX, y = data[:, :-1], data[:, -1]<br \/>\nprint(X.shape, y.shape)<br \/>\n# define model model<br \/>\nmodel = SVC()<br \/>\n# define test harness<br \/>\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)<br \/>\n# evaluate model<br \/>\nm_scores = cross_val_score(model, X, y, scoring=&#8217;accuracy&#8217;, cv=cv, n_jobs=-1, error_score=&#8217;raise&#8217;)<br \/>\nprint(&#8216;Accuracy: %.3f (%.3f)&#8217; % (mean(m_scores), std(m_scores)))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># evaluate an svm for the ionosphere dataset<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">numpy <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">mean<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">numpy <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">std<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">pandas <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">read_csv<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">model_selection <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">cross_val_score<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">model_selection <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">RepeatedStratifiedKFold<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">svm <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-i\">SVC<\/span><\/p>\n<p><span class=\"crayon-p\"># load dataset<\/span><\/p>\n<p><span class=\"crayon-v\">url<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/ionosphere.csv&#8217;<\/span><\/p>\n<p><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">read_csv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">url<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">header<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into input and output elements<\/span><\/p>\n<p><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">values<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># define model model<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">SVC<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># define test harness<\/span><\/p>\n<p><span class=\"crayon-v\">cv<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">RepeatedStratifiedKFold<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n_splits<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">10<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_repeats<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">random_state<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># evaluate model<\/span><\/p>\n<p><span class=\"crayon-v\">m_scores<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">cross_val_score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">scoring<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;accuracy&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_jobs<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">error_score<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;raise&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;Accuracy: %.3f (%.3f)&#8217;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">mean<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">m_scores<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">std<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">m_scores<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0011 seconds] --><\/p>\n<p>Running the example first loads and prepares the dataset, then evaluates the SVM model on the dataset.<\/p>\n<p><strong>Note<\/strong>: Your <a href=\"https:\/\/machinelearningmastery.com\/different-results-each-time-in-machine-learning\/\">results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n<p>In this case, we can see that the SVM with default hyperparameters achieved a mean classification accuracy of about 93.7 percent, which is skillful and close to the top performance on the problem of 94 percent.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc2398498073635\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n(351, 34) (351,)<br \/>\nAccuracy: 0.937 (0.038)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>(351, 34) (351,)<\/p>\n<p>Accuracy: 0.937 (0.038)<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>Next, let\u2019s see if we can improve performance by tuning the model hyperparameters using the scikit-optimize library.<\/p>\n<h2>Manually Tune Algorithm Hyperparameters<\/h2>\n<p>The Scikit-Optimize library can be used to tune the hyperparameters of a machine learning model.<\/p>\n<p>We can achieve this manually by using the Bayesian Optimization capabilities of the library.<\/p>\n<p>This requires that we first define a search space. In this case, this will be the hyperparameters of the model that we wish to tune, and the scope or range of each hyperparameter.<\/p>\n<p>We will tune the following <a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.svm.SVC.html\">hyperparameters of the SVM<\/a> model:<\/p>\n<ul>\n<li>\n<strong>C<\/strong>, the regularization parameter.<\/li>\n<li>\n<strong>kernel<\/strong>, the type of kernel used in the model.<\/li>\n<li>\n<strong>degree<\/strong>, used for the polynomial kernel.<\/li>\n<li>\n<strong>gamma<\/strong>, used in most other kernels.<\/li>\n<\/ul>\n<p>For the numeric hyperparameters <em>C<\/em> and <em>gamma<\/em>, we will define a log scale to search between a small value of 1e-6 and 100. <em>Degree<\/em> is an integer and we will search values between 1 and 5. Finally, the <em>kernel<\/em> is a categorical variable with specific named values.<\/p>\n<p>We can define the search space for these four hyperparameters, a list of data types from the skopt library, as follows:<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc2399512230559\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# define the space of hyperparameters to search<br \/>\nsearch_space = list()<br \/>\nsearch_space.append(Real(1e-6, 100.0, &#8216;log-uniform&#8217;, name=&#8217;C&#8217;))<br \/>\nsearch_space.append(Categorical([&#8216;linear&#8217;, &#8216;poly&#8217;, &#8216;rbf&#8217;, &#8216;sigmoid&#8217;], name=&#8217;kernel&#8217;))<br \/>\nsearch_space.append(Integer(1, 5, name=&#8217;degree&#8217;))<br \/>\nsearch_space.append(Real(1e-6, 100.0, &#8216;log-uniform&#8217;, name=&#8217;gamma&#8217;))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># define the space of hyperparameters to search<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">append<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">Real<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1e<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100.0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;log-uniform&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">name<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;C&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">append<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">Categorical<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;linear&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;poly&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;rbf&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;sigmoid&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">name<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;kernel&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">append<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-t\">Integer<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">name<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;degree&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">append<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">Real<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1e<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100.0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;log-uniform&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">name<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;gamma&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0006 seconds] --><\/p>\n<p>Note the data type, the range, and the name of the hyperparameter specified for each.<\/p>\n<p>We can then define a function that will be called by the search procedure. This is a function expected by the optimization procedure later and takes a model and set of specific hyperparameters for the model, evaluates it, and returns a score for the set of hyperparameters.<\/p>\n<p>In our case, we want to evaluate the model using repeated stratified 10-fold cross-validation on our ionosphere dataset. We want to maximize classification accuracy, e.g. find the set of model hyperparameters that give the best accuracy. By default, the process minimizes the score returned from this function, therefore, we will return one minus the accuracy, e.g. perfect skill will be (1 \u2013 accuracy) or 0.0, and the worst skill will be 1.0.<\/p>\n<p>The <em>evaluate_model()<\/em> function below implements this and takes a specific set of hyperparameters.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc239a822847379\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# define the function used to evaluate a given configuration<br \/>\n@use_named_args(search_space)<br \/>\ndef evaluate_model(**params):<br \/>\n\t# configure the model with specific hyperparameters<br \/>\n\tmodel = SVC()<br \/>\n\tmodel.set_params(**params)<br \/>\n\t# define test harness<br \/>\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)<br \/>\n\t# calculate 5-fold cross validation<br \/>\n\tresult = cross_val_score(model, X, y, cv=cv, n_jobs=-1, scoring=&#8217;accuracy&#8217;)<br \/>\n\t# calculate the mean of the scores<br \/>\n\testimate = mean(result)<br \/>\n\t# convert from a maximizing score to a minimizing score<br \/>\n\treturn 1.0 &#8211; estimate<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># define the function used to evaluate a given configuration<\/span><\/p>\n<p><span class=\"crayon-sy\">@<\/span><span class=\"crayon-e\">use_named_args<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">evaluate_model<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># configure the model with specific hyperparameters<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">SVC<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">set_params<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># define test harness<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">RepeatedStratifiedKFold<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n_splits<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">10<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_repeats<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">random_state<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># calculate 5-fold cross validation<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">cross_val_score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_jobs<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">scoring<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;accuracy&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># calculate the mean of the scores<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">estimate<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">mean<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># convert from a maximizing score to a minimizing score<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1.0<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">estimate<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0007 seconds] --><\/p>\n<p>Next, we can execute the search by calling the <em>gp_minimize()<\/em> function and passing the name of the function to call to evaluate each model and the search space to optimize.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc239b095547643\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# perform optimization<br \/>\nresult = gp_minimize(evaluate_model, search_space)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># perform optimization<\/span><\/p>\n<p><span class=\"crayon-v\">result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">gp_minimize<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">evaluate_model<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0001 seconds] --><\/p>\n<p>The procedure will run until it converges and returns a result.<\/p>\n<p>The result object contains lots of details, but importantly, we can access the score of the best performing configuration and the hyperparameters used by the best forming model.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc239c829845057\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# summarizing finding:<br \/>\nprint(&#8216;Best Accuracy: %.3f&#8217; % (1.0 &#8211; result.fun))<br \/>\nprint(&#8216;Best Parameters: %s&#8217; % (result.x))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># summarizing finding:<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;Best Accuracy: %.3f&#8217;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1.0<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">fun<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;Best Parameters: %s&#8217;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0002 seconds] --><\/p>\n<p>Tying this together, the complete example of manually tuning the hyperparameters of an SVM on the ionosphere dataset is listed below.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc239d677773031\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# manually tune svm model hyperparameters using skopt on the ionosphere dataset<br \/>\nfrom numpy import mean<br \/>\nfrom pandas import read_csv<br \/>\nfrom sklearn.model_selection import cross_val_score<br \/>\nfrom sklearn.model_selection import RepeatedStratifiedKFold<br \/>\nfrom sklearn.svm import SVC<br \/>\nfrom skopt.space import Integer<br \/>\nfrom skopt.space import Real<br \/>\nfrom skopt.space import Categorical<br \/>\nfrom skopt.utils import use_named_args<br \/>\nfrom skopt import gp_minimize<\/p>\n<p># define the space of hyperparameters to search<br \/>\nsearch_space = list()<br \/>\nsearch_space.append(Real(1e-6, 100.0, &#8216;log-uniform&#8217;, name=&#8217;C&#8217;))<br \/>\nsearch_space.append(Categorical([&#8216;linear&#8217;, &#8216;poly&#8217;, &#8216;rbf&#8217;, &#8216;sigmoid&#8217;], name=&#8217;kernel&#8217;))<br \/>\nsearch_space.append(Integer(1, 5, name=&#8217;degree&#8217;))<br \/>\nsearch_space.append(Real(1e-6, 100.0, &#8216;log-uniform&#8217;, name=&#8217;gamma&#8217;))<\/p>\n<p># define the function used to evaluate a given configuration<br \/>\n@use_named_args(search_space)<br \/>\ndef evaluate_model(**params):<br \/>\n\t# configure the model with specific hyperparameters<br \/>\n\tmodel = SVC()<br \/>\n\tmodel.set_params(**params)<br \/>\n\t# define test harness<br \/>\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)<br \/>\n\t# calculate 5-fold cross validation<br \/>\n\tresult = cross_val_score(model, X, y, cv=cv, n_jobs=-1, scoring=&#8217;accuracy&#8217;)<br \/>\n\t# calculate the mean of the scores<br \/>\n\testimate = mean(result)<br \/>\n\t# convert from a maximizing score to a minimizing score<br \/>\n\treturn 1.0 &#8211; estimate<\/p>\n<p># load dataset<br \/>\nurl = &#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/ionosphere.csv&#8217;<br \/>\ndataframe = read_csv(url, header=None)<br \/>\n# split into input and output elements<br \/>\ndata = dataframe.values<br \/>\nX, y = data[:, :-1], data[:, -1]<br \/>\nprint(X.shape, y.shape)<br \/>\n# perform optimization<br \/>\nresult = gp_minimize(evaluate_model, search_space)<br \/>\n# summarizing finding:<br \/>\nprint(&#8216;Best Accuracy: %.3f&#8217; % (1.0 &#8211; result.fun))<br \/>\nprint(&#8216;Best Parameters: %s&#8217; % (result.x))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<p>39<\/p>\n<p>40<\/p>\n<p>41<\/p>\n<p>42<\/p>\n<p>43<\/p>\n<p>44<\/p>\n<p>45<\/p>\n<p>46<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># manually tune svm model hyperparameters using skopt on the ionosphere dataset<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">numpy <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">mean<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">pandas <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">read_csv<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">model_selection <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">cross_val_score<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">model_selection <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">RepeatedStratifiedKFold<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">svm <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">SVC<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">skopt<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">space <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-t\">Integer<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">skopt<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">space <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">Real<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">skopt<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">space <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">Categorical<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">skopt<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">utils <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">use_named_args<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">skopt <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">gp<\/span><span class=\"crayon-sy\">_<\/span>minimize<\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># define the space of hyperparameters to search<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">append<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">Real<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1e<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100.0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;log-uniform&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">name<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;C&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">append<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">Categorical<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;linear&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;poly&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;rbf&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;sigmoid&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">name<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;kernel&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">append<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-t\">Integer<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">name<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;degree&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">append<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">Real<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1e<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100.0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;log-uniform&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">name<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;gamma&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># define the function used to evaluate a given configuration<\/span><\/p>\n<p><span class=\"crayon-sy\">@<\/span><span class=\"crayon-e\">use_named_args<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">evaluate_model<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># configure the model with specific hyperparameters<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">SVC<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">set_params<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># define test harness<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">RepeatedStratifiedKFold<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n_splits<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">10<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_repeats<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">random_state<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># calculate 5-fold cross validation<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">cross_val_score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_jobs<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">scoring<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8216;accuracy&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># calculate the mean of the scores<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-v\">estimate<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">mean<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-p\"># convert from a maximizing score to a minimizing score<\/span><\/p>\n<p><span class=\"crayon-h\">\t<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1.0<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">estimate<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># load dataset<\/span><\/p>\n<p><span class=\"crayon-v\">url<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/ionosphere.csv&#8217;<\/span><\/p>\n<p><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">read_csv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">url<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">header<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into input and output elements<\/span><\/p>\n<p><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">values<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># perform optimization<\/span><\/p>\n<p><span class=\"crayon-v\">result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">gp_minimize<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">evaluate_model<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">search_space<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># summarizing finding:<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;Best Accuracy: %.3f&#8217;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1.0<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">fun<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;Best Parameters: %s&#8217;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0024 seconds] --><\/p>\n<p>Running the example may take a few moments, depending on the speed of your machine.<\/p>\n<p>You may see some warning messages that you can safely ignore, such as:<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc239e725351243\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nUserWarning: The objective has been evaluated at this point before.<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>UserWarning: The objective has been evaluated at this point before.<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>At the end of the run, the best-performing configuration is reported.<\/p>\n<p><strong>Note<\/strong>: Your <a href=\"https:\/\/machinelearningmastery.com\/different-results-each-time-in-machine-learning\/\">results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n<p>In this case, we can see that configuration, reported in order of the search space list, was a modest <em>C<\/em> value, a RBF <em>kernel<\/em>, a <em>degree<\/em> of 2 (ignored by the RBF kernel), and a modest <em>gamma<\/em> value.<\/p>\n<p>Importantly, we can see that the skill of this model was approximately 94.7 percent, which is a top-performing model<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc239f933119327\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n(351, 34) (351,)<br \/>\nBest Accuracy: 0.948<br \/>\nBest Parameters: [1.2852670137769258, &#8216;rbf&#8217;, 2, 0.18178016885627174]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>(351, 34) (351,)<\/p>\n<p>Best Accuracy: 0.948<\/p>\n<p>Best Parameters: [1.2852670137769258, &#8216;rbf&#8217;, 2, 0.18178016885627174]<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>This is not the only way to use the Scikit-Optimize library for hyperparameter tuning. In the next section, we can see a more automated approach.<\/p>\n<h2>Automatically Tune Algorithm Hyperparameters<\/h2>\n<p>The Scikit-Learn machine learning library provides tools for tuning model hyperparameters.<\/p>\n<p>Specifically, it provides the <a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.model_selection.GridSearchCV.html\">GridSearchCV<\/a> and <a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.model_selection.RandomizedSearchCV.html\">RandomizedSearchCV<\/a> classes that take a model, a search space, and a cross-validation configuration.<\/p>\n<p>The benefit of these classes is that the search procedure is performed automatically, requiring minimal configuration.<\/p>\n<p>Similarly, the Scikit-Optimize library provides a similar interface for performing a Bayesian Optimization of model hyperparameters via the <a href=\"https:\/\/scikit-optimize.github.io\/modules\/generated\/skopt.BayesSearchCV.html\">BayesSearchCV class<\/a>.<\/p>\n<p>This class can be used in the same way as the Scikit-Learn equivalents.<\/p>\n<p>First, the search space must be defined as a dictionary with hyperparameter names used as the key and the scope of the variable as the value.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc23a0726468107\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# define search space<br \/>\nparams = dict()<br \/>\nparams[&#8216;C&#8217;] = (1e-6, 100.0, &#8216;log-uniform&#8217;)<br \/>\nparams[&#8216;gamma&#8217;] = (1e-6, 100.0, &#8216;log-uniform&#8217;)<br \/>\nparams[&#8216;degree&#8217;] = (1,5)<br \/>\nparams[&#8216;kernel&#8217;] = [&#8216;linear&#8217;, &#8216;poly&#8217;, &#8216;rbf&#8217;, &#8216;sigmoid&#8217;]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># define search space<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">dict<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;C&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1e<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100.0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;log-uniform&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;gamma&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1e<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100.0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;log-uniform&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;degree&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;kernel&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;linear&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;poly&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;rbf&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;sigmoid&#8217;<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0005 seconds] --><\/p>\n<p>We can then define the <em>BayesSearchCV<\/em> configuration taking the model we wish to evaluate, the hyperparameter search space, and the cross-validation configuration.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc23a1511963630\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# define evaluation<br \/>\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)<br \/>\n# define the search<br \/>\nsearch = BayesSearchCV(estimator=SVC(), search_spaces=params, n_jobs=-1, cv=cv)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># define evaluation<\/span><\/p>\n<p><span class=\"crayon-v\">cv<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">RepeatedStratifiedKFold<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n_splits<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">10<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_repeats<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">random_state<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># define the search<\/span><\/p>\n<p><span class=\"crayon-v\">search<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">BayesSearchCV<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">estimator<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">SVC<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">search_spaces<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_jobs<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0004 seconds] --><\/p>\n<p>We can then execute the search and report the best result and configuration at the end.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc23a2203377248\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# perform the search<br \/>\nsearch.fit(X, y)<br \/>\n# report the best result<br \/>\nprint(search.best_score_)<br \/>\nprint(search.best_params_)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># perform the search<\/span><\/p>\n<p><span class=\"crayon-v\">search<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># report the best result<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">search<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">best_score_<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">search<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">best_params_<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0002 seconds] --><\/p>\n<p>Tying this together, the complete example of automatically tuning SVM hyperparameters using the BayesSearchCV class on the ionosphere dataset is listed below.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc23a3605263802\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# automatic svm hyperparameter tuning using skopt for the ionosphere dataset<br \/>\nfrom pandas import read_csv<br \/>\nfrom sklearn.model_selection import cross_val_score<br \/>\nfrom sklearn.svm import SVC<br \/>\nfrom sklearn.model_selection import RepeatedStratifiedKFold<br \/>\nfrom skopt import BayesSearchCV<br \/>\n# load dataset<br \/>\nurl = &#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/ionosphere.csv&#8217;<br \/>\ndataframe = read_csv(url, header=None)<br \/>\n# split into input and output elements<br \/>\ndata = dataframe.values<br \/>\nX, y = data[:, :-1], data[:, -1]<br \/>\nprint(X.shape, y.shape)<br \/>\n# define search space<br \/>\nparams = dict()<br \/>\nparams[&#8216;C&#8217;] = (1e-6, 100.0, &#8216;log-uniform&#8217;)<br \/>\nparams[&#8216;gamma&#8217;] = (1e-6, 100.0, &#8216;log-uniform&#8217;)<br \/>\nparams[&#8216;degree&#8217;] = (1,5)<br \/>\nparams[&#8216;kernel&#8217;] = [&#8216;linear&#8217;, &#8216;poly&#8217;, &#8216;rbf&#8217;, &#8216;sigmoid&#8217;]<br \/>\n# define evaluation<br \/>\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)<br \/>\n# define the search<br \/>\nsearch = BayesSearchCV(estimator=SVC(), search_spaces=params, n_jobs=-1, cv=cv)<br \/>\n# perform the search<br \/>\nsearch.fit(X, y)<br \/>\n# report the best result<br \/>\nprint(search.best_score_)<br \/>\nprint(search.best_params_)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># automatic svm hyperparameter tuning using skopt for the ionosphere dataset<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">pandas <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">read_csv<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">model_selection <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">cross_val_score<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">svm <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">SVC<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">model_selection <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">RepeatedStratifiedKFold<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">skopt <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-i\">BayesSearchCV<\/span><\/p>\n<p><span class=\"crayon-p\"># load dataset<\/span><\/p>\n<p><span class=\"crayon-v\">url<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/ionosphere.csv&#8217;<\/span><\/p>\n<p><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">read_csv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">url<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">header<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into input and output elements<\/span><\/p>\n<p><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">values<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># define search space<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">dict<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;C&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1e<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100.0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;log-uniform&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;gamma&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1e<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100.0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;log-uniform&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;degree&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;kernel&#8217;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;linear&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;poly&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;rbf&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;sigmoid&#8217;<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-p\"># define evaluation<\/span><\/p>\n<p><span class=\"crayon-v\">cv<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">RepeatedStratifiedKFold<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n_splits<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">10<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_repeats<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">random_state<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># define the search<\/span><\/p>\n<p><span class=\"crayon-v\">search<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">BayesSearchCV<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">estimator<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">SVC<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">search_spaces<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">params<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n_jobs<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">cv<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># perform the search<\/span><\/p>\n<p><span class=\"crayon-v\">search<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># report the best result<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">search<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">best_score_<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">search<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">best_params_<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0015 seconds] --><\/p>\n<p>Running the example may take a few moments, depending on the speed of your machine.<\/p>\n<p>You may see some warning messages that you can safely ignore, such as:<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc23a4697960476\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nUserWarning: The objective has been evaluated at this point before.<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>UserWarning: The objective has been evaluated at this point before.<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>At the end of the run, the best-performing configuration is reported.<\/p>\n<p><strong>Note<\/strong>: Your <a href=\"https:\/\/machinelearningmastery.com\/different-results-each-time-in-machine-learning\/\">results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n<p>In this case, we can see that the model performed above top-performing models achieving a mean classification accuracy of about 95.2 percent.<\/p>\n<p>The search discovered a large <em>C<\/em> value, an RBF <em>kernel<\/em>, and a small <em>gamma<\/em> value.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f556a7cc23a5167280286\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n(351, 34) (351,)<br \/>\n0.9525166191832859<br \/>\nOrderedDict([(&#8216;C&#8217;, 4.8722263953328735), (&#8216;degree&#8217;, 4), (&#8216;gamma&#8217;, 0.09805881007239009), (&#8216;kernel&#8217;, &#8216;rbf&#8217;)])<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>(351, 34) (351,)<\/p>\n<p>0.9525166191832859<\/p>\n<p>OrderedDict([(&#8216;C&#8217;, 4.8722263953328735), (&#8216;degree&#8217;, 4), (&#8216;gamma&#8217;, 0.09805881007239009), (&#8216;kernel&#8217;, &#8216;rbf&#8217;)])<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>This provides a template that you can use to tune the hyperparameters on your machine learning project.<\/p>\n<h2>Further Reading<\/h2>\n<p>This section provides more resources on the topic if you are looking to go deeper.<\/p>\n<h3>Related Tutorials<\/h3>\n<h3>APIs<\/h3>\n<h2>Summary<\/h2>\n<p>In this tutorial, you discovered how to use the Scikit-Optimize library to use Bayesian Optimization for hyperparameter tuning.<\/p>\n<p>Specifically, you learned:<\/p>\n<ul>\n<li>Scikit-Optimize provides a general toolkit for Bayesian Optimization that can be used for hyperparameter tuning.<\/li>\n<li>How to manually use the Scikit-Optimize library to tune the hyperparameters of a machine learning model.<\/li>\n<li>How to use the built-in BayesSearchCV class to perform model hyperparameter tuning.<\/li>\n<\/ul>\n<p><strong>Do you have any questions?<\/strong><br \/>Ask your questions in the comments below and I will do my best to answer.<\/p>\n<div class=\"widget_text awac-wrapper\" id=\"custom_html-78\">\n<div class=\"widget_text awac widget custom_html-78\">\n<div class=\"textwidget custom-html-widget\">\n<div>\n<h2>Discover Fast Machine Learning in Python!<\/h2>\n<p><a href=\"\/machine-learning-with-python\/\" rel=\"nofollow\"><img decoding=\"async\" src=\"https:\/\/3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com\/wp-content\/uploads\/2014\/07\/MachineLearningMasteryWithPython-220px.png\" alt=\"Master Machine Learning With Python\" align=\"left\"><\/a><\/p>\n<h4>Develop Your Own Models in Minutes<\/h4>\n<p>&#8230;with just a few lines of scikit-learn code<\/p>\n<p>Learn how in my new Ebook:<br \/><a href=\"\/machine-learning-with-python\/\" rel=\"nofollow\">Machine Learning Mastery With Python<\/a><\/p>\n<p>Covers <strong>self-study tutorials<\/strong> and <strong>end-to-end projects<\/strong> like:<br \/><em>Loading data<\/em>, <em>visualization<\/em>, <em>modeling<\/em>, <em>tuning<\/em>, and much more&#8230;<\/p>\n<h4>Finally Bring Machine Learning To<br \/>Your Own Projects<\/h4>\n<p>Skip the Academics. Just Results.<\/p>\n<p><a href=\"\/machine-learning-with-python\/\" class=\"woo-sc-button  red\"><span class=\"woo-\">See What&#8217;s Inside<\/span><\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/machinelearningmastery.com\/scikit-optimize-for-hyperparameter-tuning-in-machine-learning\/<\/p>\n","protected":false},"author":0,"featured_media":188,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/187"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=187"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/187\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/188"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=187"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=187"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=187"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}