{"id":224,"date":"2020-09-12T16:02:05","date_gmt":"2020-09-12T16:02:05","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/09\/12\/hyperopt-for-automated-machine-learning-with-scikit-learn\/"},"modified":"2020-09-12T16:02:05","modified_gmt":"2020-09-12T16:02:05","slug":"hyperopt-for-automated-machine-learning-with-scikit-learn","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/09\/12\/hyperopt-for-automated-machine-learning-with-scikit-learn\/","title":{"rendered":"HyperOpt for Automated Machine Learning With Scikit-Learn"},"content":{"rendered":"<div id=\"\">\n<p>Automated Machine Learning (AutoML) refers to techniques for automatically discovering well-performing models for predictive modeling tasks with very little user involvement.<\/p>\n<p>HyperOpt is an open-source library for large scale AutoML and HyperOpt-Sklearn is a wrapper for HyperOpt that supports AutoML with HyperOpt for the popular Scikit-Learn machine learning library, including the suite of data preparation transforms and classification and regression algorithms.<\/p>\n<p>In this tutorial, you will discover how to use HyperOpt for automatic machine learning with Scikit-Learn in Python.<\/p>\n<p>After completing this tutorial, you will know:<\/p>\n<ul>\n<li>Hyperopt-Sklearn is an open-source library for AutoML with scikit-learn data preparation and machine learning models.<\/li>\n<li>How to use Hyperopt-Sklearn to automatically discover top-performing models for classification tasks.<\/li>\n<li>How to use Hyperopt-Sklearn to automatically discover top-performing models for regression tasks.<\/li>\n<\/ul>\n<p>Let\u2019s get started.<\/p>\n<div id=\"attachment_10491\" class=\"wp-caption aligncenter\">\n<img decoding=\"async\" aria-describedby=\"caption-attachment-10491\" loading=\"lazy\" class=\"size-full wp-image-10491\" src=\"https:\/\/3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com\/wp-content\/uploads\/2020\/06\/HyperOpt-for-Automated-Machine-Learning-With-Scikit-Learn.jpg\" alt=\"HyperOpt for Automated Machine Learning With Scikit-Learn\" width=\"800\" height=\"532\"><\/p>\n<p id=\"caption-attachment-10491\" class=\"wp-caption-text\">HyperOpt for Automated Machine Learning With Scikit-Learn<br \/>Photo by <a href=\"https:\/\/flickr.com\/photos\/neillwphoto\/22975134619\/\">Neil Williamson<\/a>, some rights reserved.<\/p>\n<\/div>\n<h2>Tutorial Overview<\/h2>\n<p>This tutorial is divided into four parts; they are:<\/p>\n<ol>\n<li>HyperOpt and HyperOpt-Sklearn<\/li>\n<li>How to Install and Use HyperOpt-Sklearn<\/li>\n<li>HyperOpt-Sklearn for Classification<\/li>\n<li>HyperOpt-Sklearn for Regression<\/li>\n<\/ol>\n<h2>HyperOpt and HyperOpt-Sklearn<\/h2>\n<p><a href=\"http:\/\/hyperopt.github.io\/hyperopt\/\">HyperOpt<\/a> is an open-source Python library for Bayesian optimization developed by <a href=\"https:\/\/www.linkedin.com\/in\/james-bergstra\">James Bergstra<\/a>.<\/p>\n<p>It is designed for large-scale optimization for models with hundreds of parameters and allows the optimization procedure to be scaled across multiple cores and multiple machines.<\/p>\n<p>The library was explicitly used to optimize machine learning pipelines, including data preparation, model selection, and model hyperparameters.<\/p>\n<blockquote>\n<p>Our approach is to expose the underlying expression graph of how a performance metric (e.g. classification accuracy on validation examples) is computed from hyperparameters that govern not only how individual processing steps are applied, but even which processing steps are included.<\/p>\n<\/blockquote>\n<p>\u2014 <a href=\"https:\/\/dl.acm.org\/doi\/10.5555\/3042817.3042832\">Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures<\/a>, 2013.<\/p>\n<p>HyperOpt is challenging to use directly, requiring the optimization procedure and search space to be carefully specified.<\/p>\n<p>An extension to HyperOpt was created called <a href=\"http:\/\/hyperopt.github.io\/hyperopt-sklearn\/\">HyperOpt-Sklearn<\/a> that allows the HyperOpt procedure to be applied to data preparation and machine learning models provided by the popular <a href=\"https:\/\/scikit-learn.org\/stable\/\">Scikit-Learn open-source machine learning library<\/a>.<\/p>\n<p>HyperOpt-Sklearn wraps the HyperOpt library and allows for the automatic search of data preparation methods, machine learning algorithms, and model hyperparameters for classification and regression tasks.<\/p>\n<blockquote>\n<p>\u2026 we introduce Hyperopt-Sklearn: a project that brings the benefits of automatic algorithm configuration to users of Python and scikit-learn. Hyperopt-Sklearn uses Hyperopt to describe a search space over possible configurations of Scikit-Learn components, including preprocessing and classification modules.<\/p>\n<\/blockquote>\n<p>\u2014 <a href=\"https:\/\/conference.scipy.org\/proceedings\/scipy2014\/pdfs\/komer.pdf\">Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn<\/a>, 2014.<\/p>\n<p>Now that we are familiar with HyperOpt and HyperOpt-Sklearn, let\u2019s look at how to use HyperOpt-Sklearn.<\/p>\n<h2>How to Install and Use HyperOpt-Sklearn<\/h2>\n<p>The first step is to install the HyperOpt library.<\/p>\n<p>This can be achieved using the pip package manager as follows:<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020d6027815717\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nsudo pip install hyperopt<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>sudo pip install hyperopt<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0001 seconds] --><\/p>\n<p>Once installed, we can confirm that the installation was successful and check the version of the library by typing the following command:<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>This will summarize the installed version of HyperOpt, confirming that a modern version is being used.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020dc356387277\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nName: hyperopt<br \/>\nVersion: 0.2.3<br \/>\nSummary: Distributed Asynchronous Hyperparameter Optimization<br \/>\nHome-page: http:\/\/hyperopt.github.com\/hyperopt\/<br \/>\nAuthor: James Bergstra<br \/>\nAuthor-email: james.bergstra@gmail.com<br \/>\nLicense: BSD<br \/>\nLocation: &#8230;<br \/>\nRequires: tqdm, six, networkx, future, scipy, cloudpickle, numpy<br \/>\nRequired-by:<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>Name: hyperopt<\/p>\n<p>Version: 0.2.3<\/p>\n<p>Summary: Distributed Asynchronous Hyperparameter Optimization<\/p>\n<p>Home-page: http:\/\/hyperopt.github.com\/hyperopt\/<\/p>\n<p>Author: James Bergstra<\/p>\n<p>Author-email: james.bergstra@gmail.com<\/p>\n<p>License: BSD<\/p>\n<p>Location: &#8230;<\/p>\n<p>Requires: tqdm, six, networkx, future, scipy, cloudpickle, numpy<\/p>\n<p>Required-by:<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>Next, we must install the HyperOpt-Sklearn library.<\/p>\n<p>This too can be installed using pip, although we must perform this operation manually by cloning the repository and running the installation from the local files, as follows:<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020dd914146214\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\ngit clone git@github.com:hyperopt\/hyperopt-sklearn.git<br \/>\ncd hyperopt-sklearn<br \/>\nsudo pip install .<br \/>\ncd ..<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>git clone git@github.com:hyperopt\/hyperopt-sklearn.git<\/p>\n<p>cd hyperopt-sklearn<\/p>\n<p>sudo pip install .<\/p>\n<p>cd ..<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>Again, we can confirm that the installation was successful by checking the version number with the following command:<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>This will summarize the installed version of HyperOpt-Sklearn, confirming that a modern version is being used.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020df559400342\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nName: hpsklearn<br \/>\nVersion: 0.0.3<br \/>\nSummary: Hyperparameter Optimization for sklearn<br \/>\nHome-page: http:\/\/hyperopt.github.com\/hyperopt-sklearn\/<br \/>\nAuthor: James Bergstra<br \/>\nAuthor-email: anon@anon.com<br \/>\nLicense: BSD<br \/>\nLocation: &#8230;<br \/>\nRequires: nose, scikit-learn, numpy, scipy, hyperopt<br \/>\nRequired-by:<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>Name: hpsklearn<\/p>\n<p>Version: 0.0.3<\/p>\n<p>Summary: Hyperparameter Optimization for sklearn<\/p>\n<p>Home-page: http:\/\/hyperopt.github.com\/hyperopt-sklearn\/<\/p>\n<p>Author: James Bergstra<\/p>\n<p>Author-email: anon@anon.com<\/p>\n<p>License: BSD<\/p>\n<p>Location: &#8230;<\/p>\n<p>Requires: nose, scikit-learn, numpy, scipy, hyperopt<\/p>\n<p>Required-by:<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>Now that the required libraries are installed, we can review the HyperOpt-Sklearn API.<\/p>\n<p>Using HyperOpt-Sklearn is straightforward. The search process is defined by creating and configuring an instance of the HyperoptEstimator class.<\/p>\n<p>The algorithm used for the search can be specified via the \u201c<em>algo<\/em>\u201d argument, the number of evaluations performed in the search is specified via the \u201c<em>max_evals<\/em>\u201d argument, and a limit can be imposed on evaluating each pipeline via the \u201c<em>trial_timeout<\/em>\u201d argument.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020e0544781796\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# define search<br \/>\nmodel = HyperoptEstimator(&#8230;, algo=tpe.suggest, max_evals=50, trial_timeout=120)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># define search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">HyperoptEstimator<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">algo<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">tpe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">suggest<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">max_evals<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">50<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">trial_timeout<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">120<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0002 seconds] --><\/p>\n<p>Many different optimization algorithms are available, including:<\/p>\n<ul>\n<li>Random Search<\/li>\n<li>Tree of Parzen Estimators<\/li>\n<li>Annealing<\/li>\n<li>Tree<\/li>\n<li>Gaussian Process Tree<\/li>\n<\/ul>\n<p>The \u201c<em>Tree of Parzen Estimators<\/em>\u201d is a good default, and you can learn more about the types of algorithms in the paper \u201c<a href=\"https:\/\/papers.nips.cc\/paper\/4443-algorithms-for-hyper-parameter-optimization.pdf\">Algorithms for Hyper-Parameter Optimization<\/a>. [PDF]\u201d<\/p>\n<p>For classification tasks, the \u201c<em>classifier<\/em>\u201d argument specifies the search space of models, and for regression, the \u201c<em>regressor<\/em>\u201d argument specifies the search space of models, both of which can be set to use predefined lists of models provided by the library, e.g. \u201c<em>any_classifier<\/em>\u201d and \u201c<em>any_regressor<\/em>\u201c.<\/p>\n<p>Similarly, the search space of data preparation is specified via the \u201c<em>preprocessing<\/em>\u201d argument and can also use a pre-defined list of preprocessing steps via \u201c<em>any_preprocessing<\/em>.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020e1443491792\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# define search<br \/>\nmodel = HyperoptEstimator(classifier=any_classifier(&#8216;cla&#8217;), preprocessing=any_preprocessing(&#8216;pre&#8217;), &#8230;)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># define search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">HyperoptEstimator<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">classifier<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_classifier<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;cla&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">preprocessing<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_preprocessing<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;pre&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0002 seconds] --><\/p>\n<p>For more on the other arguments to the search, you can review the source code for the class directly:<\/p>\n<p>Once the search is defined, it can be executed by calling the <em>fit()<\/em> function.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020e2813110173\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# perform the search<br \/>\nmodel.fit(X_train, y_train)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># perform the search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_train<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0001 seconds] --><\/p>\n<p>At the end of the run, the best-performing model can be evaluated on new data by calling the <em>score()<\/em> function.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020e3093780712\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# summarize performance<br \/>\nacc = model.score(X_test, y_test)<br \/>\nprint(&#8220;Accuracy: %.3f&#8221; % acc)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># summarize performance<\/span><\/p>\n<p><span class=\"crayon-v\">acc<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X_test<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_test<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Accuracy: %.3f&#8221;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">acc<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0002 seconds] --><\/p>\n<p>Finally, we can retrieve the <em>Pipeline<\/em> of transforms, models, and model configurations that performed the best on the training dataset via the <em>best_model()<\/em> function.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020e6215213266\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# summarize the best model<br \/>\nprint(model.best_model())<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># summarize the best model<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">best_model<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0001 seconds] --><\/p>\n<p>Now that we are familiar with the API, let\u2019s look at some worked examples.<\/p>\n<h2>HyperOpt-Sklearn for Classification<\/h2>\n<p>In this section, we will use HyperOpt-Sklearn to discover a model for the sonar dataset.<\/p>\n<p>The sonar dataset is a standard machine learning dataset comprised of 208 rows of data with 60 numerical input variables and a target variable with two class values, e.g. binary classification.<\/p>\n<p>Using a test harness of repeated stratified 10-fold cross-validation with three repeats, a naive model can achieve an accuracy of about 53 percent. A top-performing model can achieve accuracy on this same test harness of about 88 percent. This provides the bounds of expected performance on this dataset.<\/p>\n<p>The dataset involves predicting whether sonar returns indicate a rock or simulated mine.<\/p>\n<p>No need to download the dataset; we will download it automatically as part of our worked examples.<\/p>\n<p>The example below downloads the dataset and summarizes its shape.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020e7154080885\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# summarize the sonar dataset<br \/>\nfrom pandas import read_csv<br \/>\n# load dataset<br \/>\nurl = &#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv&#8217;<br \/>\ndataframe = read_csv(url, header=None)<br \/>\n# split into input and output elements<br \/>\ndata = dataframe.values<br \/>\nX, y = data[:, :-1], data[:, -1]<br \/>\nprint(X.shape, y.shape)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># summarize the sonar dataset<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">pandas <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">read<\/span><span class=\"crayon-sy\">_<\/span>csv<\/p>\n<p><span class=\"crayon-p\"># load dataset<\/span><\/p>\n<p><span class=\"crayon-v\">url<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv&#8217;<\/span><\/p>\n<p><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">read_csv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">url<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">header<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into input and output elements<\/span><\/p>\n<p><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">values<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0005 seconds] --><\/p>\n<p>Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 208 rows of data with 60 input variables.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>Next, let\u2019s use HyperOpt-Sklearn to find a good model for the sonar dataset.<\/p>\n<p>We can perform some basic data preparation, including converting the target string to class labels, then split the dataset into train and test sets.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020e9734848118\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# minimally prepare dataset<br \/>\nX = X.astype(&#8216;float32&#8217;)<br \/>\ny = LabelEncoder().fit_transform(y.astype(&#8216;str&#8217;))<br \/>\n# split into train and test sets<br \/>\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># minimally prepare dataset<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">astype<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;float32&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">LabelEncoder<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit_transform<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">astype<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;str&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into train and test sets<\/span><\/p>\n<p><span class=\"crayon-v\">X_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">X_test<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_test<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">train_test_split<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">test_size<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0.33<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">random_state<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0004 seconds] --><\/p>\n<p>Next, we can define the search procedure. We will explore all classification algorithms and all data transforms available to the library and use the TPE, or Tree of Parzen Estimators, search algorithm, described in \u201c<a href=\"https:\/\/papers.nips.cc\/paper\/4443-algorithms-for-hyper-parameter-optimization.pdf\">Algorithms for Hyper-Parameter Optimization<\/a>.\u201d<\/p>\n<p>The search will evaluate 50 pipelines and limit each evaluation to 30 seconds.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020ea179937544\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# define search<br \/>\nmodel = HyperoptEstimator(classifier=any_classifier(&#8216;cla&#8217;), preprocessing=any_preprocessing(&#8216;pre&#8217;), algo=tpe.suggest, max_evals=50, trial_timeout=30)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># define search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">HyperoptEstimator<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">classifier<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_classifier<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;cla&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">preprocessing<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_preprocessing<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;pre&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">algo<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">tpe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">suggest<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">max_evals<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">50<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">trial_timeout<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">30<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0002 seconds] --><\/p>\n<p>We then start the search.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020eb145172767\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# perform the search<br \/>\nmodel.fit(X_train, y_train)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># perform the search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_train<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0001 seconds] --><\/p>\n<p>At the end of the run, we will report the performance of the model on the holdout dataset and summarize the best performing pipeline.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020ec230180366\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# summarize performance<br \/>\nacc = model.score(X_test, y_test)<br \/>\nprint(&#8220;Accuracy: %.3f&#8221; % acc)<br \/>\n# summarize the best model<br \/>\nprint(model.best_model())<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># summarize performance<\/span><\/p>\n<p><span class=\"crayon-v\">acc<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X_test<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_test<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Accuracy: %.3f&#8221;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">acc<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># summarize the best model<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">best_model<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0003 seconds] --><\/p>\n<p>Tying this together, the complete example is listed below.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020ed653742271\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# example of hyperopt-sklearn for the sonar classification dataset<br \/>\nfrom pandas import read_csv<br \/>\nfrom sklearn.model_selection import train_test_split<br \/>\nfrom sklearn.preprocessing import LabelEncoder<br \/>\nfrom hpsklearn import HyperoptEstimator<br \/>\nfrom hpsklearn import any_classifier<br \/>\nfrom hpsklearn import any_preprocessing<br \/>\nfrom hyperopt import tpe<br \/>\n# load dataset<br \/>\nurl = &#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv&#8217;<br \/>\ndataframe = read_csv(url, header=None)<br \/>\n# split into input and output elements<br \/>\ndata = dataframe.values<br \/>\nX, y = data[:, :-1], data[:, -1]<br \/>\n# minimally prepare dataset<br \/>\nX = X.astype(&#8216;float32&#8217;)<br \/>\ny = LabelEncoder().fit_transform(y.astype(&#8216;str&#8217;))<br \/>\n# split into train and test sets<br \/>\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)<br \/>\n# define search<br \/>\nmodel = HyperoptEstimator(classifier=any_classifier(&#8216;cla&#8217;), preprocessing=any_preprocessing(&#8216;pre&#8217;), algo=tpe.suggest, max_evals=50, trial_timeout=30)<br \/>\n# perform the search<br \/>\nmodel.fit(X_train, y_train)<br \/>\n# summarize performance<br \/>\nacc = model.score(X_test, y_test)<br \/>\nprint(&#8220;Accuracy: %.3f&#8221; % acc)<br \/>\n# summarize the best model<br \/>\nprint(model.best_model())<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># example of hyperopt-sklearn for the sonar classification dataset<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">pandas <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">read_csv<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">model_selection <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">train_test_split<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">preprocessing <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">LabelEncoder<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">hpsklearn <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">HyperoptEstimator<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">hpsklearn <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">any_classifier<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">hpsklearn <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">any_preprocessing<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">hyperopt <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-i\">tpe<\/span><\/p>\n<p><span class=\"crayon-p\"># load dataset<\/span><\/p>\n<p><span class=\"crayon-v\">url<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/sonar.csv&#8217;<\/span><\/p>\n<p><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">read_csv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">url<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">header<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into input and output elements<\/span><\/p>\n<p><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">values<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-p\"># minimally prepare dataset<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">astype<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;float32&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">LabelEncoder<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit_transform<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">astype<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;str&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into train and test sets<\/span><\/p>\n<p><span class=\"crayon-v\">X_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">X_test<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_test<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">train_test_split<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">test_size<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0.33<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">random_state<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># define search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">HyperoptEstimator<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">classifier<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_classifier<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;cla&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">preprocessing<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_preprocessing<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;pre&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">algo<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">tpe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">suggest<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">max_evals<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">50<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">trial_timeout<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">30<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># perform the search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_train<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># summarize performance<\/span><\/p>\n<p><span class=\"crayon-v\">acc<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X_test<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_test<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Accuracy: %.3f&#8221;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">acc<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># summarize the best model<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">best_model<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0013 seconds] --><\/p>\n<p>Running the example may take a few minutes.<\/p>\n<p>The progress of the search will be reported and you will see some warnings that you can safely ignore.<\/p>\n<p>At the end of the run, the best-performing model is evaluated on the holdout dataset and the Pipeline discovered is printed for later use.<\/p>\n<p><strong>Note<\/strong>: Your <a href=\"https:\/\/machinelearningmastery.com\/different-results-each-time-in-machine-learning\/\">results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n<p>In this case, we can see that the chosen model achieved an accuracy of about 85.5 percent on the holdout test set. The Pipeline involves a gradient boosting model with no pre-processing.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020ee346361601\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nAccuracy: 0.855<br \/>\n{&#8216;learner&#8217;: GradientBoostingClassifier(ccp_alpha=0.0, criterion=&#8217;friedman_mse&#8217;, init=None,<br \/>\n                           learning_rate=0.009132299586303643, loss=&#8217;deviance&#8217;,<br \/>\n                           max_depth=None, max_features=&#8217;sqrt&#8217;,<br \/>\n                           max_leaf_nodes=None, min_impurity_decrease=0.0,<br \/>\n                           min_impurity_split=None, min_samples_leaf=1,<br \/>\n                           min_samples_split=2, min_weight_fraction_leaf=0.0,<br \/>\n                           n_estimators=342, n_iter_no_change=None,<br \/>\n                           presort=&#8217;auto&#8217;, random_state=2,<br \/>\n                           subsample=0.6844206624548879, tol=0.0001,<br \/>\n                           validation_fraction=0.1, verbose=0,<br \/>\n                           warm_start=False), &#8216;preprocs&#8217;: (), &#8216;ex_preprocs&#8217;: ()}<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>Accuracy: 0.855<\/p>\n<p>{&#8216;learner&#8217;: GradientBoostingClassifier(ccp_alpha=0.0, criterion=&#8217;friedman_mse&#8217;, init=None,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 learning_rate=0.009132299586303643, loss=&#8217;deviance&#8217;,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 max_depth=None, max_features=&#8217;sqrt&#8217;,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 max_leaf_nodes=None, min_impurity_decrease=0.0,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 min_impurity_split=None, min_samples_leaf=1,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 min_samples_split=2, min_weight_fraction_leaf=0.0,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 n_estimators=342, n_iter_no_change=None,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 presort=&#8217;auto&#8217;, random_state=2,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 subsample=0.6844206624548879, tol=0.0001,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 validation_fraction=0.1, verbose=0,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 warm_start=False), &#8216;preprocs&#8217;: (), &#8216;ex_preprocs&#8217;: ()}<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0001 seconds] --><\/p>\n<p>The printed model can then be used directly, e.g. the code copy-pasted into another project.<\/p>\n<p>Next, let\u2019s take a look at using HyperOpt-Sklearn for a regression predictive modeling problem.<\/p>\n<h2>HyperOpt-Sklearn for Regression<\/h2>\n<p>In this section, we will use HyperOpt-Sklearn to discover a model for the housing dataset.<\/p>\n<p>The housing dataset is a standard machine learning dataset comprised of 506 rows of data with 13 numerical input variables and a numerical target variable.<\/p>\n<p>Using a test harness of repeated stratified 10-fold cross-validation with three repeats, a naive model can achieve a mean absolute error (MAE) of about 6.6. A top-performing model can achieve a MAE on this same test harness of about 1.9. This provides the bounds of expected performance on this dataset.<\/p>\n<p>The dataset involves predicting the house price given details of the house suburb in the American city of Boston.<\/p>\n<p>No need to download the dataset; we will download it automatically as part of our worked examples.<\/p>\n<p>The example below downloads the dataset and summarizes its shape.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020ef720916130\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# summarize the auto insurance dataset<br \/>\nfrom pandas import read_csv<br \/>\n# load dataset<br \/>\nurl = &#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/housing.csv&#8217;<br \/>\ndataframe = read_csv(url, header=None)<br \/>\n# split into input and output elements<br \/>\ndata = dataframe.values<br \/>\nX, y = data[:, :-1], data[:, -1]<br \/>\nprint(X.shape, y.shape)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># summarize the auto insurance dataset<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">pandas <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">read<\/span><span class=\"crayon-sy\">_<\/span>csv<\/p>\n<p><span class=\"crayon-p\"># load dataset<\/span><\/p>\n<p><span class=\"crayon-v\">url<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/housing.csv&#8217;<\/span><\/p>\n<p><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">read_csv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">url<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">header<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into input and output elements<\/span><\/p>\n<p><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">values<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0005 seconds] --><\/p>\n<p>Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 63 rows of data with one input variable.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<p><!-- [Format Time: 0.0000 seconds] --><\/p>\n<p>Next, we can use HyperOpt-Sklearn to find a good model for the auto insurance dataset.<\/p>\n<p>Using HyperOpt-Sklearn for regression is the same as using it for classification, except the \u201c<em>regressor<\/em>\u201d argument must be specified.<\/p>\n<p>In this case, we want to optimize the MAE, therefore, we will set the \u201c<em>loss_fn<\/em>\u201d argument to the <em>mean_absolute_error()<\/em> function provided by the scikit-learn library.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020f1099432542\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<br \/>\n# define search<br \/>\nmodel = HyperoptEstimator(regressor=any_regressor(&#8216;reg&#8217;), preprocessing=any_preprocessing(&#8216;pre&#8217;), loss_fn=mean_absolute_error, algo=tpe.suggest, max_evals=50, trial_timeout=30)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-p\"># define search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">HyperoptEstimator<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">regressor<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_regressor<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;reg&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">preprocessing<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_preprocessing<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;pre&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">loss_fn<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">mean_absolute_error<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">algo<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">tpe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">suggest<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">max_evals<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">50<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">trial_timeout<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">30<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0003 seconds] --><\/p>\n<p>Tying this together, the complete example is listed below.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020f2634796883\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# example of hyperopt-sklearn for the housing regression dataset<br \/>\nfrom pandas import read_csv<br \/>\nfrom sklearn.model_selection import train_test_split<br \/>\nfrom sklearn.metrics import mean_absolute_error<br \/>\nfrom hpsklearn import HyperoptEstimator<br \/>\nfrom hpsklearn import any_regressor<br \/>\nfrom hpsklearn import any_preprocessing<br \/>\nfrom hyperopt import tpe<br \/>\n# load dataset<br \/>\nurl = &#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/housing.csv&#8217;<br \/>\ndataframe = read_csv(url, header=None)<br \/>\n# split into input and output elements<br \/>\ndata = dataframe.values<br \/>\ndata = data.astype(&#8216;float32&#8217;)<br \/>\nX, y = data[:, :-1], data[:, -1]<br \/>\n# split into train and test sets<br \/>\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)<br \/>\n# define search<br \/>\nmodel = HyperoptEstimator(regressor=any_regressor(&#8216;reg&#8217;), preprocessing=any_preprocessing(&#8216;pre&#8217;), loss_fn=mean_absolute_error, algo=tpe.suggest, max_evals=50, trial_timeout=30)<br \/>\n# perform the search<br \/>\nmodel.fit(X_train, y_train)<br \/>\n# summarize performance<br \/>\nmae = model.score(X_test, y_test)<br \/>\nprint(&#8220;MAE: %.3f&#8221; % mae)<br \/>\n# summarize the best model<br \/>\nprint(model.best_model())<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># example of hyperopt-sklearn for the housing regression dataset<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">pandas <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">read_csv<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">model_selection <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">train_test_split<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">metrics <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">mean_absolute_error<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">hpsklearn <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">HyperoptEstimator<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">hpsklearn <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">any_regressor<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">hpsklearn <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">any_preprocessing<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">hyperopt <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-i\">tpe<\/span><\/p>\n<p><span class=\"crayon-p\"># load dataset<\/span><\/p>\n<p><span class=\"crayon-v\">url<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/housing.csv&#8217;<\/span><\/p>\n<p><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">read_csv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">url<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">header<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># split into input and output elements<\/span><\/p>\n<p><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dataframe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">values<\/span><\/p>\n<p><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">astype<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;float32&#8217;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-p\"># split into train and test sets<\/span><\/p>\n<p><span class=\"crayon-v\">X_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">X_test<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_test<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">train_test_split<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">test_size<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0.33<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">random_state<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># define search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">HyperoptEstimator<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">regressor<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_regressor<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;reg&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">preprocessing<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-e\">any_preprocessing<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;pre&#8217;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">loss_fn<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">mean_absolute_error<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">algo<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">tpe<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">suggest<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">max_evals<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">50<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">trial_timeout<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">30<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># perform the search<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X_train<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_train<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># summarize performance<\/span><\/p>\n<p><span class=\"crayon-v\">mae<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X_test<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y_test<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;MAE: %.3f&#8221;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">mae<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># summarize the best model<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">best_model<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0013 seconds] --><\/p>\n<p>Running the example may take a few minutes.<\/p>\n<p>The progress of the search will be reported and you will see some warnings that you can safely ignore.<\/p>\n<p>At the end of the run, the best performing model is evaluated on the holdout dataset and the Pipeline discovered is printed for later use.<\/p>\n<p><strong>Note<\/strong>: Your <a href=\"https:\/\/machinelearningmastery.com\/different-results-each-time-in-machine-learning\/\">results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n<p>In this case, we can see that the chosen model achieved a MAE of about 0.883 on the holdout test set, which appears skillful. The Pipeline involves an <em>XGBRegressor<\/em> model with no pre-processing.<\/p>\n<p><strong>Note<\/strong>: for the search to use XGBoost, you must have the <a href=\"https:\/\/machinelearningmastery.com\/install-xgboost-python-macos\/\">XGBoost library installed<\/a>.<\/p>\n<p><!-- Urvanov Syntax Highlighter v2.8.13 --><\/p>\n<div id=\"urvanov-syntax-highlighter-5f5cef5a020f3121158216\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nMAE: 0.883<br \/>\n{&#8216;learner&#8217;: XGBRegressor(base_score=0.5, booster=&#8217;gbtree&#8217;,<br \/>\n             colsample_bylevel=0.5843250948679669, colsample_bynode=1,<br \/>\n             colsample_bytree=0.6635160670570662, gamma=6.923399395303031e-05,<br \/>\n             importance_type=&#8217;gain&#8217;, learning_rate=0.07021104887683309,<br \/>\n             max_delta_step=0, max_depth=3, min_child_weight=5, missing=nan,<br \/>\n             n_estimators=4000, n_jobs=1, nthread=None, objective=&#8217;reg:linear&#8217;,<br \/>\n             random_state=0, reg_alpha=0.5690202874759704,<br \/>\n             reg_lambda=3.3098341637038, scale_pos_weight=1, seed=1,<br \/>\n             silent=None, subsample=0.7194797262656784, verbosity=1), &#8216;preprocs&#8217;: (), &#8216;ex_preprocs&#8217;: ()}<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>MAE: 0.883<\/p>\n<p>{&#8216;learner&#8217;: XGBRegressor(base_score=0.5, booster=&#8217;gbtree&#8217;,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 colsample_bylevel=0.5843250948679669, colsample_bynode=1,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 colsample_bytree=0.6635160670570662, gamma=6.923399395303031e-05,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 importance_type=&#8217;gain&#8217;, learning_rate=0.07021104887683309,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 max_delta_step=0, max_depth=3, min_child_weight=5, missing=nan,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 n_estimators=4000, n_jobs=1, nthread=None, objective=&#8217;reg:linear&#8217;,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 random_state=0, reg_alpha=0.5690202874759704,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 reg_lambda=3.3098341637038, scale_pos_weight=1, seed=1,<\/p>\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 silent=None, subsample=0.7194797262656784, verbosity=1), &#8216;preprocs&#8217;: (), &#8216;ex_preprocs&#8217;: ()}<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<\/div>\n<p><!-- [Format Time: 0.0001 seconds] --><\/p>\n<h2>Further Reading<\/h2>\n<p>This section provides more resources on the topic if you are looking to go deeper.<\/p>\n<h2>Summary<\/h2>\n<p>In this tutorial, you discovered how to use HyperOpt for automatic machine learning with Scikit-Learn in Python.<\/p>\n<p>Specifically, you learned:<\/p>\n<ul>\n<li>Hyperopt-Sklearn is an open-source library for AutoML with scikit-learn data preparation and machine learning models.<\/li>\n<li>How to use Hyperopt-Sklearn to automatically discover top-performing models for classification tasks.<\/li>\n<li>How to use Hyperopt-Sklearn to automatically discover top-performing models for regression tasks.<\/li>\n<\/ul>\n<p><strong>Do you have any questions?<\/strong><br \/>Ask your questions in the comments below and I will do my best to answer.<\/p>\n<div class=\"widget_text awac-wrapper\" id=\"custom_html-78\">\n<div class=\"widget_text awac widget custom_html-78\">\n<div class=\"textwidget custom-html-widget\">\n<div>\n<h2>Discover Fast Machine Learning in Python!<\/h2>\n<p><a href=\"\/machine-learning-with-python\/\" rel=\"nofollow\"><img decoding=\"async\" src=\"https:\/\/3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com\/wp-content\/uploads\/2014\/07\/MachineLearningMasteryWithPython-220px.png\" alt=\"Master Machine Learning With Python\" align=\"left\"><\/a><\/p>\n<h4>Develop Your Own Models in Minutes<\/h4>\n<p>&#8230;with just a few lines of scikit-learn code<\/p>\n<p>Learn how in my new Ebook:<br \/><a href=\"\/machine-learning-with-python\/\" rel=\"nofollow\">Machine Learning Mastery With Python<\/a><\/p>\n<p>Covers <strong>self-study tutorials<\/strong> and <strong>end-to-end projects<\/strong> like:<br \/><em>Loading data<\/em>, <em>visualization<\/em>, <em>modeling<\/em>, <em>tuning<\/em>, and much more&#8230;<\/p>\n<h4>Finally Bring Machine Learning To<br \/>Your Own Projects<\/h4>\n<p>Skip the Academics. Just Results.<\/p>\n<p><a href=\"\/machine-learning-with-python\/\" class=\"woo-sc-button  red\"><span class=\"woo-\">See What&#8217;s Inside<\/span><\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/machinelearningmastery.com\/hyperopt-for-automated-machine-learning-with-scikit-learn\/<\/p>\n","protected":false},"author":0,"featured_media":225,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/224"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=224"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/224\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/225"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=224"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=224"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=224"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}