{"id":1419,"date":"2021-12-31T16:47:54","date_gmt":"2021-12-31T16:47:54","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/12\/31\/functional-programming-in-python\/"},"modified":"2021-12-31T16:47:54","modified_gmt":"2021-12-31T16:47:54","slug":"functional-programming-in-python","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/12\/31\/functional-programming-in-python\/","title":{"rendered":"Functional Programming In Python"},"content":{"rendered":"<div id=\"\">\n<p id=\"last-modified-info\">Last Updated on December 19, 2021<\/p>\n<p>Python is a fantastic programming language. It is likely to be your first choice for developing a machine learning or data science application. Python is interesting because it is a multi-paradigm programming language that can be used for both object-oriented and imperative programming. It has a simple syntax that is easy to read and comprehend.<\/p>\n<p>In computer science and mathematics, the solution of many problems can be more easily and naturally expressed using the functional programming style. In this tutorial, we\u2019ll discuss Python\u2019s support for the functional programming paradigm, and Python\u2019s classes and modules that help you program in this style.<\/p>\n<p>After completing this tutorial, you will know:<\/p>\n<ul>\n<li>Basic idea of functional programming<\/li>\n<li>The <code>itertools<\/code> library<\/li>\n<li>The <code>functools<\/code>\u00a0library<\/li>\n<li>Map-reduce design pattern and its possible implementation in Python<\/li>\n<\/ul>\n<p>Let\u2019s get started.<\/p>\n<div id=\"attachment_13133\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/12\/abdullahShakoortree-gdd40e365b_1920.jpg\"><img aria-describedby=\"caption-attachment-13133\" loading=\"lazy\" class=\"wp-image-13133 \" data-cfsrc=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/12\/abdullahShakoortree-gdd40e365b_1920-1024x661.jpg\" alt=\"\" width=\"628\" height=\"405\"><img decoding=\"async\" aria-describedby=\"caption-attachment-13133\" loading=\"lazy\" class=\"wp-image-13133 \" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/12\/abdullahShakoortree-gdd40e365b_1920-1024x661.jpg\" alt=\"\" width=\"628\" height=\"405\"><\/a><\/p>\n<p id=\"caption-attachment-13133\" class=\"wp-caption-text\">Functional Programming In Python <br \/>Photo by Abdullah_Shakoor, some rights reserved<\/p>\n<\/div>\n<h2 id=\"Tutorial-Overview\">Tutorial Overview<\/h2>\n<p>This tutorial is divided into 5 parts; they are:<\/p>\n<ol>\n<li>The idea of functional programming<\/li>\n<li>High order functions: Filter, map, and reduce<\/li>\n<li>Itertools<\/li>\n<li>Functools<\/li>\n<li>Map-reduce pattern<\/li>\n<\/ol>\n<h2>The idea of functional programming<\/h2>\n<p>If you have programming experience, likely you learned imperative programming. It is built with statements and manipulating variables. Functional programming is a <strong>declarative<\/strong> paradigm. It is different from imperative paradigm that programs are built by applying and composing functions. The functions here are supposed to be closer to the definition of a mathematical function, which there are <strong>no side effects<\/strong>, or simply, no access to external variables and when you call them with the same argument, they always give you the same result.<\/p>\n<p>The benefit of functional programming is to make your program less error-prone. Without the side effects, it is more predictable and easier to see the outcome. We also no need to worry about one part of the program is interfering another part.<\/p>\n<p>Many libraries adopted a functional programming paradigm. For example the following using pandas and pandas-datareader:<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8d4878981068\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nimport pandas_datareader as pdr<br \/>\nimport pandas_datareader.wb<\/p>\n<p>df = (<br \/>\n    pdr.wb<br \/>\n    .download(indicator=&#8221;SP.POP.TOTL&#8221;, country=&#8221;all&#8221;, start=2000, end=2020)<br \/>\n    .reset_index()<br \/>\n    .filter([&#8220;country&#8221;, &#8220;SP.POP.TOTL&#8221;])<br \/>\n    .groupby(&#8220;country&#8221;)<br \/>\n    .mean()<br \/>\n)<br \/>\nprint(df)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">pandas_datareader <\/span><span class=\"crayon-st\">as<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">pdr<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">pandas_datareader<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">wb<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">df<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">pdr<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">wb<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">download<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">indicator<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8220;SP.POP.TOTL&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">country<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8220;all&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">start<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">2000<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">end<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">2020<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">reset_index<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">filter<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;country&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;SP.POP.TOTL&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">groupby<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;country&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">mean<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">df<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>This gives you the following output:<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8d9584689979\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n                              SP.POP.TOTL<br \/>\ncountry<br \/>\nAfghanistan                  2.976380e+07<br \/>\nAfrica Eastern and Southern  5.257466e+08<br \/>\nAfrica Western and Central   3.550782e+08<br \/>\nAlbania                      2.943192e+06<br \/>\nAlgeria                      3.658167e+07<br \/>\n&#8230;                                   &#8230;<br \/>\nWest Bank and Gaza           3.806576e+06<br \/>\nWorld                        6.930446e+09<br \/>\nYemen, Rep.                  2.334172e+07<br \/>\nZambia                       1.393321e+07<br \/>\nZimbabwe                     1.299188e+07<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0SP.POP.TOTL<\/p>\n<p>country\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n<p>Afghanistan\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a02.976380e+07<\/p>\n<p>Africa Eastern and Southern\u00a0\u00a05.257466e+08<\/p>\n<p>Africa Western and Central\u00a0\u00a0 3.550782e+08<\/p>\n<p>Albania\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a02.943192e+06<\/p>\n<p>Algeria\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a03.658167e+07<\/p>\n<p>&#8230;\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 &#8230;<\/p>\n<p>West Bank and Gaza\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 3.806576e+06<\/p>\n<p>World\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a06.930446e+09<\/p>\n<p>Yemen, Rep.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a02.334172e+07<\/p>\n<p>Zambia\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1.393321e+07<\/p>\n<p>Zimbabwe\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1.299188e+07<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>The pandas-datareader is a useful library that helps you download data from the Internet in the realtime. The above example is to download population data from the World Bank. The result is a pandas dataframe with countries and years as index and a single column named \u201cSP.POP.TOTL\u201d for the population. Then we manipulate the dataframe step by step, and at the end, we find the average population of all countries across the years.<\/p>\n<p>We can write in this way because in pandas, most functions on the dataframe are not changing the dataframe, but to produce a new dataframe to reflect the result of the function. We call this behavior <strong>immutable<\/strong> because the input dataframe never changed. The consequence is that, we can chain up the functions to manipulate the dataframe step by step. If we have to break it into using the style of imperative programming, the above program is same as the following:<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8da538290009\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nimport pandas_datareader as pdr<br \/>\nimport pandas_datareader.wb<\/p>\n<p>df = pdr.wb.download(indicator=&#8221;SP.POP.TOTL&#8221;, country=&#8221;all&#8221;, start=2000, end=2020)<br \/>\ndf = df.reset_index()<br \/>\ndf = df.filter([&#8220;country&#8221;, &#8220;SP.POP.TOTL&#8221;])<br \/>\ngroups = df.groupby(&#8220;country&#8221;)<br \/>\ndf = groups.mean()<\/p>\n<p>print(df)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">pandas_datareader <\/span><span class=\"crayon-st\">as<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">pdr<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">pandas_datareader<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">wb<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">df<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">pdr<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">wb<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">download<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">indicator<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8220;SP.POP.TOTL&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">country<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-s\">&#8220;all&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">start<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">2000<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">end<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">2020<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">df<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">df<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">reset_index<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">df<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">df<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">filter<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;country&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;SP.POP.TOTL&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">groups<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">df<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">groupby<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;country&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">df<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">groups<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">mean<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">df<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<h2>High order functions: Filter, map, and reduce<\/h2>\n<p>Python is not a strictly functional programming language. But it is trivial to write Python in a functional style. There are three basic functions on iterables that allows us to write powerful program in a very trivial way: filter, map, and reduce.<\/p>\n<p>Filter is to select some of the elements in an iterable, such as a list. Map is to transform elements one by one. Finally, reduce is to convert the entire iterable into a different form, such as the sum of all elements or concatenating substrings in a list into a longer string. To illustrate their use, let\u2019s consider a simple task: Given a log file from the Apache web server, find the IP address that sent most requests with error code 404. If you have no idea what a log file from Apache web server looks like, the following is an example:<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8db613888344\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n89.170.74.95 &#8211; &#8211; [17\/May\/2015:16:05:27 +0000] &#8220;HEAD \/projects\/xdotool\/ HTTP\/1.1&#8221; 200 &#8211; &#8220;-&#8221; &#8220;Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko\/20100101 Firefox\/24.0&#8221;<br \/>\n95.82.59.254 &#8211; &#8211; [19\/May\/2015:03:05:19 +0000] &#8220;GET \/images\/jordan-80.png HTTP\/1.1&#8221; 200 6146 &#8220;http:\/\/www.semicomplete.com\/articles\/dynamic-dns-with-dhcp\/&#8221; &#8220;Mozilla\/5.0 (Windows NT 6.1; rv:27.0) Gecko\/20100101 Firefox\/27.0&#8221;<br \/>\n155.140.133.248 &#8211; &#8211; [19\/May\/2015:06:05:34 +0000] &#8220;GET \/images\/jordan-80.png HTTP\/1.1&#8221; 200 6146 &#8220;http:\/\/www.semicomplete.com\/blog\/geekery\/debugging-java-performance.html&#8221; &#8220;Mozilla\/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident\/5.0)&#8221;<br \/>\n68.180.224.225 &#8211; &#8211; [20\/May\/2015:20:05:02 +0000] &#8220;GET \/blog\/tags\/documentation HTTP\/1.1&#8221; 200 12091 &#8220;-&#8221; &#8220;Mozilla\/5.0 (compatible; Yahoo! Slurp; http:\/\/help.yahoo.com\/help\/us\/ysearch\/slurp)&#8221;<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>89.170.74.95 &#8211; &#8211; [17\/May\/2015:16:05:27 +0000] &#8220;HEAD \/projects\/xdotool\/ HTTP\/1.1&#8221; 200 &#8211; &#8220;-&#8221; &#8220;Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko\/20100101 Firefox\/24.0&#8221; <\/p>\n<p>95.82.59.254 &#8211; &#8211; [19\/May\/2015:03:05:19 +0000] &#8220;GET \/images\/jordan-80.png HTTP\/1.1&#8221; 200 6146 &#8220;http:\/\/www.semicomplete.com\/articles\/dynamic-dns-with-dhcp\/&#8221; &#8220;Mozilla\/5.0 (Windows NT 6.1; rv:27.0) Gecko\/20100101 Firefox\/27.0&#8221;<\/p>\n<p>155.140.133.248 &#8211; &#8211; [19\/May\/2015:06:05:34 +0000] &#8220;GET \/images\/jordan-80.png HTTP\/1.1&#8221; 200 6146 &#8220;http:\/\/www.semicomplete.com\/blog\/geekery\/debugging-java-performance.html&#8221; &#8220;Mozilla\/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident\/5.0)&#8221;<\/p>\n<p>68.180.224.225 &#8211; &#8211; [20\/May\/2015:20:05:02 +0000] &#8220;GET \/blog\/tags\/documentation HTTP\/1.1&#8221; 200 12091 &#8220;-&#8221; &#8220;Mozilla\/5.0 (compatible; Yahoo! Slurp; http:\/\/help.yahoo.com\/help\/us\/ysearch\/slurp)&#8221;<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>The above is from a bigger file located <a href=\"https:\/\/raw.githubusercontent.com\/elastic\/examples\/master\/Common%20Data%20Formats\/apache_logs\/apache_logs\">here<\/a>. These are a few lines from the log. Each line begins with the IP address of the client (i.e., the browser) and the code after \u201cHTTP\/1.1\u201d is the response status code. Normally it is 200 if the request is fulfilled. But if the browser requested something not exists in the server, the code will be 404. To find the IP address that corresponds to the most 404 requests, we can simply scan the log file line by line, find those with 404, and count the IP address to find the one with the most occurrences.<\/p>\n<p>In Python code, we can do the following. First we see how we can read the log file and extract the IP address and status code from a line:<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8dc035103234\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nimport urllib.request<br \/>\nimport re<\/p>\n<p># Read the log file, split into lines<br \/>\nlogurl = &#8220;https:\/\/raw.githubusercontent.com\/elastic\/examples\/master\/Common%20Data%20Formats\/apache_logs\/apache_logs&#8221;<br \/>\nlogfile = urllib.request.urlopen(logurl).read().decode(&#8220;utf8&#8221;)<br \/>\nlines = logfile.splitlines()<\/p>\n<p># using regular expression to extract IP address and status code from a line<br \/>\ndef ip_and_code(logline):<br \/>\n    m = re.match(r'([d.]+) .*? [.*?] &#8220;.*?&#8221; (d+) &#8216;, logline)<br \/>\n    return (m.group(1), m.group(2))<\/p>\n<p>print(ip_and_code(lines[0]))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">urllib<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">request<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-i\">re<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Read the log file, split into lines<\/span><\/p>\n<p><span class=\"crayon-v\">logurl<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;https:\/\/raw.githubusercontent.com\/elastic\/examples\/master\/Common%20Data%20Formats\/apache_logs\/apache_logs&#8221;<\/span><\/p>\n<p><span class=\"crayon-v\">logfile<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">urllib<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">request<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">urlopen<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">logurl<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">read<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">decode<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;utf8&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">lines<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">logfile<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">splitlines<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># using regular expression to extract IP address and status code from a line<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">ip_and_code<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">logline<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">m<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">re<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">match<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">r<\/span><span class=\"crayon-s\">&#8216;([d.]+) .*? [.*?] &#8220;.*?&#8221; (d+) &#8216;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">logline<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">m<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">group<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">m<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">group<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">ip_and_code<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">lines<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>then we can use a couple map() and filter() and some other functions to find the IP address:<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8de103779465\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<\/p>\n<p>import collections<\/p>\n<p>def is404(pair):<br \/>\n    return pair[1] == &#8220;404&#8221;<br \/>\ndef getIP(pair):<br \/>\n    return pair[0]<br \/>\ndef count_ip(count_item):<br \/>\n    ip, count = count_item<br \/>\n    return (count, ip)<\/p>\n<p># transform each line into (IP address, status code) pair<br \/>\nipcodepairs = map(ip_and_code, lines)<br \/>\n# keep only those with status code 404<br \/>\npairs404 = filter(is404, ipcodepairs)<br \/>\n# extract the IP address part from each pair<br \/>\nip404 = map(getIP, pairs404)<br \/>\n# count the occurrences, the result is a dictionary of IP addresses map to the count<br \/>\nipcount = collections.Counter(ip404)<br \/>\n# convert the (IP address, count) tuple into (count, IP address) order<br \/>\ncountip = map(count_ip, ipcount.items())<br \/>\n# find the tuple with the maximum on the count<br \/>\nprint(max(countip))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">collections<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">is404<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">pair<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">pair<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;404&#8221;<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">getIP<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">pair<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">pair<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">count_ip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">count_item<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">ip<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">count<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">count_item<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">count<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ip<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># transform each line into (IP address, status code) pair<\/span><\/p>\n<p><span class=\"crayon-v\">ipcodepairs<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">map<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">ip_and_code<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">lines<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># keep only those with status code 404<\/span><\/p>\n<p><span class=\"crayon-v\">pairs404<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">filter<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">is404<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ipcodepairs<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># extract the IP address part from each pair<\/span><\/p>\n<p><span class=\"crayon-v\">ip404<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">map<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">getIP<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">pairs404<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># count the occurrences, the result is a dictionary of IP addresses map to the count<\/span><\/p>\n<p><span class=\"crayon-v\">ipcount<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">collections<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">Counter<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">ip404<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># convert the (IP address, count) tuple into (count, IP address) order<\/span><\/p>\n<p><span class=\"crayon-v\">countip<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">map<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">count_ip<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ipcount<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">items<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># find the tuple with the maximum on the count<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">max<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">countip<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>Here we did not use reduce() function because we have some specialized reduce operation built-in, such as <code>max()<\/code>. But indeed, we can make a simpler program with list comprehension notation:<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8df067463151\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n&#8230;<\/p>\n<p>ipcodepairs = [ip_and_code(x) for x in lines]<br \/>\nip404 = [ip for ip,code in ipcodepairs if code==&#8221;404&#8243;]<br \/>\nipcount = collections.Counter(ip404)<br \/>\ncountip = [(count,ip) for ip,count in ipcount.items()]<br \/>\nprint(max(countip))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">ipcodepairs<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-e\">ip_and_code<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">x<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">lines<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-v\">ip404<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-e\">ip <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ip<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-e\">code <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">ipcodepairs <\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">code<\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-s\">&#8220;404&#8221;<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-v\">ipcount<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">collections<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">Counter<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">ip404<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">countip<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">count<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-v\">ip<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ip<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-e\">count <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ipcount<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">items<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">max<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">countip<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>or even write it in a single statement (but less readable):<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8e0742239719\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nimport urllib.request<br \/>\nimport re<br \/>\nimport collections<\/p>\n<p>logurl = &#8220;https:\/\/raw.githubusercontent.com\/elastic\/examples\/master\/Common%20Data%20Formats\/apache_logs\/apache_logs&#8221;<br \/>\nprint(<br \/>\n    max(<br \/>\n        [(count,ip) for ip,count in<br \/>\n            collections.Counter([<br \/>\n                ip for ip, code in<br \/>\n                [ip_and_code(x) for x in<br \/>\n                     urllib.request.urlopen(logurl)<br \/>\n                     .read()<br \/>\n                     .decode(&#8220;utf8&#8243;)<br \/>\n                     .splitlines()<br \/>\n                ]<br \/>\n                if code==&#8221;404&#8221;<br \/>\n            ]).items()<br \/>\n        ]<br \/>\n    )<br \/>\n)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">urllib<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">request<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">re<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">collections<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">logurl<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;https:\/\/raw.githubusercontent.com\/elastic\/examples\/master\/Common%20Data%20Formats\/apache_logs\/apache_logs&#8221;<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">max<\/span><span class=\"crayon-sy\">(<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">count<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-v\">ip<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ip<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-e\">count <\/span><span class=\"crayon-st\">in<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">collections<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">Counter<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">ip <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ip<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">code <\/span><span class=\"crayon-st\">in<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-e\">ip_and_code<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">x<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span class=\"crayon-v\">urllib<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">request<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">urlopen<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">logurl<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">read<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">decode<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;utf8&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">splitlines<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">code<\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-s\">&#8220;404&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">items<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<h2>Itertools in Python<\/h2>\n<p>The above example on filter, map, and reduce illustrates the ubiquity of <strong>iterables<\/strong> in Python. This includes lists, tuples, dictionaries, sets, and even generators as all of them can be <strong>iterated<\/strong> using a for-loop. In Python, we have a module named <code>itertools<\/code> that brings in more functions to manipulate (but not mutate) iterables. From\u00a0<a href=\"https:\/\/docs.python.org\/3\/library\/itertools.html\">Python\u2019s official documentation<\/a>:<\/p>\n<div class=\"woo-sc-quote\">\n<p>The module standardizes a core set of fast, memory efficient tools that are useful by themselves or in combination. Together, they form an \u201citerator algebra\u201d making it possible to construct specialized tools succinctly and efficiently in pure Python.<\/p>\n<\/div>\n<p>We\u2019ll discuss a few functions of <code>itertools<\/code> in this tutorial. When trying out the examples given below, be sure to import <code>itertools<\/code> and <code>operator<\/code> as:<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8e1818726865\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nimport itertools<br \/>\nimport operator<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">itertools<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">operator<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<h3>Infinite Iterators<\/h3>\n<p>Infinite iterators help you create sequences of infinite length as shown below.<\/p>\n<table cellspacing=\"0\" cellpadding=\"0\">\n<tbody>\n<tr>\n<td valign=\"top\"><span>Construct + Example<\/span><\/td>\n<td valign=\"top\"><span>Output<\/span><\/td>\n<\/tr>\n<tr>\n<td valign=\"top\"><code>count()<\/code><\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8e2968901896\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nstart = 0<br \/>\nstep = 100<br \/>\nfor i in itertools.count(start, step):<br \/>\n\u00a0 \u00a0 print(i)<br \/>\n\u00a0 \u00a0 if i&gt;=1000:<br \/>\n\u00a0 \u00a0 \u00a0 \u00a0 break<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-v\">start<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0<\/span><\/p>\n<p><span class=\"crayon-v\">step<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100<\/span><\/p>\n<p><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">i<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">count<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">start<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">step<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p>\u00a0<span class=\"crayon-h\"> <\/span>\u00a0<span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">i<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<span class=\"crayon-h\"> <\/span>\u00a0<span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">i<\/span><span class=\"crayon-o\">&gt;=<\/span><span class=\"crayon-cn\">1000<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p>\u00a0<span class=\"crayon-h\"> <\/span>\u00a0<span class=\"crayon-h\"> <\/span>\u00a0<span class=\"crayon-h\"> <\/span>\u00a0<span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">break<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<td valign=\"top\">\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8e3746873937\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n0<br \/>\n100<br \/>\n200<br \/>\n300<br \/>\n400<br \/>\n500<br \/>\n600<br \/>\n700<br \/>\n800<br \/>\n900<br \/>\n1000<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>0<\/p>\n<p>100<\/p>\n<p>200<\/p>\n<p>300<\/p>\n<p>400<\/p>\n<p>500<\/p>\n<p>600<\/p>\n<p>700<\/p>\n<p>800<\/p>\n<p>900<\/p>\n<p>1000<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\"><code>cycle()<\/code><\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8e4225514561\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\ncounter = 0<br \/>\ncyclic_list = [1, 2, 3, 4, 5]<br \/>\nfor i in itertools.cycle(cyclic_list):<br \/>\n    print(i)<br \/>\n    counter = counter+1<br \/>\n    if counter&gt;10:<br \/>\n        break<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-v\">counter<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0<\/span><\/p>\n<p><span class=\"crayon-v\">cyclic_list<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">i<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">cycle<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">cyclic_list<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">i<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">counter<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">counter<\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-cn\">1<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">counter<\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-cn\">10<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">break<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<td valign=\"top\">\n<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\"><code>repeat()<\/code><\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8e6263985777\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nfor i in itertools.repeat(3,5):<br \/>\n    print(i)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">i<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">repeat<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">i<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<td valign=\"top\">\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h3 id=\"Combinatoric-iterators\">Combinatoric iterators<\/h3>\n<p>You can create permutations, combinations, etc. with these iterators.<\/p>\n<table cellspacing=\"0\" cellpadding=\"0\">\n<tbody>\n<tr>\n<td valign=\"top\"><span>Construct + Example<\/span><\/td>\n<td valign=\"top\"><span>Output<\/span><\/td>\n<\/tr>\n<tr>\n<td valign=\"top\"><code>product()<\/code><\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8e8597936251\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nx = [1, 2, 3]<br \/>\ny = [&#8216;A&#8217;, &#8216;B&#8217;]<br \/>\nprint(list(itertools.product(x, y)))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-v\">x<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;A&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;B&#8217;<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">product<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<td valign=\"top\">\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8e9278824998\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n[(1, &#8216;A&#8217;), (1, &#8216;B&#8217;), (2, &#8216;A&#8217;), (2, &#8216;B&#8217;),<br \/>\n (3, &#8216;A&#8217;), (3, &#8216;B&#8217;)]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>[(1, &#8216;A&#8217;), (1, &#8216;B&#8217;), (2, &#8216;A&#8217;), (2, &#8216;B&#8217;), <\/p>\n<p> (3, &#8216;A&#8217;), (3, &#8216;B&#8217;)]<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\"><code>permutations()<\/code><\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8ea830342417\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nx = [1, 2, 3]<br \/>\nprint(list(itertools.permutations(x)))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-v\">x<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">permutations<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<td valign=\"top\">\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8eb438590598\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n[(1, 2, 3), (1, 3, 2), (2, 1, 3),<br \/>\n (2, 3, 1), (3, 1, 2), (3, 2, 1)]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>[(1, 2, 3), (1, 3, 2), (2, 1, 3), <\/p>\n<p> (2, 3, 1), (3, 1, 2), (3, 2, 1)]<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\"><code>combinations()<\/code><\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8ec378890238\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\ny = [&#8216;A&#8217;, &#8216;B&#8217;, &#8216;C&#8217;, &#8216;D&#8217;]<br \/>\nprint(list(itertools.combinations(y, 3)))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;A&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;B&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;C&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;D&#8217;<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">combinations<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<td valign=\"top\">\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8ed474942770\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n[(&#8216;A&#8217;, &#8216;B&#8217;, &#8216;C&#8217;), (&#8216;A&#8217;, &#8216;B&#8217;, &#8216;D&#8217;),<br \/>\n (&#8216;A&#8217;, &#8216;C&#8217;, &#8216;D&#8217;), (&#8216;B&#8217;, &#8216;C&#8217;, &#8216;D&#8217;)]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>[(&#8216;A&#8217;, &#8216;B&#8217;, &#8216;C&#8217;), (&#8216;A&#8217;, &#8216;B&#8217;, &#8216;D&#8217;), <\/p>\n<p> (&#8216;A&#8217;, &#8216;C&#8217;, &#8216;D&#8217;), (&#8216;B&#8217;, &#8216;C&#8217;, &#8216;D&#8217;)]<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\"><code>combinations_with_replacement()<\/code><\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8ee816510311\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nz = [&#8216;A&#8217;, &#8216;B&#8217;, &#8216;C&#8217;]<br \/>\nprint(list(itertools.combinations_with_replacement(z, 2)))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-v\">z<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8216;A&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;B&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8216;C&#8217;<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">combinations_with_replacement<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<td valign=\"top\">\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8ef155129051\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n[(&#8216;A&#8217;, &#8216;A&#8217;), (&#8216;A&#8217;, &#8216;B&#8217;), (&#8216;A&#8217;, &#8216;C&#8217;),<br \/>\n (&#8216;B&#8217;, &#8216;B&#8217;), (&#8216;B&#8217;, &#8216;C&#8217;), (&#8216;C&#8217;, &#8216;C&#8217;)]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>[(&#8216;A&#8217;, &#8216;A&#8217;), (&#8216;A&#8217;, &#8216;B&#8217;), (&#8216;A&#8217;, &#8216;C&#8217;), <\/p>\n<p> (&#8216;B&#8217;, &#8216;B&#8217;), (&#8216;B&#8217;, &#8216;C&#8217;), (&#8216;C&#8217;, &#8216;C&#8217;)]<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h3>More Useful Iterators<\/h3>\n<p>There are other iterators that stop at the end of the shorter of the two lists passed as arguments. \u00a0Some of them are described below. This is not an exhaustive list and you can <a href=\"https:\/\/docs.python.org\/3\/library\/itertools.html#itertool-functions\">see the complete list here<\/a>.<\/p>\n<h4>Accumulate()<\/h4>\n<p>Automatically creates an iterator that accumulates the result of a given operator or function, and returns the result. You can choose an operator from Python\u2019s <code>operator<\/code> \u00a0library or write your own customized operator.<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8f0210884529\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# Custom operator<br \/>\ndef my_operator(a, b):<br \/>\n    return a+b if a&gt;5 else a-b<\/p>\n<p>x = [2, 3, 4, -6]<br \/>\nmul_result = itertools.accumulate(x, operator.mul)<br \/>\nprint(&#8220;After mul operator&#8221;, list(mul_result))<br \/>\npow_result = itertools.accumulate(x, operator.pow)<br \/>\nprint(&#8220;After pow operator&#8221;, list(pow_result))<br \/>\nmy_operator_result = itertools.accumulate(x, my_operator)<br \/>\nprint(&#8220;After customized my_operator&#8221;, list(my_operator_result))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># Custom operator<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">my_operator<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-i\">b<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">else<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-i\">b<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><\/p>\n<p><span class=\"crayon-v\">x<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-v\">mul_result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">accumulate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">operator<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">mul<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;After mul operator&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">mul_result<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">pow_result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">accumulate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">operator<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">pow<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;After pow operator&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">pow_result<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">my_operator_result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">accumulate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">my_operator<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;After customized my_operator&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">my_operator_result<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8f1963989097\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nAfter mul operator [2, 6, 24, -144]<br \/>\nAfter pow operator [2, 8, 4096, 2.117582368135751e-22]<br \/>\nAfter customized my_operator [2, -1, -5, 1]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>After mul operator [2, 6, 24, -144]<\/p>\n<p>After pow operator [2, 8, 4096, 2.117582368135751e-22]<\/p>\n<p>After customized my_operator [2, -1, -5, 1]<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<h4>Starmap()<\/h4>\n<p>Apply the same operator to pairs of items.<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8f2039483957\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\npair_list = [(1, 2), (4, 0.5), (5, 7), (100, 10)]<\/p>\n<p>starmap_add_result = itertools.starmap(operator.add, pair_list)<br \/>\nprint(&#8220;Starmap add result: &#8220;, list(starmap_add_result))<\/p>\n<p>x1 = [2, 3, 4, -6]<br \/>\nx2 = [4, 3, 2, 1] <\/p>\n<p>starmap_mul_result = itertools.starmap(operator.mul, zip(x1, x2))<br \/>\nprint(&#8220;Starmap mul result: &#8220;, list(starmap_mul_result))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-v\">pair_list<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.5<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">7<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">100<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">10<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">starmap_add_result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">starmap<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">operator<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">add<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">pair_list<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Starmap add result: &#8220;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">starmap_add_result<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">x1<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-v\">x2<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">starmap_mul_result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">starmap<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">operator<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">mul<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">zip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">x1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">x2<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Starmap mul result: &#8220;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">starmap_mul_result<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8f3073899103\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nStarmap add result:  [3, 4.5, 12, 110]<br \/>\nStarmap mul result:  [8, 9, 8, -6]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">Starmap <\/span><span class=\"crayon-e\">add <\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\">\u00a0\u00a0<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4.5<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">12<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">110<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">Starmap <\/span><span class=\"crayon-e\">mul <\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\">\u00a0\u00a0<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">8<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">9<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">8<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<h4 id=\"filterfalse\">filterfalse()<\/h4>\n<p>Filter out data based on a specific criterion.<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8f4488960585\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nmy_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]<br \/>\neven_result = itertools.filterfalse(lambda x: x%2, my_list)<br \/>\nsmall_terms = itertools.filterfalse(lambda x: x&gt;=5, my_list)<br \/>\nprint(&#8216;Even result:&#8217;, list(even_result))<br \/>\nprint(&#8216;Less than 5:&#8217;, list(small_terms))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-v\">my_list<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">7<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">8<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">9<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">10<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-v\">even_result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">filterfalse<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">lambda<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">my_list<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">small_terms<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">itertools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">filterfalse<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">lambda<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-o\">&gt;=<\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">my_list<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;Even result:&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">even_result<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;Less than 5:&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">small_terms<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8f5740182588\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nEven result: [2, 4, 6, 8, 10]<br \/>\nLess than 5: [1, 2, 3, 4]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">Even <\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">8<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">10<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-e\">Less <\/span><span class=\"crayon-i\">than<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<h2>Functools in Python<\/h2>\n<p>In most programming languages, passing function as arguments or a function returning another function might be confusing or hard to work with. Python includes the <code>functools<\/code>\u00a0library that makes it easy to work with these functions. From Python\u2019s official <code>functools<\/code> documentation:<\/p>\n<div class=\"woo-sc-quote\">\n<p>The <code>functools<\/code> module is for higher-order functions: functions that act on or return other functions. In general, any callable object can be treated as a function<\/p>\n<\/div>\n<p>Here we explain a few nice features of this library. You can look at the <a href=\"https:\/\/docs.python.org\/3\/library\/functools.html\">complete list of <code>functools<\/code>\u00a0functions here<\/a>.<\/p>\n<h3>Using <code>lru_cache<\/code><\/h3>\n<p>In imperative programming languages, recursion is very expensive. Every time a function is invoked, it is evaluated, even if it is called with the same set of arguments. In Python, the <code>lru_cache<\/code>\u00a0is a decorator that can be used to cache the results of function evaluations. When the function is invoked again with the same set of arguments, the stored result is used, avoiding the extra overhead related to recursion.<\/p>\n<p>Let\u2019s look at the following example. We have the same implementation of the computation of the nth Fibonacci number with and without <code>lru_cache<\/code>. We can see that <code>fib(30<\/code>) has 31 function evaluations just as we expect because of <code>lru_cache<\/code>. The <code>fib()<\/code>\u00a0function is invoked only for n=0,1,2\u202630 and the result is stored in memory and used later. This is significantly less as compared to <code>fib_slow(30)<\/code>, with 2692537 evaluations.<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8f6381709106\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nimport functools<br \/>\n@functools.lru_cache<br \/>\ndef fib(n):<br \/>\n    global count<br \/>\n    count = count + 1<br \/>\n    return fib(n-2) + fib(n-1) if n&gt;1 else 1<\/p>\n<p>def fib_slow(n):<br \/>\n    global slow_count<br \/>\n    slow_count = slow_count + 1<br \/>\n    return fib_slow(n-2) + fib_slow(n-1) if n&gt;1 else 1<\/p>\n<p>count = 0<br \/>\nslow_count = 0<br \/>\nfib(30)<br \/>\nfib_slow(30)<\/p>\n<p>print(&#8216;With lru_cache total function evaluations: &#8216;, count)<br \/>\nprint(&#8216;Without lru_cache total function evaluations: &#8216;, slow_count)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-i\">functools<\/span><\/p>\n<p><span class=\"crayon-sy\">@<\/span><span class=\"crayon-v\">functools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">lru_cache<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">fib<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-m\">global<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">count<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">count<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">count<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">fib<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">fib<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">else<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">fib_slow<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-m\">global<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">slow_count<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">slow_count<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">slow_count<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">fib_slow<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">fib_slow<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">else<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">count<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0<\/span><\/p>\n<p><span class=\"crayon-v\">slow_count<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0<\/span><\/p>\n<p><span class=\"crayon-e\">fib<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">30<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">fib_slow<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">30<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;With lru_cache total function evaluations: &#8216;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">count<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;Without lru_cache total function evaluations: &#8216;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">slow_count<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8f7797992462\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nWith lru_cache total function evaluations:  31<br \/>\nWithout lru_cache total function evaluations:  2692537<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>With lru_cache total function evaluations:\u00a0\u00a031<\/p>\n<p>Without lru_cache total function evaluations:\u00a0\u00a02692537<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>It is worth to note that the <code>lru_cache<\/code> decorator is particularly useful when you\u2019re experimenting with machine learning problems in Jupyter notebooks. If you have a function that downloads data from the Internet, wrapping it with <code>lru_cache<\/code> can keep your download in memory and avoid downloading the same file again even if you invoked the download function multiple times.<\/p>\n<h3>Using <code>reduce()<\/code><\/h3>\n<p>Reduce is similar to the <code>itertools.accumulate()<\/code>. It applies a function repeatedly to the elements of a list and returns the result. Here are a few examples with comments to explain the working of this functions.<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8f8273171943\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# Evaluates ((1+2)+3)+4<br \/>\nlist_sum = functools.reduce(operator.add, [1, 2, 3, 4])<br \/>\nprint(list_sum)<\/p>\n<p># Evaluates (2^3)^4<br \/>\nlist_pow = functools.reduce(operator.pow, [2, 3, 4])<br \/>\nprint(list_pow)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># Evaluates ((1+2)+3)+4<\/span><\/p>\n<p><span class=\"crayon-v\">list_sum<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">functools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">reduce<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">operator<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">add<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">list_sum<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Evaluates (2^3)^4<\/span><\/p>\n<p><span class=\"crayon-v\">list_pow<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">functools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">reduce<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">operator<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">pow<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">list_pow<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>The <code>reduce()<\/code> function can accept any \u201coperators\u201d and optionally an initial value. For example, the <code>collections.Counter<\/code> function in the previous example can be implemented as follows<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8fa989628248\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nimport functools<\/p>\n<p>def addcount(counter, element):<br \/>\n    if element not in counter:<br \/>\n        counter[element] = 1<br \/>\n    else:<br \/>\n        counter[element] += 1<br \/>\n    return counter<\/p>\n<p>items = [&#8220;a&#8221;, &#8220;b&#8221;, &#8220;a&#8221;, &#8220;c&#8221;, &#8220;d&#8221;, &#8220;c&#8221;, &#8220;b&#8221;, &#8220;a&#8221;]<\/p>\n<p>counts = functools.reduce(addcount, items, {})<br \/>\nprint(counts)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">functools<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">addcount<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">counter<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">element<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">element <\/span><span class=\"crayon-st\">not<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">counter<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">counter<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">element<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">else<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">counter<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">element<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">counter<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">items<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;a&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;b&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;a&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;c&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;d&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;c&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;b&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;a&#8221;<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">counts<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">functools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">reduce<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">addcount<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">items<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">counts<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8fb202057312\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n{&#8216;a&#8217;: 3, &#8216;b&#8217;: 2, &#8216;c&#8217;: 2, &#8216;d&#8217;: 1}<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>{&#8216;a&#8217;: 3, &#8216;b&#8217;: 2, &#8216;c&#8217;: 2, &#8216;d&#8217;: 1}<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<h3>Using <code>partial()<\/code><\/h3>\n<p>There are situations when you \u00a0have a function that takes multiple arguments and some of its arguments are repeated again and again. The function <code>partial()<\/code> returns a new version of the same function with a reduced number of arguments.<\/p>\n<p>For example, if you have to compute the power of 2 repeatedly, you can create a new version of numpy\u2019s <code>power()<\/code>\u00a0function as shown below:<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8fc360194630\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nimport numpy<\/p>\n<p>power_2 = functools.partial(np.power, 2)<br \/>\nprint(&#8216;2^4 =&#8217;, power_2(4))<br \/>\nprint(&#8216;2^6 =&#8217;, power_2(6))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">numpy<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">power_2<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">functools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">partial<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">power<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;2^4 =&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">power_2<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;2^6 =&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">power_2<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">6<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<h2 id=\"Map-Reduce-Pattern\">Map-Reduce Pattern<\/h2>\n<p>We have mentioned the filter, map, and reduce functions as high order functions in a previous section. The use of map-reduce design pattern is indeed a way to help us easily make a highly scalable program. The map-reduce pattern is an abstract representation of many types of computations that manipulate lists or collections of objects. The <code>map<\/code> stage takes the input collection and maps it to an intermediate representation. The <code>reduce<\/code> step takes this intermediate representation and computes a single output from it. This design pattern is very popular in functional programming languages. Python also provides constructs to implement this design pattern in an efficient manner.<\/p>\n<h3 id=\"Map-Reduce-In-Python\">Map-Reduce In Python<\/h3>\n<p>As an illustration of the map-reduce design pattern, let\u2019s take a simple example. Suppose we want to count the numbers divisible by 3 in a list. We\u2019ll use <code>lambda<\/code> to define an anonymous function and use it to <code>map()<\/code> all items of a list to 1 or 0 depending upon whether they pass our divisibility test or not. The function <code>map()<\/code> takes as argument a function and an iterable.\u00a0Next we\u2019ll use <code>reduce()<\/code> to accumulate the overall result.<\/p>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8fe573838235\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# All numbers from 1 to 20<br \/>\ninput_list = list(range(20))<br \/>\n# Use map to see which numbers are divisible by 3<br \/>\nbool_list = map(lambda x: 1 if x%3==0 else 0, input_list)<br \/>\n# Convert map object to list<br \/>\nbool_list = list(bool_list)<br \/>\nprint(&#8216;bool_list =&#8217;, bool_list)<\/p>\n<p>total_divisible_3 = functools.reduce(operator.add, bool_list)<br \/>\nprint(&#8216;Total items divisible by 3 = &#8216;, total_divisible_3)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># All numbers from 1 to 20<\/span><\/p>\n<p><span class=\"crayon-v\">input_list<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">range<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">20<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># Use map to see which numbers are divisible by 3<\/span><\/p>\n<p><span class=\"crayon-v\">bool_list<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">map<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">lambda<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-o\">%<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">else<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">input_list<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-p\"># Convert map object to list<\/span><\/p>\n<p><span class=\"crayon-v\">bool_list<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">bool_list<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;bool_list =&#8217;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">bool_list<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">total_divisible_3<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">functools<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">reduce<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">operator<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">add<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">bool_list<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8216;Total items divisible by 3 = &#8216;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">total_divisible_3<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<div id=\"urvanov-syntax-highlighter-61cd4e621c8ff729334897\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nbool_list = [1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0]<br \/>\nTotal items divisible by 3 =  7<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>bool_list = [1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0]<\/p>\n<p>Total items divisible by 3 =\u00a0\u00a07<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>The previous example, while being very simple, illustrates how easy it is to implement <code>map-reduce<\/code>\u00a0design pattern in Python. You can solve complex and lengthy problems using the surprisingly simple and easy constructs in Python.<\/p>\n<h2 id=\"Further-Reading\">Further Reading<\/h2>\n<p>This section provides more resources on the topic if you are looking to go deeper.<\/p>\n<h3 id=\"Books\">Books<\/h3>\n<h3 id=\"API-Reference\">Python Official Documentation<\/h3>\n<h2 id=\"Summary\">Summary<\/h2>\n<p>In this tutorial, you discovered features of Python that support functional programming<\/p>\n<p>Specifically, you learned:<\/p>\n<ul>\n<li>The iterables returning finite or infinite sequences in Python using <code>itertools<\/code><\/li>\n<li>The higher order functions supported by <code>functools<\/code><\/li>\n<li>The map-reduce design pattern\u2019s implementation in Python<\/li>\n<\/ul>\n<p>Do you have any questions about Python discussed in this post? Ask your questions in the comments below and I will do my best to answer.<\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/machinelearningmastery.com\/functional-programming-in-python\/<\/p>\n","protected":false},"author":0,"featured_media":1420,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1419"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=1419"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1419\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/1420"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=1419"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=1419"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=1419"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}