{"id":2028,"date":"2022-03-28T20:43:45","date_gmt":"2022-03-28T20:43:45","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/28\/grading-our-2022-oscars-machine-learning-predictions\/"},"modified":"2022-03-28T20:43:45","modified_gmt":"2022-03-28T20:43:45","slug":"grading-our-2022-oscars-machine-learning-predictions","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/28\/grading-our-2022-oscars-machine-learning-predictions\/","title":{"rendered":"Grading our 2022 Oscars Machine Learning Predictions"},"content":{"rendered":"<div>\n<p class=\"has-text-align-justify\"><a rel=\"noreferrer noopener\" href=\"https:\/\/www.oscars.org\/oscars\/ceremonies\/2022\" target=\"_blank\"><strong>The 94th Academy Awards<\/strong><\/a> ceremony is a wrap and thanks to an unexpected turn of events it seems to have the distinction of being dubbed as <strong><a rel=\"noreferrer noopener\" href=\"https:\/\/www.latimes.com\/entertainment-arts\/movies\/story\/2022-03-27\/oscars-will-smith-coda-takeaways\" target=\"_blank\">the most chaotic of them all<\/a><\/strong>. The night began with all the familiar signs of what one would expect from a non-pandemic Oscars proceedings with the celebrities streaming in, the red carpet interviews, and the inevitable opening salvos from the night\u2019s hosts. But then, suddenly, it took a turn for the worse when Chris Rock took the stage for a serving of his no-holds-barred celebrity jokes, except this time Will Smith responded with a <a href=\"https:\/\/www.youtube.com\/watch?v=z4t1CC7-UFE\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>\u201cservice return\u201d of his own<\/strong><\/a> that will perhaps forever be remembered as the defining moment of this year\u2019s Academy Awards.  While we hope that tempers will fade in the coming days, we\u2019d like to quietly jump into the movies themselves and how we did with our predictions this year.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" data-attachment-id=\"30543\" data-permalink=\"https:\/\/blog.bigml.com\/screen-shot-2022-03-28-at-10-30-53-am\/\" data-orig-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-10.30.53-am.png\" data-orig-size=\"942,346\" data-comments-opened=\"1\" data-image-meta='{\"aperture\":\"0\",\"credit\":\"\",\"camera\":\"\",\"caption\":\"\",\"created_timestamp\":\"0\",\"copyright\":\"\",\"focal_length\":\"0\",\"iso\":\"0\",\"shutter_speed\":\"0\",\"title\":\"\",\"orientation\":\"0\"}' data-image-title=\"screen-shot-2022-03-28-at-10.30.53-am\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-10.30.53-am.png?w=300\" data-large-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-10.30.53-am.png?w=810\" src=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-10.30.53-am.png?w=942\" alt=\"\" class=\"wp-image-30543\"><\/figure>\n<p class=\"has-text-align-justify\">With modifications to our data and approach, this year\u2019s prediction challenge turned out a fruitful one as we had <a href=\"https:\/\/blog.bigml.com\/2022\/03\/25\/predicting-the-2022-oscar-winners-with-machine-learning\/\"><strong>15 out of 19 predictions correct<\/strong><\/a> for a pretty decent <strong>hit rate of 79%!<\/strong> In fact, we\u2019ve never had 15 right picks before as there were some question marks as to whether some of the more technical categories we chose to add for the first time would prove predictable. Although a single year can be misleading, we have a good reason to continue including the newly added categories going forward. <\/p>\n<p class=\"has-text-align-justify\">As usual, we have to remind ourselves that getting 15 out of 19 right with anywhere between 5 to 10 nominees for each category would be an exercise in futility if we were to engage in the random act of flipping coins instead.  Basically, we\u2019re talking about finding the one correct combination out of 30,517,600,000 possible combinations. As the tables below summarize, armed with a spreadsheet full of data, <strong>Machine Learning does a remarkable job of almost fully solving this complex puzzle despite having no preconceived notion of what a movie is!<\/strong><\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" data-attachment-id=\"30533\" data-permalink=\"https:\/\/blog.bigml.com\/screen-shot-2022-03-28-at-7-02-00-am\/\" data-orig-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-7.02.00-am.png\" data-orig-size=\"868,508\" data-comments-opened=\"1\" data-image-meta='{\"aperture\":\"0\",\"credit\":\"\",\"camera\":\"\",\"caption\":\"\",\"created_timestamp\":\"0\",\"copyright\":\"\",\"focal_length\":\"0\",\"iso\":\"0\",\"shutter_speed\":\"0\",\"title\":\"\",\"orientation\":\"0\"}' data-image-title=\"screen-shot-2022-03-28-at-7.02.00-am\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-7.02.00-am.png?w=300\" data-large-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-7.02.00-am.png?w=810\" src=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-7.02.00-am.png?w=868\" alt=\"\" class=\"wp-image-30533\"><\/figure>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" data-attachment-id=\"30554\" data-permalink=\"https:\/\/blog.bigml.com\/screen-shot-2022-03-28-at-11-37-58-am\/\" data-orig-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-11.37.58-am.png\" data-orig-size=\"874,452\" data-comments-opened=\"1\" data-image-meta='{\"aperture\":\"0\",\"credit\":\"\",\"camera\":\"\",\"caption\":\"\",\"created_timestamp\":\"0\",\"copyright\":\"\",\"focal_length\":\"0\",\"iso\":\"0\",\"shutter_speed\":\"0\",\"title\":\"\",\"orientation\":\"0\"}' data-image-title=\"screen-shot-2022-03-28-at-11.37.58-am\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-11.37.58-am.png?w=300\" data-large-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-11.37.58-am.png?w=810\" src=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-11.37.58-am.png?w=874\" alt=\"\" class=\"wp-image-30554\"><\/figure>\n<h2>Analysis and Lessons Learned<\/h2>\n<ul>\n<li>\n<p class=\"has-text-align-justify\">We found that we could have achieved even better results if we stuck to averaging the <strong><a rel=\"noreferrer noopener\" href=\"https:\/\/bigml.com\/releases\/winter-2018\" target=\"_blank\">Top 10 OptiML models<\/a><\/strong> which would yield 17 hits out of 19 for a hit rate of almost 90%. Only the Best Picture and the Best Original Screenplay results would be incorrect while Best Adapted Screenplay and Best Costume misses would be reversed. Considering how easy it is to run an <strong>OptiML<\/strong> model search (it\u2019s really one single click folks) once you have created a dataset, we can truly talk about <strong>bringing Machine Learning to the masses<\/strong>.<\/p>\n<\/li>\n<\/ul>\n<ul>\n<li>\n<p class=\"has-text-align-justify\">It\u2019s also noteworthy that the good old Logistic Regression models made up a good part of the top-performing OptiML models, which tells us <strong>complexity doesn\u2019t always add value when it comes to Machine Learning<\/strong>. Simply aim to shoot for what works. Interestingly, this is an ironic pattern we see quite often in AI\/ML-type projects in the enterprise setting, where data scientists and ML specialists sometimes unnecessarily complicate things. Note that our method for making predictions is transparent and documented for all 19 categories, which has added value of its own. In the same vein, organizations must decide for themselves where they should stand given the accuracy, complexity, risk, time-to-market, and cost tradeoff spectrums. Our experience since 2011 has shown investing in data quality and data engineering are the smart choices to overcome bottlenecks while simultaneously relying on the select batch of best-in-class methods the Machine Learning community has been able to collectively produce since the 1980s.<\/p>\n<\/li>\n<\/ul>\n<ul>\n<li>\n<p class=\"has-text-align-justify\">CODA was clearly the <a rel=\"noreferrer noopener\" href=\"https:\/\/www.sltrib.com\/artsliving\/2022\/03\/27\/how-coda-made-history\/#:~:text=%E2%80%9CCODA%E2%80%9D%20%E2%80%94%20the%20story%20of,of%20the%20COVID%2D19%20pandemic.\" target=\"_blank\">feel-good picture of the year<\/a> and it indeed delivered a storybook ending to its eventful journey capped with the BEST PICTURE OSCAR! The fact that this smallish independent film was acquired by the media heavyweight Apple for a hefty sum elevated its exposure to a worldwide level carrying its campaign to the winner\u2019s circle while marking the arrival of this new distribution model for the movie industry as one that\u2019s likely here to stay. We will probably see more CODAs being scooped up by streaming giants and plugged into their vast network effects and audiences.<\/p>\n<\/li>\n<\/ul>\n<ul>\n<li>\n<p class=\"has-text-align-justify\">As was the case in prior years, the Screenplay categories proved highly competitive. Perhaps, there\u2019s something to say about the story itself \u2013 the most basic element of any film proving the most equal competition among nominees before production budgets, casting, and an unending array of technical talent add whole new dimensions to the final equation of competitive differentiation. We\u2019ll look for new angles to tackle these categories in the next year.<\/p>\n<\/li>\n<\/ul>\n<ul>\n<li>\n<p class=\"has-text-align-justify\">This year, since we opened up our <a rel=\"noreferrer noopener\" href=\"https:\/\/blog.bigml.com\/2022\/03\/16\/can-you-beat-a-simple-machine-learning-model\/\" target=\"_blank\"><strong>Oscars Prediction Challenge<\/strong><\/a> to all of our community, quite a few of you have submitted predictions based on a wide variety of approaches such as Machine Learning modeling, gut feeling, as well as unofficially polling movie critic opinions. We\u2019re happy to report that BigML\u2019s own <strong>Alfonso Amauleyas<\/strong> got 15 of 19 picks right as well. Of the participants that didn\u2019t apply ML, <strong>Ivana Guraieb Galland<\/strong> got 5 out of 6 picks correct. We\u2019ll continue using this \u201ccrowdsourced\u201d approach in the coming years.<\/p>\n<\/li>\n<\/ul>\n<h2>History to Date Predictions Performance<\/h2>\n<p>Finally, we\u2019ve also updated the table below that compiles all our predictions between the 2018 and 2022 Oscars and the corresponding hit rates for the major categories. In addition to the Top Picks that we annually shared in our past blogs, this table lists how the accuracy metric improves if we also consider the movies that received the highest two (Top 2) or three (Top 3) scores. The Top Picks alone had an average 68% hit rate, whereas the coverage reached <strong>92% with the Top 3 taken into account<\/strong>.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" data-attachment-id=\"30534\" data-permalink=\"https:\/\/blog.bigml.com\/screen-shot-2022-03-28-at-7-01-46-am\/\" data-orig-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-7.01.46-am.png\" data-orig-size=\"873,520\" data-comments-opened=\"1\" data-image-meta='{\"aperture\":\"0\",\"credit\":\"\",\"camera\":\"\",\"caption\":\"\",\"created_timestamp\":\"0\",\"copyright\":\"\",\"focal_length\":\"0\",\"iso\":\"0\",\"shutter_speed\":\"0\",\"title\":\"\",\"orientation\":\"0\"}' data-image-title=\"screen-shot-2022-03-28-at-7.01.46-am\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-7.01.46-am.png?w=300\" data-large-file=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-7.01.46-am.png?w=810\" src=\"https:\/\/littleml.files.wordpress.com\/2022\/03\/screen-shot-2022-03-28-at-7.01.46-am.png?w=873\" alt=\"\" class=\"wp-image-30534\"><figcaption>NOTE: Original and Adapted Screenplay predictions apply only to 2019-22.<\/figcaption><\/figure>\n<p>As the\u00a0<strong><a rel=\"noreferrer noopener\" href=\"https:\/\/bigml.com\/milestones\/\" target=\"_blank\">pioneers of ML-as-a-Service<\/a><\/strong>\u00a0here at BigML, we welcome students of Machine Learning to build their own models on top of the public\u00a0<a rel=\"noreferrer noopener\" href=\"https:\/\/bigml.com\/user\/academy_awards\/gallery\/dataset\/607c9a229f6704133a0012e2\" target=\"_blank\"><strong>movies 2000-20<\/strong><\/a><strong><a rel=\"noreferrer noopener\" href=\"https:\/\/bigml.com\/user\/academy_awards\/gallery\/dataset\/607c9a229f6704133a0012e2\" target=\"_blank\">2<\/a><\/strong><a rel=\"noreferrer noopener\" href=\"https:\/\/bigml.com\/user\/academy_awards\/gallery\/dataset\/607c9a229f6704133a0012e2\" target=\"_blank\"><strong>1<\/strong><\/a><a href=\"https:\/\/bigml.com\/shared\/dataset\/mgEg7bE4rDsrShvo5XHBIx461FM\"><strong> dataset<\/strong><\/a>. It\u2019s a great way to put your Machine Learning skills to test quickly without the overhead of having to download and install many open source packages and worry about compatibility issues or hard-to-decipher error messages. It takes 1 minute to\u00a0<a rel=\"noreferrer noopener\" href=\"https:\/\/bigml.com\/accounts\/register\/\" target=\"_blank\"><strong>create a FREE account<\/strong><\/a>\u00a0and about as much time to clone the movies dataset to your account. As always, let us know how your results turn out on Twitter\u00a0<a rel=\"noreferrer noopener\" href=\"https:\/\/twitter.com\/bigmlcom\" target=\"_blank\"><strong>@bigmlcom<\/strong><\/a>,\u00a0or send us a note anytime at\u00a0<a href=\"mailto:feedback@bigml.com\"><strong>feedback@bigml.com<\/strong><\/a>!<\/p>\n<div id=\"jp-post-flair\" class=\"sharedaddy sharedaddy-dark sd-like-enabled sd-sharing-enabled\">\n<div class=\"sharedaddy sd-block sd-like jetpack-likes-widget-wrapper jetpack-likes-widget-unloaded\" id=\"like-post-wrapper-30283844-30530-62421e018bfb5\" data-src=\"\/\/widgets.wp.com\/likes\/index.html?ver=20220105#blog_id=30283844&amp;post_id=30530&amp;origin=littleml.wordpress.com&amp;obj_id=30283844-30530-62421e018bfb5&amp;domain=blog.bigml.com\" data-name=\"like-post-frame-30283844-30530-62421e018bfb5\" data-title=\"Like or Reblog\">\n<h3 class=\"sd-title\">Like this:<\/h3>\n<p><span class=\"button\"><span>Like<\/span><\/span> <span class=\"loading\">Loading&#8230;<\/span><\/p>\n<p><span class=\"sd-text-color\"><\/span><a class=\"sd-link-color\"><\/a><\/div>\n<h3 class=\"jp-relatedposts-headline\"><em>Relacionado<\/em><\/h3>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/blog.bigml.com\/2022\/03\/28\/grading-our-2022-oscars-machine-learning-predictions\/<\/p>\n","protected":false},"author":0,"featured_media":2029,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/2028"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=2028"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/2028\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/2029"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=2028"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=2028"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=2028"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}