{"id":821,"date":"2021-09-05T15:53:11","date_gmt":"2021-09-05T15:53:11","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/09\/05\/new-blog-series-memoirs-of-a-torchvision-developer\/"},"modified":"2021-09-05T15:53:11","modified_gmt":"2021-09-05T15:53:11","slug":"new-blog-series-memoirs-of-a-torchvision-developer","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/09\/05\/new-blog-series-memoirs-of-a-torchvision-developer\/","title":{"rendered":"New Blog series \u2013 Memoirs of a TorchVision developer"},"content":{"rendered":"<div>\n<ul class=\"unstyled inline blog-info\">\n<li><i class=\"icon-calendar\"><\/i> August 21, 2021<\/li>\n<li><i class=\"icon-user\"><\/i> <a href=\"https:\/\/blog.datumbox.com\/author\/bbriniotis\/\" title=\"Posts by Vasilis Vryniotis\" rel=\"author\">Vasilis Vryniotis<\/a><\/li>\n<li> <a href=\"https:\/\/blog.datumbox.com\/new-blog-series-memoirs-of-a-torchvision-developer\/#respond\">. No comments<\/a><\/li>\n<\/ul>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"770\" height=\"433\" src=\"https:\/\/blog.datumbox.com\/wp-content\/uploads\/2021\/08\/torchvision-770x433.jpg\" alt=\"\" class=\"wp-image-1022\"><\/figure>\n<p>I\u2019m starting a new blog post series about the development of PyTorch\u2019s computer vision library. I plan to discuss interesting upcoming features primarily from <a href=\"https:\/\/github.com\/pytorch\/vision\/\">TorchVision<\/a> and secondary from the PyTorch ecosystem. My target is to highlight new and in-development features and provide clarity of what\u2019s happening in between the releases. Though the format is likely to change over time, I initially plan to keep it bite-sized and offer references for those who want to dig deeper. Finally, instead of publishing articles on fixed intervals, I\u2019ll be posting when I have enough interesting topics to cover.<\/p>\n<p><strong>Disclaimer: <\/strong>The features covered will be biased towards topics I\u2019m personally interested. The PyTorch ecosystem is massive and I only have visibility over a tiny part of it. Covering (or not covering) a feature says nothing about its importance. Opinions expressed are solely my own.<\/p>\n<p>With that out of the way, let\u2019s see what\u2019s cooking:<\/p>\n<h2>Label Smoothing for CrossEntropy Loss<\/h2>\n<p>A highly requested feature on PyTorch is to <a href=\"https:\/\/github.com\/pytorch\/pytorch\/issues\/11959\">support soft targets<\/a> and <a href=\"https:\/\/github.com\/pytorch\/pytorch\/issues\/7455\">add a label smoothing option<\/a> in Cross Entropy loss. Both features target in making it easy to do Label Smoothing, with the first option offering more flexibility when Data Augmentation techniques such as <a href=\"https:\/\/arxiv.org\/abs\/1710.09412\">mixup<\/a>\/<a href=\"https:\/\/arxiv.org\/abs\/1905.04899\">cutmix<\/a> are used and the second being more performant for the simple cases. The soft targets option has already been <a href=\"https:\/\/github.com\/pytorch\/pytorch\/pull\/61044\">merged on master<\/a> by Joel Schlosser while the label_smoothing option is <a href=\"https:\/\/github.com\/pytorch\/pytorch\/pull\/63122\">being developed<\/a> by Thomas J. Fan and is currently under review.<\/p>\n<h2>New Warm-up Scheduler<\/h2>\n<p>Learning Rate warm up is a common technique used when training models but until now PyTorch didn\u2019t offer an off-the-shelf solution. Recently, Ilqar Ramazanli has <a href=\"https:\/\/github.com\/pytorch\/pytorch\/pull\/60836\">introduced<\/a> a new Scheduler supporting linear and constant warmup. Currently in progress is the work around <a href=\"https:\/\/github.com\/pytorch\/pytorch\/pull\/63457\">improving the chain-ability<\/a> and <a href=\"https:\/\/github.com\/pytorch\/pytorch\/pull\/63491\">combination<\/a> of existing schedulers.<\/p>\n<h2>TorchVision with \u201cBatteries included\u201d<\/h2>\n<p>This half we are working on adding in TorchVision popular Models, Losses, Schedulers, Data Augmentations and other utilities used to achieve state-of-the-art results. This project is aptly named \u201c<a href=\"https:\/\/github.com\/pytorch\/vision\/issues\/3911\">Batteries included<\/a>\u201d and is currently <a href=\"https:\/\/github.com\/pytorch\/vision\/projects\/1\">in progress<\/a>.<\/p>\n<p>Earlier this week, I\u2019ve <a href=\"https:\/\/github.com\/pytorch\/vision\/pull\/4293\">added a new layer<\/a> called <a href=\"https:\/\/github.com\/pytorch\/vision\/issues\/4298\">StochasticDepth<\/a> which can be used to randomly drop residual branches in residual architectures. Currently I\u2019m working on <a href=\"https:\/\/github.com\/pytorch\/vision\/issues\/980\">adding an implementation<\/a> of the popular network architecture called <a href=\"https:\/\/github.com\/pytorch\/vision\/issues\/980\">EfficientNet<\/a>. Finally, Allen Goodman is <a href=\"https:\/\/github.com\/pytorch\/vision\/pull\/4290\">currently<\/a> adding a new operator that will enable converting <a href=\"https:\/\/github.com\/pytorch\/vision\/issues\/3960\">Segmentation Masks to Bounding Boxes<\/a>.<\/p>\n<h2>Other features in-development<\/h2>\n<p>Thought we constantly make incremental improvements on the documentation, CI infrastructure and overall code quality, below I highlight some of the \u201cuser-facing\u201d <a href=\"https:\/\/github.com\/pytorch\/vision\/issues\/4187\">roadmap items<\/a> which are in-development:<\/p>\n<p>That\u2019s it! I hope you found it interesting. Any ideas on how to adapt the format or what topics to cover are very welcome. Hit me up on <a rel=\"noreferrer noopener\" href=\"https:\/\/www.linkedin.com\/in\/bbriniotis\" target=\"_blank\">LinkedIn<\/a> or <a rel=\"noreferrer noopener\" href=\"https:\/\/twitter.com\/bbriniotis\" target=\"_blank\">Twitter<\/a>.<\/p>\n<div class=\"authorsure-author-box\"><img decoding=\"async\" alt=\"\" src=\"https:\/\/secure.gravatar.com\/avatar\/0d9cf005f5d9fbbe42b6723e723a7f76?s=90&amp;r=g\" class=\"avatar avatar-90 photo\" height=\"90\" width=\"90\" loading=\"lazy\"><\/p>\n<h4>About\u00a0<a rel=\"author\" href=\"https:\/\/blog.datumbox.com\/author\/bbriniotis\/\" class=\"authorsure-author-link\">Vasilis Vryniotis<\/a><\/h4>\n<p>My name is Vasilis Vryniotis. I&#8217;m a Data Scientist, a Software Engineer, author of Datumbox Machine Learning Framework and a proud geek. <a href=\"https:\/\/blog.datumbox.com\/author\/bbriniotis\/\">Learn more<\/a><\/p>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/blog.datumbox.com\/new-blog-series-memoirs-of-a-torchvision-developer\/<\/p>\n","protected":false},"author":0,"featured_media":822,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/821"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=821"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/821\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/822"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=821"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=821"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=821"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}