{"id":418,"date":"2020-10-18T20:13:03","date_gmt":"2020-10-18T20:13:03","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/10\/18\/how-to-take-s3-backups-with-dejadup-on-ubuntu-20-10\/"},"modified":"2020-10-18T20:13:03","modified_gmt":"2020-10-18T20:13:03","slug":"how-to-take-s3-backups-with-dejadup-on-ubuntu-20-10","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/10\/18\/how-to-take-s3-backups-with-dejadup-on-ubuntu-20-10\/","title":{"rendered":"How to take S3 backups with DejaDup on Ubuntu 20.10"},"content":{"rendered":"<div>\n<ul class=\"unstyled inline blog-info\">\n<li>\n<i class=\"icon-calendar\"><\/i> October 18, 2020<\/li>\n<li>\n<i class=\"icon-user\"><\/i> <a href=\"http:\/\/blog.datumbox.com\/author\/bbriniotis\/\" title=\"Posts by Vasilis Vryniotis\" rel=\"author\">Vasilis Vryniotis<\/a>\n<\/li>\n<li> <a href=\"http:\/\/blog.datumbox.com\/how-to-take-s3-backups-with-dejadup-on-ubuntu-20-10\/#respond\">. No comments<\/a>\n<\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-large wp-image-821\" src=\"http:\/\/blog.datumbox.com\/wp-content\/uploads\/2020\/10\/ubuntu_s3_backup.png\" alt=\"\" width=\"770\" height=\"432\">DejaDup is the default backup application for Gnome. It\u2019s a GUI for duplicity, focuses on simplicity, supports incremental encrypted backups and up until recently supported a large number of cloud providers. Unfortunately as of version 42.0, <a href=\"https:\/\/wiki.gnome.org\/Apps\/DejaDup\/Clouds#Deprecated_Cloud_Storage_Services\" target=\"_blank\" rel=\"noopener noreferrer\">all major cloud providers have been removed<\/a>. Thus given that Ubuntu 20.10 ships with the specific version, any user who upgrades and has backups on Amazon S3 won\u2019t be able to access them. In this blog post, we will provide a solution that will allow you to continue taking backups on AWS S3 using DejaDup.<\/p>\n<h2>The mandatory rant (feel free to skip)<\/h2>\n<p>The removal of the cloud providers should not come as a surprise. I\u2019m not exactly sure which version of DejaDup deprecated them but it was around the release of <a href=\"https:\/\/blog.datumbox.com\/ubuntu-17-10-a-last-minute-review\/\">Ubuntu 17.10<\/a> when they were all hidden as an option. So for 3 long years, people who had backups on Amazon S3, Google Cloud Storage, Openstack Swift, Rackspace etc could still use the deprecated feature and prepare for the inevitable removal.<\/p>\n<p>So why complain you might ask? Well, first of all, when you update from an earlier version of Ubuntu to 20.10, you don\u2019t really know that the all cloud providers are removed from DejaDup. Hence if something goes wrong during the update, you won\u2019t be able to easily access your backups and restore your system.<\/p>\n<p>Another big problem is the lack of storage options on the last version of DejaDup. They decided to change their policy and support only \u201cconsumer-targeted cloud services\u201d but currently they only support Google Drive. So they eliminated all the cost efficient options for mass storage and kept only one single very expensive option. I\u2019m not really sure how this is good for the users of the application. Linux was always about having a choice (too much of it in many cases), so why not maintain multiple storage options to serve both the experience and inexperienced users? Thankfully because we are on Linux, we have option to fix this.<\/p>\n<h2>How to use Deja Dup v42+ with AWS S3<\/h2>\n<p><img decoding=\"async\" src=\"http:\/\/blog.datumbox.com\/wp-content\/uploads\/2020\/10\/DejaDup.png\"><\/p>\n<p><strong>WARNING: I have not tested thoroughly the following setup so use it at your own risk. If the computer explodes in your face, you lose your data, or your spouse takes your kids and leaves you, don\u2019t blame me.<\/strong><\/p>\n<h3>Installing s3fs fuse<\/h3>\n<p>With that out of the way, let\u2019s proceed to the fix. We will use <a href=\"https:\/\/github.com\/s3fs-fuse\/s3fs-fuse\" target=\"_blank\" rel=\"noopener noreferrer\">s3fs fuse<\/a>, a program that allows you to mount an S3 bucket via FUSE and effectively make it look like a local disk. Thankfully you don\u2019t have to compile it from source as it\u2019s on Ubuntu\u2019s repos. To install it, type the following in your terminal:<\/p>\n<p><code>sudo apt install s3fs<\/code><\/p>\n<h3>Setting up your AWS credentials file<\/h3>\n<p>Next, we need to configure your credentials. The s3fs supports two methods for authentication: an <a href=\"https:\/\/docs.aws.amazon.com\/cli\/latest\/userguide\/cli-configure-files.html\" target=\"_blank\" rel=\"noopener noreferrer\">AWS credential file<\/a> or a custom passwd file. In this tutorial we will use the first method but if you are interested for the latter feel free to view the <a href=\"https:\/\/github.com\/s3fs-fuse\/s3fs-fuse\" target=\"_blank\" rel=\"noopener noreferrer\">s3fs documentation <\/a>on Github. To setup your credentials make sure that the file ~\/.aws\/credentials contains your AWS access id and secret key. It should look like this:<\/p>\n<p><code><br \/>\n[default] <br \/>aws_access_key_id=YOUR_ACCESS_KEY_ID <br \/>aws_secret_access_key=YOUR_SECRET_ACCESS_KEY<br \/>\n<\/code><\/p>\n<h3>Mounting your bucket to your local filesystem<\/h3>\n<p>Once your have your credentials file you are ready to mount your backup bucket. If you don\u2019t remember the bucket name you can find it by visiting your <a href=\"https:\/\/console.aws.amazon.com\/console\/home\" target=\"_blank\" rel=\"noopener noreferrer\">AWS account<\/a>. To mount and unmount the bucket to\/from a specific location type:<\/p>\n<p>\n<code><br \/>\n# mount <br \/>s3fs BUCKET_NAME \/path\/to\/location <\/code><\/p>\n<p># unmount <br \/>fusermount -u \/path\/to\/location\n<\/p>\n<p>Mounting the bucket like this is only temporary and will not persist across reboots. You can add it on \/etc\/fstab but I believe this only works with the passwd file. If you want to use your AWS credentials file an easy workaround it to create a shortcut in your Startup Applications Preferences.<\/p>\n<p><img decoding=\"async\" src=\"http:\/\/blog.datumbox.com\/wp-content\/uploads\/2020\/10\/Startup.png\"><\/p>\n<p>Note that you can add a small 10 sec delay to ensure that the WiFi is connected before you try to mount the bucket. Internet access is obviously necessary for mounting it successfully. If you are behind VPNs or have other complex setups, you can also create a bash script that makes the necessary checks before you execute the mount command. Sky is the limit!<\/p>\n<h3>Configuring DejaDup<\/h3>\n<p>With the bucket mounted as a local drive, we can now easily configure DejaDup to use it. First of all we need to change the backend to local. This can be done either by using a program like dconfig or the console with the following command:<\/p>\n<p><code>gsettings set org.gnome.DejaDup backend 'local'<\/code><\/p>\n<p><img decoding=\"async\" src=\"http:\/\/blog.datumbox.com\/wp-content\/uploads\/2020\/10\/deconfig.png\"><\/p>\n<p>Finally we open DejaDup, go to preferences and point the storage location to the directory that has your S3 backup files. Make sure you select the subdirectory that contains the backup files; this is typically a subdirectory in your mount point that has name equal to your computer\u2019s hostname. Last but not least, make sure that the S3 mount directory is excluded from DejaDup! To do this, check the ignored folders in Preferences.<\/p>\n<p><img decoding=\"async\" src=\"http:\/\/blog.datumbox.com\/wp-content\/uploads\/2020\/10\/location.png\"><\/p>\n<p>That\u2019s it! Now go to your restore tab and DejaDup will be able to read your previous backups. You can also take new ones.<\/p>\n<h2>Gotchas<\/h2>\n<p>There are a few things to keep in mind in this setup:<\/p>\n<ol>\n<li>First of all, you must be connected on the internet when you mount the bucket. If you are not the bucket won\u2019t be mounted. So, I advise you instead of just calling the mount command, to write a bash script that does the necessary checks before mounting (internet connection is on, firewall allows external requests etc).<\/li>\n<li>Taking backups like that seems slower than using the old native S3 support and it is likely to generate more network traffic (mind AWS traffic costs!). This is expected because DejaDup thinks it\u2019s accessing the local file-system, so there is no need for aggressive caching or minimization of operations that cause network traffic.<\/li>\n<li>You should expect stability issues. As we said earlier, DejaDup does not know it writes data over the wire so much of the functionalities that usually exist in such setups (such as retry-on-fail) are missing. And obviously if you lose connection midway of the backup you will have to delete it and start a new one to avoid corrupting your future backups.<\/li>\n<li>Finally keep in mind that this is a very experimental setup and if you really want to have a reliable solution, you should do your own research and select something that meets your needs.<\/li>\n<\/ol>\n<p>If you have a recommendation for an Open-Source Backup solution that allows locally encrypted incremental backups, supports S3 and has an easy to use UI please leave a comment as I\u2019m more than happy to give it a try. <\/p>\n<div class=\"authorsure-author-box\">\n<img decoding=\"async\" loading=\"lazy\" alt=\"\" src=\"http:\/\/0.gravatar.com\/avatar\/0d9cf005f5d9fbbe42b6723e723a7f76?s=90&amp;r=g\" class=\"avatar avatar-90 photo\" height=\"90\" width=\"90\"><\/p>\n<h4>About\u00a0<a rel=\"author\" href=\"http:\/\/blog.datumbox.com\/author\/bbriniotis\/\" class=\"authorsure-author-link\">Vasilis Vryniotis<\/a><br \/>\n<\/h4>\n<p>My name is Vasilis Vryniotis. I&#8217;m a Data Scientist, a Software Engineer, author of Datumbox Machine Learning Framework and a proud geek. <a href=\"http:\/\/blog.datumbox.com\/author\/bbriniotis\/\">Learn more<\/a><\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>http:\/\/blog.datumbox.com\/how-to-take-s3-backups-with-dejadup-on-ubuntu-20-10\/<\/p>\n","protected":false},"author":0,"featured_media":419,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/418"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=418"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/418\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/419"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=418"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=418"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=418"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}