{"id":1155,"date":"2021-11-06T08:40:00","date_gmt":"2021-11-06T08:40:00","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/11\/06\/run-alphafold-v2-0-on-amazon-ec2\/"},"modified":"2021-11-06T08:40:00","modified_gmt":"2021-11-06T08:40:00","slug":"run-alphafold-v2-0-on-amazon-ec2","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/11\/06\/run-alphafold-v2-0-on-amazon-ec2\/","title":{"rendered":"Run AlphaFold v2.0 on Amazon EC2"},"content":{"rendered":"
\n

After the article in Nature<\/em><\/a> about the open-source of AlphaFold v2.0<\/a> on GitHub<\/a> by DeepMind<\/a>, many in the scientific and research community have wanted to try out DeepMind\u2019s AlphaFold implementation firsthand. With compute resources through Amazon Elastic Compute Cloud<\/a> (Amazon EC2) with Nvidia GPU, you can quickly get AlphaFold running and try it out yourself.<\/p>\n

In this post, I provide you with step-by-step instructions on how to install AlphaFold on an EC2 instance with Nvidia GPU.<\/p>\n

Overview of solution<\/h2>\n

The process starts with a Deep Learning Amazon Machine Image (DLAMI). After installation, we run predictions using the AlphaFold model with CASP14 samples on the instance. I also show how to create an Amazon Elastic Block Store<\/a> (Amazon EBS) snapshot for future use to reduce the effort of setting it up again and save costs.<\/p>\n

To run AlphaFold without setting up a new EC2 instance from scratch, go to the last section of this post. You can create a new EC2 instance with the provided public EBS snapshots in a short time.<\/p>\n

The total cost for the AWS resources used in this post cost is less than $100 if you finish all the steps and shut down all resources within 24 hours. If you create an EBS snapshot and store it inside your AWS account, the EBS snapshot storage cost is about $150 per month.<\/p>\n

Launch an EC2 instance with a DLAMI<\/h2>\n

In this section, I demonstrate how to set up an EC2 instance using a DLAMI from AWS. It already has lots of AlphaFold\u2019s dependencies preinstalled and saves time on the setup.<\/p>\n

    \n
  1. On the Amazon EC2 console, choose your preferred AWS Region.<\/li>\n
  2. Launch a new EC2 instance with DLAMI by searching Deep Learning AMI. I use DLAMI version 48.0 based on Ubuntu 18.04. This is the latest version at the time of this writing.\"EC2<\/li>\n
  3. Select a p3.2xlarge instance with one GPU as the instance type. If you don\u2019t have enough quota on a p3.2xlarge instance, you can increase the Amazon EC2 quota on your AWS account.\"P3<\/li>\n
  4. Configure the proper Amazon Virtual Private Cloud<\/a> (Amazon VPC) setting based on your AWS environment requirements. If this is your first time configuring your Amazon VPC, consider using the default Amazon VPC and review Get started with Amazon VPC<\/a>.<\/li>\n
  5. Set the system volume to 200 GiB, and add one new data volume of 3 TB (3072 GiB) in size.
    \"EBS<\/li>\n
  6. Make sure that the security group settings<\/a> allow you to access the EC2 instance with SSH, and the EC2 instance can reach the internet to install AlphaFold and other packages.<\/li>\n
  7. Launch the EC2 instance.<\/li>\n
  8. Wait for the EC2 instance to become ready and use SSH to access the Amazon EC2 terminal.<\/li>\n
  9. Optionally, if you have other required software for the new EC2 instance, install it now.<\/li>\n<\/ol>\n

    Install AlphaFold<\/h2>\n

    You\u2019re now ready to install AlphaFold.<\/p>\n

      \n
    1. After you use SSH to access the Amazon EC2 terminal, first update all packages:\n <\/li>\n
    2. Mount the data volume to the folder \/data<\/code>. For more details, refer to the Make an Amazon EBS volume available for use on Linux<\/a><\/li>\n
    3. Use the lsblk<\/code> command to view your available disk devices and their mount points (if applicable) to help you determine the correct device name to use:\n
      \n
      lsblk<\/code><\/pre>\n
      \"lsblk<\/p>\n<\/p><\/div>\n<\/li>\n
    4. Determine whether there is a file system on the volume. New volumes are raw block devices, and you must create a file system on them before you can mount and use them. The device is an empty volume.\n
      \n
      sudo file -s \/dev\/xvdb<\/code><\/pre>\n
      \"sudo<\/p>\n<\/p><\/div>\n<\/li>\n
    5. Create a file system on the volume and mount the volume to the \/data<\/code> folder:\n
      \n
      sudo mkfs.xfs \/dev\/xvdb\nsudo mkdir \/data\nsudo mount \/dev\/xvdb \/data\nsudo chown ubuntu:ubuntu -R \/data\ndf -h \/data<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n
    6. Install the AlphaFold dependencies and any other required tools:\n
      \n
      sudo apt install aria2 rsync git vim wget tmux tree -y<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n
    7. Create working folders, and clone the AlphaFold code from the GitHub repo:\n
      \n
      cd \/data\nmkdir -p \/data\/af_download_data\nmkdir -p \/data\/output\/alphafold\nmkdir -p \/data\/input\ngit clone https:\/\/github.com\/deepmind\/alphafold.git<\/code><\/pre>\n<\/p><\/div>\n
      You use the new volume exclusively, so the snapshot you create later has all the necessary data.<\/p>\n<\/li>\n
    8. Download the data using the provided scripts in the background. AlphaFold needs multiple genetic (sequence) database and model parameters.\n
      \n
      nohup \/data\/alphafold\/scripts\/download_all_data.sh \/data\/af_download_data &<\/code><\/pre>\n<\/p><\/div>\n
      The whole download process could take over 10 hours, so wait for it to finish. You can use the following command to monitor the download and unzip process:<\/p>\n
      \n
      du -sh \/data\/af_download_data\/*<\/code><\/pre>\n<\/p><\/div>\n
      When the download process is complete, you should have the following files in your \/data\/af_download_data<\/code> folder:<\/p>\n
      \n
      $DOWNLOAD_DIR\/                             # Total: ~ 2.2 TB (download: 438 GB)\n    bfd\/                                   # ~ 1.7 TB (download: 271.6 GB)\n        # 6 files.\n    mgnify\/                                # ~ 64 GB (download: 32.9 GB)\n        mgy_clusters_2018_12.fa\n    params\/                                # ~ 3.5 GB (download: 3.5 GB)\n        # 5 CASP14 models,\n        # 5 pTM models,\n        # LICENSE,\n        # = 11 files.\n    pdb70\/                                 # ~ 56 GB (download: 19.5 GB)\n        # 9 files.\n    pdb_mmcif\/                             # ~ 206 GB (download: 46 GB)\n        mmcif_files\/\n            # About 180,000 .cif files.\n        obsolete.dat\n    small_bfd\/                             # ~ 17 GB (download: 9.6 GB)\n        bfd-first_non_consensus_sequences.fasta\n    uniclust30\/                            # ~ 86 GB (download: 24.9 GB)\n        uniclust30_2018_08\/\n            # 13 files.\n    uniref90\/                              # ~ 58 GB (download: 29.7 GB)\n        uniref90.fasta<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n
    9. Update \/data\/alphafold\/docker\/run_docker.py<\/code> to make the configuration march the local path:\n
      \n
      vim \/data\/alphafold\/docker\/run_docker.py<\/code><\/pre>\n<\/p><\/div>\n
      With the folders you\u2019ve created, the configurations look like the following. If you set up a different folder structure in your EC2 instance, set it accordingly.<\/p>\n
      \n
      #### USER CONFIGURATION ####\n\n# Set to target of scripts\/download_all_databases.sh\nDOWNLOAD_DIR = '\/data\/af_download_data'\n\n# Name of the AlphaFold Docker image.\ndocker_image_name = 'alphafold'\n\n# Path to a directory that will store the results.\noutput_dir = '\/data\/output\/alphafold'<\/code><\/pre>\n
      \"User<\/p>\n<\/p><\/div>\n<\/li>\n
    10. Confirm the NVidia container kit is installed:\n
      \n
      sudo docker run --rm --gpus all nvidia\/cuda:11.0-base nvidia-smi<\/code><\/pre>\n<\/p><\/div>\n
      You should see similar output to the following screenshot.
      \"NVidia<\/p>\n<\/li>\n
    11. Build the AlphaFold Docker image. Make sure that the local path is \/data\/alphafold<\/code> because a .dockerignore<\/code> file is under that folder.\n
      \n
      cd \/data\/alphafold\ndocker build -f docker\/Dockerfile -t alphafold .\ndocker images<\/code><\/pre>\n<\/p><\/div>\n
      You should see the new Docker image after the build is complete.
      \"alpha<\/p>\n<\/li>\n
    12. Use pip to install all Python dependencies required by AlphaFold:\n
      \n
      pip3 install -r \/data\/alphafold\/docker\/requirements.txt<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n
    13. Go to the CASP14 target list<\/a> and copy the sequence from the plaintext link for T1050<\/a>.
      \"CASP14<\/li>\n
    14. Copy the content into a new T1050.fasta<\/code> file and save it under the \/data\/input<\/code> folder.
      \"T1050<\/li>\n
    15. You can use this same process to create a few more .fasta<\/code> files for testing under the \/data\/input<\/code> folder.
      \"fasta<\/li>\n<\/ol>\n
      Install CloudWatch monitoring for GPU (Optional)<\/h2>\n
      Optionally, you can install Amazon CloudWatch<\/a> monitoring for GPU. This requires an AWS Identity and Access Management<\/a> (IAM) role.<\/p>\n
        \n
      1. Create an Amazon EC2 IAM role<\/a> for CloudWatch and attach it to the EC2 instance.
        \"IAM<\/li>\n
      2. Change the Region in gpumon.py<\/code> if your instance is in another Region, and provide a new namespace like AlphaFold<\/code> as the CloudWatch namespace:\n
        \n
        vim ~\/tools\/GPUCloudWatchMonitor\/gpumon.py<\/code><\/pre>\n
        \"gpumon<\/p>\n<\/p><\/div>\n<\/li>\n
      3. Launch gpumon<\/code> and start sending GPU metrics to CloudWatch:\n
        \n
        source activate python3\npython ~\/tools\/GPUCloudWatchMonitor\/gpumon.py &<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n<\/ol>\n
        Use AlphaFold for prediction<\/h2>\n
        We\u2019re now ready to run predictions with AlphaFold.<\/p>\n
          \n
        1. Use the following command to run a prediction of a protein sequence from \/data\/input\/T1050.fasta<\/code>:\n
          \n
          nohup python3 \/data\/alphafold\/docker\/run_docker.py --fasta_paths=\/data\/input\/T1050.fasta --max_template_date=2020-05-14 &<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n
        2. Use the tail<\/code> command to monitor the prediction progress:\n
          The whole prediction takes a few hours to finish. When the prediction is complete, you should see the following in the output folder. In this case, <target_name><\/code> is T1050<\/code>.<\/p>\n
          \n
          <target_name>\/\n    features.pkl\n    ranked_{0,1,2,3,4}.pdb\n    ranking_debug.json\n    relaxed_model_{1,2,3,4,5}.pdb\n    result_model_{1,2,3,4,5}.pkl\n    timings.json\n    unrelaxed_model_{1,2,3,4,5}.pdb\n    msas\/\n        bfd_uniclust_hits.a3m\n        mgnify_hits.sto\n        pdb70_hits.hhr\n        uniref90_hits.sto\n<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n
        3. Change the owner of the output folder from root<\/code> so you can copy them:\n
          \n
          sudo chown ubuntu:ubuntu \/data\/output\/alphafold\/ -R<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n
        4. Use scp<\/code> to copy the output from the prediction output folder to your local folder:\n
          \n
          scp -i <ec2-key-path>.pem -r ubuntu@<ec2-ip>:\/data\/output\/alphafold\/T1050 ~\/Downloads\/<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n
        5. Use this protein 3D viewer<\/a> from NIH to view the predicted 3D structure from your result folder.<\/li>\n
        6. Select ranked_0.pdb<\/code>, which contains the prediction with the highest confidence.
          \"Open
          The following is a 3D view of the predicted structure for T1050<\/code> by AlphaFold.
          \"T1050<\/li>\n<\/ol>\n
          Create a snapshot from the data volume<\/h2>\n
          It takes time to install AlphaFold on an EC2 instance. However, the P3 instance and the EBS volume can become expensive if you keep them running all the time. You may want to have an EC2 instance ready quickly but also don\u2019t want to spend time rebuilding the environment every time you need it. An EBS snapshot helps you save both time and cost.<\/p>\n
            \n
          1. On the Amazon EC2 console, choose Volumes<\/strong> in the navigation pane under Elastic Block Store<\/strong>.<\/li>\n
          2. Filter by the EC2 instance ID.Two volumes should be listed.<\/li>\n
          3. Select the data volume with 3072 GiB in size.<\/li>\n
          4. On the Actions <\/strong>menu, choose Create snapshot.
            \"Create
            <\/strong>The snapshot takes a few hours to finish.<\/li>\n
          5. When the snapshot is complete, choose Snapshots<\/strong>, and your new snapshot should be in the list.<\/li>\n<\/ol>\n
            You can safely shut down your EC2 instance now. At this point, all the data in the data volume is safely stored in the snapshot for future use.<\/p>\n
            Recreate the EC2 instance with a snapshot<\/h2>\n
            To recreate a new EC2 instance with AlphaFold, the first steps are similar to what you did earlier when creating an EC2 instance from scratch. But instead of creating the data volume from scratch, you attach a new volume restored from the Amazon EBS snapshot.<\/p>\n
              \n
            1. Open the Amazon EC2 console and in the AWS Region of your choice, and launch a new EC2 instance with DLAMI by searching Deep Learning AMI<\/code>.<\/li>\n
            2. Choose the DLAMI based on Ubuntu 18.04.
              \"EC2<\/li>\n
            3. Select p3.2xlarge with one GPU as the instance type. If you don\u2019t have enough quota on a p3.2xlarge instance, you can increase the Amazon EC2 quota<\/a> on your AWS account.
              \"P3<\/li>\n
            4. Configure the proper Amazon VPC setting based on your AWS environment requirements. If this is your first time configuring your Amazon VPC, consider using the default Amazon VPC and review Get started with Amazon VPC<\/a>.<\/li>\n
            5. Set the system volume to 200 GiB, but don\u2019t add a new data volume.
              \"EBS<\/li>\n
            6. Make sure that the security group settings<\/a> allow you to access the EC2 instance and the EC2 instance can reach the internet to install Python and Docker packages.<\/li>\n
            7. Launch the EC2 instance.<\/li>\n
            8. Take note of which Availability Zone the instance is in and the instance ID, because you use them in a later step.
              \"restored<\/li>\n
            9. On the Amazon EC2 console, choose Snapshots<\/strong>.<\/li>\n
            10. Select the snapshot you created earlier or use the public snapshot provided.<\/li>\n
            11. On the Actions<\/strong> menu, choose Create Volume<\/strong>.
              For this post, we provide public snapshots in Regions us-east-1<\/code>, us-west-2<\/code>, and eu-west-1<\/code>. You can search public snapshots by snapshot ID: snap-0d736c6e22d0110d0<\/code> in us-east-1<\/code>,snap-080e5bbdfe190ee7e<\/code> in us-west-2<\/code>,snap-08d06a7c7c3295567<\/code> in eu-west-1<\/code>.<\/li>\n
            12. Set up the new data volume settings accordingly. Make sure that the Availability Zone is the same as the newly created EC2 instance. Otherwise, you can\u2019t mount the volume to the new EC2 instance.<\/li>\n
            13. Choose Create Volume<\/strong> to create the new data volume.
              \"restore<\/li>\n
            14. Choose Volumes<\/strong>, and you should see the newly created data volume. Its state should be available<\/code>.<\/li>\n
            15. Select the volume, and on the Actions <\/strong>menu, choose Attach volume<\/strong>.<\/li>\n
            16. Choose the newly created EC2 instance and attach the volume.
              \"Attached<\/li>\n
            17. Use SSH to access the Amazon EC2 terminal and run lsblk<\/code>. You should see that the new data volume is unmounted. In this case, it is \/dev\/xvdf<\/code>.
              \"lsblk\"<\/li>\n
            18. Determine whether there is a file system on the volume. The data volume created from the snapshot has an XFS file system on it already.\n
              \n
              sudo file -s \/dev\/xvdf<\/code><\/pre>\n
              \"sudo<\/p>\n<\/p><\/div>\n<\/li>\n
            19. Mount the new data volume to the \/data<\/code> folder:\n
              \n
              sudo mkdir \/data\nsudo mount \/dev\/xvdf \/data\nsudo chown ubuntu:ubuntu -R \/data<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n
            20. Update all packages on the system and install the dependencies. You do need to rebuild the AlphaFold Docker image.\n
              \n
              sudo apt update\nsudo apt install aria2 rsync git vim wget tmux tree -y\npip3 install -r \/data\/alphafold\/docker\/requirements.txt\n\ncd \/data\/alphafold\ndocker build -f docker\/Dockerfile -t alphafold .\ndocker images<\/code><\/pre>\n
              \"alpha<\/p>\n<\/p><\/div>\n<\/li>\n
            21. Confirm that the Nvidia container kit is installed:\n
              \n
              sudo docker run --rm --gpus all nvidia\/cuda:11.0-base nvidia-smi<\/code><\/pre>\n<\/p><\/div>\n
              You should see output like the following screenshot.
              \"NVidia<\/p>\n<\/li>\n
            22. Use the following command to run a prediction of a protein sequence from \/data\/input\/T1024.fasta<\/code>:\n
              \n
              cd \/data\nnohup python3 \/data\/alphafold\/docker\/run_docker.py --fasta_paths=\/data\/input\/T1024.fasta --max_template_date=2020-05-14 &<\/code><\/pre>\n<\/p><\/div>\n
              I use a different protein sequence because the snapshot contains the result from T1050<\/code> already. If you want to run the prediction for T1050<\/code> again, first delete or rename the existing T1050<\/code> result folder before running the new prediction.<\/p>\n<\/li>\n
            23. Use the tail command to monitor the prediction progress:\n
              The whole prediction takes a few hours to finish.<\/p>\n<\/li>\n<\/ol>\n
              Clean up<\/h2>\n
              When you finish all your predictions and copy your results locally, clean up the AWS resources to save cost. You can safely shut down the EC2 instance and delete the EBS data volume if it didn\u2019t delete when the EC2 instance was shut down. When you need to use AlphaFold again, you can follow the same process to spin up a new EC2 instance and run new predictions in a matter of minutes. And you don\u2019t incur any additional cost other than the EBS snapshot storage cost.<\/p>\n
              Conclusion<\/h2>\n
              With Amazon EC2 with Nvidia GPU and the Deep Learning AMI, you can install the new AlphaFold implementation from DeepMind and run predictions over CASP14 samples. Because you back up the data on the data volumes to point-in-time snapshots, you avoid paying for EC2 instances and EBS volumes when you don\u2019t need them. Creating an EBS volume based on the previous snapshot greatly shortens the time needed to recreate the EC2 instance with AlphaFold. Therefore, you can start running your predictions in a short amount of time.<\/p>\n
              \n
              About the Author<\/h3>\n
              \"\"<\/a>Qi Wang<\/strong> is a Sr. Solutions Architect on the Global Healthcare and Life Science team at AWS. He has over 10 years of experience working in the healthcare and life science vertical in business innovation and digital transformation. At AWS, he works closely with life science customers transforming drug discovery, clinical trials, and drug commercialization.<\/p>\n
                     \n      <\/div>\n","protected":false},"excerpt":{"rendered":"
              https:\/\/aws.amazon.com\/blogs\/machine-learning\/run-alphafold-v2-0-on-amazon-ec2\/<\/p>\n","protected":false},"author":0,"featured_media":1156,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1155"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=1155"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1155\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/1156"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=1155"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=1155"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=1155"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}