{"id":200,"date":"2020-09-10T01:23:52","date_gmt":"2020-09-10T01:23:52","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/09\/10\/visualizing-tensorflow-training-jobs-with-tensorboard\/"},"modified":"2020-09-10T01:23:52","modified_gmt":"2020-09-10T01:23:52","slug":"visualizing-tensorflow-training-jobs-with-tensorboard","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/09\/10\/visualizing-tensorflow-training-jobs-with-tensorboard\/","title":{"rendered":"Visualizing TensorFlow training jobs with TensorBoard"},"content":{"rendered":"
\n

TensorBoard<\/a> is an open-source toolkit for TensorFlow<\/a> users that allows you to visualize a wide range of useful information about your model, from model graphs; to loss, accuracy, or custom metrics; to embedding projections, images, and histograms of weights and biases.<\/p>\n

This post demonstrates how to use TensorBoard with Amazon SageMaker<\/a> training jobs, write logs from TensorFlow training scripts to Amazon Simple Storage Service<\/a> (Amazon S3), and ways to run TensorBoard: locally, using Amazon Elastic Container Service<\/a> (Amazon ECS) on AWS Fargate<\/a>, or inside of an Amazon SageMaker notebook instance<\/a>.<\/p>\n

Generating training logs using tf.summary<\/h2>\n

TensorFlow comes with a tf.summary<\/code> module to write summary data, which it uses for monitoring and visualization. The module\u2019s API provides methods to write scalars, audio, histograms, text, and image summaries, and can trace information that\u2019s useful for profiling training jobs. An example command to write the accuracy of the first step of training looks like the following:<\/p>\n

\n
tf.summary.scalar('accuracy', 0.45, step=1)<\/code><\/pre>\n<\/div>\n
To use the summary data after the training job is complete, it\u2019s important to write the files to a persistent storage. This way, you can visualize your past jobs or compare different runs during the hyperparameter tuning phase. The tf.summary<\/code> module allows you to use Amazon S3 as the destination for log files, passing the S3 bucket URI directly into the create_file_writer<\/code> method. See the following code:<\/p>\n
\n
tf.summary.create_file_writer('s3:\/\/<bucket_name>\/<prefix>')<\/code><\/pre>\n<\/div>\n
Keras<\/a> users can use keras.callbacks.TensorBoard<\/code> as one of the callbacks provided to the Model.fit()<\/code> method. This callback provides an abstraction of a low-level tf.summary<\/code> API and collects a lot of the data automatically. With TensorBoard callbacks, you can collect data to visualize training graphs, metrics plots, activation histograms, and run profiling. See the following code:<\/p>\n
\n
tb_callback = tf.keras.callbacks.TensorBoard(log_dir='s3:\/\/<bucket_name>\/<prefix>')\r\nmodel.fit(x, y, epochs=5, callbacks=[tb_callback])<\/code><\/pre>\n<\/div>\n
For a detailed example of how to collect summary data in the training scripts, see the TensorBoard Keras example notebook on the Amazon SageMaker examples GitHub repo<\/a> or inside a running Amazon SageMaker notebook instance on the Amazon SageMaker Examples<\/strong> tab. This notebook uses TensorFlow 2.2 and Keras to train a Convolutional Neural Network (CNN) to recognize images from the CIFAR-10 dataset<\/a>. Code in the notebook runs the training job locally inside the notebook instance one time, and then another 10 times during the hyperparameter tuning job. All training jobs write log files under one Amazon S3 prefix, so the log destination path for every run follows the format s3:\/\/<bucket_name><\/em>\/<project_name><\/em>\/logs\/<training_job_name><\/em><\/code><\/span>, where the project name is tensorboard_keras_cifar10<\/code>.<\/p>\n
\"\"<\/p>\n
The notebook also demonstrates how to run TensorBoard inside of the Amazon SageMaker notebook instance. This method has some limitations; for example, the TensorBoard command blocks the run of the notebook and lives as long as the notebook instance is alive, but allows you to quickly access the dashboard and make sure the training is running correctly.<\/p>\n
In the following sections, we look at other ways to run TensorBoard.<\/p>\n
Running TensorBoard on your local machine<\/h2>\n
If you want to run TensorBoard locally, the first thing you need to do is to install TensorFlow:<\/p>\n
An independent distribution of TensorBoard is also available, but it has limited functionality if run without TensorFlow. For this post, we use TensorBoard as part of the TensorFlow distribution.<\/p>\n
Assuming your AWS Command Line Interface<\/a> (AWS CLI) is installed and configured properly, we simply run TensorBoard pointing to the Amazon S3 directory containing the generated summary data:<\/p>\n
\n
AWS_REGION=eu-west-1 tensorboard --logdir s3:\/\/<bucket_name>\/tensorboard_keras_cifar10\/logs\/<\/code><\/pre>\n<\/div>\n
You must specify the region where your S3 bucket is located. You can find the right region in the list of buckets<\/a> on the Amazon S3 console.<\/p>\n
The user you use must have read access to the specified S3 bucket. For more information about securely granting access to S3 buckets to a specific user, see Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket<\/a>.<\/p>\n
You should see something similar to the following screenshot.<\/p>\n
\"\"<\/p>\n
Running TensorBoard on Amazon ECS on AWS Fargate<\/h2>\n
If you prefer to have an instance of TensorBoard permanently running and accessible to your whole team, you can deploy it as an independent application in the cloud. One of the easiest ways to do this without managing servers is AWS Fargate, a serverless compute engine for containers. The following diagram illustrates this architecture.<\/p>\n
\"\"<\/p>\n
You can deploy an example TensorBoard container image with all required roles and an Application Load Balancer by using the provided AWS CloudFormation<\/a> template:<\/p>\n
\"\"<\/a><\/p>\n
\u00a0<\/p>\n
\u00a0<\/p>\n
This template has five input parameters:<\/p>\n