Lecture Notes: CS20SI TensorFlow for Deep Learning Research


Recently, I’m watching TensorFlow tutorial videos from this YouTube Channel. The tutor Labhesh Patel introduced TensorFlow Graphs, APIs, and popular algorithms with the slides of Stanford CS20SI. I will take notes while learning this course.

Lecture 1: TensorFlow and Graph


Even though there are tons of tutorials, they are too shallow and data are not hitting the points. When the slides of CS20SI from Stanford are available for public, the tutor of this course is going to create a brief introduction of TensorFlow.

What is TensorFlow?

  • An open source software library for numerical computation using data flow graphs.
  • Originally developed by Google Brain Team to conduct Machine Learning and Deep Neural Networks Research.
  • General enough to be applicable to a wide variety of other domains as well.

TensorFlow provides an extensive suite of functions and classes that allow users to build various models from scratch.

There’s a little different between the open source version and private version of TensorFlow.

The basic is creating a graph and computing data. There are many deep learning libraries and TensorFlow is most popular.

Why TensorFlow?

Python API. Portability: CPUs or GPUs or even mobile device. Checkpoints (Model could be saved). Auto-differentiation. Visualization (TensorBoard).

Implementation: Neural Style Translation, Generative Handwriting, WaveNet: Text t Speech.

Goals of this course: Understand TensorFlow’s computation graph approach, explore TensorFlow’s built-in functions and learn how to build and structure models best suited for a deep learning project.

import tensorflow as tf

Simplified TensorFlow such as TFLearn, TFSlim or Keras could make our program more simple, but we don’t need baby TensorFlow. We are going to learn making model from scratch.


An example of Graph

Data Flow Graphs: TensorFlow separates the definition of computations from their execution.

Phase 1: Assemble a graph

Phase 2: Use a session to execute operations of your graph.


An n-dimensional matrix.

0-d tensor: scalar (number)

1-d tensor: vector

2-d tensor: matrix, and so on.


import tensorflow as tf

a = tf.add(2,3) (Note: tf automatically name the variants)

Nodes: operators, variables and constants; Edges: tensors.

Tensors are data. Data Flow->TensorFlow

If we print a now, we will get a scalar-tensor with shape = (). Nothing has been done.

If we want to get the value of a, we could create a session, assign it to variable sess so we can call it later. Within this session, evaluate the graph to fetch the value of a.

import tensorflow as tf

a = tf.add(2,3)

with tf.Session as sess:
    print sess.run(a)

When a session is defined and called, TensorFlow will find all the nodes or resources to compute the value. (All the nodes that lead to the specific node.) But the useless nodes won’t be calculated in order to save resources. However, if you really want to calculate useless nodes, you could pass all the nodes when you run the session.

sess.run(a, useless) (Since the session run API will take a list argument)

More Subgraphs

An example of Subgraphs

It’s possible to break graph into several chunks and run them parallelly across multiple CPUs or GPUs. It’s better to build disconnected subgraphs within one graph rather than build multiple graphs.

Create a graph:


To add operators to a graph, set it as default:

with g.as_default()
    x = tf.add(3,5)

sess = tf.session (graph=g)

with tf.session() as sess:

To handle the default graph:

g = tf.get_default_graph()

Don’t mix default graph and user created graphs.

Why Graphs

1. Save computation (only run subgraphs that lead to the values you want to fetch)

2. Break computation into small, differential pieces to facilitates auto-differentiation

3. Facilitate distributed computation, spread the work across multiple CPUs, GPUs, or devices.

4. Many common machine learning models are commonly taught and visualized as directed graphs already.

Published by

Siujoeng Lau

Liberty will never perish.

Leave a Reply

Your email address will not be published. Required fields are marked *