GAN (Generative Adversarial Network) is featured by two opponents: generator network and discriminator network. Generator network generates fake images from noise, trying to fool discriminator to classify them into real image, while discriminator trying to distinguish between real ones from the training set and fake images from generator.

1. Sketch of framework

2. Minimax of objective function

In the formula below, $\theta_d$ is discriminator’s parameter, while $\theta_g$ is generator’s parameter. $D_{\theta_d(\cdot)}$ is the output of discriminator which is a value between 0 and 1 meaning the likelihood of real image.

$$\min_{\substack{\theta_{g}}} \max_{\substack{\theta_{d}}} [E_{x \sim p_{data}(x)} logD_{\theta_d}(x) + E_{z \sim p_z(z)} log(1 - D_{\theta_{d}}(G_{\theta_{g}}(z))) ]$$

For discriminator, it wants to maximize the objective function so that $D_{\theta_d}(x)$ is close to 1 ($x$ is from real images) and $ D_{\theta_{d}}(G_{\theta_{g}}(z)))$ is close to zero ($G_{\theta_{g}}(z)$ is a generated fake image). On the contrary, generator wants to minimize objective such that $D_{\theta_{d}}(G_{\theta_{g}}(z)))$ is close to 1 (meaning discriminator is fooled into thinking $G(z)$ is real)

3. Algorithm

for #training iterations do
    for k steps do
         Sample minibatch of m noise samples {$z^{(1)}, ..., z^{(m)}$} from noise prior
         Sample minibatch of m samples {$x^{(1)}, ..., x^{(m)}$} from data generating distribution
         Update the discriminator by ascending its stochastic gradient:

$$\nabla_{\theta_d} \frac{1}{m}\Sigma_{i=1}^{m} [logD(x^{(i)}) + log(1 - D(G(z^{(i)}))) ]$$

     end for

         Sample minibatch of m noise samples {$z^{(1)}, ..., z^{(m)}$} from noise prior $p_g(z)$

         Update the generator by descending its stochastic gradient:

$$\nabla_{\theta_d} \frac{1}{m}\Sigma_{i=1}^{m} log(1 - D(G(z^{(i)})))$$

end for

4. Implementation Example: generate hand-written digits using Convolutional GAN

Here we use MNIST dataset which contains 60,000 training and 10,000 test images of human hand-written digits.

Sample images in MNIST dataset:

Original image dimension is $28 \times 28$, but is flattened by TensorFlow MNIST wrapper to a $784$d vector, i.e. dimension of input to network is (batch_size, $784$) where we use $128$ as batch_size in this example.

1. Discriminator: compute score for a batch of input images

Code:

def discriminator(x):
    with tf.variable_scope("discriminator"):
        x_in = tf.reshape(x, [-1, 28, 28, 1])
        # Convolutional Layer #1
        conv1 =
          tf.layers.conv2d(inputs=x_in, filters=32, kernel_size=[5, 5], padding="valid", activation=leaky_relu)
        # Pooling Layer #1
        pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)
        # Convolutional Layer #2
        conv2 =
          tf.layers.conv2d(inputs=pool1, filters=64, kernel_size=[5, 5], padding="valid", activation=leaky_relu)
        # Pooling Layer #2
        pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)
        # flatten
        pool2_flat = tf.contrib.layers.flatten(pool2)
        # fully connected layer
        fc1 = tf.layers.dense(inputs=pool2_flat, units=1024, activation=leaky_relu)
        logits = tf.layers.dense(inputs=fc1, units=1)
        return logits

where the leaky RELU activation function is defined as

def leaky_relu(x, alpha=0.01):
    return tf.maximum(alpha*x, x)

2. Generator: generate images from a random noise vector.

Code:

def generator(z):
  with tf.variable_scope("generator"):
      fc1 = tf.layers.dense(inputs=z, units=1024, activation=tf.nn.relu)
      bn1 = tf.layers.batch_normalization(fc1)
      fc2 = tf.layers.dense(inputs=bn1, units=7*7*128, activation=tf.nn.relu)
      bn2 = tf.layers.batch_normalization(fc2)
      bn2_resized = tf.reshape(bn2, (-1, 7, 7, 128))
      convt1 = tf.contrib.layers.conv2d_transpose(bn2_resized, 64, (4,4), (2,2), activation_fn=tf.nn.relu)
      bn3 = tf.layers.batch_normalization(convt1)
      img = tf.contrib.layers.conv2d_transpose(bn3, 1, (4,4), (2,2), activation_fn=tf.nn.tanh)
      return img

3. Least squares loss function for GAN

Least squares loss is a newly proposed, more stable loss function than the original GAN’s.

$$l_G = \frac{1}{2}E_{z \sim p(z)}[D(G(Z))-1)^2]$$

$$l_D = \frac{1}{2}E_{x \sim p_{data}}[(D(X)-1)^2] + \frac{1}{2}E_{z \sim p(z)}[D(G(Z))^2]$$

Here we use average instead of computing the expectation.

Code:

def lsgan_loss(score_real, score_fake):
    G_loss = 0.5*tf.reduce_mean(tf.square(score_fake - 1))
    D_loss = 0.5*tf.reduce_mean(tf.square(score_real - 1))+0.5*tf.reduce_mean(tf.square(score_fake))
    return D_loss, G_loss

Result

First 200 iteration output:

1000 Iteration:

Final output:

References

1. Goodfellow, Ian, et al. “Generative adversarial nets.” Advances in neural information processing systems. 2014.
2. http://cs231n.stanford.edu/