一.引言

1.1文字生成图像

下图是传统监督学习文字生成图像的示意图：

存在的问题：输出较多，且神经网络只能给出一中模糊的结果，给不出一个具体的结果及自己真正想生成的效果。接下来具体给出CGAN的解决方法

二.Conditional GAN

2.1Conditional GAN生成器

与传统的GAN的不同之处在于，CGAN的输入有两部分，一部分是随机噪声分布，另外一部分是条件标签输入。

2.1Conditional GAN判别器

对于输入图像X，可能是生成的也可能是伪造的，然后将标签C也同时输入。输出一个标量，可以判断x是不是真实的图片或者C和X是否匹配。

2.2Conditional GAN判别器结构

大多数论文采用的判别器架构如下：

其他判别器架构：

三.StackGAN

StackGAN的结构如下图：

CGAN可以生成带有条件的图像，但是图像精度很低，我们如果希望我们的精度更高一些，因此使用StackGAN可以更好的解决这个问题。我们先将条件跟随机噪声放入生成器，进行一个上采样操作，也就是一个逐步变大的卷积网络中。然后再将生成的伪造的64×64的图片跟真实的图像放入判别器中，进行一个逐步缩小的卷积网络中，上述操作跟传统的CGAN操作基本是一样的，紧接着我们将生成的图像作为条件与64×64的图像一并作为输入再放入到另外一个生成器中，进行放小然后放大，最终生成一个256×256的图像然后进行判别。
核心思想：想要生成高精度的图像，使用两阶段的对抗学习模型，先生成低精度图像，再逐步生成高精度图像。

四.PatchGAN

想要生成高精度图像我们也可以使用PatchGAN，就是对高精度图片进行裁剪，输入图片局部进行逐步生成，PatchGAN并不神秘，其只是一个全卷积网络而已，只是最终输出是一个特征图X，而非一个实数.它就相当于对图像先进行若干次70x70的随机剪裁，将剪裁后图像输入判别器，然后对所有输出的实数值取平均。

五.图像转图像的应用

5.1图像转图像简介

与文字生成图像类似，只是将条件换做是图片即可。类似于下图可应用用如下场景：

5.1语音增强

5.2 视频生成

六.可控数字图像生成应用及代码编写

6.1可控数字图像生成具体代码

可控数字图像生成具体代码

// An highlighted block
import tensorflow as tf
import numpy as np
import time
import os
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import  DataHandler as dh
class CGAN_MNIST(object):
    def __init__(self, noise_dim=10, img_h=32, img_w=32, img_c=1, lr=0.0004):
        """
        初始化CGAN对象
        :param noise_dim:随机噪声维度
        :param img_h: 图像高度
        :param img_w: 图像宽度
        :param img_c: 图像深度
        :param lr: 学习
        """
        self.noise_dim = noise_dim
        self.img_h = img_h
        self.img_w = img_w
        self.img_c = img_c
        self.lr = lr
        self.d_dim = 1
        self.label = tf.placeholder(dtype=tf.float32, shape=[None, 10])
        self.isTrain = tf.placeholder(dtype=tf.bool)
        self.gen_x = tf.placeholder(dtype=tf.float32, shape=[None, 1, 1, self.noise_dim])
        self.gen_out = self._init_generator(input=self.gen_x, label=self.label, istrain=self.isTrain)  # 生成数据
        self.gen_logis = self._init_discriminator(input=self.gen_out, label=self.label, isTrain=self.isTrain)
        self.x = tf.placeholder(dtype=tf.float32, shape=[None, self.img_w, self.img_h, self.img_c], name="input_data")
        self.real_logis = self._init_discriminator(input=self.x, label=self.label, isTrain=self.isTrain, reuse=True)
        self._init_train_methods()

    def _init_discriminator(self, input, label, isTrain=True, reuse=False):
        """
        初始化判别器
        :param input:输入数据op
        :param label: 输入数据标签
        :param isTrain: 是否训练状态
        :param reuse: 是否复用内部参数
        :return: 判断结果
        """
        with tf.variable_scope("discriminator", reuse=reuse):
            # hidden layer 1  input=[None,32,32,1]
            labels = tf.reshape(label, shape=[-1, 1, 1, 10])
            data_sum = tf.shape(input=input)[0]
            la = labels * tf.ones([data_sum, self.img_w, self.img_h, 10])
            input = tf.concat([input, la], axis=3)
            conv1 = tf.layers.conv2d(input, 32, [4, 4], strides=[2, 2], padding="same")
            active1 = tf.nn.leaky_relu(conv1) #[none,16,16,32]
            # layer 2
            conv2 = tf.layers.conv2d(active1, 64, [4, 4], strides=[2, 2], padding="same")
            bn2 = tf.layers.batch_normalization(conv2,training=isTrain)#[none,8,8,64]
            active2 = tf.nn.leaky_relu(bn2)#[none,8,8,64]
            # layer 3
            conv3 = tf.layers.conv2d(active2, 128, [4, 4], strides=[2, 2], padding="same")
            bn3 = tf.layers.batch_normalization(conv3,training=isTrain)
            active3 = tf.nn.leaky_relu(bn3)#[none,4,4,128]
            # out layer
            out_logis = tf.layers.conv2d(active3,1,[4,4],strides=[1,1],padding="valid") #[none,1,1,1]
        return out_logis

    def _init_generator(self, input, label, istrain=True, resue=False):
        """
        初始化生成器
        :param input:输入噪声
        :param label: 输入数据标签
        :param istrain: 是否训练状态
        :param resue: 是否复用内部参数
        :return: 生成数据op
        """
        with tf.variable_scope("generator",reuse=resue):
            # input[none,1,1,self.noise_dim]
            labels = tf.reshape(label,shape=[-1,1,1,10])
            input = tf.concat([input,labels],axis=3)
            # hidden layer 1
            conv1 = tf.layers.conv2d_transpose(input,256,[4,4],strides=(1,1),padding="valid")
            bn1 = tf.layers.batch_normalization(conv1,training=istrain)
            active1 = tf.nn.leaky_relu(bn1) #[none,4,4,256]

            conv2 = tf.layers.conv2d_transpose(active1,128,[4,4],strides=(2,2),padding="same")
            bn2 = tf.layers.batch_normalization(conv2,training=istrain)
            active2 = tf.nn.leaky_relu(bn2) #[none,8,8,128]
            # layer 3
            conv3 = tf.layers.conv2d_transpose(active2,64,[4,4],strides=(2,2),padding='same')
            bn3 = tf.layers.batch_normalization(conv3,training=istrain)
            active3 = tf.nn.leaky_relu(bn3) # [none,16,16,64]
            # out layer
            conv4 = tf.layers.conv2d_transpose(active3,self.img_c,[4,4],strides=(2,2),padding="same")
            out = tf.nn.sigmoid(conv4)  # (0,1)        [None,32,32,1]
        return out

    def _init_train_methods(self):
        """
        初始化训练方法：梯度下降方法，Session初始化,损失函数。
        :return: NONE
        """
        self.D_loss_real =tf.reduce_mean(
            tf.nn.sigmoid_cross_entropy_with_logits(logits=self.real_logis,labels=tf.ones_like(self.real_logis))
        )
        self.D_loss_fake = tf.reduce_mean(
            tf.nn.sigmoid_cross_entropy_with_logits(logits=self.gen_logis,labels=tf.zeros_like(self.gen_logis))
        )
        self.D_loss = self.D_loss_real +self.D_loss_fake
        self.G_loss = tf.reduce_mean(
            tf.nn.sigmoid_cross_entropy_with_logits(logits=self.gen_logis,labels=tf.ones_like(self.gen_logis))
        )
        total_vars = tf.trainable_variables()
        d_vars =[var for var in total_vars if var.name.startswith('discriminator')]
        g_vars = [var for var in total_vars if var.name.startswith('generator')]
        self.D_trainer = tf.train.AdamOptimizer(learning_rate=self.lr).minimize(
            self.D_loss,var_list=d_vars
        )
        self.G_trainer = tf.train.AdamOptimizer(learning_rate=self.lr).minimize(
            self.G_loss,var_list = g_vars
        )

        self.sess = tf.InteractiveSession()
        self.sess.run(tf.global_variables_initializer())
        self.saver = tf.train.Saver(max_to_keep=1)

    def gen_data(self, label, save_path="out/CGAN_MNIST/test.png"):
        """
        生成数字图像
        :param label:数字标签
        :param save_path: 保存路劲
        :return: 图像数组 numpy
        """
        batch_noise = np.random.normal(0,1,(25,1,1,self.noise_dim))
        print(np.argmax(label,axis=1))
        samples = self.sess.run(self.gen_out,feed_dict={self.gen_x:batch_noise,self.label:label,self.isTrain:True})
        # [25,32,32,1] ---->[25,32,32]
        samples =np.reshape(samples,[-1,32,32])
        fig = self.plot(samples)
        if not os.path.exists('out/CGAN_MNIST/'):
            os.makedirs('out/CGAN_MNIST/')
        plt.savefig(save_path)
        plt.close(fig)
        return samples

    def train(self, batch_size=64, itrs=100000, save_time=1000):
        """
        训练模型方法
        :param batch_size:批量采样大小
        :param itrs: 迭代次数
        :param save_time: 保存模型周期
        :return: NONE
        """
        # 读取数据
        data = dh.load_mnist_resize()
        start_time = time.time()
        test_y = data['label'][0:25]
        for i in range(itrs):
            #随机采样数据
            mask = np.random.choice(data['data'].shape[0],batch_size,replace=True)
            batch_x = data['data'][mask]
            batch_y = data['label'][mask]
            batch_noise = np.random.normal(0,1,(batch_size,1,1,self.noise_dim))
            #训练判别器
            _,D_loss_curr =self.sess.run([self.D_trainer,self.D_loss],feed_dict={
                self.x:batch_x,self.gen_x:batch_noise,self.label:batch_y,self.isTrain:True
            })
            #训练生成器
            batch_noise = np.random.normal(0,1,[batch_size,1,1,self.noise_dim])
            _,G_loss_curr = self.sess.run([self.G_trainer,self.G_loss],feed_dict={
                self.gen_x:batch_noise,self.label:batch_y,self.isTrain:True
            })
            if i%save_time ==0:
                self.gen_data(label=test_y,save_path='out/CGAN_MNIST/'+str(i).zfill(6)+".png")
                print("i:",i," D_loss:",D_loss_curr," G_loss",G_loss_curr)
                self.save()
                end_time = time.time()
                time_loss = end_time-start_time
                print("训练时间：",int(time_loss),"秒")
                start_time =time.time()
        self.sess.close()

    def save(self, path="model/CGAN_MNIST/"):
        """
        保存模型
        :param path:保存路径
        :return: NONE
        """
        self.saver.save(self.sess,save_path=path)

    def restore(self, path='model/CGAN_MNIST/'):
        """
        恢复模型
        :param path:模型保存路径
        :return: NONE
        """
        self.saver.restore(sess=self.sess,save_path=path)

    def plot(self, smaples):
        """
        绘制5*5图片
        :param smaple: numpy数组
        :return: 绘制图像
        """
        fig = plt.figure(figsize=(5,5))
        gs = gridspec.GridSpec(5,5)
        gs.update(wspace=0.05,hspace=0.05)
        for i,smaple in enumerate(smaples):
            ax = plt.subplot(gs[i])
            plt.axis("off")
            ax.set_xticklabels([])
            ax.set_yticklabels([])
            ax.set_aspect('equal')
            plt.imshow(smaple,cmap='Greys_r')
        return fig

if __name__ == '__main__':
    gan = CGAN_MNIST()
    gan.train()
    dh.img2gif(img_path="out/CGAN_MNIST/",gif_path="out/CGAN_MNIST/")

6.2仿真结果

根据仿真结果可看出随着迭代次数的增加，训练的效果越来越好，同时也是满足在指定标签下所生成的数字图像。对比传统的GAN，在复现CGAN论文的时候，将图像标签作为输入数据放入生成器中，且输入标签跟随机噪声要一致。对于判别器，将标签跟生成数据合在一起放入判别器中。

6.3图像转图像生成应用介绍

6.4图像转图像生成应用代码

图像转图像生成应用代码

// An highlighted block
import tensorflow as tf
import numpy as np
import time
import os
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import  DataHandler as dh
class CGAN_img2img(object):
    def __init__(self, img_h=64, img_w=64, img_c=3, lr=0.0004):
        """
        初始化CGAN_img2img对象
        :param img_h: 图像高度
        :param img_w: 图像宽度
        :param img_c: 图像深度
        :param lr: 学习率
        """
        self.img_h =img_h
        self.im_w =img_w
        self.img_c = img_c
        self.lr = lr
        self.d_dim =1
        self.isTrain = tf.placeholder(dtype=tf.bool)
        self.gen_x = tf.placeholder(dtype=tf.float32,shape=[None,img_h,img_w,img_c])# 生成器的输入
        self.condition = tf.placeholder(dtype=tf.float32,shape=[None,img_h,img_w,img_c]) # 输入条件
        self.gen_out =self._init_generator(input=self.gen_x,istrain=self.isTrain) #生成器输出
        self.gen_logis = self._init_discriminator(input=self.gen_out,label=self.condition,isTrain=self.isTrain)
        self.x = tf.placeholder(dtype=tf.float32,shape=[None,img_w,img_h,img_c],name="input_data")
        self.real_logis =self._init_discriminator(input=self.x,label=self.condition,isTrain=True,reuse=True)
        self._init_train_methods()
    def _init_discriminator(self, input, label, isTrain=True, reuse=False):
        """
        初始化判别器
        :param input:输入数据op
        :param label: 输入数据标签
        :param isTrain: 是否训练状态
        :param reuse: 是否复用内部参数
        :return: 判断结果
        """
        with tf.variable_scope('discriminator',reuse=reuse):
            # input [none,64,64,3]  label [none,64,64,3]
            input = tf.concat([input,label],axis=3)
            conv1 = tf.layers.conv2d(input,32,[4,4],strides=(2,2),padding="same") #[none,32,32,32]
            bn1 =tf.layers.batch_normalization(conv1,training=isTrain)
            active1 = tf.nn.leaky_relu(bn1) #[none,32,32,32]
            #layer 2
            conv2 = tf.layers.conv2d(active1,64,[4,4],strides=(2,2),padding="same") #[none,16,16,64]
            bn2 = tf.layers.batch_normalization(conv2,training=isTrain)
            active2 = tf.nn.leaky_relu(bn2) # [none,16,16,64]
            # layer 3
            conv3 = tf.layers.conv2d(active2, 128, [4, 4], strides=(2, 2), padding="same")  # [none,8,8,128]
            bn3 = tf.layers.batch_normalization(conv3, training=isTrain)
            active3 = tf.nn.leaky_relu(bn3)  # [none,8,8,128]
            # layer 4
            conv4 = tf.layers.conv2d(active3, 256, [4, 4], strides=(2, 2), padding="same")  # [none,4,4,256]
            bn4 = tf.layers.batch_normalization(conv4, training=isTrain)
            active4 = tf.nn.leaky_relu(bn4)  # [none,4,4,256]
            # out layer
            conv5 =tf.layers.conv2d(active4,1,[4,4],strides=(1,1),padding="valid") #[none,1,1,1]
            out = tf.reshape(conv5,shape=[-1,1])
        return out
    def _init_generator(self, input, istrain=True, resue=False):
        """
        初始化生成器
        :param input:输入噪声
        :param istrain: 是否训练状态
        :param resue: 是否复用内部参数
        :return: 生成数据op
        """
        with tf.variable_scope("generator",reuse=resue):
            # input [none,64,64,3]
            conv1 = tf.layers.conv2d(input,64,[4,4],strides=(2,2),padding="same") #[none,32,32,64]
            bn1 = tf.layers.batch_normalization(conv1,training=istrain)
            active1 = tf.nn.leaky_relu(bn1) #[none,32,32,64]
            # conv2
            conv2 = tf.layers.conv2d(active1, 128, [4, 4], strides=(2, 2), padding="same")  # [none,16,16,128]
            bn2 = tf.layers.batch_normalization(conv2, training=istrain)
            active2 = tf.nn.leaky_relu(bn2)  # [none,16,16,128]
            # conv3
            conv3 = tf.layers.conv2d(active2, 256, [4, 4], strides=(2, 2), padding="same")  # [none,8,8,256]
            bn3 = tf.layers.batch_normalization(conv3, training=istrain)
            active3 = tf.nn.leaky_relu(bn3)  # [none,8,8,256]
            # conv4
            conv4 = tf.layers.conv2d(active3, 512, [4, 4], strides=(2, 2), padding="same")  # [none,4,4,512]
            bn4 = tf.layers.batch_normalization(conv4, training=istrain)
            active4 = tf.nn.leaky_relu(bn4)  # [none,4,4,512]
            # deconv1
            de1 = tf.layers.conv2d_transpose(active4,256,[4,4],strides=(2,2),padding="same") #[none,8,8,256]
            de_bn1 = tf.layers.batch_normalization(de1,training=istrain)
            de_active1 = tf.nn.leaky_relu(de_bn1) #[none,8,8,256]
            # deconv2
            de2 = tf.layers.conv2d_transpose(de_active1, 128, [4, 4], strides=(2, 2), padding="same")  # [none,16,16,128]
            de_bn2 = tf.layers.batch_normalization(de2, training=istrain)
            de_active2 = tf.nn.leaky_relu(de_bn2)  # [none,16,16,128]
            # deconv3
            de3 = tf.layers.conv2d_transpose(de_active2, 64, [4, 4], strides=(2, 2), padding="same")  # [none,32,32,64]
            de_bn3 = tf.layers.batch_normalization(de3, training=istrain)
            de_active3 = tf.nn.leaky_relu(de_bn3)  # [none,32,32,64]
            # deconv4
            de4 = tf.layers.conv2d_transpose(de_active3, 3, [4, 4], strides=(2, 2), padding="same")  # [none,64,64,3]
            de_bn4 = tf.layers.batch_normalization(de4, training=istrain)
            out = tf.nn.tanh(de_bn4)  # [none,64,64,3]
        return out
    def _init_train_methods(self):
        """
        初始化训练方法：梯度下降方法，Session初始化,损失函数。
        :return: NONE
        """
        # 寻找判别器与生成器训练参数
        total_vars = tf.trainable_variables()
        d_vars = [var for var in total_vars if var.name.startswith('discriminator')]
        g_vars = [var for var in total_vars if var.name.startswith('generator')]
        # 构造损失函数
        self.D_loss_real = tf.reduce_mean(
            tf.nn.sigmoid_cross_entropy_with_logits(logits=self.real_logis,labels=tf.ones_like(self.real_logis))
        )
        self.D_loss_fake = tf.reduce_mean(
            tf.nn.sigmoid_cross_entropy_with_logits(logits=self.gen_logis,labels=tf.zeros_like(self.gen_logis))
        )
        self.D_loss = self.D_loss_fake+self.D_loss_real

        #构造生成器损失
        gen_loss_l1 = tf.reduce_mean(tf.abs(self.gen_out-self.x))
        gen_loss_d = tf.reduce_mean(
            tf.nn.sigmoid_cross_entropy_with_logits(logits=self.gen_logis,labels=tf.ones_like(self.gen_logis))
        )
        beta = 0.7
        self.G_loss = beta*gen_loss_l1 +(1-beta)*gen_loss_d
        #构造训练方法
        self.D_trainer = tf.train.AdamOptimizer(learning_rate=self.lr).minimize(
            self.D_loss,var_list=d_vars
        )
        self.G_trainer = tf.train.AdamOptimizer(learning_rate=self.lr).minimize(
            self.G_loss,var_list=g_vars
        )
        # 初始化session
        self.sess = tf.InteractiveSession()
        self.sess.run(tf.global_variables_initializer())
        self.saver = tf.train.Saver(max_to_keep=1)
    def gen_data(self,data, label, save_path="out/CGAN_img2img/test.png"):
        """
        生成36张6*6的图片，第一列生成图片，第二列输入数据，第三列真实目标值.....
        :param label:真实图片
        :param data:输入数据
        :param save_path: 保存路劲
        :return: 图像数组 numpy
        """
        samples = self.sess.run(self.gen_out,feed_dict={
            self.gen_x:data,self.isTrain:True
        })
        # samples [none,64,64,3]  float   (-1,1)
        samples = ((samples+1)/2*255).astype(np.uint8)
        label =((label+1)/2*255).astype(np.uint8)
        data = ((data+1)/2*255).astype(np.uint8)
        fig = self.plot(samples,data,label)
        if not os.path.exists("out/CGAN_img2img/"):
            os.makedirs("out/CGAN_img2img/")
        plt.savefig(save_path)
        plt.close(fig)
        return samples
    def train(self, batch_size=64, itrs=100000, save_time=1000):
        """
        训练模型方法
        :param batch_size:批量采样大小
        :param itrs: 迭代次数
        :param save_time: 保存模型周期
        :return: NONE
        """
        start_time = time.time()
        train_x,train_y = dh.read_all_facade2numpy()
        for i in range(itrs):
            mask =np.random.choice(train_x.shape[0],batch_size,replace=True)
            batch_x = train_x[mask]
            batch_y = train_y[mask]
            _,D_loss_curr = self.sess.run([self.D_trainer,self.D_loss],feed_dict={
                self.x:batch_y,self.gen_x:batch_x,self.condition:batch_x,self.isTrain:True
            })
            #训练生成器
            mask = np.random.choice(train_x.shape[0], batch_size, replace=True)
            batch_x = train_x[mask]
            batch_y = train_y[mask]
            _,G_loss_curr = self.sess.run([self.G_trainer,self.G_loss],feed_dict={
               self.x:batch_y,self.gen_x:batch_x,self.condition:batch_x,self.isTrain:True
            })
            if i%save_time ==0:
                mask = np.random.choice(train_x.shape[0],12,replace=True)
                test_x = train_x[mask]
                test_y =train_y[mask]
                self.gen_data(data=test_x,label=test_y,save_path="out/CGAN_img2img/"+str(i).zfill(6)+".png")
                print("i:",i," D_loss:",D_loss_curr," G_loss",G_loss_curr)
                self.save()
                end_time = time.time()
                time_loss = end_time-start_time
                print("时间消耗:",int(time_loss),"秒")
                start_time = time.time()
        self.sess.close()
    def save(self, path="model/CGAN_img2img/"):
        """
        保存模型
        :param path:保存路径
        :return: NONE
        """
        self.saver.save(self.sess,save_path=path)
    def restore(self, path='model/CGAN_img2img/'):
        """
        恢复模型
        :param path:模型保存路径
        :return: NONE
        """
        self.saver.restore(sess=self.sess,save_path=path)
    def plot(self, smaples,data,label):
        """
        绘制6*6图片矩阵
        :param smaples:生成图片
        :param data: 输入图片
        :param label: 真实图片
        :return: 绘制图像
        """
        fig = plt.figure(figsize=(6,6))
        gs = gridspec.GridSpec(6,6)
        gs.update(wspace=0.05,hspace=0.05)
        for i in range(12):
            # 绘制生成图片
            ax = plt.subplot(gs[3*i])
            plt.axis("off")
            ax.set_xticklabels([])
            ax.set_yticklabels([])
            ax.set_aspect("equal")
            plt.imshow(smaples[i])
            #绘制输入图片
            ax = plt.subplot(gs[3 * i+1])
            plt.axis("off")
            ax.set_xticklabels([])
            ax.set_yticklabels([])
            ax.set_aspect("equal")
            plt.imshow(data[i])
            #绘制真实图片
            ax = plt.subplot(gs[3 * i+2])
            plt.axis("off")
            ax.set_xticklabels([])
            ax.set_yticklabels([])
            ax.set_aspect("equal")
            plt.imshow(label[i])
        return  fig
if __name__ == '__main__':
    gan = CGAN_img2img()
    gan.train();

6.5仿真结果

迭代0次时的结果：

迭代5000次

迭代50000次：

迭代100000次：

对于上述仿真结果，第一列和第四列为生成图片，第二列和第五列为输入图像，第三列和第六列为真实图片。由此可见CGAN在输入图像标签和随机噪声（可不加）的作用下，生成图片的效果还是非常理想，若想要生成高精度图像可使用StackGAN，PatchGAN等。

七.总结

CGAN的重难点在于理解CGAN的框架，以及编码过程，对于CGAN，可以根据输入标签生成自己所想生成的图像，对于某些特定的工程及应用具有很强的方法性。

八.参考文献

1.I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014. 2, 3,4, 7
2.K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.
arXiv preprint arXiv:1409.1556, 2014. 7