【中英】【吴恩达课后测验】Course 2 - 改善深层神经网络 - 第三周测验

第3周测验 - 超参数调整，批量标准化，编程框架

1. 如果在大量的超参数中搜索最佳的参数值，那么应该尝试在网格中搜索而不是使用随机值，以便更系统的搜索，而不是依靠运气，请问这句话是正确的吗？

   • 【★】 错误
   • 【 】 正确

   请注意：应当尝试随机值，不要使用网格搜索，因为你不知道哪些超参数比其他的更重要。

   And to take an extreme example, let’s say that hyperparameter two was that value epsilon that you have in the denominator of the Adam algorithm. So your choice of alpha matters a lot and your choice of epsilon hardly matters.
   
   
   举一个很极端的例子，就比如在Adam算法中防止除零操作的ε的值，一般为1的负8次方，但是和学习率α相比，ε就显得不那么重要了。

2. 每个超参数如果设置得不好，都会对训练产生巨大的负面影响，因此所有的超参数都要调整好，请问这是正确的吗？

   • 【★】 错误
   • 【 】 正确

   We’ve seen in lecture that some hyperparameters, such as the learning rate, are more critical than others.
   
   
   我们在视频中讲到的比如学习率这个超参数比其他的超参数更加重要。

3. 在超参数搜索过程中，你尝试只照顾一个模型（使用熊猫策略）还是一起训练大量的模型（鱼子酱策略）在很大程度上取决于：

   • 【 】 是否使用批量（batch）或小批量优化（mini-batch optimization）
   • 【 】 神经网络中局部最小值（鞍点）的存在性
   • 【★】 在你能力范围内，你能够拥有多大的计算能力（博主注：就是高性能电脑和低性能电脑的区别）
   • 【 】 需要调整的超参数的数量

4. 如果您认为$\beta$（动量超参数）介于0.9和0.99之间，那么推荐采用以下哪一种方法来对$\beta$值进行取样？

   r = np.random.rand()
   beta = 1 - 10 ** ( - r - 1 )

   （博主注：$\beta = 1 - 10^{-r-1}$，因为$r$的取值只能在0到1之间，所以当$r$等于0时，$\beta=1 - 10^{-1} = 1 - 0.1 = 0.9$，当$r$等于1时，$\beta=1 - 10^{-2} = 1 - 0.01 = 0.99$。）

5. 找到好的超参数的值是非常耗时的，所以通常情况下你应该在项目开始时做一次，并尝试找到非常好的超参数，这样你就不必再次重新调整它们。请问这正确吗？

   • 【★】 错误
   • 【 】 正确

   请注意：模型中的细微变化可能导致您需要从头开始重新找到好的超参数。

6. 在视频中介绍的批量标准化中，如果将其应用于神经网络的第$l$层，那么您怎样进行标准化？

   • 【★】 $z^{[l]}$

   • 在标准化公式中，为什么要使用epsilon（$\epsilon$）？

     • 【★】 为了避免除零操作

   • 批处理规范中关于 $γ$ 和 $β$ 的以下哪些陈述是正确的？（博主注：只列出了正确选项）

     • 【★】它们可以在Adam、具有动量的梯度下降或RMSprop使中用，而不仅仅是用梯度下降来学习。
     • 【★】它们设定给定层的线性变量 $z^{[l]}$ 的均值和方差。

   • 在训练具有批处理规范的神经网络之后，在测试时间，在新样本上评估神经网络，您应该：

     • 【★】执行所需的标准化，在训练期间使用使用了$μ$和$σ^2$的指数加权平均值来估计mini-batches的情况。

   • 关于深度学习编程框架的这些陈述中，哪一个是正确的？

     • 【★】 通过编程框架，您可以使用比低级语言（如Python）更少的代码来编写深度学习算法。
     • 【★】 即使一个项目目前是开源的，项目的良好管理有助于确保它即使在长期内仍然保持开放，而不是仅仅为了一个公司而关闭或修改。
     • 【 】 深度学习编程框架的运行需要基于云的机器。

Week 3 Quiz - Hyperparameter tuning, Batch Normalization, Programming Frameworks

1. If searching among a large number of hyperparameters, you should try values in a grid rather than random values, so that you can carry out the search more systematically and not rely on chance. True or False?

   • [x] False
   • [ ] True

   Note: Try random values, don’t do grid search. Because you don’t know which hyperparamerters are more important than others.

   And to take an extreme example, let’s say that hyperparameter two was that value epsilon that you have in the denominator of the Adam algorithm. So your choice of alpha matters a lot and your choice of epsilon hardly matters.

2. Every hyperparameter, if set poorly, can have a huge negative impact on training, and so all hyperparameters are about equally important to tune well. True or False?

   • [x] False
   • [ ] True

   We’ve seen in lecture that some hyperparameters, such as the learning rate, are more critical than others.

3. During hyperparameter search, whether you try to babysit one model (“Panda” strategy) or train a lot of models in parallel (“Caviar”) is largely determined by:

   • [ ] Whether you use batch or mini-batch optimization
   • [ ] The presence of local minima (and saddle points) in your neural network
   • [x] The amount of computational power you can access
   • [ ] The number of hyperparameters you have to tune

4. If you think β (hyperparameter for momentum) is between on 0.9 and 0.99, which of the following is the recommended way to sample a value for beta?

   r = np.random.rand()
   beta = 1 - 10 ** (-r - 1)

5. Finding good hyperparameter values is very time-consuming. So typically you should do it once at the start of the project, and try to find very good hyperparameters so that you don’t ever have to revisit tuning them again. True or false?

   • [x] False
   • [ ] True

   Note: Minor changes in your model could potentially need you to find good hyperparameters again from scratch.

6. In batch normalization as presented in the videos, if you apply it on the lth layer of your neural network, what are you normalizing?

   • z^[l]

7. In the normalization formula, why do we use epsilon?

   • To avoid division by zero

8. Which of the following statements about γ and β in Batch Norm are true? Only correct options listed

   • They can be learned using Adam, Gradient descent with momentum, or RMSprop, not just with gradient descent.
   • They set the mean and variance of the linear variable z^[l] of a given layer.

9. After training a neural network with Batch Norm, at test time, to evaluate the neural network on a new example you should:

   • Perform the needed normalizations, use μ and σ^2 estimated using an exponentially weighted average across mini-batches seen during training.

10. Which of these statements about deep learning programming frameworks are true? (Check all that apply)

    • [x] A programming framework allows you to code up deep learning algorithms with typically fewer lines of code than a lower-level language such as Python.
    • [x] Even if a project is currently open source, good governance of the project helps ensure that the it remains open even in the long term, rather than become closed or modified to benefit only one company.
    • [ ] Deep learning programming frameworks require cloud-based machines to run.