ScalaSparkLambda “goesto” => Analysis

 1 /// Define a function AddNoise with parameters rdd and Fraction respectively.  Where rdd is an RDD composed of (BreezeDenseMatrix, BreezeDenseMatrix) tuples.  Fraction is a Double.  Returns an RDD of (BreezeDenseMatrix, BreezeDenseMatrix) tuples.  
  2 def AddNoise(rdd: RDD[(BDM[Double], BDM[Double])], Fraction: Double): RDD[(BDM[Double], BDM[Double])] = {
  3 /// Define the temporary storage of the return value, which is composed of f=>STH implemented by each element in rdd
  4 val addNoise = rdd.map { f =>
  5 /// The second part of the data in f is a BreezeDenseMatrix
  6 val features = f._2
  7 /// Generate a BreezeDenseMatrix filled with random numbers
  8 val a = BDM.rand[Double](features.rows, features.cols)
  9 /// Define a BreezeDenseMatrix a1, where the elements are bool type.  It is true if the corresponding element in a is greater than Fraction.  Otherwise false.  
 10 val a1 = a :>= Fraction
 11 /// Define an RDD d1, which is filled by the elements in a1 after the following operation: If the current element is true, it is 1.0,  Otherwise it is 0.  
 12 val d1 = a1.data.map { f => if (f == true) 1.0 else  0.0}
 13 /// Create a new BreezeDenseMatrix and fill it with the d1 elements corresponding to the corresponding positions of features.  
 14 val a2 = new BDM(features.rows, features.cols, d1)
 15 /// :* means that each element is multiplied in sequence.  Get BreezeDenseMatrix.  
 16 val features2 = features :* a2
 17 /// Returns the RDD composed of (BreezeDenseMatrix, BreezeDenseMatrix).  As the function return value, addNoise is updated.  
 18  (f._1, features2)
 19  }
 20 /// Returns the result after the operation as the function return value.  
 21  addNoise
 22 }

This code is written in Scala and runs on Spark, implementing AddNoise in the NN algorithm. Used to complete the random noise addition of DenoiseAutoencoder.

The idea is very simple, but it vividly demonstrates the usage of Spark’s map operation and Scala’s lambda operator, which is worth learning.

Code comes from sunbow0.

Personal analysis, please correct me if I am wrong.

Analysis,mamicode.com” target=”_blank”>Scala – Spark Lambda “goesto” => Analysis