Home / Questions / 278502
Accepted
Ares777
Asked: 2020-07-13 01:19:54 +0800 CST 2020-07-13 01:19:54 +0800 CST

Save the output of a Foreach to an array

  • 772

I have a file of X number of lines with this format:

1::F::1::10::48067

What I am looking for is that it reads as many lines as I pass to the following function:

case class User(userID: String, Gender: String, Occupation: String, Zipcode:
 String)

def getNthLineUser(lines: Array[String]) =  {

  val rddUsersSplit = lines.foreach(e => println(e.split("::")(0), e.split("::")(1), e.split("::")(2), e.split("::")(3)))

}

getNthLineUser(RDDusers.take(2))

What I want to do with this function is to read each of the lines and split "1::F::1::10::48067" by "::" and store this in a sequence of [User ] to later convert it to DF with the .toDF.

I don't see how to convert the output which is [Unit] to Sequence or List. What I have managed to do is simply read each line and separate it by "::" and print it to the screen.

How can I do it?

scala

1 Answers

  1. Best Answer

    The use of foreachonly looks for some collateral effect, not the transformation of a data flow. In Functional Programming it should always be avoided. Use any other functional method, such as map:

    val rddUsersSplit = lines.map{ line =>
   val Array(userID, gender, occupation, zipcode, _) = line.split("::")
   User(userID, gender, occupation, zipcode)
}.toDF
    • 1