Spark Interview Questions and Answers

In Spark, if any partition of an RDD is lost due to the failure of a worker node, that partition can be re-computed using the lineage of operations from the original fault-tolerant dataset.

Here are the uses of GraphX in Spark:

• It can be used for unifying ETL, exploratory analysis, and computation of iterative graphs within a single system.
• It can be used to present data in the form of graphs and collections while transforming and joining charts with RDD.
• It can be used for writing custom iterative graph algorithms with the help of Pregel API.

In Spark, Coalesce is just another method for partitioning the data into a data frame. This is primarily used for reducing the number of partitions inside a data frame. It is most commonly used in cases where the user wants to decrease the amount of partitions without any confusion of shuffle.

Here are some of the advantages of using Spark rather than Hadoop’s MapReduce:

• Spark is relatively easier to program and requires a lot less of actual coding than MapReduce
• Spark has an in-built interactive mode, whereas MapReduce is, by default, has only batch processing and does not have an in-built interactive mode.
• Spark uses a data abstraction, RDD, to make the features more productive, whereas MapReduce does not have any concept
• Spark executes batch processing jobs almost 10X to 100X times faster than MapReduce.
• Spark is considered as a general-purpose cluster computing engine due to its various methods for data processing such as steaming, batch processing, and machine learning, whereas MapReduce only has a Batch Engine.
• Spark consumes lower latency via partial or complete caching of results across various nodes whereas, MapReduce is disk-based and consumes a far higher latency.

Spark uses Lazy Evaluation because of the following reasons:

• It increases the manageability of the program by dividing it into smaller operations, thereby reducing the number of passes on the data by grouping operations.
• It increases the speed and saves computational and calculational overhead by computing only necessary values.
• It reduces complexities in any program by allowing users to work with an infinite data structure while drastically reducing time and space overheads.
• It optimizes the program by reducing the number of queries being run.

RDDs or Resilient Distributed Datasets are the fundamental data structure present in Spark. They are immutable and fault-tolerant in nature. There are multiple ways to create RDDs in Spark. They are:

• Creating RDD from a Seq or List using Parallelize

           RDDs can be created by taking an existing collection from a driver’s program and passing it to the Spark’s SparkContext’s parallelize () method. Here’s an example:
      val rdd=spark.sparkContext.parallelize(Seq(("Java", 10000),
      ("Python", 200000), ("Scala", 4000)))
      rdd.foreach(println)
       Output
      (Python,100000)
      (Scala,3000)
      (Java,20000)

• Creating an RDD using a text file

     Mostly, in production systems, users can generate RDDs from files by simply reading the data from the files. Let us see how:
     Val rdd = spark.sparkContext.textFile("/path/textFile.txt")
     The above line of code creates an RDD in which each record represents a line of code.

• Creating RDDs from Dataframes and DataSets

     You can easily convert any DataFrame or DataSet into an RDD. It can be done by using the rdd() method. Here’s how:
     val myRdd2 = spark.range(20).toDF().rdd
     In the above line of code, toDF() creates a DataFrame, and by calling an RDD, the range of code returns with a newly created RDD.

There are two types of RDD Operations in Spark. They are:

• Transformation: It is a type of function in which a new RDD is created from an existing RDD.
• Action: This is a type of function which is used when the user wants to work with an actual DataSet.