Apache Spark Cluster Architecture

from https://spark.apache.org/docs/latest/cluster-overview.html

            

            Spark follows the Master slave Architecture. There three type of nodes acting in spark cluster.

1. Resource Manager or cluster manager

2. Diver or master node

2. Worker node or slave node

Cluster Manager -

Manages the resource allocation and coordinating with master node to execute the jobs or tasks  in worker nodes. Managing the cluster and if one worker failed, it will allocate that work to other available node(new node) or existing node.

it supports below cluster managers

Standalone - default spark cluster manager

YARN - popular hadoop cluster manager

Mesos - a general open source cluster manager

Kubernetes -  mostly used for container orchestration, container is similar to our worker node

Diver/Master Node

    A node where main function of your application will run. it will create the spark context which is responsible for establishing and communicating to worker node through cluster manager. It is responsible to run the application and complete the jobs with of cluster manager and worker nodes.

Worker Node

    It will have the running executor in it. Executor is responsible to complete the tasks allocated by driver program. Same worker node can have multiple executor if it having that capability such as RAM and CPU.

Cache - A memory to capture and persists the variable value or datasets to reuse in tasks.it will persist till that application complete

Task - a piece of computational work such as rename, join etc.....

SparkContext (till spark 1.x.x)

• Spark Context is created in Main Driver program,
• Spark Context  is entry point of your application. it 
• makes connection with cluster, 
• managing resources with help of resource manager, 
• will create and manage RDD, accumulators, and broad casting variables,.
• coordinate with worker nodes for allocation and completion of tasks and jobs 
• managing configurations such as execution mode, memory, cluster manager etc...
• Error handling - if task or node fails, it will allocate that task to another to complete it.
• release the resources once job completed.
• one spark context allowed to create per JVM

Spark Session (from spark 2.x.x)

• it is the entry point of application from spark 2.x.x
• it responsible to create RDD, Datasets and Data Frames (more optimized for structured data processing).
• it abstraction of spark context, SQL context, Streaming Context, Hive Context. means, it will create  spark context, SQL context, Streaming Context, Hive Context internally if not exist and provide simplified interface to manage those context and configurations
• It can be created as many sessions we want. but, contexts are only one per JVM.
• Improves query execution performance through catalyst Optimizer.