Spark Jobs

 Introduction Spark JOB:-

In spark, we have two different types of computation or operations.

1. transformations

2. action

Transformations are lazily evaluated means it will be executed until and  any actions or side effects are called.

Till side effects are actions called, spark will create logical plan for those transformations as DAG. if no action called  then no transformation happen or executed.

So, this each actions will trigger the jobs or create jobs and computed. so, mostly no of jobs is equal to no of actions unless some optimization techniques applied like caching Data Frames are in intermediate results, Whole Stage Code Generation techniques, query optimization , adaptive query execution, shuffled count.

Spark jobs are set of tasks which will done by distributed computation. 

Spark job is logically divided into stages.

Stages:-

Stages are logical unit of  job. It is consist of set of tasks can run parallelly on partitions of data or executors of worker node.

One stage output or result will be input of subsequent stages.

Stages types

1. Narrow stage - create on transformations does not needs data shuffling. it can pipelined

2. Wider stage - it is created when that transformation involves data shuffling. Data shuffling is costlier operations. so, one or many stages based on complexity (example transformations join, groupbykey, countbykey, reducebykey etc......) 

A job may have no of stages and stages may have no of tasks based on data volume and partitions.

Tasks 

Task is a single unit of execution. it will be executed by executor process running in its own JVM of the worker node on the partition of the data. 

Example, data divided into 10 partitions and if you want perform count on the data, 1 job, 1 stage and 10 tasks created. Each action or transformation on single partition of data is one task.

Executors

A worker process which have its own JVM running on a worker node, intended to perform the tasks scheduled by the driver node.  it is responsible to complete the task and result will be returned to driver program.

Each executor will have its own allocated memory, CPU cores and other spark configurations.

Executor Configurations

Memory (--executor-memory) : by default, 1 GB per executor. we can increase or decrease by giving customized configuration value when submitting the spark job

CPU (--executor-cores) : by default, 1 core per executor, we can increase or decrease by giving customized configuration value when submitting the spark job

no of executors (--num-executors) :  no of executors per worker can be launched by spark, default set to 2. it is customizable as above on job submission.

Executor Garbage collector: Two types of GC, default set to Concurrent Mark and Sweep. Alternative is Garbage First Collector. can customize

Overhead Memory (--executor-memory-overhead) : it is allocated for system process such as JVM processes etc..... By default, 10 % of worker node memory allocated for this. customize

shuffle memory (--park.shuffle.memoryFraction) : by default, 384 MB allocated for Data shuffling operations and data exchange between executors. it can be customizable.

Types of Executors

Default Executor - a general purpose executor.

Coarse-Grained Executor - it used for processing large dataset or, tasks requires more memory. it will more memory allocated than default executor.

Fine-Grained Executor - it is used to process the smaller datasets. or tasks requires less memory. it is useful when smaller dataset which doesn't need data shuffling.

External Executor  - like GPU .. etc. if application needs external executor to process data.

Performance of executor will impact by following factors such as memory, CPU cores, Network bandwidth, data distribution (skewness), task complexity .. etc.