Posts

Spark Jobs

 Introduction Spark JOB:- In spark, we have two different types of computation or operations. 1. transformations 2. action Transformations are lazily evaluated means it will be executed until and  any actions or side effects are called. Till side effects are actions called, spark will create logical plan for those transformations as DAG. if no action called  then no transformation happen or executed. So, this each actions will trigger the jobs or create jobs and computed. so, mostly no of jobs is equal to no of actions unless some optimization techniques applied like caching Data Frames are in intermediate results, Whole Stage Code Generation techniques, query optimization , adaptive query execution, shuffled count. Spark jobs are set of tasks which will done by distributed computation.  Spark job is logically divided into stages. Stages:- Stages are logical unit of  job. It is consist of set of tasks can run parallelly on partitions of data or executors of work...
Post a Comment
Read more

Apache Spark Cluster Architecture

Image
 Apache Spark Cluster Architecture from https://spark.apache.org/docs/latest/cluster-overview.html                               Spark follows the Master slave Architecture. There three type of nodes acting in spark cluster. 1. Resource Manager or cluster manager 2. Diver or master node 2. Worker node or slave node Cluster Manager - Manages the resource allocation and coordinating with master node to execute the jobs or tasks  in worker nodes. Managing the cluster and if one worker failed, it will allocate that work to other available node(new node) or existing node. it supports below cluster managers Standalone - default spark cluster manager YARN - popular hadoop cluster manager Mesos - a general open source cluster manager Kubernetes -  mostly used for container orchestration, container is similar to our worker node Diver/Master Node     A node where main function o...
Post a Comment
Read more

Introduction to Apache Spark

Image
 Introduction: Spark - open source big data processing framework used for processing transforming and analyzing Big Data. Core written using Scala language. Programming language spark supports:  - Java default support,  - Scala default support,  - Python through Pyspark, - R through SparkR Spark Frame Work Architecture:- Spark Framework Components:- 1. Spark Core (core functionalities, APIs to support programming languages ) 2. Spark SQL - enables to leverage the power SQL declarative programming language to concentrating on what to do (Business Logic) , not How to Do also like imperative languages. 3. Spark Streaming - a real time or nearly real time processing, it will process even if it one record, not wait bulk processing 4. Spark MLlib - for Machine Learning purpose 5. GraphX  - helps to process graph data Spark Cluster Type - Master Slave architecture (can say Diver - worker) Features:- - provides Distributed computing(parallel processing) - in - memory pr...
Post a Comment
Read more