My spark learning

 Apache Spark Cluster Architecture from https://spark.apache.org/docs/latest/cluster-overview.html                               Spark follows the Master slave Architecture. There three type of nodes acting in spark cluster. 1. Resource Manager or cluster manager 2. Diver or master node 2. Worker node or slave node Cluster Manager - Manages the resource allocation and coordinating with master node to execute the jobs or tasks  in worker nodes. Managing the cluster and if one worker failed, it will allocate that work to other available node(new node) or existing node. it supports below cluster managers Standalone - default spark cluster manager YARN - popular hadoop cluster manager Mesos - a general open source cluster manager Kubernetes -  mostly used for container orchestration, container is similar to our worker node Diver/Master Node     A node where main function o...

 Introduction: Spark - open source big data processing framework used for processing transforming and analyzing Big Data. Core written using Scala language. Programming language spark supports:  - Java default support,  - Scala default support,  - Python through Pyspark, - R through SparkR Spark Frame Work Architecture:- Spark Framework Components:- 1. Spark Core (core functionalities, APIs to support programming languages ) 2. Spark SQL - enables to leverage the power SQL declarative programming language to concentrating on what to do (Business Logic) , not How to Do also like imperative languages. 3. Spark Streaming - a real time or nearly real time processing, it will process even if it one record, not wait bulk processing 4. Spark MLlib - for Machine Learning purpose 5. GraphX  - helps to process graph data Spark Cluster Type - Master Slave architecture (can say Diver - worker) Features:- - provides Distributed computing(parallel processing) - in - memory pr...