Introduction to Apache Spark

 Introduction:

Spark - open source big data processing framework used for processing transforming and analyzing Big Data. Core written using Scala language.

Programming language spark supports:

 - Java default support, 

- Scala default support, 

- Python through Pyspark,

- R through SparkR

Spark Frame Work Architecture:-

Spark Framework Components:-

1. Spark Core (core functionalities, APIs to support programming languages )

2. Spark SQL - enables to leverage the power SQL declarative programming language to concentrating on what to do (Business Logic) , not How to Do also like imperative languages.

3. Spark Streaming - a real time or nearly real time processing, it will process even if it one record, not wait bulk processing

4. Spark MLlib - for Machine Learning purpose

5. GraphX  - helps to process graph data

Spark Cluster Type - Master Slave architecture (can say Diver - worker)

Features:-

- provides Distributed computing(parallel processing)

- in - memory processing

- supports batch, streaming and real time processing

- supports SQL to write transformations and actions

- Lazy evaluation

- Fault tolerant

- providing RDD/ Datasets/Date Frames(optimized) simulates structured Tables

- supports different types of File Systems and Cluster Managers

Advantages :-

- Both Open Source and Enterprise solutions available.

- High performance

- handle large volume of data

- customizable configurations such as memory, partitions, nodes etc..

- built-in optimization for SQL

- flexible to run on different file systems such as HDFS, DBFS, S3, Azure Blob etc...

- easily integrate with wide range of Sources and sink

- support graph data processing

- strong community support

- highly scalable

Disadvantages :-

- in-memory makes it as relatively costlier due to high RAM capacity needed even though it provides reliable performance.

- no built - in file systems , reliance on External file systems

- Streaming is not effecient compared to other alternative streaming frameworks such as fink.

- Limited Machine Learning Algorithms making less choices for MLOps and Data Science.

- Manual Performance tuning makes need for expertise in it. less advantage for beginners  

- not optimized for small scale applications.

 

continuing Apache Cluster Architecture in next blog.