Apache
spark can be used as a solution for ingesting data and transforming
data. Spark fits well with the other components of Big Data
ecosystem. Spark lets you quickly write applications in Java, Scala,
or Python. This helps developers to create and run their applications
on their familiar programming languages. Spark runs on top of the
Hadoop Distributed File System (HDFS). Spark does in memory
computations as much as it can which results in increased speed in
data processing over Hadoop MapReduce.
Scala
Scala is a
pure-bred object-oriented language. Conceptually, every value is an
object and every operation is a method-call. Scala is also a
full-blown functional language. It has a general preference of
immutability over mutation. Scala allows a gradual, easy migration to
a more functional style. You can start to use it as a “Java without
semicolons”. Over time, you can progress to gradually eliminate
mutable state in your applications, phasing in safe functional
composition patterns instead. Scala runs on the JVM. Java and Scala
classes can be freely mixed, no matter whether they reside in
different projects or in the same. They can even mutually refer to
each other. The Scala compiler contains a subset of a Java compiler
to make sense of such recursive dependencies.
Spark
Apache
Spark is a fast and general engine for large-scale data processing.
Spark runs programs up to 100x faster than Hadoop MapReduce in
memory, or 10x faster on disk. Though Spark is written in Scala,
Spark applications can be quickly written in Java, Scala and Python.
Spark combines SQL, streaming, and complex analytics. Spark powers a
stack of high-level tools including Spark SQL, MLlib for machine
learning, GraphX, and Spark Streaming. You can combine these
libraries seamlessly in the same application. Spark runs on Hadoop,
Mesos, standalone, or in the cloud. It can access diverse data
sources including HDFS, Cassandra, HBase, and S3. Since its release,
Spark has been rapidly adopted by enterprises across a wide range of
industries. Many internet powerhouses have deployed Spark at massive
scale, collectively processing multiple petabytes of data on clusters
of over 8,000 nodes. It has quickly become the largest open source
community in big data.
Spark
Streaming
Spark
Streaming is an extension of the core Spark API that enables
scalable, high-throughput, fault-tolerant stream processing of live
data streams. Data can be ingested from many sources like Kafka,
Flume, Twitter, ZeroMQ, Kinesis or TCP sockets and can be processed
using complex algorithms expressed with high-level functions like
map, reduce, join and window. Finally, processed data can be pushed
out to file systems, databases, and live dashboards. In fact, we can
apply Spark’s machine learning and graph processing algorithms on
data streams.