Friday, 28 November 2014

How To Enter The World Of Scala Spark Streaming - Beginners Introduction

How To Enter The World Of Scala, Spark and Streaming - Beginners Introduction




Apache spark can be used as a solution for ingesting data and transforming data. Spark fits well with the other components of Big Data ecosystem. Spark lets you quickly write applications in Java, Scala, or Python. This helps developers to create and run their applications on their familiar programming languages. Spark runs on top of the Hadoop Distributed File System (HDFS). Spark does in memory computations as much as it can which results in increased speed in data processing over Hadoop MapReduce. 

 Scala

Scala is a pure-bred object-oriented language. Conceptually, every value is an object and every operation is a method-call. Scala is also a full-blown functional language. It has a general preference of immutability over mutation. Scala allows a gradual, easy migration to a more functional style. You can start to use it as a “Java without semicolons”. Over time, you can progress to gradually eliminate mutable state in your applications, phasing in safe functional composition patterns instead. Scala runs on the JVM. Java and Scala classes can be freely mixed, no matter whether they reside in different projects or in the same. They can even mutually refer to each other. The Scala compiler contains a subset of a Java compiler to make sense of such recursive dependencies.

Spark

Apache Spark is a fast and general engine for large-scale data processing. Spark runs programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. Though Spark is written in Scala, Spark applications can be quickly written in Java, Scala and Python. Spark combines SQL, streaming, and complex analytics. Spark powers a stack of high-level tools including Spark SQL, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application. Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3. Since its release, Spark has been rapidly adopted by enterprises across a wide range of industries. Many internet powerhouses have deployed Spark at massive scale, collectively processing multiple petabytes of data on clusters of over 8,000 nodes. It has quickly become the largest open source community in big data.

Spark Streaming

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ, Kinesis or TCP sockets and can be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to file systems, databases, and live dashboards. In fact, we can apply Spark’s machine learning and graph processing algorithms on data streams.