Tag: BigData

Getting started with Apache Flume

https://flume.apache.org/FlumeUserGuide.html $ flume-ng agent –conf-file flume-config/flume.conf –name agent1 Dflume.root.logger=INFO,console # # Define the different components for an agent # agent1.sources = source1 agent1.sinks = sink1 agent1.channels = channel1 # # Define the settings for source # agent1.sources.source1.type = netcat agent1.sources.source1.bind = localhost agent1.sources.source1.port = 44444 # # Define the settings for sink # agent1.sinks.sink1.type = […]

Getting started with Apache Hadoop

Installation I am going to install Hadoop on macOS Mojave (10.14.2) using brew. The installation process is quite straight forward. To install Hadoop, we will execute this following command in terminal. $ brew install hadoop This installs Hadoop version 3.1.1. Configuration We are going to edit 4 configuration files to successfully run Hadoop: hadoop-env.sh, core-site.xml, […]

Getting started with Apache Spark

Installing Apache Spark on Mac OSX $ brew install apache-spark Run this command to test your installation: $ pyspark For me, the output was this: Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 6 2017, 12:04:38) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin Type “help”, “copyright”, “credits” or “license” for more information. 2018-12-07 21:22:12 WARN […]

Back To Top