APACHE-SPARK INSTALLATION ON UBUNTU

 SPARK INSTALLATION ON UBUNTU

strep 1:

download java using the below link

https://www.oracle.com/in/java/technologies/javase/javase8-archive-downloads.html


and extract the tar file using the below command 

tar -zxvf jdk-8u202-linux-x64.tar.gz (Extract the tar file)

step 2:

spark download click the below link here is the direct download link 

         https://dlcdn.apache.org/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.tgz  in the link you can see the latest spark version 3.4.0 you can click and download.

(or)if you want to download other versions clickbelow the link you can latest and old versions

https://spark.apache.org/downloads.html

extract the tar file using the below command:

tar -zxvf <your spark path...>   

exmaple...

tar -zxvf /home/user/spark-3.4.0-bin-hadoop3.tgz

step 3:

open the bashrc file using the command "gedit .bashrc" and set your spark and java path 

export JAVA_HOME=/home/user/jdk1.8.0_202

export SPARK_HOME=/home/user/spark-3.4.0-bin-hadoop3

export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH



step 3:

in your spark conf  folder you can see the spark-env.sh-template file and take copy and paste it there only.and rename that copy as a spark-env.sh and also set $JAVA_HOME path

you can see here i copy and rename the that fie


open that rename file and set java path like below




step 4:

now you can able to start the spark using

"sbin/start-all.sh"

and using the "jps" commands you can see the spark demons are running..

there are two demons 

spark master

spark worker


 

Comments

Post a Comment

Popular posts from this blog

Data Migration Using Apache Sqoop

 Data Migration Using Apache Sqoop  Sqoop: sql + hadoop = sqoop actually sqoop is one of the component in hadoop and it was initially developed and maintained by Cloudera. sqoop is a data pipeline tool used to transfer the data between RDBMS(Relational Database Management System) to Hadoop. It can be a any rdbms data. There are three ways to import from rdbms to hadoop 1) RDBMS to HDFS 2) RDBMS to Hive 3) RDBMS to Hbase we can do export to rdbms from hive and hdfs  only  1)HDFS to RDBMS  2)Hive to RDBMS Here we use the Mysql(Rdbms) commands for import    1) MySql to HDFS bin/sqoop import --connect jdbc:mysql://localhost/db -username root -password 123 --table Persons -m1 2)Mysql to Hive bin/sqoop import --connect jdbc:mysql://localhost/db -username root -password 123 --table test --hive-table mysqltohive --create-hive-table --hive-import -m1 3)Mysql to Hbase bin/sqoop import --connect jdbc:mysql://localhost/test --username root --password root --table...
Read more

APACHE -CASSANDRA INSTALLATION ON UBUNTU

Image
  Apache -Cassandra Cassandra is a NoSQL database which is distributed and scalable. It is provided by Apache Software Foundation. Apache Cassandra is highly scalable, high performance, distributed NoSQL database. Cassandra is designed to handle huge amount of data across many commodity servers, providing high availability without a single point of failure. Cassandra has a distributed architecture which is capable to handle a huge amount of data. Data is placed on different machines with more than one replication factor to attain a high availability without a single point of failure. Step 1: Before cassandra you have to need install java1.8>=  https://www.oracle.com/in/java/technologies/javase/javase8-archive-downloads.html step 2: python is needed for  install cassandra  Install Python 2.7 sudo apt-add-repository universe sudo apt update sudo apt install python2-minimal python2 -V Step 3: Download cassandra 3.11.15  https://cassandra.apache.org/_/download.ht...
Read more