Apache Bigtop is a project for the development of packaging and test of the Hadoop ecosystem.

Deployment

Deploy a Bigtop virtual Hadoop cluster with vagrant and puppet recipe Github repo => https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet-vm

####1. Clone the Bigtop repository $ git clone https://github.com/apache/bigtop.git $ cd bigtop/bigtop-deploy/vm/vagrant-puppet-vm

####2. Install plugins $ vagrant plugin install vagrant-hostmanager
$ vagrant plugin install vagrant-cachier

####3. Configure Vagrantfile and vagrantfile.yaml

this modification is no longer needed because of the recent update in apache/bigtop git repo

The vagrantfile.yaml works as a configuration file of Vagrantfile. They handle the initialization of vagrant box automatically. The default box in vagrantfile.yaml is centos6.6 For basic deployment, there’s no need to change anything in vagrantfile.yaml However, we have to make some small modification in Vagrantfile, add two commands.

68 $script = <<SCRIPT
69 service iptables stop
70 chkconfig iptables off
+  71 puppet module install puppetlabs-stdlib
72 cat /dev/null > /etc/hosts
73 echo "Bigtop yum repo = #{repo}"
74 # Prepare puppet configuration file
75 mkdir -p /etc/puppet/hieradata
+ 76 mkdir -p /data/1 /data/2

The vagrantfile.yaml also has other options that users can specify such as: change number of nodes to provision in bigtop cluster

 num_instances: 5

see their github repo for more infomation

####4. Spin up Bigtop Cluster

 vagrant up

Smoke tests

set run_smoke_tests: true in vagrantconfig.yaml

  • this will execute utils/smoke-tests.sh while vagrant up
    • this script will install pig, hive, flume, sqoop, mahout to the cluster. (they don’t seem to have smoke tests on spark which is weird)
    • and execute default smoke tests on mapreduce and pig.
    • to run more smoke tests look at section 3. GO

####1. Spin up Cluster and ssh into it

vagrant up
vagrant ssh

####2. Export some environment variables

export HADOOP_CONF_DIR=/etc/hadoop/conf/
export BIGTOP_HOME=/bigtop-home/
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/
export HIVE_HOME=/usr/lib/hive/
export PIG_HOME=/usr/lib/pig/
export FLUME_HOME=/usr/lib/flume/
export HIVE_CONF_DIR=/etc/hive/conf/
export JAVA_HOME=/usr/lib/jvm/java-openjdk/
export MAHOUT_HOME=/usr/lib/mahout
export SQOOP_HOME=/usr/lib/sqoop
export ITEST="0.7.0"

####3. GO move to smoke tests directory and switch to root user

cd /bigtop-home/bigtop-tests/smoke-tests

for example: run smoke tests on flume and hive

./gradlew compileGroovy test -Dsmoke.tests=flume,hive --info

--info option helps debugging, can change to --debug if needed more detail