User Tools

Site Tools


Sidebar

Navigation

Join us by

user mailing list
devel mailing list


More news...

RSS

tutorial:process_and_result_of_benchmark

How to launch terasort of other test on grid5000?

To easily run a benchmark on grid5000, you have to use the image of demo on grid5000 and modificate the application-test.sh.

Download the zip file of Demo:

https://gforge.inria.fr/scm/viewvc.php/contrib/demo/tags/BlobSeer-Demo-0.3.3-1.tar.gz?root=blobseer&view=log
scp BlobSeer-Demo-0.3.3.zip USER@access.grid5000.fr:/home/USER/nancy

Do next command in your $HOME folder:

ssh nancy.g5k 
tar -xzvf BlobSeer-Demo-0.3.3-1.tar.gz  
mv BlobSeer-Demo-0.3.3 BlobSeer-Demo
cp /home/lcloatre/demo-squeeze-x64-base.env .
cp /home/lcloatre/demo-0_3_3-image-BSFS-1.2.2.tgz .

Update the env file with your user name:

vi demo-squeeze-x64-base.env

Deployment and automatic test

  • Launch the demo to reserve node and deploy image in it (j= to submit a reservation on g5k, n= number of nodes, r= number of hour of reservation):
~/BlobSeer-Demo/demo-scripts/demo-launch.sh -j true -n 5 -r 1
  • Launch the deployment of BlobSeer/BSFS (f = file system, t= type of test, c= concurrency, p= number of data provider in blobSeer, m= number of metadata provider):
cd BlobSeer-Demo/demo-scripts
./application-test.sh -f BSFS -s 3 -t terasort -c 2 -p 3 -m 1

Manual Test

Terasort

  • To make manual test, you can stop the application-test before to launch hadoop job:

And add an “echo ${master}” and “exit 0” command after the bloc:

echo "---------------------------------"
echo "----- Start Map Reduce Task -----"
echo "---------------------------------"
echo " "  
echo ${master}
exit 0
  1. You can configure properties in hadoop folder, for example set fs.bsfs.size to 8388608 (8MB) in core-site.xml, then restart all:
./hadoop-bsfs-1.2.1/bin/stop-all.sh  
./hadoop-bsfs-1.2.1/bin/start-all.sh
  1. You can launch the command now, example:
./hadoop jar ../hadoop-examples-1.2.1.jar teragen -Ddfs.block.size=536870912 -Dmapred.map.tasks=32 -Dmapred.reduce.tasks=16 -Dmapred.map.tasks.speculative.execution=true -Dmapred.compress.map.output=true 100000000 /Workloads/teragen/ter_in.10000m536870911
./hadoop jar ../hadoop-examples-1.2.1.jar  terasort -Ddfs.block.size=536870912 -Dio.file.buffer.size=32768 -Dmapred.map.tasks=32 -Dmapred.reduce.tasks=16 -Dio.sort.factor=48 -Dio.sort.record.percent=0.138 /Workloads/teragen/ter_in.10000m536870911 /Workloads/terasort/TEST_OUTPUT.10000m5368709121
./hadoop jar ../hadoop-examples-1.2.1.jar  teravalidate  /Workloads/terasort/TEST_OUTPUT.1000m536870912 /Workloads/teravalidate/TEST_VALIDATE.1000m536870912

Grep

  1. You can launch the command now, example:
./hadoop jar ../hadoop-examples-1.2.1.jar  randomtextwriter -Ddfs.block.size=1073741824 -Dtest.randomtextwrite.bytes_per_map=46777216 -Dtest.randomtextwrite.maps_per_host=16  /Workloads/grep/data_20688aa
./hadoop jar ../hadoop-examples-1.2.1.jar grep /Workloads/grep/data_20688aa /Workloads/grep/out_2as "toto"

Results

  • Here you can find differents values of parameters that has been tested:

Excel file

tutorial/process_and_result_of_benchmark.txt · Last modified: 2015/01/07 11:05 by lcloatre