User Tools

Site Tools


Sidebar

Navigation

Join us by

user mailing list
devel mailing list


More news...

RSS

tutorial:installation_and_launch

Installation and launch

You can use stable release of Demo using bash script 0.3.3 or the last release 0.3.5 using python library Execo for manage grid5000(recommanded).

Release Stable: 0.3.3:

The Demo can be downloaded here, click download link and save on the local machine of the user.

Copy it from your laptop to grid5000account:
 scp  BlobSeer-Demo-0.3.3.zip   USER@access.grid5000.fr:/home/USER/rennes/BlobSeer-Demo-0.3.3.zip

First you have to retrieve snapshot to be deployed on grid5000 and the configuration file env:

cp /home/lcloatre/demo-0_3_3-image.tgz .
cp /home/lcloatre/demo-squeeze-x64-base.env .

Update demo-squeeze-x64-base.env to target your username.

vi demo-squeeze-x64-base.env
---
name: squeeze-x64-base
...
image:
  kind: tar
  compression: gzip
  file: /home/**YOUR_USERNAME**/demo-0_3_3-image.tgz

Release 0.3.5 still in test:

The Demo can be downloaded here, click download link and save on the local machine of the user.

Copy it from your laptop to grid5000account:
 scp  BlobSeer-Demo-0.3.5.zip   USER@access.grid5000.fr:/home/USER/rennes/BlobSeer-Demo-0.3.5.zip

First you have to retrieve snapshot to be deployed on grid5000 and the configuration file env:

cp /home/lcloatre/demo-0_3_5-blobseer.tgz .
cp /home/lcloatre/demo-squeeze-x64-base.env .

Update demo-squeeze-x64-base.env to target your username.

vi demo-squeeze-x64-base.env
---
name: squeeze-x64-base
...
image:
  kind: tar
  compression: gzip
  file: /home/**YOUR_USERNAME**/demo-0_3_5-blobseer.tgz

Installation

For demo-0.3.5 you have to install execo on grid5000 by typing:

export http_proxy="http://proxy:3128" ; export https_proxy="https://proxy:3128"
easy_install --user execo

Copy the zip file to Grid5000 account HOME directory.

scp BlobSeer-Demo-0.3.5.zip <login>@access.grid5000.fr:<site>/

To install the Demo, the administrator needs only to extract 'BlobSeer-Demo-0.3.5.zip' on Grid5000.

unzip BlobSeer-Demo-0.3.5.zip

The command creates the main directory BlobSeer-Demo-0.3.5. Change the name of the directory to BlobSeer-Demo

mv BlobSeer-Demo-0.3.5 BlobSeer-Demo

Attention !!

To use the visualization functionality, the same installation process should be repeated again on the user's local machine. Moreover, gnuplot should be installed installed to draw figures.

Launch the Demo

Enter the repository BlobSeer-Demo, the user will find several subdirectories:

  1. benchmark-scripts: scripts to launch benchmark tests concurrently on a set of Grid5000 nodes.
  2. benchmark-tests: Java classes of benchmark tests.
  3. demo-sbin: scripts to deploy BlobSeer, Hadoop-BSFS/HDFS on the reserved Grid5000 nodes.
  4. demo-scripts: main programs.
  5. docs: documentations for the Demo.
  6. figure-scripts: visualization functionality.
  7. logs: repository stores the logs, output and results of all tests.

The scripts in folders 1, 2 and 3 are already installed in the Kadeploy image. Usually, the user needs to do nothing with them unless he wants to create the Demo image from scratch.

Then, the user type the command:

version 0.3.3:
~/BlobSeer-Demo/demo-scripts/demo-launch.sh -h
version 0.3.5:
python ~/BlobSeer-Demo/demo-scripts/demo-lauch-total.py -h

This command shows all the mandatory and optional parameters:

===== G5K configuration =====
-j, --job-submit                  : Set it to 'true' to submit a G5K job. Default setting is false.
-i, --job-id                      : Please either enter the job ID or submit a new job.
-n, --n-number	                  : Number of wanted G5K nodes (3 nodes as minimum).
-r, --run-time		          : Job runing time (unit is hour, 2 hours is default).
-d, --deploy			  : Set it to 'true' to deploy working environment. Default setting is false.
-h, -?, --help                    : Display this help message.

Then, simply run the script, the user can deploy a certain number of nodes with Hadoop-BSFS/HDFS installed and ready to be run.

For example if the user wants to start a new job for 2 hours on Grid5000 with 5 nodes, he can type:

version 0.3.3:
~/BlobSeer-Demo/demo-scripts/demo-launch.sh -j true -n 5 -r 2 
version 0.3.5:
python ~/BlobSeer-Demo/demo-scripts/demo-lauch-total.py -j true -n 5 -r 2 

Here is the sequence diagram explaining in some words each steps:

Then, we can see that the program start to reserve resouces on Grid5000:

Job submission is required.
Nodes number is set to 5.
Job runtime is set to 2.

***** Check input parameters *****
BSFS will be used.
----------------------
----- Submit job -----
----------------------
Request has been sent, waiting ...
[ADMISSION RULE] Modify resource description with type constraints
Generate a job key...
OAR_JOB_ID=XXXXXX
Reservation mode : waiting validation...
Reservation valid --> OK

Submission accepted, wait another 2 mins.
...
...
***** Write configuration file. *****
...
...
***** Retrieve available nodes. *****
...
...
***** Deploy environment on nodes. *****
...
...
Wait for another 1 minutes to confirm that nodes are up.

Clean temporary files.

Be careful, the process may be long since the deployment of image on nodes is quite time consuming.

If the user has already the reservation on Grid5000, he can use the job ID to re-deploy the nodes:

version 0.3.3:
~/BlobSeer-Demo/demo-scripts/demo-launch.sh -i <job-ID> -d true
version 0.3.5:
python ~/BlobSeer-Demo/demo-scripts/demo-lauch-total.py -i <job-ID> -d true 

If everything is OK…please go to next step Play with single test

Details of the Launch Process

This is the description of the launch process. To start a test, please go to the next section.

There are mainly three steps in the deployment process:

  • Reserve resources on Grid5000
  • Deploy nodes
  • Pre-configuration of each VM

The first step reserves a certain number of nodes as it is required. Since the deployment of BlobSeer on a cluster needs at least three nodes, the default value of the parameter –n-number is set to three. Basically, we use the command:

$ oarsub -r "current time" -l nodes=${--n-number},walltime="${runtime}" -t deploy

to obtain nodes.

If enough number of nodes are launched in the first step, the script proceeds to the next phase, namely, environment deployment. In this phase the reserved nodes load the prepared image and boot.

The final step is to set SSH configuration on each node to ensure that nodes can connect with each other via SSH. After the ssh configuration is correctly finished, the nodes are ready to run BlobSeer, Hadoop-BSFS/HDFS and various tests.

At the end we get this:

tutorial/installation_and_launch.txt · Last modified: 2015/01/19 11:13 by lcloatre