You are here: TWiki > Accounting Web>GratiaWebDevAndTest (06 Dec 2016, KyleGross)

Measurements and Metrics Graphing (Gratia Web) Development Guide

Metrics and Measurements provides a web graphing system designed for experts to pull data out of the Gratia system and other data sources. It is based upon GraphTool.

Installation

Since the Gratia Web have multiple dependencies, the best way to install it with its dependencies is to install it from the OSG repositories. Please follow the instructions on: Install Gratia Web from the OSG repository

Development Directions

Gratia Web is composed of 2 different projects whose source code is kept in github, the projects are OSG-Measurements-Metrics and graphtool.

The source code for OSG-Measurements-Metrics is kept here:

OSG-Measurements-Metrics uses graphtool to submit database queries and create plots. The source code for graphtool is kept here:

These 2 github projects are also eclipse/!PyDev projects.

RPM Build

Both projects contain and RPM building script called buildrpms.sh in the base folder of each project. The changelog file is also kept in the base folder of each project with the name changelog.txt. The RPM builiding configuration is set in different configuration files.

For OSG-Measurements-Metrics these files are:

  • ./setup/setup_basic.cfg
  • ./setup/setup_Db.cfg
  • ./setup/setup_static_graphs.cfg
  • ./setup/setup_Web.cfg
  • ./setup/setup.cfg

For graphtool these files are:

  • ./setup.cfg

In this files you can change the release number of the rpm that is going to be built.

[bdist_rpm]
...
release = X
...
X is the release number.

To rebuild the RPMs run the following from the base folder:

[user@client ~]$./buildrpms.sh
on The path to this directory must not contain any spaces

This will put the new copy of the RPM(s) in dist/.

To clean the temporary directories, created with the previous script, run from the base folder:

[user@client ~]$./local-clean.sh

Deploy changes on Development machine

After changes have been made to the project code those changes can be deployed on the development machine running from the base folder of each project:

[user@client ~]$./local-deploy.sh X
Where X is the release number of the project.

As well they could be uninstalled running from the base folder of each project:

[user@client ~]$./local-uninstall.sh

To deploy changes for OSG-Measurements-Metrics and graphtool at the same time the following script can be used from the base folder of OSG-Measurements-Metrics:

[user@client ~]$./local-deploy-with-graphtool.sh X Y 
Where X is the release number of OSG-Measurements-Metrics and Y is the release number of graphtool.

on Root access will be requested to run these scripts.

on graphtool uninstall script will fail if osg-measurements-metrics-db package is still installed on the machine.

on ./local-deploy.sh and ./local-deploy-with-graphtool.sh will uninstall the current RPMs on the machine, build the required rpms and install them.

Eclipse + PyDev

Installation

  1. Verify that you have Java installed
    [user@client ~]$ java -version
    *If Java is installed you must get something like:*
    java version "1.7.0_75"
    OpenJDK Runtime Environment (rhel-2.5.4.0.el6_6-x86_64 u75-b13)
    OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
    If Java is not installed, it can be downloaded from https://www.java.com.
  2. Download Eclipse from https://eclipse.org/

on Any eclipse should work, but the Eclipse IDE for Java Developers is recommended

  1. Open Eclipse (You can create a new workspace or use the default one)
  2. Go to the menu Help -> Eclipse Marketplace
  3. Search for PyDev and install PyDev (Python IDE for Eclipse)
  4. (Optional) Use the marketplace to install EasyShell, this plug-in will let you open terminals easily for the projects locations you'll work on.
  5. Go to the menu Window -> Open Perspective -> Other
  6. Select PyDev and open it
  7. A prompt will request to configure python
    • If python is installed in the default location select Quick Autoconfig
    • If not select Manual Configuration and setup python
  8. Go to Window -> Preferences -> Editors -> Text Editor
  9. Check the Show whitespace characters and Insert spaces for tabs and press OK on This will help you avoid errors caused by incorrect indentation

Import Python Projects on PyDev

If you already have a clone of the repository:

  1. Go to the menu File -> Import -> General -> Existing Projects into Workspace
  2. Browse and select the base folder of the project you want to import
  3. Select the project you want to open and press Finish

If you don't have already a clone of the repository:

  1. Setup your machine to use github with the public key configuration.
  2. Go to the menu File -> Import -> Git -> Projects from Git
  3. Select Clone URI
  4. Insert the clone SSH URI
  5. Select SSH as connection protocol on User must be git with no password
  6. Select the branches you are going to work on
  7. Select the directory where the repository is going to be cloned in your local machine and press Next
  8. Select Import existing projects and press Next
  9. Check the project you want to import and press Finish

Running/Debugging with PyDev

PyDev projects have source folders. These source folder can be recognized because they have a package symbol over the folder icon in the Package Explorer, and python packages have a package icon instead. All of the source folders in the project are included in the PYTHONPATH when running a script from eclipse.

  • Source Folders and Packages in PyDev:
    src-pkg.png

on To create a new source folder you can do right click on the project and select New -> Source Folder

on To create a package in python you must add the new folder and include an __init__.py file (even if is an empty file).

on To include code in the PYTHONPATH that is not included in the project by itself or in the system python libraries you can:

  • Do right click on the project and go to Properties -> Project References where you can select other projects that are imported in eclipse to be included in your project PYTHONPATH
  • Do right click on the project and go to Properties -> PyDev - PYTHONPATH and setup manually the PYTHONPATH

Running a Script

  1. Select or open the script you want to run (it must be in a source folder).
  2. Do right click on the script and go to Run As -> Python Run.
  3. The console will prompt displaying the output of the script and will give you the option to terminate the script.

on If the console does not prompt go to Window -> Show View -> Console

Debugging a Script

  1. Open the script you want to debug
  2. Go to the line of code where you want to start to control the program execution
  3. Over the line number on the left do right click and select Add Breakpoint
  4. Do right click on the script and go to Debug As -> Python Run.
  5. Wait until the program execution gets to the line where you added the breakpoint
  6. If Confirm Perspective Switch is prompt select Yes
  7. The Debug Perspective opens and will allow you to control the program execution as well as examine the variable contents

  • PyDev Variable Explorer:
    Variables.png

on Debug controls allow you to:

  • Step Into (shortcut F5): If a function is about to be invoked will continue the step-by-step control inside the function definition
  • Step Over (shortcut F6): Will continue the step-by-step control in the next line of code of the current script
  • Step Return (shortcut F7): Will continue the step-by-step control in the next return
  • Resume (shortcut F8): Will resume the normal program execution until the next breakpoint
  • Terminate (shortcut ctrl+F2): Will terminate the program execution

  • PyDev debug controls:
    DebugControls.png
on To return to the PyDev develpoment perspective click on PyDev on the upper right corner, or go to the menu Window -> Open Perspective -> Other and select PyDev.

  • PyDev Quick Perspective Change:
    !PyDevPerspective.png

Gratia Web Development with PyDev

OSG-Measurements-Metrics and graphtool are already configured as PyDev projects. OSG-Measurements-Metrics is configured to reference the graphtool project to include its code in the PYTHONPATH. However the graphtool PyDev project is not required to be imported if the code that need to be modified is included only in OSG-Measurements-Metrics. If graphtool PyDev project is imported in your eclipse workspace, when you run a script from the OSG-Measurements-Metrics source folder in eclipse, it will include the changes of code that you have on eclipse in the graphtool project. To avoid this and use the graphtool code installed in your computer you can close the graphtool eclipse project.

Before you run the OSG-Measurements-Metrics server from eclipse you will need to adjust the website-devel.xml file. This file is located in src/gratia/config. Replace the /home/user/git/OSG-Measurements-Metrics/static for the absolute path to the OSG-Measurements-Metrics/static folder in your machine.

<graphtool-config>
...
  <class name="static-site" type="StaticContent">
    <directory location="/gratia/static-site" name="/gratia/static-site">/home/user/git/OSG-Measurements-Metrics/static/</directory>
  </class>
...
</graphtool-config>

After this go to src/gratia/tools/gratia_web_dev.py and run the script from eclipse. This will start the server and will be available from http://localhost:8086/gratia.

on The development server will use the database configuration located at /etc/DBParam.xml.

Adding queries to GratiaWeb

Check the following documentation about queries in GratiaWeb: Adding/Modifying Queries to GratiaWeb

Comments

-- JuanFelipeMosqueraMorales - 11 Feb 2015

Topic attachments
I Attachment Action Size Date Who Comment
pngpng DebugControls.png manage 5.4 K 12 Feb 2015 - 18:44 JuanFelipeMosqueraMorales PyDev? debug controls
pngpng PyDevPerspective.png manage 12.5 K 12 Feb 2015 - 18:47 JuanFelipeMosqueraMorales PyDev? Quick Perspective Change
pngpng Variables.png manage 21.8 K 12 Feb 2015 - 18:44 JuanFelipeMosqueraMorales PyDev? Variable Explorer
pngpng src-pkg.png manage 31.2 K 12 Feb 2015 - 18:42 JuanFelipeMosqueraMorales Source Folders and Packages in PyDev?
Topic revision: r16 - 06 Dec 2016 - 18:12:35 - KyleGross
Hello, TWikiGuest
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..