Gratia Release v0.32.1c (3/31/2008 - 4/1/2008)

Overview

This release (v0.32.1c) is a maintenance release.

Main New Features:

NONE.

Additional Features:

NONE.

Technical changes affecting some of the listed features

Anticipated downtime

It is expected that this release will require the Gratia services and reporting to be unavailable for a very brief (10 minute) period at the most.

• Start: 3/31/2008
• Available: 3/31/2008
• Actual: Start - End -

The changes affecting downtime for this release are:

1. Length of time it takes to install the new software

Collectors and Databases Affected

The following Gratia collectors and databases will be converted with this release:

The following Gratia collectors and databases will NOT be converted with this release. The gratia collector will be upgraded to the new software as part of the conversion to InnoDB.

The following Gratia collectors and databases will NOT be converted with this release. These repositories contained specialized reports that have not as yet been upgraded to the new Birt V2.2 software. They will not be affected by this upgrade.

Build the v0.32.1c for distribution

1. Make sure your build area contains all committed changes.
   • cvs update
2. In gratia/build-scripts/Makefile , change the version_default to:
   • version_default = v0.33
   • commit the change

   The reason for using a version number 1 more than the current release is so that all development that occurs after this release will have the odd numbered development release (v0.33).

   The current release (v0.32.1c) will get assigned properly when the cvs export is performed after the tagging occurs.

3. Tag the release:
   • cvs tag -R v0-32-1c gratia
   This tags everything that is in your build directory and presumably tested.
4. Into a new area, export the tagged release:
   • cvs export -d gratia-v0.32.1c -r v0-32-1c gratia
5. Build it for the release (this insures that tar files are produced for VDT):
   • cd gratia-v0.32.1c/build-scripts
   • source setup-jdk15.sh
   • make release
6. Copy the built tar files to the release area:
   • scp ../target/*_v0.32.1c.tar flxi07.fnal.gov:/afs/fnal.gov/files/expwww/gratia/html/Files/
   This step is NOT being done for this point release
7. Update the version number on the services release TWiki page:
• Edit and update the TWiki variable ReleaseVersion.
This step is NOT being done for this point release

Done: 3/31/08 11:45 CDT

Upgrade and implementation

The upgrades should be single-threaded , that is, performed for each database schema one at a time.

We will perform these upgrades based on the size of the individual database schema, in ascending order.

For fermi_itb on gratia08:

1. Stop the tomcat init.d service for the Gratia collector:
   • service tomcat-fermi_itb stop

2. Save off the logs under each tomcat instance and empty the log directory.
   • date=`date '+%Y%m%d'`
   • cd /data/tomcat-fermi_itb/logs/
   • tar zcf /data/gratia_tomcat_logs_backups/tomcat-fermi_itb.$date.tgz *
   • rm -f *

3. Install the new software on the Gratia tomcat instance:
   pswd=xxx
   source=/home/weigand/cdcvs/gratia-v0.32.1c
   pgm=/home/weigand/cdcvs/gratia-v0.32.1c/common/configuration/update-gratia-local
   $pgm -d $pswd -S $source fermi_itb
4. Start the tomcat init.d service for the Gratia collector:
   • service tomcat-fermi_itb start
   • When the tomcat service initializes, it will detect any schema changes have been effected and a conversion process will begin.
   • tail the catalina.out log. When the conversion process completes the log will show the following message:
     "INFO: Server startup in _xxx_ms"

Start: 3/31/08 16:09 End: 3/31/08 16:12

For fermi_osg on gratia08:

1. Stop the tomcat init.d service for the Gratia collector:
   • service tomcat-fermi_osg stop

2. Save off the logs under each tomcat instance and empty the log directory.
   • date=`date '+%Y%m%d'`
   • cd /data/tomcat-fermi_osg/logs/
   • tar zcf /data/gratia_tomcat_logs_backups/tomcat-fermi_osg.$date.tgz *
   • rm -f *

3. Install the new software on the Gratia tomcat instance:
   pswd=xxx
   source=/home/weigand/cdcvs/gratia-v0.32.1c
   pgm=/home/weigand/cdcvs/gratia-v0.32.1c/common/configuration/update-gratia-local
   $pgm -d $pswd -S $source fermi_osg
4. Start the tomcat init.d service for the Gratia collector:
   • service tomcat-fermi_osg start
   • When the tomcat service initializes, it will detect any schema changes have been effected and a conversion process will begin.
   • tail the catalina.out log. When the conversion process completes the log will show the following message:
     "INFO: Server startup in _xxx_ms"

Start: 3/31/08 16:16 End: 3/31/08 16:18

For qcd on gratia08:

1. Stop the tomcat init.d service for the Gratia collector:
   • service tomcat-qcd stop

2. Save off the logs under each tomcat instance and empty the log directory.
   • date=`date '+%Y%m%d'`
   • cd /data/tomcat-qcd/logs/
   • tar zcf /data/gratia_tomcat_logs_backups/tomcat-qcd.$date.tgz *
   • rm -f *

3. Install the new software on the Gratia tomcat instance:
   pswd=xxx
   source=/home/weigand/cdcvs/gratia-v0.32.1c
   pgm=/home/weigand/cdcvs/gratia-v0.32.1c/common/configuration/update-gratia-local
   $pgm -d $pswd -S $source qcd
4. Start the tomcat init.d service for the Gratia collector:
   • service tomcat-qcd start
   • When the tomcat service initializes, it will detect any schema changes have been effected and a conversion process will begin.
   • tail the catalina.out log. When the conversion process completes the log will show the following message:
     "INFO: Server startup in _xxx_ms"

Start: 4/1/08 07:38 End: 4/1/08 07:40

For itb on gratia09:

1. Stop the tomcat init.d service for the Gratia collector:
   • service tomcat-itb stop

2. Save off the logs under each tomcat instance and empty the log directory.
   • date=`date '+%Y%m%d'`
   • cd /data/tomcat-itb/logs/
   • tar zcf /data/gratia_tomcat_logs_backups/tomcat-itb.$date.tgz *
   • rm -f *

3. Install the new software on the Gratia tomcat instance:
   pswd=xxx
   source=/home/weigand/cdcvs/gratia-v0.32.1c
   pgm=/home/weigand/cdcvs/gratia-v0.32.1c/common/configuration/update-gratia-local
   $pgm -d $pswd -S $source itb
4. Start the tomcat init.d service for the Gratia collector:
   • service tomcat-itb start
   • When the tomcat service initializes, it will detect any schema changes have been effected and a conversion process will begin.
   • tail the catalina.out log. When the conversion process completes the log will show the following message:
     "INFO: Server startup in _xxx_ms"

Start: 4/1/08 07:42 End: 4/1/08 07:44

For osg_integration on gratia09:

1. Stop the tomcat init.d service for the Gratia collector:
   • service tomcat-osg_integration stop

2. Save off the logs under each tomcat instance and empty the log directory.
   • date=`date '+%Y%m%d'`
   • cd /data/tomcat-osg_integration/logs/
   • tar zcf /data/gratia_tomcat_logs_backups/tomcat-osg_integration.$date.tgz *
   • rm -f *

3. Install the new software on the Gratia tomcat instance:
   pswd=xxx
   source=/home/weigand/cdcvs/gratia-v0.32.1c
   pgm=/home/weigand/cdcvs/gratia-v0.32.1c/common/configuration/update-gratia-local
   $pgm -d $pswd -S $source osg_integration
4. Start the tomcat init.d service for the Gratia collector:
   • service tomcat-osg_integration start
   • When the tomcat service initializes, it will detect any schema changes have been effected and a conversion process will begin.
   • tail the catalina.out log. When the conversion process completes the log will show the following message:
     "INFO: Server startup in _xxx_ms"

Start: 4/1/08 07:47 End: 4/1/08 07:49

Post-mortem

At this time, this will appear to be random notes. After the conversion, they may be organized.

1. osg_integration instance problems with replicated data.

   The osg_integration instance (http://gratia.opensciencegrid.org:8885/gratia-reporting/) is experiencing problems with the v0.32.1c conversion. All the other databases appear ok.

   We had turned back on, late last week, the replication of specific gratia_itb probes to osg_integration and all was doing well. Now it appears that this instance is getting into strange state when processing the replicated data.

   Symptoms:

   • the http://gratia.opensciencegrid.org:8885/gratia-administration page would never render... it would just spin
   • the hibernate log shows many many many of these errors which can be viewed in /data/gratia_tomcat_logs_backups/osg_integration, which are the logs I saved this morning

   2008-04-02 01:13:41,003 org.hibernate.util.JDBCExceptionReporter [WARN]: SQL Error: 1062, SQLState: 23000
   2008-04-02 01:13:41,003 org.hibernate.util.JDBCExceptionReporter [ERROR]: Duplicate entry 'F8F4C00A065268D4C189E6F440A33B54' for key 2
   

   I turned off ALL replication from gratia_itb to osg_integration this morning around 4/2 09:35.

   I am not certain this is related to the 0.32.1c version or is specific with replication or maybe a combination.

Major updates

-- JohnWeigand - 31 Mar 2008