RPM Development Guide

This page documents technical guidelines and details about RPM development for the OSG Software Stack. The procedures, conventions, and policies defined within are used by the OSG Software Team, and are recommended to all external developers who wish to contribute to the OSG Software Stack.

Status: Mid-December 2011: needs review

Principles

The principles below guide the design and implementation of the technical details that follow.

  • Packages should adhere to community standards (e.g., Fedora Packaging Guidelines) when possible, and significant deviations must be documented
  • Every released package must be reproducible from data stored in our system
  • Source code for software should be clearly separable from the packaging of that software
  • Upstream source files (which should not be modified) should be clearly separated from files owned by the OSG Software Team
  • Building source and binary packages from our system should be easy and efficient
  • External developers should have a clear and effective system for developing and contributing packages
  • We should use standard tools from relevant packaging and development communities when appropriate

Contributing Packages

We encourage all interested parties to contribute to OSG Software, and all the infrastructure described on this page should be friendly to external contributors.

However, not all of the infrastructure is in place yet: you must gather a few authorizations. We will be working to decrease the extra steps necessary.

  • To participate in the packaging community: You must subscribe to the osg-software@opensciencegrid.org email list. Subscribing to an OSG email list is described here.
  • To create and edit packages: Obtain access to VDT SVN.
  • To upload new source tarballs: You must have a cs.wisc.edu account with write access to the VDT source tarball directory. Email the osg-software list and request permission.

Development Infrastructure

This section documents most of what a developer needs to know about our RPM infrastructure:

  • Upstream Source Cache — a filesystem scheme for caching upstream source files
  • Revision Control System — where to get and store development files, and how they are organized
  • Build System — how to build packages from the revision control system
  • Development Tools — other tools to aid development
  • Yum Repository — the location and organization of our Yum repository, and how to promote packages through it

Upstream Source Cache

One of our principles (every released package must be reproducible from data stored in our system) creates a potential issue: If we keep all historical source data, especially upstream files like source tarballs and source RPMs, in our revision control system, we may face large checkouts and consequently long checkout and update times.

Our solution is to cache all upstream source files in a separate filesystem area, retaining historical files indefinitely. To avoid tainting upstream files, our policy is to leave them unmodified after download.

Now that we build RPMs in koji, it is considered the authoritative location for source files we have released packages from.

Locating Files in the Cache

Upstream source files are stored in the filesystem as follows:

prefix/package/version/file

where:

Symbol Definition Example
prefix Filesystem prefix, the base location of the upstream source cache At Madison: /p/vdt/public/html/upstream
package Upstream name of the source package, or some widely accepted form thereof ndt
version Upstream version string used to identify the release 3.6.4
file Upstream filename itself (unless filename conflicts force a change, which should be exceedingly rare) ndt-3.6.4.tar.gz

The authoritative cache is the VDT webserver, which is fully backed up. Other developers can create a local cache for efficient development.

Upstream source files are referenced from within the revision control system; see below for details.

Contributing Upstream Files

If you are using a local cache during development, you must make sure that any new upstream source files are cached on the VDT webserver before declaring the package to be ready for testing. You have two options:

  • If you have access to a UW–Madison CSL machine, you can scp the source files directly into the AFS locations using that machine
  • If you do not have such access, write to the osg-software list to find someone who will post the files for you

Revision Control System

All packages that the OSG Software Team releases are checked into our revision control system (currently Subversion but with the option to change at a later date).

Subversion Access

Our Subversion repository is located at:

https://vdt.cs.wisc.edu/svn

Procedure for offsite users obtaining access to Subversion

Or, from a UW–Madison Computer Sciences machine:

file:///p/condor/workspaces/vdt/svn

The current SVN directory housing our work-in-progress native package work is $repo/native (where $repo is one of the ways of accessing our SVN repository above). For example, to check out the current Subversion repository via https, do:

svn co https://vdt.cs.wisc.edu/svn/native

OSG-Owned Software

OSG-owned software goes into Subversion as follows:

software/package/section

where:

Symbol Definition Example
package Our name for the software package vdt-system-profiler
section Development section Standard Subversion sections like trunk, branches/*, and tags/*

Within a section directory, files are organized as the developer sees fit.

It is strongly recommended that each software package include a top-level Makefile with at least the following targets:

Symbol Purpose
install Install the software into final FHS locations rooted at DESTDIR
dist Create a distribution source tarball (in the current section directory) for a release
upstream Install the distribution source tarball into the upstream source cache (default to VDT AFS, but override the upstream root with UPSTREAM

Packaging Top-Level Directory Organization

The top levels of our Subversion directory hierarchy for packaging are as follows:

native/os/section/package

where:

Symbol Definition Example
os Major operating system release redhat (only supported value, currently)
section Development section Standard Subversion sections like trunk, branches/*, and tags/*
package Our standardized name for a source package ndt

Package Directory Organization

Within a source package directory, the following files (detailed in separate sections below) may exist:

README text file package notes, by and for developers
upstream/ directory references to the upstream source cache and other kinds of upstream files
osg/ directory overrides and patches of upstream files, plus new files, which contribute to the final OSG source package
_upstream_srpm_contents/ generated directory (not under revision control) expanded contents of a cached upstream source package
_upstream_tarball_contents/ generated directory (not under revision control) expanded contents of all cached upstream source tarballs
_final_srpm_contents/ generated directory (not under revision control) the final contents of the OSG source package
_build_results/ generated directory (not under revision control) OSG source and binary packages resulting from a build

README

This is a free-form text file for developers to leave notes about the package. Please document anything interesting about how you procured the upstream source, the reasons for the modifications you made, or anything else people might need to know in order to maintain the package in the future. Please document the why, not just the what.

upstream

Within the per-package directories of the revision control system, there must be a way to refer to cached files. This is done with small text files that (a) are named consistently, and (b) contain the location of the referenced file as its contents.

A reference file is named:

description.type.source

where:

Symbol Definition Example
description Descriptive label of the source of the referenced file developer, epel, emi
type Type of referenced file Use: tarball, srpm

The contents of the file match the upstream source cache path defined above, without the prefix component:

package/version/file

This way, the command-line tools that use the .source files can be given different prefixes for different build situations.

For example, the reference file for NDT’s source tarball is named developer.tarball.source and might contain:

ndt/3.6.4/ndt-3.6.4.tar.gz

The .source files may contain comments, which start with # and continue until the end of the line.

osg

The osg directory contains files that are owned by the OSG Software Team and that are used to create the final, released source package. It may contain a variety of development files:

  • An RPM .spec file, which overrides any spec file from a referenced source
  • Patch (.patch) or replacement files, which override any same-named file from the top-level directory of a referenced source
  • Other files, which must be explicitly placed into the package by the spec file

_upstream_srpm_contents

The _upstream_srpm_contents directory contains the files that are part of the upstream source package. It is generated by the package build tool and is a volatile record of the upstream source for developer use. This directory is excluded from Subversion commits.

_upstream_tarball_contents

The _upstream_tarball_contents directory contains the files that are part of the upstream source tarballs. It is generated by the package build tool if and only if the --full-extract option is passed. It is not used for anything by the build tool, but meant as a convenience to allow the developer to look inside the upstream sources (for making patches, etc.). This directory is excluded from Subversion commits.

_final_srpm_contents

The _final_srpm_contents directory contains the final files that are part of the released source package. It is generated by the package build tool and is a volatile record of a build for developer use. This directory is excluded from Subversion commits.

_build_results

The _build_results directory contains the source and binary RPMs that are produced by a local build. The directory is generated by the package build tool and is a volatile record of a build for developer use. It is excluded from Subversion commits.

Use Case 1: Packaging Our Own Code

When the OSG Software Team contributes all of the files for a package, they are located in the software project area. The Makefile therein creates an upstream source tarball in the standard location. All that remains is to define a .source reference to the upstream source tarball, and a .spec file. For example, the OSG Software Team provides a debugging utility called vdt-system-profiler to collect system information that can be sent to the OSG Software Team to aid in debugging. Its software directory contains:

vdt-system-profiler/
    Makefile
    vdt-system-profiler

Its packaging directory contains:

vdt-system-profiler/
    osg/
        vdt-system-profiler.spec
    upstream/
        vdt.tarball.source

Use Case 2: Packaging an Upstream Source Tarball

When the OSG Software Team packages an upstream source tarball, for which there is no existing package, the source tarball is referenced with a .source file and we provide a spec file and, if necessary, patches. For example, RSV is provided as a source tarball only. Its package directory contains:

osg-rsv/
    osg/
        osg-rsv.spec
    upstream/
        developer.tarball.source

Use Case 3: Passing Through a Source RPM

When the OSG Software Team simply provides a copy of an existing source RPM, it is referenced with a .source file and that is it. For example, we do not modify the xrootd source RPM from EMI. Its package directory contains:

xrootd/
    upstream/
        developer.srpm.source

Use Case 4: Modifying a Source RPM

When the OSG Software Team modifies an existing source RPM, it is referenced with a .source file and then all changes to the upstream source are contained in the osg directory. For example, if we were to ship a modified Apache HTTPD webserver package, we would use this mechanism. Its package directory contains:

apache/
    osg/
        httpd.spec
        vdt-apachectl.patch
    upstream/
        epel.srpm.source

Build Process

We use the standard RPM build tools such as rpmbuild, mock, and koji with our own wrapper scripts around them.

Officially Supported Packages

  1. All necessary information to create the package will be committed to the VDT source code repository (see below)
  2. A tool is provided that will take those files, create a source RPM, and submit it to a build system; currently we support three underlying build systems:
    • rpmbuild
    • mock
    • koji

Official builds will be done with Koji, but contributors can use options 1 or 2 to develop a package before contributing it to the OSG Software Stack.

Unsupported Contributed Packages

Unsupported contributed packages have two build options. They will be provided via a separate yum repository for unsupported contributions.

  1. Build exactly like the supported packages (above).
  2. The RPM can be built by the contributor in their own environment and shared with the OSG Software Team for manual staging into the yum repository.

Should we support Koji in the future, we'll allow Koji submissions for trusted contributors.

Build Tools

The main build tools can be checked out from SVN, under software/osg-build/trunk, or installed from the osg-build RPM in the osg-development repository. See instructions on how to use the OSG build tools

OSG Software Repository

The GOC maintains the Yum repositories that contain our source and binary RPMs. The repositories are mirrored at other institutions as well. The repositories are hosted on http://repo.grid.iu.edu and is organized in the following fashion:

http://repo.grid.iu.edu/osg-level/architecture/*.rpm

where:

Symbol Definition Example
osg-level Release level of the package (see below) osg-development, osg-testing, osg-production, osg-contrib
architecture Architecture of the built packages i386, x86_64, source

In addition, once we start having multiple major-minor versions of the VDT, and supporting multiple OS releases, we will need to reorganize the repositories to include the following information:

Symbol Definition Example
version Major-minor version of the VDT 3.0
os Major operating system release for which packages were built el5, el6

Release Levels

Every package is classified into a release level based on the amount of testing it has undergone and our confidence in its stability. When a package is first built, it goes into the lowest level (osg-development). Only members of the OSG Software Team may promote packages upward through the release levels.

See the Release Policy page for more info.

Package Signing

Packages are signed automatically by Koji after a non-scratch build.

Packaging Conventions

In addition to adhering to the Fedora Packaging Guidelines (FPG), we have a few rules and guidelines of our own:

  • When we pass-through an RPM and make any changes to it (so it has an updated package number), we construct the version-release as follows:
    • The version of the original RPM remains unchanged.
    • The release is composed of three parts: ORIGINALRELEASE.OSGRELEASE
    • We add the 'osg.el5' (or 'osg.el6') distro tag. (Use %{?dist} in the Release field)
Example: We copy package foobar-3.0.5-1 from somewhere. We need to patch it, so the full name-version-release (NVR) becomes foobar-3.0.5-1.1.osg.el5 That is, we added ".1.osg.el5" to the release number. If we update our packaging (but still base on foobar-3.0.5-1), we change to ".2.osg.el5". In the spec file, this would look like:
Release: 1.2%{?dist}
  • We may require packages that do not meet all FPG guidelines to install a file (perhaps in /usr/share/doc/package/) that documents major deviations from the guidelines that may be significant to end users.
  • We should probably provide a more user-level README file in /usr/share/doc/package as well, if a package doesn't come with some form of documentation.
  • Java-based software must be packaged according to our Java packaging policy.

Packaging for Multiple Distro Versions

Conditionalizing spec files

We are starting support for RHEL 6-based distributions in addition to RHEL 5-based ones. In order to minimize effort needed to keep the builds synchronized, instead of creating separate spec files for EL 5 and EL 6, we will be writing spec files such that they build on both distro versions. Some packages may need different build behavior between versions; RPM conditional statements will be used to handle this. The following macros are defined:

Name Value (EL5) Value (EL6)
%rhel 5 6
%el5 1 undefined or 0
%el6 undefined or 0 1
%dist .osg.el5 .osg.el6

Here's how to use them:

%if 0%{?el5}
# this code will be executed on EL 5 only
%endif
%if 0%{?el6}
# this code will be executed on EL 6 only
%endif
%if 0%{?rhel} >= 6
# this code will be executed on EL 6 and newer
%endif
(There does not seem to be an %elseif).

The syntax %{?el6} expands to the value of the %el6 macro if it is defined, and to the empty string if not; the 0 is there to keep the condition from being empty in the %if statement if the macro is not defined.

Other notes

  • Pass --el5 or --el6 to osg-build to target those distros. --el5 is the default, regardless of the distro of the machine you are running osg-build on.
  • For your final builds, use osg-build allbuild PACKAGE to build the package for all distros. This prevents packages from getting out of sync between distros due to forgetfulness.
  • Never* tag a package built for el5 with a tag beginning with el6, or vice versa.
  • Always make sure %{?dist} is at the end of the Release field for each package you make; in the past you could get away with having it be missing but now Koji will complain about duplicate NVRs when you try to build the package for multiple distros.
  • We will provide a tool to verify that the packages tagged for each distro are in sync.

* unless you really know what you are doing

External RPM Repositories

EPEL & Globus

We tell our users that they have to use/enable the EPEL repository. We also tell our uses that they use the yum-priorities plugin to make sure that our packages override packages from EPEL, in case we modify any packages.

We currently ship Globus ourselves, though our packages are based on the SRPMs that the Globus team provides (that were originally built for EPEL). This is because at the time of this writing, EPEL was shipping a version of Globus 5.0 that we considered unusable, so we started shipping 5.2 prereleases instead.

EMI

EMI 1.0 is very new. We expect to use many of their packages, but we will prefer to copy the packages to our repository instead of relying directly on the EMI repository. This will allow us to control the versions of the critical grid software our stakeholders need.

If you report bugs to EMI, you can use GGUS. Please note a few things:

  1. Please copy Alain on the ticket so he can be aware of what's happening with all of our bug reports to EMI.
  2. If you think the ticket affects other people in the group, please send a note about it to osg-software@opensciencegrid.org.
  3. You should put "EMI" in the short description and something like "Please assign to EMI Release Management" in the description.

Topic revision: r45 - 27 Jan 2012 - 20:34:03 - MatyasSelmeci
Hello, TWikiGuest
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..