source: trunk/server/doc/upgrade-tips @ 2418

Upgrading Scripts for a new Fedora distribution
===============================================

1. Gather knowledge
-------------------

You should read the Release Notes for all of the intervening
releases.  For example, here are the Fedora 13 release notes:

    http://docs.fedoraproject.org/en-US/Fedora/13/html/Release_Notes/

Because we sometimes skip releases, you should read any skipped
release's report notes.

Example:

    In Fedora 12, i586 was deprecated in favor of i686; this meant
    that any parts of Scripts that referenced i586 explicitly had to
    changed to i686.

2. Update the Scripts build environment
---------------------------------------

A large amount of the Scripts source repository is Fedora Release
specific, so when you are ramping up the new release, you will want
a new branch to do development on, before merging back upon the
official release.  You can do this with:

    svn cp svn://scripts.mit.edu/trunk \
           svn://scripts.mit.edu/branches/fcXX-dev

On the new branch, there are a number of files you will have to
update:

2.1 Mock

Mock needs to be setup for the new environment.  The first thing to do
is to update the Makefile by substituting
s/scripts-fcOLD/scripts-fcNEW/g on the /usr/bin/mock invocations.
After that, you need to go to /etc/mock and create the new cfg file
for the new scripts-fcXX-ARCH configurations (where ARCH is x86_64 and
i386).  You can base the new cfg off of the older version's, however
you will want to make the following changes:

    * Update all references to the old Fedora release to the new
      Fedora release.  This includes root, dist, mirrorlist, baseurl

    * Temporarily disabling the web.mit.edu Scripts RPM repository
      and the local RPM repository by setting enabled=0 (it's there for
      a reason!)  However, the local RPM repository is fairly painless
      to create and will come in handy when you start attempting to
      build packages that have dependencies on other scriptsified
      packages: you can set one up as scripts-build with:

        mkdir ~/mock-local
        createrepo ~/mock-local

3. Rebuild Scripts packages
---------------------------

In order to support specific extra functionality, we have scriptsified
a variety of Fedora packages.  When the base packages get upgrades,
we need to upgrade the scriptsification.  Some of the following topics
are covered in 'package-build-howto', but a new Fedora release tends
to also result in somewhat rarer situations.

As you finish building packages, you'll want to place them somewhere
so they don't get blown away on a successive mock build.  ~/mock-local
is a good choice.  The Mock RPMs will be created in:

    /var/lib/mock/$MOCK_ENV/result/

Here are some of the common troubles you'll have to deal with:

3.1 Spec patches are no longer necessary

When a Fedora release gets EOL'ed, we may continue to backport
patches for CVE's manually.  When we upgrade to a non-EOL'd release,
those patches will generally become unnecessary and can be dropped.

You can drop a modified specfile from the repository simply by
`svn rm`ing:

    * The spec patch in server/fedora/specs,
    * The source code patch in server/common/patches, and
    * The upstream_yum entry in server/fedora/Makefile

If a specfile merely bumps the version field, there may be no extra
patch (this indicates that the maintainer rebuilt the package simply
by manually dropping the new source tarball in rpmbuild/SOURCES,
which is kind of sketchy but works.  See -c 1586 for an example.)

3.2 Spec patches no longer apply

Symptom:

    $ make patch-specs
    patching file openssh.spec
    Hunk #1 succeeded at 74 with fuzz 2 (offset 11 lines).
    Hunk #2 failed at 88.
    Hunk #3 succeeded at 177 (offset 14 lines).
    Hunk #4 succeeded at 270 with fuzz 2 (offset 36 lines).
    1 out of 4 hunks failed--saving rejects to openssh.spec.rej

Fix:

    The main thing to remember is where the generated files live
    they are placed in rpmbuild/SPECS/openssh.spec{.rej,.orig}.
    A workflow for fixing them might look like:

        1. Inspect the rejects file.
        2. As much as possible, manually fix the original diff
           file in /srv/repository/server/fedora/specs
        3. If absolutely necessary, edit the rpmbuild/SPECS/openssh.spec
           file with any final changes (this is dangerous because
           this file is blown away on a successive make)
        4. Generate a new unified diff:
             diff -u openssh.spec.orig openssh.spec > \
                 /srv/repository/server/fedora/specs\openssh.spec.patch

3.3 Mock fails with no error message

Fix: You forgot to add scripts-build to the mock group.  See
     https://bugzilla.redhat.com/show_bug.cgi?id=630791
     [XXX: remove this entry when this bug is fixed]

3.4 Source patches no longer apply

Symptoms:

    Generally, you will see these error messages after Mock starts
    building (if they occur before Mock, that means it's a bug in the
    spec patch, not a source patch that the spec patch references.)

Fix:

    The error message will be from within a schroot that Mock is using.
    As a result, it's not immediately obvious where the files live.

    The easiest approach is to use rpmbuild to manually reapply the
    patches.

        rpmbuild -bp path/to/foo.spec

    If this fails complaining about a dependency, you should install
    the dependency and add it to the Makefile.

    Once you've fixed the patch, you can rerun rpmbuild after running

        make setup

    (This is useful if you can't do a full make due to another mock
    process running.)

4. "Officializing" everything
-----------------------------

web.mit.edu scripts repository (/mit/scripts/rpm-fcXX and
/mit/scripts/rpm-fcXX-testing) needs to be made.  It's quite simple;
all you need to do is copy the RPMs from the build server to there
(probably going through a trusted machine, since you don't want to put
your root tickets on a server.)  When you're done, run `createrepo -d`
on the directory.

Note that if you do a successive rebuild without bumping the Subversion
revision (via a `svn up`), the new package will have the *same* version
and yum will probably insist on using the old cached version.  You can
use `yum clean all` to reset your cache and force yum to get the latest
version.

5. Update fs sysname
--------------------

XXX out of date
Update /etc/sysconfig/openafs with an extra amd64_fedoraX_scripts and
amd64_fedoraX sysname.  The format should be evident from the existing
entries.  [XXX There might be other things you want]

6. Bind to scripts-test
-----------------------

First, make sure no other servers are bound to scripts-test (try ping).
Then, create /etc/sysconfig/network-scripts/ifcfg-eth0:0 based off of
/etc/sysconfig/network-scripts/ifcfg-eth0 but with the scripts-test
IP address 18.181.0.229.  Run `ifup eth0:0` to complete the change.

7. Testing critical infrastructure
----------------------------------

There are some important Scripts maintained applications you should test
and ensure run on the new platform.  They include:

    - http://scripts.mit.edu
    - http://scripts.mit.edu/wiki
    - http://scripts.mit.edu/trac
    - http://scripts.mit.edu/whois/
    - http://pony.scripts.mit.edu

8. Extra stuff
--------------

Fedora occasionally updates the architecture name for 32-bit; the last
such update was in Fedora 12, when i586 became i686.  Fixing this
usually just involves replacing i586 with i686 in the appropriate places
(Makefile, specfiles, /etc/mock configuration).  Note that for
hysterical raisins we still refer to our 32-bit builds as i386.
[XXX: Maybe this should change]

Until we decide that the performance impact is negligible, any new PHP
extensions other than the few we’ve whitelisted should be disabled by
emptying their .ini files in /etc/php.d.

9. Sending announcements
------------------------

Once development work has finished, we need to allow users to test
their websites on the new servers.

    SIPB Internal Testing: Send an email to scripts-team@mit.edu
    and -c sipb notifying them of testing procedure and known
    issues.

    General Testing: