Bodhi Infrastructure SOP

    Bodhi is used by Fedora developers to submit potential package updates for
   releases. From here, bodhi handles all of the dirty work, from sending
   around emails, dealing with Koji, to mashing the repositories.

   Bodhi project Trac: https://fedorahosted.org/bodhi/

   Contents

     * 1 Contact Information
     * 2 Adding a new pending release
     * 3 0-day Release Actions
     * 4 Pushing EPEL updates
     * 5 Doing a production bodhi upgrade
     * 6 Release EOL
     * 7 Troubleshooting and Resolution

          * 7.1 Empty <id/> tags in the updateinfo.xml.gz
          * 7.2 Creating tables at sqlite:/// when trying to use
            postgres!?

Contact Information

   Owner: Fedora Infrastructure Team

   Contact: #fedora-admin

   Persons: lmacken, f13

   Location: Phoenix

   Servers: admin.fedoraproject.org/updates

   Purpose: Push package updates, and handle new submissions.

Adding a new pending release

   The following commands need to be run from a bodhi checkout at the moment.
   The easiest way to do this is to log into `releng02`, and do the
   following.

 $ sudo su - masher
 $ cd bodhi
 $ make shell
 >>> Release(name='F13', long_name='Fedora 13', id_prefix='FEDORA', dist_tag='dist-f13', locked=True)
 >>> from bodhi.jobs import refresh_metrics
 >>> refresh_metrics()

0-day Release Actions

   Going from pending to a proper release in bodhi requires setting the
   Release.locked field to false:

 ssh releng02
 sudo su - masher
 cd bodhi
 make shell
 >>> Release.byName('F15').locked = False
 >>> exit()

   You may also need to disable any pre-release policy defined in the bodhi
   config in puppet.

 ssh puppet01
 cd puppet/modules/bodhi/templates
 git pull
 vi bodhi-prod.cfg.erb

   Then you can safely remove or comment out the release-specific policy:

 # Release status
 # post-beta enforces the 'Beta to Pre Release' policy defined here:
 # https://fedoraproject.org/wiki/Updates_Policy
 f15.status = 'post_beta'
 f15.post_beta.mandatory_days_in_testing = 3
 f15.post_beta.critpath.num_admin_approvals = 1
 f15.post_beta.critpath.min_karma = 2

Pushing EPEL updates

   SSH into the EPEL masher, and run the following command:

 * sudo bodhi --push --push-release=EL-4 --push-release=EL-5 -u $YOUR_USERNAME

   This will display a list of updates that are ready to be pushed. If you
   press enter, it will quit the tool and write out the package lists to
   corresponding files (eg: Testing-EL5, Stable-EL4) which can then be used
   for piping through the signing scripts.

   Once the packages are signed you can run the above `bodhi` command again
   and press `y` to begin the push. You can then keep track of the progress
   by running `tail -f /var/log/bodhi/server.log`. If a problem occurs, you
   can resume the push by running:

 * sudo bodhi --push --push-release=EL-4 --push-release=EL-5 -u $YOUR_USERNAME --resume-push

Doing a production bodhi upgrade

 for app in 01 02 03 04 5 6 07; do
   sudo func "app$app*" call command run 'yum clean metadata'
   sudo func "app$app*" call command run 'yum -y update bodhi-server'
   sudo func "app$app*" call command run '/usr/sbin/puppetd -t'
   curl -I http://app$app/updates/
   sleep 5
 done

Release EOL

   We currently remove releases from bodhi when they reach their End of Life.

   First, we take a database snapshot:

 [masher@releng02 bodhi]$ PYTHONPATH=$(pwd) python bodhi/tools/pickledb.py save

   Then, we can remove all F13 related updates and comments from the
   database:

 [masher@releng02 bodhi]$ PYTHONPATH=$(pwd) python bodhi/tools/rmrelease.py F13

Troubleshooting and Resolution

  Empty <id/> tags in the updateinfo.xml.gz

   Bodhi caches the most recent repodata for all releases in
   `/mnt/koji/mash/updates/*.repodata`. Sometimes, if a push fails or crashes
   for some reason, this can lead to corrupted metadata. So, if you see this
   message:

 * Repodata sanity check failed!
 * updateinfo.xml.gz contains empty ID tags

   You can remove the cached metadata and try resuming the push:

 * rm -fr /mnt/koji/mash/updates/el*.repodata
 * bodhi --push --push-release=EL-4 --push-release=EL-5 -u $YOUR_USERNAME --resume-push

  Creating tables at sqlite:/// when trying to use postgres!?

   When using a third-party TurboGears identity provider or visit manager,
   this may sometimes require you to enable SQLAlchemy in your configuration.
   Since bodhi v0.x uses SQLObject, this can currently confuse `tg-admin`
   initialization configurations, and will require you to temporarily comment
   out the following line when you need to run the `tg-admin sql create`
   database initialization.

 # Required by jsonfas identity provider to run, but confuses tg-admin.
 #sqlalchemy.dburi="sqlite:///"

   More Tips/tricks for working with a Bodhi instance can be found here:

   https://fedorahosted.org/bodhi/wiki/Administration