instructions.txt

dtmrepo

dtmrepo is a mrepo replacement. The main difference between the two is that mrepo uses RHN classic subscription to synchronise with Red Hat while dtmrepo uses subscription-manager. They handle non-RHN repos almost the same way so http, ftp and file-based repos should work with dtmrepo.

I. Prerequisites

Unlike mrepo, dtmrepo relies heavily on external yum utilities which means that repositories must be properly configured before they can be utilised. It is not enough that you define an epel configuration, you need to make sure that the epel yum repo is also configured. This restriction is further explained in the Configuration section below.

If you need to sync with Red Hat you will require a valid subscription. You will also need to install the subscription-manager package.

II. Installation

The syntax for installer.sh is as follows:
    installer.sh <installation dir> [repo dir] [subscribe]

Run the installer.sh script and specify the target installation directory, for example:
    ./installer.sh /usr/local/dtmrepo

Optionally, you could also specify a separate directory where you want dtmrepo to store the packages:
    ./installer.sh /usr/local/dtmrepo /var/repos

You could also automatically subscribe to Red Hat's subscription manager by passing the "subscribe" parameter:
    ./installer.sh /usr/local/dtmrepo /var/repos subscribe

The last example will run "dtmrepo -i" as described in the Configuration section below.

III. Configuration

If you wish to synchronise Red Hat packages, make sure that your dtmrepo server is registered and subscribed. Set "subscribed = 1" in dtmrepo.conf then perform the following:

    dtmrepo -i

This will register and initialise dtmrepo. During registration it will ask for your Red Hat credentials and attempt to subscribe your system using subscription-manager. It will also configure /etc/rhsm/rhsm.conf such that dtmrepo will internally manage the yum repository configuration files.

    The main configuration file for dtmrepo is INSTALLDIR/etc/dtmrepo.conf. Sample settings:
    ================================
    [global]
    rootdir = /var/dtmrepo
    confdir = /etc/dtmrepo.conf.d
    yumconf = /etc/dtmrepo-yum.conf
    keep = 3
    arch = x86_64
    subscribed = 1
    protect = java,firefox
    protectrepo = myrepo,foo*
    freezerepo = os
    ================================

    At the moment there is just one section called [global]. This section is required.
        rootdir – this is the where packages are stored. Each distro will have its own subdirectory.
        confdir -- this is where distro specific configuration files are located
        yumconf -- custom yum configuration for dtmrepo
        keep -- the number of package versions to keep. Setting "keep = 0" keeps all versions. This is probably the most sought-after feature that is missing in mrepo.
        arch -- default distro architecture. This can be overridden in the individual distro configuration files defined under confdir.
        subscribed -- the system is subscribed and wants to sync from Red Hat
        protect -- comma separated list of package patterns that will not be deleted even if they are old; patterns are compatible with grep
        protectrepo -- comma separated list of repo patterns to protect. All packages in the matching repos will be kept. Patterns are compatible with grep.
        freezerepo -- comma separated list of repos to freeze. Frozen repos will not be modified by any operation.

    Sample dtmrepo-yum.conf:
    ===============================
    [main]
    cachedir=/var/cache/dtmrepo/
    logfile=/var/log/dtmrepo.log
    obsoletes=1
    plugins=1
    reposdir=/etc/dtmrepo.repos.d
    ===============================


    The INSTALLDIR/etc/dtmrepo.conf.d directory

    This directory contains distro-specific configuration files. The configuration files must have an extension of .conf. The filename defines the distro name. For example, the distro rhel5s-x86_64 will have a configuration file called INSTALLDIR/etc/dtmrepo.conf.d/rhel5s-x86_64.conf. The distro name also corresponds to the directory REPODIR/distro where repo packages are saved during synchronisation.

    Below is a sample distro.conf configuration file:
    ====================================================
    [control]
    releasever = 5Server
    arch = x86_64,i686
    keep = 2
    protectrepo = os,optional

    [repos]
    os = rhel-5-server-rpms
    optional = rhel-5-server-optional-rpms
    supplementary = rhel-5-server-supplementary-rpms
    epel = epel-5-x86_64
    foo = local:/var/dtmrepo/foo/el5/x86_64
    misc = swalk:some-spacewalk-channel
    dev = mirror:dev-el5
    ====================================================

    There are two sections: [control] and [repos]. The [control] section may override some of the global configuration options. In the example above it overrides the the number of package versions to keep. The [repos] section defines the different repositories that will be synced. The format is "targetdir = repoid". The targetdir corresponds to the directory REPDODIR/rhel5s-x86_64/targetdir and repoid is a real yum repository that is preconfigured under INSTALLDIR/etc/dtmrepo.repos.d/.

    IMPORTANT: For all distros that rely on Red Hat subscription-manager and Spacewalk, the "releasever" and "arch" variables NEED to be defined.

    At the moment there are two types of repoid that is accepted by dtmrepo. The first type is the most common where the repoid is a real yum repo. In the example above, the line epel = epel-5-x86_64 implies that epel-5-x86_64 is a yum repo defined under INSTALLDIR/etc/dtmrepo.repos.d/ like this:

    ===============================================================================
    [epel-5-x86_64]
    name=epel-5
    mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-5&arch=x86_64
    failovermethod=priority
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-5
    ===============================================================================

    Put the repo configurations under the directory specified in "reposdir" of the custom yum configuration specified in "yumconf" of INSTALLDIR/etc/dtmrepo.conf.

    If the repoid is prefixed by "mirror:" then an update operation will mirror that repo. Local packages that are no longer present in the source repo will be deleted.

    The second type of repoid is not a real repository but just a collection of packages stored in a directory somewhere. This is specified by the identifier local:. In the example above, /var/dtmrepo/foo/el5/x86_64 need not contain a repodata. If the directory already exists where it's supposed to be under repodata (i.e. /var/dtmrepo/rhel5s-x86_64/foo) then it is left untouched otherwise a symlink that points to the source directory is created under the said repodata.

    The third type of repoid is for Spacewalk channels. For this to work, the dtmrepo server needs to be registered with Spacewalk obviously. If the distro matches the distro of the dtmrepo server then the "swalk" marker is not needed. If the distro is a "fake" distro, i.e. you are fetching EL7 packages from a EL6 dtmrepo server, then you need to register first (see Section IV) and the "swalk" marker is required.

IV. Spacewalk Registration

    This feature requires that the rhn-client-tools python modules are installed. These modules are installed under "/usr/share/rhn/up2date_client".

    The main challenge with Spacewalk is that when you register a system, it would automatically figure out the OS distro of the system and then attach the system to a base channel. Channels that are outside of this base channel are inaccessible to the system.

    For the dtmrepo server to fetch packages from other base channels we need to "fake" its distro. This is performed by doing "dtmrepo -e -d <distro>".
    Before trying to register with Spacewalk, make sure that you have your distro.conf file already configured. Below is a sample /etc/dtmrepo.conf.d/rhel7s-x86_64.conf file:

    ==============================================
    [control]
    releasever = 7Server
    arch = x86_64,i686
    keep = 3

    [repos]
    base-os = swalk:el7-base-x86_64
    =============================================

    Make sure that "releasever" matches the base channel that you want to register with. To associate this fake distro with a Spacewalk base channel, it is recommended that you get the activation key (14de4a3a0b6abcd768b34abfa4b11111 in the example below) for that base channel then execute:

    dtmrepo -e -d rhel7s-x86_64 -k 14de4a3a0b6abcd768b34abfa4b11111

    If an activation key is not supplied it will prompt for your Spacewalk username and password instead.
    This fake distro will be automatically have a profile name of "$HOSTNAME-$DISTRO-dtmrepo". Hopefully, the registration will succeed. Log in to your Spacewalk server and associate this profile with other child channels if required. You can then use these channels in the [repos] section prefixed with "swalk":

    ======================================
    optional = swalk:el7-optional-channel
    custom = swalk:el7-custom-channel
    foobar = swalk:foobar-channel
    ======================================

    To start fetching packages for the Spacewalk channels, do the usual thing:

    dtmrepo -ugv -r base-os -d rhel7-x86_64

V. Usage

dtmrepo accepts several commandline options namely:

    -a : sync all packages. Useful when you are starting to build your repo server. By default only newer packages are synced
    -c : clean up dtmrepo. Performs sanity check and disables all dtmrepo-controlled repos to avoid future errors. This does not affect the functionality of reposync at all.
    -d : comma-separated list of distros to process. If not specified then all distros are processed
    -e : enroll/register with Spacewalk. This requires -d option. Register one distro at a time for now please.
    -f : force the operation. Normally used in conjunction with -g option. Useful when you are suspecting a corrupted repodata.
    -k : activation key to use when registering with Spacewalk. Implies -e option
    -r : comma-separated list of repos to process. If not specified then all repos are processed
    -u : sync packages
    -g : generate repodata
    -m : mirror; implies -a, -u and -g
    -s : download selected packages and all dependencies. This option disables -u and temporarily protects the entire repo so that the packages are not deleted during repodata creation. Has no effect on "local:" repos. Automatically performs -g.
    -p : purge undeclared repo directories
    -f : force the operation. Normally, dtmrepo will not attempt to generate new repodata if it's newer than any package in the repo.
    -x : autofix. Red Hat subscription-manager automatically updates the subscription certificate. This option automatically updates master.repo with the new cert.
    -i : register and initialise the system. Only relevant if you wish to sync with Red Hat. Use it if the sanity check reports any error. It is smart enough to fix only the errors encountered.
    -v : run in verbose mode. Useful when dtmrepo seems to ignore you.

VI. Usage Examples

    dtmrepo -?
        show help

    dtmrepo -c
        perform sanity checks and clean up

    dtmrepo -i
        register and initialise the system

    dtmrepo -augfv
        sync all packages and force the generation of repodata for all repos and all distros and show me what you're doing. Useful during initial build of a dtmrepo server.

    dtmrepo -ug -r epel
        sync and generate repo data for epel on all distros

    dtmrepo -mv -r epel
        mirror the epel repo for all distros

    dtmrepo -u -r foo,bar
        sync foo and bar repos on all distros

    dtmrepo -s foo,bar -r foobar
        download packages foo and bar including all their dependencies from repo foobar

    dtmrepo -u -d rhel5s
        sync only the rhel5s distro

    dtmrepo -u -d rhel5s,rhel6s
        sync all repos on rhel5s and rhel6s distros

    dtmrepo -fg -r foo -d rhel5s
        generate repodata for the foo repo in the rhel5s distro

    dtmrepo -xug
        a useful command for automatic updates via cron

    dtmrepo -p -d rhel5s
        purge all repo directories that are not declared in the rhel5s distro config file

    dtmrepo -p
        purge all repo directories that are not declared in the distro config files

VII. Tips

    1. Make sure that all the repo configurations in /etc/yum.repos.d are disabled (enabled=0) except for the self referencing repo mentioned in #2 below.

    2. To update the dtmrepo server itself, create a self-referencing repo configuration under /etc/yum.repos.d. Example:

        [self-os]
        name=self-os
        baseurl=file:///var/dtmrepo/rhel6s-x86_64/os
        gpgcheck=0

    3. If you are synchronising from a repo that changes frequently you may want to disable http caching in INSTALLDIR/etc/dtmrepo-yum.conf. This allows dtmrepo to immediately detect and sync new packages.

        http_caching=none


    4. To sync multiple distros, create separate configurations under INSTALLDIR/etc/dtmrepo.conf.d. Syncing different versions of RHEL is a bit tricky because of the subscription manager certificates.
       You will need to have access to registered machines for each version that you want to sync. Grab the certificates from /etc/pki/product from each machine and copy (rename as necessary) them into
       the /etc/pki/product of the dtmrepo server. Here's how it looks like in our dtmrepo server:

       [me@dtmrepo product]$ pwd
       /etc/pki/product
       [me@dtmrepo product]$ ls
       dts.pem  el5.pem  el6.pem  el7.pem  rhscl.pem

       Make sure that dtmrepo.conf has "subscribed = 1". If the dtmrepo server has already been initialised, just delete /etc/yum.repos.d/redhat-dtmrepo.repo and run "dtmrepo -x", otherwise initialise with "dtmrepo -i".

       Create separate configurations for each distro you want to sync. Here's our dtmrepo.conf.d:

       [me@dtmrepo dtmrepo.conf.d]$ ls
       centos5-x86_64.conf  centos7-x86_64.conf  oel6s-x86_64.conf  rhel5s-i386.conf  rhel5s-x86_64.conf  rhel6s-x86_64.conf  rhel7s-x86_64.conf

       Some tidbits of the distro configs:

        [me@dtmrepo dtmrepo.conf.d]$ head rhel6s-x86_64.conf
        [control]
        arch = x86_64
        releasever = 6Server
        protectrepo = datastax

        [repos]
        os = rhel-6-server-rpms
        oracle-java = rhel-6-server-thirdparty-oracle-java-rpms
        optional = rhel-6-server-optional-rpms
        supplementary = rhel-6-server-supplementary-rpms


        [me@dtmrepo dtmrepo.conf.d]$ head rhel7s-x86_64.conf
        [control]
        arch = x86_64
        releasever = 7Server

        [repos]
        os = rhel-7-server-rpms
        oracle-java = rhel-7-server-thirdparty-oracle-java-rpms
        optional = rhel-7-server-optional-rpms
        supplementary = rhel-7-server-supplementary-rpms


VIII. genrepoconfig Tool (obsolete and no longer maintained)

    This tool is useful for generating repo configuration files for other Red Hat releases. By default, it uses INSTALLDIR/etc/dtmrepo.repos.d/redhat-dtmrepo.repo as reference but you can also pass another repo as a fourth parameter. For example, if the dtmrepo server is running EL6 but you want to pull EL5 or EL7 packages as well, you can create repo configs like so:

    genrepoconfig 5 Server x86_64 > /etc/dtmrepo.repos.d/redhat5.repo
    genrepoconfig 7 Server x86_64 > /etc/dtmrepo.repos.d/redhat7.repo
    genrepoconfig 7 Server x86_64 /etc/yum.repos.d/redhat.repo > /etc/dtmrepo.repos.d/redhat7.repo

    From here you can select what repos to sync after creating your custom configs under /etc/dtmrepo.conf.d/.

IX. Limitations

    1. To sync multiple different versions and types of RHEL (i.e. 5Server, 6Workstation, etc...) from one dtmrepo server, you will need to get the correct product certificates from Red Hat otherwise dtmrepo will only sync the RHEL product type that dtmrepo is running in. Once you have the product certificates installed "dtmrepo -x" should automatically take care of the repository configuration under /etc/yum.repos.d/redhat-dtmrepo.repo. Of course this restriction does not affect non-Red Hat repos.

    2. dtmrepo has problems dealing with broken downloads. If your internet connection is faulty and packages are not downloaded completely then the repodata generation may fail. The only way to fix this is to perform another fetch (dtmrepo -u) hoping that the packages will download completely this time. I have encountered instances where because of our pathetic internet link, the downloaded packages are zero-byte files.

    3. There's no ISO mounting support yet, unlike mrepo. I don't find it critical for my use.