Rex configuration management

Posted <2017-09-30 Sat 17:30> by Aaron S. Jackson.

Whenever our lab has bought another GPU machine, the time to make a change, such as allow a user to log in, install a package, or updating the global bashrc, has increased. We have reached a point where I don't want to do that anymore, and ideally, the configuration across all machines should be identical.

One of the things I hated doing most, mentioned above, is adding a user to the allow list in sssd. One solution would be to set up OpenLDAP to shadow the school's directory, and then overlay groups on top of that. However, LDAP is pretty enormous and I don't know much about it. The second most irritating thing is maintaining a mesh of NFS for the local SSDs. While SSDs aren't going to be very fast over a 1GbE NFS link, it is fine for convenience and the latency is still quite low. I've managed to automate most of these tasks with Rex configuration management tool.

For example, updating sssd.conf is as simple as this:

desc "Update SSSD config file";
task "sssd", group => "gpu_general", sub {
    file "/etc/sssd/sssd.conf",
        source => "files/etc/sssd/sssd.conf",
        owner  => "root",
        group  => "root",
        mode   => 600,
        on_change => sub {
            service "sssd" => "restart";
    };
    service "sssd" => ensure => 'running';
    say run "systemctl is-active sssd";
};

Updating /etc/exports is even easier:

desc "Update the NFS exports file";
task "nfs", group => "gpu_general", sub {
    file "/etc/exports",
        source => "files/etc/exports",
        on_change => sub {
            run "exportfs -av"
    };
};

And finally, updating /etc/fstab to mount the mesh of NFS mounts is a bit more complicated, but perfectly fine to manage. I've replaced the actual hostnames with something else.

desc 'Update fstab to include meshed nfs mountpoints';
task 'fstab', group => "gpu_general", sub {
    my %sysinfo = get_system_information;
    my $h = $sysinfo{'hostname'};

    my $para = "nfs  rsize=8192,wsize=8192,timeo=14,intr";
    my $gpu01  = "gpu01:/   /mnt/gpu01   $para";
    my $gpu02  = "gpu02:/   /mnt/gpu02   $para";
    my $gpu03  = "gpu03:/   /mnt/gpu03   $para";
    my $gpu04  = "gpu04:/   /mnt/gpu04   $para";

    say "Hostname: $h";

    if ($h ne "gpu01") {
        run "mkdir -p /mnt/gpu01";
        append_if_no_such_line "/etc/fstab", "$gpu01";
    }
    if ($h ne "gpu02") {
        run "mkdir -p /mnt/gpu02";
        append_if_no_such_line "/etc/fstab", "$gpu02";
    }
    if ($h ne "gpu03") {
        run "mkdir -p /mnt/gpu03";
        append_if_no_such_line "/etc/fstab", "$gpu03";
    }
    if ($h ne "gpu04") {
        run "mkdir -p /mnt/gpu04";
        append_if_no_such_line "/etc/fstab", "$gpu04";
    }

    # create a local link to the root if it is the same machine
    run "[ ! -e /mnt/$h ] && ln -s / /mnt/$h"
};

This one is particularly nice because it updates the fstab, instead of overwriting it, which is almost definitely going to kill things.

Cool!

Wanting to leave a comment?

Comments and feedback are welcome by email (aaron@nospam-aaronsplace.co.uk).

Tags: linux