Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge "neo-cluster" branch, remain backwards-compatible to the old config #50

Draft
wants to merge 41 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
159dfeb
Initial commit
lschuermann May 20, 2023
d632dd1
Add initial sns-cluster configuration module
lschuermann May 22, 2023
96694c7
Add semi-automated SNS machine installer + machine templates
lschuermann May 22, 2023
9b28e70
Add SNS62 machine configuration
lschuermann May 22, 2023
e0ac1ff
Support different installer image targets (e.g., kexec tarball)
lschuermann May 22, 2023
2dcd023
sns-cluster-install.sh: fix boot disk id detection
lschuermann May 23, 2023
23f5ed2
sns-cluster-installer.nix: add netboot installer support
lschuermann May 23, 2023
e68377c
sns-cluster-installer.nix: add filesystem support & useful packages
lschuermann May 23, 2023
f379174
sns-cluster/default: set both initialHashedPassword & hashedPassword
lschuermann May 23, 2023
914e98b
sns-cluster/default: add operator ssh keys
lschuermann May 23, 2023
2b17389
sns-cluster/default: add iPXE shell bootloader entry
lschuermann May 23, 2023
81eea93
sns-cluster/default: add useful system utilities
lschuermann May 23, 2023
8d4175a
sns-cluster/default: use the "ondemand" frequency governor by default
lschuermann May 23, 2023
d61a270
sns-cluster: add beta machine-family configuration & template
lschuermann May 23, 2023
13fbfad
machines: add SNS26 (beta) machine
lschuermann May 23, 2023
29a4149
sns-cluster: add WIP alpha machine-family configuration & template
lschuermann May 23, 2023
d8ce710
machines/sns62: fix sns-cluster module import
lschuermann May 23, 2023
f30417d
machines/sns26: add syncoid-based cluster backups
lschuermann May 23, 2023
eb232bc
sns-cluster: set ZFS permissions after setting up users
lschuermann May 23, 2023
e0bcaf2
sns-cluster: allow RAID-1 bootvol on beta, prepare other families
lschuermann May 23, 2023
f1d1224
sns-cluster: enable firewall
lschuermann May 23, 2023
71c0e57
sns-cluster: move cpuFreqGovernor setting to misc section
lschuermann May 23, 2023
f5dced6
sns-cluster: add prometheus node exporter
lschuermann May 23, 2023
809a267
machines/sns26: add prometheus monitoring server
lschuermann May 23, 2023
55f90bc
sns-cluster: enforce contactEmail to be set on users
lschuermann May 23, 2023
fc6fe4a
Add machines from cluster install party on 2023-05-24
lschuermann May 24, 2023
5119378
sns-cluster: add ipmitool to systemPackages
lschuermann May 24, 2023
76648d0
sns-cluster-installer: blacklist md_mod to avoid importing RAID arrays
lschuermann May 24, 2023
dc9d599
Add GPG-encrypted passwords (pass-compliant)
lschuermann May 26, 2023
032d8a3
Move config backups to more appropriately named folder
alevy May 27, 2023
0ff38b3
Example remote config setup, just on sns33
alevy May 27, 2023
2819404
Revert "Example remote config setup, just on sns33"
lschuermann May 29, 2023
587cc53
Revert "Move config backups to more appropriately named folder"
lschuermann May 29, 2023
48bc9f0
machines/sns26: add known-good SNS hosts to backup & monitoring
lschuermann May 29, 2023
271092c
sns-cluster-install.sh: set git clone origin URL & name + email
lschuermann May 29, 2023
64ebee3
Add sns58 machine
lschuermann May 29, 2023
3bcfce7
Add sns49 machine
lschuermann May 29, 2023
fdc84e5
machines/sns26: remote duplicate sns62 from snsHosts
lschuermann May 29, 2023
5ae2f0c
machines/sns49: fix configuration file location
lschuermann May 29, 2023
759ada8
machines/sns26: ensure that ZFS backup target file systems exist
lschuermann May 29, 2023
00c9a92
Merge branch 'neo-cluster', moving old config to `legacy`
lschuermann May 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
result
/configuration.nix
51 changes: 2 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,50 +1,3 @@
# SNS Cluster Configuration
Princeton SNS Group Cluster Configuration
=========================================

## Machine Specs

| Cluster | CPU | RAM | HDD | SSD |
|---------|------------------------|------|-------------|---------|
| Adam | 2x8 2.2GHz Intel Xeon | 64GB | 4x2TB (in RAID) | |
| Alpha | 2x4 2.3GHz AMD Opetron | 8GB | 2x1TB | - |
| Beta | 2x4 2.4GHz Intel Xeon | 12GB | 2x1TB | 150GB |
| Gamma | 2x8 2.2GHz Intel Xeon | 64GB | 1TB | 2x240GB |
| SNS57 | 2x4 2.4GHz Intel Xeon | 64GB | 2x1TB | 150GB |


### Alpha Cluster -- Old SUN Systems

1. sns3.cs.princeton.edu
2. sns4.cs.princeton.edu
3. sns6.cs.princeton.edu
4. sns7.cs.princeton.edu
5. sns8.cs.princeton.edu
6. sns9.cs.princeton.edu
7. sns14.cs.princeton.edu
7. sns15.cs.princeton.edu
8. sns17.cs.princeton.edu
9. sns20.cs.princeton.edu
10. sns21.cs.princeton.edu
11. sns24.cs.princeton.edu

### Beta Cluster -- Remote Workstations

1. sns26.cs.princeton.edu
2. sns29.cs.princeton.edu
3. sns31.cs.princeton.edu
4. sns32.cs.princeton.edu
5. sns33.cs.princeton.edu
6. sns44.cs.princeton.edu
6. sns45.cs.princeton.edu
8. sns49.cs.princeton.edu
9. sns51.cs.princeton.edu
10. sns52.cs.princeton.edu
11. ~~sns55.cs.princeton.edu~~ (needs manual intervention)
12. ~~sns55.cs.princeton.edu~~ (needs manual intervention)

## Gamma Cluster -- Temporary Reservations

1. sns59.cs.princeton.edu

## One-offs

1. sns57.cs.princeton.edu
9 changes: 0 additions & 9 deletions default.nix

This file was deleted.

1 change: 1 addition & 0 deletions default.nix
50 changes: 50 additions & 0 deletions legacy/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# SNS Cluster Configuration

## Machine Specs

| Cluster | CPU | RAM | HDD | SSD |
|---------|------------------------|------|-------------|---------|
| Adam | 2x8 2.2GHz Intel Xeon | 64GB | 4x2TB (in RAID) | |
| Alpha | 2x4 2.3GHz AMD Opetron | 8GB | 2x1TB | - |
| Beta | 2x4 2.4GHz Intel Xeon | 12GB | 2x1TB | 150GB |
| Gamma | 2x8 2.2GHz Intel Xeon | 64GB | 1TB | 2x240GB |
| SNS57 | 2x4 2.4GHz Intel Xeon | 64GB | 2x1TB | 150GB |


### Alpha Cluster -- Old SUN Systems

1. sns3.cs.princeton.edu
2. sns4.cs.princeton.edu
3. sns6.cs.princeton.edu
4. sns7.cs.princeton.edu
5. sns8.cs.princeton.edu
6. sns9.cs.princeton.edu
7. sns14.cs.princeton.edu
7. sns15.cs.princeton.edu
8. sns17.cs.princeton.edu
9. sns20.cs.princeton.edu
10. sns21.cs.princeton.edu
11. sns24.cs.princeton.edu

### Beta Cluster -- Remote Workstations

1. sns26.cs.princeton.edu
2. sns29.cs.princeton.edu
3. sns31.cs.princeton.edu
4. sns32.cs.princeton.edu
5. sns33.cs.princeton.edu
6. sns44.cs.princeton.edu
6. sns45.cs.princeton.edu
8. sns49.cs.princeton.edu
9. sns51.cs.princeton.edu
10. sns52.cs.princeton.edu
11. ~~sns55.cs.princeton.edu~~ (needs manual intervention)
12. ~~sns55.cs.princeton.edu~~ (needs manual intervention)

## Gamma Cluster -- Temporary Reservations

1. sns59.cs.princeton.edu

## One-offs

1. sns57.cs.princeton.edu
9 changes: 9 additions & 0 deletions legacy/default.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
with builtins;
let
utils = import ./utils;
machines = utils.modulesIn ./machines;
baseSNSMachine = n: (import ./machines/common.nix) { hostname = "sns${n}"; };
baseMachines = listToAttrs
(map (n: let ns = toString n; in { name = "sns${ns}"; value = baseSNSMachine ns; })
(genList (n: n + 1) 100));
in baseMachines // machines
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
38 changes: 38 additions & 0 deletions legacy/machines/sns26.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{ config, pkgs, ... }:

let
hostname = "sns26";
common = (import ./common.nix) { hostname = hostname; };
utils = import ../utils;
in {

# Import common configurat for all machines (locale, SSHd, updates...)
imports = [ common ];

# List packages installed in system profile. To search, run:
# $ nix search wget
environment.systemPackages = with pkgs; [
git
];

programs.mosh.enable = true;

networking.firewall.allowedTCPPorts = [
# Frida server
8000
];

virtualisation.docker.enable = true;

users.users.npopescu = {
isNormalUser = true;
extraGroups = [ "wheel" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "nataliepopescu";
};

users.users.leons = {
isNormalUser = true;
extraGroups = [ "wheel" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "lschuermann";
};
}
42 changes: 42 additions & 0 deletions legacy/machines/sns29.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Configured as a workstation for @alevy, primarily for testing stuff on the
# cluster. If this comment is still here after November 2020, you can take over
# this machine with a PR.

{ config, pkgs, ... }:

let
hostname = "sns29";
common = (import ./common.nix) { hostname = hostname; };
utils = import ../utils;
in {

# Import common configurat for all machines (locale, SSHd, updates...)
imports = [ common ];

# List packages installed in system profile. To search, run:
# $ nix search wget
environment.systemPackages = with pkgs; [
];

programs.mosh.enable = true;

users.mutableUsers = false;

users.users.alevy = {
isNormalUser = true;
extraGroups = [ "wheel" "snapfaas" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "alevy";
};

users.users.yue = {
isNormalUser = true;
extraGroups = [ "wheel" "snapfaas" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "tan-yue";
};

users.users.kw22 = {
isNormalUser = true;
extraGroups = [ "snapfaas" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "kw1122";
};
}
39 changes: 39 additions & 0 deletions legacy/machines/sns31.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Configured as a workstation for Jianan
# Current project users include Jianan, and Haoyu.

{ config, pkgs, ... }:

let
hostname = "sns31";
common = (import ./common.nix) { hostname = hostname; };
utils = import ../utils;
in {

# Import common configurat for all machines (locale, SSHd, updates...)
imports = [ common ];

# List packages installed in system profile. To search, run:
# $ nix search wget
environment.systemPackages = with pkgs; [
];

programs.mosh.enable = true;

users.users.haoyu = {
isNormalUser = true;
extraGroups = [ "wheel" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "Lei-Houjyu";
};

users.users.jiananl= {
isNormalUser = true;
extraGroups = [ "wheel" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "amberlu";
};

users.users.araina= {
isNormalUser = true;
extraGroups = [ "wheel" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "ashwiniraina";
};
}
27 changes: 27 additions & 0 deletions legacy/machines/sns32.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{ config, pkgs, ... }:

let
hostname = "sns32";
common = (import ./common.nix) { hostname = hostname; };
utils = import ../utils;
in {

# Import common configurat for all machines (locale, SSHd, updates...)
imports = [ common ];

# List packages installed in system profile. To search, run:
# $ nix search wget
environment.systemPackages = with pkgs; [
git
];

programs.mosh.enable = true;

virtualisation.docker.enable = true;

users.users.scaspin = {
isNormalUser = true;
extraGroups = [ "wheel" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "scaspin";
};
}
66 changes: 66 additions & 0 deletions legacy/machines/sns33.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Configured as a workstation for @lei

{ config, pkgs, ... }:

let
hostname = "sns33";
common = (import ./common.nix) { hostname = hostname; };
utils = import ../utils;
kubeMasterIP = "10.1.1.2";
kubeMasterHostname = "api.kube";
kubeMasterAPIServerPort = 6443;
in {

# Import common configurat for all machines (locale, SSHd, updates...)
imports = [ common ];

networking.extraHosts = "${kubeMasterIP} ${kubeMasterHostname}";

services.kubernetes = {
roles = ["master" "node"];
masterAddress = kubeMasterHostname;
apiserverAddress = "https://${kubeMasterHostname}:${toString kubeMasterAPIServerPort}";
easyCerts = true;
apiserver = {
securePort = kubeMasterAPIServerPort;
advertiseAddress = kubeMasterIP;
};

# use coredns
addons.dns.enable = true;

# needed if you use swap
kubelet.extraOpts = "--fail-swap-on=false";
};

# List packages installed in system profile. To search, run:
# $ nix search wget
environment.systemPackages = with pkgs; [
git zip unzip
vim tmux wget docker-compose kubectl kompose kubernetes helm
];

programs.mosh.enable = true;

virtualisation.docker.enable = true;

users.mutableUsers = false;

users.users.lei = {
isNormalUser = true;
extraGroups = [ "wheel" "docker" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "geraldleizhang";
};

users.users.leochanj = {
isNormalUser = true;
extraGroups = [ "wheel" "docker" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "leochanj105";
};

users.users.leons = {
isNormalUser = true;
extraGroups = [ "wheel" ];
openssh.authorizedKeys.keys = utils.githubSSHKeys "lschuermann";
};
}
File renamed without changes.
Loading