From: Peter Palfrader Date: Mon, 8 Jan 2018 10:07:06 +0000 (+0100) Subject: Merge branch 'master' into staging X-Git-Url: https://git.adam-barratt.org.uk/?a=commitdiff_plain;h=1eb059797393099ff8bbfd961718a0fd46f18379;p=mirror%2Fdsa-puppet.git Merge branch 'master' into staging * master: samhain ignore /etc/quagga/bgpd.conf and /etc/quagga/zebra.conf Add zebra and bgpd facters Fix a typo in previous commit Always enable page table isolation on stretch/amd64 This sudo is no longer needed Add the pre-commit hook from handel into the repo, so it is easier to use Allow adayevskaya to ssh trigger puppetmaster/handel remove obsolete entry from .gitignore Fix ProxyPassReverse Do the same for the git user Fix linger setup to use variable Add webhook things for Ganneff based on his patch let sallinen read sibelius backups add sallinen to pg server group give sallinen pg access to sibelius Redirect linux security updates to security-cdn on all mirrors And a homedir for the webhook user give gitdoadm sudo to salsa-webhook Do the linux redirect to security-cdn dance on setoguchi Two more packages for salsa Tweak shell quoting per weasel's suggestion Delete temp dir in update-fastly-ips script Use separate static component for planet.d.n vhost (rt#7018) Add planet.d.n static component (rt#7018) Add redirections for the Debian Policy manual (now in single page) merge nagios-wraps crontab into dsa-puppet-stuff move absent cron.d files to one-line statements to make grepping easier fix weblog provider fragement Move crontab weblog-provider into dsa-puppet-stuff Move crontab static-mirror into dsa-puppet-stuff Move crontab pg base backup into dsa-puppet-stuff Move crontab dchroot update into dsa-puppet-stuff Move crontab geodns boot into dsa-puppet-stuff Move crontab crazy multipath into dsa-puppet-stuff Move crontab exim virtualdomains into dsa-puppet-stuff remove stray punctuation Move crontab buildd into dsa-puppet-stuff Move crontab bacula-storage into dsa-puppet-stuff Move crontab bacula-director into dsa-puppet-stuff Move puppet-export-scheduled-shutdown into dsa-puppet-stuff move cron.d/puppet-update-fastly-ips into dsa-puppet-stuff set MAILTO=root in dsa-puppet-stuff header move munin-master crontab to dsa-puppet-stuff restart hp-health on bm-bl* if needed re-add lost cronjob line Make dsa-puppet-stuff a concat bacula-unlink-removed-volumes: do not remove .nobackup files After rotating log files, sleep a few seconds disable unprivileged BPF loading Use ftp.uk.debian.org instead of mirror.bytemark.co.uk at ARM Retire planeta.debian.net ServerAlias for planet.d.o Use https instead of http for some redirects Ignore unhealthy hosts for deciding which mirrors are the newest Handle ConnectTimeout the same as ReadTimeout for mirror-health Add lower-case redirects for all the top-level upper-case URLs on www.d.o Redirect debian.org/bugs to /Bugs (Closes: #883946) The TCP BBR module is only available on stretch and later Set referrer-policy to same-origin on debtags.d.o Enable TCP BBR on a bunch of hosts. Not all for now, but maybe we should. (re: RT#6990) Put vhost for signup.salsa.debian.org on the salsa host (re: RT#7008) Put cert for signup.salsa.debian.org on the salsa host (re: RT#7008) Install packages for salsa registration app (re: RT#7008) Fixup sources.d.n setup Add sources.d.n static vhost with redirect to sources.d.o Make redirects from {volatile,women}.d.o to www.d.o use https Remove dak's sudoers entry for code signing Add planet_master role and planet-master.d.o vhost And fix a pronoun Add comment to sudoers Allow sudo to runmirrors in the current location Make sudo set a special path for calls as archvsync user Remove philp from experimental_apache Redirect old children-distros page to new derivatives page include with the correct name set vm dirty values do extra grub for grnet-node01,grnet-node02 set elevator=deadline at grnet Add kantuser Add kantuser volume at ubc set mode of /etc/default/locale to a+r Add extra netnod servers to ferm named: add more dnsnode server ACLs Remove /etc/init.d sudo to spamassassin and amavis - listmaster can go via service(8) give %list access to service {spamassassin,amavis} {reload,restart,stop,start} sudo on listhosts: give list group access to postcat as postfix Once more with feeling Enable wsgi-py3 for tracker remove ticharich from experimental_apache group Reduce WAL retention from 21 to 14 days for bmdb1/debsources manpages: force content-type to text/plain for non-html .gz files Distinguish ssl/nossl access logs for planet-backend Revert "install newer version of devscripts" Fix planet-backend.d.o add ssl vhost for planet-backend Fix http://www.debian.org picconi and pkgmirror-csail are on stretch, remove from experimental_apache Fixup sources.d.o config Rotate fastly syslogs Reload syslog-ng after daemon.log rotation to prevent cron spam seger's dak db is on postgresql 9.6 Disable ftp:// on security-master Turn off ftp:// on ftp.debian.org Turn off ftp:// on security mirrors Add debsources role for sources.d.o serial options that work on clementi hopefully will also work on czerny Do not do serial on manda-hosts just yet puppet managed grub on celemtni, czerny Disable OCSP stapling on the default vhost Further restrict access to cgi-bin on http://popcon.d.o Remove unneeded bits from the http popcon vhost, and enable HSTS Import popcon.d.o apache vhost config Add ssl key/cert for popcon redirect www.d.o to https www: Split out onion hostname Split common-www.d.o into common-www.d.o and -inner Add a comment remove obsolete ServerAlias entries for www-other redirect www-other (i.e. debian.org, www.CC.d.o, www.d.CC) to https on www.debian.org now reject package file names that could be used to install local files. Issue reported by Julian Andres Klode. Cleanup experimental_apache role remove custom casulana rules RT#6923 - More users and groups Add mail filters for some aliases (rt#6227) always a typo prune ssh ACLs for luca add more casulana rules for br1 add masquerade rules for casulana virtual machines undo casulana custom roles fix up the custom cloud-admins rule custom rule for cloud-builds on casaluna add sudo access to group cloud-builds bmdb1 main cluster is back on timeline 1 Ensure mirror-health is restarted after the daemon-reload Drop klecker from ftp.d.o mirror-health checking mask sys-kernel-debug-tracing.mount and sys-kernel-debug.mount Add a systemd::mask Fix octal number in python script to it compiles Revert "Use RedirectPermanent instead of RewriteRule" Use RedirectPermanent instead of RewriteRule Better debian-ports.org/debian-cd redirection Drop remaining debian-ports-cd code Redirect ftp.ports.debian.org/debian-ports-cd to cdimage Update debian-ports.org/debian-cd redirection to cdimage.d.do Format weekly stunnel restart script nicer Have gobby reload its config when we change its ssl cert remove auto-cert and auto-clientcert symlinks from fileserver path fix one path Try to replace file access to auto-ca things with templates Add syncproxy addresses to ssh whitelist And more move things move ssl/clientcerts to ssl/auto-clientcerts move exim/certs to ssl/auto-certs Stop hardcoding /srv/puppet.debian.org/from-letsencrypt/ all over the place remove from-letsencrypt symlink from fileserver path Make db key loaded from a template Make gobby key loaded from a template Add tls key for gobby server Use restrict authorized_keys option for geodns remove unused modules/ssl/files/chains with the GANDI chains Use a template to get more of the from-letsencrypt certs and keys, and no longer support getting certs and chains from files/{servicecerts,chains} (which no longer holds any DSA certs) Restrict ssh to mirrors Fix ssl key template Use a template to get from-letsencrypt cert key, and no longer support getting keys from files/keys (which no longer exists anyhow) bmdb1/main on postgresql 9.6 don't spawn a shell in create-onionbalance-config Make sure onionbalance private keys are group-readable bmdb1's debsources cluster is on 9.6 Add debconf17.dc.o static component Consider ourselves unhealthy if fetching from localhost fails Use max instead of if to get biggest timestamp stop hardcoding danzi in postgres-make-base-backup Use postgres::backup_source for danzi's main pg cluster add danzi/debconf pg cluster as backup source .onion for debconf18.dc.o At least -current-live is expected to exist Add debconf18.dc.o static component serial on klecker mirror-health: have systemd restart the service when it dies mirror-health: add shutdown check mirror-health: move up-to-date check to a function Add a tiny bit of error handling for health checking Make apache listen for debian.backend.mirrors.debian.org on loopback too Add missing domain component, now with 100% more valid names Use service-looking names instead… Use hard coded list for what hosts to check Notify service when the underlying file changes or the service changes Correct path to health check status and allow access to it Make sure to start the mirror-health service Fix logic in healthy/unhealthy Status code is an int Correct variable name in systemd unit Fix name of variable (it is a timestamp, not a zone) and log a bit more Disallow redirects for health checking DynamicUser and python don't mix, apply by hand instead Format the list of hosts to check properly Use define rather than class to make this work properly Add health checking support for mirrors install newer version of devscripts fixup ferm rule for danzi update ferm rules for postgresql@danzi sudo: debconf-web group can become debconf-web user add debussy add debussy volume at ubc danzi pg is now 9.6 Revert "redirect linux updates to security-cdn" Be more defensive with mv and use --no-target-directory Refactor logging. Better python, i.e., python that actually does what it should Do not hardcode debian specifics in staticsync scripts, make them use a conffile Quote COMPONENT computation in static-mirror-run Revert "Restrict ssh to anycast and static mirrors" Restrict ssh to anycast and static mirrors Actually add the template Try pages.debian.net apache And reload networking when we add new addresses Try different filename, and set preferred-lifetime Add pages.d.n ip address Looks like bmdb1/wannabuild is back to timeline 1 wannabuild cluster on pg 9.6 fasolo on postgresql 9.6 print VSS after service restart. only restart when using more than 6g provide full path to service restart multipath on bytemark blades fix modes on qemu-system-aarch64-wrapper serial on lobos/villa serial on mirror-isc/-umn serial on byrd serial on grnet/csail node 0[12] aagaard-> conova-node01 acker -> conova-node02 Touch /srv/static.debian.org/.nobackup create /srv/static.debian.org/master static-masters create ~staticsync/static-master -> /srv/static.debian.org on static-masters And remove second /srv/static.debian.org dir from static-mirror class Move mirror-master to static-master-grnet-01 from dillon fix class Create /srv/static.debian.org on static mirrors and masters (not on sources) Move /usr/local/bin/static-update-component from static_source to statice_base, and have static_mirror include static_base instead of static_source Add static-master-grnet-01 as a static-master Do not do regex fo on variables that might not be defined yet Set /etc/environment and /etc/default/locale with puppet instead of in new-machine howto Set root alias via samhain syntax fix Move samhain_recipients to hiera Install userdir-ldap Install debian.org-recommended Set grub config on mirror-isc Add slapd service definition Restart slapd on TLS cert renew Restart repro when the sip-ws TLS cert is renewed redirect linux updates to security-cdn Put mirror-master only on klecker and mirror-isc install python-requests on salsa Add buildd to paths we facter Add debian-buildd to syncproxy rsyncd exim: treat Subject as a single line during regexp match for RT Make debian-buildd tree available over rsync for syncproxies add ruby-ldap to salsa Revert "disable different paths on mirror-conova for now" Don't set grub_do_nopat or grub_do_extra unless grub_manage is set disable different paths on mirror-conova for now mirror-conova: move syncproxy to default paths, move debian mirrors to public-* paths make a hiera setting for mirror base directory (/srv/mirrors) flatten hiera role_config/syncproxy/mirror_basedir_prefix to role_config__syncproxy/mirror_basedir_prefix Make historical mirror rsync template use the archive_root variable historical mirror: make rsyncd.conf a template Make ports mirror template use an @archive_root and @archive_cd_root variable defined in the manifest Make debug mirror template use an @archive_root variable defined in the manifest rsycnd.conf.erb: make future changes less likely to break stuff fix ruby in rsycnd.conf.erb template do not list debian-security archive Make syncproxy mirror basedir configurable in hiera, and use it in all templates. Also make the syncproxy rsync template a loop and fix debian-ports list check in the process complete transition to dedicated admin key s/8080/8181/g update salsa.d.o ProxPassReverse from port 8080 to port 8181 Add arm-conova-02.debian.org (arm64 buildd) ferm: restrict access to all buildds Make last commit work Handle disabling of addresses with extensions correctly salsa: make an /etc/ssh/userkeys/git salsa: require all granted on the document root salsa: needs apache2::rewrite give ProxyPassReverse a path salsa: update apache config remove mpt-status everywhere deploy a basic apache config for salsa enable-linger git Add python-hkdf for salsa Add amdahl.debian.org (arm64 porterbox) switch buxtehude to more puppetized pg backups buildds: add an rsync-security entry to dupload.conf fix filename Add ~/.credentials-manual.yaml to salsa ruby-dev for salsa give gitlab a random key for encrypting its DB grub: don't hardcode the list of hosts with nopat remove duplicate acker entry grub: nopat on villa, once more with feeling grub: nopat on villa villa on stretch, no more experimental_apache Make insecure_ssl a role ssl/ca-global: add certs recently removed from nss to blacklist ssl/ca-global: add ANSSI and CNNIC to the blacklist Fix some paths in the SSL config comments Also apply the ca-global blacklist on godard Disable the usual SSL setup for godard ssl/ca-global: blacklist SPI/StartCom/WoSign CAs Start moving vittoria over to puppetized pg backup firewall: Start moving vittoria over to puppetized pg backup remove temporary dc17 access to vittoria Start moving vittoria over to puppetized pg backup Maintain /etc/nagios/dsa-check-backuppg.conf with puppet use ttyS1 on storace also in grub use ttyS1 on storace rsync-ssh-wrap: also allow uploads to SecurityUploadQueue vsftp::site wants a root parameter, even when disabling it remove ftp_upload role from suchon put an ssl cert on salsa add symlink security upload host: /etc/ssh/userkeys/dak should exist security upload ftp server: disallow directory listings and download security upload host: enable ftp Install ansible so the team can deploy their service Add git user to group redis fix service home path make make_base_backups +x Avoid undefined use of $grub_do_ifnames switch salsa db to postgres::backup_cluster manual entries for melartin for fw, authkeys, and make-base-backup should no longer be necessary Start with puppetizing postgres cluster backup configuration. for now, only deal with melartin remove use of "ensure => $servicefiles" with a servicefiles variable we have never defined in this context There is no bugsmaster role anymore. Remove remaining users next step in getting salsa pg backed up actually add pg's sshkeys-manual ship pg backup sshkeys in puppet salsa: allow postgresql connections from backuphosts through firewall pg: put postgres ssh keys onto backup server move roles::postgresql_server to postgres::backup_source add a comment explaining postgresql_server Create .nobackup flag in non-hardcoded datadir salsa: Make sure we use pg 9.6, and listen on * Add salsa-admin@d.o create salsa database with puppet new concat no longer works with source => on jessie hosts. Switch to content => template in the one use of that Update concat Update stdlib newer pg module salsa: more mail setup salsa: set mail username and password salsa: plan to deploy database with puppet, write out credentials to a .yaml file salsa: no yarn handling Add actual postgresl module from puppetlabs Add postgresl module from puppetlabs Start with salsa.debian.org role/module Add godard to salsa.debian.org role --- 1eb059797393099ff8bbfd961718a0fd46f18379