muug.ca debian mirror 403'ing
The muug.ca/mirror/debian/ mirror is 403'ing on IPv4 and IPv6. Other mirror directories look ok at a glance (fedora, debian-archive) Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/
Not sure how/why that happened, but the directory had execute permission turned off for everyone but the owner. (All the subdirectories seemed fine, as did the other directories under /mirror/.) I've turned execute permission back on for all, so things should be OK now. Hopefully it's not an upstream problem, which will just get reset on the next rsync... (That would be odd, though, given those particular permissions. You'd think that would just prevent all access to the source site.) Gilbert On 22/09/2016 10:07 AM, Theodore Baschak wrote:
The muug.ca/mirror/debian/ <http://muug.ca/mirror/debian/> mirror is 403'ing on IPv4 and IPv6.
Other mirror directories look ok at a glance (fedora, debian-archive)
Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/
-- Manitoba UNIX User Group E-mail: <gedetil@muug.mb.ca> c/o Gilbert E. Detillieux Web: http://www.muug.mb.ca/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609
Thanks for fixing this so quickly Gilbert! Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/
On Sep 22, 2016, at 10:18 AM, Gilbert E. Detillieux <gedetil@cs.umanitoba.ca> wrote:
Not sure how/why that happened, but the directory had execute permission turned off for everyone but the owner. (All the subdirectories seemed fine, as did the other directories under /mirror/.) I've turned execute permission back on for all, so things should be OK now.
Hopefully it's not an upstream problem, which will just get reset on the next rsync... (That would be odd, though, given those particular permissions. You'd think that would just prevent all access to the source site.)
Gilbert
On 22/09/2016 10:07 AM, Theodore Baschak wrote:
The muug.ca/mirror/debian/ <http://muug.ca/mirror/debian/> mirror is 403'ing on IPv4 and IPv6.
Other mirror directories look ok at a glance (fedora, debian-archive)
Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/
-- Manitoba UNIX User Group E-mail: <gedetil@muug.mb.ca> c/o Gilbert E. Detillieux Web: http://www.muug.mb.ca/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609 _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Looks like this happened again, it must be an upstream permissions error thats being propagated thru the mirroring process. I gather from a conversation with Adam that the mirroring hierarchy is rather complex as well. Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/ On Thu, Sep 22, 2016 at 10:18 AM, Gilbert E. Detillieux < gedetil@cs.umanitoba.ca> wrote:
Not sure how/why that happened, but the directory had execute permission turned off for everyone but the owner. (All the subdirectories seemed fine, as did the other directories under /mirror/.) I've turned execute permission back on for all, so things should be OK now.
Hopefully it's not an upstream problem, which will just get reset on the next rsync... (That would be odd, though, given those particular permissions. You'd think that would just prevent all access to the source site.)
Gilbert
On 22/09/2016 10:07 AM, Theodore Baschak wrote:
The muug.ca/mirror/debian/ <http://muug.ca/mirror/debian/> mirror is 403'ing on IPv4 and IPv6.
Other mirror directories look ok at a glance (fedora, debian-archive)
Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/
-- Manitoba UNIX User Group E-mail: <gedetil@muug.mb.ca> c/o Gilbert E. Detillieux Web: http://www.muug.mb.ca/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609 _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
OK, I've reset the execute bits again. I've also checked the upstream site, and it seems OK now, so hopefully this have now been corrected by them and will no longer be an issue. Gilbert On 23/09/2016 10:05 AM, Theodore Baschak wrote:
Looks like this happened again, it must be an upstream permissions error thats being propagated thru the mirroring process. I gather from a conversation with Adam that the mirroring hierarchy is rather complex as well.
Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/
On Thu, Sep 22, 2016 at 10:18 AM, Gilbert E. Detillieux <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>> wrote:
Not sure how/why that happened, but the directory had execute permission turned off for everyone but the owner. (All the subdirectories seemed fine, as did the other directories under /mirror/.) I've turned execute permission back on for all, so things should be OK now.
Hopefully it's not an upstream problem, which will just get reset on the next rsync... (That would be odd, though, given those particular permissions. You'd think that would just prevent all access to the source site.)
Gilbert
On 22/09/2016 10:07 AM, Theodore Baschak wrote:
The muug.ca/mirror/debian/ <http://muug.ca/mirror/debian/> <http://muug.ca/mirror/debian/> mirror is 403'ing on IPv4 and IPv6.
Other mirror directories look ok at a glance (fedora, debian-archive)
-- Manitoba UNIX User Group E-mail: <gedetil@muug.mb.ca> c/o Gilbert E. Detillieux Web: http://www.muug.mb.ca/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609
Looks like this problem is on-going still. I've moved the MUUG.ca mirror down to #2 on my list to keep it from being a show-stopper on my end. Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/ On Fri, Sep 23, 2016 at 10:45 AM, Gilbert E. Detillieux < gedetil@cs.umanitoba.ca> wrote:
OK, I've reset the execute bits again. I've also checked the upstream site, and it seems OK now, so hopefully this have now been corrected by them and will no longer be an issue.
Gilbert
On 23/09/2016 10:05 AM, Theodore Baschak wrote:
Looks like this happened again, it must be an upstream permissions error thats being propagated thru the mirroring process. I gather from a conversation with Adam that the mirroring hierarchy is rather complex as well.
Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/
On Thu, Sep 22, 2016 at 10:18 AM, Gilbert E. Detillieux <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>> wrote:
Not sure how/why that happened, but the directory had execute permission turned off for everyone but the owner. (All the subdirectories seemed fine, as did the other directories under /mirror/.) I've turned execute permission back on for all, so things should be OK now.
Hopefully it's not an upstream problem, which will just get reset on the next rsync... (That would be odd, though, given those particular permissions. You'd think that would just prevent all access to the source site.)
Gilbert
On 22/09/2016 10:07 AM, Theodore Baschak wrote:
The muug.ca/mirror/debian/ <http://muug.ca/mirror/debian/> <http://muug.ca/mirror/debian/> mirror is 403'ing on IPv4 and IPv6.
Other mirror directories look ok at a glance (fedora, debian-archive)
-- Manitoba UNIX User Group E-mail: <gedetil@muug.mb.ca> c/o Gilbert E. Detillieux Web: http://www.muug.mb.ca/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609 _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
On 2016-09-23 Theodore Baschak wrote:
Looks like this happened again, it must be an upstream permissions error thats being propagated thru the mirroring process.
OK, I spent a few minutes and whipped up a perl script (kludge) that should solve this problem until upstream mirrors get fixed. Using inotify I watch for a perm change, check if the perms are not at least 755 and if not I chmod 755. Using inotify should result in basically zero overhead (no polling, no cron-ing, etc). I also put in a systemd wrapper I like to use that allows me to capture all stdout/err to a central log file without having to program it in each script. Not sure if systemd finally fixed this shortcoming, but at least check (maybe 1-2 years ago) they hadn't. (Did I mention I hate systemd yet?) Systemd unit is muug-debian-mirror-dir-perm-kludge. So this little script should restart on every reboot. It's logging all action to /var/log/debian-mirror-dir-perm-kludge.log so we can ensure it's not going wonky and/or pinpoint what rsync run / mirror is screwing it up by comparing times. You can easily test with (root) chmod g-r /ARRAY/mirror/debian then cat /var/log/debian-mirror-dir-perm-kludge.log then ll -d /ARRAY/mirror/debian Note, I suppose there is a possibility that rsync could do something mental like recheck the perm immediately or fight with my script in some way. That should show up in the logs. Perhaps a short (5s?) delay in my script before chmod might alleviate any issue. We'll see if it's required. The script could easily be expanded to watch all mirror dirs, but probably best to only use on an as-needed basis. I was cautious about security but anything like this just adds complexity and thus decreases security. #cat /usr/local/sbin/debian-mirror-dir-perm-kludge #!/usr/bin/perl -w $naughtydir='/ARRAY/mirror/debian'; use Linux::Inotify2; use POSIX qw(strftime); $|=1; printf strftime("%Y-%m-%d %H:%M:%S",localtime)." starting\n"; my $inotify=new Linux::Inotify2 or die "unable to create new inotify object: $!"; $inotify->watch($naughtydir,IN_ATTRIB|IN_ONLYDIR|IN_DONT_FOLLOW, sub { my $e=shift; print "events were lost\n" if $e->IN_Q_OVERFLOW; ($perm)=(stat $naughtydir)[2] or die "cannot stat: $!"; $perm&=07777; # see if perms got wonkyized if (($perm&0755)!=0755) { die "hanky panky" if !-d $naughtydir or -l $naughtydir; printf strftime("%Y-%m-%d %H:%M:%S",localtime)." updated perms (was: %o)\n",$perm; chmod 0755,$naughtydir or die "could not chmod: $!"; } }); 1 while $inotify->poll;
I switched the debian sync over to their main US site (ftp.us.debian.org). Hopefully we don't get the same issues from the US country mirrors. I don't know which is worse, using the round-robin country address, or a specific mirror out of the country list. On the one hand you might get a bad mirror in the list every once and a while. One the other hand, if you pick a single mirror that becomes unreliable, then you have constant problems and you're constantly reconfiguring to swap mirrors. One of the most annoying things to deal with when running an independent mirror is distros that want like 5 levels of mirror tiering and wont give out master access to more than one or two country mirrors. There's virtually no accountability on these country mirrors to provide reliable service, and at the same time any offer of an additional country mirror is generally scoffed at because "the current ones are good enough" even when you're able to provide more space, and more bandwidth. Oh and don't get me started on push mirroring... I also resumed syncing of debian-archive and debian-backports, which are also switched to the US country mirrors now. -- Wyatt Zacharias On Wed, Sep 28, 2016 at 5:53 PM, Trevor Cordes <trevor@tecnopolis.ca> wrote:
On 2016-09-23 Theodore Baschak wrote:
Looks like this happened again, it must be an upstream permissions error thats being propagated thru the mirroring process.
OK, I spent a few minutes and whipped up a perl script (kludge) that should solve this problem until upstream mirrors get fixed. Using inotify I watch for a perm change, check if the perms are not at least 755 and if not I chmod 755. Using inotify should result in basically zero overhead (no polling, no cron-ing, etc).
I also put in a systemd wrapper I like to use that allows me to capture all stdout/err to a central log file without having to program it in each script. Not sure if systemd finally fixed this shortcoming, but at least check (maybe 1-2 years ago) they hadn't. (Did I mention I hate systemd yet?) Systemd unit is muug-debian-mirror-dir-perm-kludge.
So this little script should restart on every reboot.
It's logging all action to /var/log/debian-mirror-dir-perm-kludge.log so we can ensure it's not going wonky and/or pinpoint what rsync run / mirror is screwing it up by comparing times.
You can easily test with (root) chmod g-r /ARRAY/mirror/debian then cat /var/log/debian-mirror-dir-perm-kludge.log then ll -d /ARRAY/mirror/debian
Note, I suppose there is a possibility that rsync could do something mental like recheck the perm immediately or fight with my script in some way. That should show up in the logs. Perhaps a short (5s?) delay in my script before chmod might alleviate any issue. We'll see if it's required.
The script could easily be expanded to watch all mirror dirs, but probably best to only use on an as-needed basis. I was cautious about security but anything like this just adds complexity and thus decreases security.
#cat /usr/local/sbin/debian-mirror-dir-perm-kludge #!/usr/bin/perl -w
$naughtydir='/ARRAY/mirror/debian';
use Linux::Inotify2; use POSIX qw(strftime); $|=1;
printf strftime("%Y-%m-%d %H:%M:%S",localtime)." starting\n";
my $inotify=new Linux::Inotify2 or die "unable to create new inotify object: $!";
$inotify->watch($naughtydir,IN_ATTRIB|IN_ONLYDIR|IN_DONT_FOLLOW, sub { my $e=shift; print "events were lost\n" if $e->IN_Q_OVERFLOW;
($perm)=(stat $naughtydir)[2] or die "cannot stat: $!"; $perm&=07777;
# see if perms got wonkyized if (($perm&0755)!=0755) { die "hanky panky" if !-d $naughtydir or -l $naughtydir; printf strftime("%Y-%m-%d %H:%M:%S",localtime)." updated perms (was: %o)\n",$perm; chmod 0755,$naughtydir or die "could not chmod: $!"; } });
1 while $inotify->poll; _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
participants (4)
-
Gilbert E. Detillieux -
Theodore Baschak -
Trevor Cordes -
Wyatt Zacharias