April 2016 - Roundtable

IBM "clicky" keyboards
by Adam Thompson 13 Oct '16

13 Oct '16

Enough people have wondered/asked/complained to me about this that I'm posting this now as a public service. IBM Model "M" keyboards are still available, *NEW*, today. They are expensive, but they are the original design that you can use as a melee weapon. The catch is that they don't say "IBM" or even "Lexmark" on them. They are available for purchase from the manufacturer, Unicomp, who can be found online at http://www.pckeyboard.com/. You can also find some vintage NIB units from time to time at http://www.clickykeyboards.com/. There are also a number of other manufacturers now making similar, but not quite as good, keyboards. - CVT Inc., the maker of the Avant Stellar (I own two of them), which is the direct descendant of the Northgate Omnikey, seems to have restructured and no trace of their keyboard manufacturing operation can be found online. However, Northgate keyboards are still available new-in-box from (this is a horrible site, beware) http://www.northgate-keyboard-repair.com/. - The Happy Hacking keyboard (now owned by Fujitsu, apparently) is equally comforting to some people despite having a totally different feel. They, and many others, can be had from http://www.elitekeyboards.com/ - Das Keyboard - Anything using Cherry MX Green, Blue, or White keyswitches. The "green" switches apparently are the closest anyone's come yet to emulating the IBM/Lexmark/Unicomp switches... and they can be had in MUCH cheaper keyboards, like the Rosewill RK-9000 (http://www.newegg.ca/Product/Product.aspx?Item=N82E16823201040 and http://techreport.com/review/23405/rosewill-rk-9000-series-mechanical-ke yboards-reviewed). - And there are an increasing number (yes, again, after the big die-off ca. 2009) of speciality manufacturers of "ergonomic" keyboards that are making clicky keyswitches available as an option. One of the better ones is a tiny shop in Ontario, but I can't find the name right now. References: http://en.wikipedia.org/wiki/Model_M_keyboard http://en.wikipedia.org/wiki/Unicomp http://deskthority.net/wiki/Cherry_MX -Adam Thompson athompso(a)athompso.net

7 12

very strange DNS errors
by Trevor Cordes 12 Jun '16

12 Jun '16

Some recurring email bounces have tipped me off to a really strange DNS issue on the boxes I admin. I really need help on this as it's impacting a real customer in a real way: delayed emails. Often, and somewhat randomly, DNS will fail to resolve a domain. Sometimes it will fail to resolve it for >4 hours and trigger a diagnostic bounce from my sendmail. When I run tests at the command line (dig) on the domains in question (the ones I've seen in email bounces) I will often very quickly get a resolve failure, but usually 5s later another dig to the same domain will resolve 100% ok! Every box I am testing on has a similar config with BIND named 9.10 (9.8 on one box) running as the local recursive resolver. /etc/resolv.conf on all is 127.0.0.1. So that means every lookup that isn't cached is going to the root NS's. When it fails to resolve, named.log logs an entry like: 20-Apr-2016 00:37:28.276 query-errors: debug 1: client 127.0.0.1#33971 (artscouncil.mb.ca): query failed (SERVFAIL) for artscouncil.mb.ca/IN/A at query.c:7769 20-Apr-2016 00:37:28.276 query-errors: debug 2: fetch completed at resolver.c:3658 for artscouncil.mb.ca/A in 0.215778: SERVFAIL/success [domain:artscouncil.mb.ca,referral:1,restart:3,qrysent:2,timeout:0,lame:0,neterr:2,badresp:0,adberr:0,findfail:0,valfail:0] For manual tests it only logs one, for the real-life sendmail problem ones, I'll see dozen/hundreds of the same thing trying hour after hour (usually around the sendmail queue retry times). One of my boxes is *extremely* well connected in the US, and while it seems to have errors slightly less often, it still has them. All the rest are on various levels of Shaw or MTS, res or business. This seems to have just started popping up maybe 6 months ago. It feels like it's getting worse. I've setup an easy test, on the actual domains with the most problems: rndc flush dig +short sportmanitoba.ca dig +short gymcan.org dig +short brandoneagles.ca dig +short interactivegym.org dig +short artscouncil.mb.ca rndc flush dig +trace sportmanitoba.ca dig +trace gymcan.org dig +trace brandoneagles.ca dig +trace interactivegym.org dig +trace artscouncil.mb.ca Maybe some others can run those tests on their boxes (but only if you're running BIND as caching resolver, which many/most people won't be). Here's where it gets interesting the +short tests I can get to fail at least 1 of the domains (at random) about 1/8 of the time! On at least 5 different boxes out there! But +trace has never failed once on any box or any domain. It's like +trace does something different, maybe slowing the process down or something, that allows it to always succeed. (Failure for the +short is a missing line in output, +trace you have to look at the domain/ip returned near the bottom.) So, is it just these particular domains?? Something wrong on their (DNS) side? Or is it more domains, not just these? Is there any way to diagnose what exactly is failing? I find it bizarre that *all* of these domains regularly go down for 4+ hours causing an email bounce!?! Or is there something horribly wrong on my BIND caching DNS servers? Perhaps they are slow, and I'm my BINDs are just not waiting long enough. Is there a way to tell BIND to be more patient waiting for DNS packets to come in? Maybe it's something regarding IPv6? (I'm doing this all in IPv4 and have no current interest in 6. And I'm only looking for A records.) I have packet traces of the above sample commands when the lookups fail, but I can't really figure out what it's doing, other than one boatload of traffic for a tiny dns query. I can provide a trace privately on demand if you think you can help. Even more odd, I never seem to have a problem with interactive things like web browsing. If this was a problem with all domains, I should see this in firefox all the time, but I don't. Maybe Firefox doesn't even obey resolv.conf and does its own thing, or retries heavily itself? I also checked to ensure my iptables aren't dropping packets related to this. Lastly, answers of "just use 8.8.8.8" aren't helpful because I also need to handle dynamic local, and in some cases, external DNS (often with multiple views), all in the same BIND/box (and I like uniformity across boxes for ease of admin). Sure, I could try another resolver, but I see no reason BIND can't be made to work, as it has for me for 20 years. And if this is a BIND bug, I want to submit it to help solve it. Thanks guys!

5 14

process accouting, fields meaning?
by Grigory Shamov 27 Apr '16

27 Apr '16

Hi All, Following up on the Adam's old presentation on Linux process accounting. Does anyone know the meaning of fields in the pacct file? The utility dump-acct dumps it in a plaintext lines separated by | . Like this: sh | 0.0| 0.0| 11.0| 0| 0|107328.0| 19284 3258|Tue Apr 26 04:17:07 2016 First three floats are runtimes, then UID|GID; last is the date it ended. Googling brought me "memory" and "io" for the two other fields but I am still at loss. What is memory here? And why io is two numbers (read/write? Which is which?) Does anyone know? (I am trying to avoid reading the sources of dump-acct or system headers). Thanks! -- Grigory Shamov Westgrid/ComputeCanada Site Lead University of Manitoba E2-588 EITC Building, (204) 474-9625

2 2

web access abuse of lisa
by Trevor Cordes 26 Apr '16

26 Apr '16

I noticed stomething strange. Lisa's web hits (as per access logs) were through the roof since around the site changeover date. Weird. I glanced at them and noticed we're getting tons of hits from just 1 user: 216.98.56.20 - - [24/Apr/2016:05:20:11 -0500] "GET /pub/epel/6/x86_64/repodata/repomd.xml HTTP/1.1" 301 332 "-" "urlgrabber/3.9.1 yum/3.2.29" All the same! Of 8434659 total current access.log hits, 8243763 (97.7%) were this same guy! Many per second! For a couple of weeks now (but not before that!). An ip lookup says this is Ubisoft in Montreal. Looks like someone has a misconfig on their box. Should we contact them about fixing this? If have added that IP to an iptables DROP rule on lisa. I just did this now and they sent 49 more hits and then stopped. Their runaway ps must have been looking for success before continuing. I guess I'll leave it in for a while then we can take it out? Looks like this IP has hit the new server but only a few times; i.e. normal looking access. Perhaps when the switchover occurred and the redirects were put in place it made their client go mental... Maybe it never dropped its http connection this whole time! Lastly, fail2ban-server is often in the top 5 ps's in top on lisa, but I don't see any fail2ban rule in iptables? Does it only create a rule once it gets something to put in? I thought it made a blank table that it filled up as needed, not no table at all. Maybe it's not working?

1 0

Re: [RndTbl] [Board] muug.ca hostname?
by Trevor Cordes 17 Apr '16

17 Apr '16

> On April 16, 2016 12:50:43 AM CDT, Trevor Cordes > <trevor(a)tecnopolis.ca> wrote: > >on muug.ca (new server): > > > >#hostname > >muug > > > >This valid? Done for a reason? Should it be muug.ca? On 2016-04-16 Adam Thompson wrote: > Hostname is - and always has - been specified as the bare name, not > the FQDN. Hmm, that's opposite to what I've understood for 25 years, and what I've done on every machine I've ever been responsible for (all of which that have "worked"). But you made me think about it a bit, and why I do it this way. Thinking back I distinctly recall that my personal AIX box in '95 would freak out (like not boot, or take forever to boot) when the hostname wasn't FQDN. Back then it was tricky because I didn't actually own a domain name to use, so I had to make something up. I've done it this way ever since. Thinking more about what is right/wrong *now*, I find: http://serverfault.com/questions/331936/setting-the-hostname-fqdn-or-short-… Which basically hints that it's a) vendor specific (i.e. RH vs Deb), and b) personal preference. No consensus (there or elsewhere) by a mile, but it appears there is a bias towards hostname NOT being FQDN. For me, being an all-Fedora guy, if "RHEL deployment guide" says use FQDN, then that's a good enough excuse for me! And I've never had any issues... I would do a survey of machines I can access, but all I can access are ones I've built, along with lisa.muug.mb.ca, and lisa says: #hostname lisa.muug.mb.ca I'd be interested to see how the computers at, say, the U of M answer that question... (Strangely, much of my computing preference to this very day stems from interesting quirks of U of M's setup from my long days as a student.) > YP/NIS domain, not DNS. Any apps that break this way are broken, > period. The reason I brought it up is that apache on new muug.ca was guessing its host/domainname wrong and doing wonky things with rewrites until I added a ServerName directive to make it explicit. But that might have been a red herring with regards to this whole question. Final note, maybe the One Systemd To Rule Them All will settle this once and for all, as I'm pretty sure hostname is now under systemd purview (gobbled up like everything else)...

2 1

Fwd: [MBIX-Members] MBIX AGM May 4th, 2016
by Adam Thompson 16 Apr '16

16 Apr '16

FYI. -Adam (one of the outgoing directors) -------- Forwarded Message -------- Subject: [MBIX-Members] MBIX AGM May 4th, 2016 Date: Tue, 12 Apr 2016 21:12:56 -0500 From: Bill Reid <billreid(a)shaw.ca> Reply-To: MBIX Members <members(a)lists.mbix.ca> To: members(a)lists.mbix.ca <members(a)lists.mbix.ca> *Manitoba Internet Exchange Annual General Meeting* Wednesday May 4^th , 2016 4 to 6 p.m. AGM to start at 4:30 p.m. King's Head Pub <http://www.kingshead.ca/> Open to all interested in IXs MBIX will be holding its 4^th AGM next month. There are 4 Board of Director positions open for election. Jacques Latour from CIRA will give an update on Canadian IXs. We will have a social time before and after the meeting. Drinks and snacks will be provided by CIRA and MBIX. Look forward to seeing you on May 4^th , Bill

7 12

poor drive/array performance
by Trevor Cordes 14 Apr '16

14 Apr '16

I have an new SSD in a system setup as linux md RAID-1 but with the 2nd drive missing at the moment. No LVM. FS is ext4, very recently created and less than half full. ext4 creation had no tweaks (just default options). md creation had no tweaks. I'm getting extremely bizarre low FS performance. cd /tmp dd if=/dev/zero of=zzzz conv=fdatasync bs=1M count=250 262144000 bytes (262 MB) copied, 21.3281 s, 12.3 MB/s again... 262144000 bytes (262 MB) copied, 35.6929 s, 7.3 MB/s again... 262144000 bytes (262 MB) copied, 26.1413 s, 10.0 MB/s This SSD is rated at 450MB/s write. Sites on the net say I may only realize 100MB/s. I would be overjoyed with 100MB/s. I'm not looking for blazing speed, just not garbage speed. The system had a rust drive previously that had the same problem. When I replaced that, I also recreated the RAID, and the FS. Strangely, this drive is also part of another array for /boot, and that one has 2 drives, one of them spinning rust. And that array/FS is faster!? Huh? : 262144000 bytes (262 MB) copied, 14.6355 s, 17.9 MB/s 262144000 bytes (262 MB) copied, 13.4979 s, 19.4 MB/s 262144000 bytes (262 MB) copied, 9.99149 s, 26.2 MB/s I'm baffled. I really am. hdparm's speed tests show it to behave super fast for reads: Timing buffered disk reads: 1010 MB in 3.00 seconds = 336.32 MB/sec Timing cached reads: 26774 MB in 2.00 seconds = 13412.64 MB/sec I've checked hdparm -i: Model=KINGSTON SV300S37A480G, FwRev=603ABBF0, SerialNo=50026B7763021118 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4 BuffType=unknown, BuffSize=unknown, MaxMultSect=1, MultSect=1 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=937703088 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=yes: unknown setting WriteCache=enabled Drive conforms to: unknown: ATA/ATAPI-2,3,4,5,6,7 udma6 is the correct mode for something like this I do believe. smartctl shows correct sata speeds: ATA Version is: ATA8-ACS, ACS-2 T13/2015-D revision 3 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Kernel is fairly new: 4.4.6-200.fc22.i686+PAE Hardware is server class Intel Xeon E3 with S1200V3RP board, a fairly new model. My only thoughts now are firmware problems, bios setting problems, or cable problem. I need to go onsite to check all three. Anything I am missing? I have lots of extremely similar systems out there *without SSDs* and they all show FS write speeds of no slower than 60MB/s! Weird!!

1 1

Warning: OVH VPS 2016 line missing feature
by Adam Thompson 10 Apr '16

10 Apr '16

OVH has a new line of SSD-backed KVM-hosted VPS servers starting at C$4.99/month, with good performance, decent availability, pretty much an all-round good deal. Except for one little thing... No IPv6 support. At all. According to the tech support rep I just spoke to, it's because the new 2016 lineup runs on OpenStack. I'm a little unclear on how this makes it "impossible" to provide IPv6 functionality, but that's their claim. It's 2016: new IPv4 addresses are no longer available. IPv6 is supported by all major content providers (except Twitter, I think)- but OVH doesn't consider this important, apparently. FYI: their SoYouStart and OVH Dedicated Server lines still provide IPv6 service; it's only the new 2016 VPS system that doesn't. Ironically, the older-generation (2014) system *did* support IPv6, but I can't order that any more. Grrr. -Adam

3 4

Re: [RndTbl] programatically determine if DNS is down or blocked?
by wyatt＠prairieturtle.ca 08 Apr '16

08 Apr '16

dig has parameters to force cache bypass. Daryl ----- Reply message ----- From: "Trevor Cordes" <trevor(a)tecnopolis.ca> To: "Robert Keizer" <robert(a)keizer.ca> Cc: "Continuation of Round Table discussion" <roundtable(a)muug.mb.ca> Subject: [RndTbl] programatically determine if DNS is down or blocked? Date: Wed, Apr 6, 2016 15:41 On 2016-04-06 Robert Keizer wrote: > Why don't you use dig+grep+sed in bash? Ya, that's one of the ideas (dig). It looks like dig will differentiate between named down, vs blocked 53, vs simple invalid domain. I'm trying to see if there are other ways also, perhaps ones that don't require a fork. And the dig will send out real queries to real servers, but I guess it will be tempered by the cache? I'll have to test what dig does in the different failure modes when a) resolving a domain that isn't cached vs b) resolving one that is cached. _______________________________________________ Roundtable mailing list Roundtable(a)muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable

5 5

Fw: [Bug 740188] the --title option is nonfunctional
by Trevor Cordes 07 Apr '16

07 Apr '16

Here's a great example of the snootiness, and complete lack of "getting it", of Gnome/systemd people, and why so many people are at odds with them. Tons of people commented on this bug: how asinine it was that --title was removed. The workarounds are ok if you're interactive *in* the window, but what if you want to launch a bunch of windows from a script, and have each one named? Who are the people being l33t here??? At least (for the first time I can remember when watching the dealings of these people), the "old fart" "unix admins" get their way. Begin forwarded message: Date: Thu, 07 Apr 2016 18:25:09 +0000 From: "gnome-terminal" (GNOME Bugzilla) <bugzilla(a)gnome.org> Subject: [Bug 740188] the --title option is nonfunctional https://bugzilla.gnome.org/show_bug.cgi?id=740188 Debarshi Ray <rishi.is(a)lostca.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rishi.is(a)lostca.se --- Comment #22 from Debarshi Ray <rishi.is(a)lostca.se> --- commit 156b85533f42d8c7e6cf6258ce3e7fe4ec8f1754 Author: Christian Persch <chpe(a)gnome.org> Date: Tue Feb 16 18:59:09 2016 +0100 client: legacy: Reimplement deprecated --title option This reverts commit 0033eca2909ddccf3358c1f089bce05d3473b123. Reimplement the deprecated --title client option. It seems it is absolutely required by all those super l33t unix admins that are above using a lowly command like printf(1) or the even simpler xtermcontrol(1). https://bugzilla.gnome.org/show_bug.cgi?id=740188

1 0