- Roundtable - muug.ca

HPC sysadmin job at UVic
by Grigory Shamov 23 Aug '17

23 Aug '17

Hi, Not sure if it is the appropriate list. A sysadmin job, University of Victoria. https://uvic.mua.hrdepartment.com/hr/ats/Posting/view/3034 -- Grigory Shamov WestGrid Site Lead / HPC Specialist University of Manitoba E2-588 EITC Building, (204) 474-9625

1 0

Best update problem ever
by Glen Ditchfield 18 Aug '17

18 Aug '17

"When I run apt-get upgrade on my Ubuntu 17.04 machine, sometimes it starts Dwarf Fortress by itself." -- https://askubuntu.com/questions/938606/dwarf-fortress-starting-during-apt-g…

3 2

mail server help
by Trevor Cordes 18 Aug '17

18 Aug '17

I just started getting some weirdness with some email sent to my mail servers. Greylisting "come back later" (451) messages are being interpreted (it appears) by remote MTAs as a 5xx level error, meaning they immediately abort send attempts and bounce the email! I have never seen this before. I've seen it from two separate remote MTAs so far, but from reports I'm getting in, it's happening on more. Here's what I get from umce3cip03.ad.umanitoba.ca: Remote Server returned '554 5.0.0 <[50.71.247.87] #5.0.0 smtp; 5.1.0 - Unknown address error 550-'<foobar(a)tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later' (delivery attempts: 0)>' On my server sendmail log I see: Aug 16 08:43:15 pog sendmail[14379]: v7GDhFav014379: Milter: to=<foobar(a)tecnopolis.ca>, reject=451 4.7.1 Greylisting in action, please come back later So umce3cip03.ad.umanitoba.ca appears to be taking my 451 and turning it into a 500/554/510 permanent error. I also see this from yahoomail, but that's a different situation because yahoo is always considered "broken mta" in milter-greylist and we must whitelist all of their servers... but, the usual/previous symptom was yahoo would keep retrying as normal, just with a different IP each time, thus never passing greylisting. And now yahoo is doing the same thing as umce3cip03.ad.umanitoba.ca above (when I haven't yet whitelisted the particular IP that day, grrr): Sorry, we were unable to deliver your message to the following address. <foo(a)tecnopolis.ca>: 550: <foo(a)tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later So at least 2 MTAs, probably more, are changing 451 greylist into 5xx. Is there some new massive change out there to basically take greylist MTAs as broken? Is there a way to find out what MTA (or outsourced service provider) umce3cip03.ad.umanitoba.ca is using? Perhaps there is just one brand that unilaterally decided on this action? I can find no google hits on any of this. :-( There is an easy workaround/gotcha: if people wait the greylist timeout (usually 2 to 20 minutes) and then resend the same to-from-ip tuple email, it will go through as their server will have been whitelisted in the interim! But that makes it harder to troubleshoot this problem because it then becomes transient. If any MUUGers have problems with MUUG mailing lists bouncing, please resend your email after 1 hr if it hasn't showed up yet (you can check the website mail list archives section) and/or please email me directly with your bounce email message (twice, wait 20 mins!) so I can solve this for both MUUG servers and my own. If this problem is a) permanent/deliberate, and b) widespread, I think that spells the death of greylisting (grrrrr...) and the nearing of "spf strict" enabling.

3 5

git subtree pull error
by Trevor Cordes 17 Aug '17

17 Aug '17

Hi to all gitters who helped me a couple of years back. I'm implementing a subtree using git subtree (better fits my model than submodules), but I'm trying to do my 1st merge/pull (after the initial subtree import) and I'm getting a weird error: git subtree pull -P Nub nub master * branch master -> FETCH_HEAD fatal: refusing to merge unrelated histories I'm conceptually confused as to why it's saying this. The whole subtree command is basically the same as when I setup the remote and did the initial bits. So why aren't the subtree histories related? Or is it confusing the entire project with the subtree? The google hits I get aren't really helping me since they aren't really specific to subtree. If I add a squash: git subtree pull --squash -P Nub nub master Then it works and I can commit. If I take out squash and add --allow-unrelated-histories it barfs because subtree doesn't understand --allow-unrelated-histories, though I guess I could do the separate parts manually and merge with --allow-unrelated-histories. However, in my understanding, I don't want squash nor feel I should require allow-unrelated-histories? Am I wrong? The reason I don't want squash is that I want all the subtree history mixed into my main project because I want to be able to bisect on the entire project+subtree if I ever need to (they are extremely tightly coupled). Plus, I don't understand why squash solves the "unrelated histories" error! That makes me nervous as to what is going on under the hood, and I'd like to ensure I am doing this correctly before I commit my entire project to a specific course of action. Anyone understand this git subtree stuff better than me who can take a mo to explain? Thanks!!!

1 0

Intel watchdog in linux?
by Trevor Cordes 06 Aug '17

06 Aug '17

I have an Intel S1200BT-family server board which I'm pretty sure has a "watchdog" option in the BIOS. I find myself wanting this feature while working through a kernel hang bug (64-bit this time!). The board is considered an entry-level (E3) server board so while it does have many server bells, it's missing some of the higher-end whistles. Does anyone know if Linux has support for this watchdog? Where do I start looking in terms of packages/programs to support this? I'm hoping this is old hat to someone here... Thanks!

2 1

Apache MPM choice for PHP + HTTP2
by Trevor Cordes 24 Jul '17

24 Jul '17

Trying to get HTTP2 support working in Apache, easy enough. Except when I restart apache it complains that mod_http2 is not compatible with mpm prefork and turns http2 off. OK, I go back to figure out why I'm running prefork. Turns out that still seems to be the easiest/best way to work with PHP via mod_php. Some research reveals talks from 5+ years ago that say that not every PHP module that may be compiled in (in a normal distro) is thread safe, so all of the threaded mpm's are out of the question. Doh. But some people (but not anyone official!) say that the above is dated and PHP should be fine with threaded mpm's. This isn't the type of thing I want to throw out there and see if users can deadlock my apache! Who to believe... So, what are MUUGers who use apache and PHP using as their mpm? Anyone successfully using a threaded mpm in this setup on complicated PHP sites without problems? It looks like the only other option (short of running 2 servers + rev proxy) is php-fpm, which I'm not familiar with but appears to run PHP outside of apache in its own daemon and talk directly with apache via fastcgi or similar. From what I read it shouldn't kill performance but I thought the whole point of mod_php was to put php inside apache for speed. And, no, I'm not interested in moving to nginx at this time (and from benchmarks I saw it's no faster for PHP anyhow). :-) Thanks!

2 1

32->64-bit OS upgrades for Fedora/RHEL systems WORKING!
by Trevor Cordes 24 Jul '17

24 Jul '17

Something I've asked about in the past on this list, and at MUUG meetings, as some may remember: How to upgrade an entire system from 32-bit to 64-bit x86_64 in-place, over a network, off-site. Specifically for Fedora (and RHEL). And ensuring the resulting system was "clean" without a mess everywhere. Well, I finally developed a method and have successfully upgraded all the boxes I'm responsible for. And for not a single one was I required to go on-site! Yay! I wasn't sure it was possible, as all Fedora forums said it was hopeless, and any instructions out there were for like Fedora 5 or 10 and relied on features and workarounds that are no longer present. Most people are like "just reinstall from DVD". Great if you're on-site and your system isn't heavily tweaked. Even then, you're looking at blowing a day (at least) on the upgrade. I couldn't do that for 20+ off-site systems! So I found new hidden methods and workarounds and tested and honed them to come up with a reliable procedure. And it worked! I now have boxes that have been successfully upgraded, using yum/dnf, from ~FC2 through to F24+ and multiple iterations of hardware from P1 to today's Xeon; never with any on-site upgrade or reinstall! And there's no mess & no leftovers! Try that on Windows!! (Boy, do I love Linux!) I'm mentioning it here for two reasons: 1) curiosity, as I'm sure at least someone wants to hear about the outcome of this loony proposition; and 2) I recall at least 2 members telling me they had a similar need (see, it's not just me!). I thought about doing a presentation or something but it would be so narrowly-focused that it would bore the 99% of have no need of this. So for now I'm keeping it under wraps and will be selling it as a service to those who have the need (gotta keep my computer company in business, after all, it is my day job!). So if you want info, email me privately; special uber-discount for MUUGers! P.S. The only other distro I could find with easy 32->64 instructions was Arch. Other than that it appears hopeless using online resources, though I'm pretty sure you could do the upgrade easily on Gentoo also.

1 0

Re: [RndTbl] IP ID field
by Vijay Sankar 20 Jul '17

20 Jul '17

Quoting roundtable-request(a)muug.ca: > Send Roundtable mailing list submissions to > roundtable(a)muug.ca > > To subscribe or unsubscribe via the World Wide Web, visit > https://muug.ca/mailman/listinfo/roundtable > or, via email, send a message with subject or body 'help' to > roundtable-request(a)muug.ca > > You can reach the person managing the list at > roundtable-owner(a)muug.ca > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Roundtable digest..." > > > Today's Topics: > > 1. IP ID field (Vijay Sankar) > 2. Re: IP ID field (Trevor Cordes) > 3. Re: IP ID field (Robert Keizer) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 20 Jul 2017 05:17:38 -0500 > From: Vijay Sankar <vsankar(a)foretell.ca> > To: roundtable(a)muug.ca > Subject: [RndTbl] IP ID field > Message-ID: > <20170720051738.Horde.Njt9ul6yxkwX9F4SgaRrt_f(a)server3.foretell.ca> > Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes > > I am a bit confused about IP ID and was wondering about the following. > > Is it normal to have the same IP ID for the initial SYN packet from > different source IP addresses? There is no fragmentation issues in > this case since it is only 40 bytes and I see this same IP ID only > with attempts to establish a session to 161/TCP. > > I read through RFCs (mostly 6861 and 4413) but not sure. Please let me > know if you can give me any clues or suggestions. > > Thanks very much, > > Vijay > -- > Vijay Sankar, M.Eng., P.Eng. > ForeTell Technologies Limited > vsankar(a)foretell.ca > > > > ------------------------------ > > Message: 2 > Date: Thu, 20 Jul 2017 05:29:47 -0500 > From: Trevor Cordes <trevor(a)tecnopolis.ca> > To: roundtable(a)muug.ca > Subject: Re: [RndTbl] IP ID field > Message-ID: <20170720052947.3d308709(a)pog.tecnopolis.ca> > Content-Type: text/plain; charset=US-ASCII > > On 2017-07-20 Vijay Sankar wrote: >> I am a bit confused about IP ID and was wondering about the following. >> >> Is it normal to have the same IP ID for the initial SYN packet from >> different source IP addresses? There is no fragmentation issues in >> this case since it is only 40 bytes and I see this same IP ID only >> with attempts to establish a session to 161/TCP. > > Off the top of my head, and without consulting anything (I can do that > later), I recall reading something about this being OS specific. Some > OS's randomize, some start with whatever. I think it can be used to > determine what OS is hitting you in some cases. My guess would be > older OS's don't randomize. Or I'm completely out to lunch at this late > hour... > > > ------------------------------ > > Message: 3 > Date: Thu, 20 Jul 2017 08:10:00 -0500 > From: Robert Keizer <robert(a)keizer.ca> > To: roundtable(a)muug.ca > Subject: Re: [RndTbl] IP ID field > Message-ID: <581b99bc-697e-a196-c4ba-00ab38af6a3e(a)keizer.ca> > Content-Type: text/plain; charset="utf-8" > > This might be useful. I had bookmarked it years and years ago because I > thought it was neat. > > http://lcamtuf.coredump.cx/oldtcp/tcpseq.html > > > Rob > > On 2017-07-20 5:29 AM, Trevor Cordes wrote: >> On 2017-07-20 Vijay Sankar wrote: >>> I am a bit confused about IP ID and was wondering about the following. >>> >>> Is it normal to have the same IP ID for the initial SYN packet from >>> different source IP addresses? There is no fragmentation issues in >>> this case since it is only 40 bytes and I see this same IP ID only >>> with attempts to establish a session to 161/TCP. >> Off the top of my head, and without consulting anything (I can do that >> later), I recall reading something about this being OS specific. Some >> OS's randomize, some start with whatever. I think it can be used to >> determine what OS is hitting you in some cases. My guess would be >> older OS's don't randomize. Or I'm completely out to lunch at this late >> hour... >> _______________________________________________ >> Roundtable mailing list >> Roundtable(a)muug.ca >> https://muug.ca/mailman/listinfo/roundtable > > >

1 0

IP ID field
by Vijay Sankar 20 Jul '17

20 Jul '17

I am a bit confused about IP ID and was wondering about the following. Is it normal to have the same IP ID for the initial SYN packet from different source IP addresses? There is no fragmentation issues in this case since it is only 40 bytes and I see this same IP ID only with attempts to establish a session to 161/TCP. I read through RFCs (mostly 6861 and 4413) but not sure. Please let me know if you can give me any clues or suggestions. Thanks very much, Vijay -- Vijay Sankar, M.Eng., P.Eng. ForeTell Technologies Limited vsankar(a)foretell.ca

3 2

Attempted murder of sysadmin
by Trevor Cordes 17 Jul '17

17 Jul '17

I've been thinking about this a lot lately, as each new buzzword layer enters the scene (SaaS, IaaS, MS, VZ, cloud, containers, ...). Each time one of these technologies comes along I look at it and say "boy, is that stupid / inefficient / costly". Of course, I'm thinking as a sysadmin. A couple of years ago it dawned on me that the whole point of these techs was to eliminate sysadmins. These techs *are* inefficient in that they all take more CPU cycles / RAM / storage; they are more costly in that they aren't as cheap as I can do it myself (i.e. cloud vs dedicated server). But that's me, computer expert, who can do everything myself (from hw to os to sw) very quickly. Not CEO who doesn't even know what an OS is. I mean, look at docker or flatpacks or even VMs. These things are the opposite of what we spent 30 years working against... duplication. So now we have the same/similar 100-thousand OS files duplicated everywhere. Now they want to undo the entire concept of shared libraries and essentially make libs static again (well, shared but distribute specific lib versions with the apps, duplicating out the wazoo). For an efficiency purist it seems insane. So what gives? *** It's to eliminate/reduce one of the most highly paid workers in IT: sysadmins. Or at least, to (finally) allow the overseas outsourcing of sysadmin duties. As MUUGers, a lot of us are sysadmins. We should care. Personally, I'm not worried (as I'm sure many of you aren't), as our skillsets are so diversified, we can't be pigeonholed, and, as required, we can be the masters of any new technology. My only question is one of curiosity, will they succeed in murdering most sysadmins? Will they save their $100kUS$/yr/sysadmin by giving it all to AWS? Will people accept the obvious inefficiencies of flatpacks in order to delete the cost of people? Actually, I'm dubious, as I've read these buzzwords in trade pubs ever since I started in tech ~1997, every single one claiming to be the admin-killing panacea, and they came and went, and most turned out to be waaay overblown. I'll be shocked if they actually succeed in killing sysadmins this time. Anyone else thinking about these things? Bonus question: What happens the day AWS (or whatever) screws up / has a major data loss / some other calamity? You sure are putting a lot of trust in *their* sysadmins, who may not be as smart as you are! Decent new article about it all this here: https://www.informationweek.com/devops/rip-systems-administrator-welcome-de…

5 12