OK, this is a new one...
Updated my kernel and rebooted tonight. Everything came up fine.
Except dovecot (IMAP server). It complained: Error: service(imap-login): listen(*, 993) failed: Address already in use
(993=imaps)
OK. Except:
#netstat -tulpn | grep 993 <nothing> #lsof -i:993 <nothing> #ss -t -l 'sport = 993' <nothing>
A wasted hour later I did:
netstat -n |grep 993
Sure enough I had a tcp connection between my workstation and my file server on 993 to 2049. grep 2049 /etc/services... eureka!
2049 is nfs.
From what I've figured out, nfs, which I have mount something on boot,
randomly grabbed 993 before dovecot did, and was holding it as long as the nfs fs was mounted locally.
And because nfs is in-kernel it didn't show up in lsof et al.
I umounted it, waited for TIME_WAIT to expire and then I could start dovecot.
*** This has never happened before on dozens of reboots... I guess it's pretty much luck / race condition what port nfs gets, especially with systemd possibly trying to do the mount and/or dovecot in parallel. I have no idea when it tries to do the mounts, but I'm guessing before dovecot.
Note, I'm using NFSv4 which (unlike older NFS) uses TCP stateful in a persistent connection. Pretty much everything you know about v2/v3 throw away when it comes to using v4.
I poked around nfs docs and google and found this option:
resvport / noresvport Specifies whether the NFS client should use a privileged source port when communicating with an NFS server for this mount point. If this option is not specified, or the resvport option is specified, the NFS client uses a privileged source port. If the noresvport option is specified, the NFS client uses a non-privileged source port. This option is supported in kernels 2.6.28 and later. Refer to the SECURITY CONSIDERATIONS section for important details.
OK, down there is a section:
Using non-privileged source ports
Which basically says using privileged ports is for security so normal users can't make/fake their own daemons and pretend to be any user they want. OK, so I know that's a bit hokey from a sec standpoint, but it's better than nothing, I guess. Regardless, having nfs client pick a random high port might hurt me also because I use some of them for my own purposes/daemons and I need them open also.
The docs say this:
"The exact range of privileged source ports that can be chosen is set by a pair of sysctls to avoid choosing a well-known port, such as the port used by ssh."
Which would be perfect, except I can't find any further reference to these magic sysctls. I tried searching my box's /proc and /sys fs's for nfs & ports, nfs & priv, nfs & mount, etc. Couldn't find anything relevant.
It would be really nice if I could specify the local source port, or at least specify the list of no-no ports using this elusive, promised, "sysctl". Anyone have any ideas?
What a strange bug to run into. Chance of 1 in 1024 and it had to hit me.
On 2016-10-24 Trevor Cordes wrote:
It would be really nice if I could specify the local source port, or at least specify the list of no-no ports using this elusive, promised, "sysctl". Anyone have any ideas?
Sigh... as often the case, after writing my email I found the answer, this was a tough one! Had to use the source patch logs to get some hints. I didn't want to grep the files in sys and proc for nfs, but sunrpc.
#find /proc/ /sys/ | grep sunrpc |grep port /proc/sys/sunrpc/max_resvport /proc/sys/sunrpc/min_resvport
#tail -c+1 /proc/sys/sunrpc/m??_resvport ==> /proc/sys/sunrpc/max_resvport <== 1023
==> /proc/sys/sunrpc/min_resvport <== 665
Also 2 files in: /sys/module/sunrpc/parameters/m??_resvport
Might need to tweak.
Tada! I'll just tweak those to a tiny range I know for sure I won't use at all, and still be under 1024.
Note, each mount takes another of those ports, so if you mount a large number of remote fs's then you need a big enough range. I only mount 1 NFS mount, so I'm good with a tiny (1?) range.
A bit presumptuous of them to assume anything over 665 is "safe"!!
Moral of the story, if you use NFSv4 and you run critical services over port 664, you better set these sysctl's to avoid this problem.
Aside: I may also be forgetting my raw socket programming, but I didn't think using a source port to connect to something external (i.e. as an initiator) tied up that port completely keeping others from bind/listen()ing on it? I thought as long as you were the only bind/listen()er it didn't matter what ports other progs took? I could be massively wrong on this...
Also, most distros now seem to have a service called 'portreserve' that runs very early in the boot process, grabs all the critical ports and then lets them go one by one as other services start up and ask for them. I recall from my reading that RPC and NFS were the primary reason this thing existed. -Adam
-----Original Message----- From: roundtable-bounces@muug.mb.ca [mailto:roundtable- bounces@muug.mb.ca] On Behalf Of Trevor Cordes Sent: October 24, 2016 03:58 To: roundtable@muug.mb.ca Subject: Re: [RndTbl] wacky NFS port problem
On 2016-10-24 Trevor Cordes wrote:
It would be really nice if I could specify the local source port, or at least specify the list of no-no ports using this elusive, promised, "sysctl". Anyone have any ideas?
Sigh... as often the case, after writing my email I found the answer, this was a tough one! Had to use the source patch logs to get some hints. I didn't want to grep the files in sys and proc for nfs, but sunrpc.
#find /proc/ /sys/ | grep sunrpc |grep port /proc/sys/sunrpc/max_resvport /proc/sys/sunrpc/min_resvport
#tail -c+1 /proc/sys/sunrpc/m??_resvport ==> /proc/sys/sunrpc/max_resvport <== 1023
==> /proc/sys/sunrpc/min_resvport <== 665
Also 2 files in: /sys/module/sunrpc/parameters/m??_resvport
Might need to tweak.
Tada! I'll just tweak those to a tiny range I know for sure I won't use at all, and still be under 1024.
Note, each mount takes another of those ports, so if you mount a large number of remote fs's then you need a big enough range. I only mount 1 NFS mount, so I'm good with a tiny (1?) range.
A bit presumptuous of them to assume anything over 665 is "safe"!!
Moral of the story, if you use NFSv4 and you run critical services over port 664, you better set these sysctl's to avoid this problem.
Aside: I may also be forgetting my raw socket programming, but I didn't think using a source port to connect to something external (i.e. as an initiator) tied up that port completely keeping others from bind/listen()ing on it? I thought as long as you were the only bind/listen()er it didn't matter what ports other progs took? I could be massively wrong on this... _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
On 2016-10-24 Adam Thompson wrote:
Also, most distros now seem to have a service called 'portreserve' that runs very early in the boot process, grabs all the critical ports and then lets them go one by one as other services start up and ask for them. I recall from my reading that RPC and NFS were the primary reason this thing existed. -Adam
Ya, I think systemd has something like that... not sure why it's not working by default, although how would it know I have imap on vs just pop and imaps vs just imap. Seems like an impossible problem since lower layers shouldn't be parsing higher layer conf files...
I guess I'll hit the systemd docs, if they exist, about this feature, to see if that could do it for me. However, I'm glad I also have my other workaround as I don't trust systemd not to mess up their feature (change syntax, semantics, etc) in the future as they've done to me sooooo many times.
Ya, I think systemd has something like that... Seems like an impossible problem since lower layers shouldn't be parsing higher layer conf files...
Um... we are talking about system, here. However, IIRC, you had to enumerate the ports yourself in /etc/default/portreserve or something like that.
I guess I'll hit the systemd docs, if they exist, about this feature, to see if that could do it for me. However, I'm glad I also have my other workaround as I don't trust systemd not to mess up their feature (change syntax, semantics, etc) in the future as they've done to me sooooo many times.
Welcome to Windows :-). (Feel free to switch to *BSD, although maybe not FreeBSD since they're in the middle of completely reorganizing their init system, too...)
-Adam
-
Adam Thompson -
Trevor Cordes