I had looked at it.... But your suggestion made me google just "nf_conntrack" which lead me to "conntrack-tools" "that allow system administrators interact with the Connection Tracking System,"
Now we are getting somewhere...
I can see that there is state information for that IP address but sadly, deleting all the state information for that IP (conntrack -D -s xxx.xxx.xxx.xxx) did not solve the problem. There must be something deeper going on. Perhaps the limit stuff was just a red herring... The state tracking looks normal.
John