Is there a way to tell after the fact what command(s) an "at" command ran? I got an email about an error in a cron job and it appears to be an at command that was queued up. The error is suspicious, and I don't remember queueing this at command (though I do use at from time to time).
I can find in logs that it was atd running it under my user id, but the at queue is now empty and I can't find a way to retroactively see what commands/script tried to run. I'm doing a full fs search on some of the error strings but I am not hopeful.
Is there a way to find out more about what at ran? Are there options that in the future would give me more log/debug output so I can more easily answer this type of question?
Doing some cursory Googles (I don't use at much myself either, except when upgrading salt minions) it seems like atd doesn't log very much at all, unless you run jobs with the -m option. Another suggestion was to wrap the command being run with at in a wrapper to syslog:
ex: #!/bin/bash logger -i -t mycmd Starting /bin/somecommand logger -i -t mycmd Completed exit 0
It looks like you're out of luck for determining what happened in the past, but going into the future either the -m or wrapper options might work for you.
Theodore Baschak https://ciscodude.net/ https://theodorebaschak.com/
On Fri, Feb 26, 2016 at 1:59 PM, Trevor Cordes trevor@tecnopolis.ca wrote:
Is there a way to tell after the fact what command(s) an "at" command ran? I got an email about an error in a cron job and it appears to be an at command that was queued up. The error is suspicious, and I don't remember queueing this at command (though I do use at from time to time).
I can find in logs that it was atd running it under my user id, but the at queue is now empty and I can't find a way to retroactively see what commands/script tried to run. I'm doing a full fs search on some of the error strings but I am not hopeful.
Is there a way to find out more about what at ran? Are there options that in the future would give me more log/debug output so I can more easily answer this type of question? _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Here's the precise debug email I got from cron:
sh: line 69: warning: here-document at line 68 delimited by end-of-file (wanted `marcinDELIMITER04dba0c8') /bin/sh: 1marcinDELIMITER04dba0c8: ambiguous redirect
It spooks me a little bit as that's a funky here-doc string to look for, and WTF with the redirect? Doesn't look like anything I would have scripted (I rarely use heredocs).
The problem with requiring the caller of at to do something to log at commands is that that won't help track/log if someone manages to inject a malicious at command. The above evidence smells to me of some compromised program allowing someone to set a timed at job as my user. But it could just be me being paranoid...
On Sat, 27 Feb 2016, Trevor Cordes wrote:
Here's the precise debug email I got from cron:
sh: line 69: warning: here-document at line 68 delimited by end-of-file (wanted `marcinDELIMITER04dba0c8') /bin/sh: 1marcinDELIMITER04dba0c8: ambiguous redirect
I thought I'd seen that here-doc delimiter before. It's standard for the at command. When entered at the command line the at command puts the user's input in a here-document for the atd to execute.
Once upon a time there was a bug https://bugzilla.redhat.com/show_bug.cgi?id=486844 that led me to remember what I've observed on the at job queue.
It spooks me a little bit as that's a funky here-doc string to look for, and WTF with the redirect? Doesn't look like anything I would have scripted (I rarely use heredocs).
I've seen some utilities that don't care for the vagaries of different cron daemons use an at command at the end of the current at job to schedule their next run. The first one is usually started by an init script or whatever equivalent in the distro in use.
The problem with requiring the caller of at to do something to log at commands is that that won't help track/log if someone manages to inject a malicious at command. The above evidence smells to me of some compromised program allowing someone to set a timed at job as my user. But it could just be me being paranoid...
-Daryl
On 2016-02-27 Daryl F wrote:
sh: line 69: warning: here-document at line 68 delimited by end-of-file (wanted `marcinDELIMITER04dba0c8') /bin/sh: 1marcinDELIMITER04dba0c8: ambiguous redirect
I thought I'd seen that here-doc delimiter before. It's standard for the at command. When entered at the command line the at command puts the user's input in a here-document for the atd to execute.
Thanks to all who helped (jd, come visit us at a meeting sometime)! I left a complete disk scan for marcinDELIMITER running and came back and independently ran across the exact same thing as Daryl just before I read his reply! Nailed it!
Yes, from the disk scan, hexdumping all the results for my perusal, an odd one stood out. Byte offsets + debugfs reverse decoding allowed me to determine that string is used in /usr/bin/at! WTF? Searching that substring (without the hex after it, hence why I didn't find it before) led me to the same bz.
Phew, so it's not some guy named marcin p0wning me. It probably is an at job I wrote ages ago that somehow messed up the incredibly lame and odd way that at is handling this stuff. Perhaps it was an at job I moved over from my old 32b box to the new 64b install and at is unhappy with such a raw migration. Now I have to remember what the at job was about, and I almost exclusively use at for notes-to-self reminders!
That's some great memory there Daryl remembering that odd delimiter... I knew MUUGers would be able to help! Of course, I doubt I'll ever forget it now, too; after being bitten by it.
-
Daryl F -
Theodore Baschak -
Trevor Cordes