As discussed at the meeting, here's what my oom_killer messages look like (see below).
Each time it happened it was a different program "invoking" oom-killer, no rhyme or reason. One time it invoked oom-killer 170 times in 4 minutes before I issued a reboot.
The only weird mem related options I am using are: # try to improve garbage smb/nfs performance net.core.rmem_max=8388608 net.core.wmem_max=8388608 net.core.rmem_default=262143 net.core.wmem_default=262143
# test to enable more disk blocks to remain in cache vm.swappiness=80
# 121205 test 30 to enable more disk blocks to remain in cache longer, seems to immediately help # 131022 switched back to 100 to see if can fix "huge cache slows io down to a crawl after a few days" issue: http://unix.stackexchange.com/questions/94079/massive-unpredictable-i-o-perf... vm.vfs_cache_pressure=100
I will take out all the above and reboot into the new kernel to see if that helps. I was using them for specific reasons, but they aren't critical.
My "taint" is due to prop nvidia driver as there's a current bad bug that makes myth unusable with nouveau.
Feb 10 14:02:40 pog kernel: [40455.970964] xfce4-panel invoked oom-killer: gfp_mask=0x42d0, order=2, oom_score_adj=0 Feb 10 14:02:40 pog kernel: [40455.970970] xfce4-panel cpuset=/ mems_allowed=0 Feb 10 14:02:40 pog kernel: [40455.970975] CPU: 3 PID: 1962 Comm: xfce4-panel Tainted: PF O 3.12.9-201.fc19.i686.PAE #1 Feb 10 14:02:40 pog kernel: [40455.970977] Hardware name: /D975XBX2, BIOS BX97520J.86A.2797.2007.1008.1941 10/08/2007 Feb 10 14:02:40 pog kernel: [40455.970979] 00000000 00000000 e9881c1c c09afcd2 e9bc7980 e9881c74 c09ad927 c0b6070c Feb 10 14:02:40 pog kernel: [40455.970985] e9bc7bb8 000042d0 00000002 00000000 e9881c48 c0c93b14 00000000 e9881cb8 Feb 10 14:02:40 pog kernel: [40455.970991] e9881c90 c0523996 00000206 e9881c74 c0695aa3 000042d0 00000002 0000002e Feb 10 14:02:40 pog kernel: [40455.970996] Call Trace: Feb 10 14:02:40 pog kernel: [40455.971001] [<c09afcd2>] dump_stack+0x41/0x52 Feb 10 14:02:40 pog kernel: [40455.971017] [<c09ad927>] dump_header.isra.11+0x73/0x1b0 Feb 10 14:02:40 pog kernel: [40455.971022] [<c0523996>] ? do_try_to_free_pages+0x4c6/0x4f0 Feb 10 14:02:40 pog kernel: [40455.971026] [<c0695aa3>] ? ___ratelimit+0x83/0xf0 Feb 10 14:02:40 pog kernel: [40455.971030] [<c0515c5b>] oom_kill_process+0x16b/0x2b0 Feb 10 14:02:40 pog kernel: [40455.971034] [<c063647c>] ? security_capable_noaudit+0x1c/0x30 Feb 10 14:02:40 pog kernel: [40455.971038] [<c045dd6a>] ? has_capability_noaudit+0x1a/0x30 Feb 10 14:02:40 pog kernel: [40455.971041] [<c051618b>] out_of_memory+0x22b/0x260 Feb 10 14:02:40 pog kernel: [40455.971044] [<c051a580>] __alloc_pages_nodemask+0x830/0x8b0 Feb 10 14:02:40 pog kernel: [40455.971048] [<c08bba65>] sock_alloc_send_pskb+0x1d5/0x370 Feb 10 14:02:40 pog kernel: [40455.971052] [<c095dad6>] unix_stream_sendmsg+0x2f6/0x3d0 Feb 10 14:02:40 pog kernel: [40455.971056] [<c08b7191>] sock_aio_write+0xd1/0xf0 Feb 10 14:02:40 pog kernel: [40455.971059] [<c0562796>] do_sync_readv_writev+0x56/0x80 Feb 10 14:02:40 pog kernel: [40455.971062] [<c08b70c0>] ? sock_sendpage+0x40/0x40 Feb 10 14:02:40 pog kernel: [40455.971065] [<c056367c>] do_readv_writev+0x9c/0x280 Feb 10 14:02:40 pog kernel: [40455.971068] [<c08b70c0>] ? sock_sendpage+0x40/0x40 Feb 10 14:02:40 pog kernel: [40455.971087] [<f7ef3bc3>] ? nlm_lookup_file+0x83/0x230 [lockd] Feb 10 14:02:40 pog kernel: [40455.971090] [<c08b8e6d>] ? SyS_recvfrom+0x2d/0x30 Feb 10 14:02:40 pog kernel: [40455.971092] [<c05638f7>] vfs_writev+0x37/0x60 Feb 10 14:02:40 pog kernel: [40455.971095] [<c0563a19>] SyS_writev+0x49/0xb0 Feb 10 14:02:40 pog kernel: [40455.971099] [<c09bcdcd>] sysenter_do_call+0x12/0x28 Feb 10 14:02:40 pog kernel: [40455.971100] Mem-Info: Feb 10 14:02:40 pog kernel: [40455.971102] DMA per-cpu: Feb 10 14:02:40 pog kernel: [40455.971104] CPU 0: hi: 0, btch: 1 usd: 0 Feb 10 14:02:40 pog kernel: [40455.971106] CPU 1: hi: 0, btch: 1 usd: 0 Feb 10 14:02:40 pog kernel: [40455.971108] CPU 2: hi: 0, btch: 1 usd: 0 Feb 10 14:02:40 pog kernel: [40455.971110] CPU 3: hi: 0, btch: 1 usd: 0 Feb 10 14:02:40 pog kernel: [40455.971111] Normal per-cpu: Feb 10 14:02:40 pog kernel: [40455.971113] CPU 0: hi: 186, btch: 31 usd: 30 Feb 10 14:02:40 pog kernel: [40455.971115] CPU 1: hi: 186, btch: 31 usd: 0 Feb 10 14:02:40 pog kernel: [40455.971117] CPU 2: hi: 186, btch: 31 usd: 0 Feb 10 14:02:40 pog kernel: [40455.971118] CPU 3: hi: 186, btch: 31 usd: 30 Feb 10 14:02:40 pog kernel: [40455.971120] HighMem per-cpu: Feb 10 14:02:40 pog kernel: [40455.971121] CPU 0: hi: 186, btch: 31 usd: 0 Feb 10 14:02:40 pog kernel: [40455.971123] CPU 1: hi: 186, btch: 31 usd: 0 Feb 10 14:02:40 pog kernel: [40455.971125] CPU 2: hi: 186, btch: 31 usd: 0 Feb 10 14:02:40 pog kernel: [40455.971126] CPU 3: hi: 186, btch: 31 usd: 31 Feb 10 14:02:40 pog kernel: [40455.971130] active_anon:316483 inactive_anon:108472 isolated_anon:0 Feb 10 14:02:40 pog kernel: [40455.971130] active_file:202470 inactive_file:254743 isolated_file:0 Feb 10 14:02:40 pog kernel: [40455.971130] unevictable:0 dirty:1 writeback:0 unstable:0 Feb 10 14:02:40 pog kernel: [40455.971130] free:992094 slab_reclaimable:168477 slab_unreclaimable:6636 Feb 10 14:02:40 pog kernel: [40455.971130] mapped:46706 shmem:3667 pagetables:4214 bounce:0 Feb 10 14:02:40 pog kernel: [40455.971130] free_cma:0 Feb 10 14:02:40 pog kernel: [40455.971138] DMA free:3148kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15928kB managed:5892kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:2332kB slab_unreclaimable:104kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Feb 10 14:02:40 pog kernel: [40455.971140] lowmem_reserve[]: 0 781 8085 8085 Feb 10 14:02:40 pog kernel: [40455.971151] Normal free:47224kB min:3564kB low:4452kB high:5344kB active_anon:0kB inactive_anon:0kB active_file:184kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:890872kB managed:800188kB mlocked:0kB dirty:0kB writeback:0kB mapped:10908kB shmem:0kB slab_reclaimable:671576kB slab_unreclaimable:26440kB kernel_stack:4024kB pagetables:584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:293 all_unreclaimable? yes Feb 10 14:02:40 pog kernel: [40455.971160] lowmem_reserve[]: 0 0 58433 58433 Feb 10 14:02:40 pog kernel: [40455.971191] HighMem free:3918004kB min:512kB low:8840kB high:17172kB active_anon:1265932kB inactive_anon:433888kB active_file:809696kB inactive_file:1019016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7479456kB managed:7479456kB mlocked:0kB dirty:4kB writeback:0kB mapped:175888kB shmem:14668kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:16272kB unstable:0kB bounce:6066568kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 10 14:02:40 pog kernel: [40455.971199] lowmem_reserve[]: 0 0 0 0 Feb 10 14:02:40 pog kernel: [40455.971226] DMA: 21*4kB (UE) 7*8kB (EM) 2*16kB (M) 1*32kB (U) 4*64kB (UE) 1*128kB (U) 2*256kB (M) 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 3148kB Feb 10 14:02:40 pog kernel: [40455.971241] Normal: 3002*4kB (UEM) 4305*8kB (UEMR) 13*16kB (EM) 2*32kB (R) 2*64kB (R) 0*128kB 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47104kB Feb 10 14:02:40 pog kernel: [40455.971252] HighMem: 16681*4kB (UM) 9810*8kB (UM) 4930*16kB (UM) 2511*32kB (UM) 5928*64kB (UM) 2979*128kB (UM) 1204*256kB (UM) 422*512kB (UM) 120*1024kB (UM) 47*2048kB (MR) 515*4096kB (M) = 3918004kB Feb 10 14:02:40 pog kernel: [40455.971267] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Feb 10 14:02:40 pog kernel: [40455.971268] 462536 total pagecache pages Feb 10 14:02:40 pog kernel: [40455.971271] 1636 pages in swap cache Feb 10 14:02:40 pog kernel: [40455.971273] Swap cache stats: add 17955, delete 16319, find 1932/2275 Feb 10 14:02:40 pog kernel: [40455.971274] Free swap = 8330636kB Feb 10 14:02:40 pog kernel: [40455.971275] Total swap = 8388604kB Feb 10 14:02:40 pog kernel: [40455.993273] 2293759 pages RAM Feb 10 14:02:40 pog kernel: [40455.993278] 2066946 pages HighMem Feb 10 14:02:40 pog kernel: [40455.993280] 225202 pages reserved Feb 10 14:02:40 pog kernel: [40455.993281] 1381019 pages shared Feb 10 14:02:40 pog kernel: [40455.993282] 930549 pages non-shared Feb 10 14:02:40 pog kernel: [40455.993284] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Feb 10 14:02:40 pog kernel: [40455.993297] [ 304] 0 304 60742 2440 72 0 0 systemd-journal Feb 10 14:02:40 pog kernel: [40455.993300] [ 324] 0 324 3013 465 8 0 -1000 systemd-udevd Feb 10 14:02:40 pog kernel: [40455.993303] [ 330] 0 330 3227 121 5 0 0 lvmetad Feb 10 14:02:40 pog kernel: [40455.993308] [ 484] 0 484 919 257 8 0 0 alsactl Feb 10 14:02:40 pog kernel: [40455.993311] [ 505] 288 505 16860 1304 17 9 0 milter-greylist Feb 10 14:02:40 pog kernel: [40455.993317] [ 511] 0 511 7894 387 10 0 0 rsyslogd Feb 10 14:02:40 pog kernel: [40455.993320] [ 514] 0 514 1007 246 5 0 0 irqbalance Feb 10 14:02:40 pog kernel: [40455.993323] [ 515] 0 515 1642 228 9 0 0 sh Feb 10 14:02:40 pog kernel: [40455.993326] [ 516] 0 516 718 124 6 0 0 mdadm Feb 10 14:02:40 pog kernel: [40455.993329] [ 517] 0 517 1642 229 8 0 0 sh Feb 10 14:02:40 pog kernel: [40455.993332] [ 519] 0 519 2025 453 7 0 0 smartd Feb 10 14:02:40 pog kernel: [40455.993335] [ 520] 0 520 1027 400 4 0 0 systemd-logind Feb 10 14:02:40 pog kernel: [40455.993338] [ 522] 81 522 1031 427 5 2 -900 dbus-daemon Feb 10 14:02:40 pog kernel: [40455.993341] [ 534] 0 534 643 74 6 0 0 gpm Feb 10 14:02:40 pog kernel: [40455.993344] [ 572] 32 572 1204 245 5 0 0 rpcbind Feb 10 14:02:40 pog kernel: [40455.993347] [ 577] 0 577 2782 1190 13 26 0 udp-sgs Feb 10 14:02:40 pog kernel: [40455.993352] [ 578] 0 578 3177 1552 17 0 0 dynamic-ip-upda Feb 10 14:02:40 pog kernel: [40455.993355] [ 596] 0 596 2507 326 21 1 0 saslauthd Feb 10 14:02:40 pog kernel: [40455.993358] [ 597] 0 597 2507 329 21 1 0 saslauthd Feb 10 14:02:40 pog kernel: [40455.993361] [ 598] 0 598 2507 333 21 1 0 saslauthd Feb 10 14:02:40 pog kernel: [40455.993363] [ 599] 0 599 2507 308 21 1 0 saslauthd Feb 10 14:02:40 pog kernel: [40455.993366] [ 600] 0 600 2507 333 21 1 0 saslauthd Feb 10 14:02:40 pog kernel: [40455.993370] [ 865] 0 865 5239 1809 13 206 0 dhclient Feb 10 14:02:40 pog kernel: [40455.993373] [ 1221] 0 1221 1642 229 8 0 0 sh Feb 10 14:02:40 pog kernel: [40455.993375] [ 1222] 0 1222 1642 229 8 0 0 sh Feb 10 14:02:40 pog kernel: [40455.993378] [ 1223] 0 1223 1642 217 8 13 0 sh Feb 10 14:02:40 pog kernel: [40455.993381] [ 1226] 0 1226 3145 1461 16 29 0 mailwarnings Feb 10 14:02:40 pog kernel: [40455.993385] [ 1227] 0 1227 3012 822 19 0 0 fetchmail Feb 10 14:02:40 pog kernel: [40455.993389] [ 1228] 0 1228 1826 131 6 5 0 vsftpd Feb 10 14:02:40 pog kernel: [40455.993392] [ 1229] 0 1229 7587 5131 29 62 0 ebay-getter-ser Feb 10 14:02:40 pog kernel: [40455.993396] [ 1240] 0 1240 2146 564 13 3 0 pushover Feb 10 14:02:40 pog kernel: [40455.993399] [ 1248] 0 1248 10837 533 10 0 0 apcupsd Feb 10 14:02:40 pog kernel: [40455.993402] [ 1249] 0 1249 2685 659 8 0 -1000 sshd Feb 10 14:02:40 pog kernel: [40455.993405] [ 1259] 43 1259 1135 367 9 82 0 xfs Feb 10 14:02:40 pog kernel: [40455.993408] [ 1264] 603 1264 856 293 8 0 0 thttpd Feb 10 14:02:40 pog kernel: [40455.993411] [ 1279] 0 1279 824 340 4 0 0 dovecot Feb 10 14:02:40 pog kernel: [40455.993414] [ 1304] 0 1304 806 183 4 0 0 atd Feb 10 14:02:40 pog kernel: [40455.993417] [ 1310] 27 1310 1643 308 9 0 0 mysqld_safe Feb 10 14:02:40 pog kernel: [40455.993420] [ 1414] 0 1414 1086 179 7 0 0 agetty Feb 10 14:02:40 pog kernel: [40455.993425] [ 1416] 0 1416 1086 180 7 0 0 agetty Feb 10 14:02:40 pog kernel: [40455.993428] [ 1419] 0 1419 1086 180 7 0 0 agetty Feb 10 14:02:40 pog kernel: [40455.993430] [ 1420] 0 1420 1086 179 7 0 0 agetty Feb 10 14:02:40 pog kernel: [40455.993433] [ 1421] 0 1421 1086 179 7 0 0 agetty Feb 10 14:02:40 pog kernel: [40455.993436] [ 1422] 0 1422 1260 499 14 19 0 login Feb 10 14:02:40 pog kernel: [40455.993439] [ 1564] 41 1564 4595 1349 20 82 0 mailmanctl Feb 10 14:02:40 pog kernel: [40455.993442] [ 1582] 41 1582 4565 1981 22 45 0 python Feb 10 14:02:40 pog kernel: [40455.993445] [ 1583] 41 1583 4551 1914 21 114 0 python Feb 10 14:02:40 pog kernel: [40455.993448] [ 1584] 41 1584 4568 2023 21 36 0 python Feb 10 14:02:40 pog kernel: [40455.993451] [ 1585] 41 1585 4586 2041 21 52 0 python Feb 10 14:02:40 pog kernel: [40455.993453] [ 1586] 41 1586 4569 1997 21 38 0 python Feb 10 14:02:40 pog kernel: [40455.993457] [ 1587] 41 1587 4586 2069 21 28 0 python Feb 10 14:02:40 pog kernel: [40455.993461] [ 1588] 41 1588 4585 2047 22 24 0 python Feb 10 14:02:40 pog kernel: [40455.993464] [ 1589] 41 1589 4552 1973 21 50 0 python Feb 10 14:02:40 pog kernel: [40455.993467] [ 1622] 27 1622 186685 37079 111 1683 0 mysqld Feb 10 14:02:40 pog kernel: [40455.993470] [ 1623] 25 1623 24371 8928 32 355 0 named Feb 10 14:02:40 pog kernel: [40455.993473] [ 1624] 97 1624 783 225 4 0 0 anvil Feb 10 14:02:40 pog kernel: [40455.993476] [ 1625] 0 1625 816 259 4 0 0 log Feb 10 14:02:40 pog kernel: [40455.993479] [ 1627] 0 1627 979 417 5 0 0 config Feb 10 14:02:40 pog kernel: [40455.993481] [ 1648] 0 1648 5113 538 10 17 0 nmbd Feb 10 14:02:40 pog kernel: [40455.993484] [ 1649] 0 1649 5103 327 10 20 0 nmbd Feb 10 14:02:40 pog kernel: [40455.993487] [ 1690] 0 1690 7621 1105 17 0 0 smbd Feb 10 14:02:40 pog kernel: [40455.993490] [ 1692] 0 1692 1642 228 9 0 0 sh Feb 10 14:02:40 pog kernel: [40455.993495] [ 1693] 0 1693 8659 2861 30 97 0 ebay-reminder Feb 10 14:02:40 pog kernel: [40455.993498] [ 1696] 0 1696 28241 16858 56 695 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993501] [ 1745] 0 1745 3583 712 10 0 0 sendmail Feb 10 14:02:40 pog kernel: [40455.993505] [ 1762] 0 1762 7621 711 17 0 0 smbd Feb 10 14:02:40 pog kernel: [40455.993508] [ 1764] 0 1764 1439 30 4 0 0 rpc.rquotad Feb 10 14:02:40 pog kernel: [40455.993511] [ 1765] 0 1765 690 61 4 0 0 rpc.idmapd Feb 10 14:02:40 pog kernel: [40455.993514] [ 1766] 0 1766 3246 834 10 0 0 cupsd Feb 10 14:02:40 pog kernel: [40455.993517] [ 1767] 0 1767 1395 224 5 0 0 rpc.mountd Feb 10 14:02:40 pog kernel: [40455.993520] [ 1781] 51 1781 3458 393 9 11 0 sendmail Feb 10 14:02:40 pog kernel: [40455.993523] [ 1783] 110 1783 8441 1266 14 1 0 colord Feb 10 14:02:40 pog kernel: [40455.993526] [ 1793] 177 1793 5758 3003 14 56 0 dhcpd Feb 10 14:02:40 pog kernel: [40455.993531] [ 1794] 0 1794 1844 352 6 0 0 crond Feb 10 14:02:40 pog kernel: [40455.993533] [ 1795] 38 1795 1441 454 5 0 0 ntpd Feb 10 14:02:40 pog kernel: [40455.993536] [ 1801] 0 1801 3017 1873 16 23 0 nss_pcache Feb 10 14:02:40 pog kernel: [40455.993539] [ 1809] 48 1809 29534 17156 58 1265 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993542] [ 1811] 48 1811 29250 17260 57 865 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993545] [ 1812] 48 1812 29245 17205 57 867 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993548] [ 1818] 0 1818 6698 728 19 19 0 console-kit-dae Feb 10 14:02:40 pog kernel: [40455.993551] [ 1890] 500 1890 2146 770 11 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993555] [ 1915] 500 1915 818 271 7 0 0 sh Feb 10 14:02:40 pog kernel: [40455.993558] [ 1934] 500 1934 913 152 7 0 0 xinit Feb 10 14:02:40 pog kernel: [40455.993561] [ 1935] 500 1935 42199 16283 94 540 0 X Feb 10 14:02:40 pog kernel: [40455.993564] [ 1942] 500 1942 18364 3132 44 196 0 xfce4-session Feb 10 14:02:40 pog kernel: [40455.993567] [ 1949] 500 1949 929 110 4 0 0 dbus-launch Feb 10 14:02:40 pog kernel: [40455.993570] [ 1950] 500 1950 1143 301 5 8 0 dbus-daemon Feb 10 14:02:40 pog kernel: [40455.993573] [ 1955] 500 1955 1992 130 7 8 0 ssh-agent Feb 10 14:02:40 pog kernel: [40455.993576] [ 1958] 500 1958 1723 509 14 1 0 xfconfd Feb 10 14:02:40 pog kernel: [40455.993579] [ 1961] 500 1961 860 155 10 20 0 gpg-agent Feb 10 14:02:40 pog kernel: [40455.993582] [ 1962] 500 1962 41956 4980 60 55 0 xfce4-panel Feb 10 14:02:40 pog kernel: [40455.993585] [ 1964] 500 1964 4555 1959 28 7 0 sawfish Feb 10 14:02:40 pog kernel: [40455.993588] [ 1965] 500 1965 21550 3786 47 83 0 xfdesktop Feb 10 14:02:40 pog kernel: [40455.993591] [ 1969] 500 1969 7133 589 20 0 0 gvfsd Feb 10 14:02:40 pog kernel: [40455.993594] [ 1970] 500 1970 17360 9459 33 378 0 gkrellm Feb 10 14:02:40 pog kernel: [40455.993597] [ 1971] 500 1971 15135 9364 31 432 0 gkrellm Feb 10 14:02:40 pog kernel: [40455.993600] [ 1979] 500 1979 11733 688 22 1 0 gvfsd-fuse Feb 10 14:02:40 pog kernel: [40455.993603] [ 1980] 500 1980 38509 3475 49 53 0 xfce4-power-man Feb 10 14:02:40 pog kernel: [40455.993606] [ 1982] 500 1982 16295 3746 42 103 0 xfce4-verve-plu Feb 10 14:02:40 pog kernel: [40455.993609] [ 1990] 500 1990 13591 3019 40 191 0 panel-6-systray Feb 10 14:02:40 pog kernel: [40455.993612] [ 1991] 500 1991 17420 4407 46 21 0 panel-10-weathe Feb 10 14:02:40 pog kernel: [40455.993616] [ 2000] 0 2000 9815 890 30 0 0 upowerd Feb 10 14:02:40 pog kernel: [40455.993619] [ 2091] 500 2091 1519 631 16 0 0 xscreensaver Feb 10 14:02:40 pog kernel: [40455.993622] [ 2093] 500 2093 15845 2675 40 202 0 polkit-gnome-au Feb 10 14:02:40 pog kernel: [40455.993625] [ 2094] 500 2094 16635 3133 45 128 0 xfsettingsd Feb 10 14:02:40 pog kernel: [40455.993628] [ 2098] 500 2098 38870 2127 54 0 0 pulseaudio Feb 10 14:02:40 pog kernel: [40455.993631] [ 2101] 500 2101 3762 634 26 0 0 gconf-helper Feb 10 14:02:40 pog kernel: [40455.993634] [ 2103] 500 2103 2034 745 15 26 0 gconfd-2 Feb 10 14:02:40 pog kernel: [40455.993637] [ 2105] 500 2105 11149 640 17 0 0 at-spi-bus-laun Feb 10 14:02:40 pog kernel: [40455.993640] [ 2109] 500 2109 937 334 4 0 0 dbus-daemon Feb 10 14:02:40 pog kernel: [40455.993644] [ 2112] 500 2112 4420 689 14 0 0 at-spi2-registr Feb 10 14:02:40 pog kernel: [40455.993647] [ 2124] 500 2124 48376 7963 73 68 0 gnome-terminal- Feb 10 14:02:40 pog kernel: [40455.993650] [ 2127] 500 2127 572 165 4 0 0 gnome-pty-helpe Feb 10 14:02:40 pog kernel: [40455.993653] [ 2128] 500 2128 1356 785 10 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993656] [ 2145] 500 2145 1264 462 5 0 0 su Feb 10 14:02:40 pog kernel: [40455.993659] [ 2149] 0 2149 1344 639 10 133 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993662] [ 2165] 0 2165 6023 563 14 0 0 synergys Feb 10 14:02:40 pog kernel: [40455.993665] [ 2207] 500 2207 818 250 7 0 0 open-windows Feb 10 14:02:40 pog kernel: [40455.993668] [ 2210] 500 2210 1176 597 10 3 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993671] [ 2232] 500 2232 1408 838 10 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993674] [ 2248] 500 2248 1200 632 9 1 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993677] [ 2253] 500 2253 1800 737 16 0 0 xterm Feb 10 14:02:40 pog kernel: [40455.993680] [ 2270] 500 2270 1105 491 9 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993682] [ 2276] 500 2276 797 397 8 0 0 nano Feb 10 14:02:40 pog kernel: [40455.993685] [ 2283] 500 2283 21572 15970 57 542 0 daily Feb 10 14:02:40 pog kernel: [40455.993688] [ 2304] 500 2304 176236 42720 227 760 0 firefox Feb 10 14:02:40 pog kernel: [40455.993691] [ 2368] 48 2368 29245 16978 57 777 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993694] [ 2489] 500 2489 172363 75676 285 3569 0 mythfrontend Feb 10 14:02:40 pog kernel: [40455.993697] [ 2529] 0 2529 4202 773 17 0 -900 udisks-daemon Feb 10 14:02:40 pog kernel: [40455.993700] [ 2530] 0 2530 1830 114 16 0 -900 udisks-daemon Feb 10 14:02:40 pog kernel: [40455.993703] [ 2555] 500 2555 1356 785 10 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993706] [ 2628] 273 2628 1996 700 8 0 0 imap-login Feb 10 14:02:40 pog kernel: [40455.993708] [ 2630] 0 2630 1261 211 6 0 0 ssl-params Feb 10 14:02:40 pog kernel: [40455.993711] [ 2632] 500 2632 1606 613 6 25 0 imap Feb 10 14:02:40 pog kernel: [40455.993714] [ 7572] 500 7572 1359 791 10 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993718] [ 7610] 500 7610 54425 10691 96 0 0 plugin-containe Feb 10 14:02:40 pog kernel: [40455.993721] [ 7914] 500 7914 1357 785 9 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993724] [ 7931] 500 7931 2587 814 8 0 0 ssh Feb 10 14:02:40 pog kernel: [40455.993727] [ 9044] 276 9044 67121 55494 133 0 0 clamd Feb 10 14:02:40 pog kernel: [40455.993730] [ 9065] 290 9065 11914 546 14 0 0 clamav-milter Feb 10 14:02:40 pog kernel: [40455.993733] [ 9126] 23 9126 4522 583 13 0 0 squid Feb 10 14:02:40 pog kernel: [40455.993736] [ 9128] 23 9128 15573 11593 31 0 0 squid Feb 10 14:02:40 pog kernel: [40455.993739] [ 9132] 23 9132 960 176 4 0 0 unlinkd Feb 10 14:02:40 pog kernel: [40455.993741] [ 9298] 48 9298 29245 17858 57 271 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993745] [24205] 48 24205 29187 17883 57 210 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993748] [24206] 48 24206 29245 17310 57 353 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993751] [24207] 48 24207 29038 16873 57 363 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993754] [24522] 48 24522 29117 17220 57 353 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993757] [24536] 0 24536 638 251 4 0 0 vgetty Feb 10 14:02:40 pog kernel: [40455.993760] [24582] 48 24582 29245 17293 57 352 0 /usr/sbin/httpd Feb 10 14:02:40 pog kernel: [40455.993763] [ 3504] 500 3504 47214 6098 78 0 0 Thunar Feb 10 14:02:40 pog kernel: [40455.993766] [ 3509] 500 3509 7452 960 22 0 0 gvfs-udisks2-vo Feb 10 14:02:40 pog kernel: [40455.993769] [ 3511] 0 3511 13259 1005 15 0 0 udisksd Feb 10 14:02:40 pog kernel: [40455.993772] [ 3515] 106 3515 18824 2056 19 0 0 polkitd Feb 10 14:02:40 pog kernel: [40455.993775] [ 3538] 500 3538 9585 766 22 0 0 gvfsd-trash Feb 10 14:02:40 pog kernel: [40455.993778] [ 3566] 500 3566 1425 856 11 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993781] [ 3616] 500 3616 75855 6894 92 0 0 audacious Feb 10 14:02:40 pog kernel: [40455.993784] [ 3714] 500 3714 14299 3854 42 0 0 rep Feb 10 14:02:40 pog kernel: [40455.993786] [ 3752] 500 3752 50242 10092 64 0 0 claws-mail Feb 10 14:02:40 pog kernel: [40455.993789] [ 3758] 500 3758 2079 908 8 0 0 imap Feb 10 14:02:40 pog kernel: [40455.993792] [ 3763] 506 3763 1234 657 6 0 0 imap Feb 10 14:02:40 pog kernel: [40455.993795] [ 3776] 500 3776 1239 662 10 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993798] [ 3796] 500 3796 1358 787 10 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993801] [ 3814] 500 3814 2587 812 8 0 0 ssh Feb 10 14:02:40 pog kernel: [40455.993804] [ 3886] 500 3886 1357 788 10 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993807] [ 3909] 500 3909 1265 462 5 0 0 su Feb 10 14:02:40 pog kernel: [40455.993810] [ 3913] 0 3913 1374 802 11 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993813] [ 4068] 0 4068 1965 1149 13 0 0 expunge Feb 10 14:02:40 pog kernel: [40455.993816] [ 4071] 500 4071 1358 788 10 0 0 tcsh Feb 10 14:02:40 pog kernel: [40455.993819] [ 4105] 500 4105 2587 815 8 0 0 ssh Feb 10 14:02:40 pog kernel: [40455.993822] [ 4398] 0 4398 3626 1059 10 0 0 sendmail Feb 10 14:02:40 pog kernel: [40455.993825] Out of memory: Kill process 2489 (mythfrontend) score 19 or sacrifice child Feb 10 14:02:40 pog kernel: [40455.993830] Killed process 2489 (mythfrontend) total-vm:689452kB, anon-rss:246640kB, file-rss:56064kB