reliable way of counting real CPUs under Linux?
Hi All, What would be the subj? Suppose you have a PC. It can have Intel or AMD CPUs. It can have HyperThreading on, and even enabled. And it can have one or two, or several sockets. Would there be a universal method to count real physical cores? Say, passing /proc/cpuinfo through a magic filter, and not failing on AMD Opterons too? -- Grigory Shamov HPC Analyst, Westgrid/Compute Canada E2-588 EITC Building, University of Manitoba (204) 474-9625
On 17/04/2014 2:39 PM, Grigory Shamov wrote:
Hi All,
What would be the subj? Suppose you have a PC. It can have Intel or AMD CPUs. It can have HyperThreading on, and even enabled. And it can have one or two, or several sockets.
Would there be a universal method to count real physical cores? Say, passing /proc/cpuinfo through a magic filter, and not failing on AMD Opterons too?
An IT World article from a year ago dealt with that sort of thing... http://www.itworld.com/operating-systems/340223/counting-processors-your-lin... A quick Google search also came up with many articles and forum posts on the topic, some more vague and general than others... http://www.richweb.com/cpu_info https://www.ibm.com/developerworks/community/blogs/brian/entry/linux_show_th... http://heidydogdog.appspot.com/?p=105001 https://access.redhat.com/site/discussions/480953 http://www.linuxforums.org/forum/red-hat-fedora-linux/180236-how-do-i-get-ph... Hope these help! -- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609
Hi Gilbert, Thanks! There is a lot of advice there for sure. For example, The 'ht' flag is always there on our AMD Opterons (Shaghais and Barcelonas), even they do not have HT enabled. I'd think that 'ht' it is always there in any SMP-enabled linux kernel? -- Grigory Shamov HPC Analyst, Westgrid/Compute Canada E2-588 EITC Building, University of Manitoba (204) 474-9625 On 14-04-17 3:13 PM, "Gilbert E. Detillieux" <gedetil@cs.umanitoba.ca> wrote:
On 17/04/2014 2:39 PM, Grigory Shamov wrote:
Hi All,
What would be the subj? Suppose you have a PC. It can have Intel or AMD CPUs. It can have HyperThreading on, and even enabled. And it can have one or two, or several sockets.
Would there be a universal method to count real physical cores? Say, passing /proc/cpuinfo through a magic filter, and not failing on AMD Opterons too?
An IT World article from a year ago dealt with that sort of thing...
http://www.itworld.com/operating-systems/340223/counting-processors-your-l inux-box?source=ITWNLE_nlt_linux_2013-02-05
A quick Google search also came up with many articles and forum posts on the topic, some more vague and general than others...
http://www.richweb.com/cpu_info https://www.ibm.com/developerworks/community/blogs/brian/entry/linux_show_ the_number_of_cpu_cores_on_your_system17?lang=en http://heidydogdog.appspot.com/?p=105001 https://access.redhat.com/site/discussions/480953 http://www.linuxforums.org/forum/red-hat-fedora-linux/180236-how-do-i-get- physical-cpu-socket-count-rhel.html
Hope these help!
-- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609 _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
On 17/04/2014 4:01 PM, Grigory Shamov wrote:
Hi Gilbert,
Thanks!
There is a lot of advice there for sure. For example, The 'ht' flag is always there on our AMD Opterons (Shaghais and Barcelonas), even they do not have HT enabled. I'd think that 'ht' it is always there in any SMP-enabled linux kernel?
More likely, it's always there for any processor that supports HT, whether it's currently enabled or not. -- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609
Dear Gilbert, On 14-04-17 4:33 PM, "Gilbert E. Detillieux" <gedetil@cs.umanitoba.ca> wrote:
On 17/04/2014 4:01 PM, Grigory Shamov wrote:
Hi Gilbert,
Thanks!
There is a lot of advice there for sure. For example, The 'ht' flag is always there on our AMD Opterons (Shaghais and Barcelonas), even they do not have HT enabled. I'd think that 'ht' it is always there in any SMP-enabled linux kernel?
More likely, it's always there for any processor that supports HT, whether it's currently enabled or not.
Do you know if AMD Opterons support HT? model name : Quad-Core AMD Opteron(tm) Processor 2376 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 *ht* etc. etc. model name : Six-Core AMD Opteron(tm) Processor 2431 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 *ht* etc. etc. -- Grigory Shamov HPC Analyst, Westgrid/Compute Canada E2-588 EITC Building, University of Manitoba (204) 474-9625
On 14-04-21 09:02 AM, Grigory Shamov wrote:
Do you know if AMD Opterons support HT?
model name : Quad-Core AMD Opteron(tm) Processor 2376 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 *ht* etc. etc.
model name : Six-Core AMD Opteron(tm) Processor 2431 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 *ht* etc. etc.
Not historically, but apparently some newer ones may. The output you show above certainly suggests that they do. To count physical sockets, physical cores, and virtual cores (aka HT), look at the "physical id", "siblings", "core id", and "cpu cores" values in /proc/cpuinfo. If "siblings" > "cpu cores", HT (or something similar) is turned on and functioning. Actual # of physical cores should be #-of-distinct-physical-ids * "cpu cores". Note that some very strange systems may not always have constant "cpu cores" value for every physical socket. I don't think you'll encounter this situation in your work, but better to not make assumptions. -- -Adam Thompson athompso@athompso.net
This article from several years ago suggests AMD processors did not support hyper-threading... http://www.ducea.com/2006/06/23/linux-tips-how-to-find-out-if-a-your-cpu-sup... ... but suggests they may in the future. A talk with my CPU-savvy co-worker confirmed that AMD has never supported HT, and likely wouldn't ever. So, that raises the question: What exactly does the "ht" flag in /proc/cpuinfo mean, then, if it's not the existence of HT? This thread offers some insight... https://groups.google.com/forum/#!topic/fa.linux.kernel/rAttTaRINKw So, the "ht" flag seems to just indicate a processor's ability to detect siblings, whether they're actual cores or just HT. Misleading, yes, but they chose to leave it that way rather than break the ABI. <rant> Let's leave an obscure, misleading flag in a particular /proc file alone, to avoid breaking things for some programmer, somewhere, who may be doing some weird test that would be better done in some other way. But, on the other hand, let's completely change the way all process and device initialization happens in a very fundamental way, regardless of how many scripts and 3rd-party services break as a result... Gotta love Linux! (OK, that may be an unfair juxtaposition, since we're talking about kernel vs user space, and different groups of people supporting these, with radically different priorities and methods. But still... Ugh!) </rant> Gilbert On 21/04/2014 9:02 AM, Grigory Shamov wrote:
Dear Gilbert,
On 14-04-17 4:33 PM, "Gilbert E. Detillieux" <gedetil@cs.umanitoba.ca> wrote:
On 17/04/2014 4:01 PM, Grigory Shamov wrote:
Hi Gilbert,
Thanks!
There is a lot of advice there for sure. For example, The 'ht' flag is always there on our AMD Opterons (Shaghais and Barcelonas), even they do not have HT enabled. I'd think that 'ht' it is always there in any SMP-enabled linux kernel?
More likely, it's always there for any processor that supports HT, whether it's currently enabled or not.
Do you know if AMD Opterons support HT?
model name : Quad-Core AMD Opteron(tm) Processor 2376 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 *ht* etc. etc.
model name : Six-Core AMD Opteron(tm) Processor 2431 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 *ht* etc. etc.
-- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609
I was just looking at https://github.com/opscode/ohai/blob/master/lib/ohai/plugins/linux/cpu.rband realized it's related to this conversation. It's how Chef enumerates CPUs on Linux and lines 42-44 is where it differentiates a real cpu vs a logical one. Sean On Tue, Apr 22, 2014 at 3:55 PM, Gilbert E. Detillieux < gedetil@cs.umanitoba.ca> wrote:
This article from several years ago suggests AMD processors did not support hyper-threading...
http://www.ducea.com/2006/06/23/linux-tips-how-to-find-out- if-a-your-cpu-supports-ht-hyper-threading/
... but suggests they may in the future. A talk with my CPU-savvy co-worker confirmed that AMD has never supported HT, and likely wouldn't ever.
So, that raises the question: What exactly does the "ht" flag in /proc/cpuinfo mean, then, if it's not the existence of HT? This thread offers some insight...
https://groups.google.com/forum/#!topic/fa.linux.kernel/rAttTaRINKw
So, the "ht" flag seems to just indicate a processor's ability to detect siblings, whether they're actual cores or just HT. Misleading, yes, but they chose to leave it that way rather than break the ABI.
<rant> Let's leave an obscure, misleading flag in a particular /proc file alone, to avoid breaking things for some programmer, somewhere, who may be doing some weird test that would be better done in some other way. But, on the other hand, let's completely change the way all process and device initialization happens in a very fundamental way, regardless of how many scripts and 3rd-party services break as a result...
Gotta love Linux! (OK, that may be an unfair juxtaposition, since we're talking about kernel vs user space, and different groups of people supporting these, with radically different priorities and methods. But still... Ugh!) </rant>
Gilbert
On 21/04/2014 9:02 AM, Grigory Shamov wrote:
Dear Gilbert,
On 14-04-17 4:33 PM, "Gilbert E. Detillieux" <gedetil@cs.umanitoba.ca> wrote:
On 17/04/2014 4:01 PM, Grigory Shamov wrote:
Hi Gilbert,
Thanks!
There is a lot of advice there for sure. For example, The 'ht' flag is always there on our AMD Opterons (Shaghais and Barcelonas), even they do not have HT enabled. I'd think that 'ht' it is always there in any SMP-enabled linux kernel?
More likely, it's always there for any processor that supports HT, whether it's currently enabled or not.
Do you know if AMD Opterons support HT?
model name : Quad-Core AMD Opteron(tm) Processor 2376 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 *ht* etc. etc.
model name : Six-Core AMD Opteron(tm) Processor 2431 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 *ht* etc. etc.
-- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
-- Sean Walberg <sean@ertw.com> http://ertw.com/
participants (4)
-
Adam Thompson -
Gilbert E. Detillieux -
Grigory Shamov -
Sean Walberg