Re: [gentoo-user] Limit number of cores used by emerge?

[Mark Knecht]

On Fri, Sep 30, 2011 at 10:28 AM, Paul Hartman
<paul.hartman+gentoo@×××××.com> wrote:
> On Fri, Sep 30, 2011 at 11:25 AM, Mark Knecht <markknecht@×××××.com> wrote:
>> Hi,
>>   Is there a portage option that will limit the number of cores used
>> by emerge? For instance, in a chroot on a 12 core machine I want to
>> limit emerge to not using more than 3 cores?
>>
>>   If possible, I'd also like to limit the total disk bandwidth
>> consumption during emerge. For instance, when untarring a big file to
>> do the emerge at times the disk consumption gets to high and the
>> machine becomes laggy. Is there an option that addresses this?
>>
>>   These questions are mostly about being able to update a chroot
>> mid-day without other tasks slowing down too much. I don't care how
>> long the chroot really takes to get a huge emerge done, but rathe just
>> keeping the machine very responsive while it's happening. I already
>> use:
>>
>> MAKEOPTS="-j3"
>> PORTAGE_NICENESS="15"
>>
>> which helps (I think) but it doesn't totally address either of the issues above.
>
> If your MAKEOPTS is -j3 then it's not going to use more than 3 cores
> at a time but it will touch all 12 cores throughout the process
> because of the normal load balancing. If you want it to use only 3
> specific cores, you would need to set the processor affinity (usually
> done using the "taskset" command from sys-apps/util-linux).
>
> For the disk I/O you can set an ionice in your make.conf like:
> PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"
>
> Salt to taste. :)
>

Thanks to all who have answered so far. I'll give the ionice command a
try immediately. I'll try to address my view of other comments in a
single post to keep the traffic down.

My experience with -j3 is not that it limits emerge to 3 CPU cores but
rather it limits emerge to running 3 _jobs_. There's a big difference.
1 job can use 12 cores if gcc spawns a lot of stuff in parallel, which
in my experience it does. It's this parallel spawn running up to 12
cores in use which causes the machine to lag. Even when not setting
the -j option at all which results in a single package being emerged
at one time, I often use 6-12 cores in use for short periods of time
as gcc builds that package. This is completely appropriate for the
host machine. When I do updates I want them to get done ASAP. On the
other hand, for a chroot it's too much power.

Concerning which core is used, I don't care.

Concerning the 'bouncing around' aspect of multi-core processors,
that's primarily a CPU function of where it has cached data & code as
well as thermal management on chip. Internal hardware is generally
controlling what CPU gets used for what as far as I know. However even
a single emerge job (1 package being emerged) could move around from
core to core depending on what work is getting done. It's
understandable that a driver might be fundamentally associated with a
specific core's immediate L1 or L2 cache, whereas the emerge job
itself could be using other cores.

Of course, everything above is just my limited understanding of what
the hardware is doing and could be completely wrong.

Thanks,
Mark