[gentoo-dev] Portage, Jobs, and Niceness

[Chris White]

Hi all,

-- Niceness --

  While waiting for kde to compile (read as hours), I decided to take a chance and talk about niceness and jobs this time.  Ok so, we'll start with niceness first:

if portage.settings.has_key("PORTAGE_NICENESS"):
        try:
                os.nice(int(portage.settings["PORTAGE_NICENESS"]))
        except SystemExit, e:
                raise # Needed else can't exit
        except Exception,e:
                print "!!! Failed to change nice value to '"+str(portage.setting
s["PORTAGE_NICENESS"])+"'"
                print "!!!",e

the setting PORTAGE_NICENESS in make.conf sets up the portage niceness.  This code gives you an idea of how it is done.  if it sees PORTAGE_NICENESS, it simply uses the os module's nice function to set the niceness.  Now.. niceness is kind of weird.. so let's demonstrate using American Idol(tm)!  Alright.. so you have your two wannabe singers here, ProcessPest and ProcessPrincess.  Now.. Simon is the cpu here, having to judge which singer he wants more (it is safe to note here that one doesn't suck).  Now, instead of showing who he likes more, he shows who he hates less!  So, we have something like this:

nice -n -19 ProcessPrincess
nice -n 10 ProcessPest

So Simon says to ProcessPrincess "You suck -19, which means I like you, you get more cpu"
So Simon says to ProcessPest "You suck 10, which means you suck, you get screwed"

Now back to our regularly scheduled technical form... Ok, so now that we know what's going on, niceness means how "nice" you are to other programs.  The more "nice" you are, the more willing you are to share your cpu to other programs.  The less "nice" you are, the more you want to hog it to yourself.  Now, when a process is set niceness, it does it by INCREMENT, not setting.  This basically means that if you do this:

PORTAGE_NICENESS=10

portage will increase your niceness from its original (normally 0 unless nice was already applied to it) by 10, giving you 10 niceness.  Now if you do it negative:

PORTAGE_NICENESS=-3

it increases by a negative number.. so it DECREASES portage niceness.  In this case, it would be -3, which is generally an ok value.  Note that the values go from -20 to 20, and that while you can increase niceness no problem, if you want to increases, you're probably going to need to be root.  Note to BE CAREFUL ABOUT NICENESS.  Remember, the same niceness that portage gets will be thusly applied to its child processes.  That means if you're running -j5 with PORTAGE_NICENESS=-19, subprocesses will all recieve -19 niceness too.  That's a lot of programs trying to grab for the cpu (I can tell you from experience that it can happen..)!  Only try this if you have some spectacular multi processor elite setup.  So that's what you get with niceness.  Now let's look at jobs:

-- Jobs --

Then there's jobs.  Now, this is extremly usefull for people that realized "Oh crap, I'm in the middle of emerge-ing such and such package and I just realized I need all my cpu for something more important (homework maybe?)".  No need to fear!  Bash has this nice little deal called jobs.  Let's take a look:

Scenario

<ChrisWhite> Ahh crud!  My xorg-x11 compile is halfway through and I just noticed this security patch I need to get through asap!  What to do! (or I'm compiling kde and realize I missed an assignment at school!).  Nice little handy bash function called jobs can take care of this.  To work with jobs, simply press CTRL+Z during the compile.  This stops the process as so:

(please note I was lazy here and removed my japanese error messages.. now you're not dreaming)

making imake with BOOTSTRAPCFLAGS= and CROSSCOMPILEFLAGS=-DCROSSCOMPILEDIR="" in config/imake
make[2]: Leaving directory `/var/tmp/portage/xorg-x11-6.8.99.8/work/xc/config/imake'
rm -f ./config/makedepend/Makefile.proto
./config/imake/imake -I./config/cf  -s ./config/makedepend/Makefile.proto -f ./config/makedepend/Imakefile -DTOPDIR=../.. -DCURDIR=./config/makedepend
./config/cf/Imake.tmpl:109  include 
                 Imakefile.c:35 :
./config/cf/linux.cf:390: Warning: "BuildLibGlxWithoutPIC" redefined
./config/cf/site.def:44  include ,
                 ./config/cf/Imake.tmpl:46 
                 Imakefile.c:35 :
./config/cf/host.def:63: Warning: this is the location of the previous definition
cd ./config/makedepend && rm -f -r Makefile Makefile.dep makedepend *.o bootstrap
cd ./config/makedepend && make -f Makefile.proto bootstrap
make[2]: Entering directory `/var/tmp/portage/xorg-x11-6.8.99.8/work/xc/config/makedepend'
make[2]: `bootstrap'
make[2]: Leaving directory `/var/tmp/portage/xorg-x11-6.8.99.8/work/xc/config/makedepend'
./config/imake/imake -I./config/cf  -s ./config/imake/Makefile.proto -f ./config/imake/Imakefile -DTOPDIR=../.. -DCURDIR=./config/imake -DBootStrap

[1]+  Stopped                 emerge xorg-x11

And it stopped!  Now that you've done your stuff, you'd like it to run again.  First off, we do a jobs listing like such:

secures chris # jobs
[1]+  Stopped                 emerge xorg-x11

Job 1 is listed here as our stopped emerge xorg-x11.  Please note that jobs are PER SESSION.  This means you can't just pull another console up and expect the current console's jobs to show up in it.  One session, one job list.  So, you have two options:

1) fg 1
2) bg 1

the first puts the process in the foreground
the second puts the process in the background

Let's pull it to the foreground in this case:

secures chris # fg 1
emerge xorg-x11
./config/cf/Imake.tmpl:109  include 
                 Imakefile.c:35 :
./config/cf/linux.cf:390: : "BuildLibGlxWithoutPIC" redefined
./config/cf/site.def:44  include 
                 ./config/cf/Imake.tmpl:46 ,
                 Imakefile.c:35 :
./config/cf/host.def:63: : this is the location of the previous definition
cd ./config/imake && make -f Makefile.proto bootstrapdepend
make[2]: Entering directory `/var/tmp/portage/xorg-x11-6.8.99.8/work/xc/config/imake'
make[2]: `bootstrapdepend' 
make[2]: Leaving directory `/var/tmp/portage/xorg-x11-6.8.99.8/work/xc/config/imake'
cd ./config/imake && make -w -f Makefile.ini BOOTSTRAPCFLAGS="" CC="cc" bootstrap
make[2]: Entering directory `/var/tmp/portage/xorg-x11-6.8.99.8/work/xc/config/imake'
make[2]: `bootstrap' 
make[2]: Leaving directory `/var/tmp/portage/xorg-x11-6.8.99.8/work/xc/config/imake'
cd ./config/imake && make -f Makefile.proto imakeonly
make[2]: Entering directory `/var/tmp/portage/xorg-x11-6.8.99.8/work/xc/config/imake'
make[2]: `imakeonly' 
make[2]: Leaving directory `/var/tmp/portage/xorg-x11-6.8.99.8/work/xc/config/imake'
+ rm -f xmakefile.bak
+ mv xmakefile xmakefile.bak

and viola!  Our compile moves right on!  Please note that jobs is NOT A PROGRAM, it's a part of the shell.  I know bash does it.. I'm not sure if others do.  That's it for now, hope this becomes usefull to some people.

Chris White