Re: [gentoo-alt] multilib / multijob / parallel build problem

[Greg Turner]

Well, like I said I'm quite plausibly confused, as it's been a while
since I paid any mind to this.

But I do recall that there was some tricky race that could crop up if
the readers and the writers didn't synchronize "just so".  IIRC
zmedico posted a test-case for it to one of these lists, probably -dev
or this one.

I think it was some issue along the lines that, under certain
circumstances, if the writer started stuffing data in before the
reader started listening, the first message(s?) down the pipe would be
lost.

Even more vaguely, I seem to recall that somehow, the act of bash
creating, specifically, a bidirectional pipe prevented the problem --
I think, some magic happened with fork() and the cloned bidi pipe
handle where the pipe was considered to "have a reader" right away and
therefore (if the pipe buffer filled up?) writers would block until
the reader got around to reading... something like that.

Anyhow I could definitely be completely wrong, and even if I'm not,
it's quite possible that by cloning the reader and writer pipe handles
you'll achieve the same result.

A way to test for this would be to create a simple multijob
test-ebuild, where the first job returns a failure code, but all the
others (preferably a crapload of super-quick ones, I guess) succeed,
and modify multiprocessing.eclass, by inserting a 30-second sleep in
multijob_finish_one just before the "read" statement.  If
multijob_finish in your test e-build correctly returned a nonzero
result, then you could be sure that the first message was not ignored
despite the delayed read.

(This is assuming it's the first write that gets dropped... I could be
wrong about that, i.e., it could be the second, or the first one after
some buffer fills up, or I could just be spewing complete nonsense :)
)

On Mon, Oct 7, 2013 at 12:36 AM, Alan Hourihane <alanh@×××××××××××.uk> wrote:
> On 10/07/13 01:11, Greg Turner wrote:
>>
>> Alan wrote:
>>>
>>> - redirect_alloc_fd mj_control_fd "${pipe}"
>>> + redirect_alloc_fd mj_write_fd "${pipe}"
>>> + redirect_alloc_fd mj_read_fd "${pipe}"
>>
>> The one thing I'd be looking to verify is that there's no race where
>> where the reader must block before the writer starts producing data.
>>
>> You could test that hypothesis by sleeping for a while right before
>> the reader starts listening.
>>
>> If that were the case -- I kinda suspect it is -- then unfortunately,
>> I think the writer has to spin, waiting for the reader to block... not
>> a pretty situation, so let's hope I'm confused.
>>
>
> I don't understand. There is no race, the above two commands just get two
> different file descriptors for the read end and the write end. The FIFO is
> opened in non-blocking IO mode anyway, as the FIFO is still opened in
> read/write mode to get this behaviour. So the actual read from one fd, and
> the write from another fd behaves no differently to reading and writing from
> the same fd.
>
> Alan.
>