[gentoo-user] Re: how does a pipe work? Which process wait for which one, or they don't actually wait each other?

[Caster]

张|武 wrote:
> Hello. This might be OT but I am pretty interested in this and being
> unlucky not able to find a real in-depth explanation of pipe on the
> Internet.
>

I think "man 7 pipe" gives quite precise explanation for this.

> How does pipe actually work? I mean, when there is a pipe like this:
> $ appA | appB
> What happen if appA produced output when appB is still busy processing
> the data and did not require any data from input?
>
> possibility 1) as long as appA can get the resource (CPU?), it simply
> keep outputing data, and this data is cached in memory, as long as there
> is enough memory, and will finally feed to appB when appB finished his
> business and begin to accept more data;
>
> possibility 2) as long as appB stop requiring data, appA is suspended,
> the resource goes to appB. appA is only given resource (CPU?) when appB
> finished his business and begin to accept more data;
>

I would say "both is true", except the limitation of (1) is not whole
memory, but pipe buffer size (according to man it's 64k). When this buffer
is full, (2) happens.

> In my case appA gets the data from another host who have very short
> timeout settings, appB is used to compress the data obtained from appA.
> the compression is very difficult, usually at 30Kbps, the network is
> very fast, around 10Mbps. appB compress the data tunck by tunck, if
> Linux actually works in mode 2), the network connection is dropped when
> the interval of two tuncks of appB compressing data is longer then the
> network timeout setting. appA acutally don't know how to restart
> connection from where it was dropped, thus understanding this difference
> makes sense to me.

If it's a problem, either appA or appB should be threaded, one thread
buffering the the input in own memory (not relying on pipe buffer size) and
second thread making the output (appA) or compression (appB).

Caster