1 |
On cygwin, there is a problem with bi-directional pipe support in bash. |
2 |
|
3 |
I used to solve this with an ugly reversion in portage and an |
4 |
ultra-simple stubbification patch for multiprocessing.eclass (both |
5 |
serialized everything). |
6 |
|
7 |
However, this really sucked for numerous reasons, including the obvious |
8 |
one: it makes stuff slow as hell. |
9 |
|
10 |
So I've reworked everything to use uni-directional named pipes. |
11 |
|
12 |
In portage I have: |
13 |
|
14 |
diff --git a/bin/helper-functions.sh b/bin/helper-functions.sh |
15 |
index c7400fa..87f3120 100644 |
16 |
--- a/bin/helper-functions.sh |
17 |
+++ b/bin/helper-functions.sh |
18 |
@@ -7,6 +7,36 @@ |
19 |
|
20 |
source "${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}"/isolated-functions.sh |
21 |
|
22 |
+# try real hard to figure out if this is a cygwin host; cache results. |
23 |
+this_host_is_cygwin() { |
24 |
+ if [[ -n ${_this_host_is_cygwin} ]] ; then |
25 |
+ return $_this_host_is_cygwin |
26 |
+ fi |
27 |
+ [[ -x ${EPREFIX}/usr/bin/uname ]] && \ |
28 |
+ [[ $( ${EPREFIX}/usr/bin/uname -o 2>/dev/null ) == Cygwin* ]] && \ |
29 |
+ export _this_host_is_cygwin=0 && return 0 |
30 |
+ [[ -x /usr/bin/uname ]] && \ |
31 |
+ [[ $( /usr/bin/uname -o 2>/dev/null ) == Cygwin* ]] && \ |
32 |
+ export _this_host_is_cygwin=0 && return 0 |
33 |
+ [[ -x /bin/uname ]] && \ |
34 |
+ [[ $( /bin/uname -o 2>/dev/null ) == Cygwin* ]] && \ |
35 |
+ export _this_host_is_cygwin=0 && return 0 |
36 |
+ # hail-mary before we resort to envvars |
37 |
+ [[ $( uname -o 2>/dev/null ) == Cygwin* ]] && \ |
38 |
+ export _this_host_is_cygwin=0 && return 0 |
39 |
+ |
40 |
+ [[ -n ${CHOST} ]] && \ |
41 |
+ [[ ${CHOST} == *-cygwin* ]] && \ |
42 |
+ export _this_host_is_cygwin=0 && return 0 |
43 |
+ [[ -n ${CTARGET} ]] && \ |
44 |
+ [[ ${CTARGET} == *-cygwin* ]] && \ |
45 |
+ export _this_host_is_cygwin=0 && return 0 |
46 |
+ |
47 |
+ # either it ain't cygwin or something is very broken. |
48 |
+ export _this_host_is_cygwin=1 |
49 |
+ return 1 |
50 |
+} |
51 |
+ |
52 |
# |
53 |
# API functions for doing parallel processing |
54 |
# |
55 |
@@ -19,25 +49,51 @@ numjobs() { |
56 |
|
57 |
multijob_init() { |
58 |
# Setup a pipe for children to write their pids to when they finish. |
59 |
- mj_control_pipe=$(mktemp -t multijob.XXXXXX) |
60 |
- rm "${mj_control_pipe}" |
61 |
- mkfifo "${mj_control_pipe}" |
62 |
- redirect_alloc_fd mj_control_fd "${mj_control_pipe}" |
63 |
+ export mj_control_pipe=$(mktemp -t multijob.XXXXXX) |
64 |
rm -f "${mj_control_pipe}" |
65 |
+ mkfifo "${mj_control_pipe}" |
66 |
+ |
67 |
+ if ! this_host_is_cygwin ; then |
68 |
+ redirect_alloc_fd mj_control_fd "${mj_control_pipe}" |
69 |
+ rm -f "${mj_control_pipe}" |
70 |
+ fi |
71 |
|
72 |
# See how many children we can fork based on the user's settings. |
73 |
mj_max_jobs=$(numjobs) |
74 |
mj_num_jobs=0 |
75 |
} |
76 |
|
77 |
+# make sure someone called multijob_init |
78 |
+multijob_assert() { |
79 |
+ if this_host_is_cygwin ; then |
80 |
+ [[ -z ${mj_control_pipe} ]] && \ |
81 |
+ die "multijob initialization required" |
82 |
+ [[ $( file -b "${mj_control_pipe}" ) != fifo* ]] && \ |
83 |
+ die "multijob fifo gone" |
84 |
+ else |
85 |
+ [[ -z ${mj_control_fd} ]] && \ |
86 |
+ die "multijob initialization required" |
87 |
+ fi |
88 |
+} |
89 |
+ |
90 |
multijob_child_init() { |
91 |
- trap 'echo ${BASHPID} $? >&'${mj_control_fd} EXIT |
92 |
+ multijob_assert |
93 |
+ if this_host_is_cygwin ; then |
94 |
+ trap 'echo ${BASHPID} $? >'${mj_control_pipe} EXIT |
95 |
+ else |
96 |
+ trap 'echo ${BASHPID} $? >&'${mj_control_fd} EXIT |
97 |
+ fi |
98 |
trap 'exit 1' INT TERM |
99 |
} |
100 |
|
101 |
multijob_finish_one() { |
102 |
local pid ret |
103 |
- read -r -u ${mj_control_fd} pid ret |
104 |
+ multijob_assert |
105 |
+ if this_host_is_cygwin ; then |
106 |
+ read -r pid ret < ${mj_control_pipe} |
107 |
+ else |
108 |
+ read -r -u ${mj_control_fd} pid ret |
109 |
+ fi |
110 |
: $(( --mj_num_jobs )) |
111 |
return ${ret} |
112 |
} |
113 |
@@ -70,6 +126,9 @@ multijob_post_fork() { |
114 |
redirect_alloc_fd() { |
115 |
local var=$1 file=$2 redir=${3:-"<>"} |
116 |
|
117 |
+ [[ "${redir}" == "<>" ]] && this_host_is_cygwin && \ |
118 |
+ die "Cygwin bash has broken <> bidirectional redirection support." |
119 |
+ |
120 |
if [[ $(( (BASH_VERSINFO[0] << 8) + BASH_VERSINFO[1] )) -ge $(( (4 << |
121 |
8) + 1 )) ]] ; then |
122 |
# Newer bash provides this functionality. |
123 |
eval "exec {${var}}${redir}'${file}'" |
124 |
-- |
125 |
(Here's hoping Thunderbird didn't reflow the above; apologies in advance |
126 |
if it did) |
127 |
|
128 |
In multiprocessing.eclass, I do something fairly similar, also relying |
129 |
on named pipes rather than file-descriptor cloning. |
130 |
|
131 |
I was wondering if anyone in the know could comment on how correct or |
132 |
incorrect the above actually is? For example, is either portage |
133 |
multijob or mutliprocessing.eclass expected to work if someone nests |
134 |
multijob_init's (because I'm pretty sure mine won't -- FTR, this seems |
135 |
pretty naughty, given that both get their parallelism defaults from |
136 |
MAKEOPTS)? |
137 |
|
138 |
Finally, to be clear, I'm not wanting this to go in to git or |
139 |
anything... at least, certainly not as of now. Just seeking |
140 |
constructive criticism so my code isn't horribly broken, since there |
141 |
aren't a ton of multi{jobs,processing} clients, ATM, for me to test against. |
142 |
|
143 |
-gmt |
144 |
|
145 |
Postscript: status of this bug in cygwin, for anyone who may be |
146 |
interested/googling: |
147 |
|
148 |
I've tried to look into this, just a little bit, because if I run an |
149 |
ultra-simple bidirectional pipe test in C, it almost kinda-sorta works |
150 |
(although some weird pump-priming seems to be required ... blah). |
151 |
|
152 |
bash's code for this doesn't exactly make for light reading, nor does |
153 |
cygwin's named pipe code, although the latter is at least not mixed up |
154 |
with a bunch of parser/generator code. Given that I'm not even sure the |
155 |
problem isn't already fixed upstream, it didn't seem worth the effort. |
156 |
|
157 |
The matter has been mentioned on the cygwin mailing list, and cgf, |
158 |
cygwin lead dev and implementation SME, has stated he doesn't want the |
159 |
named-pipe code churning for now. So, unless the fix is already |
160 |
in-repo, but unreleased (my over-under on this would be maybe 3:1 |
161 |
against; I'm too lazy to check), it's likely to stay this way for a while. |
162 |
|
163 |
My personal game-plan is: just wait until the next cygwin core release |
164 |
and see if it's fixed. Until then, I have what seem to be tolerable |
165 |
work-arounds for my overlay, as described above. If the next release |
166 |
fails to resolve the problem, then I might revisit and see if I can't |
167 |
find a solution, or at least a work-around in bash. |