Gentoo Archives: gentoo-user

From: Dr Rainer Woitok <rainer.woitok@×××××.com>
To: gentoo-user@l.g.o
Subject: [gentoo-user] [SOLVED] Re: Google and "fetchmail" + "ssmtp"
Date: Wed, 06 Apr 2022 17:26:40
Message-Id: 25165.52546.952360.921586@tux.local
1 Greetings,
2
3 On Friday, 2022-03-18 11:50:33 +0100, I myself wrote:
4
5 > Nikos,
6 >
7 > On Thursday, 2022-03-17 19:04:04 +0200, you wrote:
8 >
9 > > ...
10 > > http://mmogilvi.users.sourceforge.net/software/oauthbearer.html
11 > > ...
12 >
13 > Really interesting reading. Thanks for the pointer. And also thanks to
14 > the other responders.
15
16 Matthew Ogilvie's write-up at the above URL was not only interesting but
17 also extremely helpful. It basically covers four points:
18
19 1. Background information on OAuth 2.
20
21 2. A how-to about setting up one's Google mail account for OAuth 2 and a
22 Python script (requiring a small configuration file), which gets you
23 your first two OAuth tokens: an "access token" which has to be passed
24 to Google as a password, but which has only a life time of 60 minut-
25 es, and a "refresh token" which is valid until you change your Google
26 password or maybe also until Google decides enough being enough and
27 starts rejecting your refresh requests with "HTTP Error 400: Bad Re-
28 quest" messages which you'll find at the bottom of the Python trace-
29 back from Matthew's script. As long as the refresh token is valid
30 you will have to use this script on a rather regular basis to get new
31 valid access tokens.
32
33 3. Gentoo-ready patches for "net-mail/fetchmail" up to at least version
34 6.4.13 which however will only support IMAP and will not work in dae-
35 mon mode, thus requiring a "cron" job for fetching mail.
36
37 Matthew also points out that Gentoo offers a still masked "net-mail/
38 fetchmail" version 7.0.0_alpha9-r1 which supports OAuth 2, POP3, and
39 daemon mode.
40
41 4. A description how to tweak "mail-mta/postfix" so it supports OAuth 2.
42 This involves installing a "cyrus-sasl-xoauth2" plugin not available
43 in the Gentoo repository, a complex configuration setup for a full-
44 fledged MTA which above all is dynamically changing (at least, if you
45 want to handle outgoing mail for more than one user), special "sudo-
46 ers" rules for these mailing users, and "cron" jobs to periodically
47 send their mail off to their Google accounts.
48
49 Following part 2 of this guide I soon had my Google account OAuth-ready.
50 The Python script from Matthew requires a configuration file the path to
51 which has to be explicitly specified in every call. My configuration
52 file resides at "~/.../oauth.cfg" and contains:
53
54 client_id=...
55 client_secret=...
56 access_token_file=/home/rainer/.../oauth-access-token
57 refresh_token_file=/home/rainer/.../oauth-refresh-token
58 max_age_sec=1800
59
60 The first two lines specify the Google project just created, the next
61 two lines define the absolute paths to the two token files to be used,
62 and the last line sets the access token age before which the token will
63 not be renewed. DO NOT try to add empty or comment lines to this file!
64
65 Regarding the adaption of "fetchmail" in part 3 I opted for installing
66 version 7.0.0_alpha9-r1, since I was used to POP3 and daemon mode [1].
67 Version 7 is still masked, so to be able to emerge it execute (in the
68 first "echo" command replace "amd64" with YOUR architecture):
69
70 # mkdir -p /etc/portage/package.accept_keywords
71 # echo '>=net-mail/fetchmail-7.0.0_alpha9-r1 ~amd64' \
72 >> /etc/portage/package.accept_keywords/Oauth
73 # mkdir -p /etc/portage/package.unmask
74 # echo '>=net-mail/fetchmail-7.0.0_alpha9-r1' \
75 >> /etc/portage/package.unmask/OAuth
76
77 [1] If you do not like installing an alpha-version, want to use POP3,
78 but do not insist on daemon mode, I've meanwhile found out that Gen-
79 too also provides a package "net-mail/mpop" from the same author as
80 the MTA I decided to install instead of "mail-mta/postfix". Howev-
81 er, Gentoo-wise this package is almost two years behind due to the
82 lack of a Gentoo maintainer, so you might want to install it direct-
83 ly from
84
85 https://marlam.de/mpop/
86
87 Of course the new "fetchmail" version also introduced new configuration
88 items. Here is my new "~/.fetchmailrc" file (the "authenticate" direct-
89 ive as well as the last three lines are new):
90
91 set daemon 60
92 set invisible
93 set no syslog
94
95 poll pop.gmail.com
96 protocol POP3
97 service 995
98 authenticate oauthbearer
99
100 username "rainer.woitok@×××××.com"
101 dropdelivered
102 fetchall
103 no keep
104 mda "/usr/bin/procmail -pf %F"
105 passwordfile "/home/rainer/.../oauth-access-token"
106 sslmode wrapped
107 sslcertck
108
109 Even though I run "fetchmail" in daemon mode, it's not running all the
110 time. It's started once when I log in, but my backup script which I use
111 at least once a day will again terminate it before doing anything else,
112 so my mailbox will never change between my last backup and hibernating
113 my laptop.
114
115 And yes, I almost never shutdown my laptop, but rather suspend or hib-
116 ernate it. Thus I'll possibly need a fresh access token when I manually
117 start "fetchmail", but it's not necessary to repeatedly update it via
118 "cron" while "fetchmail" isn't running at all. The same holds when re-
119 suming from suspension or hibernation: only when "fetchmail" is running
120 the age of the access token should be checked and potentially a new one
121 requested.
122
123 So I'll have to use Matthew's "oauth.py" script in three ways:
124
125 1. Running it the first time or starting over to get both, a new refresh
126 token and a new access token.
127
128 2. Unconditionally checking the age of the access token and optionally
129 requesting a new one, regardless of whether or not "fetchmail" is ex-
130 ecuting.
131
132 3. Only checking the age of the access token and optionally requesting a
133 new one, if "fetchmail" is really executing.
134
135 This leads to wrapper script "~/.../oauth.sh" which knows the options it
136 has to pass to script "oauth.py":
137
138 #! /bin/bash -u
139 #
140 # Script to retrieve new OAuth tokens from Google mail.
141 #
142 # There are two types of OAuth tokens, an "access token" which only has
143 # a life time of 60 minutes and is required for both, fetching mail from
144 # and sending mail via Google mail, and a "refresh token" which is valid
145 # until changing one's Google mail password or until Google decides it
146 # has been used long enough, and is required to get a new "access tok-
147 # en".
148 #
149 # The action performed by this script depends on its first argument:
150 #
151 # "init": Get a (new) refresh token together with a new access token.
152 # This causes an URL to be sent to standard output, which you
153 # have to point your browser to. At Google's mail login page
154 # use your mail address and password to log into your mail ac-
155 # count, agree to accessing your project, copy the authorizat-
156 # ion code then provided, paste it after the text "Enter ver-
157 # ification code:" in the terminal window where this script is
158 # still running, and hit Enter.
159 #
160 # "update": Update the access token, except when it is younger than the
161 # maximum age specified in configuration file "oauth.cfg".
162 #
163 # no arg: Only when "fetchmail" is running update the access token as
164 # if "update" has been specified.
165
166 dir=$(dirname $(realpath "$0")) # Symlink-free abs path to script dir.
167 cfg="--config_file=$dir/oauth.cfg" # Configuration file option.
168
169 if [[ "${1-}" = init ]]
170 then "$dir/oauth.py" "$cfg" --obtain_refresh_token_file
171 elif pgrep -u $USER fetchmail > /dev/null || [[ "${1-}" = update ]]
172 then "$dir/oauth.py" "$cfg" --auto_refresh
173 fi
174
175 Now keeping the above comment regarding "init" in mind I executed:
176
177 $ ~/.../oauth.sh init
178
179 to get my first OAuth 2 access tokens. I then created:
180
181 1. A new "fetchmail" alias (which will also be run upon login):
182
183 $ alias fetchmail='~/.../oauth.sh update ; fetchmail'
184
185 2. A "crontab" entry for my own userid (don't forget to add yourself to
186 group "cron"):
187
188 # When "fetchmail" is started, it will request a new "OAuth" access tok-
189 # en from Google as soon as the current token is older than 30 minutes.
190 # And as long as "fetchmail" is running, this "cron" job, too, will re-
191 # new the "OAuth" access token when it is older than 30 minutes. Since
192 # we start this "cron" job three times an hour, this will result in acc-
193 # ess tokens (except for the first one, which will be refreshed after 30
194 # to 50 minutes) being refreshed every 40 minutes. This fits well with
195 # the maximum life time of 60 minutes for any access token.
196
197 11,31,51 * * * * $HOME/.../oauth.sh
198
199 3. The hook script "/lib64/elogind/system-sleep/oauth-token.sh" for the
200 "elogind" service:
201
202 #! /bin/bash -u
203 #
204 # When resuming from hibernation or suspension update Google's OAuth to-
205 # ken for user "rainer".
206 #
207 # This script will be called by "elogind" with exactly two arguments:
208 #
209 # $1: Either "pre" or "post".
210 # $2: Either "hibernate", "hybrid-sleep", "suspend", or "suspend-then-
211 # hibernate".
212 #
213 # The "sleep" command is required to give the host some time connecting
214 # the WiFi before running "oauth.sh", and taking this nap in the back-
215 # ground prevents it from delaying "elogind" while firing up the WiFi:
216
217 [[ "$1" = pre ]] || (sleep 10 ; sudo -u rainer ~rainer/.../oauth.sh) &
218
219 Now it's time to just start "fetchmail" via its new alias and to catch
220 up with one's incoming mail ...
221
222 However, I did not follow Matthew's suggestions in part 4 regarding the
223 configuration of "mail-mta/postfix" because the configuration effort for
224 a full-fledged, system wide MTA as well as adding another "ebuild" file
225 to my local overlay seemed like overkill to me. Up to now I was using
226 "ssmtp" for sending mail, and if I remember correctly, the first "s" in
227 this name was referring to "simple". Its configuration file just con-
228 tained eight lines. Looking around I found Gentoo package "mail-mta/
229 msmtp", which supports OAuth 2 and also a user local configuration file
230 at "~/.msmtprc". Here are the relevant lines from mine:
231
232 # Name the only account defined here "default", so it overrides the def-
233 # ault account defined in file "/etc/msmtprc":
234
235 account default
236
237 auth oauthbearer
238 domain gmail.com
239 host smtp.gmail.com
240 passwordeval cat ~/.../oauth-access-token
241 port 587
242 protocol smtp
243 tls on
244 tls_certcheck on
245 tls_starttls on
246 user rainer.woitok@×××××.com
247
248 Now the only thing left to do is telling one's MUA not to pipe outgoing
249 mail directly to the standard "/usr/sbin/sendmail", but rather to script
250 "~/.../msmtp.sh" instead, so we can refresh the OAuth access token, if
251 necessary, before really sending off the mail. Script "~/.../msmtp.sh"
252 contains:
253
254 #! /usr/bin/bash -u
255 #
256 # Before sending the mail refresh the access token, if necessary, and
257 # since the mail to be sent is provided on standard input, route stand-
258 # ard input around the call to "oauth.sh":
259
260 { cat >&3 ; ~/.../oauth.sh update ; } 3>&1 | msmtp "$@"
261
262 Should you MUA not provide a way for specifying the programme to which
263 outgoing mail is piped, you can simply modify the symbolic link "/usr/
264 sbin/sendmail" (which has just been set by package "mail-mta/msmtp" to
265 point to "../bin/msmtp"):
266
267 # cd /usr/sbin
268 # rm sendmail
269 # ln -s ~rainer/.../msmtp.sh sendmail
270
271 That's all, happy mailing ... :-)
272
273 Sincerely,
274 Rainer
275
276 PS: This mail was sent to you using "mail-mta/msmtp" set-up the way de-
277 scribed above ... :-)