Re: [gentoo-project] call for agenda items -- council meeting 2021-05-09

[Alec Warner]

On Sat, May 1, 2021 at 8:14 AM Andreas K. Huettel <dilfridge@g.o> wrote:
>
> > Please respond to this message with any items you would like us to
> > discuss or vote on. The agenda will be sent to this list a week from
> > now (2021-05-02).
>
> There's rumors going around that Agostino's tinderbox basically
> already burned through our entire 2021 AWS Open Source credit budget
> and has overdrawn it significantly.

Practically speaking I manage the AWS budget. If you are unhappy with
how it's allocated; feel free to follow up with me. Developers who use
AWS don't get a ton of say in the spending; I decide how much money we
have, how many machines to buy, how big they are, etc. Developers
can't even see their own cost usage.

>
> Independent of whether this rumor is true or not, I would like to
> discuss
> * how we can avoid such a situation in the future,
> * and how we can fairly handle the distribution of limited infra
> resources that may end up costing us money

Ack, so I'll respond with the same things I said on IRC (hopefully, haha.)

The AWS program runs November 2020 - November 2021. We were provided
25,000$ in credits; these are promotional credits so we cannot do
things like use Cost Savings Plans or Reserved Instances (two ways to
reduce cost on AWS.)

This is our second year in the program. In the first year we spent
very little of the budget (I think under 5000$) and I viewed this as
something of a failure; we should be in a position to use more of the
budget. When the program was renewed in November I wanted to spend
more of the budget.

When the plan restarted I gave folks more VMs that were bigger; with
the idea that we would both track utilization (are the VMs being used)
and cost (are we on track to spend well.)
Current program state is as follows:
 - We fund 4 VMs.
 - We have spent 14,200 of 25,000 credits.
 - We are 5/7ths through the 12 month program (it ends Nov 31 2021)
 - We have 10,800 credits left, for 7 months.

The plan then:
 - At our current spend, we will run out of credits.
 - 7 / 10,800 ~= 1500$ / mo average spend we need to hit.
 - Our current spend is ~$4000 / mo.
 - We have to cut our spending significantly.

The risk:
 - We could run out of credits and incur costs. Assuming we did
nothing, we spend approximately 4000$ a month. We have 7 months left;
so that is 28000$ - 10800 (our remaining credits) leaves us with a
bill of about 18000$.
 - We could end up not getting the program renewed at the end of the
cycle, and never get additional credits.

Both of these would likely result in us needing to relocate services
to other providers. It's a known risk (and we have other onPrem
providers that we have similar risks for.) I suspect practically we
would rent hardware from OVH or Hetzner and migrate services to those.
Unlike AWS, those providers cost money; which is one reason why we
would prefer to spend the AWS credits instead.

For specific projects using the credits we could likely self-fund a
physical machine. These machines are  not free (folks were estimating
something like 60 euro a month to lease one.) My map here is "I'd
rather spend AWS credits than pay 60 euro a month." If the credits run
out or we don't get renewed in the program for 2022 we can talk about
other options. Maybe we should lease a machine for stage building (as
an example.)

My process for resource allocation:
 - We currently have more resources than needs. While this is not true
of AWS credits specifically; it is true of money in general.
 - This leads us to a low barrier for resource allocation. There is
not scarcity, so the process to get resources is not intended to be
much of a barrier.
 - Once there is scarcity, we will have different conversations.

One risk I have heard is "why don't you reserve some of the credits
for if something cool comes up?" or "how would we fund some cool new
idea?" and the answer is "with actual money." The Foundation has
enough money[0] to weather most of the bad outcomes (e.g. we could
fund AWS for a month while we migrate away, we could buy hardware, we
could rent hardware, etc. etc.)

I consider the AWS credits as part of our cash-like reserves. We are
spending them first because they expire and because they are not
actually cash. We still have more cash than expenses; so in general I
don't see a ton of pressure on Foundation expenses (and thus there is
not intended to be a complex process for spending money[1].)

I hope this helps; please follow up with me if you have additional
questions or concerns.

-A

[0] We paid our taxes for the past few years and we have already filed
our 2020 taxes; so there are no known large outstanding obligations.
You can see our 2020 numbers here:
https://wiki.gentoo.org/wiki/Foundation:Gentoo_Foundation_Finances_FY2020
[1] There is an argument about spending Foundation resources 'well'
(for some definition of that word) and while I empathize with it quite
a bit, currently I don't think Foundation supports development very
much at all; so my thought process is "try to support more development
in general" and if we are not supporting development "well" (e.g. we
think there is waste, etc) we can iterate on that. Get it working
first, optimize second; if you will.

>
> Cheers,
> Andreas
>
> --
> Andreas K. Hüttel
> dilfridge@g.o
> Gentoo Linux developer
> (council, toolchain, base-system, perl, libreoffice)