Re: [gentoo-dev] the state of dev-lang/lua

[Robin H. Johnson]

On Sat, Mar 23, 2019 at 04:23:27PM -0500, William Hubbs wrote:
> Hi all,
>
> Soon I will be working on fixing up the state of dev-lang/lua, and there
> are a couple of things I want to mention.
>
> The first thing is liblua as a shared library. If you are using lua
> internally in a program, upstream strongly recommends not linking it
> this way; it is supposed to be statically linked into the executable.
> Because of this, and because of the amount of custom patching we do to
> maintain liblua as a shared library, I plan to stop creating the shared
> library.
Please don't go back to static libraries. Look at the other major
distros, all of them shipped shared Lua as the primary method.

I'm just going to link to the Ubuntu examples here, but Debian, Fedora &
Arch are similar:
https://packages.ubuntu.com/source/cosmic/lua5.1
https://packages.ubuntu.com/source/cosmic/lua5.2
https://packages.ubuntu.com/source/cosmic/lua5.3
https://packages.ubuntu.com/cosmic/amd64/liblua5.1-0/filelist
https://packages.ubuntu.com/cosmic/amd64/liblua5.2-0/filelist
https://packages.ubuntu.com/cosmic/amd64/liblua5.3-0/filelist

Ubuntu & Debian DO ship the static library .a file, as part of
the liblua5.X-dev binary package.

That said, the pkgconfig files we output for Lua5.2 & Lua5.3 do have
errors in some cases, bringing in the non-slotted include directory, and
trying to use the non-slotted library soname; which need to be fixed
properly.

Upstream's original objections about static libraries mostly revolved
around the lack of registers on i386 with -fPIC, as well as subtle
breakage when the library SONAME was just generically 'liblua', without
much version information, as this broke when functions were dropped.

> I'm a bit undecided still about slotting lua. I'm sure we
> need subslots so we can force rebuilds when new lua releases enter the
> tree. However, I'm still unsure whether we need slots. I don't know of
> many things in the tree that are locked to a specific version
> of lua (there is only one package based on an irc conversation I
> had this week).
> Does anyone have any thoughts?
Lua needs first class slots, just like Python & Ruby, not just subslots.
Changing between versions can be a major undertaking.

I think the slots to start with should be:
- lua5.1
- lua5.2
- lua5.3
- luajit5.1 (this is basically an alternative implementation of Lua5.1,
much like pypy implements Python2).

Porting between versions is non-trivial, as the API has changed in
incompatible ways.

There are two tricks to make porting easier, but they are not
guaranteed. 
1. USE=deprecated on Lua5.2/5.3 offers most of the prior version APIs;
   mostly from Lua5.3 back to Lua5.2 & Lua5.1 with the LUA_COMPAT_5_2 &
   LUA_COMPAT_5_1 compile-time defines.
2. In the other direction, there is 'lua-compat-5.3', which we don't
   have packaged; It offers parts of Lua-5.3 functionality to
   Lua-5.1/5.2. I'll quote from the docs:
   "This does not make Lua 5.2 (or even Lua 5.1) entirely compatible
   with Lua 5.3, but it brings the API closer to that of Lua 5.3."
   That project maintains a good list of what's not implemented:
   https://github.com/keplerproject/lua-compat-5.3#whats-not-implemented

> Are there more packages in the tree that are locked to a specific version of lua?
Yes, several, and more on that in a second.

I think packages will need one of two variables:
LUA_SINGLE_TARGET
LUA_TARGETS

LUA_TARGETS is probably only going to be used by the dev-lua/ modules.
LUA_SINGLE_TARGET is probably going to be used by MOST packages that
include Lua as an embedded scripting language; of which I'll discuss a
few below.

The pain in Lua thusfar I see comes when one of the packages with
embedded Lua scripting needs to consume a system-wide Lua module.

Some Lua modules provide special-case handling for luajit, to ensure
best performance.

I personally work on net-proxy/haproxy, which supports ONLY Lua 5.3. It
relies on some functionality that cannot be back-ported to earlier
releases (heavy usage of coroutine/yield & uint64)

I'm also about to add back sys-apps/likwid to the tree, for which the
upstream bundles Lua 5.3, but the code explicitly is written to support
a system-wide copy of Lua 5.2 or Lua 5.3. The ebuild now uses the
system-wide copy of Lua 5.3. likwid ships as both a set of Lua scripts
that bring in a special C library, as well as Lua scripting placed back
into that C library.

x11-wm/awesome is a good example of a package with support for multiple
versions of Lua. For a long time upstream only supported Lua 5.1, but in
the recent major versions of Awesome, the upstream dev has Lua 5.2 & Lua
5.3 support with a test environment. I'm running on my own local ebuild
with Awesome linked against Lua 5.3 as I write this.

dev-db/redis is a good example of where Upstream tried to port from 5.1
to newer versions, but did an incomplete job:
- Try to compile it with Lua5.3 and it falls over numeric types.
- Try to compile it with Lua5.2 and it falls over the
  lua_open/lua_newstate.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@g.o
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136