Time |
Nick |
Message |
01:13 |
|
tekakutli joined #minetest-dev |
01:42 |
HuguesRoss |
Can someone clarify the relationship of a couple PRs for me? I'm looking at #10486 and #11438, both aim (at least in part) to fix issue #8601. However, the latter of the two has an approval and is assigned to a milestone while the former hasn't had any activity in ~6 months but seems to be waiting on review...should we consider closing it, or has it been left open for a reason? |
01:42 |
ShadowBot |
https://github.com/minetest/minetest/issues/10486 -- Fix optional dependencies causing cascading failures with dependent mods by IceDragon200 |
01:42 |
ShadowBot |
https://github.com/minetest/minetest/issues/11438 -- Mods: Improve mod loading priorities (2nd) by SmallJoker |
01:42 |
ShadowBot |
https://github.com/minetest/minetest/issues/8601 -- Optional dependencies being overridden/turned into hard dependencies |
01:46 |
MTDiscord |
<Jonathon> the latter before it was closed and reopened a few? times said it superseeded the other |
01:46 |
HuguesRoss |
Yes, that's what it looks like |
01:47 |
HuguesRoss |
Today, it's open with a milestone and one approval, so my initial instinct is that it'll probably be the one that gets in |
01:49 |
MTDiscord |
<Jonathon> maybe its like the fonts, both are left open till one wins (gets merged) |
02:10 |
|
v-rob joined #minetest-dev |
02:23 |
|
tekakutli joined #minetest-dev |
03:29 |
|
queria joined #minetest-dev |
05:00 |
|
MTDiscord joined #minetest-dev |
05:27 |
|
m42uko_ joined #minetest-dev |
05:40 |
|
MTDiscord joined #minetest-dev |
05:40 |
|
nrz joined #minetest-dev |
05:40 |
|
asdflkj_sh joined #minetest-dev |
05:41 |
|
nrz joined #minetest-dev |
05:42 |
|
Taoki joined #minetest-dev |
06:13 |
|
v-rob joined #minetest-dev |
07:18 |
|
v-rob joined #minetest-dev |
07:34 |
|
ivanbu joined #minetest-dev |
08:11 |
|
tekakutli joined #minetest-dev |
08:42 |
|
v-rob joined #minetest-dev |
09:20 |
|
proller joined #minetest-dev |
10:34 |
|
appguru joined #minetest-dev |
10:40 |
|
appguru joined #minetest-dev |
11:06 |
|
HuguesRoss joined #minetest-dev |
11:36 |
|
tech_exorcist joined #minetest-dev |
11:43 |
|
proller joined #minetest-dev |
12:19 |
|
appguru joined #minetest-dev |
12:23 |
|
appguru1 joined #minetest-dev |
12:49 |
|
gaelysam joined #minetest-dev |
13:35 |
|
appguru joined #minetest-dev |
13:36 |
MTDiscord |
<luatic> HuguesRoss: be careful |
13:36 |
MTDiscord |
<luatic> this has the potential to cause massive breakage |
13:36 |
MTDiscord |
<luatic> and I fear Krock's implementation might be incorrect |
13:38 |
sfan5 |
that's why it's in the milestone for 5.6 ;) |
13:39 |
MTDiscord |
<luatic> I'll try to sum up the issue |
13:39 |
MTDiscord |
<luatic> Because the way I see it, we're still lacking expected behavior for the handling of "optional" dependencies |
13:41 |
MTDiscord |
<luatic> The docs say the following: "Like a dependency, but no error if the mod doesn't exist." |
13:41 |
MTDiscord |
<luatic> So what should happen if there are circular optional dependencies among enabled mods? |
13:41 |
|
HuguesRoss joined #minetest-dev |
13:42 |
MTDiscord |
<luatic> Simplest (and most understandable) way would be to throw an error, effectively turning "optional" dependencies into "hard" dependencies if the mod exists. |
13:43 |
MTDiscord |
<luatic> Alternative is trying to maximize the number of fulfilled optional dependencies |
13:44 |
MTDiscord |
<luatic> This might lead to "weird" (unexpected) behavior that a mod is an enabled, optional dependency, yet loads after it's dependant. |
13:46 |
MTDiscord |
<luatic> This is effectively a topological sorting |
13:48 |
MTDiscord |
<luatic> See modlib for an example Lua implementation: https://github.com/appgurueu/modlib/blob/master/minetest/misc.lua#L242-L323 |
13:49 |
|
Wuzzy joined #minetest-dev |
13:49 |
sfan5 |
'turning "optional" dependencies into "hard" dependencies if the mod exists' <- wasn't this the exact behaviour that was considered bad? |
13:50 |
|
calcul0n joined #minetest-dev |
13:55 |
erlehmann |
the toposort thing seems true from my POV |
13:55 |
erlehmann |
i agree with luatic that the semantics of optional dependencies are at least a bit weird |
13:58 |
sfan5 |
how do other package managers solve this? |
13:59 |
MTDiscord |
<luatic> sfan5: I believe package managers don't bother usually, as they just install the required deps (often you must install "recommended" opt. deps yourself) |
14:00 |
sfan5 |
hm yeah I guess this exact problem does not apply to them |
14:00 |
|
Taoki joined #minetest-dev |
14:03 |
MTDiscord |
<luatic> As said, we must define how we want optional dependencies to work |
14:03 |
MTDiscord |
<luatic> It's odd if optional dependencies don't get fulfilled arbitrarily in circular cases |
14:04 |
MTDiscord |
<luatic> I don't like Krock's PR as it isn't optimal |
14:05 |
MTDiscord |
<luatic> A recursive solution like #10486 is the way to go |
14:05 |
ShadowBot |
https://github.com/minetest/minetest/issues/10486 -- Fix optional dependencies causing cascading failures with dependent mods by IceDragon200 |
14:05 |
erlehmann |
sfan5 package managers have required and different levels of optional: suggests, enhances, recommends |
14:06 |
erlehmann |
sfan5 i believe only required is a strict dependency in terms of topological sort. everything else does not care about loading order. for example, ghostscript does not have to be installed before gimp, as it is merely recommended. |
14:07 |
MTDiscord |
<luatic> MT's optional dependencies often very much care about loading order |
14:07 |
erlehmann |
i agree with luatic that a recursive solution is probably the way to go. i have seen a lot of ”1. toposort 2. load” approaches turn out badly. |
14:33 |
rubenwardy |
<sfan5> 'turning "optional" dependencies into "hard" dependencies if the mod exists' <- wasn't this the exact behaviour that was considered bad? |
14:33 |
rubenwardy |
The problem was because the optional dependencies was then disabled, I believe |
14:34 |
rubenwardy |
I'm in favour of making failing to load rather than disabling mods when there are missing dependencies |
14:35 |
rubenwardy |
if an optional dependency exists, then it should be considered a hard dependency from that point on. So if you get a dependency cycle, fail to load rather than randomly disabling mods |
14:35 |
MTDiscord |
<Warr1024> Silent failures are bad in general, an starting up a world with requested mods not actually loaded would be a case of that. |
14:35 |
rubenwardy |
yeah. It's confusing to users |
14:38 |
rubenwardy |
<rubenwardy> The problem was because the optional dependencies was then disabled, I believe |
14:38 |
rubenwardy |
more accurately, the problem is with mods being disabled in a chain. If you error instead, it becomes a lot more usable |
14:40 |
MTDiscord |
<luatic> That's exactly what I wanted to hear |
14:40 |
MTDiscord |
<Warr1024> If I tell it to load a mod and that mod fails to load, then everything should just abort at that stage. It's not even relevant how mods should respond to their dependencies being present but not loaded because failing to load exactly the set of mods I demanded is already a fatal failure case. |
14:41 |
MTDiscord |
<luatic> 100% agreed, I even filed a bug as the errors sometimes don't get logged to chat |
14:41 |
rubenwardy |
I had a security issue on CTF because of this |
14:41 |
rubenwardy |
the dependency system silently disabled the mod that adds moderation features, such as locking chat commands behind a threshold of activity |
14:42 |
MTDiscord |
<Warr1024> It makes upgrades a nightmare too, because any time you change anything about your setup, mod version or engine version, you run the risk of one brick disappearing and taking out the whole wall without you even realizing. |
14:42 |
rubenwardy |
I'd like to refactor the mod/pkg system to use a single code base, rather than being split into C++ and Lua. Then I'd like to update the Select Mods to call the dependency resolver and highlight any mods that are missing deps or have cycles |
14:42 |
MTDiscord |
<Warr1024> Basically having any mod fail to load is already a fatal error, it's just that we have a human-in-the-loop process for actually detecting it. |
14:59 |
|
queria joined #minetest-dev |
15:40 |
nore |
Just giving my $0.02 on the issue: for me, the "intuitive" semantics of "A soft-depends on B" is: "if A and B are both enabled (not only exist), then B is loaded before A" |
15:41 |
MTDiscord |
<Warr1024> I had assumed that "exist" in this case meant "exist within the set of mods allowed to be loaded in this world" |
15:41 |
nore |
I need to check minetest.get_modpath, but I expect minetest.get_modpath to return nil if the mod provided as argument is present but not enabled |
15:42 |
nore |
yep, same |
15:44 |
nore |
For me, all functions that check if some mods exist or this kind of things should behave identically if the mod is in the mod folder but not enabled or if the mod is not in the mod folder |
15:47 |
nore |
(also, I can write a topological ordering with linear complexity if needed) |
15:48 |
|
erlehmann joined #minetest-dev |
15:48 |
|
v-rob joined #minetest-dev |
15:49 |
MTDiscord |
<luatic> nore: We're considering pathological cases here, so the question is: What if A and B soft-depend on each other (cycle) and are both enabled? Should it throw an error or just silently "violate" one of the two optional dependencies? |
15:50 |
MTDiscord |
<luatic> I prefer the error |
15:51 |
MTDiscord |
<Benrob0329> Optional Deps are still deps, they need to make sense or there's no right way to resolve them. |
15:51 |
MTDiscord |
<luatic> Yeah, but the word "optional" insinuates that they needn't be resolved |
15:51 |
erlehmann |
i believe the first step is to clarify what “optional dependency” *means* |
15:51 |
nore |
according to the semantics I gave above, an error should occur |
15:51 |
MTDiscord |
<luatic> erlehmann: absolutely |
15:51 |
MTDiscord |
<luatic> the docs are too vague currently |
15:52 |
erlehmann |
i *think* that it indeed involves “if this exists, load it before the current thing” |
15:52 |
MTDiscord |
<luatic> https://github.com/minetest/minetest/blob/master/doc/lua_api.txt#L209 |
15:52 |
nore |
that's why I wanted to give a precise semantics for it |
15:52 |
MTDiscord |
<luatic> "Like a dependency, but no error if the mod doesn't exist." |
15:52 |
MTDiscord |
<luatic> So according to this, it becomes a hard dependency if the mod "exists" (is enabled). |
15:52 |
erlehmann |
luatic which is not precise enough, as you pointed out yeah |
15:52 |
erlehmann |
debatable |
15:52 |
MTDiscord |
<Benrob0329> Luatic: that seems sane and expected to me |
15:52 |
erlehmann |
because “no error” does not fit with circular deps |
15:53 |
nore |
the "no error" is only if the mod doesn't exist |
15:53 |
MTDiscord |
<Benrob0329> Doesn't exist is not the same as exists and is broken |
15:53 |
erlehmann |
ah sorry |
15:53 |
MTDiscord |
<luatic> it is "no error if the mod doesn't exist" |
15:53 |
erlehmann |
nore luatic i should take a nap sorry. |
15:53 |
MTDiscord |
<luatic> the only issue I see with the current wording is the "exists" |
15:54 |
MTDiscord |
<luatic> it implies that even disabled mods turn soft dependencies into hard dependencies |
15:54 |
MTDiscord |
<luatic> WAIT |
15:54 |
nore |
yeah, the semantics of exists should be clarified to mean that the mod should be enabled |
15:54 |
MTDiscord |
<luatic> according to this, the issue is not a bug |
15:54 |
MTDiscord |
<luatic> #8601 is not a bug LMAO |
15:54 |
ShadowBot |
https://github.com/minetest/minetest/issues/8601 -- Optional dependencies being overridden/turned into hard dependencies |
15:55 |
erlehmann |
if minetest is made into a movie, it needs to be directed by m. night shyamalamadingdong |
15:55 |
erlehmann |
twists upon twists |
15:56 |
nore |
https://github.com/minetest/minetest/blob/master/doc/lua_api.txt#L4569 |
15:56 |
MTDiscord |
<luatic> https://github.com/minetest/minetest/issues/8601#issuecomment-1011190822 |
15:56 |
nore |
So minetest.get_modpath is well-documented |
15:56 |
MTDiscord |
<luatic> Apparently "exists = installed" as expect |
15:57 |
MTDiscord |
<luatic> Funnily enough this forces you to enable optional dependencies, doesn't it? |
15:57 |
nore |
I'd still say it is a bug, exists = installed means installing a new mod without enabling it can prevent worlds for loading |
15:57 |
MTDiscord |
<luatic> Actually the bug might be that mods still load despite some of their optional dependencies being disabled |
15:58 |
nore |
*from |
15:58 |
MTDiscord |
<luatic> The question is: What do we consider expected behavior? |
15:58 |
MTDiscord |
<luatic> The docs are rather clear (yet silly) on this |
15:59 |
MTDiscord |
<GoodClover> I interpret optional_deoendency as "load after if enabled". |
16:00 |
MTDiscord |
<GoodClover> Loading an optional dependancy after the dependent mod would be stupid and likely to break things. |
16:11 |
erlehmann |
i have a salomonic solution |
16:11 |
erlehmann |
you have the entire cdb |
16:11 |
erlehmann |
you have the dependency tree of the entire cdb |
16:11 |
MTDiscord |
<luatic> hehe |
16:11 |
MTDiscord |
<luatic> I like this |
16:11 |
MTDiscord |
<luatic> will do |
16:11 |
erlehmann |
surely rubenwardy can give us access to the cdb thing |
16:11 |
MTDiscord |
<luatic> we don't need access |
16:11 |
erlehmann |
and then we can empirically figure out what people actually think |
16:11 |
MTDiscord |
<luatic> the API suffices |
16:11 |
erlehmann |
ah well but then you have to scrape it? |
16:12 |
erlehmann |
but yeah |
16:12 |
MTDiscord |
<luatic> I don't have to scrape it |
16:12 |
MTDiscord |
<luatic> The JSON API fully returns depends |
16:12 |
erlehmann |
i think the solution is to try to figure out how people emprically use stuff |
16:12 |
MTDiscord |
<luatic> I don't use it any more than any normal MT client I believe |
16:12 |
erlehmann |
luatic the thing is, you have to do it for *all* packages |
16:12 |
erlehmann |
this is kind of a big task |
16:12 |
erlehmann |
figure out which circular optional dependencies exist |
16:12 |
MTDiscord |
<luatic> it's not |
16:13 |
erlehmann |
if none exist |
16:13 |
MTDiscord |
<luatic> only thousands of mod |
16:13 |
erlehmann |
we are lucky |
16:13 |
MTDiscord |
<luatic> finding a cycle is linear time |
16:13 |
erlehmann |
i mean it is a big task compared to just defining something arbitrarily |
16:13 |
erlehmann |
but i think it is the best way forward for the discussion |
16:13 |
MTDiscord |
<luatic> aww too bad, https://content.minetest.net/api/packages/?type=mod doesn't contain the depends |
16:13 |
MTDiscord |
<luatic> erlehmann: I'm wondering now |
16:14 |
erlehmann |
wondering what |
16:14 |
MTDiscord |
<luatic> should I check only among mods? |
16:14 |
MTDiscord |
<luatic> Or mods + games? |
16:14 |
MTDiscord |
<luatic> games + games kinda makes no sense... |
16:14 |
erlehmann |
i would check among games first. because they constitute mods that are definitely used together. |
16:15 |
erlehmann |
and then try to find cycles in the rest |
16:17 |
MTDiscord |
<luatic> I wouldn't check inside games |
16:17 |
MTDiscord |
<luatic> That's hardly possible without.. actual scraping |
16:17 |
|
Fixer joined #minetest-dev |
16:17 |
MTDiscord |
<luatic> Just among different CDB packages |
16:18 |
MTDiscord |
<luatic> Basically, what if you enabled all mods - would there be an optional dependency cycle? |
16:18 |
MTDiscord |
<Warr1024> Some games do have optional depends on other stuff, so they can be involved in cycles, at least... |
16:37 |
MTDiscord |
<luatic> https://gist.github.com/appgurueu/31b5b7a6ac5ae5f34b4b3054341763fd - Lua PoC for a mod load order implementation, test it out |
16:50 |
erlehmann |
luatic i hereby want to warn you: if you actually enable about a thousand mods, minetest takes very long to start |
16:50 |
erlehmann |
even if those are just empty |
16:50 |
erlehmann |
and contain nothing but meta info |
16:51 |
erlehmann |
okay and an empty init.lua bc otherwise nothing works |
16:51 |
erlehmann |
> -- Determines the load order by finding roots for the mod forest and then doing a DFS |
16:51 |
erlehmann |
poetic! |
16:52 |
MTDiscord |
<luatic> erlehmann: don't worry, I'll just simulate it |
16:52 |
MTDiscord |
<luatic> The current algo is most likely trash |
17:00 |
|
tech_exorcist joined #minetest-dev |
17:05 |
Pexin |
out of order optional loads cannot be an option because mods are not like linux packages for example in that elements of mods can be specifically overridden by later loading mods |
17:09 |
rubenwardy |
luatic, erlehmann: https://content.minetest.net/api/dependencies/ |
17:10 |
MTDiscord |
<luatic> rubenwardy: thanks, found that already |
17:10 |
rubenwardy |
I added it just now |
17:10 |
MTDiscord |
<luatic> this is my script so far |
17:10 |
MTDiscord |
<luatic> https://cdn.discordapp.com/attachments/747163566800633906/930871419368263690/scan-cdb.lua |
17:10 |
MTDiscord |
<luatic> NVM msiread |
17:10 |
erlehmann |
neat rubenwardy |
17:11 |
MTDiscord |
<luatic> thought you meant the per-package dependency API |
17:11 |
rubenwardy |
this one returns for all packages, which will be more efficient for that usecase |
17:11 |
MTDiscord |
<luatic> I guess this will speed things up ;) |
17:11 |
rubenwardy |
it supports filtering as well |
17:11 |
rubenwardy |
https://content.minetest.net/api/dependencies/?type=game&type=mod |
17:12 |
MTDiscord |
<luatic> yay |
17:12 |
rubenwardy |
maybe it should filter out txps by default, as they can't have deps |
17:12 |
MTDiscord |
<luatic> argh this is tricky because of forks |
17:13 |
rubenwardy |
yeah, CDB also won't tell you which specific mod in a package depends on what |
17:17 |
|
fluxionary joined #minetest-dev |
17:17 |
|
tech_exorcist joined #minetest-dev |
17:21 |
MTDiscord |
<luatic> looks like there's a bug in my PoC |
17:24 |
MTDiscord |
<luatic> bruh ruben, why the pagination |
17:24 |
MTDiscord |
<luatic> is there are a way to set per_page? |
17:28 |
MTDiscord |
<luatic> erlehmann, rubenwardy: looks like we've got our first circular dependency "moreblocks<-moretrees<-flowerpot<-frame<-moreores<-basic_materials<-moreblocks" |
17:28 |
MTDiscord |
<Warr1024> pagination is often done to limit the ability of a single request to impact server resources |
17:29 |
rubenwardy |
takes 15 seconds to load unpaginate |
17:29 |
rubenwardy |
+d |
17:29 |
MTDiscord |
<luatic> ah alright |
17:39 |
Krock |
will merge #11942 in 10 minutes |
17:39 |
ShadowBot |
https://github.com/minetest/minetest/issues/11942 -- Fix NodeDef backwards compatibility to 5.3.0 by SmallJoker |
17:43 |
MTDiscord |
<luatic> Alright, so here are the cycles my script found among only mods: |
17:44 |
MTDiscord |
<luatic> moreblocks<-moretrees<-mg<-moreores<-basic_materials<-moreblocks<-ethereal<-techpack |
17:44 |
MTDiscord |
<luatic> technic<-appliances<-clothing<-skinsdb<-i3<-technic legendary_armor<-legendary_ore<-legendary_armor |
17:46 |
MTDiscord |
<Warr1024> that last one is legendary_facepalm... |
17:46 |
MTDiscord |
<Warr1024> obviously-same-author should know better than to create a trivial loop within their own stuff |
17:47 |
rubenwardy |
are you ignoring modpacks there? |
17:48 |
MTDiscord |
<luatic> Warr1024: yeah, the two mods have each other as optional dependencies |
17:48 |
MTDiscord |
<luatic> rubenwardy: ?type=mod |
17:48 |
MTDiscord |
<luatic> here it is |
17:48 |
MTDiscord |
<luatic> https://cdn.discordapp.com/attachments/747163566800633906/930880968812822579/scan-cdb.lua |
17:48 |
Krock |
merging |
17:49 |
rubenwardy |
you want to use mod.provides, not mod.name |
17:49 |
rubenwardy |
otherwise you won't detect cycles with mods in modpacks that don't have the same name as the modpack |
17:50 |
MTDiscord |
<luatic> So ?type=mod returns modpacks too... |
17:50 |
rubenwardy |
yeah |
17:50 |
MTDiscord |
<luatic> sigh, i need this at a mod level |
17:50 |
rubenwardy |
CDB doesn't have that info |
17:50 |
MTDiscord |
<luatic> yeah |
17:50 |
MTDiscord |
<luatic> I'd have to scrape the mods etc. |
17:51 |
MTDiscord |
<luatic> But I think just ignoring modpack is good enough for now |
17:51 |
MTDiscord |
<luatic> if #mod.provides == 1 and mod.provides[1] == mod.name then ... end |
17:52 |
MTDiscord |
<luatic> the "technic" one was a false positive |
17:52 |
MTDiscord |
<luatic> so we have VE's couple mods (probably more in modpacks) and someone who doesn't know how to use opt. deps (the legendary guy) |
17:57 |
MTDiscord |
<luatic> lol, this code is mutually exclusive: https://github.com/gitgithu/legendary_armor/blob/main/init.lua#L250-L256 and https://github.com/gitgithu/legendary_ore/blob/main/init.lua#L31-L40 |
17:58 |
MTDiscord |
<luatic> I guess we needn't care about the legendary stuff, it's poorly made - a direct cyclic optional dependency simply is unreasonable. |
18:26 |
|
Evergreen5 joined #minetest-dev |
18:27 |
|
bigfoot547 joined #minetest-dev |
18:28 |
|
ROllerozxa_ joined #minetest-dev |
18:29 |
|
[0] joined #minetest-dev |
18:30 |
|
Calinou_ joined #minetest-dev |
18:38 |
|
tech_exorcist joined #minetest-dev |
18:38 |
|
fluxionary joined #minetest-dev |
18:38 |
|
Taoki joined #minetest-dev |
18:38 |
|
Wuzzy joined #minetest-dev |
18:38 |
|
HuguesRoss joined #minetest-dev |
18:38 |
|
asdflkj_sh joined #minetest-dev |
18:38 |
|
MTDiscord joined #minetest-dev |
18:38 |
|
Yad joined #minetest-dev |
18:38 |
|
Pexin joined #minetest-dev |
18:38 |
|
ShadowBot joined #minetest-dev |
18:38 |
|
behalebabo joined #minetest-dev |
18:38 |
|
wsor joined #minetest-dev |
18:38 |
|
Krock joined #minetest-dev |
19:23 |
|
tech_exorcist_ joined #minetest-dev |
19:27 |
|
Fixer_ joined #minetest-dev |
19:35 |
|
olliy joined #minetest-dev |
19:44 |
|
proller joined #minetest-dev |
20:24 |
|
tech_exorcist joined #minetest-dev |
20:28 |
|
x2048 joined #minetest-dev |
20:36 |
|
Alias joined #minetest-dev |
21:16 |
|
v-rob joined #minetest-dev |
23:11 |
|
Fixer joined #minetest-dev |
23:33 |
|
YuGiOhJCJ joined #minetest-dev |
23:44 |
|
v-rob joined #minetest-dev |