| Time |
Nick |
Message |
| 01:41 |
|
v-rob joined #minetest-dev |
| 04:00 |
|
MTDiscord joined #minetest-dev |
| 06:10 |
|
v-rob joined #minetest-dev |
| 06:30 |
sfan5 |
"In practice, the direct mode is safe to use in the absolute majority of cases." |
| 06:37 |
erle |
yeah, this is what about everyone says who can't be arsed to implement dependency tracking correctly. that it's not a problem “in practice”. |
| 06:37 |
erle |
the majority of them are liars. |
| 06:38 |
erle |
because there exists very few software to figure out if it is a problem. |
| 06:39 |
erle |
and every software that *i* know that can help you to figure out if it is a problem does this either by implementing correct dependency tracking – in which case you do not hae the problem … |
| 06:39 |
erle |
… or by fully rebuilding it and comparing it with a partial build, which only works if you a) have reproducible builds b) can afford the time to wait for a full rebuild. |
| 06:40 |
erle |
(none of which applies here AFAIK) |
| 06:41 |
sfan5 |
the truth is people do not want software whose correctness can be mathematically proven but software that just does what they want |
| 06:42 |
erle |
it's a bit more subtle than that i think |
| 06:42 |
erle |
i think the truth is that they want software that they *think* does what they want |
| 06:43 |
erle |
and almost all dependency-tracking bugs resulting in miscompilation result in a somewhat working binary |
| 06:43 |
erle |
almost all that i have seen |
| 06:43 |
erle |
it's just that the binary does not correspond to the source, or to what you'd get with a full rebuild |
| 06:43 |
erle |
so unless people have a good testsuite or something else to figure it out, they will not notice in the majority of cases |
| 06:47 |
|
v-rob joined #minetest-dev |
| 06:47 |
|
v-rob joined #minetest-dev |
| 06:49 |
erle |
anyways, as long as minetest dependency tracking is as broken as i have detailed in #11749 … doing incremental builds in the CI is only going to lead to exactly the bugs i have mentioned. |
| 06:49 |
ShadowBot |
https://github.com/minetest/minetest/issues/11749 -- CMake does not capture all dependencies, causing erroneous incremental builds & build failures |
| 06:50 |
erle |
and i strongly doubt that people have interest in CI builds that are not matching the source |
| 06:52 |
erle |
Zughy[m] that issue is still unconfirmed. given you aim to do triage, could you confirm it maybe? i have included steps to reproduce it there, over half a year ago. |
| 06:56 |
sfan5 |
I have already told you that using a git hook to mess with file timestamps is not a bug |
| 06:56 |
sfan5 |
besides, shouldn't you be filing this on CMake's or make's bug tracker? Minetest does not implement its own build system (not on the level that it could influence dependency tracking) |
| 07:01 |
|
calcul0n joined #minetest-dev |
| 07:40 |
erle |
sfan5 the git hook only *exposes* the bug |
| 07:41 |
erle |
it creates a situation where it occurs very reliably |
| 07:41 |
erle |
and no, make or cmake are not the right targets for two reasons |
| 07:41 |
erle |
first, make can not be fixed. the architecture does not allow it. |
| 07:43 |
erle |
cmake can also not be fixed (i spoke to someone who tried years ago and the entire – racy – solution was bigger than what i could do), but IIRC someone (josiah_wi maybe?) confirmed some of the dependencies are simply undeclared. it matters very little what cmake can do if the dependencies are not even declared correctly. |
| 07:44 |
erle |
and by “racy solution” i mean “it looks like a solution but is so racy it's useless” |
| 07:48 |
erle |
anyways, i have offered to help fix dependency issues and make at least incremental builds faster (it is likely that it would make full rebuilds faster too, because correct dependencies allow better parallelization – parallel builds are a basically several incremental builds interleaved), but for that there needs to be at least some level of interest from coredevs. |
| 07:49 |
erle |
and as long as the reply to “here is a git hook to reproduce the bug that people run into randomly” is something like “just don't use that git hook” i feel not taken seriously on this. |
| 07:50 |
erle |
i mean, i have immediately stopped using that hook after i found out that it does not save me time, obviously. and i still run into these kinds of bugs all the time. |
| 07:53 |
erle |
anyways, as i wrote in the issue: “Please try to actually understand and reproduce the issues instead of telling me they do not exist or are not important.” – given that it is still tagged “unconfirmed” after half a year and multiple people complaining about build problems (even in this very channel) I doubt there is much interest in making incremental builds correct. |
| 07:54 |
erle |
and if incremental builds are not correct, you can't expect ccache to work in the first place |
| 08:58 |
sfan5 |
if make and cmake can't be fixed what can we even do to fix this? |
| 08:59 |
sfan5 |
what are the concrete steps to fix dependency issues? where are the current dependency issues even? |
| 08:59 |
sfan5 |
I also asked this months ago on the issue and you did not answer me |
| 09:03 |
sfan5 |
and the reason I want these answered ahead of time is that there is no point in keeping an issue open we cannot possibly ever fix |
| 09:32 |
|
Taoki joined #minetest-dev |
| 09:50 |
erle |
here is what i have done in the past, for other projects, to fix similar dependency issues: a) figure out what commands cmake / make run b) plug those into a build system that can infer dependencies c) graph the result |
| 09:51 |
erle |
my usual showcase is https://github.com/linleyh/liberation-circuit |
| 09:54 |
erle |
last time i checked, you could build it with make, do, or redo. make does the usual thing: declare deps beforehand, with all the bugs that result from it. (in minetests case, cmake is declaring the deps). do always rebuilds everything unconditionally. redo builds first and infers any dependency relationships from the build. |
| 09:54 |
erle |
let me check how many deps i found there that make did not handle |
| 09:55 |
|
appguru joined #minetest-dev |
| 09:56 |
|
HuguesRoss joined #minetest-dev |
| 10:01 |
erle |
so for liberation circuit, it seems to be at least a bit over 2000 dependencies that make does not track (and can not track at all, if i am not mistaken). the total number of dependencies seems to be a bit over 5500 and i'd be very surprised if make tracks all of them. |
| 10:02 |
erle |
caveat: i have not put much effort into this right now, because people often do not believe that actually tracking dependencies produces such large numbers if they can get away with listing maybe 50 or so in their makefile (or so they think) |
| 10:04 |
erle |
caveat 2: compiler options to output dependencies for makefiles typically do not output anything close the full set of dependencies. |
| 10:05 |
erle |
to put it into perspective, sloccount reports about 90.5k lines of C code for liberation circuit. for minetest, i believe the number was about 130k. |
| 10:05 |
erle |
(excluding irrlichtmt) |
| 10:07 |
erle |
therefore, given the failures in incremental builds that me and others have observed and that it's technically impossible for the current build system setup to track all dependencies, i conclude that there are probably hundreds (maybe a few thousand) dependency relations that are not acknowledged right now by the build process. |
| 10:10 |
erle |
to be any more convincing i would need to rebuild minetest with redo and then use redo-dot to spit out a huge graphviz file. yet, at that point i would have already converted minetest to use a build system that could build minetest without cmake. |
| 10:11 |
sfan5 |
thanks for that information |
| 10:12 |
erle |
and i vaguely remember that my claim “to fix entire categories of dependency bugs once and for all, you might have have to ditch cmake” was not taken well. |
| 10:12 |
erle |
though i might be wrong on that, not sure |
| 10:12 |
erle |
as i said, IIRC josiah_wi claimed to have found and fixed some dependency bugs by making cmake generate ninja files |
| 10:14 |
erle |
the thing is, having more accurate dependency information makes builds faster in the same way that having more accurate typing informaiton makes programs faster. constraints allow you to avoid useless busywork. |
| 10:14 |
sfan5 |
the thing is I'm pretty sure there is absolutely no consensus for switching away from cmake |
| 10:15 |
erle |
well, i'd be happy if anyone found how to fix it within cmake |
| 10:16 |
erle |
i just know that the last person i encountered that actually tried fixing cmake produced something that a) did not work well b) was larger than my entire redo implementation (which is about 400 lines of bourne shell, do is about 100 lines of bourne shell) |
| 10:17 |
erle |
my goal is not to use any secific buily system, my goal is to a) have faster builds b) have less pain when bisecting |
| 10:17 |
erle |
buily → build |
| 10:18 |
erle |
as before, i suggest the microsoft paper “build systems a la carte” as an overview for how to somewhat objectively evaluate build systems. https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf |
| 10:19 |
erle |
it is a bit weird though. the first build system they explain is make, then comes … excel. it makes sense in context. |
| 10:20 |
erle |
(excels calc engine is an interactive dynamic dependency resolver, more powerful than make if i am not mistaken) |
| 10:22 |
erle |
the feature that would benefit minetest most is probably what they call the “early cutoff optimization” |
| 10:22 |
erle |
simply stated: don't build what you don't need to build. |
| 10:24 |
erle |
anyways, basically any tool that tries to 1. get dependencies 2. topologically sort them 3. build them in that order … is either subtly or very obviously wrong. |
| 10:30 |
erle |
while this might seem stupid again (seriously, which instructions to reproduce a bug do not seem stupid), one of the easiest way to expose such build systems is to a) make rules that change source files during the builds that are usey for two or more target files b) build that unholy mess in parallel |
| 10:30 |
erle |
and yes, i don't suggest to do such a thing, ever. except to figure out if a build system is cheating. |
| 10:32 |
erle |
“cheating” as in: the toposort approach makes it much easier to score high in benchmarks by avoiding considering a huge amount of dependencies. crucially though, it can avoid *too much* – and the easiest (and probably stupidest by far) way to expose that is to change source files during builds. |
| 10:33 |
|
YuGiOhJCJ joined #minetest-dev |
| 10:33 |
sfan5 |
rubenwardy: is MTG findable in content db? I can't find it right now |
| 10:33 |
sfan5 |
but I remember that it used to be |
| 10:34 |
erle |
sfan5, the cdb version is strictly minetest 5.5, maybe that is the issue? |
| 10:35 |
erle |
like, i doubt it shows up with any other version, even newer ones |
| 10:35 |
erle |
(not tested rn, just noticed the max version) |
| 10:36 |
sfan5 |
ah true |
| 10:36 |
sfan5 |
but have we had a version bump yet |
| 10:36 |
sfan5 |
or how does cdb know? |
| 10:36 |
erle |
i have no idea, isn't the version bumb right after the release? |
| 10:36 |
erle |
bump |
| 10:37 |
sfan5 |
the version yes |
| 10:37 |
sfan5 |
internal things like the protocol version are bumped when necessary or at least once just before release |
| 10:54 |
|
YuGiOhJCJ joined #minetest-dev |
| 10:58 |
rubenwardy |
It's done both on protocol and version name |
| 11:00 |
rubenwardy |
If mtg is forwards compatible, I could remove the upper limit |
| 11:01 |
rubenwardy |
Done |
| 11:01 |
|
YuGiOhJCJ joined #minetest-dev |
| 11:01 |
MTDiscord |
<luatic> MTG is forwards compatible unless Minetest is not backwards compatible. |
| 11:03 |
Zughy[m] |
I thought Gentoo was a Linux distro (the label) #12276 |
| 11:03 |
ShadowBot |
https://github.com/minetest/minetest/issues/12276 -- Cannot turn off development test mode |
| 11:04 |
MTDiscord |
<luatic> Gentoo is a Linux distro, but the issue is presumably unrelated to the fact that Gentoo is a Linux distro. |
| 11:05 |
MTDiscord |
<luatic> TBH the issue looks like a "possible close" to me. |
| 11:11 |
sfan5 |
Zughy[m]: you could argue for or against the label in different ways but I had the exact thought luatic described |
| 11:11 |
sfan5 |
? |
| 11:16 |
sfan5 |
rubenwardy: can you put the new repo into the engine group (permissions) |
| 11:16 |
rubenwardy |
done |
| 11:18 |
sfan5 |
hm why can I access part of the repo settings, that's new |
| 11:18 |
sfan5 |
"Your site is published at http://www.minetest.net/benchmarks/" uhhhhh |
| 11:19 |
rubenwardy |
I granted permissions for that |
| 11:20 |
rubenwardy |
as I thought you might need it |
| 11:20 |
MTDiscord |
<luatic> A benchmark page sounds lovely |
| 11:20 |
sfan5 |
ah |
| 11:20 |
rubenwardy |
afaict, you can't revert back to minetest.github.io |
| 11:20 |
sfan5 |
yeah but it's not supposed to go there |
| 11:20 |
rubenwardy |
if that's what you were intending |
| 11:20 |
sfan5 |
but it doesn't look like we have a choice |
| 11:20 |
rubenwardy |
you can ask c55 for a subdomain and then enter it as a custom domain |
| 11:21 |
rubenwardy |
like blog.minetest.net |
| 11:21 |
MTDiscord |
<luatic> what's up with the blog BTW, april is missing |
| 11:22 |
MTDiscord |
<luatic> will it just jump to may and fix the numbering issue? |
| 11:22 |
rubenwardy |
it's WIP |
| 11:22 |
rubenwardy |
I've done most of it, MisterE just has a few more things to add - but he's busy |
| 11:54 |
sfan5 |
somehow the binaries built by irrlichtmt's CI are 1 MB larger than the local ones I used to build, weird |
| 11:55 |
rubenwardy |
debug symbols? |
| 11:56 |
sfan5 |
yeah thats it |
| 12:47 |
sfan5 |
merging #12255 and #12278 soon if the CI of the latter passes |
| 12:47 |
ShadowBot |
https://github.com/minetest/minetest/issues/12255 -- Sort out some issues with our CI setup by sfan5 |
| 12:47 |
ShadowBot |
https://github.com/minetest/minetest/issues/12278 -- Build-related stuff by sfan5 |
| 12:47 |
|
Fixer joined #minetest-dev |
| 12:51 |
|
proller joined #minetest-dev |
| 12:55 |
erle |
sfan5 btw in case you read the ”build systems a la carte” i'd be very interested in what opinions (if any, but i bet you have a lot) about build systems you reconsider afterwards |
| 13:04 |
erle |
the approach i advocate is what they name “SHAKE” btw |
| 13:04 |
erle |
with a bit of “fabricate” mixed in for good measure |
| 13:04 |
erle |
(i strace the compiler to get the ”real” dependencies) |
| 13:12 |
sfan5 |
I haven't |
| 13:17 |
|
Fixer_ joined #minetest-dev |
| 13:20 |
|
Fixer joined #minetest-dev |
| 13:37 |
|
proller joined #minetest-dev |
| 15:08 |
|
CoolJar10 joined #minetest-dev |
| 15:11 |
|
CoolJar10 joined #minetest-dev |
| 15:29 |
sfan5 |
btw since it got the supported by coredev label someone should review #11654, it's just a small feature |
| 15:29 |
ShadowBot |
https://github.com/minetest/minetest/issues/11654 -- [no squash] Implement tool use sound by sfan5 |
| 16:08 |
|
v-rob joined #minetest-dev |
| 18:10 |
appguru |
sfan5: can I close game#2807? |
| 18:10 |
ShadowBot |
https://github.com/minetest/minetest_game/issues/2807 -- Compress all .obj files by ExeVirus |
| 18:10 |
sfan5 |
if you want to |
| 18:34 |
appguru |
does Minetest currently do any kind of batching on outgoing packets? when I call hud_add a million times, will I get a million UDP packets? |
| 18:36 |
sfan5 |
packets merging is not supported |
| 18:36 |
sfan5 |
-s |
| 18:37 |
MTDiscord |
<Warr1024> MT seems to rely on the overhead for UDP packets being relatively small, and leaves efficient batch handling up to the underlying network. |
| 18:38 |
erle |
> when I call hud_add a million times |
| 18:38 |
erle |
maybe you shouldn't |
| 18:38 |
appguru |
obviously I'm interested in abnormal usage |
| 18:38 |
appguru |
abusing HUD elements as pixels |
| 18:39 |
appguru |
i.e. this is just for fun |
| 18:39 |
appguru |
to push the engine limitations |
| 18:39 |
rubenwardy |
a better example would be entity/object properties |
| 18:39 |
appguru |
also erle, aren't you usually the one to point out engine breakage in ridiculous edge cases ;) |
| 18:39 |
rubenwardy |
if you batched them, you could send multiple objects in one package and only the properties that changed |
| 18:39 |
rubenwardy |
*packet |
| 18:40 |
appguru |
pretty sure it does currently batch some obj attrs |
| 18:40 |
rubenwardy |
which would reduce the number of packets quite a lot without making big packets |
| 18:40 |
appguru |
agreed |
| 18:40 |
sfan5 |
active object messages are batched |
| 18:40 |
erle |
appguru i am also usually the one advocating strict limits and better architecture to ensure that such cases do not happen. |
| 18:41 |
appguru |
strict architecture is the opposite of fun |
| 18:41 |
erle |
programming is not fun though. it's horrible. |
| 18:42 |
erle |
appguru if you want to have pixels in the hud, how about you garbage collect texture modifiers? then you can go nuts with it. |
| 18:42 |
appguru |
I'm aware |
| 18:43 |
appguru |
But implementing proper garbage collection there doesn't seem easy to me |
| 18:44 |
MTDiscord |
<Warr1024> Programming IS fun. And horrible. You don't last long in the programming world if you don't have a properly warped sense of fun. |
| 18:44 |
erle |
to the contrary, i lasted long because it is hard to demotivate me further at this point. |
| 18:45 |
erle |
furthermore i don't trust people who program for fun. what if it stops being fun? |
| 18:45 |
erle |
:P |
| 18:46 |
MTDiscord |
<luatic> alright I tried it |
| 18:46 |
MTDiscord |
<luatic> and now my Minetest is frozen |
| 18:47 |
Pexin |
never pick a career you enjoy. once it is a career, and you are on a shedule, and have requirements, you WILL hate it. it would be a shame to hate what you used to love. (and back to #minetest now) |
| 18:47 |
erle |
anyways, appguru IMO garbage collection will be needed for anything super-dynamic with pixels. |
| 18:48 |
appguru |
erle: yes, and HUD elements are garbage collected |
| 18:48 |
MTDiscord |
<Warr1024> HUD elements are also garbage, collected. ? |
| 18:48 |
erle |
appguru you still can't do this efficiently |
| 18:48 |
erle |
even if you batch all the stuff |
| 18:49 |
MTDiscord |
<Warr1024> At minimum you still have to send at least the equivalent of {hud_change:{text:"brown.png"}} to the client for each pixel, which sounds like a pretty painful way to encode an image :-) |
| 18:49 |
erle |
yes |
| 18:50 |
appguru |
Indeed it is |
| 18:51 |
appguru |
But implementing GC for texmods is painful as well |
| 18:51 |
appguru |
(ideally dynamic media would be GC'd) |
| 18:51 |
appguru |
I would prefer not to have to use the [png texmod as a workaround |
| 18:51 |
erle |
the [png texmod is still the stupidest thing ever |
| 18:52 |
MTDiscord |
<Warr1024> If I had media GC then I'd probably more or less immediately expand the skybox navigation features in NodeCore. |
| 18:52 |
appguru |
it's the closest we are to going nuts with dynamic media, because texmod GC is easier than media GC |
| 18:53 |
erle |
appguru, look, as an interesting case, take mcl_maps, which displays a 128 × 128 texture. each update of that texture is 128 × 128 × 4 bytes being filled with a decoded texture. that's 64k uncompressed. unless you are standing on a featureless plains, this image will take about 4 to 12 k over the network with the current implementation. and now think about how many pixels you want to update how often. |
| 18:53 |
MTDiscord |
<Warr1024> Is it really that much easier? It seemed to me like the lion's share of the GC problem is stuff that'd be common to all of them anyway. |
| 18:53 |
erle |
why is media gc not easy btw? |
| 18:54 |
MTDiscord |
<Warr1024> The assumption of "once loaded, always loaded" is kind of distributed throughout MT now, for one thing |
| 18:54 |
MTDiscord |
<Warr1024> and of course one consequence of that is that nobody has needed to track usage, since it never mattered before |
| 18:57 |
erle |
wouldn't it be similarly hard for texmods too then? |
| 18:57 |
erle |
after all, their stuff gets turned into pixels |
| 18:58 |
erle |
i mean gc-ing the texmod string woudn't do a whole much |
| 18:59 |
MTDiscord |
<Warr1024> Yeah, I mean, the point is that we'd have to identify that the computed texture is probably not in use anymore, remove the image and texture objects corresponding to it, and also ensure it returns to a state where if the engine encounters it again it would know to rebuild it. |
| 19:00 |
MTDiscord |
<Warr1024> I think though that just verifying that it's not being used right now would be tricky, because not everything that might be holding a reference to it will necessarily make that known. |
| 19:01 |
erle |
i wonder what the worst case is right now. |
| 19:04 |
erle |
the thing is, if i right now would make mcl_maps push a texture every second, the client would needd about 4MB RAM per minute minimum to keep up. |
| 19:04 |
|
v-rob joined #minetest-dev |
| 19:04 |
erle |
which is obviously why i don't do that |
| 19:05 |
erle |
still, after an hour of playing it would be “only” 240 MB, so maybe it isn't too bad? |
| 19:05 |
erle |
like if all of that is cleared at the end of the session obv |
| 19:09 |
|
Alias joined #minetest-dev |
| 19:49 |
|
CoolJar10 joined #minetest-dev |
| 19:59 |
Zughy[m] |
#10392 possible close? |
| 19:59 |
ShadowBot |
https://github.com/minetest/minetest/issues/10392 -- Add media alias |
| 20:01 |
rubenwardy |
from the issue title, I expected that to be a texture pack thing |
| 20:01 |
erle |
it is not |
| 20:02 |
erle |
it is someone saying they want to rename their files and have other stuff not break |
| 20:02 |
erle |
like other mods that depend on having that file |
| 20:02 |
erle |
i think this is bonkers and the commenters seem to think too |
| 20:03 |
erle |
like, just don't rename files, problem solved |
| 20:06 |
rubenwardy |
merging #12115 and #12258 in 5 |
| 20:06 |
ShadowBot |
https://github.com/minetest/minetest/issues/12115 -- Default chat clickable weblinks by Froggo8311 |
| 20:06 |
ShadowBot |
https://github.com/minetest/minetest/issues/12258 -- Add benchmarks for json string serialize/deserialize by paradust7 |
| 20:11 |
MTDiscord |
<luatic> or if you do want to rename files, copy them instead |
| 20:11 |
MTDiscord |
<luatic> does Minetest support symlinks for media? |
| 20:37 |
Zughy[m] |
is #6914 still a thing? |
| 20:37 |
ShadowBot |
https://github.com/minetest/minetest/issues/6914 -- Leveled nodeboxes lack leveled selection boxes |
| 20:43 |
|
Cosmosis[m] joined #minetest-dev |
| 20:48 |
Zughy[m] |
ok, so: 4 days ago there were 200 "Feature request" issues with no "@ <something>" label. Now there are 0 |
| 20:49 |
Zughy[m] |
? |
| 20:51 |
Zughy[m] |
200 more unlabelled issues remaining (mostly unconfirmed bugs) and then this hell is over |
| 20:53 |
MTDiscord |
<Warr1024> then you get to finally move on to the hell of triaging the steady trickle of new issues as they arrive. |
| 20:54 |
Zughy[m] |
I'm already doing that |
| 20:56 |
Zughy[m] |
I must say, you core devs are batshit crazy for having done this all by yourselves for years. Like... just ask for help |
| 20:58 |
|
v-rob joined #minetest-dev |
| 22:34 |
|
panwolfram joined #minetest-dev |
| 23:10 |
|
AliasAlreadyTake joined #minetest-dev |
| 23:12 |
|
proller joined #minetest-dev |
| 23:17 |
|
erle joined #minetest-dev |
