Minetest logo

IRC log for #minetest-dev, 2022-05-06

| Channels | #minetest-dev index | Today | | Google Search | Plaintext

All times shown according to UTC.

Time Nick Message
01:41 v-rob joined #minetest-dev
04:00 MTDiscord joined #minetest-dev
06:10 v-rob joined #minetest-dev
06:30 sfan5 "In practice, the direct mode is safe to use in the absolute majority of cases."
06:37 erle yeah, this is what about everyone says who can't be arsed to implement dependency tracking correctly. that it's not a problem “in practice”.
06:37 erle the majority of them are liars.
06:38 erle because there exists very few software to figure out if it is a problem.
06:39 erle and every software that *i* know that can help you to figure out if it is a problem does this either by implementing correct dependency tracking – in which case you do not hae the problem …
06:39 erle … or by fully rebuilding it and comparing it with a partial build, which only works if you a) have reproducible builds b) can afford the time to wait for a full rebuild.
06:40 erle (none of which applies here AFAIK)
06:41 sfan5 the truth is people do not want software whose correctness can be mathematically proven but software that just does what they want
06:42 erle it's a bit more subtle than that i think
06:42 erle i think the truth is that they want software that they *think* does what they want
06:43 erle and almost all dependency-tracking bugs resulting in miscompilation result in a somewhat working binary
06:43 erle almost all that i have seen
06:43 erle it's just that the binary does not correspond to the source, or to what you'd get with a full rebuild
06:43 erle so unless people have a good testsuite or something else to figure it out, they will not notice in the majority of cases
06:47 v-rob joined #minetest-dev
06:47 v-rob joined #minetest-dev
06:49 erle anyways, as long as minetest dependency tracking is as broken as i have detailed in #11749 … doing incremental builds in the CI is only going to lead to exactly the bugs i have mentioned.
06:49 ShadowBot https://github.com/minetest/minetest/issues/11749 -- CMake does not capture all dependencies, causing erroneous incremental builds & build failures
06:50 erle and i strongly doubt that people have interest in CI builds that are not matching the source
06:52 erle Zughy[m] that issue is still unconfirmed. given you aim to do triage, could you confirm it maybe? i have included steps to reproduce it there, over half a year ago.
06:56 sfan5 I have already told you that using a git hook to mess with file timestamps is not a bug
06:56 sfan5 besides, shouldn't you be filing this on CMake's or make's bug tracker? Minetest does not implement its own build system (not on the level that it could influence dependency tracking)
07:01 calcul0n joined #minetest-dev
07:40 erle sfan5 the git hook only *exposes* the bug
07:41 erle it creates a situation where it occurs very reliably
07:41 erle and no, make or cmake are not the right targets for two reasons
07:41 erle first, make can not be fixed. the architecture does not allow it.
07:43 erle cmake can also not be fixed (i spoke to someone who tried years ago and the entire – racy – solution was bigger than what i could do), but IIRC someone (josiah_wi maybe?) confirmed some of the dependencies are simply undeclared. it matters very little what cmake can do if the dependencies are not even declared correctly.
07:44 erle and by “racy solution” i mean “it looks like a solution but is so racy it's useless”
07:48 erle anyways, i have offered to help fix dependency issues and make at least incremental builds faster (it is likely that it would make full rebuilds faster too, because correct dependencies allow better parallelization – parallel builds are a basically several incremental builds interleaved), but for that there needs to be at least some level of interest from coredevs.
07:49 erle and as long as the reply to “here is a git hook to reproduce the bug that people run into randomly” is something like “just don't use that git hook” i feel not taken seriously on this.
07:50 erle i mean, i have immediately stopped using that hook after i found out that it does not save me time, obviously. and i still run into these kinds of bugs all the time.
07:53 erle anyways, as i wrote in the issue: “Please try to actually understand and reproduce the issues instead of telling me they do not exist or are not important.” – given that it is still tagged “unconfirmed” after half a year and multiple people complaining about build problems (even in this very channel) I doubt there is much interest in making incremental builds correct.
07:54 erle and if incremental builds are not correct, you can't expect ccache to work in the first place
08:58 sfan5 if make and cmake can't be fixed what can we even do to fix this?
08:59 sfan5 what are the concrete steps to fix dependency issues? where are the current dependency issues even?
08:59 sfan5 I also asked this months ago on the issue and you did not answer me
09:03 sfan5 and the reason I want these answered ahead of time is that there is no point in keeping an issue open we cannot possibly ever fix
09:32 Taoki joined #minetest-dev
09:50 erle here is what i have done in the past, for other projects, to fix similar dependency issues: a) figure out what commands cmake / make run b) plug those into a build system that can infer dependencies c) graph the result
09:51 erle my usual showcase is https://github.com/linleyh/liberation-circuit
09:54 erle last time i checked, you could build it with make, do, or redo. make does the usual thing: declare deps beforehand, with all the bugs that result from it. (in minetests case, cmake is declaring the deps). do always rebuilds everything unconditionally. redo builds first and infers any dependency relationships from the build.
09:54 erle let me check how many deps i found there that make did not handle
09:55 appguru joined #minetest-dev
09:56 HuguesRoss joined #minetest-dev
10:01 erle so for liberation circuit, it seems to be at least a bit over 2000 dependencies that make does not track (and can not track at all, if i am not mistaken). the total number of dependencies seems to be a bit over 5500 and i'd be very surprised if make tracks all of them.
10:02 erle caveat: i have not put much effort into this right now, because people often do not believe that actually tracking dependencies produces such large numbers if they can get away with listing maybe 50 or so in their makefile (or so they think)
10:04 erle caveat 2: compiler options to output dependencies for makefiles typically do not output anything close the full set of dependencies.
10:05 erle to put it into perspective, sloccount reports about 90.5k lines of C code for liberation circuit. for minetest, i believe the number was about 130k.
10:05 erle (excluding irrlichtmt)
10:07 erle therefore, given the failures in incremental builds that me and others have observed and that it's technically impossible for the current build system setup to track all dependencies, i conclude that there are probably hundreds (maybe a few thousand) dependency relations that are not acknowledged right now by the build process.
10:10 erle to be any more convincing i would need to rebuild minetest with redo and then use redo-dot to spit out a huge graphviz file. yet, at that point i would have already converted minetest to use a build system that could build minetest without cmake.
10:11 sfan5 thanks for that information
10:12 erle and i vaguely remember that my claim “to fix entire categories of dependency bugs once and for all, you might have have to ditch cmake” was not taken well.
10:12 erle though i might be wrong on that, not sure
10:12 erle as i said, IIRC josiah_wi claimed to have found and fixed some dependency bugs by making cmake generate ninja files
10:14 erle the thing is, having more accurate dependency information makes builds faster in the same way that having more accurate typing informaiton makes programs faster. constraints allow you to avoid useless busywork.
10:14 sfan5 the thing is I'm pretty sure there is absolutely no consensus for switching away from cmake
10:15 erle well, i'd be happy if anyone found how to fix it within cmake
10:16 erle i just know that the last person i encountered that actually tried fixing cmake produced something that a) did not work well b) was larger than my entire redo implementation (which is about 400 lines of bourne shell, do is about 100 lines of bourne shell)
10:17 erle my goal is not to use any secific buily system, my goal is to a) have faster builds b) have less pain when bisecting
10:17 erle buily → build
10:18 erle as before, i suggest the microsoft paper “build systems a la carte” as an overview for how to somewhat objectively evaluate build systems. https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf
10:19 erle it is a bit weird though. the first build system they explain is make, then comes … excel. it makes sense in context.
10:20 erle (excels calc engine is an interactive dynamic dependency resolver, more powerful than make if i am not mistaken)
10:22 erle the feature that would benefit minetest most is probably what they call the “early cutoff optimization”
10:22 erle simply stated: don't build what you don't need to build.
10:24 erle anyways, basically any tool that tries to 1. get dependencies 2. topologically sort them 3. build them in that order … is either subtly or very obviously wrong.
10:30 erle while this might seem stupid again (seriously, which instructions to reproduce a bug do not seem stupid), one of the easiest way to expose such build systems is to a) make rules that change source files during the builds that are usey for two or more target files b) build that unholy mess in parallel
10:30 erle and yes, i don't suggest to do such a thing, ever. except to figure out if a build system is cheating.
10:32 erle “cheating” as in: the toposort approach makes it much easier to score high in benchmarks by avoiding considering a huge amount of dependencies. crucially though, it can avoid *too much* – and the easiest (and probably stupidest by far) way to expose that is to change source files during builds.
10:33 YuGiOhJCJ joined #minetest-dev
10:33 sfan5 rubenwardy: is MTG findable in content db? I can't find it right now
10:33 sfan5 but I remember that it used to be
10:34 erle sfan5, the cdb version is strictly minetest 5.5, maybe that is the issue?
10:35 erle like, i doubt it shows up with any other version, even newer ones
10:35 erle (not tested rn, just noticed the max version)
10:36 sfan5 ah true
10:36 sfan5 but have we had a version bump yet
10:36 sfan5 or how does cdb know?
10:36 erle i have no idea, isn't the version bumb right after the release?
10:36 erle bump
10:37 sfan5 the version yes
10:37 sfan5 internal things like the protocol version are bumped when necessary or at least once just before release
10:54 YuGiOhJCJ joined #minetest-dev
10:58 rubenwardy It's done both on protocol and version name
11:00 rubenwardy If mtg is forwards compatible, I could remove the upper limit
11:01 rubenwardy Done
11:01 YuGiOhJCJ joined #minetest-dev
11:01 MTDiscord <luatic> MTG is forwards compatible unless Minetest is not backwards compatible.
11:03 Zughy[m] I thought Gentoo was a Linux distro (the label) #12276
11:03 ShadowBot https://github.com/minetest/minetest/issues/12276 -- Cannot turn off development test mode
11:04 MTDiscord <luatic> Gentoo is a Linux distro, but the issue is presumably unrelated to the fact that Gentoo is a Linux distro.
11:05 MTDiscord <luatic> TBH the issue looks like a "possible close" to me.
11:11 sfan5 Zughy[m]: you could argue for or against the label in different ways but I had the exact thought luatic described
11:11 sfan5 🤷
11:16 sfan5 rubenwardy: can you put the new repo into the engine group (permissions)
11:16 rubenwardy done
11:18 sfan5 hm why can I access part of the repo settings, that's new
11:18 sfan5 "Your site is published at http://www.minetest.net/benchmarks/" uhhhhh
11:19 rubenwardy I granted permissions for that
11:20 rubenwardy as I thought you might need it
11:20 MTDiscord <luatic> A benchmark page sounds lovely
11:20 sfan5 ah
11:20 rubenwardy afaict, you can't revert back to minetest.github.io
11:20 sfan5 yeah but it's not supposed to go there
11:20 rubenwardy if that's what you were intending
11:20 sfan5 but it doesn't look like we have a choice
11:20 rubenwardy you can ask c55 for a subdomain and then enter it as a custom domain
11:21 rubenwardy like blog.minetest.net
11:21 MTDiscord <luatic> what's up with the blog BTW, april is missing
11:22 MTDiscord <luatic> will it just jump to may and fix the numbering issue?
11:22 rubenwardy it's WIP
11:22 rubenwardy I've done most of it, MisterE just has a few more things to add - but he's busy
11:54 sfan5 somehow the binaries built by irrlichtmt's CI are 1 MB larger than the local ones I used to build, weird
11:55 rubenwardy debug symbols?
11:56 sfan5 yeah thats it
12:47 sfan5 merging #12255 and #12278 soon if the CI of the latter passes
12:47 ShadowBot https://github.com/minetest/minetest/issues/12255 -- Sort out some issues with our CI setup by sfan5
12:47 ShadowBot https://github.com/minetest/minetest/issues/12278 -- Build-related stuff by sfan5
12:47 Fixer joined #minetest-dev
12:51 proller joined #minetest-dev
12:55 erle sfan5 btw in case you read the ”build systems a la carte” i'd be very interested in what opinions (if any, but i bet you have a lot) about build systems you reconsider afterwards
13:04 erle the approach i advocate is what they name “SHAKE” btw
13:04 erle with a bit of “fabricate” mixed in for good measure
13:04 erle (i strace the compiler to get the ”real” dependencies)
13:12 sfan5 I haven't
13:17 Fixer_ joined #minetest-dev
13:20 Fixer joined #minetest-dev
13:37 proller joined #minetest-dev
15:08 CoolJar10 joined #minetest-dev
15:11 CoolJar10 joined #minetest-dev
15:29 sfan5 btw since it got the supported by coredev label someone should review #11654, it's just a small feature
15:29 ShadowBot https://github.com/minetest/minetest/issues/11654 -- [no squash] Implement tool use sound by sfan5
16:08 v-rob joined #minetest-dev
18:10 appguru sfan5: can I close game#2807?
18:10 ShadowBot https://github.com/minetest/minetest_game/issues/2807 -- Compress all .obj files by ExeVirus
18:10 sfan5 if you want to
18:34 appguru does Minetest currently do any kind of batching on outgoing packets? when I call hud_add a million times, will I get a million UDP packets?
18:36 sfan5 packets merging is not supported
18:36 sfan5 -s
18:37 MTDiscord <Warr1024> MT seems to rely on the overhead for UDP packets being relatively small, and leaves efficient batch handling up to the underlying network.
18:38 erle > when I call hud_add a million times
18:38 erle maybe you shouldn't
18:38 appguru obviously I'm interested in abnormal usage
18:38 appguru abusing HUD elements as pixels
18:39 appguru i.e. this is just for fun
18:39 appguru to push the engine limitations
18:39 rubenwardy a better example would be entity/object properties
18:39 appguru also erle, aren't you usually the one to point out engine breakage in ridiculous edge cases ;)
18:39 rubenwardy if you batched them, you could send multiple objects in one package and only the properties that changed
18:39 rubenwardy *packet
18:40 appguru pretty sure it does currently batch some obj attrs
18:40 rubenwardy which would reduce the number of packets quite a lot without making big packets
18:40 appguru agreed
18:40 sfan5 active object messages are batched
18:40 erle appguru i am also usually the one advocating strict limits and better architecture to ensure that such cases do not happen.
18:41 appguru strict architecture is the opposite of fun
18:41 erle programming is not fun though. it's horrible.
18:42 erle appguru if you want to have pixels in the hud, how about you garbage collect texture modifiers? then you can go nuts with it.
18:42 appguru I'm aware
18:43 appguru But implementing proper garbage collection there doesn't seem easy to me
18:44 MTDiscord <Warr1024> Programming IS fun.  And horrible.  You don't last long in the programming world if you don't have a properly warped sense of fun.
18:44 erle to the contrary, i lasted long because it is hard to demotivate me further at this point.
18:45 erle furthermore i don't trust people who program for fun. what if it stops being fun?
18:45 erle :P
18:46 MTDiscord <luatic> alright I tried it
18:46 MTDiscord <luatic> and now my Minetest is frozen
18:47 Pexin never pick a career you enjoy. once it is a career, and you are on a shedule, and have requirements, you WILL hate it. it would be a shame to hate what you used to love. (and back to #minetest now)
18:47 erle anyways, appguru IMO garbage collection will be needed for anything super-dynamic with pixels.
18:48 appguru erle: yes, and HUD elements are garbage collected
18:48 MTDiscord <Warr1024> HUD elements are also garbage, collected. 😏
18:48 erle appguru you still can't do this efficiently
18:48 erle even if you batch all the stuff
18:49 MTDiscord <Warr1024> At minimum you still have to send at least the equivalent of {hud_change:{text:"brown.png"}} to the client for each pixel, which sounds like a pretty painful way to encode an image :-)
18:49 erle yes
18:50 appguru Indeed it is
18:51 appguru But implementing GC for texmods is painful as well
18:51 appguru (ideally dynamic media would be GC'd)
18:51 appguru I would prefer not to have to use the [png texmod as a workaround
18:51 erle the [png texmod is still the stupidest thing ever
18:52 MTDiscord <Warr1024> If I had media GC then I'd probably more or less immediately expand the skybox navigation features in NodeCore.
18:52 appguru it's the closest we are to going nuts with dynamic media, because texmod GC is easier than media GC
18:53 erle appguru, look, as an interesting case, take mcl_maps, which displays a 128 × 128 texture. each update of that texture is 128 × 128 × 4 bytes being filled with a decoded texture. that's 64k uncompressed. unless you are standing on a featureless plains, this image will take about 4 to 12 k over the network with the current implementation. and now think about how many pixels you want to update how often.
18:53 MTDiscord <Warr1024> Is it really that much easier?  It seemed to me like the lion's share of the GC problem is stuff that'd be common to all of them anyway.
18:53 erle why is media gc not easy btw?
18:54 MTDiscord <Warr1024> The assumption of "once loaded, always loaded" is kind of distributed throughout MT now, for one thing
18:54 MTDiscord <Warr1024> and of course one consequence of that is that nobody has needed to track usage, since it never mattered before
18:57 erle wouldn't it be similarly hard for texmods too then?
18:57 erle after all, their stuff gets turned into pixels
18:58 erle i mean gc-ing the texmod string woudn't do a whole much
18:59 MTDiscord <Warr1024> Yeah, I mean, the point is that we'd have to identify that the computed texture is probably not in use anymore, remove the image and texture objects corresponding to it, and also ensure it returns to a state where if the engine encounters it again it would know to rebuild it.
19:00 MTDiscord <Warr1024> I think though that just verifying that it's not being used right now would be tricky, because not everything that might be holding a reference to it will necessarily make that known.
19:01 erle i wonder what the worst case is right now.
19:04 erle the thing is, if i right now would make mcl_maps push a texture every second, the client would needd about 4MB RAM per minute minimum to keep up.
19:04 v-rob joined #minetest-dev
19:04 erle which is obviously why i don't do that
19:05 erle still, after an hour of playing it would be “only” 240 MB, so maybe it isn't too bad?
19:05 erle like if all of that is cleared at the end of the session obv
19:09 Alias joined #minetest-dev
19:49 CoolJar10 joined #minetest-dev
19:59 Zughy[m] #10392 possible close?
19:59 ShadowBot https://github.com/minetest/minetest/issues/10392 -- Add media alias
20:01 rubenwardy from the issue title, I expected that to be a texture pack thing
20:01 erle it is not
20:02 erle it is someone saying they want to rename their files and have other stuff not break
20:02 erle like other mods that depend on having that file
20:02 erle i think this is bonkers and the commenters seem to think too
20:03 erle like, just don't rename files, problem solved
20:06 rubenwardy merging #12115 and #12258 in 5
20:06 ShadowBot https://github.com/minetest/minetest/issues/12115 -- Default chat clickable weblinks by Froggo8311
20:06 ShadowBot https://github.com/minetest/minetest/issues/12258 -- Add benchmarks for json string serialize/deserialize by paradust7
20:11 MTDiscord <luatic> or if you do want to rename files, copy them instead
20:11 MTDiscord <luatic> does Minetest support symlinks for media?
20:37 Zughy[m] is #6914 still a thing?
20:37 ShadowBot https://github.com/minetest/minetest/issues/6914 -- Leveled nodeboxes lack leveled selection boxes
20:43 Cosmosis[m] joined #minetest-dev
20:48 Zughy[m] ok, so: 4 days ago there were 200 "Feature request" issues with no "@ <something>" label. Now there are 0
20:49 Zughy[m] 🎉
20:51 Zughy[m] 200 more unlabelled issues remaining (mostly unconfirmed bugs) and then this hell is over
20:53 MTDiscord <Warr1024> then you get to finally move on to the hell of triaging the steady trickle of new issues as they arrive.
20:54 Zughy[m] I'm already doing that
20:56 Zughy[m] I must say, you core devs are batshit crazy for having done this all by yourselves for years. Like... just ask for help
20:58 v-rob joined #minetest-dev
22:34 panwolfram joined #minetest-dev
23:10 AliasAlreadyTake joined #minetest-dev
23:12 proller joined #minetest-dev
23:17 erle joined #minetest-dev

| Channels | #minetest-dev index | Today | | Google Search | Plaintext