Time Nick Message 06:30 sfan5 "In practice, the direct mode is safe to use in the absolute majority of cases." 06:37 erle yeah, this is what about everyone says who can't be arsed to implement dependency tracking correctly. that it's not a problem “in practice”. 06:37 erle the majority of them are liars. 06:38 erle because there exists very few software to figure out if it is a problem. 06:39 erle and every software that *i* know that can help you to figure out if it is a problem does this either by implementing correct dependency tracking – in which case you do not hae the problem … 06:39 erle … or by fully rebuilding it and comparing it with a partial build, which only works if you a) have reproducible builds b) can afford the time to wait for a full rebuild. 06:40 erle (none of which applies here AFAIK) 06:41 sfan5 the truth is people do not want software whose correctness can be mathematically proven but software that just does what they want 06:42 erle it's a bit more subtle than that i think 06:42 erle i think the truth is that they want software that they *think* does what they want 06:43 erle and almost all dependency-tracking bugs resulting in miscompilation result in a somewhat working binary 06:43 erle almost all that i have seen 06:43 erle it's just that the binary does not correspond to the source, or to what you'd get with a full rebuild 06:43 erle so unless people have a good testsuite or something else to figure it out, they will not notice in the majority of cases 06:49 erle anyways, as long as minetest dependency tracking is as broken as i have detailed in #11749 … doing incremental builds in the CI is only going to lead to exactly the bugs i have mentioned. 06:49 ShadowBot https://github.com/minetest/minetest/issues/11749 -- CMake does not capture all dependencies, causing erroneous incremental builds & build failures 06:50 erle and i strongly doubt that people have interest in CI builds that are not matching the source 06:52 erle Zughy[m] that issue is still unconfirmed. given you aim to do triage, could you confirm it maybe? i have included steps to reproduce it there, over half a year ago. 06:56 sfan5 I have already told you that using a git hook to mess with file timestamps is not a bug 06:56 sfan5 besides, shouldn't you be filing this on CMake's or make's bug tracker? Minetest does not implement its own build system (not on the level that it could influence dependency tracking) 07:40 erle sfan5 the git hook only *exposes* the bug 07:41 erle it creates a situation where it occurs very reliably 07:41 erle and no, make or cmake are not the right targets for two reasons 07:41 erle first, make can not be fixed. the architecture does not allow it. 07:43 erle cmake can also not be fixed (i spoke to someone who tried years ago and the entire – racy – solution was bigger than what i could do), but IIRC someone (josiah_wi maybe?) confirmed some of the dependencies are simply undeclared. it matters very little what cmake can do if the dependencies are not even declared correctly. 07:44 erle and by “racy solution” i mean “it looks like a solution but is so racy it's useless” 07:48 erle anyways, i have offered to help fix dependency issues and make at least incremental builds faster (it is likely that it would make full rebuilds faster too, because correct dependencies allow better parallelization – parallel builds are a basically several incremental builds interleaved), but for that there needs to be at least some level of interest from coredevs. 07:49 erle and as long as the reply to “here is a git hook to reproduce the bug that people run into randomly” is something like “just don't use that git hook” i feel not taken seriously on this. 07:50 erle i mean, i have immediately stopped using that hook after i found out that it does not save me time, obviously. and i still run into these kinds of bugs all the time. 07:53 erle anyways, as i wrote in the issue: “Please try to actually understand and reproduce the issues instead of telling me they do not exist or are not important.” – given that it is still tagged “unconfirmed” after half a year and multiple people complaining about build problems (even in this very channel) I doubt there is much interest in making incremental builds correct. 07:54 erle and if incremental builds are not correct, you can't expect ccache to work in the first place 08:58 sfan5 if make and cmake can't be fixed what can we even do to fix this? 08:59 sfan5 what are the concrete steps to fix dependency issues? where are the current dependency issues even? 08:59 sfan5 I also asked this months ago on the issue and you did not answer me 09:03 sfan5 and the reason I want these answered ahead of time is that there is no point in keeping an issue open we cannot possibly ever fix 09:50 erle here is what i have done in the past, for other projects, to fix similar dependency issues: a) figure out what commands cmake / make run b) plug those into a build system that can infer dependencies c) graph the result 09:51 erle my usual showcase is https://github.com/linleyh/liberation-circuit 09:54 erle last time i checked, you could build it with make, do, or redo. make does the usual thing: declare deps beforehand, with all the bugs that result from it. (in minetests case, cmake is declaring the deps). do always rebuilds everything unconditionally. redo builds first and infers any dependency relationships from the build. 09:54 erle let me check how many deps i found there that make did not handle 10:01 erle so for liberation circuit, it seems to be at least a bit over 2000 dependencies that make does not track (and can not track at all, if i am not mistaken). the total number of dependencies seems to be a bit over 5500 and i'd be very surprised if make tracks all of them. 10:02 erle caveat: i have not put much effort into this right now, because people often do not believe that actually tracking dependencies produces such large numbers if they can get away with listing maybe 50 or so in their makefile (or so they think) 10:04 erle caveat 2: compiler options to output dependencies for makefiles typically do not output anything close the full set of dependencies. 10:05 erle to put it into perspective, sloccount reports about 90.5k lines of C code for liberation circuit. for minetest, i believe the number was about 130k. 10:05 erle (excluding irrlichtmt) 10:07 erle therefore, given the failures in incremental builds that me and others have observed and that it's technically impossible for the current build system setup to track all dependencies, i conclude that there are probably hundreds (maybe a few thousand) dependency relations that are not acknowledged right now by the build process. 10:10 erle to be any more convincing i would need to rebuild minetest with redo and then use redo-dot to spit out a huge graphviz file. yet, at that point i would have already converted minetest to use a build system that could build minetest without cmake. 10:11 sfan5 thanks for that information 10:12 erle and i vaguely remember that my claim “to fix entire categories of dependency bugs once and for all, you might have have to ditch cmake” was not taken well. 10:12 erle though i might be wrong on that, not sure 10:12 erle as i said, IIRC josiah_wi claimed to have found and fixed some dependency bugs by making cmake generate ninja files 10:14 erle the thing is, having more accurate dependency information makes builds faster in the same way that having more accurate typing informaiton makes programs faster. constraints allow you to avoid useless busywork. 10:14 sfan5 the thing is I'm pretty sure there is absolutely no consensus for switching away from cmake 10:15 erle well, i'd be happy if anyone found how to fix it within cmake 10:16 erle i just know that the last person i encountered that actually tried fixing cmake produced something that a) did not work well b) was larger than my entire redo implementation (which is about 400 lines of bourne shell, do is about 100 lines of bourne shell) 10:17 erle my goal is not to use any secific buily system, my goal is to a) have faster builds b) have less pain when bisecting 10:17 erle buily → build 10:18 erle as before, i suggest the microsoft paper “build systems a la carte” as an overview for how to somewhat objectively evaluate build systems. https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf 10:19 erle it is a bit weird though. the first build system they explain is make, then comes … excel. it makes sense in context. 10:20 erle (excels calc engine is an interactive dynamic dependency resolver, more powerful than make if i am not mistaken) 10:22 erle the feature that would benefit minetest most is probably what they call the “early cutoff optimization” 10:22 erle simply stated: don't build what you don't need to build. 10:24 erle anyways, basically any tool that tries to 1. get dependencies 2. topologically sort them 3. build them in that order … is either subtly or very obviously wrong. 10:30 erle while this might seem stupid again (seriously, which instructions to reproduce a bug do not seem stupid), one of the easiest way to expose such build systems is to a) make rules that change source files during the builds that are usey for two or more target files b) build that unholy mess in parallel 10:30 erle and yes, i don't suggest to do such a thing, ever. except to figure out if a build system is cheating. 10:32 erle “cheating” as in: the toposort approach makes it much easier to score high in benchmarks by avoiding considering a huge amount of dependencies. crucially though, it can avoid *too much* – and the easiest (and probably stupidest by far) way to expose that is to change source files during builds. 10:33 sfan5 rubenwardy: is MTG findable in content db? I can't find it right now 10:33 sfan5 but I remember that it used to be 10:34 erle sfan5, the cdb version is strictly minetest 5.5, maybe that is the issue? 10:35 erle like, i doubt it shows up with any other version, even newer ones 10:35 erle (not tested rn, just noticed the max version) 10:36 sfan5 ah true 10:36 sfan5 but have we had a version bump yet 10:36 sfan5 or how does cdb know? 10:36 erle i have no idea, isn't the version bumb right after the release? 10:36 erle bump 10:37 sfan5 the version yes 10:37 sfan5 internal things like the protocol version are bumped when necessary or at least once just before release 10:58 rubenwardy It's done both on protocol and version name 11:00 rubenwardy If mtg is forwards compatible, I could remove the upper limit 11:01 rubenwardy Done 11:01 MTDiscord MTG is forwards compatible unless Minetest is not backwards compatible. 11:03 Zughy[m] I thought Gentoo was a Linux distro (the label) #12276 11:03 ShadowBot https://github.com/minetest/minetest/issues/12276 -- Cannot turn off development test mode 11:04 MTDiscord Gentoo is a Linux distro, but the issue is presumably unrelated to the fact that Gentoo is a Linux distro. 11:05 MTDiscord TBH the issue looks like a "possible close" to me. 11:11 sfan5 Zughy[m]: you could argue for or against the label in different ways but I had the exact thought luatic described 11:11 sfan5 ? 11:16 sfan5 rubenwardy: can you put the new repo into the engine group (permissions) 11:16 rubenwardy done 11:18 sfan5 hm why can I access part of the repo settings, that's new 11:18 sfan5 "Your site is published at http://www.minetest.net/benchmarks/" uhhhhh 11:19 rubenwardy I granted permissions for that 11:20 rubenwardy as I thought you might need it 11:20 MTDiscord A benchmark page sounds lovely 11:20 sfan5 ah 11:20 rubenwardy afaict, you can't revert back to minetest.github.io 11:20 sfan5 yeah but it's not supposed to go there 11:20 rubenwardy if that's what you were intending 11:20 sfan5 but it doesn't look like we have a choice 11:20 rubenwardy you can ask c55 for a subdomain and then enter it as a custom domain 11:21 rubenwardy like blog.minetest.net 11:21 MTDiscord what's up with the blog BTW, april is missing 11:22 MTDiscord will it just jump to may and fix the numbering issue? 11:22 rubenwardy it's WIP 11:22 rubenwardy I've done most of it, MisterE just has a few more things to add - but he's busy 11:54 sfan5 somehow the binaries built by irrlichtmt's CI are 1 MB larger than the local ones I used to build, weird 11:55 rubenwardy debug symbols? 11:56 sfan5 yeah thats it 12:47 sfan5 merging #12255 and #12278 soon if the CI of the latter passes 12:47 ShadowBot https://github.com/minetest/minetest/issues/12255 -- Sort out some issues with our CI setup by sfan5 12:47 ShadowBot https://github.com/minetest/minetest/issues/12278 -- Build-related stuff by sfan5 12:55 erle sfan5 btw in case you read the ”build systems a la carte” i'd be very interested in what opinions (if any, but i bet you have a lot) about build systems you reconsider afterwards 13:04 erle the approach i advocate is what they name “SHAKE” btw 13:04 erle with a bit of “fabricate” mixed in for good measure 13:04 erle (i strace the compiler to get the ”real” dependencies) 13:12 sfan5 I haven't 15:29 sfan5 btw since it got the supported by coredev label someone should review #11654, it's just a small feature 15:29 ShadowBot https://github.com/minetest/minetest/issues/11654 -- [no squash] Implement tool use sound by sfan5 18:10 appguru sfan5: can I close game#2807? 18:10 ShadowBot https://github.com/minetest/minetest_game/issues/2807 -- Compress all .obj files by ExeVirus 18:10 sfan5 if you want to 18:34 appguru does Minetest currently do any kind of batching on outgoing packets? when I call hud_add a million times, will I get a million UDP packets? 18:36 sfan5 packets merging is not supported 18:36 sfan5 -s 18:37 MTDiscord MT seems to rely on the overhead for UDP packets being relatively small, and leaves efficient batch handling up to the underlying network. 18:38 erle > when I call hud_add a million times 18:38 erle maybe you shouldn't 18:38 appguru obviously I'm interested in abnormal usage 18:38 appguru abusing HUD elements as pixels 18:39 appguru i.e. this is just for fun 18:39 appguru to push the engine limitations 18:39 rubenwardy a better example would be entity/object properties 18:39 appguru also erle, aren't you usually the one to point out engine breakage in ridiculous edge cases ;) 18:39 rubenwardy if you batched them, you could send multiple objects in one package and only the properties that changed 18:39 rubenwardy *packet 18:40 appguru pretty sure it does currently batch some obj attrs 18:40 rubenwardy which would reduce the number of packets quite a lot without making big packets 18:40 appguru agreed 18:40 sfan5 active object messages are batched 18:40 erle appguru i am also usually the one advocating strict limits and better architecture to ensure that such cases do not happen. 18:41 appguru strict architecture is the opposite of fun 18:41 erle programming is not fun though. it's horrible. 18:42 erle appguru if you want to have pixels in the hud, how about you garbage collect texture modifiers? then you can go nuts with it. 18:42 appguru I'm aware 18:43 appguru But implementing proper garbage collection there doesn't seem easy to me 18:44 MTDiscord Programming IS fun. And horrible. You don't last long in the programming world if you don't have a properly warped sense of fun. 18:44 erle to the contrary, i lasted long because it is hard to demotivate me further at this point. 18:45 erle furthermore i don't trust people who program for fun. what if it stops being fun? 18:45 erle :P 18:46 MTDiscord alright I tried it 18:46 MTDiscord and now my Minetest is frozen 18:47 Pexin never pick a career you enjoy. once it is a career, and you are on a shedule, and have requirements, you WILL hate it. it would be a shame to hate what you used to love. (and back to #minetest now) 18:47 erle anyways, appguru IMO garbage collection will be needed for anything super-dynamic with pixels. 18:48 appguru erle: yes, and HUD elements are garbage collected 18:48 MTDiscord HUD elements are also garbage, collected. ? 18:48 erle appguru you still can't do this efficiently 18:48 erle even if you batch all the stuff 18:49 MTDiscord At minimum you still have to send at least the equivalent of {hud_change:{text:"brown.png"}} to the client for each pixel, which sounds like a pretty painful way to encode an image :-) 18:49 erle yes 18:50 appguru Indeed it is 18:51 appguru But implementing GC for texmods is painful as well 18:51 appguru (ideally dynamic media would be GC'd) 18:51 appguru I would prefer not to have to use the [png texmod as a workaround 18:51 erle the [png texmod is still the stupidest thing ever 18:52 MTDiscord If I had media GC then I'd probably more or less immediately expand the skybox navigation features in NodeCore. 18:52 appguru it's the closest we are to going nuts with dynamic media, because texmod GC is easier than media GC 18:53 erle appguru, look, as an interesting case, take mcl_maps, which displays a 128 × 128 texture. each update of that texture is 128 × 128 × 4 bytes being filled with a decoded texture. that's 64k uncompressed. unless you are standing on a featureless plains, this image will take about 4 to 12 k over the network with the current implementation. and now think about how many pixels you want to update how often. 18:53 MTDiscord Is it really that much easier? It seemed to me like the lion's share of the GC problem is stuff that'd be common to all of them anyway. 18:53 erle why is media gc not easy btw? 18:54 MTDiscord The assumption of "once loaded, always loaded" is kind of distributed throughout MT now, for one thing 18:54 MTDiscord and of course one consequence of that is that nobody has needed to track usage, since it never mattered before 18:57 erle wouldn't it be similarly hard for texmods too then? 18:57 erle after all, their stuff gets turned into pixels 18:58 erle i mean gc-ing the texmod string woudn't do a whole much 18:59 MTDiscord Yeah, I mean, the point is that we'd have to identify that the computed texture is probably not in use anymore, remove the image and texture objects corresponding to it, and also ensure it returns to a state where if the engine encounters it again it would know to rebuild it. 19:00 MTDiscord I think though that just verifying that it's not being used right now would be tricky, because not everything that might be holding a reference to it will necessarily make that known. 19:01 erle i wonder what the worst case is right now. 19:04 erle the thing is, if i right now would make mcl_maps push a texture every second, the client would needd about 4MB RAM per minute minimum to keep up. 19:04 erle which is obviously why i don't do that 19:05 erle still, after an hour of playing it would be “only” 240 MB, so maybe it isn't too bad? 19:05 erle like if all of that is cleared at the end of the session obv 19:59 Zughy[m] #10392 possible close? 19:59 ShadowBot https://github.com/minetest/minetest/issues/10392 -- Add media alias 20:01 rubenwardy from the issue title, I expected that to be a texture pack thing 20:01 erle it is not 20:02 erle it is someone saying they want to rename their files and have other stuff not break 20:02 erle like other mods that depend on having that file 20:02 erle i think this is bonkers and the commenters seem to think too 20:03 erle like, just don't rename files, problem solved 20:06 rubenwardy merging #12115 and #12258 in 5 20:06 ShadowBot https://github.com/minetest/minetest/issues/12115 -- Default chat clickable weblinks by Froggo8311 20:06 ShadowBot https://github.com/minetest/minetest/issues/12258 -- Add benchmarks for json string serialize/deserialize by paradust7 20:11 MTDiscord or if you do want to rename files, copy them instead 20:11 MTDiscord does Minetest support symlinks for media? 20:37 Zughy[m] is #6914 still a thing? 20:37 ShadowBot https://github.com/minetest/minetest/issues/6914 -- Leveled nodeboxes lack leveled selection boxes 20:48 Zughy[m] ok, so: 4 days ago there were 200 "Feature request" issues with no "@ " label. Now there are 0 20:49 Zughy[m] ? 20:51 Zughy[m] 200 more unlabelled issues remaining (mostly unconfirmed bugs) and then this hell is over 20:53 MTDiscord then you get to finally move on to the hell of triaging the steady trickle of new issues as they arrive. 20:54 Zughy[m] I'm already doing that 20:56 Zughy[m] I must say, you core devs are batshit crazy for having done this all by yourselves for years. Like... just ask for help