| Time |
Nick |
Message |
| 02:11 |
|
Baytuch joined #minetest-dev |
| 02:16 |
|
Baytuch joined #minetest-dev |
| 02:24 |
|
Baytuch joined #minetest-dev |
| 02:49 |
|
Baytuch joined #minetest-dev |
| 03:03 |
|
Alias joined #minetest-dev |
| 03:04 |
schwarzwald[m] |
Maybe the need for better documentation of the development workflow can be represented as its own issue. |
| 03:07 |
|
Baytuch joined #minetest-dev |
| 03:10 |
|
Baytuch joined #minetest-dev |
| 03:11 |
schwarzwald[m] |
Sfan made a comment on the PR I believe, to the effect that build systems have maintenance costs. I'm not an expert either, but I've spent some time just reading about CMake and trying out simple things. My current analysis is that the cost of identifying those dependencies and coming up with a maintainable build architecture is very high. Doing it in a way that is really maintainable and robust probably involves implementing our |
| 03:11 |
schwarzwald[m] |
own CMake tooling that abstracts the build logic, and writing detailed documentation so that future build maintainers won't introduce more similar bugs. |
| 03:11 |
schwarzwald[m] |
Is that worth it? |
| 03:17 |
schwarzwald[m] |
Coming up with the idea is the hardest part. If we can describe such a system I will play around with it. |
| 03:19 |
|
Baytuch joined #minetest-dev |
| 03:19 |
schwarzwald[m] |
I don't think that's as expensive as switching the build system BTW. |
| 03:26 |
schwarzwald[m] |
Brainstorming here: one really low-short-term-cost fix is to force rebuilds automatically. The long term cost is huge because most of the time it works to not rebuild. Maybe a compromise could be found. |
| 03:26 |
schwarzwald[m] |
Better yet, if compile times can be optimized significantly, this becomes a non-issue. I don't know what that would entail. |
| 04:00 |
|
MTDiscord joined #minetest-dev |
| 04:12 |
|
Baytuch joined #minetest-dev |
| 04:19 |
|
Baytuch joined #minetest-dev |
| 04:53 |
|
Baytuch joined #minetest-dev |
| 05:06 |
|
Baytuch joined #minetest-dev |
| 05:16 |
|
Baytuch joined #minetest-dev |
| 05:21 |
|
Baytuch joined #minetest-dev |
| 06:01 |
|
calcul0n joined #minetest-dev |
| 06:02 |
|
Baytuch joined #minetest-dev |
| 07:36 |
|
cranezhou joined #minetest-dev |
| 08:20 |
|
erle joined #minetest-dev |
| 08:58 |
|
appguru joined #minetest-dev |
| 09:18 |
|
Baytuch joined #minetest-dev |
| 09:53 |
|
HuguesRoss6 joined #minetest-dev |
| 10:10 |
|
Fixer joined #minetest-dev |
| 12:04 |
|
erle joined #minetest-dev |
| 12:14 |
erle |
“Sfan made a comment on the PR I believe, to the effect that build systems have maintenance costs.” – that's undeniably true, but i maintain it is a ridiculous over-estimation here, most likely either because it is coming from a place of “i do not want to dive deep into the issue” or because, yes, this is indeed a really hard if not impossible problem to solve if you insist on identifying the dependencies b |
| 12:14 |
erle |
efore the build instead of generating them as a side effect. |
| 12:14 |
erle |
it's a graph traversal problem |
| 12:19 |
erle |
schwarzwald[m], basically, what we have with cmake is something described in section 2.1, what makes this problem hard to solve. section 2.3 shows how to solve it easily: https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf |
| 12:21 |
erle |
this is the kind of problem which is one or two orders of magnitude more code if you choose the wrong approach. |
| 12:24 |
erle |
schwarzwald[m] have you implemented or developed on a build system? |
| 12:39 |
schwarzwald[m] |
I have not. |
| 12:41 |
schwarzwald[m] |
> what makes this problem hard to solve |
| 12:41 |
schwarzwald[m] |
That's exactly what I'm saying. |
| 12:43 |
schwarzwald[m] |
I haven't considered implementing a custom build system or switching to some other build system as solutions. Sfan5 made it clear those are not desirable solutions. |
| 12:43 |
schwarzwald[m] |
You've implemented a build system before, so if you really think you know how to fix this, explain how instead of linking academic papers and telling other people it's easy. |
| 12:44 |
schwarzwald[m] |
The only constraint is that we need to use CMake, but I'm sure that's easy for you. |
| 12:44 |
erle |
that's bullshit |
| 12:44 |
erle |
the problem is that build systems have different power, depending on the approaches to the problem |
| 12:45 |
erle |
choosing a different algorithm to figure out dependencies makes solving this trivial – and cmake can not be modified to do that |
| 12:45 |
erle |
what's so hard to understand about that? |
| 12:45 |
schwarzwald[m] |
Nothing. That's what I've been saying since last night. |
| 12:46 |
erle |
yeah, but then how do you get to ”The only constraint is that we need to use CMake, but I'm sure that's easy for you.” ? |
| 12:46 |
schwarzwald[m] |
Sarcasm. |
| 12:46 |
erle |
oh okay |
| 12:46 |
erle |
sorry then |
| 12:47 |
schwarzwald[m] |
It sounds like we're agreed then that fixing this without switching build systems is probably not in scope. |
| 12:47 |
schwarzwald[m] |
I do still think the workflow should be documented though, maybe in the dev wiki or something. |
| 12:48 |
erle |
i have a suspicion that “documenting the workflow” is a task that no one is going to do |
| 12:49 |
erle |
because some problems are evidently unsolvable without ”rebuild everything every time” |
| 12:49 |
erle |
(while using cmake) |
| 12:50 |
erle |
i have asked for advice on how to bisect across a wide range of versions several times now and not gotten any good answers. maybe no one is actually doing automatic bisection and it's all manual. |
| 12:51 |
schwarzwald[m] |
Did no one mention git bisect? |
| 12:51 |
erle |
look, the problem for git bisect is that you need reliable builds |
| 12:51 |
erle |
if your build fails because of some problems with the build system, then bisect is worth nothing |
| 12:52 |
erle |
you can of course manually build every time, figuring out the correct magical incantation |
| 12:52 |
erle |
but it is not going to make it an easy or even automatic thing |
| 12:53 |
schwarzwald[m] |
> rebuild everything every time |
| 12:53 |
schwarzwald[m] |
Every time you switch branches, anyway. |
| 12:53 |
erle |
also, just to point it out again: “cmake does not capture all dependencies” is in scope without switching build systems. cmake can be told to evaluate directories or files directly. just because no one has done it, it does not mean no one can do it. |
| 12:53 |
schwarzwald[m] |
Including detaching HEAD to an earlier commit. |
| 12:53 |
erle |
schwarzwald[m] yes. now the problem is that if you use the same build dir as before, cmake can stumble over itself too. |
| 12:53 |
MTDiscord |
<luatic> Heh, I have an idea but I CBA to implement it |
| 12:53 |
erle |
you can end up in a situation in which you need to delete some files |
| 12:53 |
schwarzwald[m] |
We should represent our dependency tree correctly insofar as CMake's design philosophy permits. |
| 12:54 |
erle |
luatic tell me about your idea |
| 12:54 |
MTDiscord |
<luatic> You could tie MT commits to IrrlichtMT commits based on their date |
| 12:54 |
erle |
schwarzwald[m] this is what i am saying and why i am so frustrated by how sfan5 is like “this can not be solved perfectly, so i close the issue” |
| 12:54 |
schwarzwald[m] |
And I would recommend not merely discouraging in-tree builds, but completely forbidding them in the build setup. |
| 12:54 |
schwarzwald[m] |
luatic: Those issues are technically caused not by CMake, but by improper version management. |
| 12:55 |
MTDiscord |
<luatic> And then automatically have a script that checks out IrrlichtMT to the proper revision |
| 12:55 |
MTDiscord |
<luatic> schwarzwald[m]: Definitely agreed |
| 12:55 |
erle |
luatic matching MT commits to irrlichtMT is not the issue. the CI scripts by sfan5 do this already. |
| 12:55 |
MTDiscord |
<luatic> nice |
| 12:55 |
MTDiscord |
<luatic> you'll still have to rebuild both |
| 12:55 |
schwarzwald[m] |
The Minetest build setup already checks the IrrlichtMt version tag and throws an error if the API is incompatible. |
| 12:55 |
sfan5 |
you can easily check util/buildbot/buildwin32.sh for the correct irrlichtmt version that is guaranteed to build |
| 12:56 |
schwarzwald[m] |
Nobody updated the API properly so it doesn't work. Not the build system's fault. |
| 12:56 |
erle |
yes, but that is not a cmake or build system issue, it's an issue with people who have evidently little sense of how to manage separate code bases together determining how to manage separate code bases together. |
| 12:56 |
erle |
most people just use submodules and don't bother |
| 12:56 |
schwarzwald[m] |
Submodules would have the same problem. |
| 12:56 |
erle |
you can see it all over github with projects that need project-specific libraries that they update in lockstep |
| 12:57 |
sfan5 |
I have no idea what the topic of discussion is at this point |
| 12:57 |
schwarzwald[m] |
We basically agreed that it's a lot of work to try to fix the CMake dependency thing. |
| 12:58 |
schwarzwald[m] |
I would still like to hear erle's opinion on reducing compile time as an alternative solution. |
| 12:58 |
erle |
schwarzwald[m] you want an example? |
| 12:58 |
MTDiscord |
<luatic> What do you think, how large improvements to compile time are possible? |
| 12:58 |
schwarzwald[m] |
No, I want solutions. |
| 12:59 |
schwarzwald[m] |
luatic: I have no idea, but I'm also really curious. I haven't tried to profile a build before. |
| 12:59 |
erle |
luatic for full rebuilds it's probably very little. for incremental rebuilds, it's the difference between ”recompile everything that has a changed timestamp” (e.g. everything, if you switch a branch in git) and ”recompile the minimum set of things”. that suffices for you? |
| 13:00 |
erle |
i switch branches a lot, so i run into this problem all the time |
| 13:00 |
schwarzwald[m] |
I'm talking about reducing the compile time for a full rebuild. |
| 13:00 |
schwarzwald[m] |
On the assumption that we're going to recommend a clean rebuild every time you switch branches. |
| 13:00 |
schwarzwald[m] |
(out-of source though) |
| 13:00 |
schwarzwald[m] |
s//-/ |
| 13:02 |
erle |
schwarzwald[m] i have written the build rules for liberation circuit, a C++ game https://github.com/linleyh/liberation-circuit – avery pennarun has optimized them later on. the game can be built with both redo (or do) and make. it is a bit simpler than minetest, but an excellent showcase of how one would solve these kinds of problems. |
| 13:03 |
erle |
i wanted to recommend it as a case of how cmake stacks up, but it turns out the maintainer has ultimately chosen to remove the cmake build option lmao |
| 13:03 |
erle |
but that's just the last commit |
| 13:03 |
erle |
schwarzwald[m] does that satisfy your curiousity? |
| 13:03 |
schwarzwald[m] |
What, no. |
| 13:03 |
erle |
why? it's a large C++ project |
| 13:04 |
erle |
it has thousands of dependencies and thousands of non-existence dependencies and it can be built using both approaches (recursive top-down and toposort bottom-up) |
| 13:05 |
erle |
schwarzwald[m] what exactly is a ”solution” for you? |
| 13:06 |
erle |
if you want me to figure it out for minetest – i am not doing that as long as sfan5 is saying that no possible solution will be accepted and no other coredev steps up and says they would be willing to endorse something if it solves the dependency problems better than the current thing. |
| 13:07 |
schwarzwald[m] |
I think we've identified 3 problems: |
| 13:07 |
schwarzwald[m] |
- version management |
| 13:07 |
schwarzwald[m] |
- improper CMake dependency management (fixable) |
| 13:07 |
schwarzwald[m] |
- nonexistant dependencies (unfixable) |
| 13:07 |
erle |
no, there is at least one more |
| 13:07 |
erle |
also it is not ”nonexistant dependencies” |
| 13:07 |
schwarzwald[m] |
I can never remember the right term. :) |
| 13:08 |
erle |
it is non-existence dependencies. a dependency on the non-existence or existence of a file at a path. |
| 13:08 |
schwarzwald[m] |
Which one did I miss? |
| 13:08 |
erle |
unnecessary rebuilds |
| 13:08 |
erle |
fixing that actually lead to this whole issue |
| 13:08 |
schwarzwald[m] |
That doesn't follow from the other 3? |
| 13:08 |
erle |
nope |
| 13:09 |
schwarzwald[m] |
What's the root cause of the unecessary rebuilds, then? |
| 13:09 |
erle |
as it turns out, as long as make over-rebuilds stuff that does not need to be rebuilt, you rarely run into the case where you notice the missing dependencies |
| 13:09 |
erle |
as soon as you force make to only rebuild stuff that is in its dependency graph, the build does not work |
| 13:10 |
schwarzwald[m] |
I'm not here to talk about theoretical build system design. This is the Minetest dev channel. |
| 13:10 |
erle |
basically, the ”make unnecessarily rebuilds stuff all the time” bug masks the other ones to a large extent |
| 13:10 |
schwarzwald[m] |
If you want to talk about this we should PM. |
| 13:10 |
erle |
this is not theoretical, i have posted code to the issue that prevents make from rebuilding stuff all the time |
| 13:10 |
erle |
and then i ran into the other issues |
| 13:10 |
schwarzwald[m] |
What belongs here is concrete "how do we get from here to there" discussion. |
| 13:12 |
schwarzwald[m] |
You haxxed CMake. You're not supposed to go and manually interfere with the build setup like that. Of course it broke. |
| 13:12 |
erle |
look, solving version management is a people problem. submodules exist, other solutions exist, several people have suggested them over time, coredevs never agreed on anything. |
| 13:12 |
erle |
that is not true |
| 13:12 |
schwarzwald[m] |
Also, anyone who is not using Ninja should switch because so far it's been way faster, especially when checking out new branches. |
| 13:13 |
schwarzwald[m] |
Another concrete thing we can do: update the documentation to recommend using Ninja. I can PR that later. |
| 13:14 |
erle |
yes, ninja does a bunch of things better than make, i know that. but crucially, it rejected the following PR in 2015, which would make it possible for ninja to actually solve a large part of the unnecessary rebuilds: https://github.com/ninja-build/ninja/pull/929 |
| 13:14 |
erle |
schwarzwald[m] i do not believe you are going to have luck with this, as sfan5 said that adding a build system is not an option. |
| 13:14 |
erle |
i totally agree that ninja is better than what we have now |
| 13:14 |
erle |
i can still run circles around it with a very small shell script |
| 13:14 |
schwarzwald[m] |
You think I'm at odds with sfan5 here? |
| 13:15 |
schwarzwald[m] |
sfan5 When you have time, what are your thoughts on recommending Ninja builds in our README? |
| 13:15 |
erle |
well, you need to convince someone |
| 13:15 |
erle |
there are other devs |
| 13:15 |
erle |
but sfan5 did not say ”i do not want redo” or “i do not want what erle suggests”, but “no additional build systems” |
| 13:16 |
erle |
here https://github.com/minetest/minetest/issues/11749?notification_referrer_id=NT_kwDN7emwMjYyNTU1NTc1MTo2MDkwNQ#issuecomment-1192385010 |
| 13:17 |
erle |
“We will not add a second build system” |
| 13:17 |
MTDiscord |
<ROllerozxa> would be cool if the README could recommend using ninja with cmake (-G Ninja) alongside also changing it to recommend putting cmake build files inside of a build/ folder instead of doing a in-source build |
| 13:17 |
erle |
i totally agree. anything would be better than what is happening now. |
| 13:18 |
erle |
ROllerozxa don't forget the cleanup instructions once cmake paints itself into a corner |
| 13:18 |
erle |
you need to delete some files, CMakeCache.txt or so? |
| 13:19 |
ROllerozxa |
rm -rf build/ & remake the cmake build folder? |
| 13:19 |
ROllerozxa |
or I'm not sure what you mean |
| 13:19 |
schwarzwald[m] |
See, someone knows how it works. ^ |
| 13:21 |
schwarzwald[m] |
I work on a really slow computer so my build times are like half an hour. How long does it take for you ROllerozxa? Is it good? A little annoying? Really frustrating? |
| 13:21 |
MTDiscord |
<luatic> Only works if you didn't build in-source ^.^ |
| 13:21 |
ROllerozxa |
15 mins for a full build |
| 13:21 |
ROllerozxa |
with irrlicht included |
| 13:21 |
erle |
30+ minutes for me as well |
| 13:21 |
MTDiscord |
<luatic> < 2 mins for a full build here, Ryzen is really doing wonders; used to take half an hour on my previous laptop though |
| 13:22 |
schwarzwald[m] |
Awesome, so one solution we forgot to consider is all the dev's buying Ryzens. |
| 13:22 |
schwarzwald[m] |
s/dev's/devs/ |
| 13:23 |
schwarzwald[m] |
That's pretty good build time, although because Minetest supports old hardware it's going to be built on (relatively) old hardware probably. |
| 13:23 |
erle |
by the way, regarding complexity: the cmake build rules for minetest source (SLOC ~132k lines) are 18kb. by contrast, the redo build rules for liberation circuit (SLOC ~62k lines) are 1.5kb. |
| 13:24 |
MTDiscord |
<luatic> who said build rules scale linearly |
| 13:24 |
erle |
look, i have no better example. i just wanted to point out why i think the complexity that people know from cmake does not have to be taken as a given. |
| 13:25 |
schwarzwald[m] |
Our build system code is about as clean as our production code. Just saying. ^^ |
| 13:25 |
MTDiscord |
<luatic> hehe |
| 13:25 |
erle |
schwarzwald[m] well spoken hehehehe |
| 13:25 |
MTDiscord |
<luatic> appropriately clean™️ |
| 13:26 |
erle |
schwarzwald[m] i personally think that for any game many devs just have vastly better hardware than the majority of players or casual devs. it takes an unusual amount of empathy to actually step down from that high horse and make it better for the commoners. minetest is no exception. |
| 13:27 |
erle |
but that's a people problem and you can't replace the devs. so you have to improve the tooling to make it bearable. |
| 13:28 |
erle |
schwarzwald[m] ROllerozxa good luck on your ninja suggestions. i do not have high hopes, but i wish you well. |
| 13:28 |
erle |
schwarzwald[m] btw, how much faster is ninja vs make on your hardware for a full rebuild? |
| 13:31 |
ROllerozxa |
well it'd just be a readme build instruction change, hopefully one for the better :P |
| 13:32 |
erle |
ROllerozxa well, if that would fly, i could just maintain a separate set of build scripts and make a PR to mention them in the README as well i guess. |
| 13:32 |
erle |
it would kill the maintenance argument dead |
| 13:34 |
ROllerozxa |
adding `ninja-build` to the build dependencies and `-G Ninja` to cmake does not exactly ruin the workflow for people who would look past that and use the regular unix makefile generator anyways |
| 13:35 |
ROllerozxa |
but at the same time, I do `-G Ninja` on any projects even if the README doesn't state it so ¯\_(ツ)_/¯ |
| 13:36 |
|
vampirefrog joined #minetest-dev |
| 13:36 |
erle |
ROllerozxa yes, and adding ”redo” as a possible build command also does not prevent anyone from using cmake, so i don't get the opposition. but you know, maintenance costs! :P |
| 13:37 |
ROllerozxa |
well unix makefiles/ninja are both generators provided by cmake so it works from the same build files |
| 13:37 |
ROllerozxa |
and I assume 'do' or 'redo' is like scrapping cmake in favour of something else? |
| 13:37 |
erle |
that kinda reduces the scope of things that ninja can solve. because if cmake does not know dependencies, how is it going to tell ninja about them? |
| 13:38 |
erle |
so i predict that any improvements that ninja has over cmake are going to run into the same issue of “cmake does not know all the dependency relations” eventually. |
| 13:39 |
erle |
ROllerozxa nah, it's more like adding a bunch of very small shell scripts (that may even call cmake helpers) that guarantee a fast and correct incremental build. |
| 13:40 |
erle |
just look into the man pages i wrote, you can figure it out within 30 minutes probably how it would help |
| 13:40 |
erle |
the fact that cmake is such a huge abomination may be a reason for the fear of tech debt |
| 13:40 |
schwarzwald[m] |
Hey, if it improves it that much, that's great! But there's more to making changes than telling people to read man pages. |
| 13:42 |
erle |
schwarzwald[m] the only thing that is necessary for me to make a proof-of-concept is one coredev saying ”if you manage to improve build times and/or build reliability, i am willing to at least look at it”. so far, i can only count one core dev who says “even if you managed to improve build times and/or build reliability, i am not willing to look at it – in fact, i am actively against anything you suggested so far” |
| 13:42 |
erle |
you have to solve the human factor first |
| 13:43 |
erle |
and i don't think it is unreasonable for sfan5 to have absolutely no interest in maintaining any of this. |
| 13:43 |
erle |
but then again the current cmake thing is also basically unmaintained, so … |
| 13:44 |
erle |
schwarzwald[m] you have any idea how to find dependencies that cmake misses? |
| 13:44 |
erle |
i mean i have a bunch, but they were all rejected. maybe you can come up with something new. |
| 13:44 |
schwarzwald[m] |
It's an open source project, so if we can find enough people who have similar priorities we could probably get a group together and start cleaning stuff up. |
| 13:45 |
erle |
nope, you can see in the map-enlargement PR from proller how that goes. you can do a LOT of work for nothing if you don't get buy-in from coredevs. |
| 13:45 |
erle |
not saying that prollers solution was acceptable, but it was rejected on principle AFAIK, not on its technical merits |
| 13:45 |
schwarzwald[m] |
One way to find dependencies CMake misses is to make sure we aren't overspecifying our dependencies. You can have CMake generate a dot graph of the project's dependency tree, and that should be a good place to start. |
| 13:46 |
erle |
please tell me how to do that. |
| 13:46 |
erle |
i mean, redo can do this and i have never heard about other systems that can actually do that in a useful way. |
| 13:47 |
erle |
okay, i did this: cmake '--graphviz=deps.dot' |
| 13:48 |
erle |
OH LOL |
| 13:48 |
erle |
that graph is laughably simple |
| 13:48 |
erle |
that explains A LOT |
| 13:48 |
schwarzwald[m] |
Yep, I had just looked that up, but I guess you found it. |
| 13:48 |
schwarzwald[m] |
It sounds like it was already useful in revealing some problems. |
| 13:48 |
schwarzwald[m] |
That sure was quick. |
| 13:49 |
erle |
how can i get the full graph? |
| 13:49 |
schwarzwald[m] |
Do you maybe need to give it the build directory? |
| 13:50 |
schwarzwald[m] |
`cmake -S . -B path/to/build --graphviz=deps.dot`? |
| 13:50 |
erle |
nah, i mean with all files |
| 13:51 |
erle |
look, if cmake truly only knows about what is in this graph, it is underestimating the dependencies by about an order of magnitude. and i do not think it is THAT bad. |
| 13:51 |
erle |
maybe two orders of magnitude |
| 13:52 |
schwarzwald[m] |
erle: I don't think it shows individual source file dependencies |
| 13:52 |
schwarzwald[m] |
It should show all library dependencies. |
| 13:53 |
erle |
that's not useful |
| 13:53 |
erle |
also it's about a third or so of what it should be |
| 13:53 |
schwarzwald[m] |
Well, that sounds like something we can fix. |
| 13:53 |
erle |
good luck. i bet the 18kb of cmake rules do nothing else than trying to figure out which library is where lol |
| 13:54 |
schwarzwald[m] |
There's a lot of undocumented setup etc. as I recall. |
| 13:54 |
schwarzwald[m] |
I just need to know which library dependencies are missing. |
| 13:58 |
erle |
look, this graph shows about ~40 dependency nodes. for comparison, for liberation circuit (a project of 1/3rd the size of minetest), redo-dot generates a graph with over 400 dependency nodes and over 200 non-existence dependency nodes and over 5000 edges. |
| 13:58 |
erle |
(with less than 10% of the build rules, just to point that out again) |
| 13:59 |
erle |
if cmake is truly not knowing *that much* about the dependency tree (which i do not think is true), then you can replace it in a week or so with literally anything and come out ahead on maintenance costs |
| 13:59 |
erle |
so please tell me how to get the *real* graph |
| 14:01 |
schwarzwald[m] |
https://cmake.org/cmake/help/latest/module/CMakeGraphVizOptions.html |
| 14:01 |
schwarzwald[m] |
CMake actually does not know the source files its dependencies depend on. |
| 14:01 |
erle |
yes, we need them to actually estimate the scale of the problem |
| 14:01 |
schwarzwald[m] |
It does know the source files Minetest directly needs, but they aren't treated as dependencies. |
| 14:02 |
erle |
uh what |
| 14:02 |
erle |
why |
| 14:02 |
erle |
if i change one of those files, the target is rebuilt |
| 14:02 |
erle |
like if you change something in cloud.cpp then it will rebuild cloud.o and relink |
| 14:02 |
erle |
so how can cmake not know about that |
| 14:11 |
erle |
schwarzwald[m], how to specify the options? and don't point me to the docs, give me an example command line |
| 14:44 |
|
appguru joined #minetest-dev |
| 14:57 |
sfan5 |
<schwarzwald[m]> sfan5 When you have time, what are your thoughts on recommending Ninja builds in our README? |
| 14:57 |
sfan5 |
we can do that |
| 14:57 |
sfan5 |
along with recommending out of tree builds |
| 14:58 |
|
kilbith joined #minetest-dev |
| 14:59 |
sfan5 |
btw speaking of ninja this is quite annoying https://github.com/ninja-build/ninja/issues/174 (haven't tried the cmake option for that yet) |
| 16:48 |
|
fluxionary joined #minetest-dev |
| 17:05 |
erle |
so it turns out that i already had a working proof-of-concept of rebuilding minetest 5.4.1 in a manner that tries to capture all dependencies correctly and figure out the set of minimal rebuilds, using redo and two shell scripts that together were together about 50 lines. what a shame that such a tiny, but very useful addition would never be accepted! |
| 17:05 |
erle |
(i am updating them right now) |
| 17:09 |
erle |
given what i have learned about this problem, i think it would take maybe 500 to 1000 lines more to get to feature parity with the ~4700 lines of cmake configuration. |
| 17:09 |
erle |
(i.e. working on all platforms) |
| 17:18 |
erle |
this also means that i can make cmake and make behave 100% correctly, by using the dependency information gained using that |
| 17:19 |
erle |
(which is pointless, because the shell scripts that gather the dependency information already leave a correctly built minetest binary as a side effect) |
| 17:20 |
erle |
schwarzwald[m] are you interested? |
| 17:26 |
erle |
celeron55 HuguesRoss6 rubenwardy Krock if i can demonstrate a way to build minetest on linux in a way that is faster for incremental builds, will never rebuild things unnecessarily just because its timestamp changed (e.g. git branch switch), and will have next to no miscompilations (due to increased coverage of dependencies), is anyone of you willing to take a look at it and possibly support a PR regarding that? |
| 17:31 |
Krock |
CMake exists since 2000 and I believe its system is good enough for what we need. I've yet to encounter a miscompile in first place |
| 17:33 |
erle |
Krock to encounter a miscompile, try bisecting a bug that lies between 5.5.0 and 5.4.1, using incremental rebuilds every time. switching between branches that have diverged a lot also does it. preventing make from needlessly rebuilding things also does it. you probably have never *noticed* a miscompile, but they happen all the time. |
| 17:33 |
celeron55 |
erle: it has to be cross platform and easy to maintain by the existing team, there's no resources to maintain more build systems |
| 17:33 |
HuguesRoss |
I'm not opposed to looking at anything on principle really, but I can't guarantee I'll support it without knowing what it is |
| 17:34 |
HuguesRoss |
...but if it's a unity build, I can give you a hard no immediately |
| 17:34 |
erle |
i simply want someone to look at it, not commitment for support for a cat-in-the-bag |
| 17:34 |
erle |
HuguesRoss, what is unity? |
| 17:35 |
celeron55 |
erle: but really, practically nobody seems to be complaining about incremental build times |
| 17:35 |
HuguesRoss |
It's a method of compiling C++ faster, sometimes also called blobbing. We do it at work, essentially it merges all source file into a single file pre-compilation in order to reduce file access times during the build process. FastBuild uses this as its basis, and it *is* indeed much faster but there are severe trade-offs |
| 17:35 |
erle |
celeron55 yeah i get that, but would you be willing to look at a proof-of-concept? |
| 17:35 |
erle |
HuguesRoss that sounds horrible. no, the only thing i do is compile smarter, by not doing the stupid toposort thing that cmake does. |
| 17:36 |
celeron55 |
if i have the time, yes, an efficient build system interests me. but i'm quite busy so can't promise |
| 17:36 |
HuguesRoss |
The biggest being that the code you compile is no longer necessarily the code you wrote, because cpp files can affect each other |
| 17:36 |
erle |
celeron55, i don't want a promise, just something else than the sfan5 stance “i do not have this problem and i reject all of your solutions without looking at them because i do not even want to care” |
| 17:36 |
celeron55 |
erle: it's interesting for future projects at least |
| 17:37 |
erle |
HuguesRoss it seems you know more than me about compiling. so i guess i can show you when i have something that works and you can tell me its faults? |
| 17:37 |
|
Yad joined #minetest-dev |
| 17:37 |
celeron55 |
in terms of the logistics of developing and building MT, it's probably not beneficial |
| 17:38 |
HuguesRoss |
ehh, I wouldn't say that. My knowledge of compiling is extremely specialized to the tech used in AAA games, so it probably won't help here |
| 17:38 |
erle |
i think you underestimate the amount of improvements that are possible. |
| 17:38 |
celeron55 |
so whether it's interesting depends on your focus. we'd like people here to focus on MT of course |
| 17:39 |
erle |
celeron55 right now my problem is that “git bisect” is basically useless when builds can a) randomly miscompile (you will not find the buggy revision that way) b) take more than 10 minutes even if only a little bit has changed |
| 17:39 |
erle |
that's why i am focusing on incremental builds |
| 17:39 |
erle |
and correctness of dependencies |
| 17:40 |
erle |
but it will basically never affect you if you only skip around git HEAD |
| 17:40 |
celeron55 |
erle: if you make a blog post about the results, i'm sure many will read it. whether people will actually go and read the code and try it, that's harder to do |
| 17:43 |
erle |
HuguesRoss interesting. from your toolchain experience, do you know anything about correct dependency handling that i may not? i mean what i do is extremely crude – it's just that as long as i am faster than ninja for common cases, i am not really motivated to speed a build up further. |
| 17:43 |
erle |
HuguesRoss i am especially interested in the graphviz output of dependency graphs. the cmake one i tried earlier is kinda useless. |
| 17:44 |
HuguesRoss |
Probably not anything useful, I work at a scale where dependency graphs can be accurately described as 'lovecraftian' |
| 17:44 |
erle |
oh but that's exactly what i mean |
| 17:44 |
HuguesRoss |
I have yet to find a visualization method that works for those |
| 17:45 |
erle |
the one i use is relatively simple: redo-dot takes the name of a source file and then shows everything that depends on it and all its dependencies and non_existence dependencies. you could of course do second-order dependencies here, but it worked for my purpose. |
| 17:45 |
erle |
in a web interface, you can have a similar view using d3.js or so |
| 17:46 |
HuguesRoss |
Ah, source dependencies |
| 17:48 |
HuguesRoss |
I was thinking modules. I actually worked on tooling for that once, but it's proprietary to my employer. IIRC we used a custom analysis program based on clang to generate a database of files, types, methods, variables, macros, etc... basically every 'thing' in the code, shoved it all into a giant ElasticSearch database then used custom GUI tooling |
| 17:48 |
HuguesRoss |
to query it and display local usage |
| 17:49 |
HuguesRoss |
Not sure that's necessary for a project like this tho lol |
| 17:49 |
erle |
well, i just shove everything that can be a dependency into a file |
| 17:49 |
erle |
and i have never worked on projects where that was not feasible |
| 17:49 |
erle |
target compilation depends on CPPFLAGS? they get an extra file which can be timestamped and hashed |
| 17:50 |
erle |
same for build rules |
| 17:50 |
HuguesRoss |
It's feasible for most use-cases but not ours |
| 17:50 |
HuguesRoss |
We have too much code for it |
| 17:50 |
erle |
the most common build rule error i see is not rebuilding target files when their build rules change, because they do not depend on the build rules |
| 17:51 |
HuguesRoss |
That comes back to my point--most of my experience is with projects at roughly the top-end of codebase sizes, so most of the tricks I know do not scale down well |
| 17:51 |
erle |
“scale down” is a weird thing to say with the size of minetests codebase. how big in terms of SLOC count are the projects, millions, tens of millions? |
| 17:51 |
HuguesRoss |
Tens of millions |
| 17:52 |
erle |
i see |
| 17:52 |
HuguesRoss |
Hundreds of modules, dozens of compilation targets per module |
| 17:52 |
HuguesRoss |
Enough to need a distributed compilation network for devs |
| 17:53 |
erle |
maybe that's a reason why people say ninja et al are fast while i can run circles around it with a small shell script. i can not guarantee that my solution would work for projects exceeding a miliion lines of code. |
| 17:53 |
erle |
simply because i have never run into that. a few hundred thousand lines of code, that works. |
| 17:54 |
HuguesRoss |
maybe! We don't use ninja either so I can't comment, but certainly it's important to distinguish between 'is fast' and 'stays fast'. The former is probably more important for projects like MT if that was the metric to compare |
| 17:54 |
erle |
for example, few people think about that you can actually run out of space when invoking a compiler the usual way. i assume that is something you have encountered HuguesRoss? |
| 17:54 |
erle |
like run out of space for filenames on the command line etc. |
| 17:55 |
HuguesRoss |
nah, we're ok in that regard |
| 17:56 |
HuguesRoss |
iirc we use files to pass around filenames, and we have terabytes of disk space and many gigs of ram so we're good on that end |
| 17:56 |
HuguesRoss |
I don't recall all the specifics though, haven't had to care for about a year and a half |
| 17:57 |
erle |
we have never needed a compilation network at work, but years ago i managed to cut a ~2 hour jenkins build to about 5 minutes. unfortunately, i found a dependency error where not-cleaning-up-the-build-dir and relying-on-a-previously-built-artifact interacted badly as soon as a) that artifact was no longer built b) the work dir on jenkins was cleaned up. fun times! |
| 17:57 |
erle |
basically, not tracking dependencies meant that the CI was totally fine, but for months the thing had never been able to be built from scratch |
| 17:57 |
HuguesRoss |
Yeah, that's always fun |
| 17:58 |
HuguesRoss |
I think the worst issues I've seen are when you discover some part of the build that's order-sensitive |
| 17:58 |
erle |
given that you mentioned AAA games, what's the usual way to figure out the shader instruction limit? |
| 17:58 |
HuguesRoss |
no idea, not my department |
| 17:58 |
erle |
ah, sorry |
| 17:58 |
HuguesRoss |
My knowledge is purely tools dev and some pipeline dev |
| 17:58 |
erle |
i see |
| 17:59 |
HuguesRoss |
With a bit of accumulated base-level knowledge from schooling |
| 17:59 |
HuguesRoss |
I know all the basics, but nothing advanced outside of my specialization |
| 17:59 |
erle |
well, sometimes people have weird blobs of knowledge |
| 17:59 |
HuguesRoss |
True! |
| 18:00 |
HuguesRoss |
If you ever have a GPU OOM after changing build systems, let me know |
| 18:02 |
erle |
it's pretty trivial to make the intel integrated GPUs go OOM, they don't have much memory. but they handle it kinda-gracefully, as in rendering becomes really slow first (same as with exceeding the shader instruction limit) and only falls over and die if you push it. |
| 18:02 |
erle |
i have no knowledge of the internals, but compared to other things that just refuse to work if you poke them it seems sensible |
| 18:02 |
erle |
or maybe they are not really OOM when they become slow |
| 18:03 |
erle |
HuguesRoss so what was the story behind that? |
| 18:03 |
HuguesRoss |
yeah, so as usual the culprit was unity builds |
| 18:03 |
HuguesRoss |
Basically, after changing build systems the game would run your PC out of VRAM with no code having changed... |
| 18:04 |
erle |
oops |
| 18:04 |
erle |
btw, what is your opinion on shared precompiled headers? avery pennarun made liberation circuit use that and i wonder if there are any downsides. |
| 18:04 |
HuguesRoss |
They seem fine, as long as you maintain them |
| 18:05 |
erle |
well, it does speed up the build |
| 18:07 |
HuguesRoss |
back to the story. After a lot of digging, we discovered that two unrelated parts of the codebase were defining unrelated #defines with the same name. Because of unity builds, these two headers that were never concurrently included in any file *but both could apply to some files when they combine*. So by changing build systems, the order of the |
| 18:07 |
HuguesRoss |
compiled files changed and the defined values were reversed... causing a debug flag to trip and render a sphere every ~1 meter in the city simultaneously |
| 18:07 |
HuguesRoss |
And that's how you lose 4 gigs of VRAM by changing build systems |
| 18:08 |
HuguesRoss |
Well, probably more than 4. But that's all we had at the time XD |
| 18:10 |
erle |
oh, hmm, seems i have an include problem |
| 18:14 |
erle |
HuguesRoss i may have asked this before, but how do you handle non-existence dependencies? e.g. changing fallback paths or so. do you use build systems that are like what kragen posted in #minetest recently, where you have te specify every input or else it will not be visible to the build? |
| 18:15 |
erle |
i ask because the minetest build i am trying currently straces the compiler, which is not exactly the fastest way to do it |
| 18:16 |
erle |
the overhead is low enough though, and it is orders of magnitude less complexity than all the filesystem tricks i know |
| 18:19 |
HuguesRoss |
I think if a dep is missing we just fail, our build environments are generally preset down to the drive letter |
| 18:20 |
HuguesRoss |
And before you ask "isn't that incredibly brittle?" yes |
| 18:20 |
HuguesRoss |
it is |
| 18:21 |
erle |
well, that's one way to do it |
| 18:21 |
erle |
literally all embedded devs i know do it like that |
| 18:21 |
erle |
sometimes even without incremental rebuilds, so one less chance for error |
| 18:21 |
HuguesRoss |
It certainly has its upsides |
| 18:21 |
HuguesRoss |
Downside is, if any non-prog needs to compile in a hurry they have a lot of hoops to jump through |
| 18:22 |
HuguesRoss |
thankfully, not that common |
| 18:22 |
erle |
i once asked mitch altman about if he sees the use for a 100% correct tiny build system and he was like “well, back in the 80ies, of course. but nowadays, my builds take 2 seconds, so i just rebuild everything every time!” |
| 18:24 |
erle |
can anyone help me with this compilation error? https://mister-muffin.de/p/XWrK.txt |
| 18:24 |
erle |
> fatal error: irrTypes.h: No such file or directory |
| 18:24 |
erle |
which include directory did i forget? |
| 18:26 |
erle |
the file is in ./lib/irrlichtmt/include/irrTypes.h … so maybe relative specifications are useless here |
| 18:27 |
erle |
oh, indeed |
| 18:28 |
erle |
because the file is not in src, but src/client |
| 18:49 |
sfan5 |
does anyone have opinions on https://github.com/minetest/minetest/pull/12571#issuecomment-1193106245 |
| 18:50 |
|
YuGiOhJCJ joined #minetest-dev |
| 18:54 |
rubenwardy |
could the irrlicht manifest say the lowest irrlicht version it's compatible with? |
| 18:54 |
rubenwardy |
really, this should be done with MAJOR increments |
| 18:54 |
erle |
what rubenwardy says |
| 18:55 |
rubenwardy |
== is also not a big deal |
| 18:55 |
erle |
i have noticed that many people are embarassed about increasing the major version on every incompatible increment |
| 18:55 |
erle |
but really, it's much more honest to increase the version if you are going to increase the incompatibility |
| 18:57 |
sfan5 |
irrlicht revisions are pretty much never compatible |
| 18:57 |
erle |
yeah but that's on them |
| 18:57 |
sfan5 |
so == it is |
| 18:57 |
erle |
you can do better |
| 18:58 |
sfan5 |
ok |
| 18:58 |
erle |
if you are doing that, what exactly is the argument against adding irrlichtmt as a submodule again? that would solve it too |
| 18:58 |
erle |
i mean, there was probably some drawback, but i forgot |
| 18:58 |
sfan5 |
irrlichtmt is essentially being kept just like a submodule |
| 18:58 |
sfan5 |
hence all the incompatiblities and updating in (almost) lockstep |
| 18:59 |
erle |
yes, but once you decide for lockstep updating, why not make it a submodule for real |
| 18:59 |
erle |
it's not like users need a choice which version to use |
| 18:59 |
erle |
or do they? |
| 19:00 |
erle |
(if there was a wide range of compatible version, a submodule would make less sense obv) |
| 19:02 |
sfan5 |
current git and 5.6.0 will technically be compatible with 1.9.0mt6 and 1.9.0mt7, however I don't want people using mt6 because that is missing bugfixes that have been made in the meantime |
| 19:22 |
MTDiscord |
<x2048> erle: That SM code.... compiles on Ubuntu with gcc -o sm main.c -I/usr/include -I/usr/include/GL -lm -lGL -lGLU -lglut -lc |
| 19:22 |
MTDiscord |
<x2048> you need freeglut3-dev installed |
| 19:23 |
erle |
x'll try |
| 19:24 |
MTDiscord |
<x2048> any questions on that -> #minetest, ok? |
| 19:25 |
erle |
ok |
| 19:33 |
|
appguru joined #minetest-dev |
| 19:49 |
sfan5 |
will merge #12571 #12572 once builds pass |
| 19:49 |
ShadowBot |
https://github.com/minetest/minetest/issues/12571 -- [don't squash] Bump IrrlichtMt to newest revision by sfan5 |
| 19:49 |
ShadowBot |
https://github.com/minetest/minetest/issues/12572 -- Let core.get_mod_storage be called multiple times by TurkeyMcMac |
| 19:56 |
|
x2048 joined #minetest-dev |
| 20:01 |
erle |
could anyone tell me whatis wrong in the following compile? https://mister-muffin.de/p/is9_.txt |
| 20:01 |
erle |
if i get that one last thing working, i have my sub-100-lines-with-correct-and-fast-incremental-build proof-of-concept |
| 20:02 |
erle |
and can present you the entire dependency graph that it generates |
| 20:02 |
schwarzwald[m] |
Did you mean to invoke `g++`? You wrote `c++` in the command. |
| 20:03 |
schwarzwald[m] |
erle: ^ |
| 20:04 |
erle |
schwarzwald[m], nope, that's not it |
| 20:04 |
erle |
; readlink -f /usr/bin/c++ |
| 20:04 |
erle |
/usr/bin/i686-linux-gnu-g++-10 |
| 20:24 |
sfan5 |
why are you invoking the preprocessor for linking? |
| 20:25 |
sfan5 |
but really if you need this kind of help to write your proof of concept I don't see a build system that can match CMake happening |
| 20:33 |
erle |
you forget maybe that while i have a lot of build system domain knowledge, i am behind most in this channel on C++ knowledge. my solution is not C++-specific and simply not knowing how to mash object files together does not invalidate my knowledge of other areas. |
| 20:35 |
sfan5 |
anyway you can check src/CMakeFiles/minetest.dir/link.txt for what cmake does, going by your error you have an empty string argument somewhere |
| 20:42 |
erle |
thx |
| 20:53 |
|
YuGiOhJCJ joined #minetest-dev |
| 21:00 |
erle |
sfan5, the problem was indeed an empty CFLAGS (i had renamed it to CPPFLAGS). what exactly do you mean with the preprocessor? i mean, using c++ usually works, is it bad somehow? |
| 21:01 |
sfan5 |
was me being confused about c++ / g++, sorry |
| 21:01 |
sfan5 |
`cpp` would be the preprocessor |
| 21:01 |
sfan5 |
(for both C and C++) |
| 21:12 |
|
proller joined #minetest-dev |
| 21:26 |
|
fluxionary joined #minetest-dev |
| 22:21 |
|
vampirefrog joined #minetest-dev |
| 22:34 |
|
panwolfram joined #minetest-dev |
