| Time |
Nick |
Message |
| 00:00 |
|
federalagent joined #minetest |
| 00:00 |
|
Megaf joined #minetest |
| 00:00 |
|
olliy joined #minetest |
| 00:00 |
|
olliy joined #minetest |
| 00:01 |
|
Leopold joined #minetest |
| 00:04 |
|
panwolfram joined #minetest |
| 00:05 |
|
frostsnow joined #minetest |
| 00:05 |
|
mazes_80 joined #minetest |
| 00:18 |
|
fluxionary joined #minetest |
| 00:45 |
|
Leopold joined #minetest |
| 01:12 |
|
Lesha_Vel joined #minetest |
| 01:16 |
|
fling joined #minetest |
| 01:20 |
|
smk joined #minetest |
| 01:25 |
|
TheCoffeMaker joined #minetest |
| 01:26 |
|
fluxionary joined #minetest |
| 01:37 |
|
fling_ joined #minetest |
| 02:07 |
|
HumanGeek joined #minetest |
| 02:12 |
|
HelenasaurusRex joined #minetest |
| 03:12 |
|
proller joined #minetest |
| 03:29 |
|
[MTMatrix] joined #minetest |
| 03:34 |
|
[MTMatrix] joined #minetest |
| 03:46 |
|
thelounge650 joined #minetest |
| 03:47 |
|
[MTMatrix]_ joined #minetest |
| 03:48 |
thelounge650 |
test |
| 03:48 |
[MTMatrix]_ |
<MisterE>Well, it looks like the new matrix bridge is working, on the test run |
| 03:50 |
|
[MTMatrix] joined #minetest |
| 03:55 |
|
[MTMatrix] joined #minetest |
| 03:56 |
|
MisterE123 joined #minetest |
| 03:56 |
MisterE123 |
test |
| 03:59 |
MTDiscord |
<mistere_123> oh thats a problem: on matrix, I cant see who is speaking from discord |
| 04:00 |
|
MTDiscord joined #minetest |
| 04:01 |
|
[MTMatrix] joined #minetest |
| 04:02 |
MTDiscord |
<mistere_123> Is this any better? |
| 04:02 |
[MTMatrix] |
|MisterE|nope |
| 04:05 |
|
[MTMatrix] joined #minetest |
| 04:06 |
MTDiscord |
<mistere_123> test again (sorry) |
| 04:09 |
|
[MTMatrix] joined #minetest |
| 04:12 |
MisterE123 |
Matrix bridge is down again due to discord user ids not showing up in matrix |
| 04:15 |
|
fluxionary_ joined #minetest |
| 04:52 |
|
liceDibrarian joined #minetest |
| 05:46 |
|
liceDibrarian joined #minetest |
| 06:04 |
|
calcul0n joined #minetest |
| 06:25 |
|
calcul0n_ joined #minetest |
| 06:26 |
|
jaca122 joined #minetest |
| 06:58 |
|
calcul0n_ joined #minetest |
| 07:00 |
|
pingus joined #minetest |
| 07:35 |
|
calcul0n_ joined #minetest |
| 08:07 |
|
s20 joined #minetest |
| 08:46 |
|
appguru joined #minetest |
| 08:47 |
|
Desour joined #minetest |
| 09:05 |
|
jaca122 joined #minetest |
| 09:15 |
|
definitelya joined #minetest |
| 09:35 |
|
YuGiOhJCJ joined #minetest |
| 09:58 |
|
s20 joined #minetest |
| 11:38 |
|
TomTom joined #minetest |
| 11:55 |
|
vampirefrog joined #minetest |
| 12:06 |
|
est31 joined #minetest |
| 12:12 |
|
est31 joined #minetest |
| 12:26 |
|
s20 joined #minetest |
| 12:53 |
|
qqq joined #minetest |
| 13:46 |
|
behalebabo joined #minetest |
| 13:49 |
|
s20 joined #minetest |
| 14:03 |
|
appguru joined #minetest |
| 14:17 |
Krock |
Sokomine: there are a few open issues about errors and also pull requests on https://github.com/Sokomine/mg_villages/issues . Might you have time to look at them? Alternatively it could also be arranged to move the repository to minetest-mods so that multiple contributors could maintain it |
| 14:17 |
Krock |
including you, of course. |
| 14:26 |
|
olliy joined #minetest |
| 14:27 |
|
TheCoffeMaker joined #minetest |
| 14:27 |
erle |
public service announcement: mm3d is a good 3d modeling program (but you need to use the C locale, because it is confused about commas and dots in file formats when saving otherwise) |
| 14:28 |
erle |
also don't be surprised if installing mm3d gets you blender, it's the stupidest dependency in debian ever lol |
| 14:36 |
|
TheCoffeMaker_ joined #minetest |
| 15:10 |
|
TheCoffeMaker joined #minetest |
| 15:19 |
|
TheCoffeMaker_ joined #minetest |
| 15:52 |
muurkha |
erle: thanks! |
| 15:52 |
muurkha |
erle: glad to hear about your text rendering library and the goatseing |
| 15:56 |
muurkha |
mm3d would benefit from a tutorial I think |
| 16:01 |
erle |
muurkha i hope kilbith is gone for good? |
| 16:01 |
erle |
i mean what i heard was broadly that he was always a total ass to everyone |
| 16:01 |
ROllerozxa |
kilbith is almost certainly gone for good |
| 16:01 |
erle |
nice |
| 16:02 |
|
Malo95 joined #minetest |
| 16:02 |
erle |
hey, were anyone of you at ccc camp? if so, did we maybe meet? also, anyone going to be at congress if there is one? |
| 16:02 |
Malo95 |
hey has anyone noticed skinsdb is offline? http://minetest.fensta.bplaced.net/ |
| 16:02 |
Malo95 |
whats the deal with this, has a mirror popped up or are we without a skins dabatase currently? |
| 16:02 |
muurkha |
I think you drove away kilbith this time |
| 16:02 |
muurkha |
seems unlikely he's gone for good tho |
| 16:03 |
Malo95 |
whats going on with kilbith |
| 16:03 |
ROllerozxa |
Malo95: it's been down for a while, unfortunate but inevitable considering it's been hosted on a freehost |
| 16:03 |
Malo95 |
is there any backup |
| 16:03 |
muurkha |
he's been gone since the last time he argued with erlehmann |
| 16:03 |
Malo95 |
any mirror |
| 16:03 |
ROllerozxa |
muurkha: well he's banned and I don't think a ban appeal is on the table any time soon for him considering his behaviour |
| 16:03 |
Malo95 |
whats the original author say, any comment from him? |
| 16:03 |
muurkha |
ROllerozxa: that's good to hear |
| 16:04 |
muurkha |
but he could join with another name if he wanted to grief people |
| 16:05 |
Malo95 |
So currently there is no mirror or any way for people to get skins in skinsdb? |
| 16:05 |
ROllerozxa |
Malo95: the hoster has been MIA for a while afaik, so something like this was pretty inevitable coming |
| 16:05 |
Malo95 |
The mod itself ships with no skins, so if the db site it down there is also no place to download the skins. |
| 16:05 |
Malo95 |
Ah ok so the guy is MIA too, damn. |
| 16:05 |
Malo95 |
Some one should have backed it all up but |
| 16:06 |
Malo95 |
it's not been down that long |
| 16:06 |
ROllerozxa |
I'm sure some people have a backup of the skins database they can share |
| 16:06 |
MTDiscord |
<luatic> I have one, but I don't know if I can share it :P |
| 16:06 |
ROllerozxa |
but it was also filled with copyright infringement... |
| 16:18 |
MTDiscord |
<bla8722> Malo95: you can also try internet archive´s wayback machine |
| 16:20 |
|
mazes_80 joined #minetest |
| 16:25 |
|
sparky4 joined #minetest |
| 16:33 |
MTDiscord |
<wsor4035> I also could share a copy, not sure how legal it is though |
| 16:34 |
|
Fleckenstein joined #minetest |
| 16:52 |
|
Fleckenstein joined #minetest |
| 16:59 |
Sokomine |
Krock: my problem is that i really dislike 2fa. i've forgotten my passphrase for the 2nd or 3rd time. i can still log into github but would have to change the key again. for now, i've put mg_villages on a gitlab site hosted by the server admin of the server your land |
| 16:59 |
Sokomine |
that's a gitea instance and works password-only. but i have no idea how to sync the repositories |
| 17:02 |
MTDiscord |
<wsor4035> if you set up remotes in your local repo, you can do git push gitea && git push github for example |
| 17:03 |
MTDiscord |
<wsor4035> or if that gitea instance has woodpecker ci or whatever the ci for gitea is called, can make an action on commit that pushes the updates to github |
| 17:03 |
Sokomine |
yes. but my problem is that i can't push from my local repo to github anymore as that requires a key. the password for the key is cached locally. thus i type it only once in a while. and i forget it in the meantime |
| 17:04 |
Sokomine |
hm. if the site could push the updates to github that'd be very fine. doesn't have to be automaticly. i'd be perfectly fine with doing that manually |
| 17:04 |
Krock |
Sokomine: hmm .. since when is 2FA enforced? I did not and probably will not set this up |
| 17:05 |
Sokomine |
if i could tell github on its website that i want to sync the mg_villages repo with https://gitea.your-land.de/Sokomine/mg_villages that'd be great |
| 17:05 |
MTDiscord |
<wsor4035> https://github.blog/2023-03-09-raising-the-bar-for-software-security-github-2fa-begins-march-13/ |
| 17:05 |
Sokomine |
it was for me. i couldn't proceed without setting it up |
| 17:08 |
Krock |
hmm.. https://i.postimg.cc/wMVVFrPx/grafik.png |
| 17:09 |
|
fluxionary joined #minetest |
| 17:13 |
MTDiscord |
<bla8722> Sokomine: https://docs.gitea.com/next/usage/repo-mirror#setting-up-a-push-mirror-from-gitea-to-github |
| 17:14 |
Sokomine |
krock: hm. strange |
| 17:18 |
Sokomine |
bla: that does seem to be a good solution. looking into it right now |
| 17:35 |
|
fluxionary joined #minetest |
| 17:37 |
Sokomine |
ok...says it's syncinc now. hope it works! that'd be a great relief |
| 17:53 |
MTDiscord |
<bla8722> looks like it worked |
| 17:53 |
Sokomine |
HAH! bla solved it! my mg_villages github repositiory is up to date again :-)))) |
| 17:53 |
|
appguru joined #minetest |
| 17:53 |
Sokomine |
rubenwardy: mg_villages is available and working again :-))) |
| 17:54 |
Sokomine |
that's a huge relief. pushing from the command line with password is way easier than that *censored* 2fa github uses |
| 17:55 |
MTDiscord |
<bla8722> now go fix something to check if the automatic mirroring works too lol |
| 17:55 |
Sokomine |
but mg_villages is fixed so far :-) at least the manual update worked :-)) |
| 17:57 |
Krock |
that's great. thank you for taking care of it. |
| 17:58 |
rubenwardy |
awesome! published on cdb again |
| 18:00 |
Krock |
I wonder whether there's an option on github to indicate that it's a mirror |
| 18:00 |
MTDiscord |
<wsor4035> you can turn issues off at least |
| 18:00 |
Krock |
well that does not solve the issues though |
| 18:00 |
Sokomine |
i was really stuck there. remembering a password that i have to use once every couple of month? not much chance :-( |
| 18:01 |
rubenwardy |
is the gitlab repo public? |
| 18:01 |
Krock |
Sokomine: use KeepassXC to keep track of your passwords |
| 18:01 |
Sokomine |
2fa is...it has its points, but most of the time it decreases security considerably |
| 18:01 |
rubenwardy |
use a password manager, increases security by allowing your passwords to be randomly generated and all different |
| 18:01 |
Sokomine |
well, there's some caching of passphrases involved on the command line somehow. but that makes forgetting just easier |
| 18:01 |
rubenwardy |
2fa improves security for those that don't have a password manager |
| 18:01 |
Sokomine |
that's a good sum-up, yes |
| 18:04 |
Sokomine |
still ought to offer a modpack again. but perhaps with contentdb that is less needed |
| 18:05 |
rubenwardy |
CDB automatically installs deps, so there's no need to have a modpack with mg_villages and handle schematic |
| 18:07 |
Sokomine |
that's good. but there are a few optional dependencies that are good to have but not necessary - like cottages (gives way more village types) and some additional village types that can be installed |
| 18:12 |
MinetestBot |
[git] RisingLeaf -> minetest/minetest: Do not render objects that are invisble into the shadow map 6601515 https://github.com/minetest/minetest/commit/660151572fdf848ee8416eea480df42a3b317df4 (2023-08-26T18:12:17Z) |
| 18:13 |
erle |
i did a talk on build system failure modes at ccc camp 2023 https://media.ccc.de/v/camp2023-57415-fantastic_build_system_failure_modes_and_how_to_fix_them |
| 18:14 |
erle |
this is relevant to minetest because it is the project i know which has the most miscompiles i ever encountered |
| 18:14 |
erle |
muurkha you might be interested |
| 18:15 |
erle |
TL;DR: constructing a DAG for a build, then toposorting and then scheduling build steps is impossible for the general case, because you do not have enough information to construct the full DAG at the time you want to construct it. a build should rather be seen as a function that can be partially evaluated and needs to be iterated until it reaches a fixed point. think: recompile latex until the page numbers and references match up. |
| 18:17 |
erle |
this means that no amount of fixing and testing can make DAG-toposort systems (e.g. make) get the general case right. it is easy to avoid all of these problems by using a top-down recursive build system instead. |
| 18:17 |
erle |
i don't expect anyone to ditch cmake after the talk, but it can help to figure out when it does not make sense to spend work on fixing cmake problems once you recognize the problem is unfixable. |
| 18:18 |
muurkha |
hmm, because for example you don't know which header files something is going to #include until you try to compile it, and some of your header files are dynamically generated? |
| 18:18 |
erle |
yes, that is one case |
| 18:18 |
erle |
i believe ninja has special cased c and c++ header files for that |
| 18:18 |
erle |
but there is a simpler example |
| 18:18 |
erle |
consider a target that needs to be always out of date at the time it is needed |
| 18:19 |
erle |
e.g. a website footer that includes the date the page was built |
| 18:19 |
muurkha |
that sounds like a hack to get the build system to do non-build-system things |
| 18:19 |
erle |
the DAG thing schedules it once and every page has the same timestamp at the bottom |
| 18:19 |
erle |
this is trivially wrong |
| 18:19 |
muurkha |
do you have an example that wouldn't violate build reproducibility? |
| 18:20 |
erle |
nothing violates build reproducibility, as long as you rebuild until you reach the fixed point, which you need to do anyway all the time |
| 18:20 |
|
Talkless joined #minetest |
| 18:20 |
muurkha |
including the date the page was built violates build reproducibility! |
| 18:20 |
erle |
the simple case here you build using make and then use the output of gcc -MD -MF to make a new makefile and then rebuild using that is a case where you need 2 steps to get to the fixed point |
| 18:21 |
erle |
muurkha then have the subtarget include the name of the parent target somewhere or so |
| 18:21 |
erle |
i mean then obviously it needs to be rebuilt every single time |
| 18:22 |
erle |
in any case, the DAG is a nice abstraction for a problem that is adjacent to, but ultimately not really, the problem a build system usually tries to solve |
| 18:22 |
muurkha |
to me it sounds like you're saying "sometimes people want to do arbitrary computation that can access any data in the world as part of their compilation" |
| 18:22 |
erle |
that is true |
| 18:22 |
erle |
but what about the latex example thing |
| 18:23 |
muurkha |
where you have to rebuild until the reference page numbers stop changing? |
| 18:23 |
erle |
no external information, but you have to shake it until the page numbers match up with the references |
| 18:23 |
erle |
yes |
| 18:23 |
erle |
that is intuitively a fixed point |
| 18:23 |
muurkha |
it is |
| 18:23 |
erle |
and you have to rebuild *every* project until you reach a fixed point, it is only that most of the time you reach it in one or two steps, so the DAG thing kinda seems to work |
| 18:23 |
erle |
ultimately it does not though |
| 18:23 |
muurkha |
"work" is undefined here |
| 18:23 |
erle |
just watch the talk and then talk to me about it afterwards? otherwise i repeat myself a lot |
| 18:24 |
erle |
the last part about redo is only so that people know it is possible |
| 18:24 |
erle |
to solve the issues with 4 primitves |
| 18:24 |
muurkha |
if "work" means "compute what you want" then yes sometimes people want to compute things that aren't a conventional build system |
| 18:24 |
muurkha |
I don't usually watch talks |
| 18:24 |
erle |
i suggest to read the microsoft paper “build systems a la carte” |
| 18:24 |
erle |
for the purpose of my talk, excel is a build system |
| 18:25 |
erle |
it has inputs (cells), rules (formulas), outputs (cells) |
| 18:25 |
erle |
and incremental rebuilds |
| 18:25 |
erle |
also recursion etc. |
| 18:25 |
erle |
in any case, the most common answer to my assertion is “then just don't do that” but people are doing it |
| 18:26 |
erle |
you don't have enough info to construct a correct DAG in one pass unless you are being very careful about everything, which i have so far only seen in the embedded space (where you have maybe 12 dependencies overall. a modern C hello world has probably more.) |
| 18:27 |
erle |
muurkha, are you familiar with DJB redo design (not the apenwarr one, it has a DAG cheating device built-in i think) |
| 18:28 |
erle |
if yes, that solves all problems i mention in the talk, so you don't have to watch it |
| 18:31 |
erle |
muurkha thanks for the reproducibility thing btw. but i build my webpage using redo, so it's a real problem ;) |
| 18:31 |
erle |
muurkha do you have maybe something on build systems in your books? |
| 18:32 |
erle |
muurkha, this is the microsoft paper https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf |
| 18:34 |
|
Fleckenstein joined #minetest |
| 18:36 |
erle |
muurkha the “wanting to compute arbitrary stuff” thing is very very common, even in reproducible builds. the errors i address are not about reproducibility of a single build, but about getting the same result between an incremental and a full build. |
| 18:36 |
erle |
for that you need the repeated evaluation fixed point top-down thing to make it easy |
| 18:37 |
erle |
if you don't want incremental builds, you can always rebuild everything from zero every time |
| 18:37 |
erle |
which is trivially correct, unless your build system has no fixed point |
| 18:37 |
erle |
i have seen makefiles where the correct binary fell out only every second time |
| 18:37 |
erle |
because oscillations lol |
| 18:42 |
|
HelenasaurusRex joined #minetest |
| 19:02 |
|
est31 joined #minetest |
| 19:02 |
|
gxt joined #minetest |
| 19:04 |
muurkha |
I'm vaguely familiar with DJB's redo, thanks to apenwarr and to you |
| 19:05 |
muurkha |
I keep meaning to read the SPJ paper |
| 19:07 |
muurkha |
I agree that Excel is a build system; I've written a bit about them (though arguably nothing written on the topic by someone who hasn't read the SPJ paper merits being read) |
| 19:08 |
muurkha |
probably the most relevant piece I've written is https://dercuano.github.io/notes/dependency-bibliography.html |
| 19:08 |
muurkha |
but there's also https://dercuano.github.io/notes/blob-computation.html and https://dercuano.github.io/notes/blob-computation-notes.html |
| 19:11 |
muurkha |
you'll probably also be interested in https://dercuano.github.io/topics/self-adjusting-computation.html |
| 19:14 |
|
mrkubax10 joined #minetest |
| 19:18 |
|
mrkubax10 joined #minetest |
| 19:23 |
erle |
muurkha the apenwarr implementation of redo caches dependency checks in ways that make it easier to implement, but ultimately result in nasty surprises. i wrote my implementation only because apenwarr would not fix these things (because it would make the implementation arguably correct, but slower). |
| 19:31 |
erle |
muurkha, ccache is subtly wrong |
| 19:31 |
erle |
basically it uses the wrong cache key |
| 19:31 |
erle |
i forgot what the correct one was |
| 19:32 |
erle |
but you sometimes get false positives in the cache |
| 19:32 |
erle |
which means a wrong build or no artifact at all |
| 19:44 |
|
liceDibrarian joined #minetest |
| 19:47 |
|
HelenasaurusRex joined #minetest |
| 19:52 |
|
diceLibrarian joined #minetest |
| 19:58 |
muurkha |
erle: not surprising |
| 19:58 |
muurkha |
this reminds me of the C vs. Lisp debates of 35 years ago |
| 20:12 |
|
jaca122 joined #minetest |
| 20:17 |
erle |
muurkha in what ways? |
| 20:17 |
erle |
i am 35, so |
| 20:17 |
erle |
no memories |
| 20:21 |
|
liceDibrarian joined #minetest |
| 20:32 |
erle |
muurkha so i came up with this tool yesterday, redo-gcc |
| 20:32 |
erle |
basically it invokes gcc and straces the compiler |
| 20:32 |
erle |
and records every file gcc did not find as a non-existence dependency and every file it found as a dependency |
| 20:32 |
erle |
the latter according to -MD -MF |
| 20:33 |
erle |
good idea? |
| 20:37 |
|
appguru joined #minetest |
| 21:05 |
|
proller joined #minetest |
| 21:05 |
|
TomTom joined #minetest |
| 21:10 |
|
proller joined #minetest |
| 21:15 |
muurkha |
erle: yeah, there are a lot of notes on that kind of thing in Dercuano |
| 21:16 |
muurkha |
I think it's a good idea |
| 21:23 |
erle |
muurkha so how do you deal with non-existence dependencies normally? |
| 21:23 |
erle |
e.g. the target being invalidated if a new file or new build rule is created |
| 22:20 |
MTDiscord |
<vejou> does this message show on matrix |
| 22:24 |
MTDiscord |
<mistere_123> No |
| 22:24 |
MTDiscord |
<mistere_123> The bridge is currently down. We might have a fix in a few hours. Or not, who knows |
| 22:28 |
MTDiscord |
<vejou> oh i thought the whole thing was supposed to end |
| 22:28 |
MTDiscord |
<vejou> did that change |
| 22:29 |
MTDiscord |
<mistere_123> Not sure what you mean... |
| 22:29 |
MTDiscord |
<vejou> i thought whatever we were using for the bridge was gonna stop servicing |
| 22:31 |
MTDiscord |
<vejou> https://discord.com/channels/369122544273588224/747163566800633906/1144928932383162379 |
| 22:32 |
|
panwolfram joined #minetest |
| 22:33 |
|
[MTMatrix] joined #minetest |
| 22:35 |
muurkha |
erle: I think nonexistence dependencies are a really interesting question |
| 22:36 |
[MTMatrix] |
MisterE | Surprise: the matrix bridge worketh, thanks luatic. |
| 22:36 |
muurkha |
thanks luatic! |
| 22:38 |
muurkha |
one crude way to deal with them is to treat them as read dependencies on the directory containing the nonexistent file. this is "sound", in the sense that it will correctly rebuilding things every time rebuilding is necessary, but not "complete" in the sense of *only* rebuilding things for which rebuilding is necessary |
| 22:38 |
muurkha |
completeness is generally considered an infeasible goal because it requires distinguishing changes to source files that affect the compiler's output from changes that don't, which in general requires solving the halting problem for the compiler |
| 22:41 |
|
[MTMatrix] joined #minetest |
| 22:41 |
muurkha |
but recompiling the entire world every time you add a new file to /usr/include might be too far from completeness to be useful |
| 22:42 |
MTDiscord |
<mistere_123> sorry, one more test |
| 22:42 |
|
fluxionary joined #minetest |
| 22:42 |
muurkha |
in things like https://www.mail-archive.com/kragen-tol@canonical.org/msg00146.html "Make for URLs, or dependency-directed compositing" nonexistence dependencies are straightforward |
| 22:42 |
erle |
muurkha my redo implementation does ne deps correctiy i think |
| 22:42 |
erle |
you can't really model them properly as normal deps |
| 22:43 |
erle |
also just to be clear, make is unfixable |
| 22:44 |
muurkha |
yeah, I wasn't suggesting literally using make |
| 22:44 |
erle |
every DAG-toposort system i have seen can not really accomodate NE deps |
| 22:44 |
muurkha |
but in that setup, GETting a resource that is currently nonexistent produces a representation with a 404 error code, and as long as that representation is a valid representation for that resource, you don't trigger a rebuild |
| 22:44 |
muurkha |
it doesn't require any special handling |
| 22:45 |
erle |
looks like an adjacent, but different problem |
| 22:45 |
erle |
i solved that with redo-stamp |
| 22:45 |
erle |
which can declare a target semantically up to date while it is being built |
| 22:45 |
muurkha |
how? |
| 22:45 |
erle |
this is a bit weird |
| 22:45 |
erle |
but exit code 123 means “even though this target was being scheduled for rebuild, during the rebuild it turned out it was actually fine as it is” |
| 22:46 |
erle |
so the build system then works with the updated info that the target was up to date all along |
| 22:46 |
erle |
p sure you can not do that with the toposort |
| 22:46 |
muurkha |
btw, I think it would serve you well rhetorically to stop claiming that system X is "incorrect" because, although it solves problem X' correctly as the author intended, it doesn't solve problem Y' that you think is more important |
| 22:46 |
erle |
a simple use case: you have something which is byte-wise different but semantically identical |
| 22:47 |
erle |
library got recompiled with or without new symbols stuff or so |
| 22:47 |
muurkha |
aha, now I understand. yes, that seems like a good approach |
| 22:47 |
muurkha |
although you can do it in a purer DAG system by adding an extra build step which extracts out the semantics into a new bytewise identical file |
| 22:48 |
erle |
muurkha in terms of build systems and redo in particular i use “incorrect” usually for optimization attempts that result in build differentials from the most naive implementation. |
| 22:49 |
muurkha |
that approach, although it's more expensive and hard to work into existing systems, does have the advantage that it prevents a build step that incorrectly† returns 123 from incorrectly† preventing later things from being built |
| 22:49 |
erle |
wdym |
| 22:49 |
erle |
i do not use “incorrect” for stuff that merely takes longer btw |
| 22:49 |
muurkha |
well, suppose you have a library which may be built with or without debugging information |
| 22:50 |
erle |
yes |
| 22:50 |
muurkha |
maybe that debugging information doesn't end up in your final executable and so it doesn't matter |
| 22:50 |
muurkha |
as an alternative to having the build step that builds the library return 123 to say that all that has changed is the inconsequential presence of this debugging information |
| 22:51 |
muurkha |
you can add another build step that makes a stripped version of the library |
| 22:51 |
muurkha |
and then link with the stripped version of the library |
| 22:51 |
muurkha |
which will be bytewise identical if all that changed was the no-longer-present debugging information |
| 22:52 |
muurkha |
a blob-hash-based rebuild system will detect this and not bother to relink the executable |
| 22:52 |
erle |
an example of my usage of “incorrect”: take the apenwarr redo-always. it does, contrary to its name, not declare a target as always out of date in that particular implementation of redo, because of some caching. every implementation of redo that is simpler and every description of the command i have seen though claims that the target is always out of date. the difference is a caching layer that solves problem X' which is |
| 22:52 |
erle |
not problem X, which is solved according to the documentation and also solved by every simpler implementation. |
| 22:53 |
erle |
apenwarr probably thought that it would not make a difference at first, but the unit tests for apenwarr redo only work because something reaches inside the state database of the build system at runtime and forces it to *actually* rebuild always. |
| 22:53 |
|
LizzyFleck joined #minetest |
| 22:53 |
erle |
so, do you have a better term for “programmer added more complexity and is now solving a different problem than the original one” |
| 22:53 |
muurkha |
but I don't think apenwarr had the objective of behaving indistinguishably from a simple build script in shell; if he did he would have accepted your bug report |
| 22:54 |
|
LizzyFleck joined #minetest |
| 22:54 |
muurkha |
I think he wanted to write a build system that he liked better than make |
| 22:55 |
erle |
yeah, but he ended up trying to write a top-down recursive build system in terms of a DAG-toposort-bottom-up one which is generally not possible |
| 22:55 |
erle |
and the reason he does not accept my report is basically “why would anyone do this” |
| 22:56 |
erle |
i mean i know why apenwarr does this, parallelism is way easier to do with X' than with X |
| 22:56 |
erle |
and you can cheat a lot on dependency checks |
| 22:56 |
erle |
but ultimately, it results in incremental builds with apenwarr redo that do not match a full rebuild |
| 22:56 |
erle |
don't quote me on that, i'm sleepy |
| 22:57 |
erle |
but i consider a tool not doing what it's own documentation suggests as incorrect |
| 22:57 |
erle |
oh and it's also a composition issue |
| 22:57 |
erle |
you basically can not “nest” invocations of redo (which is why the unit testing does not work) |
| 22:57 |
|
Fleckenstein joined #minetest |
| 22:58 |
erle |
in any case, it does no longer solve the original problem, but a related one that is easier to solve |
| 22:58 |
erle |
lots of software does this with good reasons |
| 22:58 |
erle |
take fast inverse square root |
| 22:58 |
erle |
it's incorrect (and illegal) |
| 22:58 |
erle |
but it did exactly that |
| 22:58 |
erle |
so how would you call the scenario “programmer choses to solve X' instead of X” in general? |
| 22:58 |
erle |
i mean, it is not always wrong to do so |
| 23:00 |
erle |
i suggest to look at the implementation yourself to get the picture btw |
| 23:01 |
muurkha |
erle: I think "why would anyone do this" can be translated as "I'm not trying to solve the problem you correctly identify that this software does not solve" |
| 23:01 |
erle |
or strace it and find out it does way fewer dependency checks than it logically would have to do |
| 23:02 |
muurkha |
I think the general term for that scenario is "Worse is Better" |
| 23:02 |
erle |
yeah, but in this place, better is better |
| 23:02 |
muurkha |
you could be right about that, but it doesn't go without saying |
| 23:03 |
muurkha |
I mean it isn't always true so it requires a convincing demonstration in any particular case |
| 23:04 |
erle |
my implementation is a fraction of the code and does not have a dependency on python or sqlite and IIRC it is faster for everything but parallel builds where jobs take less than 1 second. |
| 23:04 |
erle |
something like that |
| 23:04 |
muurkha |
nice |
| 23:04 |
erle |
the funny thing is that if you benchmark it you will notice that apenwarr redo can still be faster |
| 23:05 |
erle |
but the reason is because it cheats on dependency checks i think |
| 23:05 |
erle |
i can do the same: check no dependency, output an empty file |
| 23:05 |
erle |
suddenly i am the fastest |
| 23:06 |
erle |
in any case, i think i am a “better correct than fast” person and apenwarr is a “better fast than correct” person and i just lucked into a constructive proof that nope you don't need sqlite or so many lines of python. |
| 23:06 |
erle |
i am not so sure about the parallelism though |
| 23:06 |
erle |
if you know about the jobserver thing (with the fifo tokens), maybe you could look into it |
| 23:06 |
erle |
currently i am busywaiting with sleep 1 |
| 23:06 |
erle |
which is certainly not optimal |
| 23:07 |
erle |
also apenwarr redo has this output interleaving for parallel builds which i find nice |
| 23:08 |
muurkha |
not familiar with the jobserver thing |
| 23:08 |
muurkha |
have you looked at how GNU tail works on Linux, btw? |
| 23:08 |
erle |
oh, this is a trick you'll like |
| 23:08 |
muurkha |
tail -f I mean |
| 23:08 |
erle |
so you want to schedule N jobs in parallel |
| 23:09 |
erle |
you create a fifo, write N tokens into it (bytes? i forgot) |
| 23:09 |
erle |
each job starting tries to read a token from the fifo |
| 23:09 |
erle |
each job done writes one there |
| 23:09 |
erle |
so you start all the jobs |
| 23:09 |
muurkha |
that's a good idea. so a pipe is a semaphore |
| 23:09 |
muurkha |
(with a maximum capacity of PIPE_BUF, but that's good enough for almost all cases) |
| 23:09 |
erle |
i heard a girl i explained this to say the same thing a few days ago |
| 23:10 |
erle |
so for recursive builds this is a bit weird, because you don't want to block on subtargets |
| 23:10 |
erle |
so sometimes you need additional arithmetic on when to add or remove tokens |
| 23:10 |
erle |
muurkha, i wrote this some time back, it might do what i described http://news.dieweltistgarnichtso.net/bin/jobserver |
| 23:12 |
erle |
but look into my redo-ifchange to see it in action |
| 23:13 |
erle |
i wonder why i did the argument thing that way |
| 23:13 |
erle |
i probably had a reason |
| 23:13 |
erle |
no idea |
| 23:16 |
erle |
muurkha i got the fifo approach from simon richter (?) years ago when we walked through berlin at night. i believe that he mentioned some software uses it. maybe the make jobserver? no idea |
| 23:16 |
muurkha |
for a recursive build, when you block a task because it's missing a prerequisite, maybe you could put a token back into the pipe |
| 23:17 |
muurkha |
for as long as you're blocked |
| 23:17 |
erle |
maybe i am doing that, look at redo-ifchange |
| 23:17 |
muurkha |
that way only the actively compiling tasks are counted |
| 23:18 |
muurkha |
not familiar with this unlink shell command |
| 23:18 |
erle |
it calls unlink(2) |
| 23:18 |
erle |
to unlink, and possibly delete, a file |
| 23:18 |
muurkha |
well, presumably, but isn't that what rm does? |
| 23:19 |
muurkha |
we should probably have a place to discuss this where it would be on topic |
| 23:19 |
muurkha |
maybe #scannedinavian |
| 23:19 |
erle |
it is much safer than rm and also possibly more efficient, because it does not do anything else than accept a single file that can not be named --help or --version |
| 23:20 |
erle |
rm -rf argument accidents go wild |
| 23:20 |
erle |
unlink at most deletes one file |
| 23:20 |
muurkha |
though maybe it's healthy for sfan5 to see you having a conversation where he isn't flaming you ;) |
| 23:20 |
muurkha |
apparently unlink is, much to my surprise, in coreutils |
| 23:21 |
erle |
i always use it instead of rm -rf if i can do that |
| 23:21 |
erle |
i wonder if minetest is still using a vulnerable libpng hehe |
| 23:21 |
erle |
i did mention that before some release |
| 23:22 |
muurkha |
pretty sure the copy of minetest I'm running on here is linked with the Debian system libpng |
| 23:22 |
erle |
ah |
| 23:22 |
erle |
only windows users at risk then hehe |
| 23:22 |
muurkha |
libpng16.so.16 => /lib/x86_64-linux-gnu/libpng16.so.16 (0x00007fc84b1fa000) |
| 23:22 |
muurkha |
yeah, fuck them anyway |
| 23:23 |
erle |
muurkha btw, if i did not mention it, uncompressed TGA fed through zlib is one of the best image formats for textures of the type and size of minetest. i found that when writing my text rendering library months ago. |
| 23:23 |
muurkha |
that's very surprising, the Paeth compressor in PNG doesn't buy you aanything? |
| 23:23 |
erle |
it was indeed surprising |
| 23:23 |
erle |
and it also holds true for bigger image sizes, up to a limit |
| 23:24 |
erle |
i believe there are two reasons for it |
| 23:24 |
muurkha |
I mean zlib is written by two of the authors of libpng, which uses it for compression |
| 23:24 |
erle |
first, TGA has *way* lower overhead, no checksums etc. we already know that even an uncompressed TGA can beat a small PNG on filesize, but the sizes have to be like 8×8 or so. |
| 23:25 |
erle |
second, the compression in PNG is done by chunk if i am not mistaken. |
| 23:25 |
erle |
which is obviously suboptimal |
| 23:25 |
erle |
oh and third |
| 23:25 |
erle |
the whole framing structure of PNG is hard to compress because of the checksums |
| 23:26 |
erle |
i think |
| 23:26 |
muurkha |
hmm, apparently I was wrong about that; Gailly and Adler aren't credited in libpng |
| 23:26 |
erle |
basically, whatever was done at the quake era (put TGA in PK3 archives, i.e. basically zip files) is the best possible solution for small textures (for surprisingly large values of small) |
| 23:27 |
erle |
i only ever found that out when i rendered a lot of text to a png and tried pngcrush and zlibbing a tga |
| 23:27 |
erle |
sorry |
| 23:27 |
erle |
i mean when i rendered a lot of text to a TGA, then converted it to PNG and then both compressed the TGA and used pngcrush on the PNG |
| 23:28 |
erle |
it's also a write-speed issue |
| 23:28 |
muurkha |
hmm, have you tried feeding uncompressed 8×8 TGAs through something like ncompress? |
| 23:28 |
erle |
whatever ncompress is, it is not in the lua api of minetest |
| 23:28 |
muurkha |
LZW |
| 23:28 |
erle |
also 8×8 is not the size i am going for |
| 23:28 |
erle |
give me a moment |
| 23:30 |
muurkha |
zlib.compress(b'') in Python 3 gives an 8-byte output |
| 23:31 |
muurkha |
compress </dev/null outputs 3 bytes |
| 23:32 |
muurkha |
echo xyz | compress outputs 8 bytes, but echo xxx | compress outputs 7 |
| 23:32 |
muurkha |
there's also shoco but it's mostly intended for ASCII (for which it guarantees zero expansion) |
| 23:34 |
erle |
i have whole writeup on the issue already, need to publish it |
| 23:34 |
erle |
as it is highly unintuitive for most (including me) |
| 23:35 |
erle |
so lets take flowers_waterlily_bottom.png from minetest_game, the thing i have here is 327 bytes and does not get smaller when i use pngcrush -brute |
| 23:36 |
muurkha |
that's 40 times larger than the zlib overhead |
| 23:37 |
erle |
so i chose that picture and saved it using mtpaint as a TGA image, probably not optimal, but bear with me here |
| 23:38 |
erle |
the TGA is 390 bytes, but when using zlib-flate -compress on it, i have 302 bytes |
| 23:38 |
erle |
and now the interesting thing is that crushing a PNG takes a lot of time |
| 23:38 |
erle |
but dumping a bitmap and compressing it not |
| 23:38 |
erle |
because the png crushers often try a lot of options |
| 23:39 |
erle |
the image is atypical though |
| 23:40 |
muurkha |
I wonder if you could beat it by Paeth-predicting the TGA and zlibbing the residuals |
| 23:40 |
muurkha |
TGA overhead seems to be 18 bytes |
| 23:40 |
erle |
png overhead is like 70 bytes |
| 23:40 |
erle |
most minetest textures are a bit more than 100 bytes |
| 23:40 |
erle |
go figure ^^ |
| 23:41 |
muurkha |
I mean, I wonder if Paeth-predicting the TGA would make the zlib output smaller or larger |
| 23:42 |
muurkha |
a different approach would be to put a bunch of textures into one sprite sheet |
| 23:42 |
erle |
let's try flowers_dandelion_white.png, 142 bytes. pngcrush makes it one byte bigger, so i use optipng, which does not optimize it any further. |
| 23:42 |
muurkha |
if they were all the same width, which is common for textures, you could store one height per texture |
| 23:44 |
erle |
flowers_dandelion_white.tga is 324 bytes, but becomes 98 (!) bytes after zlib-flate -compress |
| 23:45 |
erle |
this is the typical case |
| 23:45 |
erle |
this is actually a case of worse is better i guess |
| 23:45 |
erle |
because zlib + tga reader is much less complexity than having a png reader or optimizer |
| 23:45 |
erle |
you get faster reading and writing speed and smaller filesize |
| 23:46 |
erle |
so basically just by implementing a reader for tga.z in minetest you could reduce filesizes of most textures by about 30% |
| 23:47 |
erle |
and it would be massively faster than the optipng orgies that mods have to run nowadays |
| 23:47 |
erle |
as my concern is with signs, i of course want it for ingame texture generation |
| 23:48 |
erle |
also, if anyone had put any thought into minetest.encode_png() it would probably not exist (i keep saying that basically since it was introduced), because IIRC it does not use any predictor, so it is always worse than just zipping up the pixels |
| 23:49 |
erle |
the spritesheet thing is obviously a good idea, yes – but you lose some composability there |
| 23:50 |
erle |
btw, if you go implement tga.z, also implement obj.z or so |
| 23:50 |
erle |
i think obj was the 3d text format, right? |
| 23:51 |
erle |
muurkha, did you ever implement bidi text? |
| 23:52 |
rubenwardy |
I feel like this would be better as a PM |
| 23:53 |
erle |
the build system thing, probably. the TGA thing, not at all. this is good stuff. |
| 23:53 |
erle |
and relevant to minetest, btw |
| 23:54 |
erle |
for example, if you put textures in item meta (which i think you do), they can be 30% smalller with that trick |
| 23:54 |
erle |
smaller meta = less lag |
| 23:54 |
erle |
especially if you put *generated* textures in item meta, since you prob won't call optipng in your lua mods |
| 23:55 |
erle |
anyways, if “you can have the same texture quality, but save 30% on transfer size” is not relevant, i don't know why ppl optimize their pngs at all ^^ |
| 23:56 |
erle |
thank you for coming to my TED talk :P |
| 23:56 |
muurkha |
erle: no, I haven't done bidi. or even hyphenation and justification. we should probably take this somewhere where it's not off-topic |
| 23:56 |
erle |
nah, i don't want to talk about it |
| 23:57 |
erle |
it's just my minetest pure-lua unicode text rendering library does bidi text (badly, because incomplete) |
| 23:57 |
erle |
and i wondered if i can hand it to someone who can do that correctly |
| 23:57 |
erle |
;) |
| 23:57 |
erle |
i'll probably just release it in the broken state, as i haven't worked on it in a while |
| 23:57 |
erle |
it can do every character in unifont and also combining characters, that should be enough to get better signs |
| 23:58 |
muurkha |
that sounds awesome! |
| 23:58 |
muurkha |
I'd love to have combining characters in my signs |
| 23:58 |
erle |
the only issue is you need to provide your own unifont.hex and unifont_upper.hex |
| 23:58 |
muurkha |
although last I checked I can't actually type accented letters in Minetest, even precomposed |
| 23:59 |
erle |
that's an issue for someone else. i can render them in both forms. |
| 23:59 |
erle |
also i'm pretty sure there is some possiblity to optimize tga_encoder a tiny bit, but i have not done that either |
