| Time |
Nick |
Message |
| 00:00 |
pgimeno |
well, that's not really a representative example, we're not talking canonical base64, we're talking image formats |
| 00:00 |
pgimeno |
the Python decoder is not strict, but it is robust in that it doesn't crash |
| 00:00 |
erlehmann |
pgimeno i give you the base64 example because it neatly shows a) standard libs are a joke b) you don't *need* a robust decoder (whatever that would be) |
| 00:01 |
erlehmann |
pgimeno, it will also happily consume garbage and tell you that it decoded it as base64 |
| 00:01 |
erlehmann |
all while it definitely has error codes for wrong padding or non-base64-input |
| 00:01 |
erlehmann |
this one is just something that slipped through |
| 00:01 |
pgimeno |
and what would the consequences be? that you'd have a binary string, not that the decoder crashed |
| 00:02 |
erlehmann |
well … lets say you want to blacklist input that decodes to specific binary |
| 00:02 |
erlehmann |
suddenly there will be a way to get around that |
| 00:03 |
pgimeno |
you need to block it post-decoding obviously |
| 00:03 |
erlehmann |
LOL no |
| 00:03 |
pgimeno |
why not? |
| 00:03 |
erlehmann |
at that point your decoder has been exploited already |
| 00:03 |
pgimeno |
if it's not robust, sure |
| 00:03 |
erlehmann |
i hope it is obvious to you that it is *way* easier to put the regex before the decoder than to make sure the decoder is correct for all kinds of crap. |
| 00:03 |
erlehmann |
is it? |
| 00:04 |
pgimeno |
what is? the jpeg library? I imagine it is |
| 00:04 |
erlehmann |
i have actually encountered a system at work years ago that validated after decoding |
| 00:04 |
erlehmann |
it used ffmpeg |
| 00:05 |
erlehmann |
little known feature of ffmpeg, you can feed it synthesizer instructions |
| 00:05 |
pgimeno |
if you're going to mention an example of a decoder that craps on bad input, you're using a bad example |
| 00:05 |
erlehmann |
so i made the sample rate VERY high and encoded a few seconds of silence |
| 00:05 |
pgimeno |
again, bad example |
| 00:05 |
erlehmann |
the disk on the testing system was full before it could ever figure out what happened |
| 00:05 |
erlehmann |
because i made the *decoder* write a huge wave file |
| 00:06 |
erlehmann |
with a few bytes of input |
| 00:06 |
erlehmann |
pgimeno come on, almost all parsers are low-quality trash. |
| 00:06 |
pgimeno |
erlehmann: I don't care, it is still an example of a decoder that is not robust enough |
| 00:06 |
pgimeno |
libjpeg is well tested and scrutinized for vulnerabilities |
| 00:07 |
erlehmann |
have you read the LANGSEC papers? they argue pretty well that the best way to get a robust decoder is … having the first step reject everything you do not want to process because it is malformed. |
| 00:07 |
erlehmann |
which is what i propose |
| 00:07 |
erlehmann |
bc that means you need no error handling in the rest of the code |
| 00:08 |
erlehmann |
basically much less effort |
| 00:08 |
erlehmann |
after all only valid stuff will be decoded |
| 00:08 |
pgimeno |
in this project you can't validate everything, adding a JPEG validator is out of scope of the project |
| 00:08 |
pgimeno |
so is adding a PNG validator |
| 00:08 |
pgimeno |
you need to rely on the robustness of the decoders |
| 00:08 |
erlehmann |
why |
| 00:09 |
erlehmann |
look no one can get this right |
| 00:09 |
pgimeno |
because JPEG is very complex, and PNG is very complex, and validating them is about as much work as decoding them if not more |
| 00:09 |
erlehmann |
unless they follow very specific rules on how to consume input, that is |
| 00:09 |
pgimeno |
so you have to write a decoding library just to validate them |
| 00:10 |
pgimeno |
Minetest is not an image library, it's a game engine |
| 00:10 |
erlehmann |
as i said, you only need to *accept* the features that you can actually process already |
| 00:10 |
pgimeno |
and which features are those for JPEG files? |
| 00:10 |
pgimeno |
or better said, which features would you leave out in case of a JPEG? |
| 00:11 |
erlehmann |
for a game engine that dls most stuff in the beginning … probably progressive mode. don't tell hecks! |
| 00:11 |
pgimeno |
if you want to validate it, you need to decode the Huffman or arithmetic coding, check the DCT parameters and whatnot |
| 00:11 |
erlehmann |
exif decoding too |
| 00:12 |
pgimeno |
so you're going to reject files with exif? |
| 00:12 |
erlehmann |
if minetest would already reject such files, i'd very much make sure they never get to the decoder |
| 00:13 |
pgimeno |
files with exif info??? are you serious? |
| 00:13 |
pgimeno |
you're the same guy who shouted at the developers for removing support for TGA because of the compatibility break, right? |
| 00:14 |
erlehmann |
you obviously read my statement but did not comprehend |
| 00:14 |
erlehmann |
*if minetest would already reject such files*, i'd very much make sure they never get to the decoder |
| 00:14 |
erlehmann |
i have no idea if it does or not |
| 00:14 |
pgimeno |
files with exif info should be accepted, and the exif info ignored |
| 00:14 |
erlehmann |
but what i want to say is that the validator code must protect the decoder from anything the decoder would have to have error handling for |
| 00:15 |
pgimeno |
and for that, in the case of JPEG, it needs to be a JPEG decoder |
| 00:15 |
erlehmann |
if the decoder ignores exif, then that's totally ok too |
| 00:15 |
pgimeno |
and that's out of scope of this project |
| 00:15 |
erlehmann |
pgimeno why would it be out of scope for minetest to not crash? |
| 00:15 |
pgimeno |
because libjpeg takes care of that already |
| 00:16 |
pgimeno |
with your image, gimp does not crash, feh does not crash, both use libjpeg |
| 00:16 |
erlehmann |
uh, no? |
| 00:16 |
erlehmann |
yeah that is bc gimp and feh are not stupid |
| 00:16 |
pgimeno |
feh refused (not crashed) your first image, and swallows the second without problems |
| 00:16 |
erlehmann |
but they were stupid in the past |
| 00:17 |
erlehmann |
i have generated pictures myself that crashed feh |
| 00:17 |
erlehmann |
using afl-fuzz |
| 00:17 |
pgimeno |
yeah, and they both use the same library that minetest is using, and DON'T VALIDATE THE JPEG |
| 00:17 |
erlehmann |
that is because the error in minetest can be mitigated in other ways |
| 00:18 |
erlehmann |
look, i have generated malformed jpegs at my job and written a report about it |
| 00:18 |
pgimeno |
ok, I guess you're agreeing with me that libjpeg is robust enough but there's still a problem with not replacing images with errors with a placeholder |
| 00:18 |
erlehmann |
it is extremely easy to crash apps, even if they use a good library |
| 00:19 |
erlehmann |
because apps do stupid stuff like “allocate height×width×depths bytes” |
| 00:19 |
erlehmann |
after the library has parsed the stuff |
| 00:19 |
erlehmann |
the validator would of course prevent that by setting sane limits – before any of the business logic is called |
| 00:21 |
pgimeno |
so you don't need a full fledged validator, just to check that the resolution is reasonable and little else |
| 00:21 |
erlehmann |
pgimeno consider the case of a not malformed jpeg that is legit 32000×32000 pixels big. what should happen and where? |
| 00:22 |
pgimeno |
attempt to load - no error: accept, error: reject and replace |
| 00:22 |
erlehmann |
i say it should be rejected by a validator before any pixels are decoded |
| 00:22 |
erlehmann |
because otherwise you have a buffer full of trash you need to get rid of |
| 00:22 |
erlehmann |
do you agree? |
| 00:22 |
pgimeno |
no |
| 00:22 |
pgimeno |
gimp opens it, feh opens it |
| 00:23 |
pgimeno |
why should minetest refuse to open it? |
| 00:23 |
erlehmann |
in minetest i mean |
| 00:23 |
MTDiscord |
<Jordach> can you stop imagining imaginary problems |
| 00:23 |
erlehmann |
pgimeno, because 32000×32000×3 would be about 3 gigabytes of pixels |
| 00:24 |
pgimeno |
and? that fits in my memory |
| 00:24 |
erlehmann |
not here lol |
| 00:24 |
erlehmann |
and if it fits in yours, i just do it a 100 times |
| 00:24 |
erlehmann |
the code needs to handle it |
| 00:25 |
v-rob |
Then you get std::bad_alloc. Then handle it, if you want to. What's the problem? |
| 00:25 |
erlehmann |
well, do you see no problem with the fact that you maybe allocated a buffer and filled it with decoded pixels BEFORE rejecting the input? |
| 00:25 |
erlehmann |
it can be, depending on the input, extremely wasteful and slow |
| 00:25 |
v-rob |
Why should we assume the input is incorrect? |
| 00:26 |
v-rob |
It won't do us any harm to assume so and give the decoder allocated memory |
| 00:26 |
erlehmann |
i already said we should not, before the decoder needs to be a recognizer that only lets stuff we want through |
| 00:26 |
v-rob |
What don't we want? |
| 00:26 |
v-rob |
If it's invalid input, that's what the decoder's for |
| 00:27 |
erlehmann |
lol ok i give up |
| 00:28 |
erlehmann |
v-rob what do you want me to do, figure out ways to crash servers and clients until someone takes my propoal seriously? |
| 00:28 |
erlehmann |
i mean i can probably easily automate the creation of bad inputs with a fuzzer |
| 00:28 |
v-rob |
It WON'T CRASH THE SERVER if we use the decoder correctly. |
| 00:29 |
erlehmann |
if |
| 00:30 |
erlehmann |
have you tried my pixelflood mod btw? |
| 00:30 |
v-rob |
So we don't need a validator like you've been arguing. We just need to fix any bugs in our code. |
| 00:30 |
erlehmann |
the state of the art to i fix those bugs is including validators actually |
| 00:30 |
erlehmann |
postels principle about robustness is false, this has been proven in the last 10 years i think |
| 00:32 |
erlehmann |
pgimeno v-rob have you ever used afl-fuzz? |
| 00:32 |
v-rob |
Look: If GIMP doesn't do it, Minetest 100% won't do it. And pgimeno has established that GIMP doesn't |
| 00:33 |
pgimeno |
erlehmann: please stop. We have a bug in our hands now. After #11624 is fixed, feel free to report more bugs using a fuzzer or whatever you want, but for now enough is enough |
| 00:33 |
ShadowBot |
https://github.com/minetest/minetest/issues/11624 -- Minetest does not start properly with memory sanitizer enabled (use of uninitialized-value) |
| 00:34 |
erlehmann |
should stop going on your nerves about the proper way to handle input or stop posting stack traces and stuff that crashes clients? |
| 00:34 |
erlehmann |
or both |
| 00:35 |
pgimeno |
the first |
| 00:35 |
erlehmann |
pgimeno, i am very much interested in how that will be solved because i hope i learn something about it and also because memory sanitizer will be a big help. |
| 00:35 |
erlehmann |
ok! |
| 00:37 |
erlehmann |
pgimeno v-rob btw is there a public list of known exploits that work against minetest? |
| 00:37 |
erlehmann |
i mean not in code execution but rather getting around anticheat and stuff |
| 00:37 |
v-rob |
There's plenty of GitHub issues about anticheat. It's not very good |
| 00:38 |
v-rob |
Security exploit reports are only visible to core devs. |
| 00:39 |
erlehmann |
well surely you can tell me if there are open security issues right now or how many. basically i want to know if the list is huge or tiny. |
| 00:39 |
erlehmann |
bc if it is huge, i obv don't need to add more to it |
| 00:40 |
erlehmann |
i would only risk posting dupes |
| 00:42 |
v-rob |
It's not very big to my knowledge, but that's probably because people don't go around testing Minetest for security holes. If you do find a real vulnerability (not like signed overflow), report it privately (https://github.com/minetest/minetest/security/policy), not with a public bug report. |
| 00:46 |
erlehmann |
so what you are saying is i should try to exploit a case of signed overflow |
| 00:46 |
erlehmann |
and then you take me seriously |
| 00:46 |
erlehmann |
considering it is in the mapgen … unlikely |
| 00:46 |
pgimeno |
right? |
| 00:47 |
erlehmann |
dw sooner or later i find something else |
| 00:48 |
v-rob |
If you do manage to exploit signed overflow, I would be impressed, but we would fix it, yes. |
| 00:49 |
erlehmann |
hmm, but surely i would have to hold on to at least one working exploit if i find it? |
| 00:49 |
erlehmann |
for leverage i mean |
| 00:49 |
erlehmann |
like if a server would require me to run SSCSMs then i could send executable code back |
| 00:50 |
erlehmann |
it would only be fair |
| 00:50 |
erlehmann |
i think that is a good plan, but i would of course reveal it after some time |
| 00:51 |
erlehmann |
v-rob are in-game actions considered exploits too? |
| 00:51 |
erlehmann |
like dupes, teleports, privilege escalation, lagging other players out of the game |
| 00:52 |
erlehmann |
i mean coordinate leaks aren't |
| 00:52 |
erlehmann |
(considered exploits by core devs) |
| 00:53 |
erlehmann |
so, would “anyone can destroy arbitrary blocks” (a hypothetical, the techniques i know are fixed) have to be posted in secret? |
| 00:53 |
v-rob |
Some really big ones might be (like particularly bad privilege escalation such that servers could be really damaged), but flying and teleports aren't. Cheat clients can already do those way too easily. |
| 00:54 |
erlehmann |
well i *can't* arbitrarily teleport by *just* overriding privs, so i guess you know something i don't |
| 00:55 |
erlehmann |
it exceeds the anticheat speed limit i guess |
| 00:56 |
erlehmann |
v-rob, btw are there cheat clients beyond dragofire and waspsaliva that matter? |
| 00:56 |
erlehmann |
i mean and the thing that specing is making |
| 00:56 |
erlehmann |
(or is that also ws?) |
| 00:56 |
v-rob |
Anticheat's pretty buggy, I believe, but I'm not really an expert. You'd have to ask someone else for more specific info on cheating |
| 00:56 |
erlehmann |
dragofire → dragonfire |
| 00:56 |
v-rob |
I don't know anything about cheat clients besides the fact that they exist |
| 00:56 |
erlehmann |
ah ok so it was a guess |
| 00:57 |
erlehmann |
well anticheat is pretty good in preventing movement faster than about 5n/s without physics override |
| 00:57 |
erlehmann |
i think |
| 00:57 |
erlehmann |
except downwards, bc limiting falling speed would be a bit silly |
| 00:57 |
erlehmann |
and upwards you can go i think 50n/s or so, bc trampolines |
| 00:59 |
erlehmann |
by physics override i mean if you get a potion of speed or so |
| 00:59 |
erlehmann |
then you can be faster |
| 00:59 |
erlehmann |
in fact, anticheat is so good at enforcing the speed limit that if you jump on some bouncy node from high enough, when you bounce back you will hit an invisibl ceiling |
| 01:00 |
specing |
erlehmann: my client still exists, but it is deprecated in favor of porting to ws |
| 01:00 |
erlehmann |
kinde ruined the fun when i encoundered a REALLY big trampoline on a server |
| 01:00 |
erlehmann |
specing, cool i hope it makes both you and cora happy! |
| 01:01 |
specing |
yeah, sadly I haven't been playing much MT lately (other than passive afk) |
| 01:02 |
erlehmann |
same :/ |
| 01:04 |
erlehmann |
specing, query |
| 01:18 |
pgimeno |
ok, so the problem is that output_message is called, but error_exit isn't, therefore the image is only partially loaded |
| 01:19 |
erlehmann |
and then no replacement i generated? |
| 01:19 |
erlehmann |
is |
| 01:21 |
pgimeno |
no because the image is believed to be valid, because error_exit is the one which should have marked it as invalid |
| 01:22 |
pgimeno |
the result is an incompletely decoded image |
| 01:22 |
erlehmann |
lul |
| 01:22 |
erlehmann |
cool result! |
| 01:22 |
erlehmann |
:) |
| 01:23 |
pgimeno |
this is fine with gimp and fte because they can read partially decoded images, but Irrlicht does not handle it so gracefully it seems |
| 01:27 |
erlehmann |
so the next level for me is crashing gimp or feh |
| 01:27 |
pgimeno |
yep |
| 01:27 |
erlehmann |
pgimeno, which versions of those tools do you have on which platform? |
| 01:29 |
pgimeno |
feh 2.18-2, gimp 2.8.18-1+deb9u1, libjpeg62-turbo 1:1.5.1-2 |
| 02:28 |
|
queria^clone joined #minetest-dev |
| 02:29 |
|
Menchers joined #minetest-dev |
| 02:34 |
|
queria^clone joined #minetest-dev |
| 02:52 |
|
tekakutli joined #minetest-dev |
| 02:52 |
|
Extex joined #minetest-dev |
| 03:10 |
|
v-rob joined #minetest-dev |
| 04:00 |
|
MTDiscord joined #minetest-dev |
| 04:23 |
|
tekakutli joined #minetest-dev |
| 05:41 |
|
v-rob joined #minetest-dev |
| 07:08 |
|
tekakutli joined #minetest-dev |
| 07:15 |
|
specing_ joined #minetest-dev |
| 07:15 |
|
tekakutli joined #minetest-dev |
| 08:06 |
|
tekakutli joined #minetest-dev |
| 09:21 |
|
tekakutli joined #minetest-dev |
| 09:51 |
rubenwardy |
merging #11598 in 10 |
| 09:51 |
ShadowBot |
https://github.com/minetest/minetest/issues/11598 -- Readd TGA to the list of valid texture formats. by rollerozxa |
| 09:58 |
|
cheapie joined #minetest-dev |
| 10:12 |
sfan5 |
rubenwardy: ping |
| 10:12 |
rubenwardy |
lol |
| 10:23 |
|
calcul0n_ joined #minetest-dev |
| 10:35 |
|
tech_exorcist joined #minetest-dev |
| 10:45 |
erlehmann |
rubenwardy can i have the archive of all of contentdb to iterate over it? |
| 10:45 |
erlehmann |
rubenwardy i found the stats that appgurueu made interesting and stuff like “how much is this API used” might be interesting |
| 10:45 |
erlehmann |
so i'd like to search it |
| 10:53 |
rubenwardy |
https://minetest.rubenwardy.com/all_cdb_releases_2021_09_09.tar |
| 10:53 |
rubenwardy |
https://minetest.rubenwardy.com/all_cdb_releases_2021_09_09.txt |
| 10:53 |
erlehmann |
thx |
| 11:04 |
|
hendursaga joined #minetest-dev |
| 11:09 |
|
tech_exorcist joined #minetest-dev |
| 11:11 |
|
tekakutli joined #minetest-dev |
| 11:32 |
|
Kimapr[i] joined #minetest-dev |
| 11:56 |
erlehmann |
pgimeno, https://godbolt.org/z/brEMqqYe8 ;) |
| 11:57 |
pgimeno |
so? |
| 12:02 |
erlehmann |
i find it funny |
| 12:05 |
pgimeno |
still off-topic |
| 12:09 |
erlehmann |
ok |
| 12:16 |
sfan5 |
we don't have a formspec label-like element that will confine the text to a defined box (by means of wordwrap, not scrolling), do we? |
| 12:18 |
pgimeno |
the cause of irr#67 turns out not to be the malformedness of the image, but the size. I get the same backtrace with a valid 32000x32000 image. |
| 12:18 |
ShadowBot |
https://github.com/minetest/irrlicht/issues/67 -- Pixelflood 2 – Segfault Boogaloo: Sending lottapixel.jpg from server segfaults client upon connection |
| 12:20 |
rubenwardy |
We don't sfan5, we have been using read-only with scrolling |
| 12:20 |
rubenwardy |
With my wordwrap PR, I wanted to make this possible |
| 12:21 |
rubenwardy |
Read-only text areas* |
| 12:21 |
sfan5 |
doesn't guiStaticText already have word wrapping though |
| 12:22 |
sfan5 |
https://0x0.st/-xdb.png something like this should be trivial to prevent by just setting bounds |
| 12:27 |
pgimeno |
same issue with a 32000x32000 valid PNG, same backtrace |
| 12:30 |
pgimeno |
the backtrace: http://www.formauri.es/personal/pgimeno/pastes/backtrace-image-crash.txt |
| 12:30 |
pgimeno |
would someone please post that info in the report? |
| 12:32 |
rubenwardy |
the word wrapping has bugs and will truncate |
| 12:32 |
rubenwardy |
if you only want to limit by width it's perfectly fine |
| 12:33 |
|
adfeno_ left #minetest-dev |
| 12:33 |
rubenwardy |
there's currently about 4 different word wrapping implementations in Minetest, I was intending to unify them and test it. But text is hard |
| 12:33 |
rubenwardy |
especially if you want support for RTL and Arabic, which I'm completely ignoring |
| 12:34 |
rubenwardy |
in Arabic, removing characters can make text longer - which is a massive pain when it comes to word wrapping |
| 12:34 |
rubenwardy |
although, it may only be within a word in which case it's possible implement without break-word |
| 12:35 |
rubenwardy |
anyway - adding a simple wrapping label makes sense |
| 12:35 |
rubenwardy |
as in, it makes sense to add it using current word wrapping as it's still useful |
| 12:36 |
rubenwardy |
(The two bugs I found where broken support for \r\n, and for soft hyphens) |
| 12:37 |
rubenwardy |
-h |
| 12:51 |
MTDiscord |
<josiah_wi> What's the right behavior if you're sending a packet and the data array to be copied from is empty (results in copying from null pointer). |
| 12:52 |
MTDiscord |
<josiah_wi> Or rather the data array doesn't exist. |
| 12:55 |
sfan5 |
this is too vague without linking the code you are looking at |
| 13:02 |
rubenwardy |
A lot of the time, data is prefixed with a size |
| 13:03 |
rubenwardy |
But yeah too vague |
| 13:05 |
MTDiscord |
<josiah_wi> Sorry, I got it backwards. The destination array is null hehe. |
| 13:07 |
MTDiscord |
<josiah_wi> networking.cpp line 66, the memcpy destination can be null. std::vector.data documentation says this may be the case if the vector is empty. |
| 13:08 |
MTDiscord |
<josiah_wi> I'm surprised this doesn't cause the networking to fail fatally lol. |
| 13:25 |
rubenwardy |
> If either dest or src is an invalid or null pointer, the behavior is undefined, even if count is zero. |
| 13:25 |
rubenwardy |
definitely wrong |
| 13:25 |
rubenwardy |
fix pls |
| 13:25 |
rubenwardy |
memory errors in networking code is asking for trouble |
| 13:28 |
celeron55 |
it wouldn't be a bad idea to write a fuzzer test that is capable of triggering these by chance |
| 13:28 |
celeron55 |
and run it in CI |
| 13:30 |
rubenwardy |
yeah |
| 13:41 |
sfan5 |
what is an "invalid" pointer? |
| 13:42 |
sfan5 |
I don't think a pointer to the end of a buffer is invalid, you may just not write to it |
| 13:42 |
sfan5 |
(thinking of a different case) |
| 13:47 |
erlehmann |
well you may not advance once past the end? |
| 13:49 |
erlehmann |
celeron55, have you looked at anon5 minetest go lib? |
| 13:49 |
pgimeno |
invalid is any pointer that doesn't point to allocated memory, for example a pointer to freed memory |
| 14:11 |
|
calcul0n__ joined #minetest-dev |
| 14:28 |
|
Fixer joined #minetest-dev |
| 14:39 |
|
v-rob joined #minetest-dev |
| 14:41 |
pgimeno |
well I was wrong, it's more complex than that: https://timsong-cpp.github.io/cppwp/n3337/basic.stc.dynamic.safety - there are even cases where a pointer may have a legal value and be invalid >.< |
| 14:45 |
sfan5 |
rubenwardy: cppreference says what you quoted but idk where they got that from |
| 14:45 |
sfan5 |
I checked the manpage and googled "posix memcpy" and neither talk about edge cases |
| 14:46 |
rubenwardy |
I copied from ccpreference |
| 14:46 |
rubenwardy |
lol, fuck the CCP |
| 14:48 |
sfan5 |
"Where an argument declared as size_t n specifies the length of the array for a |
| 14:48 |
sfan5 |
function, n can have the value zero on a call to that function. Unless explicitly stated |
| 14:48 |
sfan5 |
otherwise in the description of a particular function in this subclause, pointer arguments |
| 14:48 |
sfan5 |
on such a call shall still have valid values, as described in 7.1.4. " |
| 14:48 |
sfan5 |
uh oh I didn't expect it to paste like that; anyway that's from the C99 standard |
| 14:48 |
erlehmann |
C and C++ are two different languages that only superficially look similar though |
| 14:49 |
erlehmann |
for example, in C you can hang forever doing nothing |
| 14:50 |
erlehmann |
in C++, that is UB and will probably be optimized out |
| 14:50 |
erlehmann |
i.e. a loop with no side effects may be assumed to terminate |
| 15:38 |
erlehmann |
i have a question |
| 15:38 |
erlehmann |
you do not use git submodule |
| 15:39 |
erlehmann |
but you have a subdirectory with a git repo that absolutely *needs* to be in sync for most of the build |
| 15:39 |
erlehmann |
WHY |
| 15:39 |
erlehmann |
it is extremely frustrating to have a project like that |
| 15:39 |
erlehmann |
so how are you actually building older versions? |
| 15:39 |
erlehmann |
do you have some other tooling for it that keeps the dirs in sync?? |
| 15:41 |
sfan5 |
I'm pretty sure relevant changes so far were put inside ifdefs |
| 15:41 |
sfan5 |
so you could build todays MT with 1.9.0mt0 |
| 15:41 |
sfan5 |
only obstacle for doing it the other way is that irrmt removed some headers that older MT included (despite not needing them) |
| 15:42 |
erlehmann |
wdym ifdefs |
| 15:42 |
sfan5 |
grep the src folder for REVISION |
| 15:42 |
erlehmann |
uh |
| 15:43 |
erlehmann |
so how would i build an arbitrary commit, like what is the command for it? |
| 15:43 |
erlehmann |
i mean the CI needs to do it |
| 15:43 |
erlehmann |
so i bet there is |
| 15:43 |
sfan5 |
what?! |
| 15:44 |
erlehmann |
let me figure out something and get back to you |
| 15:48 |
pgimeno |
worth considering: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt |
| 15:50 |
erlehmann |
haha, oh wow, i think i figured it out. remember the thing about building way too much even though the dependencies are not needed, because cmake & make are stupid? turns out that the dependency tree seems to be underspecified in ome way, so there exist revision ranges where if you build on one commit and then rebuild on another, it will bug out. |
| 15:51 |
erlehmann |
usually the “stupid” behaviour of make results in rebuilding enough stuff so that this does not happen. |
| 15:52 |
sfan5 |
are you sure that's not your git workaround tricking make into misbehaving? |
| 15:52 |
erlehmann |
well, the git workaround trick actually exposes that flaw |
| 15:52 |
erlehmann |
think about it, if you rebuild almost everything, you mask the underspecified dep tree for most of the time |
| 15:52 |
sfan5 |
I'm sure the make authors are willing to take your bug report |
| 15:53 |
erlehmann |
well i think the culprit is cmake |
| 15:53 |
|
Extex joined #minetest-dev |
| 15:53 |
erlehmann |
by making make only rebuild what it strictly needs to according to what cmake claims … the build breaks |
| 15:57 |
erlehmann |
it happens in many projects that they underspecify dependencies. very common example: files always have to be recompiled when their build rules change, which means for make, they need to depend on the makefile that contains the build rule for them. try any random project, chances are high you can change build rules and the target will not be recompiled. |
| 15:58 |
erlehmann |
modern systems implicitly record such dependencies after the build. |
| 15:58 |
erlehmann |
(because you can build build rules, so you might not know before the build) |
| 16:13 |
erlehmann |
sfan5 wow sarcam thats creative |
| 16:13 |
|
v-rob joined #minetest-dev |
| 16:42 |
|
proller joined #minetest-dev |
| 17:14 |
|
v-rob joined #minetest-dev |
| 17:41 |
|
calcul0n joined #minetest-dev |
| 18:01 |
|
erlehmann joined #minetest-dev |
| 18:01 |
|
erlehmann joined #minetest-dev |
| 18:09 |
MTDiscord |
<luatic> Am I seeing an insertion sort every client frame for z-index handling? This ought to be illegal. |
| 18:10 |
|
Extex joined #minetest-dev |
| 18:10 |
MTDiscord |
<Noodlemire> Why every frame? |
| 18:14 |
sfan5 |
for hud rendering |
| 18:14 |
Krock |
and wield mesh, and for the inventory as well I guess |
| 18:16 |
MTDiscord |
<josiah_wi> Insertion sort is fast on small arrays, isn't it? That doesn't seem like a problem in itself. |
| 18:16 |
erlehmann |
can anyone tell me how to prevent mapgen griefing when loading a map from a world download? |
| 18:23 |
sfan5 |
set it to singlenode, also: #minetest |
| 18:24 |
MTDiscord |
<luatic> Insertion sort is O(n²) |
| 18:24 |
MTDiscord |
<luatic> A sorted set should be used, elements should be inserted by their z-index there as they are received. |
| 18:26 |
erlehmann |
sfan5 no i think there is a bug here |
| 18:26 |
erlehmann |
sfan5 already generated map should not be overwritten by mapgen? |
| 18:26 |
erlehmann |
so i am asking what to change about the *code* to make it not do that |
| 18:26 |
erlehmann |
i am willing to write that patch myself |
| 18:26 |
sfan5 |
it will, that's how mapgen works |
| 18:27 |
sfan5 |
assuming you are referring to small overlaps between blocks that are written to by the mapgen even though the block is already generated |
| 18:27 |
erlehmann |
no those are not small overlaps actually |
| 18:28 |
MTDiscord |
<luatic> You shouldn't use suboptimal data structures and algorithms when using the optimal ones comes at practically zero cost, because they are already premade in the standard libraries. |
| 18:28 |
erlehmann |
sfan5, almost any map import i have seen that does not use singlenode is heavily corrupted |
| 18:28 |
erlehmann |
and it does happen after the map is loaded |
| 18:28 |
MTDiscord |
<josiah_wi> Luatic, Python uses insertion sort on small list segments because it's faster than merge sort on average in those cases. |
| 18:29 |
erlehmann |
sfan5, so i have one world dl with a base that i would love to host, bc the server is gone, but if i visit the base, it is replaced by untouched terrain. |
| 18:29 |
erlehmann |
like, first it is there, and then the mapgen replaces it piece by piece. |
| 18:30 |
MTDiscord |
<luatic> The square root of 1k is about 32 |
| 18:30 |
MTDiscord |
<luatic> So if you want to do a 32² grid using HUD elements, you'll have insertion sort do 1 million comparisons. |
| 18:31 |
sfan5 |
erlehmann: if it works with singlenode the issue is not in general but will be with that specific mapgen |
| 18:31 |
erlehmann |
sfan5 do you think it could be possible to mark those areas in world downloads as already generated somehow? or is the info what was generated stored somewhere else? i am willing to create tooling to enable ppl to host stuff from world dls. |
| 18:31 |
erlehmann |
ah |
| 18:31 |
erlehmann |
well i guess singlenode is really dumb and others emerge huge areas |
| 18:32 |
MTDiscord |
<Warr1024> Mapgen from neighboring mapblocks sometimes "bleeds" into already generated areas. Things like caves and dungeons can carve into those areas. I think the mapgen stuff assumes that terrain cannot be both adjacent to ungenerated areas AND actually meaningfully developed |
| 18:32 |
sfan5 |
you also have to consider that mapgen works in chunks (by default 5x5x5 blocks) |
| 18:32 |
erlehmann |
well if it was only so small i would not car |
| 18:32 |
MTDiscord |
<Warr1024> The 5x5x5 thing helps but does NOT fully account for it. |
| 18:32 |
erlehmann |
care |
| 18:33 |
erlehmann |
after all the terrain around that base is fully explored, also the sky |
| 18:33 |
erlehmann |
so there is no ungenerated thing next to it |
| 18:33 |
MTDiscord |
<Warr1024> 5 blocks is 80 nodes, plus there's the potential for bleed from one chunk into another, so you really need a 160 node margin around each structure you want to preserve. |
| 18:33 |
erlehmann |
oof |
| 18:33 |
sfan5 |
more like 96 |
| 18:34 |
erlehmann |
why 96 |
| 18:34 |
sfan5 |
1 chunk plus 1 block for the bleeding |
| 18:34 |
MTDiscord |
<Warr1024> yeah, ideally bleed-over shouldn't be more than 1 mapblock... |
| 18:34 |
erlehmann |
i mean i already know it is possible to race the mapgen while it is generating stuff, manually |
| 18:34 |
erlehmann |
like if you time it right you can … make it do different things, in a predictable way |
| 18:35 |
erlehmann |
so it is definitely somehow buggy |
| 18:35 |
erlehmann |
(i think i should not be able to influence terrain during generation so it pops in differently by performing ingame actions) |
| 18:36 |
erlehmann |
(which are mostly running to the right spot to make it overwrite itself in the right manner) |
| 18:37 |
erlehmann |
Warr1024 kay27 once mentioned a runaway generation bug that mods can trigger by adding structures or decorations, fun times. |
| 18:37 |
erlehmann |
as i understand it: mapgen triggers, mod places a thing upon generation, that thing needs something emerged for a shell, ad nauseam |
| 18:38 |
MTDiscord |
<Warr1024> Mapgen could theoretically be made to never touch already-generated blocks, but it would probably require simulating all of the terrain feature generation that originates in areas several mapblocks away in each direction and then clipping the resulting nodes to the local area, and then re-running those for the actual areas that contain them, as well as other neighbors ... so it was probably not done that way for speed. |
| 18:38 |
erlehmann |
well what i do not get, why is mapgen overwriting blocks that are NOT ignore |
| 18:39 |
MTDiscord |
<Warr1024> Because a structure that originates in a neighboring block might supposed to have had some structures that bleed into the nearby already generated ones, so they're basically allowed to do that in order to only have to generate that structure once. |
| 18:40 |
erlehmann |
yeah ok but that leads both to runaway generation and mapgen griefing |
| 18:40 |
MTDiscord |
<Warr1024> It really killed me on dungeons, because I wanted to run a special LBM on dungeons to generate terrain, but LBMs only fire when a block is loaded, but mapgen can write changes into blocks that are already loaded, and so I had nodes that escaped the LBM and I actually needed an ABM to catch them too. |
| 18:40 |
erlehmann |
i bet someone stole a node like that too |
| 18:40 |
MTDiscord |
<Warr1024> Yeah, it feels like a shitty trade-off. I just don't know how shitty the OTHER way would have been. |
| 18:40 |
erlehmann |
dungeon in a box |
| 18:41 |
erlehmann |
wellllll … the thing iss mapgen griefing sometimes does stuff like “turn all water at a place into lava” |
| 18:41 |
MTDiscord |
<Warr1024> ¯_(ツ)_/¯ |
| 18:42 |
MTDiscord |
<Warr1024> Oh, yeah, there CAN be some rather ugly mapgen griefs, yeah. |
| 18:42 |
erlehmann |
i think it is basically guaranteed if you have any kind of lua post-generation thingy |
| 18:43 |
erlehmann |
(which is also unreliable and griefed by mapgen) |
| 18:43 |
erlehmann |
where would you modify the code to make it not overwrite already generated terrain? |
| 18:43 |
MTDiscord |
<Warr1024> Certainly not "any" kind, but it's definitely a risk you have to consider carefully when designing mapgen hook stuff. |
| 18:43 |
erlehmann |
well wuzzy just says the same as sfan5, use singlenode |
| 18:44 |
MTDiscord |
<Warr1024> Just "not write already generated terrain" is not really enough to make mapgen not broken though, because just doing that, you'll end up with broken caves depending on the order you explored the area (I could swear MC caves used to have that problem in early beta...?) |
| 18:45 |
erlehmann |
well right now it is possible to build stuff in an area and get it overwritten a short or long time later when an adjacent block is loaded |
| 18:45 |
MTDiscord |
<Warr1024> "use singlenode" is what I do usually |
| 18:46 |
erlehmann |
the problem i am having right now: the admin of a minetest server run by user donations decided to shut it down. i have a large world download of > 1.5G, which is about 10% or so of the world. if there was no server griefing, i could spin that up and input the seed into mapgen v7. |
| 18:46 |
MTDiscord |
<Warr1024> If you have the original mapgen params and seed and such, one thing you COULD try is to let the griefing happen, explore the areas surrounding the territory you want to preserve, then re-overwrite the mapblocks from your original snapshot now that the surrounding terrain margin is already established. That would allow you to continue the map from a snapshot without having to have huge voids for unexplored areas. |
| 18:47 |
erlehmann |
yeah i have that |
| 18:47 |
erlehmann |
but how to re-overwrite stuff? can the engine combine 2 world dl? |
| 18:47 |
MTDiscord |
<Warr1024> So basically get a list of all the mapblocks, then run something to generate the terrain at least like 96 nodes in all directions around each point that currently exists in your snapshot. Then just copy the snapshot blocks over whatever mapgen spits out. |
| 18:47 |
MTDiscord |
<Warr1024> To combine them you gotta know a little basic sqlite. |
| 18:48 |
erlehmann |
do you? |
| 18:48 |
MTDiscord |
<Warr1024> I don't know the exact commands offhand but I could probably look them up. |
| 18:49 |
erlehmann |
it could make it possible to reconstruct the thing from user downloads maybe |
| 18:49 |
MTDiscord |
<Warr1024> I have this gist that does SOME map manipulation, not necessarily what you want, but it demonstrates how you can run sqlite commands that transfer data between 2 sqlite dbs https://gist.github.com/Warr1024/ae760d8d330cb06ef00a048dda89e251 |
| 18:50 |
MTDiscord |
<Warr1024> tbh yeah, it might actually be pretty cool for someone to make a utility that takes multiple user saves, and a map seed, and constructs a playable continuation of a server world, and handles stuff like latest-timestamp rules for merging the blocks. |
| 18:50 |
sfan5 |
to find out what causes the mapgen overwrite bug you probably have to investigate it yourself |
| 18:51 |
MTDiscord |
<Warr1024> Can't really rely on running the MT mapgen code on its own, though; you'd need to do it in actual MT, i.e. with all the necessary mods loaded. |
| 18:51 |
sfan5 |
the solution so far is "use singlenode" and nobody ever wanted more |
| 18:51 |
sfan5 |
did you set the correct world seed in map_meta.txt btw? |
| 18:51 |
MTDiscord |
<Warr1024> We've wanted more, we just haven't wanted more hard enough to make a difference yet... |
| 18:55 |
|
Pexin joined #minetest-dev |
| 18:57 |
|
longerstaff13 joined #minetest-dev |
| 19:01 |
|
ssieb joined #minetest-dev |
| 19:01 |
erlehmann |
Warr1024 it is easy in this case, the mods are mineclonia |
| 19:05 |
erlehmann |
sfan5, yes i think i did. where else would i set the seed? |
| 19:05 |
sfan5 |
nowhere, that's the right place |
| 19:08 |
erlehmann |
Warr1024 wait that sync thing what is missing from it? |
| 19:09 |
|
Desour joined #minetest-dev |
| 19:09 |
erlehmann |
btw, regarding cavegen griefing, cavegen only cares about ground content or not? |
| 19:11 |
MTDiscord |
<Warr1024> What my sync thing does is update the dest sqlite to match the src one exactly |
| 19:11 |
MTDiscord |
<Warr1024> That script MIGHT be close to what you want ... the "delete" part you don't need though. |
| 19:12 |
MTDiscord |
<Warr1024> If "dest" is the world that you did all the mapgenning on and "src" is the database that has the actual original snapshot data, then I think that might be the correct script you want after all |
| 19:12 |
MTDiscord |
<Warr1024> make sure you make a backup of the snapshot before you try any script, just in case you get src and dest mixed up. |
| 19:16 |
erlehmann |
yes |
| 19:17 |
|
specing joined #minetest-dev |
| 19:17 |
MTDiscord |
<Warr1024> I figured there was a good chance I didn't really have to tell you to back shit up first :-) |
| 19:18 |
MTDiscord |
<Warr1024> But I also figured you might appreciate hearing it said anyway :-D |
| 19:19 |
erlehmann |
please act like i am an idiot because i might very well be |
| 19:19 |
erlehmann |
by which i mean i could be distracted or sleepy |
| 19:19 |
erlehmann |
or just dumb |
| 20:03 |
|
appguru joined #minetest-dev |
| 22:09 |
|
fluxionary joined #minetest-dev |
| 22:12 |
|
Desour joined #minetest-dev |
| 22:15 |
|
Extex joined #minetest-dev |
| 22:22 |
|
MisterE joined #minetest-dev |
| 22:33 |
|
v-rob joined #minetest-dev |
| 23:25 |
|
Alias2 joined #minetest-dev |
