Minetest logo

IRC log for #minetest-dev, 2021-09-15

| Channels | #minetest-dev index | Today | | Google Search | Plaintext

All times shown according to UTC.

Time Nick Message
00:00 pgimeno well, that's not really a representative example, we're not talking canonical base64, we're talking image formats
00:00 pgimeno the Python decoder is not strict, but it is robust in that it doesn't crash
00:00 erlehmann pgimeno i give you the base64 example because it neatly shows a) standard libs are a joke b) you don't *need* a robust decoder (whatever that would be)
00:01 erlehmann pgimeno, it will also happily consume garbage and tell you that it decoded it as base64
00:01 erlehmann all while it definitely has error codes for wrong padding or non-base64-input
00:01 erlehmann this one is just something that slipped through
00:01 pgimeno and what would the consequences be? that you'd have a binary string, not that the decoder crashed
00:02 erlehmann well … lets say you want to blacklist input that decodes to specific binary
00:02 erlehmann suddenly there will be a way to get around that
00:03 pgimeno you need to block it post-decoding obviously
00:03 erlehmann LOL no
00:03 pgimeno why not?
00:03 erlehmann at that point your decoder has been exploited already
00:03 pgimeno if it's not robust, sure
00:03 erlehmann i hope it is obvious to you that it is *way* easier to put the regex before the decoder than to make sure the decoder is correct for all kinds of crap.
00:03 erlehmann is it?
00:04 pgimeno what is? the jpeg library? I imagine it is
00:04 erlehmann i have actually encountered a system at work years ago that validated after decoding
00:04 erlehmann it used ffmpeg
00:05 erlehmann little known feature of ffmpeg, you can feed it synthesizer instructions
00:05 pgimeno if you're going to mention an example of a decoder that craps on bad input, you're using a bad example
00:05 erlehmann so i made the sample rate VERY high and encoded a few seconds of silence
00:05 pgimeno again, bad example
00:05 erlehmann the disk on the testing system was full before it could ever figure out what happened
00:05 erlehmann because i made the *decoder* write a huge wave file
00:06 erlehmann with a few bytes of input
00:06 erlehmann pgimeno come on, almost all parsers are low-quality trash.
00:06 pgimeno erlehmann: I don't care, it is still an example of a decoder that is not robust enough
00:06 pgimeno libjpeg is well tested and scrutinized for vulnerabilities
00:07 erlehmann have you read the LANGSEC papers? they argue pretty well that the best way to get a robust decoder is … having the first step reject everything you do not want to process because it is malformed.
00:07 erlehmann which is what i propose
00:07 erlehmann bc that means you need no error handling in the rest of the code
00:08 erlehmann basically much less effort
00:08 erlehmann after all only valid stuff will be decoded
00:08 pgimeno in this project you can't validate everything, adding a JPEG validator is out of scope of the project
00:08 pgimeno so is adding a PNG validator
00:08 pgimeno you need to rely on the robustness of the decoders
00:08 erlehmann why
00:09 erlehmann look no one can get this right
00:09 pgimeno because JPEG is very complex, and PNG is very complex, and validating them is about as much work as decoding them if not more
00:09 erlehmann unless they follow very specific rules on how to consume input, that is
00:09 pgimeno so you have to write a decoding library just to validate them
00:10 pgimeno Minetest is not an image library, it's a game engine
00:10 erlehmann as i said, you only need to *accept* the features that you can actually process already
00:10 pgimeno and which features are those for JPEG files?
00:10 pgimeno or better said, which features would you leave out in case of a JPEG?
00:11 erlehmann for a game engine that dls most stuff in the beginning … probably progressive mode. don't tell hecks!
00:11 pgimeno if you want to validate it, you need to decode the Huffman or arithmetic coding, check the DCT parameters and whatnot
00:11 erlehmann exif decoding too
00:12 pgimeno so you're going to reject files with exif?
00:12 erlehmann if minetest would already reject such files, i'd very much make sure they never get to the decoder
00:13 pgimeno files with exif info??? are you serious?
00:13 pgimeno you're the same guy who shouted at the developers for removing support for TGA because of the compatibility break, right?
00:14 erlehmann you obviously read my statement but did not comprehend
00:14 erlehmann *if minetest would already reject such files*, i'd very much make sure they never get to the decoder
00:14 erlehmann i have no idea if it does or not
00:14 pgimeno files with exif info should be accepted, and the exif info ignored
00:14 erlehmann but what i want to say is that the validator code must protect the decoder from anything the decoder would have to have error handling for
00:15 pgimeno and for that, in the case of JPEG, it needs to be a JPEG decoder
00:15 erlehmann if the decoder ignores exif, then that's totally ok too
00:15 pgimeno and that's out of scope of this project
00:15 erlehmann pgimeno why would it be out of scope for minetest to not crash?
00:15 pgimeno because libjpeg takes care of that already
00:16 pgimeno with your image, gimp does not crash, feh does not crash, both use libjpeg
00:16 erlehmann uh, no?
00:16 erlehmann yeah that is bc gimp and feh are not stupid
00:16 pgimeno feh refused (not crashed) your first image, and swallows the second without problems
00:16 erlehmann but they were stupid in the past
00:17 erlehmann i have generated pictures myself that crashed feh
00:17 erlehmann using afl-fuzz
00:17 pgimeno yeah, and they both use the same library that minetest is using, and DON'T VALIDATE THE JPEG
00:17 erlehmann that is because the error in minetest can be mitigated in other ways
00:18 erlehmann look, i have generated malformed jpegs at my job and written a report about it
00:18 pgimeno ok, I guess you're agreeing with me that libjpeg is robust enough but there's still a problem with not replacing images with errors with a placeholder
00:18 erlehmann it is extremely easy to crash apps, even if they use a good library
00:19 erlehmann because apps do stupid stuff like “allocate height×width×depths bytes”
00:19 erlehmann after the library has parsed the stuff
00:19 erlehmann the validator would of course prevent that by setting sane limits – before any of the business logic is called
00:21 pgimeno so you don't need a full fledged validator, just to check that the resolution is reasonable and little else
00:21 erlehmann pgimeno consider the case of a not malformed jpeg that is legit 32000×32000 pixels big. what should happen and where?
00:22 pgimeno attempt to load - no error: accept, error: reject and replace
00:22 erlehmann i say it should be rejected by a validator before any pixels are decoded
00:22 erlehmann because otherwise you have a buffer full of trash you need to get rid of
00:22 erlehmann do you agree?
00:22 pgimeno no
00:22 pgimeno gimp opens it, feh opens it
00:23 pgimeno why should minetest refuse to open it?
00:23 erlehmann in minetest i mean
00:23 MTDiscord <Jordach> can you stop imagining imaginary problems
00:23 erlehmann pgimeno, because 32000×32000×3 would be about 3 gigabytes of pixels
00:24 pgimeno and? that fits in my memory
00:24 erlehmann not here lol
00:24 erlehmann and if it fits in yours, i just do it a 100 times
00:24 erlehmann the code needs to handle it
00:25 v-rob Then you get std::bad_alloc. Then handle it, if you want to. What's the problem?
00:25 erlehmann well, do you see no problem with the fact that you maybe allocated a buffer and filled it with decoded pixels BEFORE rejecting the input?
00:25 erlehmann it can be, depending on the input, extremely wasteful and slow
00:25 v-rob Why should we assume the input is incorrect?
00:26 v-rob It won't do us any harm to assume so and give the decoder allocated memory
00:26 erlehmann i already said we should not, before the decoder needs to be a recognizer that only lets stuff we want through
00:26 v-rob What don't we want?
00:26 v-rob If it's invalid input, that's what the decoder's for
00:27 erlehmann lol ok i give up
00:28 erlehmann v-rob what do you want me to do, figure out ways to crash servers and clients until someone takes my propoal seriously?
00:28 erlehmann i mean i can probably easily automate the creation of bad inputs with a fuzzer
00:28 v-rob It WON'T CRASH THE SERVER if we use the decoder correctly.
00:29 erlehmann if
00:30 erlehmann have you tried my pixelflood mod btw?
00:30 v-rob So we don't need a validator like you've been arguing. We just need to fix any bugs in our code.
00:30 erlehmann the state of the art to i fix those bugs is including validators actually
00:30 erlehmann postels principle about robustness is false, this has been proven in the last 10 years i think
00:32 erlehmann pgimeno v-rob have you ever used afl-fuzz?
00:32 v-rob Look: If GIMP doesn't do it, Minetest 100% won't do it. And pgimeno has established that GIMP doesn't
00:33 pgimeno erlehmann: please stop. We have a bug in our hands now. After #11624 is fixed, feel free to report more bugs using a fuzzer or whatever you want, but for now enough is enough
00:33 ShadowBot https://github.com/minetest/minetest/issues/11624 -- Minetest does not start properly with memory sanitizer enabled (use of uninitialized-value)
00:34 erlehmann should stop going on your nerves about the proper way to handle input or stop posting stack traces and stuff that crashes clients?
00:34 erlehmann or both
00:35 pgimeno the first
00:35 erlehmann pgimeno, i am very much interested in how that will be solved because i hope i learn something about it and also because memory sanitizer will be a big help.
00:35 erlehmann ok!
00:37 erlehmann pgimeno v-rob btw is there a public list of known exploits that work against minetest?
00:37 erlehmann i mean not in code execution but rather getting around anticheat and stuff
00:37 v-rob There's plenty of GitHub issues about anticheat. It's not very good
00:38 v-rob Security exploit reports are only visible to core devs.
00:39 erlehmann well surely you can tell me if there are open security issues right now or how many. basically i want to know if the list is huge or tiny.
00:39 erlehmann bc if it is huge, i obv don't need to add more to it
00:40 erlehmann i would only risk posting dupes
00:42 v-rob It's not very big to my knowledge, but that's probably because people don't go around testing Minetest for security holes. If you do find a real vulnerability (not like signed overflow), report it privately (https://github.com/minetest/minetest/security/policy), not with a public bug report.
00:46 erlehmann so what you are saying is i should try to exploit a case of signed overflow
00:46 erlehmann and then you take me seriously
00:46 erlehmann considering it is in the mapgen … unlikely
00:46 pgimeno right?
00:47 erlehmann dw sooner or later i find something else
00:48 v-rob If you do manage to exploit signed overflow, I would be impressed, but we would fix it, yes.
00:49 erlehmann hmm, but surely i would have to hold on to at least one working exploit if i find it?
00:49 erlehmann for leverage i mean
00:49 erlehmann like if a server would require me to run SSCSMs then i could send executable code back
00:50 erlehmann it would only be fair
00:50 erlehmann i think that is a good plan, but i would of course reveal it after some time
00:51 erlehmann v-rob are in-game actions considered exploits too?
00:51 erlehmann like dupes, teleports, privilege escalation, lagging other players out of the game
00:52 erlehmann i mean coordinate leaks aren't
00:52 erlehmann (considered exploits by core devs)
00:53 erlehmann so, would “anyone can destroy arbitrary blocks” (a hypothetical, the techniques i know are fixed) have to be posted in secret?
00:53 v-rob Some really big ones might be (like particularly bad privilege escalation such that servers could be really damaged), but flying and teleports aren't. Cheat clients can already do those way too easily.
00:54 erlehmann well i *can't* arbitrarily teleport by *just* overriding privs, so i guess you know something i don't
00:55 erlehmann it exceeds the anticheat speed limit i guess
00:56 erlehmann v-rob, btw are there cheat clients beyond dragofire and waspsaliva that matter?
00:56 erlehmann i mean and the thing that specing is making
00:56 erlehmann (or is that also ws?)
00:56 v-rob Anticheat's pretty buggy, I believe, but I'm not really an expert. You'd have to ask someone else for more specific info on cheating
00:56 erlehmann dragofire → dragonfire
00:56 v-rob I don't know anything about cheat clients besides the fact that they exist
00:56 erlehmann ah ok so it was a guess
00:57 erlehmann well anticheat is pretty good in preventing movement faster than about 5n/s without physics override
00:57 erlehmann i think
00:57 erlehmann except downwards, bc limiting falling speed would be a bit silly
00:57 erlehmann and upwards you can go i think 50n/s or so, bc trampolines
00:59 erlehmann by physics override i mean if you get a potion of speed or so
00:59 erlehmann then you can be faster
00:59 erlehmann in fact, anticheat is so good at enforcing the speed limit that if you jump on some bouncy node from high enough, when you bounce back you will hit an invisibl ceiling
01:00 specing erlehmann: my client still exists, but it is deprecated in favor of porting to ws
01:00 erlehmann kinde ruined the fun when i encoundered a REALLY big trampoline on a server
01:00 erlehmann specing, cool i hope it makes both you and cora happy!
01:01 specing yeah, sadly I haven't been playing much MT lately (other than passive afk)
01:02 erlehmann same :/
01:04 erlehmann specing, query
01:18 pgimeno ok, so the problem is that output_message is called, but error_exit isn't, therefore the image is only partially loaded
01:19 erlehmann and then no replacement i generated?
01:19 erlehmann is
01:21 pgimeno no because the image is believed to be valid, because error_exit is the one which should have marked it as invalid
01:22 pgimeno the result is an incompletely decoded image
01:22 erlehmann lul
01:22 erlehmann cool result!
01:22 erlehmann :)
01:23 pgimeno this is fine with gimp and fte because they can read partially decoded images, but Irrlicht does not handle it so gracefully it seems
01:27 erlehmann so the next level for me is crashing gimp or feh
01:27 pgimeno yep
01:27 erlehmann pgimeno, which versions of those tools do you have on which platform?
01:29 pgimeno feh 2.18-2, gimp 2.8.18-1+deb9u1, libjpeg62-turbo 1:1.5.1-2
02:28 queria^clone joined #minetest-dev
02:29 Menchers joined #minetest-dev
02:34 queria^clone joined #minetest-dev
02:52 tekakutli joined #minetest-dev
02:52 Extex joined #minetest-dev
03:10 v-rob joined #minetest-dev
04:00 MTDiscord joined #minetest-dev
04:23 tekakutli joined #minetest-dev
05:41 v-rob joined #minetest-dev
07:08 tekakutli joined #minetest-dev
07:15 specing_ joined #minetest-dev
07:15 tekakutli joined #minetest-dev
08:06 tekakutli joined #minetest-dev
09:21 tekakutli joined #minetest-dev
09:51 rubenwardy merging #11598 in 10
09:51 ShadowBot https://github.com/minetest/minetest/issues/11598 -- Readd TGA to the list of valid texture formats. by rollerozxa
09:58 cheapie joined #minetest-dev
10:12 sfan5 rubenwardy: ping
10:12 rubenwardy lol
10:23 calcul0n_ joined #minetest-dev
10:35 tech_exorcist joined #minetest-dev
10:45 erlehmann rubenwardy can i have the archive of all of contentdb to iterate over it?
10:45 erlehmann rubenwardy i found the stats that appgurueu made interesting and stuff like “how much is this API used” might be interesting
10:45 erlehmann so i'd like to search it
10:53 rubenwardy https://minetest.rubenwardy.com/all_cdb_releases_2021_09_09.tar
10:53 rubenwardy https://minetest.rubenwardy.com/all_cdb_releases_2021_09_09.txt
10:53 erlehmann thx
11:04 hendursaga joined #minetest-dev
11:09 tech_exorcist joined #minetest-dev
11:11 tekakutli joined #minetest-dev
11:32 Kimapr[i] joined #minetest-dev
11:56 erlehmann pgimeno, https://godbolt.org/z/brEMqqYe8 ;)
11:57 pgimeno so?
12:02 erlehmann i find it funny
12:05 pgimeno still off-topic
12:09 erlehmann ok
12:16 sfan5 we don't have a formspec label-like element that will confine the text to a defined box (by means of wordwrap, not scrolling), do we?
12:18 pgimeno the cause of irr#67 turns out not to be the malformedness of the image, but the size. I get the same backtrace with a valid 32000x32000 image.
12:18 ShadowBot https://github.com/minetest/irrlicht/issues/67 -- Pixelflood 2 – Segfault Boogaloo: Sending lottapixel.jpg from server segfaults client upon connection
12:20 rubenwardy We don't sfan5, we have been using read-only with scrolling
12:20 rubenwardy With my wordwrap PR, I wanted to make this possible
12:21 rubenwardy Read-only text areas*
12:21 sfan5 doesn't guiStaticText already have word wrapping though
12:22 sfan5 https://0x0.st/-xdb.png something like this should be trivial to prevent by just setting bounds
12:27 pgimeno same issue with a 32000x32000 valid PNG, same backtrace
12:30 pgimeno the backtrace: http://www.formauri.es/personal/pgimeno/pastes/backtrace-image-crash.txt
12:30 pgimeno would someone please post that info in the report?
12:32 rubenwardy the word wrapping has bugs and will truncate
12:32 rubenwardy if you only want to limit by width it's perfectly fine
12:33 adfeno_ left #minetest-dev
12:33 rubenwardy there's currently about 4 different word wrapping implementations in Minetest, I was intending to unify them and test it. But text is hard
12:33 rubenwardy especially if you want support for RTL and Arabic, which I'm completely ignoring
12:34 rubenwardy in Arabic, removing characters can make text longer - which is a massive pain when it comes to word wrapping
12:34 rubenwardy although, it may only be within a word in which case it's possible implement without break-word
12:35 rubenwardy anyway - adding a simple wrapping label makes sense
12:35 rubenwardy as in, it makes sense to add it using current word wrapping as it's still useful
12:36 rubenwardy (The two bugs I found where broken support for \r\n, and for soft hyphens)
12:37 rubenwardy -h
12:51 MTDiscord <josiah_wi> What's the right behavior if you're sending a packet and the data array to be copied from is empty (results in copying from null pointer).
12:52 MTDiscord <josiah_wi> Or rather the data array doesn't exist.
12:55 sfan5 this is too vague without linking the code you are looking at
13:02 rubenwardy A lot of the time, data is prefixed with a size
13:03 rubenwardy But yeah too vague
13:05 MTDiscord <josiah_wi> Sorry, I got it backwards. The destination array is null hehe.
13:07 MTDiscord <josiah_wi> networking.cpp line 66, the memcpy destination can be null. std::vector.data documentation says this may be the case if the vector is empty.
13:08 MTDiscord <josiah_wi> I'm surprised this doesn't cause the networking to fail fatally lol.
13:25 rubenwardy > If either dest or src is an invalid or null pointer, the behavior is undefined, even if count is zero.
13:25 rubenwardy definitely wrong
13:25 rubenwardy fix pls
13:25 rubenwardy memory errors in networking code is asking for trouble
13:28 celeron55 it wouldn't be a bad idea to write a fuzzer test that is capable of triggering these by chance
13:28 celeron55 and run it in CI
13:30 rubenwardy yeah
13:41 sfan5 what is an "invalid" pointer?
13:42 sfan5 I don't think a pointer to the end of a buffer is invalid, you may just not write to it
13:42 sfan5 (thinking of a different case)
13:47 erlehmann well you may not advance once past the end?
13:49 erlehmann celeron55, have you looked at anon5 minetest go lib?
13:49 pgimeno invalid is any pointer that doesn't point to allocated memory, for example a pointer to freed memory
14:11 calcul0n__ joined #minetest-dev
14:28 Fixer joined #minetest-dev
14:39 v-rob joined #minetest-dev
14:41 pgimeno well I was wrong, it's more complex than that: https://timsong-cpp.github.io/cppwp/n3337/basic.stc.dynamic.safety - there are even cases where a pointer may have a legal value and be invalid >.<
14:45 sfan5 rubenwardy: cppreference says what you quoted but idk where they got that from
14:45 sfan5 I checked the manpage and googled "posix memcpy" and neither talk about edge cases
14:46 rubenwardy I copied from ccpreference
14:46 rubenwardy lol, fuck the CCP
14:48 sfan5 "Where  an  argument  declared  as size_t n specifies  the  length  of  the  array  for  a
14:48 sfan5 function, n can  have  the  value  zero  on  a  call  to  that  function. Unless  explicitly  stated
14:48 sfan5 otherwise in the description of a particular function in this subclause, pointer arguments
14:48 sfan5 on  such  a  call  shall  still  have  valid  values,  as  described  in  7.1.4. "
14:48 sfan5 uh oh I didn't expect it to paste like that; anyway that's from the C99 standard
14:48 erlehmann C and C++ are two different languages that only superficially look similar though
14:49 erlehmann for example, in C you can hang forever doing nothing
14:50 erlehmann in C++, that is UB and will probably be optimized out
14:50 erlehmann i.e. a loop with no side effects may be assumed to terminate
15:38 erlehmann i have a question
15:38 erlehmann you do not use git submodule
15:39 erlehmann but you have a subdirectory with a git repo that absolutely *needs* to be in sync for most of the build
15:39 erlehmann WHY
15:39 erlehmann it is extremely frustrating to have a project like that
15:39 erlehmann so how are you actually building older versions?
15:39 erlehmann do you have some other tooling for it that keeps the dirs in sync??
15:41 sfan5 I'm pretty sure relevant changes so far were put inside ifdefs
15:41 sfan5 so you could build todays MT with 1.9.0mt0
15:41 sfan5 only obstacle for doing it the other way is that irrmt removed some headers that older MT included (despite not needing them)
15:42 erlehmann wdym ifdefs
15:42 sfan5 grep the src folder for REVISION
15:42 erlehmann uh
15:43 erlehmann so how would i build an arbitrary commit, like what is the command for it?
15:43 erlehmann i mean the CI needs to do it
15:43 erlehmann so i bet there is
15:43 sfan5 what?!
15:44 erlehmann let me figure out something and get back to you
15:48 pgimeno worth considering: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt
15:50 erlehmann haha, oh wow, i think i figured it out. remember the thing about building way too much even though the dependencies are not needed, because cmake & make are stupid? turns out that the dependency tree seems to be underspecified in ome way, so there exist revision ranges where if you build on one commit and then rebuild on another, it will bug out.
15:51 erlehmann usually the “stupid” behaviour of make results in rebuilding enough stuff so that this does not happen.
15:52 sfan5 are you sure that's not your git workaround tricking make into misbehaving?
15:52 erlehmann well, the git workaround trick actually exposes that flaw
15:52 erlehmann think about it, if you rebuild almost everything, you mask the underspecified dep tree for most of the time
15:52 sfan5 I'm sure the make authors are willing to take your bug report
15:53 erlehmann well i think the culprit is cmake
15:53 Extex joined #minetest-dev
15:53 erlehmann by making make only rebuild what it strictly needs to according to what cmake claims … the build breaks
15:57 erlehmann it happens in many projects that they underspecify dependencies. very common example: files always have to be recompiled when their build rules change, which means for make, they need to depend on the makefile that contains the build rule for them. try any random project, chances are high you can change build rules and the target will not be recompiled.
15:58 erlehmann modern systems implicitly record such dependencies after the build.
15:58 erlehmann (because you can build build rules, so you might not know before the build)
16:13 erlehmann sfan5 wow sarcam thats creative
16:13 v-rob joined #minetest-dev
16:42 proller joined #minetest-dev
17:14 v-rob joined #minetest-dev
17:41 calcul0n joined #minetest-dev
18:01 erlehmann joined #minetest-dev
18:01 erlehmann joined #minetest-dev
18:09 MTDiscord <luatic> Am I seeing an insertion sort every client frame for z-index handling? This ought to be illegal.
18:10 Extex joined #minetest-dev
18:10 MTDiscord <Noodlemire> Why every frame?
18:14 sfan5 for hud rendering
18:14 Krock and wield mesh, and for the inventory as well I guess
18:16 MTDiscord <josiah_wi> Insertion sort is fast on small arrays, isn't it? That doesn't seem like a problem in itself.
18:16 erlehmann can anyone tell me how to prevent mapgen griefing when loading a map from a world download?
18:23 sfan5 set it to singlenode, also: #minetest
18:24 MTDiscord <luatic> Insertion sort is O(n²)
18:24 MTDiscord <luatic> A sorted set should be used, elements should be inserted by their z-index there as they are received.
18:26 erlehmann sfan5 no i think there is a bug here
18:26 erlehmann sfan5 already generated map should not be overwritten by mapgen?
18:26 erlehmann so i am asking what to change about the *code* to make it not do that
18:26 erlehmann i am willing to write that patch myself
18:26 sfan5 it will, that's how mapgen works
18:27 sfan5 assuming you are referring to small overlaps between blocks that are written to by the mapgen even though the block is already generated
18:27 erlehmann no those are not small overlaps actually
18:28 MTDiscord <luatic> You shouldn't use suboptimal data structures and algorithms when using the optimal ones comes at practically zero cost, because they are already premade in the standard libraries.
18:28 erlehmann sfan5, almost any map import i have seen that does not use singlenode is heavily corrupted
18:28 erlehmann and it does happen after the map is loaded
18:28 MTDiscord <josiah_wi> Luatic, Python uses insertion sort on small list segments because it's faster than merge sort on average in those cases.
18:29 erlehmann sfan5, so i have one world dl with a base that i would love to host, bc the server is gone, but if i visit the base, it is replaced by untouched terrain.
18:29 erlehmann like, first it is there, and then the mapgen replaces it piece by piece.
18:30 MTDiscord <luatic> The square root of 1k is about 32
18:30 MTDiscord <luatic> So if you want to do a 32² grid using HUD elements, you'll have insertion sort do 1 million comparisons.
18:31 sfan5 erlehmann: if it works with singlenode the issue is not in general but will be with that specific mapgen
18:31 erlehmann sfan5 do you think it could be possible to mark those areas in world downloads as already generated somehow? or is the info what was generated stored somewhere else? i am willing to create tooling to enable ppl to host stuff from world dls.
18:31 erlehmann ah
18:31 erlehmann well i guess singlenode is really dumb and others emerge huge areas
18:32 MTDiscord <Warr1024> Mapgen from neighboring mapblocks sometimes "bleeds" into already generated areas.  Things like caves and dungeons can carve into those areas.  I think the mapgen stuff assumes that terrain cannot be both adjacent to ungenerated areas AND actually meaningfully developed
18:32 sfan5 you also have to consider that mapgen works in chunks (by default 5x5x5 blocks)
18:32 erlehmann well if it was only so small i would not car
18:32 MTDiscord <Warr1024> The 5x5x5 thing helps but does NOT fully account for it.
18:32 erlehmann care
18:33 erlehmann after all the terrain around that base is fully explored, also the sky
18:33 erlehmann so there is no ungenerated thing next to it
18:33 MTDiscord <Warr1024> 5 blocks is 80 nodes, plus there's the potential for bleed from one chunk into another, so you really need a 160 node margin around each structure you want to preserve.
18:33 erlehmann oof
18:33 sfan5 more like 96
18:34 erlehmann why 96
18:34 sfan5 1 chunk plus 1 block for the bleeding
18:34 MTDiscord <Warr1024> yeah, ideally bleed-over shouldn't be more than 1 mapblock...
18:34 erlehmann i mean i already know it is possible to race the mapgen while it is generating stuff, manually
18:34 erlehmann like if you time it right you can … make it do different things, in a predictable way
18:35 erlehmann so it is definitely somehow buggy
18:35 erlehmann (i think i should not be able to influence terrain during generation so it pops in differently by performing ingame actions)
18:36 erlehmann (which are mostly running to the right spot to make it overwrite itself in the right manner)
18:37 erlehmann Warr1024 kay27 once mentioned a runaway generation bug that mods can trigger by adding structures or decorations, fun times.
18:37 erlehmann as i understand it: mapgen triggers, mod places a thing upon generation, that thing needs something emerged for a shell, ad nauseam
18:38 MTDiscord <Warr1024> Mapgen could theoretically be made to never touch already-generated blocks, but it would probably require simulating all of the terrain feature generation that originates in areas several mapblocks away in each direction and then clipping the resulting nodes to the local area, and then re-running those for the actual areas that contain them, as well as other neighbors ... so it was probably not done that way for speed.
18:38 erlehmann well what i do not get, why is mapgen overwriting blocks that are NOT ignore
18:39 MTDiscord <Warr1024> Because a structure that originates in a neighboring block might supposed to have had some structures that bleed into the nearby already generated ones, so they're basically allowed to do that in order to only have to generate that structure once.
18:40 erlehmann yeah ok but that leads both to runaway generation and mapgen griefing
18:40 MTDiscord <Warr1024> It really killed me on dungeons, because I wanted to run a special LBM on dungeons to generate terrain, but LBMs only fire when a block is loaded, but mapgen can write changes into blocks that are already loaded, and so I had nodes that escaped the LBM and I actually needed an ABM to catch them too.
18:40 erlehmann i bet someone stole a node like that too
18:40 MTDiscord <Warr1024> Yeah, it feels like a shitty trade-off.  I just don't know how shitty the OTHER way would have been.
18:40 erlehmann dungeon in a box
18:41 erlehmann wellllll … the thing iss mapgen griefing sometimes does stuff like “turn all water at a place into lava”
18:41 MTDiscord <Warr1024> ¯_(ツ)_/¯
18:42 MTDiscord <Warr1024> Oh, yeah, there CAN be some rather ugly mapgen griefs, yeah.
18:42 erlehmann i think it is basically guaranteed if you have any kind of lua post-generation thingy
18:43 erlehmann (which is also unreliable and griefed by mapgen)
18:43 erlehmann where would you modify the code to make it not overwrite already generated terrain?
18:43 MTDiscord <Warr1024> Certainly not "any" kind, but it's definitely a risk you have to consider carefully when designing mapgen hook stuff.
18:43 erlehmann well wuzzy just says the same as sfan5, use singlenode
18:44 MTDiscord <Warr1024> Just "not write already generated terrain" is not really enough to make mapgen not broken though, because just doing that, you'll end up with broken caves depending on the order you explored the area (I could swear MC caves used to have that problem in early beta...?)
18:45 erlehmann well right now it is possible to build stuff in an area and get it overwritten a short or long time later when an adjacent block is loaded
18:45 MTDiscord <Warr1024> "use singlenode" is what I do usually
18:46 erlehmann the problem i am having right now: the admin of a minetest server run by user donations decided to shut it down. i have a large world download of > 1.5G, which is about 10% or so of the world. if there was no server griefing, i could spin that up and input the seed into mapgen v7.
18:46 MTDiscord <Warr1024> If you have the original mapgen params and seed and such, one thing you COULD try is to let the griefing happen, explore the areas surrounding the territory you want to preserve, then re-overwrite the mapblocks from your original snapshot now that the surrounding terrain margin is already established.  That would allow you to continue the map from a snapshot without having to have huge voids for unexplored areas.
18:47 erlehmann yeah i have that
18:47 erlehmann but how to re-overwrite stuff? can the engine combine 2 world dl?
18:47 MTDiscord <Warr1024> So basically get a list of all the mapblocks, then run something to generate the terrain at least like 96 nodes in all directions around each point that currently exists in your snapshot.  Then just copy the snapshot blocks over whatever mapgen spits out.
18:47 MTDiscord <Warr1024> To combine them you gotta know a little basic sqlite.
18:48 erlehmann do you?
18:48 MTDiscord <Warr1024> I don't know the exact commands offhand but I could probably look them up.
18:49 erlehmann it could make it possible to reconstruct the thing from user downloads maybe
18:49 MTDiscord <Warr1024> I have this gist that does SOME map manipulation, not necessarily what you want, but it demonstrates how you can run sqlite commands that transfer data between 2 sqlite dbs https://gist.github.com/Warr1024/ae760d8d330cb06ef00a048dda89e251
18:50 MTDiscord <Warr1024> tbh yeah, it might actually be pretty cool for someone to make a utility that takes multiple user saves, and a map seed, and constructs a playable continuation of a server world, and handles stuff like latest-timestamp rules for merging the blocks.
18:50 sfan5 to find out what causes the mapgen overwrite bug you probably have to investigate it yourself
18:51 MTDiscord <Warr1024> Can't really rely on running the MT mapgen code on its own, though; you'd need to do it in actual MT, i.e. with all the necessary mods loaded.
18:51 sfan5 the solution so far is "use singlenode" and nobody ever wanted more
18:51 sfan5 did you set the correct world seed in map_meta.txt btw?
18:51 MTDiscord <Warr1024> We've wanted more, we just haven't wanted more hard enough to make a difference yet...
18:55 Pexin joined #minetest-dev
18:57 longerstaff13 joined #minetest-dev
19:01 ssieb joined #minetest-dev
19:01 erlehmann Warr1024 it is easy in this case, the mods are mineclonia
19:05 erlehmann sfan5, yes i think i did. where else would i set the seed?
19:05 sfan5 nowhere, that's the right place
19:08 erlehmann Warr1024 wait that sync thing what is missing from it?
19:09 Desour joined #minetest-dev
19:09 erlehmann btw, regarding cavegen griefing, cavegen only cares about ground content or not?
19:11 MTDiscord <Warr1024> What my sync thing does is update the dest sqlite to match the src one exactly
19:11 MTDiscord <Warr1024> That script MIGHT be close to what you want ... the "delete" part you don't need though.
19:12 MTDiscord <Warr1024> If "dest" is the world that you did all the mapgenning on and "src" is the database that has the actual original snapshot data, then I think that might be the correct script you want after all
19:12 MTDiscord <Warr1024> make sure you make a backup of the snapshot before you try any script, just in case you get src and dest mixed up.
19:16 erlehmann yes
19:17 specing joined #minetest-dev
19:17 MTDiscord <Warr1024> I figured there was a good chance I didn't really have to tell you to back shit up first :-)
19:18 MTDiscord <Warr1024> But I also figured you might appreciate hearing it said anyway :-D
19:19 erlehmann please act like i am an idiot because i might very well be
19:19 erlehmann by which i mean i could be distracted or sleepy
19:19 erlehmann or just dumb
20:03 appguru joined #minetest-dev
22:09 fluxionary joined #minetest-dev
22:12 Desour joined #minetest-dev
22:15 Extex joined #minetest-dev
22:22 MisterE joined #minetest-dev
22:33 v-rob joined #minetest-dev
23:25 Alias2 joined #minetest-dev

| Channels | #minetest-dev index | Today | | Google Search | Plaintext