Time Nick Message 00:00 pgimeno well, that's not really a representative example, we're not talking canonical base64, we're talking image formats 00:00 pgimeno the Python decoder is not strict, but it is robust in that it doesn't crash 00:00 erlehmann pgimeno i give you the base64 example because it neatly shows a) standard libs are a joke b) you don't *need* a robust decoder (whatever that would be) 00:01 erlehmann pgimeno, it will also happily consume garbage and tell you that it decoded it as base64 00:01 erlehmann all while it definitely has error codes for wrong padding or non-base64-input 00:01 erlehmann this one is just something that slipped through 00:01 pgimeno and what would the consequences be? that you'd have a binary string, not that the decoder crashed 00:02 erlehmann well … lets say you want to blacklist input that decodes to specific binary 00:02 erlehmann suddenly there will be a way to get around that 00:03 pgimeno you need to block it post-decoding obviously 00:03 erlehmann LOL no 00:03 pgimeno why not? 00:03 erlehmann at that point your decoder has been exploited already 00:03 pgimeno if it's not robust, sure 00:03 erlehmann i hope it is obvious to you that it is *way* easier to put the regex before the decoder than to make sure the decoder is correct for all kinds of crap. 00:03 erlehmann is it? 00:04 pgimeno what is? the jpeg library? I imagine it is 00:04 erlehmann i have actually encountered a system at work years ago that validated after decoding 00:04 erlehmann it used ffmpeg 00:05 erlehmann little known feature of ffmpeg, you can feed it synthesizer instructions 00:05 pgimeno if you're going to mention an example of a decoder that craps on bad input, you're using a bad example 00:05 erlehmann so i made the sample rate VERY high and encoded a few seconds of silence 00:05 pgimeno again, bad example 00:05 erlehmann the disk on the testing system was full before it could ever figure out what happened 00:05 erlehmann because i made the *decoder* write a huge wave file 00:06 erlehmann with a few bytes of input 00:06 erlehmann pgimeno come on, almost all parsers are low-quality trash. 00:06 pgimeno erlehmann: I don't care, it is still an example of a decoder that is not robust enough 00:06 pgimeno libjpeg is well tested and scrutinized for vulnerabilities 00:07 erlehmann have you read the LANGSEC papers? they argue pretty well that the best way to get a robust decoder is … having the first step reject everything you do not want to process because it is malformed. 00:07 erlehmann which is what i propose 00:07 erlehmann bc that means you need no error handling in the rest of the code 00:08 erlehmann basically much less effort 00:08 erlehmann after all only valid stuff will be decoded 00:08 pgimeno in this project you can't validate everything, adding a JPEG validator is out of scope of the project 00:08 pgimeno so is adding a PNG validator 00:08 pgimeno you need to rely on the robustness of the decoders 00:08 erlehmann why 00:09 erlehmann look no one can get this right 00:09 pgimeno because JPEG is very complex, and PNG is very complex, and validating them is about as much work as decoding them if not more 00:09 erlehmann unless they follow very specific rules on how to consume input, that is 00:09 pgimeno so you have to write a decoding library just to validate them 00:10 pgimeno Minetest is not an image library, it's a game engine 00:10 erlehmann as i said, you only need to *accept* the features that you can actually process already 00:10 pgimeno and which features are those for JPEG files? 00:10 pgimeno or better said, which features would you leave out in case of a JPEG? 00:11 erlehmann for a game engine that dls most stuff in the beginning … probably progressive mode. don't tell hecks! 00:11 pgimeno if you want to validate it, you need to decode the Huffman or arithmetic coding, check the DCT parameters and whatnot 00:11 erlehmann exif decoding too 00:12 pgimeno so you're going to reject files with exif? 00:12 erlehmann if minetest would already reject such files, i'd very much make sure they never get to the decoder 00:13 pgimeno files with exif info??? are you serious? 00:13 pgimeno you're the same guy who shouted at the developers for removing support for TGA because of the compatibility break, right? 00:14 erlehmann you obviously read my statement but did not comprehend 00:14 erlehmann *if minetest would already reject such files*, i'd very much make sure they never get to the decoder 00:14 erlehmann i have no idea if it does or not 00:14 pgimeno files with exif info should be accepted, and the exif info ignored 00:14 erlehmann but what i want to say is that the validator code must protect the decoder from anything the decoder would have to have error handling for 00:15 pgimeno and for that, in the case of JPEG, it needs to be a JPEG decoder 00:15 erlehmann if the decoder ignores exif, then that's totally ok too 00:15 pgimeno and that's out of scope of this project 00:15 erlehmann pgimeno why would it be out of scope for minetest to not crash? 00:15 pgimeno because libjpeg takes care of that already 00:16 pgimeno with your image, gimp does not crash, feh does not crash, both use libjpeg 00:16 erlehmann uh, no? 00:16 erlehmann yeah that is bc gimp and feh are not stupid 00:16 pgimeno feh refused (not crashed) your first image, and swallows the second without problems 00:16 erlehmann but they were stupid in the past 00:17 erlehmann i have generated pictures myself that crashed feh 00:17 erlehmann using afl-fuzz 00:17 pgimeno yeah, and they both use the same library that minetest is using, and DON'T VALIDATE THE JPEG 00:17 erlehmann that is because the error in minetest can be mitigated in other ways 00:18 erlehmann look, i have generated malformed jpegs at my job and written a report about it 00:18 pgimeno ok, I guess you're agreeing with me that libjpeg is robust enough but there's still a problem with not replacing images with errors with a placeholder 00:18 erlehmann it is extremely easy to crash apps, even if they use a good library 00:19 erlehmann because apps do stupid stuff like “allocate height×width×depths bytes” 00:19 erlehmann after the library has parsed the stuff 00:19 erlehmann the validator would of course prevent that by setting sane limits – before any of the business logic is called 00:21 pgimeno so you don't need a full fledged validator, just to check that the resolution is reasonable and little else 00:21 erlehmann pgimeno consider the case of a not malformed jpeg that is legit 32000×32000 pixels big. what should happen and where? 00:22 pgimeno attempt to load - no error: accept, error: reject and replace 00:22 erlehmann i say it should be rejected by a validator before any pixels are decoded 00:22 erlehmann because otherwise you have a buffer full of trash you need to get rid of 00:22 erlehmann do you agree? 00:22 pgimeno no 00:22 pgimeno gimp opens it, feh opens it 00:23 pgimeno why should minetest refuse to open it? 00:23 erlehmann in minetest i mean 00:23 MTDiscord can you stop imagining imaginary problems 00:23 erlehmann pgimeno, because 32000×32000×3 would be about 3 gigabytes of pixels 00:24 pgimeno and? that fits in my memory 00:24 erlehmann not here lol 00:24 erlehmann and if it fits in yours, i just do it a 100 times 00:24 erlehmann the code needs to handle it 00:25 v-rob Then you get std::bad_alloc. Then handle it, if you want to. What's the problem? 00:25 erlehmann well, do you see no problem with the fact that you maybe allocated a buffer and filled it with decoded pixels BEFORE rejecting the input? 00:25 erlehmann it can be, depending on the input, extremely wasteful and slow 00:25 v-rob Why should we assume the input is incorrect? 00:26 v-rob It won't do us any harm to assume so and give the decoder allocated memory 00:26 erlehmann i already said we should not, before the decoder needs to be a recognizer that only lets stuff we want through 00:26 v-rob What don't we want? 00:26 v-rob If it's invalid input, that's what the decoder's for 00:27 erlehmann lol ok i give up 00:28 erlehmann v-rob what do you want me to do, figure out ways to crash servers and clients until someone takes my propoal seriously? 00:28 erlehmann i mean i can probably easily automate the creation of bad inputs with a fuzzer 00:28 v-rob It WON'T CRASH THE SERVER if we use the decoder correctly. 00:29 erlehmann if 00:30 erlehmann have you tried my pixelflood mod btw? 00:30 v-rob So we don't need a validator like you've been arguing. We just need to fix any bugs in our code. 00:30 erlehmann the state of the art to i fix those bugs is including validators actually 00:30 erlehmann postels principle about robustness is false, this has been proven in the last 10 years i think 00:32 erlehmann pgimeno v-rob have you ever used afl-fuzz? 00:32 v-rob Look: If GIMP doesn't do it, Minetest 100% won't do it. And pgimeno has established that GIMP doesn't 00:33 pgimeno erlehmann: please stop. We have a bug in our hands now. After #11624 is fixed, feel free to report more bugs using a fuzzer or whatever you want, but for now enough is enough 00:33 ShadowBot https://github.com/minetest/minetest/issues/11624 -- Minetest does not start properly with memory sanitizer enabled (use of uninitialized-value) 00:34 erlehmann should stop going on your nerves about the proper way to handle input or stop posting stack traces and stuff that crashes clients? 00:34 erlehmann or both 00:35 pgimeno the first 00:35 erlehmann pgimeno, i am very much interested in how that will be solved because i hope i learn something about it and also because memory sanitizer will be a big help. 00:35 erlehmann ok! 00:37 erlehmann pgimeno v-rob btw is there a public list of known exploits that work against minetest? 00:37 erlehmann i mean not in code execution but rather getting around anticheat and stuff 00:37 v-rob There's plenty of GitHub issues about anticheat. It's not very good 00:38 v-rob Security exploit reports are only visible to core devs. 00:39 erlehmann well surely you can tell me if there are open security issues right now or how many. basically i want to know if the list is huge or tiny. 00:39 erlehmann bc if it is huge, i obv don't need to add more to it 00:40 erlehmann i would only risk posting dupes 00:42 v-rob It's not very big to my knowledge, but that's probably because people don't go around testing Minetest for security holes. If you do find a real vulnerability (not like signed overflow), report it privately (https://github.com/minetest/minetest/security/policy), not with a public bug report. 00:46 erlehmann so what you are saying is i should try to exploit a case of signed overflow 00:46 erlehmann and then you take me seriously 00:46 erlehmann considering it is in the mapgen … unlikely 00:46 pgimeno right? 00:47 erlehmann dw sooner or later i find something else 00:48 v-rob If you do manage to exploit signed overflow, I would be impressed, but we would fix it, yes. 00:49 erlehmann hmm, but surely i would have to hold on to at least one working exploit if i find it? 00:49 erlehmann for leverage i mean 00:49 erlehmann like if a server would require me to run SSCSMs then i could send executable code back 00:50 erlehmann it would only be fair 00:50 erlehmann i think that is a good plan, but i would of course reveal it after some time 00:51 erlehmann v-rob are in-game actions considered exploits too? 00:51 erlehmann like dupes, teleports, privilege escalation, lagging other players out of the game 00:52 erlehmann i mean coordinate leaks aren't 00:52 erlehmann (considered exploits by core devs) 00:53 erlehmann so, would “anyone can destroy arbitrary blocks” (a hypothetical, the techniques i know are fixed) have to be posted in secret? 00:53 v-rob Some really big ones might be (like particularly bad privilege escalation such that servers could be really damaged), but flying and teleports aren't. Cheat clients can already do those way too easily. 00:54 erlehmann well i *can't* arbitrarily teleport by *just* overriding privs, so i guess you know something i don't 00:55 erlehmann it exceeds the anticheat speed limit i guess 00:56 erlehmann v-rob, btw are there cheat clients beyond dragofire and waspsaliva that matter? 00:56 erlehmann i mean and the thing that specing is making 00:56 erlehmann (or is that also ws?) 00:56 v-rob Anticheat's pretty buggy, I believe, but I'm not really an expert. You'd have to ask someone else for more specific info on cheating 00:56 erlehmann dragofire → dragonfire 00:56 v-rob I don't know anything about cheat clients besides the fact that they exist 00:56 erlehmann ah ok so it was a guess 00:57 erlehmann well anticheat is pretty good in preventing movement faster than about 5n/s without physics override 00:57 erlehmann i think 00:57 erlehmann except downwards, bc limiting falling speed would be a bit silly 00:57 erlehmann and upwards you can go i think 50n/s or so, bc trampolines 00:59 erlehmann by physics override i mean if you get a potion of speed or so 00:59 erlehmann then you can be faster 00:59 erlehmann in fact, anticheat is so good at enforcing the speed limit that if you jump on some bouncy node from high enough, when you bounce back you will hit an invisibl ceiling 01:00 specing erlehmann: my client still exists, but it is deprecated in favor of porting to ws 01:00 erlehmann kinde ruined the fun when i encoundered a REALLY big trampoline on a server 01:00 erlehmann specing, cool i hope it makes both you and cora happy! 01:01 specing yeah, sadly I haven't been playing much MT lately (other than passive afk) 01:02 erlehmann same :/ 01:04 erlehmann specing, query 01:18 pgimeno ok, so the problem is that output_message is called, but error_exit isn't, therefore the image is only partially loaded 01:19 erlehmann and then no replacement i generated? 01:19 erlehmann is 01:21 pgimeno no because the image is believed to be valid, because error_exit is the one which should have marked it as invalid 01:22 pgimeno the result is an incompletely decoded image 01:22 erlehmann lul 01:22 erlehmann cool result! 01:22 erlehmann :) 01:23 pgimeno this is fine with gimp and fte because they can read partially decoded images, but Irrlicht does not handle it so gracefully it seems 01:27 erlehmann so the next level for me is crashing gimp or feh 01:27 pgimeno yep 01:27 erlehmann pgimeno, which versions of those tools do you have on which platform? 01:29 pgimeno feh 2.18-2, gimp 2.8.18-1+deb9u1, libjpeg62-turbo 1:1.5.1-2 09:51 rubenwardy merging #11598 in 10 09:51 ShadowBot https://github.com/minetest/minetest/issues/11598 -- Readd TGA to the list of valid texture formats. by rollerozxa 10:12 sfan5 rubenwardy: ping 10:12 rubenwardy lol 10:45 erlehmann rubenwardy can i have the archive of all of contentdb to iterate over it? 10:45 erlehmann rubenwardy i found the stats that appgurueu made interesting and stuff like “how much is this API used” might be interesting 10:45 erlehmann so i'd like to search it 10:53 rubenwardy https://minetest.rubenwardy.com/all_cdb_releases_2021_09_09.tar 10:53 rubenwardy https://minetest.rubenwardy.com/all_cdb_releases_2021_09_09.txt 10:53 erlehmann thx 11:56 erlehmann pgimeno, https://godbolt.org/z/brEMqqYe8 ;) 11:57 pgimeno so? 12:02 erlehmann i find it funny 12:05 pgimeno still off-topic 12:09 erlehmann ok 12:16 sfan5 we don't have a formspec label-like element that will confine the text to a defined box (by means of wordwrap, not scrolling), do we? 12:18 pgimeno the cause of irr#67 turns out not to be the malformedness of the image, but the size. I get the same backtrace with a valid 32000x32000 image. 12:18 ShadowBot https://github.com/minetest/irrlicht/issues/67 -- Pixelflood 2 – Segfault Boogaloo: Sending lottapixel.jpg from server segfaults client upon connection 12:20 rubenwardy We don't sfan5, we have been using read-only with scrolling 12:20 rubenwardy With my wordwrap PR, I wanted to make this possible 12:21 rubenwardy Read-only text areas* 12:21 sfan5 doesn't guiStaticText already have word wrapping though 12:22 sfan5 https://0x0.st/-xdb.png something like this should be trivial to prevent by just setting bounds 12:27 pgimeno same issue with a 32000x32000 valid PNG, same backtrace 12:30 pgimeno the backtrace: http://www.formauri.es/personal/pgimeno/pastes/backtrace-image-crash.txt 12:30 pgimeno would someone please post that info in the report? 12:32 rubenwardy the word wrapping has bugs and will truncate 12:32 rubenwardy if you only want to limit by width it's perfectly fine 12:33 rubenwardy there's currently about 4 different word wrapping implementations in Minetest, I was intending to unify them and test it. But text is hard 12:33 rubenwardy especially if you want support for RTL and Arabic, which I'm completely ignoring 12:34 rubenwardy in Arabic, removing characters can make text longer - which is a massive pain when it comes to word wrapping 12:34 rubenwardy although, it may only be within a word in which case it's possible implement without break-word 12:35 rubenwardy anyway - adding a simple wrapping label makes sense 12:35 rubenwardy as in, it makes sense to add it using current word wrapping as it's still useful 12:36 rubenwardy (The two bugs I found where broken support for \r\n, and for soft hyphens) 12:37 rubenwardy -h 12:51 MTDiscord What's the right behavior if you're sending a packet and the data array to be copied from is empty (results in copying from null pointer). 12:52 MTDiscord Or rather the data array doesn't exist. 12:55 sfan5 this is too vague without linking the code you are looking at 13:02 rubenwardy A lot of the time, data is prefixed with a size 13:03 rubenwardy But yeah too vague 13:05 MTDiscord Sorry, I got it backwards. The destination array is null hehe. 13:07 MTDiscord networking.cpp line 66, the memcpy destination can be null. std::vector.data documentation says this may be the case if the vector is empty. 13:08 MTDiscord I'm surprised this doesn't cause the networking to fail fatally lol. 13:25 rubenwardy > If either dest or src is an invalid or null pointer, the behavior is undefined, even if count is zero. 13:25 rubenwardy definitely wrong 13:25 rubenwardy fix pls 13:25 rubenwardy memory errors in networking code is asking for trouble 13:28 celeron55 it wouldn't be a bad idea to write a fuzzer test that is capable of triggering these by chance 13:28 celeron55 and run it in CI 13:30 rubenwardy yeah 13:41 sfan5 what is an "invalid" pointer? 13:42 sfan5 I don't think a pointer to the end of a buffer is invalid, you may just not write to it 13:42 sfan5 (thinking of a different case) 13:47 erlehmann well you may not advance once past the end? 13:49 erlehmann celeron55, have you looked at anon5 minetest go lib? 13:49 pgimeno invalid is any pointer that doesn't point to allocated memory, for example a pointer to freed memory 14:41 pgimeno well I was wrong, it's more complex than that: https://timsong-cpp.github.io/cppwp/n3337/basic.stc.dynamic.safety - there are even cases where a pointer may have a legal value and be invalid >.< 14:45 sfan5 rubenwardy: cppreference says what you quoted but idk where they got that from 14:45 sfan5 I checked the manpage and googled "posix memcpy" and neither talk about edge cases 14:46 rubenwardy I copied from ccpreference 14:46 rubenwardy lol, fuck the CCP 14:48 sfan5 "Where an argument declared as size_t n specifies the length of the array for a 14:48 sfan5 function, n can have the value zero on a call to that function. Unless explicitly stated 14:48 sfan5 otherwise in the description of a particular function in this subclause, pointer arguments 14:48 sfan5 on such a call shall still have valid values, as described in 7.1.4. " 14:48 sfan5 uh oh I didn't expect it to paste like that; anyway that's from the C99 standard 14:48 erlehmann C and C++ are two different languages that only superficially look similar though 14:49 erlehmann for example, in C you can hang forever doing nothing 14:50 erlehmann in C++, that is UB and will probably be optimized out 14:50 erlehmann i.e. a loop with no side effects may be assumed to terminate 15:38 erlehmann i have a question 15:38 erlehmann you do not use git submodule 15:39 erlehmann but you have a subdirectory with a git repo that absolutely *needs* to be in sync for most of the build 15:39 erlehmann WHY 15:39 erlehmann it is extremely frustrating to have a project like that 15:39 erlehmann so how are you actually building older versions? 15:39 erlehmann do you have some other tooling for it that keeps the dirs in sync?? 15:41 sfan5 I'm pretty sure relevant changes so far were put inside ifdefs 15:41 sfan5 so you could build todays MT with 1.9.0mt0 15:41 sfan5 only obstacle for doing it the other way is that irrmt removed some headers that older MT included (despite not needing them) 15:42 erlehmann wdym ifdefs 15:42 sfan5 grep the src folder for REVISION 15:42 erlehmann uh 15:43 erlehmann so how would i build an arbitrary commit, like what is the command for it? 15:43 erlehmann i mean the CI needs to do it 15:43 erlehmann so i bet there is 15:43 sfan5 what?! 15:44 erlehmann let me figure out something and get back to you 15:48 pgimeno worth considering: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt 15:50 erlehmann haha, oh wow, i think i figured it out. remember the thing about building way too much even though the dependencies are not needed, because cmake & make are stupid? turns out that the dependency tree seems to be underspecified in ome way, so there exist revision ranges where if you build on one commit and then rebuild on another, it will bug out. 15:51 erlehmann usually the “stupid” behaviour of make results in rebuilding enough stuff so that this does not happen. 15:52 sfan5 are you sure that's not your git workaround tricking make into misbehaving? 15:52 erlehmann well, the git workaround trick actually exposes that flaw 15:52 erlehmann think about it, if you rebuild almost everything, you mask the underspecified dep tree for most of the time 15:52 sfan5 I'm sure the make authors are willing to take your bug report 15:53 erlehmann well i think the culprit is cmake 15:53 erlehmann by making make only rebuild what it strictly needs to according to what cmake claims … the build breaks 15:57 erlehmann it happens in many projects that they underspecify dependencies. very common example: files always have to be recompiled when their build rules change, which means for make, they need to depend on the makefile that contains the build rule for them. try any random project, chances are high you can change build rules and the target will not be recompiled. 15:58 erlehmann modern systems implicitly record such dependencies after the build. 15:58 erlehmann (because you can build build rules, so you might not know before the build) 16:13 erlehmann sfan5 wow sarcam thats creative 18:09 MTDiscord Am I seeing an insertion sort every client frame for z-index handling? This ought to be illegal. 18:10 MTDiscord Why every frame? 18:14 sfan5 for hud rendering 18:14 Krock and wield mesh, and for the inventory as well I guess 18:16 MTDiscord Insertion sort is fast on small arrays, isn't it? That doesn't seem like a problem in itself. 18:16 erlehmann can anyone tell me how to prevent mapgen griefing when loading a map from a world download? 18:23 sfan5 set it to singlenode, also: #minetest 18:24 MTDiscord Insertion sort is O(n²) 18:24 MTDiscord A sorted set should be used, elements should be inserted by their z-index there as they are received. 18:26 erlehmann sfan5 no i think there is a bug here 18:26 erlehmann sfan5 already generated map should not be overwritten by mapgen? 18:26 erlehmann so i am asking what to change about the *code* to make it not do that 18:26 erlehmann i am willing to write that patch myself 18:26 sfan5 it will, that's how mapgen works 18:27 sfan5 assuming you are referring to small overlaps between blocks that are written to by the mapgen even though the block is already generated 18:27 erlehmann no those are not small overlaps actually 18:28 MTDiscord You shouldn't use suboptimal data structures and algorithms when using the optimal ones comes at practically zero cost, because they are already premade in the standard libraries. 18:28 erlehmann sfan5, almost any map import i have seen that does not use singlenode is heavily corrupted 18:28 erlehmann and it does happen after the map is loaded 18:28 MTDiscord Luatic, Python uses insertion sort on small list segments because it's faster than merge sort on average in those cases. 18:29 erlehmann sfan5, so i have one world dl with a base that i would love to host, bc the server is gone, but if i visit the base, it is replaced by untouched terrain. 18:29 erlehmann like, first it is there, and then the mapgen replaces it piece by piece. 18:30 MTDiscord The square root of 1k is about 32 18:30 MTDiscord So if you want to do a 32² grid using HUD elements, you'll have insertion sort do 1 million comparisons. 18:31 sfan5 erlehmann: if it works with singlenode the issue is not in general but will be with that specific mapgen 18:31 erlehmann sfan5 do you think it could be possible to mark those areas in world downloads as already generated somehow? or is the info what was generated stored somewhere else? i am willing to create tooling to enable ppl to host stuff from world dls. 18:31 erlehmann ah 18:31 erlehmann well i guess singlenode is really dumb and others emerge huge areas 18:32 MTDiscord Mapgen from neighboring mapblocks sometimes "bleeds" into already generated areas. Things like caves and dungeons can carve into those areas. I think the mapgen stuff assumes that terrain cannot be both adjacent to ungenerated areas AND actually meaningfully developed 18:32 sfan5 you also have to consider that mapgen works in chunks (by default 5x5x5 blocks) 18:32 erlehmann well if it was only so small i would not car 18:32 MTDiscord The 5x5x5 thing helps but does NOT fully account for it. 18:32 erlehmann care 18:33 erlehmann after all the terrain around that base is fully explored, also the sky 18:33 erlehmann so there is no ungenerated thing next to it 18:33 MTDiscord 5 blocks is 80 nodes, plus there's the potential for bleed from one chunk into another, so you really need a 160 node margin around each structure you want to preserve. 18:33 erlehmann oof 18:33 sfan5 more like 96 18:34 erlehmann why 96 18:34 sfan5 1 chunk plus 1 block for the bleeding 18:34 MTDiscord yeah, ideally bleed-over shouldn't be more than 1 mapblock... 18:34 erlehmann i mean i already know it is possible to race the mapgen while it is generating stuff, manually 18:34 erlehmann like if you time it right you can … make it do different things, in a predictable way 18:35 erlehmann so it is definitely somehow buggy 18:35 erlehmann (i think i should not be able to influence terrain during generation so it pops in differently by performing ingame actions) 18:36 erlehmann (which are mostly running to the right spot to make it overwrite itself in the right manner) 18:37 erlehmann Warr1024 kay27 once mentioned a runaway generation bug that mods can trigger by adding structures or decorations, fun times. 18:37 erlehmann as i understand it: mapgen triggers, mod places a thing upon generation, that thing needs something emerged for a shell, ad nauseam 18:38 MTDiscord Mapgen could theoretically be made to never touch already-generated blocks, but it would probably require simulating all of the terrain feature generation that originates in areas several mapblocks away in each direction and then clipping the resulting nodes to the local area, and then re-running those for the actual areas that contain them, as well as other neighbors ... so it was probably not done that way for speed. 18:38 erlehmann well what i do not get, why is mapgen overwriting blocks that are NOT ignore 18:39 MTDiscord Because a structure that originates in a neighboring block might supposed to have had some structures that bleed into the nearby already generated ones, so they're basically allowed to do that in order to only have to generate that structure once. 18:40 erlehmann yeah ok but that leads both to runaway generation and mapgen griefing 18:40 MTDiscord It really killed me on dungeons, because I wanted to run a special LBM on dungeons to generate terrain, but LBMs only fire when a block is loaded, but mapgen can write changes into blocks that are already loaded, and so I had nodes that escaped the LBM and I actually needed an ABM to catch them too. 18:40 erlehmann i bet someone stole a node like that too 18:40 MTDiscord Yeah, it feels like a shitty trade-off. I just don't know how shitty the OTHER way would have been. 18:40 erlehmann dungeon in a box 18:41 erlehmann wellllll … the thing iss mapgen griefing sometimes does stuff like “turn all water at a place into lava” 18:41 MTDiscord ¯_(ツ)_/¯ 18:42 MTDiscord Oh, yeah, there CAN be some rather ugly mapgen griefs, yeah. 18:42 erlehmann i think it is basically guaranteed if you have any kind of lua post-generation thingy 18:43 erlehmann (which is also unreliable and griefed by mapgen) 18:43 erlehmann where would you modify the code to make it not overwrite already generated terrain? 18:43 MTDiscord Certainly not "any" kind, but it's definitely a risk you have to consider carefully when designing mapgen hook stuff. 18:43 erlehmann well wuzzy just says the same as sfan5, use singlenode 18:44 MTDiscord Just "not write already generated terrain" is not really enough to make mapgen not broken though, because just doing that, you'll end up with broken caves depending on the order you explored the area (I could swear MC caves used to have that problem in early beta...?) 18:45 erlehmann well right now it is possible to build stuff in an area and get it overwritten a short or long time later when an adjacent block is loaded 18:45 MTDiscord "use singlenode" is what I do usually 18:46 erlehmann the problem i am having right now: the admin of a minetest server run by user donations decided to shut it down. i have a large world download of > 1.5G, which is about 10% or so of the world. if there was no server griefing, i could spin that up and input the seed into mapgen v7. 18:46 MTDiscord If you have the original mapgen params and seed and such, one thing you COULD try is to let the griefing happen, explore the areas surrounding the territory you want to preserve, then re-overwrite the mapblocks from your original snapshot now that the surrounding terrain margin is already established. That would allow you to continue the map from a snapshot without having to have huge voids for unexplored areas. 18:47 erlehmann yeah i have that 18:47 erlehmann but how to re-overwrite stuff? can the engine combine 2 world dl? 18:47 MTDiscord So basically get a list of all the mapblocks, then run something to generate the terrain at least like 96 nodes in all directions around each point that currently exists in your snapshot. Then just copy the snapshot blocks over whatever mapgen spits out. 18:47 MTDiscord To combine them you gotta know a little basic sqlite. 18:48 erlehmann do you? 18:48 MTDiscord I don't know the exact commands offhand but I could probably look them up. 18:49 erlehmann it could make it possible to reconstruct the thing from user downloads maybe 18:49 MTDiscord I have this gist that does SOME map manipulation, not necessarily what you want, but it demonstrates how you can run sqlite commands that transfer data between 2 sqlite dbs https://gist.github.com/Warr1024/ae760d8d330cb06ef00a048dda89e251 18:50 MTDiscord tbh yeah, it might actually be pretty cool for someone to make a utility that takes multiple user saves, and a map seed, and constructs a playable continuation of a server world, and handles stuff like latest-timestamp rules for merging the blocks. 18:50 sfan5 to find out what causes the mapgen overwrite bug you probably have to investigate it yourself 18:51 MTDiscord Can't really rely on running the MT mapgen code on its own, though; you'd need to do it in actual MT, i.e. with all the necessary mods loaded. 18:51 sfan5 the solution so far is "use singlenode" and nobody ever wanted more 18:51 sfan5 did you set the correct world seed in map_meta.txt btw? 18:51 MTDiscord We've wanted more, we just haven't wanted more hard enough to make a difference yet... 19:01 erlehmann Warr1024 it is easy in this case, the mods are mineclonia 19:05 erlehmann sfan5, yes i think i did. where else would i set the seed? 19:05 sfan5 nowhere, that's the right place 19:08 erlehmann Warr1024 wait that sync thing what is missing from it? 19:09 erlehmann btw, regarding cavegen griefing, cavegen only cares about ground content or not? 19:11 MTDiscord What my sync thing does is update the dest sqlite to match the src one exactly 19:11 MTDiscord That script MIGHT be close to what you want ... the "delete" part you don't need though. 19:12 MTDiscord If "dest" is the world that you did all the mapgenning on and "src" is the database that has the actual original snapshot data, then I think that might be the correct script you want after all 19:12 MTDiscord make sure you make a backup of the snapshot before you try any script, just in case you get src and dest mixed up. 19:16 erlehmann yes 19:17 MTDiscord I figured there was a good chance I didn't really have to tell you to back shit up first :-) 19:18 MTDiscord But I also figured you might appreciate hearing it said anyway :-D 19:19 erlehmann please act like i am an idiot because i might very well be 19:19 erlehmann by which i mean i could be distracted or sleepy 19:19 erlehmann or just dumb