Time Nick Message 00:38 est31 PTAL https://github.com/est31/minetest/commit/dc2e97deb8c73d1d1da9f702dc4e774bab8f2898 00:43 est31 hmmmm, ? 00:44 est31 this is the code I've removed: https://github.com/est31/minetest/commit/5e507c9829942c434a6f1ae7a4f3a488c7e50bef#diff-ac9ac0a3a5e432320dd8784aaeeef672L1092 00:44 est31 in the regressing commit 00:45 est31 the commit now adds it back 00:46 est31 I'll push it in 30 minutes. 04:11 paramat docs added will merge in 20mins https://github.com/minetest/minetest/pull/3359 04:22 paramat now merging 04:32 paramat done 05:19 paramat hmmmm https://github.com/minetest/minetest/pull/3364 Mgfractal: Create a choice of 4 mandelbrot formulas 06:30 VanessaE so, can someone tell me what THIS means? 06:30 VanessaE terminate called after throwing an instance of 'FileNotGoodException' 06:30 VanessaE what(): LevelDB error: IO error: lock /home/minetest/.minetest/worlds/Creative_World/map.db/LOCK: Resource temporarily unavailable 06:31 VanessaE one of my servers went down for a number of hours (while I was afk), then just randomly came back up on its own after I don't know how many restart attempts 06:31 paramat a plan to fix the problem with low decoration densities: if 'area * nval' is between 0 and 1, instead of rounding 'deco count' down to 0, calculate a chance for 'deco count' to be 1. i'll work on this 06:32 sfan5 VanessaE: lock file is held by another process 06:33 VanessaE could a server crash result in a stale lock file? 06:33 VanessaE (because otherwise I can't think of any way that what you suggest can happen at all) 06:35 paramat sfan5 any comments? https://github.com/minetest/minetest_game/pull/719 06:35 sfan5 I'm not exactly sure how lock files work 06:35 sfan5 so maybe 06:36 VanessaE ok. 06:36 sfan5 paramat: +1 06:36 sfan5 paramat: since when do we have a "catch_up" param? 06:36 VanessaE this is what I *think* the chain of crashes started with: http://pastebin.ubuntu.com/13245765/ 06:36 paramat since i created it =) 06:37 sfan5 looking at the "write failed" by xban 06:38 sfan5 maybe a disc hiccup? 06:38 VanessaE well that "write failed" has been happening for a few days now, with no explanation 06:39 VanessaE permissions and ownership are file and the xban db is readable 06:39 cheapie Apparently no disk errors were logged, either by the kernel or the drives. 06:39 VanessaE ^^^ and none of the other servers on the machine were affected. 06:39 cheapie So the xban DB is readable... is it writeable? 06:40 VanessaE yep. 06:40 VanessaE -rw-r--r-- 1 minetest minetest 79060 Nov 13 01:39 /home/minetest/.minetest/worlds/Creative_World/xban.db 06:41 cheapie Are the other servers doing the xban "write failed"? 06:41 * VanessaE checks 06:41 VanessaE yes 06:41 cheapie OK, so I don't think that's related. 06:43 VanessaE is something preventing files being written to the world dir? 06:44 VanessaE (mod security is not enabled) 07:36 paramat hmmmm another PR https://github.com/minetest/minetest/pull/3365 Decoration API: Allow very low density decorations. bbl 08:50 celeron55 wait what 08:50 celeron55 why is the emerge queue not a queue and instead just an std::map indexed by block positions 08:51 celeron55 that messes up the load priorities 08:52 celeron55 the block selecting and sending code is very careful to first load more important things and later less important things, and then the emergethread just shits all over it 08:55 celeron55 this makes it so that if player A is standing at a position which is sorted first in an std::map compared to a position where player B is, player A's block loads are very much prioritized over player B's 08:55 celeron55 uncontrollably much 08:56 celeron55 basically player B gets whatever is left over from player A, if any 08:59 celeron55 wait no, actually it's working fine 08:59 celeron55 the queue is just per-thread while the data is global for... no reason 09:02 hmmmm there 09:02 hmmmm is a reason 09:03 est31 src/emerge.{cpp,h} is full of weirdnesses, until commit 0850d3bb930a where hmmmm cleaned it up, EmergeManager had a vector named "emergethread". 09:04 est31 singular 09:04 celeron55 i don't think that's weird but whatever 09:05 hmmmm that's because at that point in my life i was still debating if an array should be a singular or plural 09:06 hmmmm it kind of made sense when i wrote the multithreaded emergemanager because emergethread[4] was like saying "Emerge Thread #4" 09:06 celeron55 hmmmm: not that i really care, but what is the reason? maybe i need to tell it to somebody some day 09:07 hmmmm the map is global for uniqueness 09:07 hmmmm i thought that was kind of obvious with the "update_existing" bit of logic in pushBlockEmergeData() 09:08 celeron55 but it needs to find the thread to which it was queued previously anyway, right? 09:08 hmmmm ?? why 09:09 celeron55 i don't find it doing that though; maybe it doesn't 09:09 hmmmm in any case you might notice that I'm on a minetest break 09:09 hmmmm when I come back I want to fix the EmergeThread disk/generate queue distinction 09:10 celeron55 what's happening then if enqueueBlockEmerge is spammed by the same block position all the time, like the block selection and sending code does? 09:10 hmmmm this is going to be kind of big because I'm going to have to begin storing which blocks exist in a very efficient manner 09:10 hmmmm bloom filter probably 09:10 hmmmm what's happening? it's getting updated 09:11 celeron55 where does it see that it's already queued and doesn't add it to the thread's queue each time 09:11 hmmmm if it's getting enqueued with an option that was not present before, it'll set that flag 09:12 hmmmm oh this is not good 09:12 hmmmm this is a regression 09:12 hmmmm there should be a return false after this line https://github.com/minetest/minetest/blob/master/src/emerge.cpp#L409 09:13 hmmmm i wonder why nobody said anything about the emerge thread being super slow lately... did this bug have any effect in practice, I wonder? 09:14 celeron55 okay; in that case i also need to make an enum for the return value so that enqueueBlockEmerge can distinct between queue being full and block being already in queue (the block selecting and sending system obviously needs that information) 09:15 celeron55 distinguish* (how does this word even work) 09:15 est31 hmmmm, it can be fancier, can't it? 09:15 est31 e.g. old_flags = bedata.flags; 09:15 est31 bedata.flags |= flags; 09:16 est31 return bedata.flags == old_flags 09:16 hmmmm that's kinda redundant seeing as you already have a boolean parameter telling you if it existed or not... 09:17 hmmmm so 09:18 hmmmm I am going to just make the 5 second fix for this 09:18 hmmmm i realize it'll merge conflict with celeron's thing but this is pretty important imo 09:22 celeron55 what boolean parameter 09:23 celeron55 oh you answered to est31 09:24 celeron55 hmmmm: do realize that if you just make that return false, then you break the return value of enqueueBlockEmerge because then it will return false if the block existed in the queue in addition to returning false when the queue was full 09:24 hmmmm ? 09:24 hmmmm why is that broken 09:25 hmmmm maybe it's broken for your purposes, but i thought we established that you're going to change the return value to an enumeration 09:25 celeron55 enqueueBlockEmerge should return false only if the queue is full, because that is what the block selecting and sending code expects; it's the only code that calls the function 09:25 celeron55 if it gets false, it stops adding stuff to the queue 09:25 hmmmm the *current* code returns false? 09:25 celeron55 i am talking about what is in upstream right now 09:25 hmmmm ohh okay 09:26 hmmmm i see 09:26 hmmmm well minetest hasn't exploded yet so I'll let it go till tomorrow I really need to go to sleep 09:27 celeron55 the thread-specific queues handle the spam fairly well because they check the in-memory MapBlock as the very first thing they do 09:28 celeron55 and because they have no length limits 09:28 est31 with the "boolean param" you mean update_existing ? 09:30 est31 hrmm I guess my "fancy solution" isnt fancy at all 09:30 est31 but outright wrong 09:30 est31 simple return false is 100% ok 09:47 paramat can anyone review https://github.com/minetest/minetest/pull/3364 ? since hmmmmm is taking a break 09:58 kilbith paramat, have you an opinion on these headers 09:58 kilbith https://github.com/minetest/minetest/pull/3275#issuecomment-154723731 09:58 kilbith https://github.com/minetest/minetest/pull/3275#issuecomment-155092512 09:59 est31 If we change the header image, then I'd favor your second one. 10:00 paramat ah nice i didn't see these 10:02 paramat i like the first one because it's more universal so more suitable for an engine 10:02 kilbith thanks for the input both 11:23 celeron55 i updated this again: https://github.com/celeron55/minetest/commits/far_map_wip 11:23 celeron55 it now should be more responsive at choosing what it's generating even on slow systems 14:30 celeron55 http://i.imgur.com/WCftPPS.jpg 14:30 celeron55 this bug is interesting for sure 14:31 celeron55 in that screenshot, it has decided that various kinds of water should generally look like various kinds of bricks 14:31 celeron55 but at that one point, it should look like water, and instead trees should be yellow and dirt should be whatever the hell that is 14:50 celeron55 valgrind doesn't report anything being weird 14:51 est31 then its super w3ird 14:54 celeron55 i tried making the texture atlases to contain 64 nodes each instead of 256, and got a different result 14:54 celeron55 this means node ids are not truncated to 8 bits and the atlases are getting mixed up 14:55 celeron55 it's very weird 14:55 celeron55 it's not like i'm not trying to solve this 14:55 celeron55 no bug just seems to be found anywhere 15:05 celeron55 (and the problem is fixed once i make the atlas image large enough to fit all nodes) 15:23 celeron55 now i even tried generating meshes in the main thread 15:23 celeron55 needless to say, this has no effect whatsoever 15:24 celeron55 interestingly my farmap-optimized mesh generator is so fast that it doesn't even create any FPS spikes 15:26 celeron55 well okay, it does 15:27 celeron55 (something like 20ms each i guess) 15:34 est31 what precisely is the atlas? 15:37 est31 hmm as it seems generating it causes to pile up so much RAM so that my OOM killer kicks in 15:37 est31 and kills KDE 15:38 est31 well, thats nice, better than my system freezing and me having to sysrq 15:38 celeron55 you need to tune the range down if you don't have many gigabytes of free RAM 15:39 est31 I didnt even spawn 15:39 est31 I was inside theloading screen 15:39 est31 at "generating atlas" 15:39 celeron55 and each node in a game takes about 50kB of space in the texture atlas 15:39 est31 but yeah I admit I've tried dreambuilder 15:40 celeron55 it has like 20k 15:40 est31 trivial optimisation would be to use texture names instead of node ids 15:40 est31 there are less or equal numer of textures as nodes 15:41 celeron55 it's done already 15:41 celeron55 that's why it actually isn't 50kB as long as nodes share the same textures 15:42 celeron55 that can be helped with a special shader that allows the atlas to not contain repeated textures 15:43 celeron55 it's bound to be a problem without shaders though 15:45 est31 why is a shader needed for that? 15:46 celeron55 each farnode face needs to have 4x4 nodes repeated on it (and then there's the 16x16 LOD level too) 15:46 est31 ah I see 15:47 est31 so its taking 4x times of RAM? 15:47 celeron55 as it is now, the atlas consists of 8x8 repetitions of node faces with pre-baked lighting effects on them 15:47 celeron55 because with 4x4 the neighboring nodes would bleed to them when rendered 15:47 celeron55 so it takes the middle 4x4 and it looks fine 15:47 celeron55 so each node uses 128x128px of atlas space 15:48 celeron55 i mean 15:48 celeron55 not each node but each texture that is used by some node 15:49 est31 so it pre-bakes lighting? 15:49 est31 how much lighting levels do we have, 2^4? 15:49 celeron55 not that kind of lighting 15:49 celeron55 just some cheats so that it looks more like it's 4x4 nodes 15:52 celeron55 updating an large image into a texture is so slow in irrlicht that it can't be done at runtime so the atlas has to have all nodes on it that will be used 15:53 celeron55 slowness wouldn't be an issue if it could be threaded but of course it can't 15:53 celeron55 and once you make them small, you lose all the benefit from not having to switch textures when rendering 15:54 celeron55 or have to work around it with more complex systems 15:56 est31 I really like the idea of sending divs_per_mb over the network 15:56 celeron55 TL;DR: rendering a lot of stuff at once takes a lot of RAM or half of a lot of RAM plus very complex code 15:56 celeron55 or a quarter of a lot of RAM and even more complex code 15:57 est31 I like the shader idea 15:59 celeron55 of course i need to somehow make this such that it doesn't just crash and die if you open dreambuilder on an old computer with farmap enabled 15:59 celeron55 that might be a matter of just giving up on it but anyway 15:59 est31 it can do some calculations 15:59 est31 it knows all the nodedef and which textures it uses 15:59 est31 so take the rough size, and add it all up 16:00 est31 if larger than 50% of the free RAM on the system, dont use farmap 16:00 est31 perhaps overrideable with a setting, farmap_no_shader_ram_limit or so 16:00 celeron55 i mean, it should be able to just make an atlas with like 2x2 pixels per node 16:01 est31 thats even smarter 16:01 celeron55 so that it can render stuff with at least some colors 16:05 est31 but from trying it in minimal game it really looks great 16:13 celeron55 i think i'll actually default the far map texture atlas to something less than 16x16 per simulated node face; it's kind of excessive for creating the effect to begin with anyway 16:13 celeron55 8x8 seems to work fine 16:13 celeron55 one can always configure it higher if he wants quality over quantity 16:14 celeron55 or has the RAM to support both 16:39 celeron55 the nice thing about this setting is that by setting it low enough, the bug that weirdly swaps the atlas texture doesn't happen as you only have a single atlas texture 8) 17:09 celeron55 pushed the new setting 17:45 VanessaE celeron55: it's looking pretty good now. texture swapping glitch is pretty much gone. did you notice though that on newly-generated terrain, the farmap blocks are dark along the edges of not-yet-generated areas? 17:55 VanessaE fwiw, under Dreambuilder with HDX256, my machine only uses 2.4 GB or so before it creates the farmap atlases. after I get into a world (in this case, local copy of VE-Survival), call it 4 GB 17:55 VanessaE so I don't think you actually need to worry TOO much about RAM usage 17:57 VanessaE and your map loading speed is greatly improved now - even this local copy of VE-S loads at a tolerable speed once it gets past the initial 2 mins worth of crap that technic mod does on startup (and that's an achievement ;) ) 18:05 VanessaE bbl 19:08 rubenwardy game#726 19:10 rubenwardy ShadowBot! 19:11 rubenwardy Issue - Split default into smaller mods - https://github.com/minetest/minetest_game/issues/726 22:09 who_wants_some https://imgur.com/a/F7SuR lol 22:09 who_wants_some sorry, wrong channel 22:09 hmmmm ehh, this is the right channel i'd say 22:10 hmmmm is this reproducible? 22:10 who_wants_some it is mod problem, not related to minetest core developement, sorry 22:10 hmmmm ok