Time Nick Message 09:54 VanessaE so, I've been over nearly every byte of biome_lib code, rewriting a few things here and there, and still can't figure out what's causing occasional empty mapblocks. 09:55 VanessaE can someone please take a look? (https://gitlab.com/VanessaE/biome_lib/-/blob/master/init.lua) 13:23 MTDiscord What's the deal with using biome_lib:method()s instead of biome_lib.method()s when you reference buome_lib but not self? 13:23 VanessaE old. 13:23 VanessaE that's all 13:24 VanessaE I've been changing over the latter as I make little reworks here and there 13:24 VanessaE but mostly the colon notation is simply because it's old :) 13:24 VanessaE when I started it, I wasn't all that good at Lua 13:28 MTDiscord Hmm, I also make some methods dual local/library, i.e the ones I want to share with dependents but not necessarily allow them to override. That way I can reference the local one internally, which is not only faster at runtime, but more importantly easier to read without the library name plastered everywhere... 13:28 MTDiscord Always takes a while to get past stylistic things before I can start to actually understand the logic. 13:29 VanessaE right 13:30 VanessaE I think I'll finish-up transitioning it all over to period notation with my next set of changes (and just provide shims for old mods) 13:30 MTDiscord Does this thing really infer the types of nodes that can be on the surface based on the first mapblock it sees actually generated? 13:30 VanessaE no 13:31 MTDiscord Trying to figure out what https://gitlab.com/VanessaE/biome_lib/-/blob/master/init.lua#L435 is all about 13:31 VanessaE each block that's emerged by the mapgeg gets logged, scanned, populated, and then removed from the log 13:31 VanessaE mapgen* 13:32 VanessaE I'm using the position hash as a key to a table that holds everything having to do with the mapblock at that position 13:34 VanessaE the idea being that each mapblock gets a full workup before it's removed from the block log, then on to the next 13:35 MTDiscord Oh, you can't use a local because you do this across multiple ticks? That's kinda convoluted... 13:36 VanessaE yeah 13:37 VanessaE it has to be spread-out in order to not lag 13:37 MTDiscord I don't think it pays to be too "polite" and split work up too much across multiple ticks, unless you pay attention to dtime and consider that other mods may not be as polite back to you. If dtime creeps up but you still do only a tiny amount of work your queue can end up relatively starved... 13:37 VanessaE I do. 13:37 VanessaE https://gitlab.com/VanessaE/biome_lib/-/blob/master/init.lua#L490 13:37 VanessaE as you can see, it can be configured to run multiple steps per tick 13:37 MTDiscord Feels like this function could have been more readable as a coro. 13:38 MTDiscord Nothing exposes every little code readability problem quite like trying to read it on a phone screen. 13:38 VanessaE haha 13:38 VanessaE I can't do too much about that :S 13:38 VanessaE :D 13:39 MTDiscord Oh wait, your dtime check is backwards. 13:39 MTDiscord If dtime is larger, you do less work, not more 13:41 MTDiscord You do a fixed number of entries per step but if some other mod is eating up time, biome_lib basically freezes itself until that mod quiesces, but there's no guarantee it ever will. 13:42 MTDiscord I guess that kind of behavior shouldn't be unexpected considering how you run your servers... 13:42 VanessaE huh 13:43 VanessaE how should I do it then? the way dtime works has always confused me 13:43 VanessaE I always thought it was the real amount of time the last server tick used 13:44 VanessaE i.e. if it's getting higher, mods are (in general) sucking-up server time 13:44 celeron55 it's the amount of time since the last tick (or, well, i guess MT usually calls them steps) 13:45 celeron55 if it increases from the set amount, that means there's overload 13:45 VanessaE the idea in my case being that biome_lib should step out of the way if ticks are taking too long 13:45 MTDiscord Yes. It represents the amount of time you need to catch up on. Unfortunately, managing it is fairly complex. If you're too polite then you can starve, but if you're too aggressive you can cause a meltdown. 13:46 MTDiscord NodeCore optics are supposed to run on a fixed 12Hz timing. I do as many iterations as dtime tells me I should ... but I have a hard limit to ensure optics themselves don't cause a meltdown. 13:46 celeron55 i'd say it's best to neither try to catch up nor step out of the way 13:47 VanessaE so in the case of this code, perhaps I should just drop the dtime check and keep the run ratio and per-step features 13:47 MTDiscord Also stepping out of the way doesn't necessarily help since the expensive tick has already happened anyway... 13:48 VanessaE good point. 13:48 MTDiscord Does the run ratio really help all that much either? Might as well just run every tick, if there is work to do ... You've already done a lot to keep the per-tick work level down. 13:49 MTDiscord Just spreading the workload out across multiple ticks is already a great lag-killer, and it's hard to substantively improve on that without risking doing more harm than good. 13:49 celeron55 it seems the amount of work to be done is already set and what needs to be done is smooth it out as much as possible, but never to not do it 13:50 VanessaE I put it there for really slow machines, though on my PC and servers, I don't use it 13:50 MTDiscord On really slow machines, biome_lib would be so polite to other mods that gameplay would be impacted and players could see uninitialized blocks. 13:51 MTDiscord Though I don't think that's the cause of what we're seeing... 13:51 VanessaE nopw 13:51 MTDiscord Btw, I don't see where you're doing your neighboring block checks... 13:51 VanessaE even when it comes up with these empty blocks, the block list does eventually run dry once I stop running around 13:52 VanessaE oh that, I split the neighboring-blocks check into a different branch 13:52 VanessaE to be explored later 13:53 VanessaE so this'll be the new globalstep callback then: https://pastebin.com/rEfX6c84 13:55 VanessaE celeron55: did you see the root prob we're trying to solve though? 13:57 celeron55 i did notice, don't have any insight to it 13:57 VanessaE ok. 13:58 celeron55 but given how your system works, maybe the mapgen sometimes overwrites your blocks? 13:58 VanessaE that's what I'm wondering 13:59 celeron55 as far as i'm aware it will read and when it's done then overwrite the bordering mapblocks of the generated chunk 13:59 VanessaE but if it's overwriting blocks altered *after* they're emerged, isn't that a bug? 14:00 celeron55 decorations and some other stuff extend over the chunk, and they're allowed one mapblock 14:01 celeron55 and as it's in a different thread using its own buffer it doesn't have any idea what's happening meanwhile 14:01 celeron55 unless i fail to remember a way it was made to avoid this 14:02 VanessaE I guess I need to call upon the mapgen to always emerge all 6 neighbors of a given chunk before logging it? 14:02 celeron55 so effectively the border mapblocks are half-generated until the neighboring chunk is generated 14:02 VanessaE (using the emerge area call) 14:03 celeron55 but i don't think that status is available to anything other than the emerge thread 14:04 celeron55 as a workaround, maybe, but ideally there would be some kind of events or flags or something to allow knowing when something is actually fully done 14:04 celeron55 also still i'm still not sure i remember right 14:05 celeron55 maybe it's already handled smehow 14:05 celeron55 +o 14:05 VanessaE I remember there being a one-block "grey area" around a chunk for mud flow but that's pretty old now, and I don't recall that deleting/overwriting the border areas 14:23 VanessaE if I call `minetest.emerge_area()`, will it in turn trigger the regular `on_generated()`? 14:24 sfan5 if the chunk(s) do not already exist it will trigger the mapgen and everything it usually does 14:28 MTDiscord You have to accept that some blocks must be able to be in a half-generated state though, because if you try to emerge the neighboring blocks whenever these happen, you'll just effectively recursively queue up emergence of the entire universe. 14:28 VanessaE but how can I *detect* this "half-generated" state? 14:36 MTDiscord Haha 14:36 MTDiscord I'd be surprised if you actually can... 14:36 MTDiscord You can tell when a block MIGHT be half-generated based on the fact that at least one neighbor isn't emerged 14:37 VanessaE I tried that. 14:37 MTDiscord if the neighbor is emerged it shouldn't be able to be in a half-assed state 14:37 VanessaE check the extra-map-checks branch, you can see what I did. 14:37 VanessaE it didn't work 14:37 VanessaE the check for neighbors happens too quickly 14:38 MTDiscord about the only thing I can think of beyond that is to (1) impose an artificial delay to give the worker thread a chance to win the race, and then (2) pray that whatever arbitrary value you picked is sufficient to work but not so bad that a fast-flying player could see behind the curtain. 14:38 VanessaE it's like the core mapgen falls "behind" the Lua side 14:38 MTDiscord I absolutely abhor fine-tuning problems, but without some way to properly check the block state... 14:39 MTDiscord unfortunately I don't think that blocks themselves are aware of half-generated-ness, nor does the engine likely actually track this; it's probably just an emergent state. 14:41 MTDiscord I really don't see how my neighbor mapblock check should have failed entirely, though ... unless the game thinks that those mapblocks are generated and all ignores/air because they were included in the boundary region of the mapblock you're checking. 14:41 MTDiscord Maybe (1) either check for any ignores instead of just nil, or (2) try extending the check to 2 mapblocks around. 14:42 MTDiscord (2) might be the most reliable given that hypothesis. 14:42 VanessaE minetest.get_node_or_nil() returns nil if it gets an "ignore" node. 14:42 MTDiscord No it doesn't 14:42 VanessaE no?> 14:42 MTDiscord it returns nil if the node is not loaded 14:42 VanessaE https://github.com/minetest/minetest/blob/master/doc/lua_api.txt#L4875 14:42 VanessaE yeah, not loaded == ignore 14:43 MTDiscord no, ignores are not just for not loaded areas 14:43 MTDiscord they're also for areas that have ignores 14:43 MTDiscord you can have ignores explicitly set in an actually-loaded mapblock 14:43 VanessaE wat 14:44 VanessaE https://github.com/minetest/minetest/blob/master/doc/lua_api.txt#L4871-L4876 this disagrees with you 14:44 MTDiscord The thought here is that this might be how the engine handles blocks that aren't generated yet but already have structures intruding into them from neighboring mapgen. 14:44 MTDiscord Huh, never noticed there was a bulk_set_node call ... but I don't see what that has to do with ignores. 14:46 VanessaE that might be something worth looking into later. 14:47 MTDiscord Well, you can easily test whether get_node_or_nil ever returns an ignore. You could even just patch the API func itself to throw a log message. 14:51 MTDiscord Besides, what would be the point of having both get_node with ignores and get_node_or_nil if they were exactly equal? 14:52 VanessaE I imagine so that you can do stuff like, `if minetest.get_node_or_nil(pos) then...` without checking for a node name 14:53 VanessaE i.e. a bool versus string test 14:53 MTDiscord ^ 14:53 VanessaE I guess string handling/comparisons are slightly slower than bools in Lua 14:53 MTDiscord It's more about convenience I'd say 14:54 MTDiscord Okay, I just did the empirical test and proved that get_node_or_nil can return an ignore nodes 14:54 MTDiscord String comparison in Lua is constant time usually 14:54 MTDiscord So all debate about why they would do a thing they didn't do is moot 14:54 VanessaE Warr: wait what? 14:54 MTDiscord ah yes, minetest at it again... 14:54 MTDiscord I think the problem here is that mapblocks can't be in a half-generated state 14:54 MTDiscord they can actually be in a 1/3-generated and 2/3-generated state 14:55 MTDiscord Yeah, its not a guaranteed function iirc 14:55 MTDiscord 2/3 meaning they have terrain but not necessarily structures from neighbors, and 1/3 meaning they have ONLY structures from neighbors but no terrain yet. 14:55 MTDiscord get_node_or_nil exists to differentiate between nodes that aren't loaded and nodes that are ignores, i.e. not initialized 14:56 MTDiscord and apparently nodes can be loaded and initialized in either order 14:56 MTDiscord I'm not sure why mapblocks are allowed to be emerged when they aren't finished yet, but ? 14:57 VanessaE so you're suggesting, what, check if or_nil().name == "ignore" and call that "not fully emerged"? 14:57 MTDiscord Expanding the nil check to also include ignores (or really just using get_node and ignores) would SORTA work, unless you happened to unluckily pick a node to test in the neighboring mapblock that had content in it spilled over from the loaded area. You could check mulitple, but there's still no absolute guarantee and that's expensive. 14:57 MTDiscord So better would be to just test 2 mapblocks away, since I don't think the spillover region is allowed to be larger than 1 mapblock... 14:58 MTDiscord _or_nil() ~= nil and _or_nil().name == "ignore" could test for at least this half-emerged case, but it'd be too spotty to rely on; spottiness is itself what we're trying to fix here, so that'd be a mitigation, not a fix. I say we shoot for a fix first, and fall back to a mitigation if we can't actually fix it. 14:59 MTDiscord just check 32 nodes away instead of 16 and retest. Code change should be simple, and there's a chance it will work. 14:59 MTDiscord If it DOES work, then we can stand around debating why it works and whether it reasonably should, but at least we can do it with biome_lib being likely free from one more bug :-) 15:00 VanessaE haha 15:00 MTDiscord If you hate hacky workarounds then the minetest universe has some bad news for you :-/ 15:00 VanessaE hacky workarounds are fine if they're reliable and cheap. 15:01 VanessaE (cheap == doesn't gulp down CPU, in this case 15:01 VanessaE ) 15:01 MTDiscord You always need some kinda hacky stuff when you're dealing with edge-cases. The only real alternative is to avoid the edge cases, and sometimes all that really does is move the edge inwards. 15:02 MTDiscord In effect we're sort of doing that, because you now won't try to run the mapblocks on the actual edge, creating a new edge of engine-initialized-but-not-biomelib-initialized blocks 15:02 MTDiscord they'll be farther in from the mapgen boundary, but hopefully 1 more mapblock in isn't enough to affect the experience 15:02 MTDiscord On the plus side, you should have eventual consistency restored. 15:08 VanessaE well c55 did say earlier that the maximum overlap is 1 mapblock, so poking at the middle of a chunk is guaranteed to never be "half emerged" 15:17 VanessaE there's another corner case though: a chunk out at the edge of the player distance limit. generated in full, but then the next one out past is not generated at all. suppose the fully-generated one has already been "plantified". what happens to it when the player walks forward, and that slightly more distant mapblock finally generates? 15:17 VanessaE more distant chunk I mean. 15:18 VanessaE or rather, what happens to the overlap area 15:33 VanessaE Warr: https://pastebin.com/vs8E760R something like this? (scroll to line 427 of the paste) 15:35 VanessaE and... fail. 15:38 VanessaE well not utter fail 15:39 VanessaE what was long strips of unpopulated terrain is now a single mapblock here or there. and the debug output shows it IS detecting not-loaded neighbors 15:41 MTDiscord Is it possible that this issue doesn't exist for users of a VoxelManip because that forces Minetest to emerge the area into a buffer? 15:42 MTDiscord Maybe either call emerge, or use the LVM equivalents of get_ and set_node? 15:42 VanessaE mmmh 15:43 * VanessaE looks at celeron55 15:44 MTDiscord https://github.com/minetest/minetest/blob/master/doc/lua_api.txt#L4050 15:46 MTDiscord sees set_lighting() 15:48 MTDiscord Does this set a temporary light amount, or an actual fixed amount for both day and night respectively? 15:49 VanessaE I assume it's temporary, until something comes along and alters the map 15:53 MTDiscord So, it looks like it lets you set the current natural lighting for an area, but only within an on_generated 15:55 MTDiscord But wouldn't that just set the param1? If so, why does this function exist and why is it limited to on_generated? 15:55 MTDiscord You can set the param1 with a different function from anywhere 15:56 MTDiscord Presumably if you use the mapgen vmanip, you can avoid this problem, but some of the things VaE is trying to do in her API might be too annoying to do in VM-land. 15:57 MTDiscord I think the mapgen voxelmanip is probably the one owned by the mg thread, so manipulations done on it are not subject to the same race cond. 15:57 MTDiscord It sounds like we've got the edge cases cleaned up, though, and we're now down to corner cases... 15:57 MTDiscord So, I said, check the 6-neigborhood ... I think we MAY have to do the 26 15:57 VanessaE 6 is expensive enough 15:57 VanessaE look at the paste 15:58 MTDiscord well, you're actually paying for 12 15:58 VanessaE 26 would slow things down so much it would pretty well kill the benefits of spreading-out the load. 15:58 MTDiscord because you're retrieving the nodes redundantly 15:58 MTDiscord I don't think it'd be a big change, since VMs have equivalents to get and set_node and the whole thing will be loaded. 15:58 MTDiscord also, you're talking about 20 node checks per mapblock; that's not expensive 15:59 MTDiscord ABM neighbor checks do 26 node checks per node, that's expensive. 26 per mapblock is just under 1/1000 as expensive 15:59 MTDiscord If it'd avoid the problem entirely it might be worth a shot 15:59 MTDiscord But thats just my opinion :-) 15:59 VanessaE Ben: but what happens when all actions have been done on a block and it's time to write it out, but in the meantime the user has already started digging/building in the hot area? bang, bye bye house. 15:59 VanessaE (or whatever) 15:59 MTDiscord Plus, we're talking about mapgen cases. You can afford a lot more expense to get mapgen right, because you only mapgen once per block, so it's both a one-time cost, and you also have more impetus to get it right because any mistakes will last a long time 16:00 VanessaE well it isn't just once. 16:00 VanessaE it's continuous 16:00 VanessaE if a block appears to have unloaded neighbors, it has to be moved to the end of the queue 16:00 VanessaE which means all the blocks out at the edge of the generated area are gonna be doing that 16:01 MTDiscord for dx = -32, 32, 32 do for dy = -32, 32, 32 do for dz = -32, 32 32 do if minetest.get_node({x = pos.x + dx, y = pos.y + dy, z = pos.z + dz}).name == "ignore" then return end end end end return true 16:01 MTDiscord Ve: No? If you're doing this in on_generated the whole thing will still be ignore until it finishes 16:01 VanessaE checking for not-emerged neighbors over and over and repeatedly being moved to the end of the queue 16:01 VanessaE Ben: nope. we're getting "half-emerged" blocks due to chunk overlap[ 16:01 VanessaE the engine is passing me blocks that are not safe to write to 16:02 MTDiscord Thats not what I was referring to 16:02 MTDiscord We are arguing about microseconds for a thing that may or may not actually fix your problem. First find out if it fixes the problem. Then find out if the microseconds you pay for it are worth it. 16:02 VanessaE Ben: maybe not but that's what Warr and I'm dealing with and what Warr is trying to offer help on :) 16:03 MTDiscord Sometimes making shit work just comes at a cost, and you have to decide whether making it work right is worth the cost. Maybe there's an optimization you can find later, maybe not. 16:03 VanessaE Warr: did you see the paste though? 16:03 MTDiscord For one thing, if you want an easy optimization, just MRU-cache all known-loaded mapblocks and you can do the lookup there instead of hitting the API. 16:03 MTDiscord if you mean vs8E760R then yes, I looked at it 16:03 VanessaE ok 16:03 MTDiscord you're checking 12 neighbors (actually 6 neighbors twice each) 16:04 VanessaE Warr: yeah. and it DID help 16:04 MTDiscord yeah, you're down from edges to just corners 16:04 MTDiscord that's why I think the 26 neighbors will help 16:04 VanessaE ok 16:04 MTDiscord take a look at the code I pasted 16:04 VanessaE I saw 16:04 MTDiscord er, wrote I guess 16:04 VanessaE just trying to figure out if there's a cleaner way is all 16:04 MTDiscord Unrolling the loops might make it a tad faster on PUC lua, though I expect the different would be immeasurable on JIT. 16:05 MTDiscord if by "cleaner" you mean "more clever" / "optimized" then maybe. If you mean conceptually simpler, probably not. 16:05 VanessaE I mean less kludgy :) 16:06 MTDiscord Do you mean less kludgey conceptually or do you mean less ugly code-wise? 16:06 MTDiscord far as I know, nested loops are about the cleanest way to do it starting from zero. 16:06 MTDiscord You could pre-build a table of the 26-neighborhood and walk it once, and that'd be locally cleaner but globally mesiser. 16:06 MTDiscord unless you get some kind of reuse of the 26 neighbors 16:07 MTDiscord I actually also check the center pos because I figured the extra code complexity of checking for that special case was not worth the theoretical perf savings (you incur overhead for the check on every iteration to compensate for 1 fewer API call, so that shrinks the impact) but you can add that in if you want; it COULD be helpful. 16:08 MTDiscord Anyway, find a shitty solution first before you worry about finding a good one. There's no value in a good non-solution, so movement solution-ward is more important than movement good-ward. 16:09 VanessaE good point. 16:09 MTDiscord Once you confirm that the 26 neighbor scan works, one thing you can try is trimming it down to just the 8 corners. That should work assuming that mapchunks are always more than 1 mapblock wide, and you already make that assumption very strongly elsewhere in the code. 16:09 sfan5 does it really matter if a few mapblocks are missing plants? 16:10 sfan5 because personally I'm sure I'd have not bothered with this 16:10 MTDiscord It matters a hell of a lot more than whether we spend 100 microseconds vs 110. 16:10 MTDiscord it's quite noticeable when the plants are dense. 16:10 MTDiscord If people weren't already bothered by it then it'dn't've been reported in the first place. 16:11 VanessaE sfan5: it matters because it means either the engine is broken or my code is, though in this case it's the engine. it should *not* be returning blocks (==triggering on_generated()) for areas that have overlap until after the overlap has been satisfied. 16:11 MTDiscord I wouldn't go so far as to call this a "bug" in the engine; it's more like UB tbh 16:12 sfan5 I'm not saying it's not a bug, just question the tradeoff of unfixed end result <-> time spent on workarounds 16:12 sfan5 questioning* 16:12 VanessaE it's not just one mapblock here or there though 16:12 MTDiscord I think MT has a few things like that, where behavior at boundary cases is just sort of not guaranteed by the engine, and any assumptions we make about it behaving the way we think it should aren't actually reliable... 16:12 VanessaE https://github.com/minetest/minetest/issues/11134 16:12 VanessaE take a look at the first screenshot 16:12 VanessaE this is what happens without any workarounds. 16:13 MTDiscord Anyway, for the 8 corner checks, you can just make the step size of my loops 64 instead of 32 and that will hit the 8 corners already. Since it's only 8, it'd also be feasible to hand-unroll if that's your thing. 16:14 MTDiscord Frankly it's easy to justify spending a ton of effort fixing minor problems when you consider how much time this community in general pours into fixing non-problems. 16:14 VanessaE I wonder... does the overlap extend vertically? 16:14 VanessaE or could I just check the 8 X/Z neighbors? 16:14 MTDiscord If you mean "does MT treat some dimensions specially so that I don't need to worry about the Y axis" then I'd say no, don't rely on that assumption. 16:15 MTDiscord If something can happen on X or Z then assume it can happen on all 3 axes. 16:15 VanessaE no, I mean the mud flow feature that originally gave rise to this overlap thing 16:16 MTDiscord Whether the feature has an effect on the Y axis, I'm pretty sure the underlying mechanisms don't prevent overlap on the Y axis. Trees can other schematics/decorations can also cross the Y boundaries. 16:16 VanessaE btw, checking all 26 (well 27) does appear to completely eliminate the empty blocks. 16:16 MTDiscord Mudflow predates current mapgens, but current mapgens offer some features that almost certainly rely on the behavior, regardless of whether the reason it was originally created for is even valid or not anymore. 16:17 VanessaE https://pastebin.com/macWZYAn there's the working code 16:17 MTDiscord Try the 8 corners. Bump the loop step from 32 to 64. If that works too, that's already a big optimization and gets you back close to where you were with 6 (only actually working now) 16:17 VanessaE (line 427) 16:17 MTDiscord If 8 doesn't work though then I posit that even the 26 is perfectly livable. 16:18 MTDiscord If you start measuring lagspikes with this code that you didn't see before, then we can revisit it. 16:18 MTDiscord Also, try either (1) using get_node, or (2) storing get_node_or_nil in a local, so you only have to call the API once. 16:19 MTDiscord Lua's official optimization guide suggests that often recalculating something can be more efficient than storing it, but I think for something like an actual function call, esp. an API call, it's probably better to store it in a local... 16:20 MTDiscord I dunno if get_node vs. get_node_or_nil has any meaningful performance difference, but heuristically I'd expect them to be close enough to the same that it's very unlikely to matter, and get_node already collapses different kinds of ignores into a single check. 16:22 MTDiscord https://pastebin.com/ttKkSPKi is the "call get_node_or_nil only once" route I'm thinking 16:22 VanessaE yeah, I'm on it 16:23 MTDiscord If it still works, we can see if the unrolling route is better ... I guess whether it's "cleaner" or not depends a bit on the overall style you're going for in the code... 16:24 MTDiscord I tend to prefer that code is closer to its conceptual form and has as little baked-in optimization as tolerable, but there are places where it makes enough difference that that ideal would come at too high a cost. 16:25 VanessaE 8 corners (plus center) seems to work perfectly 16:25 VanessaE https://pastebin.com/6wC4WZe0 16:25 MTDiscord This sort of check only really works for detecting "mapgen done" cases in specific cases where mapgen is done contiguously as a function of player exploration; if you just try to manually emerge stuff or use force-loads, probably it could result in some areas that stay queued forever, until a player explores the surrounding areas. 16:27 MTDiscord It looks good to me. I think assuming you don't discover any new problems in your testing, it's release-worthy, and would be good to get out to anyone using it ASAP so at least the known issue is dealt with. 16:27 VanessaE *nod* 16:28 MTDiscord It's possible it could be better optimized but I kinda doubt that optimizations could have enough of an impact to make it worth holding up a release, or even making a hotfix release; probably better to just worry about optimizations if you do an "optimization pass" on it later. 16:28 MTDiscord Like, I mean, across the whole mod, using the profiler and such... 16:28 VanessaE there's one more optimization I need to add though - a flag to pause processing the block log if a mapblock comes up in it that it's seen before but had moved to the end, until the user triggers generation of more new terrain 16:29 VanessaE which would stop it from continuously churning when the only things left in the log are the blocks out at the edges of the terrain 16:30 MTDiscord Ah, what I do for that sort of thing is usually move the queue into a "batch" variable, recreate the queue as an empty list, and then process through the batch, so that things can be added to the queue without affecting the batch; this also allows me to avoid having to use the possibly-expensive table.remove call, since I can just walk the frozen batch array by index. 16:31 MTDiscord Normally I do this for stuff where the batch doesn't live past the end of a function call (i.e. I don't have to store it outside of function-locals) but you could easily store the batch somewhere and just make sure you end the current-tick processing when the batch is empty, i.e. only capture a batch of work at the very beginning of a tick if the current batch is empty. 16:31 MTDiscord Depending on how you handle it, it's sort of a wash anyway though. 16:31 MTDiscord Like, there are a bunch of ways you can do this which are probably all about equivalent. 16:32 MTDiscord If you have only one block in the queue and you try to do 100 blocks per tick, and you end up just processing that 1 block 100 times, yeah, that'd feel really silly :-) 16:32 VanessaE heh 16:32 VanessaE it's not 100 blocks per tick tho 16:32 VanessaE it's 100 *actions* per tick 16:32 VanessaE or whatever number 16:33 VanessaE (which under Dreambuilder, with all the flora its mods add, would be roughly 2 complete mapblocks per tick, as the action list is 50-some items long) 16:37 VanessaE another I'd like to do, but it would be kinda expensive, would be to periodically [re-]sort the map block log (say once every 5s), to put map blocks closest to the player(s) early in the list, so that they populate first before the more distant ones 16:40 MTDiscord Interesting; that implies that as you add more and more mods that depend on biome_lib, the farther behind it's possible to make the processing queue become... 16:40 VanessaE yes 16:40 VanessaE but you can compensate for that by increasing the actions per tick setting 16:40 VanessaE up to the limit of your CPU anwyay 16:41 VanessaE my old PC here can churn through up to around 500 actions per tick in singleplayer 16:41 VanessaE I figure each person/admin should set that variable to whatever works best for their hardware and mod collection 16:50 MTDiscord Sure, it can be fine-tuned. I just think that if you don't explicitly tune a system, it should have sane defaults, and if "sane" is dependent on circumstances, then the defaults ideally should be too. 16:52 VanessaE I aimed for that, actually. the default of 100 actions/tick is about the speed it used to be when the dtime stuff was all screwed up until a couple days ago 16:52 VanessaE (it was a mess) 16:52 MTDiscord Yeah, finding good adaptive optimum defaults is sometimes pretty hard. It's just beautiful when you can make a system work that way. 16:52 VanessaE though since to the user it feels much faster than before, when it used to process an entire chunk at a time 16:54 MTDiscord I found one little microoptimization to walking a vmanip that helped me out a ton on cutting down mapgen latency. Much of the time I'm stuck doing chunk-at-a-time because I'm doing everything inside on_generated. 16:54 * VanessaE ponders how to implement the pause/resume idea 16:54 MTDiscord There's that "area" thing that lets you calculate offset within the content data based on x/y/z coords. The trick is that I do that only once per linear "walk" through the volume 16:54 VanessaE if chunk-at-a-time is a prob, maybe you could do what I did and split the chunk into blocks? 16:55 MTDiscord If I did that then I'd have to suffer through all the same half-loaded-ness and asynchrony shit you've had to put up with, and I'm not sure if the cost is worth it to me. 16:55 VanessaE haha no 16:55 MTDiscord It's a very small piece of all I have to maintain, and tbh I don't think mapgenning happens often enough to bother me. 16:56 MTDiscord If I added a lot of biome diversity to my game, then that could happen, but I sorta don't like the idea of categorical biomes anyway. 16:56 VanessaE the half-loaded-ness was happening before, in chunk-at-a-time mode too 16:56 VanessaE ok that's our official term for MT mapgen. half-loaded-ness :D 16:57 MTDiscord half-assed-ness might be better :-) 16:57 VanessaE lol 16:57 MTDiscord esp. since it's not really quite "half" 16:57 MTDiscord It feels more like "schematic spillover but no terrain" is like 1/3 and "terrain but no schematic spillover" is like 2/3... 16:57 MTDiscord or "mud flow" maybe, though I'm not sure if that's still a thing...? 16:58 MTDiscord but schematic sure as hell are... 16:59 MTDiscord By far the biggest cost I pay in mapgen is when I have to walk the entire vmanip and do manipulations on every node in it, especially when I have to do neighbor checks. In NodeCore I have a rule that Lode Ore is not allowed to be exposed to air, so I have to detect and convert these, and that includes a neighbor check against every matching node in the chunk. That's probably peak expensive 16:59 MTDiscord It can be done in at least reasonable time though, at least. I'm not sure if breaking it down would be worth it anyway. 17:00 MTDiscord Often you have to make latency/throughput trade-offs, and doing it all in the mapgen callback optimizes throughput at the cost of latency. Sure, you may get lagspikes while exploring, but you can maintain a higher average movement rate that way. 17:02 MTDiscord Performance/scalability trade-offs seem to be one of the more counterintuitive trade-offs for a lot of devs, apparently; I've seen a few 10+-year professional engineers who were puzzled by the distinction. It might be a thing you'll have to consider though when you think about what kind of "optimized" you're shooting for. 17:03 VanessaE well the way my code is written, you could set the queue run ratio to like, -6000 or so, and you'd get the same effect -- everything for a given mapblock would get done in one big glob before the next one is even recorded into the log 17:05 VanessaE (that's roughly a whole chunk's worth of content in DB) 17:15 VanessaE Warr: it appears to be sufficient to do check if minetest.get_node(pos).name == "ignore", without the nil check 17:16 VanessaE so now we know, "ignore" is half-loaded 17:19 VanessaE but nil can be half-loaded too. 17:20 MTDiscord It is a little bit humorous how far you have to check around a thing for at-least-partial-loadedness before you can determine whether a thing is itself almost-certainly-fully-loaded or not yet... 17:21 VanessaE heh 17:22 MTDiscord and yeah, we're at the "almost certainly" standard, not absolutely certain, because you can never really be absolutely certain when you can only really verify empirically but not exhaustively, and that you're in that situation in the first place because of sorta-nonsensical behavior or a general lack of guarantees... 17:23 MTDiscord We have to test 2 mapblocks out because currently we rely on finding a definitely not initialized at all mapblock for the ignore check to always work. 17:31 VanessaE Warr: I added a comment to https://github.com/minetest/minetest/issues/11134 , care to chime in? 17:34 MTDiscord I guess the core issue to me is the fact that parts of mapgen are happening in a background thread that may take a snapshot of map data (to be modified by the mapgen thread) and then write those mapblocks back with modifications based on that snapshot, while apparently some of the same mapblocks may have been made available to the lua state for things like set_node stuff in the meantime, and thus changes made in the main game thread 17:34 MTDiscord are not honored by the mapgen thread. 17:35 MTDiscord It could be possible for them to avoid this problem by just merging the mapblocks (e.g. read from the game-state one wherever the mapgen one has ignores) during the "apply changes from mg thread" step. It could be a bit expensive, maybe (if they're doing a zero-copy reference replacement on the mapblocks) but it might be necessary. It could be possible to track a dirty-flag on both sides or something to determine if this actually 17:35 MTDiscord needs to be done so we don't have to pay the price in cases where it's not relevant. 17:36 VanessaE I meant add your remarks to the issue :) 17:36 MTDiscord A lot of MTE's problem is the way they give you a Phillips screwdriver, but they only test it on steel screws, and as soon as you try using it on a brass screw, they tell you to see if you can reproduce the problem on a steel one instead. 17:37 MTDiscord Eh, I guess I could add those remarks... 17:37 VanessaE haha 17:37 MTDiscord Not sure if they're well-enough informed. 17:38 MTDiscord I guess I sorta just don't want to have to try digging into the engine code itself and verify what's going on. It's just what my intuition is telling me is happening, and at least what we've observed is consistent with that intuitive understanding, but it's not necessarily exclusive. 17:40 VanessaE only question now is, since "not n or n.name == "ignore" means a half-emerged block, how do I check if a block is really, truly, honest-to-G*d not emerged at all 17:40 VanessaE (so that mapgen can finish out to the edges of the player's view and the block log can run dry) 17:42 MTDiscord I suspect you can't 17:43 VanessaE didn't think so. 17:43 MTDiscord Your queue may in fact never actually run dry 17:43 MTDiscord There might be ways you could optimize away having to run the queue every damn cycle, like doing some kind of partitioning with an "are there at least players reasonably close to this" check or something. 17:44 VanessaE yeah 17:44 MTDiscord Also, in the extremal case where you've explored literally the entire world, you'd have to add explicit special checks for mapgen_limit 17:44 VanessaE that gets back to what I was musing about earlier 17:44 MTDiscord except that as far as I can tell, the actual math for mapgen_limit is intractable ? 17:44 MTDiscord also, mapgen_limit can be raised later, so you don't necessarily want to let those mapblocks leave the queue either... 17:45 VanessaE if I could somehow sort the block log every now and then, putting them in order by distance to any player 17:45 MTDiscord I guess the biggest issue to worry about is that your queue will grow in proportion to the surface area of the explored region, so it could become quite large eventually 17:46 VanessaE nah 17:46 VanessaE not a big worry there 17:46 VanessaE it only needs a couple dozen bytes per mapblock 17:46 MTDiscord eh, well, it worries me a bit, especially since you have to manage persisting this massive queue. 17:47 MTDiscord Assuming you explore the entire world uniformly, you can have up to 100M mapblocks in the queue, representing the very edges of the map. Those would actually never get fully biome_lib_bed. 17:47 VanessaE true 17:48 MTDiscord You could create a pathological case that'd probably have orders of magnitude more, but that's maybe not worth worrying about realistically. 17:48 VanessaE but that'd still be only a couple of gigs, and it would take weeks to explore the world to that degree 17:49 MTDiscord1 A couple of GB is kinda a lot to potentially be writing back to mod_storage or whatever every 5.3 seconds... 17:50 MTDiscord tbh it feels like maybe an inside-out approach would make more sense, like having on_generated replace one node within the mapblock with a special node that indicates "initialization not done" so that the state of mapblocks is guaranteed durable, then having logic to scan and find these and do the initialization as needed. 17:51 MTDiscord I wonder if node meta is accessible during mapgen? I've really never tried that, and I'd be worried about its threadsafeness. 17:51 VanessaE oh that's a funky one 17:52 MTDiscord Also it's a pain to try to replace a node in a mapblock with another node because you'd want to be able to replace the original node afterwards, and there's no good way to store the original node in a way you can definitely recover it without metadata, I think. 17:53 VanessaE if you write to the map FAST enough, say placing a grid of fenceposts or torches or something else thin and easy to see past, you'll get corrupted data under them, but only in the nodes immediately under -- i.e. the corruption will follow the same grid pattern, while at a larger scale following the pattern of those formerly-bare strips of terrain we just worked around 17:53 MTDiscord Basically I'm trying to think if there's a way to do this ACID-transactionally under the assumption that the underlying thing mechanism always either saves or doesn't save a set of mapblock changes. 17:53 VanessaE like, grass will inexplicably turn to stone or gravel, under the fence/torch/etc 17:54 MTDiscord Wait, what, is this another different bug? 17:54 VanessaE it's some oddity I ran into a few days ago while banging my head against the bare mapblocks issue we just solved 17:54 VanessaE I just chalked it up to MT being stupid ;) 17:55 MTDiscord Something that happens on the fringe, or can happen anywhere? 17:55 VanessaE at the time, I didn't see a pattern, but after today's discussion, it was most definitely on chunk edges 17:56 VanessaE and you'd see complaints in the console about node meta not being found or some such 17:56 MTDiscord huh, neighbor data corruption is ... not something I have a theory about, even... 17:56 MTDiscord I've seen a lot of console complaints about shit failing to load, like node timers not deserializing right... 17:57 VanessaE yeah, this was nothing important. I might dink around later and see if I can reproduce it just to get you a screenshot 17:57 MTDiscord That's one of the reasons why nearly every node timer I use needs to have a "patrol" ABM that makes sure it can't be lost. I expect my worlds to "self-heal" and never allow themselves to become stuck indefinitely in a state where they're not applying the behavior they're supposed to be. 18:07 sfan5 VanessaE: you don't have num_emerge_threads > 1 do you? 18:08 VanessaE nope 18:08 sfan5 good to know 18:08 VanessaE it's not set at all, so default 1 18:08 VanessaE wouldn't do much good anyway since Lua hooks into it 18:09 sfan5 true 18:09 MTDiscord I assume you can't set it to 0 and force synchronous emerges. That'd "solve" your mapgen problem, probably, at the expense of causing others... 18:10 MTDiscord Though if you have a 4-core server that's running 4 relatively steadily-busy worlds on it, then the multithreading stuff might not be as valuable to you as you might think... 18:11 MTDiscord If I'm looking for the basics, like just some tall grass / flowers / trees, i.e. use-cases that the standard decoration API already offers, is there any advantage to using biome_lib for it instead? Or was biome_lib created before that stuff was available standard? 18:13 MTDiscord Heh, I guess I HOPE there's no reason I should want to use it in that case anyway, since it looks like it'd be license-incompatible with pretty much everything I release anyway. 18:31 VanessaE it was created long before decorations existed, but imho its API is more flexible. 18:41 Jordach nothing like an afternoon of porting code 20:10 VanessaE Warr1024: https://gitlab.com/VanessaE/biome_lib/-/commit/e346fd599f3877a89618c7b811fea3ae6d7e0117 20:17 VanessaE I still feel like handling of the blocks that can't yet be populated is a little.. kludgy (particularly when it comes to re-adding them to the main list to try hitting them again) but this is 10x better than before :) 20:17 VanessaE and I'm wiped-out. 20:17 VanessaE I/O error reading /dev/brain: device not ready 20:20 Krock attempting to unmount /dev/brain