Time Nick Message 01:27 sfan5 v-rob: a mapblock cannot have more than 4096 nodes hence it will never need more than 16-bit ids 01:29 v-rob Oh yeah, duh. That makes sense 01:29 sfan5 it's kind of obvious once you get it 01:37 VanessaE I've forgotten.. apart from the time and effort that'd be needed, what stands in the way of it? 01:38 sfan5 conving devs that this is a good idea is kind of a point 01:38 VanessaE I kinda file that into "time and effort" :) 01:39 VanessaE i.e. it's not a technical reason :) 01:42 VanessaE that being said, you know I think it's a good idea, plus you'd be upgrading the range one last time (i.e. likely it'll never again be needed). but the downside to that is that frankly the client is already pretty slow to start-up if there's an assload of node defs to process. 01:42 VanessaE (I don't need it, even DB only has like 20k node defs, but SOMEONE will eventually) 01:46 MTDiscord there are already servers that walk the line with it 01:46 VanessaE I think of it as a "just go for overkill" opportunity, since it's not the engine's job to prevent the user or admin doing something moronic like registering a million superfluous nodes. 01:47 VanessaE (though it IS the engine's job to provide features that help the modder avoid having to do so) 01:50 MTDiscord if something like https://github.com/minetest/minetest/issues/9193 was addressed/had a PR for it, you could cut way down on node registrations used for stuff like farming, machines with active states 09:37 MTDiscord I'd like it if we could arbitrarily override most node def fields using metadata 09:38 MTDiscord For example tiles: You could just override a tile if you wanted signs 09:39 MTDiscord The problem is just that the flat k/v storage of meta and the Lua table format of node defs don't go together well... 09:48 sfan5 the problem is that it complicates the mesh building because suddenly you have to consider metadata and comparing node IDs is no longer of any use 14:51 MTDiscord At that point, why not just optimize for that entirely? If you're going to have in-map definitions you might as well take the concept and run with it IMO 14:53 hecks do away with CIDs and have immediate-mode node defs of some sort? 14:53 hecks like per-block palettes 14:53 hecks oh but i have a better idea 14:53 hecks map data layers 14:54 MTDiscord Honestly I'm still partial to predefined nodestates as a param2type or param3type that overrides specific fields but doesn't add to the nodedef count. 14:54 hecks where data isn't interleaved any more but rather stored in parallel, and there can be more layers than just content,param1,param2 14:55 MTDiscord So you wind up with block overlays over areas? 14:55 MTDiscord I can see that working 14:55 hecks perhaps 14:56 hecks i can't actually think of a good reason why low level map data is interleaved, doesn't that make it harder to compress? 14:57 sfan5 it's not interleaved in the mapblock format 14:57 hecks oh cool 14:58 hecks so it would be even less trouble to just add more layers 14:59 hecks do any optimizations exist to handle blank mapblocks? 14:59 hecks like when param2 is all 0 in a block; sure i know that zlib crushes this 15:00 hecks but having a special case like this would make extra layers easier to swallow, because they would cost very little if unused 15:01 MTDiscord I'd like it if maps were just plain Lua tables 15:01 sfan5 no such thing currently 15:02 hecks lua tables eat too much ram 15:02 MTDiscord they probably do 15:02 hecks meta is basically tables, but there is a good reason why the rest is a short and a couple bytes 15:03 hecks the low data of a blocksize 16 block fits into d-cache entirely on basically any processor 15:05 hecks anyway it shouldn't be hard to add some sort of representation for null layers in a block 15:07 hecks interally I see that map data is interleaved in ram, a mapblock is just an array of mapnodes which are u16, u8, u8 15:07 hecks so this would have to go first 15:16 VanessaE first blow-out the node defs limit :) 15:19 hecks well this would make that easier 15:23 VanessaE I don't see how 15:23 VanessaE the map format isn't the limiting factor 15:31 hecks it's not about the format, it's about the ram layout 15:31 hecks you can't just keep stuffing this sequentially laid out struct with more bits 15:32 hecks although sure, extra 16 bits for content wouldn't be that damaging 15:33 hecks and a u32 param0 would be pretty hard to run out of - but of course, it would bloat network traffic 15:33 hecks so maybe we ought to ask why do we want so many node IDs and is there a better way of doing it 15:33 hecks like a visual ID 15:33 MTDiscord Isn't the network traffic using the same mapblock serialization as the on-disk format? 15:34 hecks maybe? but what exactly does it send 15:34 hecks if param2 isn't modified, does it avoid sending that? 15:34 MTDiscord Most of the extra bits you add in there would get squashed down by zlib anyway... 15:34 hecks deflate is good but don't overestimate it 15:35 hecks i'm basically poking around for a way to add an extra param that costs nothing if it isn't used 15:35 MTDiscord If we end up using the same format as on-disk anyway, then the same reason we wouldn't have to change the on-disk format anyway would prevent the network from being affected. 15:35 MTDiscord I can't see why we wouldn't want to reuse the on-disk format for the network either... 15:36 hecks it sure begs the question why is the ram layout interleaved if serialization is parallel 15:36 MTDiscord ...CPU cache strategy? 15:37 hecks how often do we iterate needing all three params? 15:37 hecks with the work being so tight that cache becomes the bottleneck 15:42 MTDiscord That's a good question, and makes me wonder if e.g. ABMs could become a bit faster if we tried it the non-interleaved way instead... 15:44 hecks Warr1024: speak of the devil, check your PR 15:44 MTDiscord Interleaved makes sense for e.g. set_node type operations where it's assumed that all the attributes for a node will be processed at the same time, but cache layout is really more relevant for bulk cases, isn't it? And interleaved would probably be worse most of the time. 15:44 hecks correct, interleaving makes sense when you know you'll be using everything 15:44 sfan5 mesh building needs all three, any kind of Lua interaction uses all three, liquid updates require two 15:44 MTDiscord Oh, yeah, I've been thinking about that ... but I'm not sure how much of a diff it'd make, and it seems a bit worse in a Least Surprise sense. 15:45 hecks also cache optimization does not mean you have to interleave everything, i think? 15:45 MTDiscord (re: 11405) 15:45 hecks just ensure that things are more or less sequential and processed in big chunks 15:45 sfan5 mapgen sets two, light updates then come and use the other one 15:45 hecks Warr1024: table construction is slow even in luajit, consider it 15:45 sfan5 vmanips deinterlave but they also read/write all three 15:48 MTDiscord Re: de-interleaving in the liquid API, I'm on the fence about it, because my own use-case would work about equally well in both ways. I feel like the existing naive approach is more intuitive, which is of some benefit, but batch APIs are as far as I know largely unprecedented anyway, so now is a time when we could establish an alternative pattern ... I think if other core devs weighed in, or other game/mod-makers could describe 15:48 MTDiscord use-cases, it might help tip the balance. 15:48 hecks it'll be harder to change it to parallel once it's merged 15:48 hecks and it's thousands of table constructions less 15:49 sfan5 if only there was a way to get data into lua without creating a copy ;) 15:49 MTDiscord indeed, but without sacrificing portability... 15:50 MTDiscord c55 and sfan5 have already weighed in on the existing PR, so if they say they would prefer the parallel arrays approach instead of the nested one, I can make that change pretty quickly. 15:50 hecks start bundling the puc-lua ffi port? :o 15:51 MTDiscord If we could do FFI on all platforms that would be awesome ... though I still need to figure out what this API should look like in the meantime at least :-) 15:52 MTDiscord sfan5, do you have a preference for 11405 between callback({array of {pos, node}}) vs callback({array of pos}, {array of node})? The latter would probably be faster... 15:53 sfan5 we have few bulk APIs to compare this to/as precedent but I think the latter approach is useful here 15:53 hecks the difference is basically (3n) tables vs (2n+2) tables constructed per call 15:54 MTDiscord Okay, I'll make the change then. 15:54 hecks a table occupies a few dozen bytes minimum even if empty; unsunk GC allocations also cause extra lag on LJ 16:10 MTDiscord Okay, wtf: I'm creating 2 tables in a row, walking the list once, pushing values to both tables ... and when I run the callback somehow there's a 0 in my stack between the tables that's causing the callback running to crap out. 16:13 MTDiscord Maybe that's the "mode" argument and it's getting shoved in the wrong place because I'm passing the wrong nargs or stack arrangement... 16:18 MTDiscord Heh, whatever it was I fixed it (maybe I was running an out of date binary). 16:59 pgimeno reminder: FFI is available everywhere where LuaJIT is supported, and is independent of JIT. There's even an FFI library for pure Lua, written by Facebook: https://github.com/facebookarchive/luaffifb 17:01 pgimeno sorry, not written by Facebook 17:03 pgimeno that said, offering FFI poses security risks, but these could probably be alleviated with range-checking accessors 17:11 MTDiscord or why not let let system permissions handle binary access 17:36 MTDiscord I found a different FFI library a while back that looked pretty good for PUC Lua, it aimed to address some issues with other libraries. One sec while I dig it up 17:37 MTDiscord https://github.com/q66/cffi-lua 17:41 MTDiscord Correction: For any PUC Compatible Lua implementation (LuaJIT included) 18:15 MTDiscord Is there an issue for 5.4.0 minetest clients mistaking a huddef text field for an image field on 5.4.1 servers?? 18:15 MTDiscord *? 18:22 pgimeno @Benrob0329 LuaJIT's FFI has the advantage that it FLIES when it's jit-compiled. Having a compatible replacement for PUC Lua is just to (possibly, though unlikely) calm the voices that say that PUC Lua support should not be dropped. 18:30 hecks ffi would be good for an inversion of the api, it may even be faster on puc 18:30 hecks although i understand that builtin is already mostly structured in this way 18:32 hecks still such an inverted api would basically be a fraction of the code to maintain 18:32 hecks it's easier to maintain a binding to dllexported C functions than it is to manipulate the VM using the usual lua api 18:41 MTDiscord nvm on the hud, was an issue on my side 19:45 v-rob You know, when c55 was doing tests of content_t as 32-bit and 64-bit, the 64-bit would be useless. Many systems only have 4 GB or less of RAM, so you'd run out of RAM long before hitting even the u32 limit (since 2^32 == 4 GB). If it were possible, a 24-bit content_t would be ideal since it would save a byte while not being possible to run out of them. But the only system I know that has 24-bit integers is the Zilog eZ80 :P 19:47 pgimeno things are not always like they seem, LuaJIT has a limit of 2 GB of RAM in a 32 bit Linux and 1 GB in a 64 bit Linux 19:48 pgimeno (LuaJIT 2.0 that is) 19:48 v-rob I'm talking C++-side 19:48 v-rob `MapNode` 19:48 specing > while not being possible to run out of them. 19:48 specing hold my beer 19:50 specing If the limit was >>32k,then there would be a lot more tablesaw-supported nodes 19:50 specing perhaps even finer nodes 19:50 specing e.g. 3-stairs or something like that 19:50 v-rob I know, u16 is too small, but u32 is overkill and will never be fully used 19:51 v-rob Well, I guess a 16 < x < 32-bit content_t theoretically could be done with an `&` without being too slow. After all, u16s are often loaded from u32s with an `&` internally on many modern CPUs anyway. 19:51 specing I also found that there is a lack of 1/16 slab nodes when you want to build something completely out of them 19:51 specing e.g. theres no slope node that would start at 1/16 and go to full size 19:58 v-rob Hmm, a good implementation for MapNode might be a bit-packed u64, like `class MapNode { u64 m_value; u32 getContent() { return m_value & CONTENT_BITS /* e.g. could be 24 bits */; } void setContent(u32 value) { m_value |= value & CONTENT_BITS } /* More methods for param fields, allowing arbitrary numbers of params up to the 64-bit limit */ }` if the bitwise operations don't slow things down. 19:58 sfan5 the compiler can do that for you for free btw 19:59 v-rob Sure, but not with arbitrary numbers of param fields. 19:59 v-rob Oops, setContent is wrong. But the gist is there 20:05 v-rob I still want better param fields since that seems to me to be a better solution than registering tons of nodes. After all, `ContentFeatures` is a really fat struct and takes up a lot of RAM. Params can handle it so much cleaner. 20:06 v-rob Imagine if "facedir" was implemented as 32 separate nodes for every direction. For the same reason, I think other things should be moved to param fields. 20:06 specing Yeah, the slabs should probably be params 20:07 specing same node, different texture 20:08 specing and then we probably wouldn't need >16bit node ids 20:08 v-rob Then again, a problem with params is that they only apply to nodes in the world, not nodes in the inventory 20:09 specing Would it be hard to add that to itemstacks? 20:09 sfan5 no 20:09 v-rob But do we want facedir preserved in itemstacks? I don't think so. 20:09 specing ofc not 20:09 specing but the texture info yes 20:10 v-rob Ugh, there's no simple solution 20:10 sfan5 uh 20:10 sfan5 it all goes through Lua anyway so it'd be just as easy to have it only apply to nodes where it makes sense 20:11 sfan5 and giving that choice to mods 20:11 MTDiscord v-rob: One could also argue that most systems made within the past 5 years or so have > 4GB of ram, so I don't think that its really an argument. A better argument might be that having 4GB of nodedefs is stupid, but having the next biggest INT doesn't seem too bad to my (naive) thoughts. 20:12 v-rob My thoughts on 24-bit integers is to leave more space for params since 32-bit isn't really necessary 20:12 v-rob Otherwise, I'd just leave it at 32 for simplicity 20:14 specing sfan5: yeah 20:14 specing sfan5: register_node_variation(node_string, { ... }) 20:17 v-rob Yeah, an alternative to making more params is to make it easier for Lua, certainly. 20:20 specing Btw, I've heard someone say that it is possible to do C/C++ mods 20:20 specing is there an example? 20:21 v-rob It's called editing the C++ code and compiling it, I think :P 20:27 v-rob It's too bad params can't be less hardcoded. At least adding multiple nodedefs if fully controllable by Lua. 20:28 v-rob *if -> is 20:28 v-rob Oh well, I'll blame all our problems on SSCSM or something and forget about nodes for now. 20:31 specing v-rob: oh, ok 20:32 specing I think ya'll are overcomplicating SSCSMs 20:32 specing just make a central contentDB for them and make server owners publish them there, then they get reviewed and servers can request them to be [down]loaded 20:33 sfan5 sure we'll "just" put lots of effort into a solution that doesn't scale 20:34 v-rob Um, core devs already have how much trouble reviewing just C++ code? Why should we expect other people to volunteer to do it with a bunch of random people's code? 20:34 specing you can think about a scalable solution after it becomes unmanageable 20:34 specing right now you've been delaying this great player experience improvement for 5 years already 20:34 v-rob I don't think such a thing will work for more than a month, tops 20:35 specing you can tap the review power of other server owners, as many will also want to reuse it on theirs 20:36 specing if you just implement direct download then it'll become a mess to track SSCSMs across servers and it will result in many duplicates being developed 20:36 v-rob As I've said before, I believe SSCSM needs to be experimentally implemented under a disabled-by-default setting with a big, fat, warning label. Then, we can do everything at our leisure, slowly. Security issues can be tackled one at a time instead of all at once. Monolithic PRs aren't the solution. 20:38 v-rob I'd do it myself, but I know very little about security, and I'm already in the process of rewriting the entire GUI, which is no small task. 20:39 specing you don't have to do much about security if its centralised and reviewed 20:42 sfan5 who's gonna school random community members on how to review security-critical code? you? 20:42 specing actually yeah, I'm thinking about starting SSCSM support in my fork of MT 20:44 specing sfan5: and don't say you don't have several trustworthy server owners who could take over this task 20:44 sfan5 trust is not the issue, it's ability 20:45 v-rob Security is notoriously hard to get right 20:45 v-rob Professionals mess it up all the time 20:45 v-rob I don't expect volunteers to not mess up, and all it takes is one mess up. 20:45 v-rob Better to do the security issues once, in the engine. 20:46 specing v-rob: why not do both? 20:46 v-rob Because it's not feasible. 20:46 specing that's what you say 20:49 v-rob There are thousands of released mods, and many get updated all the time. If SSCSM is half as useful as it's been promoted, a significant portion of them will have it. Who's going to review them all and do it without mistakes? People who make SSCSM mods, client users, and server users can all exploit security issues. It's dangerous, and it will be too much of a bottleneck. 20:51 v-rob If that's how SSCSM would work, I would never use any of them. I don't want any possibility of my computer being exploited, thank you. 20:51 rubenwardy RE: CSMs on ContentDB: It often takes a week or more for new packages to be reviewed when editors are busy, it's not very feasible to require reviewing all new releases 20:52 rubenwardy also c55 requires that connecting a server doesn't require any third party services 20:53 sfan5 is CDB a third party? ? 20:53 rubenwardy third-party in regards to the server 21:32 MTDiscord Yeah definitely not for the review thing as a solution. Much more for taking things slowly and making it buyer beware. Hopefully in 1-2 years its a halfway decent api, but for now, it could simply be chat sounds haha