Minetest logo

IRC log for #minetest-dev, 2021-07-04

| Channels | #minetest-dev index | Today | | Google Search | Plaintext

All times shown according to UTC.

Time Nick Message
01:07 v-rob joined #minetest-dev
01:27 sfan5 v-rob: a mapblock cannot have more than 4096 nodes hence it will never need more than 16-bit ids
01:29 v-rob Oh yeah, duh. That makes sense
01:29 sfan5 it's kind of obvious once you get it
01:37 VanessaE I've forgotten..  apart from the time and effort that'd be needed, what stands in the way of it?
01:38 sfan5 conving devs that this is a good idea is kind of a point
01:38 VanessaE I kinda file that into "time and effort" :)
01:39 VanessaE i.e. it's not a technical reason :)
01:41 kilbith joined #minetest-dev
01:42 VanessaE that being said, you know I think it's a good idea, plus you'd be upgrading the range one last time (i.e. likely it'll never again be needed).  but the downside to that is that frankly the client is already pretty slow to start-up if there's an assload of node defs to process.
01:42 VanessaE (I don't need it, even DB only has like 20k node defs, but SOMEONE will eventually)
01:46 MTDiscord <Jonathon> there are already servers that walk the line with it
01:46 VanessaE I think of it as a "just go for overkill" opportunity, since it's not the engine's job to prevent the user or admin doing something moronic like registering a million superfluous nodes.
01:47 VanessaE (though it IS the engine's job to provide features that help the modder avoid having to do so)
01:50 MTDiscord <Jonathon> if something like https://github.com/minetest/minetest/issues/9193 was addressed/had a PR for it, you could cut way down on node registrations used for stuff like farming, machines with active states
02:28 v-rob joined #minetest-dev
02:30 lhofhansl joined #minetest-dev
02:39 kilbith joined #minetest-dev
03:10 v-rob joined #minetest-dev
03:16 kilbith joined #minetest-dev
03:17 kilbith_ joined #minetest-dev
04:29 v-rob joined #minetest-dev
06:01 v-rob joined #minetest-dev
06:56 Extex joined #minetest-dev
07:11 hecks joined #minetest-dev
07:31 tekakutli joined #minetest-dev
08:39 Fixer joined #minetest-dev
08:40 YuGiOhJCJ joined #minetest-dev
09:27 specing_ joined #minetest-dev
09:37 MTDiscord <appguru> I'd like it if we could arbitrarily override most node def fields using metadata
09:38 MTDiscord <appguru> For example tiles: You could just override a tile if you wanted signs
09:39 MTDiscord <appguru> The problem is just that the flat k/v storage of meta and the Lua table format of node defs don't go together well...
09:48 sfan5 the problem is that it complicates the mesh building because suddenly you have to consider metadata and comparing node IDs is no longer of any use
10:01 calcul0n_ joined #minetest-dev
10:45 entuland joined #minetest-dev
11:03 behalebabo joined #minetest-dev
14:19 v-rob joined #minetest-dev
14:51 MTDiscord <Benrob0329> At that point, why not just optimize for that entirely? If you're going to have in-map definitions you might as well take the concept and run with it IMO
14:53 hecks do away with CIDs and have immediate-mode node defs of some sort?
14:53 hecks like per-block palettes
14:53 hecks oh but i have a better idea
14:53 hecks map data layers
14:54 MTDiscord <Benrob0329> Honestly I'm still partial to predefined nodestates as a param2type or param3type that overrides specific fields but doesn't add to the nodedef count.
14:54 hecks where data isn't interleaved any more but rather stored in parallel, and there can be more layers than just content,param1,param2
14:55 MTDiscord <Benrob0329> So you wind up with block overlays over areas?
14:55 MTDiscord <Benrob0329> I can see that working
14:55 hecks perhaps
14:56 hecks i can't actually think of a good reason why low level map data is interleaved, doesn't that make it harder to compress?
14:57 sfan5 it's not interleaved in the mapblock format
14:57 hecks oh cool
14:58 hecks so it would be even less trouble to just add more layers
14:59 hecks do any optimizations exist to handle blank mapblocks?
14:59 hecks like when param2 is all 0 in a block; sure i know that zlib crushes this
15:00 hecks but having a special case like this would make extra layers easier to swallow, because they would cost very little if unused
15:01 MTDiscord <appguru> I'd like it if maps were just plain Lua tables
15:01 sfan5 no such thing currently
15:02 hecks lua tables eat too much ram
15:02 MTDiscord <appguru> they probably do
15:02 hecks meta is basically tables, but there is a good reason why the rest is a short and a couple bytes
15:03 hecks the low data of a blocksize 16 block fits into d-cache entirely on basically any processor
15:05 hecks anyway it shouldn't be hard to add some sort of representation for null layers in a block
15:07 hecks interally I see that map data is interleaved in ram, a mapblock is just an array of mapnodes which are u16, u8, u8
15:07 hecks so this would have to go first
15:16 VanessaE first blow-out the node defs limit :)
15:19 hecks well this would make that easier
15:23 VanessaE I don't see how
15:23 VanessaE the map format isn't the limiting factor
15:31 hecks it's not about the format, it's about the ram layout
15:31 hecks you can't just keep stuffing this sequentially laid out struct with more bits
15:32 hecks although sure, extra 16 bits for content wouldn't be that damaging
15:33 hecks and a u32 param0 would be pretty hard to run out of - but of course, it would bloat network traffic
15:33 hecks so maybe we ought to ask why do we want so many node IDs and is there a better way of doing it
15:33 hecks like a visual ID
15:33 MTDiscord <Warr1024> Isn't the network traffic using the same mapblock serialization as the on-disk format?
15:34 hecks maybe? but what exactly does it send
15:34 hecks if param2 isn't modified, does it avoid sending that?
15:34 MTDiscord <Warr1024> Most of the extra bits you add in there would get squashed down by zlib anyway...
15:34 hecks deflate is good but don't overestimate it
15:35 hecks i'm basically poking around for a way to add an extra param that costs nothing if it isn't used
15:35 MTDiscord <Warr1024> If we end up using the same format as on-disk anyway, then the same reason we wouldn't have to change the on-disk format anyway would prevent the network from being affected.
15:35 MTDiscord <Warr1024> I can't see why we wouldn't want to reuse the on-disk format for the network either...
15:36 hecks it sure begs the question why is the ram layout interleaved if serialization is parallel
15:36 MTDiscord <Warr1024> ...CPU cache strategy?
15:36 kilbith joined #minetest-dev
15:37 hecks how often do we iterate needing all three params?
15:37 hecks with the work being so tight that cache becomes the bottleneck
15:42 MTDiscord <Warr1024> That's a good question, and makes me wonder if e.g. ABMs could become a bit faster if we tried it the non-interleaved way instead...
15:44 hecks Warr1024: speak of the devil, check your PR
15:44 MTDiscord <Warr1024> Interleaved makes sense for e.g. set_node type operations where it's assumed that all the attributes for a node will be processed at the same time, but cache layout is really more relevant for bulk cases, isn't it?  And interleaved would probably be worse most of the time.
15:44 hecks correct, interleaving makes sense when you know you'll be using everything
15:44 sfan5 mesh building needs all three, any kind of Lua interaction uses all three, liquid updates require two
15:44 MTDiscord <Warr1024> Oh, yeah, I've been thinking about that ... but I'm not sure how much of a diff it'd make, and it seems a bit worse in a Least Surprise sense.
15:45 hecks also cache optimization does not mean you have to interleave everything, i think?
15:45 MTDiscord <Warr1024> (re: 11405)
15:45 hecks just ensure that things are more or less sequential and processed in big chunks
15:45 sfan5 mapgen sets two, light updates then come and use the other one
15:45 hecks Warr1024: table construction is slow even in luajit, consider it
15:45 sfan5 vmanips deinterlave but they also read/write all three
15:48 MTDiscord <Warr1024> Re: de-interleaving in the liquid API, I'm on the fence about it, because my own use-case would work about equally well in both ways.  I feel like the existing naive approach is more intuitive, which is of some benefit, but batch APIs are as far as I know largely unprecedented anyway, so now is a time when we could establish an alternative pattern ... I think if other core devs weighed in, or other game/mod-makers could describe
15:48 MTDiscord use-cases, it might help tip the balance.
15:48 hecks it'll be harder to change it to parallel once it's merged
15:48 hecks and it's thousands of table constructions less
15:49 Extex joined #minetest-dev
15:49 sfan5 if only there was a way to get data into lua without creating a copy ;)
15:49 MTDiscord <Warr1024> indeed, but without sacrificing portability...
15:50 MTDiscord <Warr1024> c55 and sfan5 have already weighed in on the existing PR, so if they say they would prefer the parallel arrays approach instead of the nested one, I can make that change pretty quickly.
15:50 hecks start bundling the puc-lua ffi port? :o
15:51 MTDiscord <Warr1024> If we could do FFI on all platforms that would be awesome ... though I still need to figure out what this API should look like in the meantime at least :-)
15:52 MTDiscord <Warr1024> sfan5, do you have a preference for 11405 between callback({array of {pos, node}}) vs callback({array of pos}, {array of node})?  The latter would probably be faster...
15:53 sfan5 we have few bulk APIs to compare this to/as precedent but I think the latter approach is useful here
15:53 hecks the difference is basically (3n) tables vs (2n+2) tables constructed per call
15:54 MTDiscord <Warr1024> Okay, I'll make the change then.
15:54 hecks a table occupies a few dozen bytes minimum even if empty; unsunk GC allocations also cause extra lag on LJ
15:59 Noisytoot joined #minetest-dev
16:10 MTDiscord <Warr1024> Okay, wtf: I'm creating 2 tables in a row, walking the list once, pushing values to both tables ... and when I run the callback somehow there's a 0 in my stack between the tables that's causing the callback running to crap out.
16:13 MTDiscord <Warr1024> Maybe that's the "mode" argument and it's getting shoved in the wrong place because I'm passing the wrong nargs or stack arrangement...
16:18 MTDiscord <Warr1024> Heh, whatever it was I fixed it (maybe I was running an out of date binary).
16:59 pgimeno reminder: FFI is available everywhere where LuaJIT is supported, and is independent of JIT. There's even an FFI library for pure Lua, written by Facebook: https://github.com/facebookarchive/luaffifb
17:01 pgimeno sorry, not written by Facebook
17:03 pgimeno that said, offering FFI poses security risks, but these could probably be alleviated with range-checking accessors
17:11 MTDiscord <Jordach> or why not let let system permissions handle binary access
17:13 pouellet joined #minetest-dev
17:36 MTDiscord <Benrob0329> I found a different FFI library a while back that looked pretty good for PUC Lua, it aimed to address some issues with other libraries. One sec while I dig it up
17:37 MTDiscord <Benrob0329> https://github.com/q66/cffi-lua
17:41 MTDiscord <Benrob0329> Correction: For any PUC Compatible Lua implementation (LuaJIT included)
17:54 appguru joined #minetest-dev
18:15 MTDiscord <Lone_Wulf> Is there an issue for 5.4.0 minetest clients mistaking a huddef text field for an image field on 5.4.1 servers??
18:15 MTDiscord <Lone_Wulf> *?
18:22 pgimeno @Benrob0329 LuaJIT's FFI has the advantage that it FLIES when it's jit-compiled. Having a compatible replacement for PUC Lua is just to (possibly, though unlikely) calm the voices that say that PUC Lua support should not be dropped.
18:30 hecks ffi would be good for an inversion of the api, it may even be faster on puc
18:30 hecks although i understand that builtin is already mostly structured in this way
18:32 hecks still such an inverted api would basically be a fraction of the code to maintain
18:32 hecks it's easier to maintain a binding to dllexported C functions than it is to manipulate the VM using the usual lua api
18:41 MTDiscord <Lone_Wulf> nvm on the hud, was an issue on my side
19:19 v-rob joined #minetest-dev
19:45 v-rob You know, when c55 was doing tests of content_t as 32-bit and 64-bit, the 64-bit would be useless. Many systems only have 4 GB or less of RAM, so you'd run out of RAM long before hitting even the u32 limit (since 2^32 == 4 GB). If it were possible, a 24-bit content_t would be ideal since it would save a byte while not being possible to run out of them. But the only system I know that has 24-bit integers is the Zilog eZ80 :P
19:47 pgimeno things are not always like they seem, LuaJIT has a limit of 2 GB of RAM in a 32 bit Linux and 1 GB in a 64 bit Linux
19:48 pgimeno (LuaJIT 2.0 that is)
19:48 v-rob I'm talking C++-side
19:48 v-rob `MapNode`
19:48 specing > while not being possible to run out of them.
19:48 specing hold my beer
19:50 specing If the limit was >>32k,then there would be a lot more tablesaw-supported nodes
19:50 specing perhaps even finer nodes
19:50 specing e.g. 3-stairs or something like that
19:50 v-rob I know, u16 is too small, but u32 is overkill and will never be fully used
19:51 v-rob Well, I guess a 16 < x < 32-bit content_t theoretically could be done with an `&` without being too slow. After all, u16s are often loaded from u32s with an `&` internally on many modern CPUs anyway.
19:51 specing I also found that there is a lack of 1/16 slab nodes when you want to build something completely out of them
19:51 specing e.g. theres no slope node that would start at 1/16 and go to full size
19:58 v-rob Hmm, a good implementation for MapNode might be a bit-packed u64, like `class MapNode { u64 m_value; u32 getContent() { return m_value & CONTENT_BITS /* e.g. could be 24 bits */; } void setContent(u32 value) { m_value |= value & CONTENT_BITS } /* More methods for param fields, allowing arbitrary numbers of params up to the 64-bit limit */ }` if the bitwise operations don't slow things down.
19:58 sfan5 the compiler can do that for you for free btw
19:59 v-rob Sure, but not with arbitrary numbers of param fields.
19:59 v-rob Oops, setContent is wrong. But the gist is there
20:05 v-rob I still want better param fields since that seems to me to be a better solution than registering tons of nodes. After all, `ContentFeatures` is a really fat struct and takes up a lot of RAM. Params can handle it so much cleaner.
20:06 v-rob Imagine if "facedir" was implemented as 32 separate nodes for every direction. For the same reason, I think other things should be moved to param fields.
20:06 specing Yeah, the slabs should probably be params
20:07 specing same node, different texture
20:08 specing and then we probably wouldn't need >16bit node ids
20:08 v-rob Then again, a problem with params is that they only apply to nodes in the world, not nodes in the inventory
20:09 specing Would it be hard to add that to itemstacks?
20:09 sfan5 no
20:09 v-rob But do we want facedir preserved in itemstacks? I don't think so.
20:09 specing ofc not
20:09 specing but the texture info yes
20:10 v-rob Ugh, there's no simple solution
20:10 sfan5 uh
20:10 sfan5 it all goes through Lua anyway so it'd be just as easy to have it only apply to nodes where it makes sense
20:11 sfan5 and giving that choice to mods
20:11 MTDiscord <Benrob0329> v-rob: One could also argue that most systems made within the past 5 years or so have > 4GB of ram, so I don't think that its really an argument. A better argument might be that having 4GB of nodedefs is stupid, but having the next biggest INT doesn't seem too bad to my (naive) thoughts.
20:12 v-rob My thoughts on 24-bit integers is to leave more space for params since 32-bit isn't really necessary
20:12 v-rob Otherwise, I'd just leave it at 32 for simplicity
20:14 specing sfan5: yeah
20:14 specing sfan5: register_node_variation(node_string, { ... })
20:17 v-rob Yeah, an alternative to making more params is to make it easier for Lua, certainly.
20:20 specing Btw, I've heard someone say that it is possible to do C/C++ mods
20:20 specing is there an example?
20:21 v-rob It's called editing the C++ code and compiling it, I think :P
20:27 v-rob It's too bad params can't be less hardcoded. At least adding multiple nodedefs if fully controllable by Lua.
20:28 v-rob *if -> is
20:28 v-rob Oh well, I'll blame all our problems on SSCSM or something and forget about nodes for now.
20:31 specing v-rob: oh, ok
20:32 specing I think ya'll are overcomplicating SSCSMs
20:32 specing just make a central contentDB for them and make server owners publish them there, then they get reviewed and servers can request them to be [down]loaded
20:33 sfan5 sure we'll "just" put lots of effort into a solution that doesn't scale
20:34 v-rob Um, core devs already have how much trouble reviewing just C++ code? Why should we expect other people to volunteer to do it with a bunch of random people's code?
20:34 specing you can think about a scalable solution after it becomes unmanageable
20:34 specing right now you've been delaying this great player experience improvement for 5 years already
20:34 v-rob I don't think such a thing will work for more than a month, tops
20:35 specing you can tap the review power of other server owners, as many will also want to reuse it on theirs
20:36 specing if you just implement direct download then it'll become a mess to track SSCSMs across servers and it will result in many duplicates being developed
20:36 v-rob As I've said before, I believe SSCSM needs to be experimentally implemented under a disabled-by-default setting with a big, fat, warning label. Then, we can do everything at our leisure, slowly.  Security issues can be tackled one at a time instead of all at once.  Monolithic PRs aren't the solution.
20:38 v-rob I'd do it myself, but I know very little about security, and I'm already in the process of rewriting the entire GUI, which is no small task.
20:39 specing you don't have to do much about security if its centralised and reviewed
20:42 sfan5 who's gonna school random community members on how to review security-critical code? you?
20:42 specing actually yeah, I'm thinking about starting SSCSM support in my fork of MT
20:44 specing sfan5: and don't say you don't have several trustworthy server owners who could take over this task
20:44 sfan5 trust is not the issue, it's ability
20:45 v-rob Security is notoriously hard to get right
20:45 v-rob Professionals mess it up all the time
20:45 v-rob I don't expect volunteers to not mess up, and all it takes is one mess up.
20:45 v-rob Better to do the security issues once, in the engine.
20:46 specing v-rob: why not do both?
20:46 v-rob Because it's not feasible.
20:46 specing that's what you say
20:49 v-rob There are thousands of released mods, and many get updated all the time. If SSCSM is half as useful as it's been promoted, a significant portion of them will have it. Who's going to review them all and do it without mistakes? People who make SSCSM mods, client users, and server users can all exploit security issues. It's dangerous, and it will be too much of a bottleneck.
20:51 v-rob If that's how SSCSM would work, I would never use any of them. I don't want any possibility of my computer being exploited, thank you.
20:51 rubenwardy RE: CSMs on ContentDB: It often takes a week or more for new packages to be reviewed when editors are busy, it's not very feasible to require reviewing all new releases
20:52 rubenwardy also c55 requires that connecting a server doesn't require any third party services
20:53 sfan5 is CDB a third party? 🤔
20:53 rubenwardy third-party in regards to the server
21:31 specing joined #minetest-dev
21:32 MTDiscord <exe_virus> Yeah definitely not for the review thing as a solution. Much more for taking things slowly and making it buyer beware. Hopefully in 1-2 years its a halfway decent api, but for now, it could simply be chat sounds haha
21:49 queria^afk joined #minetest-dev
22:05 Extex joined #minetest-dev
22:10 v-rob joined #minetest-dev
22:35 v-rob joined #minetest-dev
23:15 Extex joined #minetest-dev
23:34 v-rob joined #minetest-dev
23:47 AliasAlreadyTake joined #minetest-dev

| Channels | #minetest-dev index | Today | | Google Search | Plaintext