| Time |
Nick |
Message |
| 01:07 |
|
v-rob joined #minetest-dev |
| 01:27 |
sfan5 |
v-rob: a mapblock cannot have more than 4096 nodes hence it will never need more than 16-bit ids |
| 01:29 |
v-rob |
Oh yeah, duh. That makes sense |
| 01:29 |
sfan5 |
it's kind of obvious once you get it |
| 01:37 |
VanessaE |
I've forgotten.. apart from the time and effort that'd be needed, what stands in the way of it? |
| 01:38 |
sfan5 |
conving devs that this is a good idea is kind of a point |
| 01:38 |
VanessaE |
I kinda file that into "time and effort" :) |
| 01:39 |
VanessaE |
i.e. it's not a technical reason :) |
| 01:41 |
|
kilbith joined #minetest-dev |
| 01:42 |
VanessaE |
that being said, you know I think it's a good idea, plus you'd be upgrading the range one last time (i.e. likely it'll never again be needed). but the downside to that is that frankly the client is already pretty slow to start-up if there's an assload of node defs to process. |
| 01:42 |
VanessaE |
(I don't need it, even DB only has like 20k node defs, but SOMEONE will eventually) |
| 01:46 |
MTDiscord |
<Jonathon> there are already servers that walk the line with it |
| 01:46 |
VanessaE |
I think of it as a "just go for overkill" opportunity, since it's not the engine's job to prevent the user or admin doing something moronic like registering a million superfluous nodes. |
| 01:47 |
VanessaE |
(though it IS the engine's job to provide features that help the modder avoid having to do so) |
| 01:50 |
MTDiscord |
<Jonathon> if something like https://github.com/minetest/minetest/issues/9193 was addressed/had a PR for it, you could cut way down on node registrations used for stuff like farming, machines with active states |
| 02:28 |
|
v-rob joined #minetest-dev |
| 02:30 |
|
lhofhansl joined #minetest-dev |
| 02:39 |
|
kilbith joined #minetest-dev |
| 03:10 |
|
v-rob joined #minetest-dev |
| 03:16 |
|
kilbith joined #minetest-dev |
| 03:17 |
|
kilbith_ joined #minetest-dev |
| 04:29 |
|
v-rob joined #minetest-dev |
| 06:01 |
|
v-rob joined #minetest-dev |
| 06:56 |
|
Extex joined #minetest-dev |
| 07:11 |
|
hecks joined #minetest-dev |
| 07:31 |
|
tekakutli joined #minetest-dev |
| 08:39 |
|
Fixer joined #minetest-dev |
| 08:40 |
|
YuGiOhJCJ joined #minetest-dev |
| 09:27 |
|
specing_ joined #minetest-dev |
| 09:37 |
MTDiscord |
<appguru> I'd like it if we could arbitrarily override most node def fields using metadata |
| 09:38 |
MTDiscord |
<appguru> For example tiles: You could just override a tile if you wanted signs |
| 09:39 |
MTDiscord |
<appguru> The problem is just that the flat k/v storage of meta and the Lua table format of node defs don't go together well... |
| 09:48 |
sfan5 |
the problem is that it complicates the mesh building because suddenly you have to consider metadata and comparing node IDs is no longer of any use |
| 10:01 |
|
calcul0n_ joined #minetest-dev |
| 10:45 |
|
entuland joined #minetest-dev |
| 11:03 |
|
behalebabo joined #minetest-dev |
| 14:19 |
|
v-rob joined #minetest-dev |
| 14:51 |
MTDiscord |
<Benrob0329> At that point, why not just optimize for that entirely? If you're going to have in-map definitions you might as well take the concept and run with it IMO |
| 14:53 |
hecks |
do away with CIDs and have immediate-mode node defs of some sort? |
| 14:53 |
hecks |
like per-block palettes |
| 14:53 |
hecks |
oh but i have a better idea |
| 14:53 |
hecks |
map data layers |
| 14:54 |
MTDiscord |
<Benrob0329> Honestly I'm still partial to predefined nodestates as a param2type or param3type that overrides specific fields but doesn't add to the nodedef count. |
| 14:54 |
hecks |
where data isn't interleaved any more but rather stored in parallel, and there can be more layers than just content,param1,param2 |
| 14:55 |
MTDiscord |
<Benrob0329> So you wind up with block overlays over areas? |
| 14:55 |
MTDiscord |
<Benrob0329> I can see that working |
| 14:55 |
hecks |
perhaps |
| 14:56 |
hecks |
i can't actually think of a good reason why low level map data is interleaved, doesn't that make it harder to compress? |
| 14:57 |
sfan5 |
it's not interleaved in the mapblock format |
| 14:57 |
hecks |
oh cool |
| 14:58 |
hecks |
so it would be even less trouble to just add more layers |
| 14:59 |
hecks |
do any optimizations exist to handle blank mapblocks? |
| 14:59 |
hecks |
like when param2 is all 0 in a block; sure i know that zlib crushes this |
| 15:00 |
hecks |
but having a special case like this would make extra layers easier to swallow, because they would cost very little if unused |
| 15:01 |
MTDiscord |
<appguru> I'd like it if maps were just plain Lua tables |
| 15:01 |
sfan5 |
no such thing currently |
| 15:02 |
hecks |
lua tables eat too much ram |
| 15:02 |
MTDiscord |
<appguru> they probably do |
| 15:02 |
hecks |
meta is basically tables, but there is a good reason why the rest is a short and a couple bytes |
| 15:03 |
hecks |
the low data of a blocksize 16 block fits into d-cache entirely on basically any processor |
| 15:05 |
hecks |
anyway it shouldn't be hard to add some sort of representation for null layers in a block |
| 15:07 |
hecks |
interally I see that map data is interleaved in ram, a mapblock is just an array of mapnodes which are u16, u8, u8 |
| 15:07 |
hecks |
so this would have to go first |
| 15:16 |
VanessaE |
first blow-out the node defs limit :) |
| 15:19 |
hecks |
well this would make that easier |
| 15:23 |
VanessaE |
I don't see how |
| 15:23 |
VanessaE |
the map format isn't the limiting factor |
| 15:31 |
hecks |
it's not about the format, it's about the ram layout |
| 15:31 |
hecks |
you can't just keep stuffing this sequentially laid out struct with more bits |
| 15:32 |
hecks |
although sure, extra 16 bits for content wouldn't be that damaging |
| 15:33 |
hecks |
and a u32 param0 would be pretty hard to run out of - but of course, it would bloat network traffic |
| 15:33 |
hecks |
so maybe we ought to ask why do we want so many node IDs and is there a better way of doing it |
| 15:33 |
hecks |
like a visual ID |
| 15:33 |
MTDiscord |
<Warr1024> Isn't the network traffic using the same mapblock serialization as the on-disk format? |
| 15:34 |
hecks |
maybe? but what exactly does it send |
| 15:34 |
hecks |
if param2 isn't modified, does it avoid sending that? |
| 15:34 |
MTDiscord |
<Warr1024> Most of the extra bits you add in there would get squashed down by zlib anyway... |
| 15:34 |
hecks |
deflate is good but don't overestimate it |
| 15:35 |
hecks |
i'm basically poking around for a way to add an extra param that costs nothing if it isn't used |
| 15:35 |
MTDiscord |
<Warr1024> If we end up using the same format as on-disk anyway, then the same reason we wouldn't have to change the on-disk format anyway would prevent the network from being affected. |
| 15:35 |
MTDiscord |
<Warr1024> I can't see why we wouldn't want to reuse the on-disk format for the network either... |
| 15:36 |
hecks |
it sure begs the question why is the ram layout interleaved if serialization is parallel |
| 15:36 |
MTDiscord |
<Warr1024> ...CPU cache strategy? |
| 15:36 |
|
kilbith joined #minetest-dev |
| 15:37 |
hecks |
how often do we iterate needing all three params? |
| 15:37 |
hecks |
with the work being so tight that cache becomes the bottleneck |
| 15:42 |
MTDiscord |
<Warr1024> That's a good question, and makes me wonder if e.g. ABMs could become a bit faster if we tried it the non-interleaved way instead... |
| 15:44 |
hecks |
Warr1024: speak of the devil, check your PR |
| 15:44 |
MTDiscord |
<Warr1024> Interleaved makes sense for e.g. set_node type operations where it's assumed that all the attributes for a node will be processed at the same time, but cache layout is really more relevant for bulk cases, isn't it? And interleaved would probably be worse most of the time. |
| 15:44 |
hecks |
correct, interleaving makes sense when you know you'll be using everything |
| 15:44 |
sfan5 |
mesh building needs all three, any kind of Lua interaction uses all three, liquid updates require two |
| 15:44 |
MTDiscord |
<Warr1024> Oh, yeah, I've been thinking about that ... but I'm not sure how much of a diff it'd make, and it seems a bit worse in a Least Surprise sense. |
| 15:45 |
hecks |
also cache optimization does not mean you have to interleave everything, i think? |
| 15:45 |
MTDiscord |
<Warr1024> (re: 11405) |
| 15:45 |
hecks |
just ensure that things are more or less sequential and processed in big chunks |
| 15:45 |
sfan5 |
mapgen sets two, light updates then come and use the other one |
| 15:45 |
hecks |
Warr1024: table construction is slow even in luajit, consider it |
| 15:45 |
sfan5 |
vmanips deinterlave but they also read/write all three |
| 15:48 |
MTDiscord |
<Warr1024> Re: de-interleaving in the liquid API, I'm on the fence about it, because my own use-case would work about equally well in both ways. I feel like the existing naive approach is more intuitive, which is of some benefit, but batch APIs are as far as I know largely unprecedented anyway, so now is a time when we could establish an alternative pattern ... I think if other core devs weighed in, or other game/mod-makers could describe |
| 15:48 |
MTDiscord |
use-cases, it might help tip the balance. |
| 15:48 |
hecks |
it'll be harder to change it to parallel once it's merged |
| 15:48 |
hecks |
and it's thousands of table constructions less |
| 15:49 |
|
Extex joined #minetest-dev |
| 15:49 |
sfan5 |
if only there was a way to get data into lua without creating a copy ;) |
| 15:49 |
MTDiscord |
<Warr1024> indeed, but without sacrificing portability... |
| 15:50 |
MTDiscord |
<Warr1024> c55 and sfan5 have already weighed in on the existing PR, so if they say they would prefer the parallel arrays approach instead of the nested one, I can make that change pretty quickly. |
| 15:50 |
hecks |
start bundling the puc-lua ffi port? :o |
| 15:51 |
MTDiscord |
<Warr1024> If we could do FFI on all platforms that would be awesome ... though I still need to figure out what this API should look like in the meantime at least :-) |
| 15:52 |
MTDiscord |
<Warr1024> sfan5, do you have a preference for 11405 between callback({array of {pos, node}}) vs callback({array of pos}, {array of node})? The latter would probably be faster... |
| 15:53 |
sfan5 |
we have few bulk APIs to compare this to/as precedent but I think the latter approach is useful here |
| 15:53 |
hecks |
the difference is basically (3n) tables vs (2n+2) tables constructed per call |
| 15:54 |
MTDiscord |
<Warr1024> Okay, I'll make the change then. |
| 15:54 |
hecks |
a table occupies a few dozen bytes minimum even if empty; unsunk GC allocations also cause extra lag on LJ |
| 15:59 |
|
Noisytoot joined #minetest-dev |
| 16:10 |
MTDiscord |
<Warr1024> Okay, wtf: I'm creating 2 tables in a row, walking the list once, pushing values to both tables ... and when I run the callback somehow there's a 0 in my stack between the tables that's causing the callback running to crap out. |
| 16:13 |
MTDiscord |
<Warr1024> Maybe that's the "mode" argument and it's getting shoved in the wrong place because I'm passing the wrong nargs or stack arrangement... |
| 16:18 |
MTDiscord |
<Warr1024> Heh, whatever it was I fixed it (maybe I was running an out of date binary). |
| 16:59 |
pgimeno |
reminder: FFI is available everywhere where LuaJIT is supported, and is independent of JIT. There's even an FFI library for pure Lua, written by Facebook: https://github.com/facebookarchive/luaffifb |
| 17:01 |
pgimeno |
sorry, not written by Facebook |
| 17:03 |
pgimeno |
that said, offering FFI poses security risks, but these could probably be alleviated with range-checking accessors |
| 17:11 |
MTDiscord |
<Jordach> or why not let let system permissions handle binary access |
| 17:13 |
|
pouellet joined #minetest-dev |
| 17:36 |
MTDiscord |
<Benrob0329> I found a different FFI library a while back that looked pretty good for PUC Lua, it aimed to address some issues with other libraries. One sec while I dig it up |
| 17:37 |
MTDiscord |
<Benrob0329> https://github.com/q66/cffi-lua |
| 17:41 |
MTDiscord |
<Benrob0329> Correction: For any PUC Compatible Lua implementation (LuaJIT included) |
| 17:54 |
|
appguru joined #minetest-dev |
| 18:15 |
MTDiscord |
<Lone_Wulf> Is there an issue for 5.4.0 minetest clients mistaking a huddef text field for an image field on 5.4.1 servers?? |
| 18:15 |
MTDiscord |
<Lone_Wulf> *? |
| 18:22 |
pgimeno |
@Benrob0329 LuaJIT's FFI has the advantage that it FLIES when it's jit-compiled. Having a compatible replacement for PUC Lua is just to (possibly, though unlikely) calm the voices that say that PUC Lua support should not be dropped. |
| 18:30 |
hecks |
ffi would be good for an inversion of the api, it may even be faster on puc |
| 18:30 |
hecks |
although i understand that builtin is already mostly structured in this way |
| 18:32 |
hecks |
still such an inverted api would basically be a fraction of the code to maintain |
| 18:32 |
hecks |
it's easier to maintain a binding to dllexported C functions than it is to manipulate the VM using the usual lua api |
| 18:41 |
MTDiscord |
<Lone_Wulf> nvm on the hud, was an issue on my side |
| 19:19 |
|
v-rob joined #minetest-dev |
| 19:45 |
v-rob |
You know, when c55 was doing tests of content_t as 32-bit and 64-bit, the 64-bit would be useless. Many systems only have 4 GB or less of RAM, so you'd run out of RAM long before hitting even the u32 limit (since 2^32 == 4 GB). If it were possible, a 24-bit content_t would be ideal since it would save a byte while not being possible to run out of them. But the only system I know that has 24-bit integers is the Zilog eZ80 :P |
| 19:47 |
pgimeno |
things are not always like they seem, LuaJIT has a limit of 2 GB of RAM in a 32 bit Linux and 1 GB in a 64 bit Linux |
| 19:48 |
pgimeno |
(LuaJIT 2.0 that is) |
| 19:48 |
v-rob |
I'm talking C++-side |
| 19:48 |
v-rob |
`MapNode` |
| 19:48 |
specing |
> while not being possible to run out of them. |
| 19:48 |
specing |
hold my beer |
| 19:50 |
specing |
If the limit was >>32k,then there would be a lot more tablesaw-supported nodes |
| 19:50 |
specing |
perhaps even finer nodes |
| 19:50 |
specing |
e.g. 3-stairs or something like that |
| 19:50 |
v-rob |
I know, u16 is too small, but u32 is overkill and will never be fully used |
| 19:51 |
v-rob |
Well, I guess a 16 < x < 32-bit content_t theoretically could be done with an `&` without being too slow. After all, u16s are often loaded from u32s with an `&` internally on many modern CPUs anyway. |
| 19:51 |
specing |
I also found that there is a lack of 1/16 slab nodes when you want to build something completely out of them |
| 19:51 |
specing |
e.g. theres no slope node that would start at 1/16 and go to full size |
| 19:58 |
v-rob |
Hmm, a good implementation for MapNode might be a bit-packed u64, like `class MapNode { u64 m_value; u32 getContent() { return m_value & CONTENT_BITS /* e.g. could be 24 bits */; } void setContent(u32 value) { m_value |= value & CONTENT_BITS } /* More methods for param fields, allowing arbitrary numbers of params up to the 64-bit limit */ }` if the bitwise operations don't slow things down. |
| 19:58 |
sfan5 |
the compiler can do that for you for free btw |
| 19:59 |
v-rob |
Sure, but not with arbitrary numbers of param fields. |
| 19:59 |
v-rob |
Oops, setContent is wrong. But the gist is there |
| 20:05 |
v-rob |
I still want better param fields since that seems to me to be a better solution than registering tons of nodes. After all, `ContentFeatures` is a really fat struct and takes up a lot of RAM. Params can handle it so much cleaner. |
| 20:06 |
v-rob |
Imagine if "facedir" was implemented as 32 separate nodes for every direction. For the same reason, I think other things should be moved to param fields. |
| 20:06 |
specing |
Yeah, the slabs should probably be params |
| 20:07 |
specing |
same node, different texture |
| 20:08 |
specing |
and then we probably wouldn't need >16bit node ids |
| 20:08 |
v-rob |
Then again, a problem with params is that they only apply to nodes in the world, not nodes in the inventory |
| 20:09 |
specing |
Would it be hard to add that to itemstacks? |
| 20:09 |
sfan5 |
no |
| 20:09 |
v-rob |
But do we want facedir preserved in itemstacks? I don't think so. |
| 20:09 |
specing |
ofc not |
| 20:09 |
specing |
but the texture info yes |
| 20:10 |
v-rob |
Ugh, there's no simple solution |
| 20:10 |
sfan5 |
uh |
| 20:10 |
sfan5 |
it all goes through Lua anyway so it'd be just as easy to have it only apply to nodes where it makes sense |
| 20:11 |
sfan5 |
and giving that choice to mods |
| 20:11 |
MTDiscord |
<Benrob0329> v-rob: One could also argue that most systems made within the past 5 years or so have > 4GB of ram, so I don't think that its really an argument. A better argument might be that having 4GB of nodedefs is stupid, but having the next biggest INT doesn't seem too bad to my (naive) thoughts. |
| 20:12 |
v-rob |
My thoughts on 24-bit integers is to leave more space for params since 32-bit isn't really necessary |
| 20:12 |
v-rob |
Otherwise, I'd just leave it at 32 for simplicity |
| 20:14 |
specing |
sfan5: yeah |
| 20:14 |
specing |
sfan5: register_node_variation(node_string, { ... }) |
| 20:17 |
v-rob |
Yeah, an alternative to making more params is to make it easier for Lua, certainly. |
| 20:20 |
specing |
Btw, I've heard someone say that it is possible to do C/C++ mods |
| 20:20 |
specing |
is there an example? |
| 20:21 |
v-rob |
It's called editing the C++ code and compiling it, I think :P |
| 20:27 |
v-rob |
It's too bad params can't be less hardcoded. At least adding multiple nodedefs if fully controllable by Lua. |
| 20:28 |
v-rob |
*if -> is |
| 20:28 |
v-rob |
Oh well, I'll blame all our problems on SSCSM or something and forget about nodes for now. |
| 20:31 |
specing |
v-rob: oh, ok |
| 20:32 |
specing |
I think ya'll are overcomplicating SSCSMs |
| 20:32 |
specing |
just make a central contentDB for them and make server owners publish them there, then they get reviewed and servers can request them to be [down]loaded |
| 20:33 |
sfan5 |
sure we'll "just" put lots of effort into a solution that doesn't scale |
| 20:34 |
v-rob |
Um, core devs already have how much trouble reviewing just C++ code? Why should we expect other people to volunteer to do it with a bunch of random people's code? |
| 20:34 |
specing |
you can think about a scalable solution after it becomes unmanageable |
| 20:34 |
specing |
right now you've been delaying this great player experience improvement for 5 years already |
| 20:34 |
v-rob |
I don't think such a thing will work for more than a month, tops |
| 20:35 |
specing |
you can tap the review power of other server owners, as many will also want to reuse it on theirs |
| 20:36 |
specing |
if you just implement direct download then it'll become a mess to track SSCSMs across servers and it will result in many duplicates being developed |
| 20:36 |
v-rob |
As I've said before, I believe SSCSM needs to be experimentally implemented under a disabled-by-default setting with a big, fat, warning label. Then, we can do everything at our leisure, slowly. Security issues can be tackled one at a time instead of all at once. Monolithic PRs aren't the solution. |
| 20:38 |
v-rob |
I'd do it myself, but I know very little about security, and I'm already in the process of rewriting the entire GUI, which is no small task. |
| 20:39 |
specing |
you don't have to do much about security if its centralised and reviewed |
| 20:42 |
sfan5 |
who's gonna school random community members on how to review security-critical code? you? |
| 20:42 |
specing |
actually yeah, I'm thinking about starting SSCSM support in my fork of MT |
| 20:44 |
specing |
sfan5: and don't say you don't have several trustworthy server owners who could take over this task |
| 20:44 |
sfan5 |
trust is not the issue, it's ability |
| 20:45 |
v-rob |
Security is notoriously hard to get right |
| 20:45 |
v-rob |
Professionals mess it up all the time |
| 20:45 |
v-rob |
I don't expect volunteers to not mess up, and all it takes is one mess up. |
| 20:45 |
v-rob |
Better to do the security issues once, in the engine. |
| 20:46 |
specing |
v-rob: why not do both? |
| 20:46 |
v-rob |
Because it's not feasible. |
| 20:46 |
specing |
that's what you say |
| 20:49 |
v-rob |
There are thousands of released mods, and many get updated all the time. If SSCSM is half as useful as it's been promoted, a significant portion of them will have it. Who's going to review them all and do it without mistakes? People who make SSCSM mods, client users, and server users can all exploit security issues. It's dangerous, and it will be too much of a bottleneck. |
| 20:51 |
v-rob |
If that's how SSCSM would work, I would never use any of them. I don't want any possibility of my computer being exploited, thank you. |
| 20:51 |
rubenwardy |
RE: CSMs on ContentDB: It often takes a week or more for new packages to be reviewed when editors are busy, it's not very feasible to require reviewing all new releases |
| 20:52 |
rubenwardy |
also c55 requires that connecting a server doesn't require any third party services |
| 20:53 |
sfan5 |
is CDB a third party? ? |
| 20:53 |
rubenwardy |
third-party in regards to the server |
| 21:31 |
|
specing joined #minetest-dev |
| 21:32 |
MTDiscord |
<exe_virus> Yeah definitely not for the review thing as a solution. Much more for taking things slowly and making it buyer beware. Hopefully in 1-2 years its a halfway decent api, but for now, it could simply be chat sounds haha |
| 21:49 |
|
queria^afk joined #minetest-dev |
| 22:05 |
|
Extex joined #minetest-dev |
| 22:10 |
|
v-rob joined #minetest-dev |
| 22:35 |
|
v-rob joined #minetest-dev |
| 23:15 |
|
Extex joined #minetest-dev |
| 23:34 |
|
v-rob joined #minetest-dev |
| 23:47 |
|
AliasAlreadyTake joined #minetest-dev |
