Time Nick Message 00:00 lhofhansl but why do we need to store them. When the block is loaded time is ticking. When we load a block we set the time to now. 00:00 lhofhansl I guess we do not want to miss timers firing. 00:00 lhofhansl when a block got unloaded and then reloaded 00:02 sfan5 https://github.com/minetest/minetest/blob/3176daee79a8e91bd16fc275704f49fe1648db32/src/serverenvironment.cpp#L947-L961 00:02 sfan5 anyway, bedtime for me 00:04 hecks waaaait 00:04 hecks one question 00:04 hecks if I make a packet unreliable on the server side only, will the clients explode? 00:05 lhofhansl sfan5: heh, good night. 00:05 sfan5 the client doesn't care which channel a packet arrives on or if it's set reliable 00:05 hecks so I can apply this dirty fix server side? 00:06 sfan5 yes 00:06 hecks thanks =] 00:23 lhofhansl hecks: what's the "dirty fix"? 00:24 hecks make animations unreliable, for my queue problems 00:25 hecks it's not the end of the world if an animation fails to play, it is if lost anim packets clog up the reliable queue 00:25 hecks lhofhansl, which continent are you on? 00:28 MTDiscord probably europe 00:28 hecks hm, it seems that the dirty fix worked 00:29 hecks I'm starting to doubt if animation packets ever needed reliable delivery 00:30 hecks oh wait 00:32 hecks okay, now it's confirmed that it works 00:38 pgimeno would database compression changes have an effect on server bandwidth? 00:39 hecks you could always compress harder in transit if that were ever an issue 00:44 MTDiscord So, if I were to make a PR that forces .x, .obj, and .b3d to go through zlib compression and decompression to help with bandwidth load for new clients, would that create a breaking change requiring version 6.x? 00:45 MTDiscord Thinking is that compression for these using standard deflate is about 30% of original size or smaller. 00:51 hecks ah, crap, making animations unreliable did not entirely fix this 00:52 hecks something is still creating a black hole in the packet queue 01:08 hecks okay, so in many cases, only the last packet really matters 01:09 hecks maybe if the queue grows too large, the server could walk the whole thing and squash it to only what is actually important? 01:12 lhofhansl hecks: I'm US west coast. 01:53 hecks this is odd 01:53 hecks we tested the same game on the same connection months before, it worked and didn't chug this badly 02:00 hecks has anybody touched SAO messages, the packet queue or default settings related to bandwidth? 02:03 lhofhansl I increased the default packet queue sizes recently. 02:13 hecks what were they before? 02:18 lhofhansl https://github.com/minetest/minetest/pull/10627 02:18 lhofhansl If that causes a lag we have a serious problem. 02:59 hecks it did get better when I nerfed that 03:00 hecks I might still have to fix the queue thing... it's not going to go away no matter how you budget the up bandwidth 03:01 hecks it would be very helpful if you were to connect again and tell me if there's any difference 12:41 zughy[m] uhm, a ton of PRs result being updated 3 hours ago, we're talking about 5 pages of PRs, when they actually didn't. What happened exactly? 12:42 zughy[m] the only thing happened at that time is this: https://github.com/minetest/minetest/commit/ecd4f45318e7e510ebe4ebe8420ea739122d2edf 12:43 zughy[m] exactly one minute before 12:43 sfan5 github bug, it does that when you force-push master 19:00 Krock will merge #9247, #10633 and #10663 in 15 minutes 19:01 ShadowBot https://github.com/minetest/minetest/issues/9247 -- Fix MSAA stripes by HybridDog 19:01 ShadowBot https://github.com/minetest/minetest/issues/10633 -- Cross-reference the node level manipulation functions by Oblomov 19:01 ShadowBot https://github.com/minetest/minetest/issues/10663 -- Cleanup shader generation code by numberZero 19:15 Krock merging 19:17 Krock last PR failed. new merge conflict \o/ 19:24 lhofhansl After 10663 I 19:25 lhofhansl let's have a look at #10688 19:25 ShadowBot https://github.com/minetest/minetest/issues/10688 -- Remove hardcoded matrices from OpenGL shaders. by lhofhansl 19:34 Krock !tell lhofhansl None, though I'm not familiar with shaders to judge whether it's a good concept 19:34 ShadowBot Krock: O.K. 20:24 hecks Tell me about packet channels, how hard would it be to allocate a new one and move some traffic to it? 20:32 Krock easy 20:32 Krock see networkprotocol.h or so 20:34 Krock hecks: clientopcodes.cpp, and server respectively 20:40 hecks so you just change the channel number and that makes a new queue? 20:40 Krock yes 20:40 hecks thanks 20:40 hecks I'm gonna experiment with splitting the big queue because most of those messages do not need to wait for one another 20:41 Krock please keep in mind that some actions must be in order so that mods work properly, such as object properties and attachments 20:42 hecks I know, but those things do not need to wait, for example, on chat or forms 20:42 hecks or skybox operations, camera parameters 20:44 MTDiscord hello: was wondering if i can get some input from the core devs on https://github.com/minetest/minetest/pull/10697 if you dont mind? 20:44 hecks the server sends out an order of magnitude more data per client than it receives, and this out stream must be managed wisely because it's too easy to clog it beyond repair 20:57 celeron55 i'm a bit freaked out about https://github.com/minetest/minetest/pull/10627 actually 20:58 Krock more traffic 20:58 celeron55 there's a reason for the original limit, and it's exactly to allow the server to work in most network environments 20:59 celeron55 really shouldn't just increase it based on some tests in a local or good network 20:59 celeron55 and based on tests that don't include trying to animate stuff at the same time or such 21:00 celeron55 and yes, the network stack is crap, something like enet would handle more stuff better, but not infinitely better 21:01 celeron55 it's 100% certain that if you optimize for the rate of sending mapblocks without looking any other statistics or trying to do anything else at the same time, everything else _will_ work badly with the configuration 21:03 celeron55 it's like pressing the gas until you can't steer 21:04 celeron55 it could be dynamic though 21:10 Krock the mapblock sending is dynamic to the connected players 21:11 Krock -> https://github.com/minetest/minetest/pull/6393 21:14 hecks https://pastebin.com/SbWtmqV8 What do you think of these categories for s->c traffic? 21:14 Krock looks like an ENUM is needed in the code 21:14 hecks for the channels? 21:15 hecks I can do that 21:15 Krock yeah 21:15 hecks but how's this for a draft of message categories, any logical conflicts here? 21:15 Krock I'm sure there will be some obscure edge-cases but overall it seems okay 21:18 Krock what happens if you spawn an entity and set its position right after? 21:19 Krock afaik it's contained in TOCLIENT_ACTIVE_OBJECT_MESSAGES, so that would be okay 21:22 Krock or sending a formspec after creating the node. item lists should re-generate themselves on refresh, though that might only happen after resizing the window or pressing a button 21:24 Krock to be fair, most of those packets are only used once, especially the auth stuff will never be called twice per session 21:35 lhofhansl Looks like I should just undo #10627...? Too much discussion about it. 21:35 ShadowBot https://github.com/minetest/minetest/issues/10627 -- Increase limit for simultaneous blocks sent per client and the meshgen cache. by lhofhansl 21:36 lhofhansl The intent was to allow the server CPU to be busy, and not just idly wait for queues to empty out. 21:37 lhofhansl That is especially an issue with larger client viewing_range (and matching server max_block_send_distance). The current limits seem to low for that. 21:40 Krock depends on the network structure. celeron55, what would you suggest there? 21:41 lhofhansl Very related to what hecks is looking at. A single queue for everything is bad. 21:41 hecks I'm already breaking it up 21:41 lhofhansl But there's a PR on client stutter already, hecks seen packet build-up, celeron's concern, etc. Perhaps better to revert and then regroup. 21:42 lhofhansl What I'm looking at is getting MT ready for larger viewing_distances. 21:42 hecks well, what I'm doing is only tangentially related to your default limit bump, it's a much more fundamental issue 21:43 hecks the limit bump makes it a little more visible by taking up upload bandwidth from the server, I've had it happen before that too 21:44 hecks lhofhansl: we could cooperate on the LOD and map renderer rewrite when everything else is ready for it 21:44 lhofhansl hecks: Yep! 21:45 hecks the last one was a false start because the block management just wasn't good enough to handle it 21:46 lhofhansl Perhaps have a separate CLIENT_BLOCKSIZE and then build client blocks from the server blocks (with MAP_BLOCKSIZE). 21:47 lhofhansl Can be done off the main thread. But then other things are tricky. 21:47 hecks let's just do the octree thing 21:48 hecks blocksize 16 is fine for generation and transfer, it's enough to merge LODs into superblocks 21:48 hecks and then do some atlasing tricks for the closest blocks 21:48 lhofhansl cool 21:48 hecks I know that atlasing has been tried and flopped before but I got an idea to make it viable 21:48 lhofhansl main goal would to reduce the number of draw calls on the client. 21:50 lhofhansl I can easily draw a scene at range 700 at 60 FPS if set the MAP_BLOCKSIZE to 64. Same number of vertices, etc, just fewer draw calls (it does keep my graphics card more busy) 21:50 lhofhansl celeron55: so revert #10627 for now? 21:50 ShadowBot https://github.com/minetest/minetest/issues/10627 -- Increase limit for simultaneous blocks sent per client and the meshgen cache. by lhofhansl 21:51 hecks we've basically done this whole drawcall and blocksize investigation like half a year ago, so everyone knows what's up 21:51 lhofhansl It's just a default anyway. Folks can increase it for fast networks. 21:52 lhofhansl hecks: that one was news to me, but I was gone for a bit too. 21:52 lhofhansl I realized that MT cannot keep graphics card busy due to all the drawcalls. 21:53 hecks yup, we're wasting gigaflops of perfectly good silicon 21:53 hecks as for that default limit thing, I'm not sure if the logic was right on that one 21:53 lhofhansl how so? 21:53 hecks I mean, there's nothing wrong if the server can generate blocks faster than they're being sent 21:54 lhofhansl well, it doesn't. It then waits for the queue to empty, so when I increased that value the map would be sent to the client faster. 21:54 hecks but as you can see, the upload bandwidth budget is serious business 21:55 lhofhansl yeah, but that's another issue. That increase brought load time for a scene from 35s to 18s (that on a local network). 21:56 lhofhansl Anyway, as I said, happy to revert that change. 21:56 hecks maybe it doesn't need reverting, hang on 21:56 lhofhansl * hangs on 21:57 hecks you'd be better off asking people who run actual servers and not 1 mbps poverty boxes 21:57 lhofhansl or dialup :) 21:58 hecks there's actually another issue here 21:59 hecks do network messages have any concept of priority at all? 21:59 hecks like gameplay before blocks, close blocks before far blocks 22:00 Krock block updates are already handled this way 22:00 hecks but do they wait for gameplay? 22:01 Krock I don't understand. why would block updates wait? 22:01 Krock they're already a separate channel 22:02 hecks they shouldn't wait in that sense, but if the server is short on bandwidth, it could prioritize certain channels 22:02 hecks throttle the block updates a bit to prevent other things from feeling laggy 22:02 Krock uhm.. I guess it just sends whatever's waiting 22:02 hecks yup, and that might need to change 22:03 hecks okay, let's build this monstrosity 22:04 hecks segfault =] 22:10 lhofhansl Also there's #10616, which increases (doubles) the default emerge queue sizes, and only allows 1/2 of the total queue size for emege requests on behalf of active objects. 22:10 ShadowBot https://github.com/minetest/minetest/issues/10616 -- Increase default emerge queue limits and limit enqueue requests for active blocks. by lhofhansl 22:10 hecks ah, so there *is* a CHANNEL_COUNT 22:16 hecks okay, localhost seems happy 22:56 hecks well I'll be damned, this might be working 22:57 hecks lhofhansl: you got time to try the server again? 23:01 lhofhansl not now, sorry. What did you change? 23:02 hecks https://github.com/hecktest/minetest/tree/queue-death 23:03 hecks there are now 16 channels instead of 3 23:04 hecks chat, audio, physics, entities, modchannels, camera, ambience, fx do not wait on one another 23:04 hecks it seems to prevent a hopeless choke, at least on my setup 23:05 sfan5 https://pastebin.com/SbWtmqV8 What do you think of these categories for s->c traffic? 23:05 sfan5 seems sane from a quick look but just throwing more channels at the problem isn't done lightly 23:06 hecks well guess what, I just did it and it worked 23:06 sfan5 I didn't say it doesn't work 23:06 hecks =] 23:06 sfan5 Krock: do you think TOCLIENT_NODEMETA_CHANGED should be on channel 2? I can make a commit if you agree. 23:07 sfan5 current ones for reference: https://github.com/minetest/minetest/blob/master/src/network/serveropcodes.cpp#L215 23:16 hecks sfan5: but what's the problem exactly? Are channels expensive? Are you worried about things being too out of sync? 23:17 sfan5 someone needs to bother to write the compatibility code 23:17 hecks hm, wait, do clients need a rebuild? 23:18 sfan5 there's an assert for {channel count from the packet} < CHANNEL_COUNT in the networking code 23:18 sfan5 also I'd rather have the packet loss issue fixed (also https://github.com/minetest/minetest/issues/9107) than adding 16+ channels for lots of things that could ordinarily be just fine in one 23:19 hecks that's not what I'm trying to solve 23:19 hecks #10694 23:19 ShadowBot https://github.com/minetest/minetest/issues/10694 -- Reliable packet queue can't recover after getting jammed 23:19 sfan5 it's somewhat related 23:20 sfan5 or maybe not actually 23:20 hecks not really, even if you deal with packet loss, it won't help the "dropped for one second" case 23:20 sfan5 ignore that 23:20 hecks this is like when you get a "connection interrupted" in quake 23:20 sfan5 yeah perhaps the server knowing about which packets it can throw away would be better for that 23:20 hecks minetest currently just hammers the client with more packets during that time and that can't ever be caught up with 23:20 hecks that would be a step 2 to fixing this, compacting the queue after a soft disconnect 23:21 hecks but splitting the queues is also a logically sound idea, why the hell is chat waiting on entity attachment 23:21 hecks constraining the channel count with an assert may have been a very stupid idea 23:22 hecks but let's just eat the compat problem, and make it so that further channel count bumps do not cost as much 23:22 sfan5 it's not just an assert, the client code has exactly as many queues as CHANNEL_COUNT 23:22 sfan5 so channels[3] points into the wild west (and the compiler might yell at you) 23:22 hecks well then, it might need some reengineering to make it dynamic 23:23 hecks I don't expect more fundamental channel count changes, but 23:23 hecks certain "channels" like entities and modchannels are a whole new beast; ideally they should be dynamically allocated 23:23 sfan5 I can understand modchannels but how would you split up entities? 23:24 hecks well, logically, two entities that do not interact with one another should not wait on one another 23:25 sfan5 right, so one per entity 23:25 hecks but I also have a bad feeling about attachments 23:25 hecks that area might need extra sanity checks if entities are desynced from one another 23:25 hecks other than that, entities do not seem to interact with one another much 23:26 hecks one channel per entity is okay, I guess entity messages mostly make sense to arrive in order 23:26 hecks and instead, I'd prune the queue aggressively to reduce the bandwidth there 23:27 hecks so let's see... change all the channel arrays to vectors 23:28 hecks negotiate MAX_CHANNELS between the client and the server 23:28 hecks if the client uses an old protocol, emulate the old 3-channel setup 23:28 hecks it's trivial to make a legacy channel map 23:29 hecks does that satisfy compat? sfan5 23:29 sfan5 it does 23:30 sfan5 (there's nothing to negotiate though, the server says it wants 255 channels and the client can decide if it likes that or not) 23:31 hecks oh, honestly, it could just be a map<> and not a vector 23:32 hecks but, but 23:32 hecks the channel count is a u8 right? 23:32 sfan5 yes 23:32 hecks and this is a part of the packet format? 23:32 sfan5 yes 23:32 sfan5 are you suggesting more than 255 might make sense 23:33 hecks yes, for the entity and modchannel dynamic allocation 23:33 hecks let's utf8 this then: channel 255 is an escape for a bigger channel id 23:34 hecks ideally if the channel array is now a map or something like that, entities and modchannels can each get a whole range to allocate from 23:34 hecks or, actually, there's another way 23:35 hecks wait let me check if the id isn't a part of the packet in both cases 23:36 hecks okay so the active object messages have a u16 id right at the beginning 23:37 hecks and modchannels are keyed by string 23:37 hecks making modchannels dynamic can wait 23:38 hecks so, I guess it would be acceptable to just make another queue map keyed by object id and leave it at that 23:39 hecks I'm guessing that the u16 is an absolute limit for all active objects observed by players on the whole server, + the players themselves? 23:40 sfan5 yes but minus the "observed" 23:40 sfan5 or rather s|observed|loaded/considered active| 23:41 hecks that's a sad limitation but oh well, it can be dealt with later 23:41 hecks no server exists yet that could hit it but it's realistically reachable 23:43 hecks the biggest currently active server has 92 players; this translates to a budget of 712 entities per player, still a reasonable count 23:45 hecks but who knows what the future will bring =]