Time Nick Message 01:02 Pope_Rigby Is there a way to see which mod an item comes from? 01:11 MinetestBot sabino: Aug-06 06:46 UTC which version of the `pipeworks` mod are you running? And, about the wrench: which fork of `technic` are you using? afaik in the https://github.com/mt-mods/technic fork the wrench is almost completely rewritten and should work for almost everything chest-like 01:12 sabino hello! there is a version of minetest on microsoft store. Is the official? 01:12 sabino https://www.microsoft.com/store/productId/9NZ5P9FHDG1M 01:12 MTDiscord no 01:12 MTDiscord It wouldn't be 4 releases behind if it were official 01:13 MTDiscord Also it would almost certainly not cost money 01:15 sabino yeah, it was that I think. But the people shouldn't sell a project if they dont has a copyright 04:47 celeron55 yeah that's ridiculous. feels like M$ is trying to gouge my eyes out 04:49 celeron55 i wonder how easy it is to upload something there 04:50 celeron55 i think it could be worth it to me for the low price of $1/copy 04:50 MTDiscord krita, some other projects charge on the ms store, steam, for people who want the convince/want to donate to the project 04:51 MTDiscord that specific listing has been reported multiple times 04:51 celeron55 i really don't want to endorse that store though, or any other store 04:51 celeron55 so maybe 10$ just to keep people away 04:51 MTDiscord relevant https://github.com/minetest/minetest/issues/12605 04:54 MTDiscord krita charges $15 us, so $10 wouldnt be that bad https://apps.microsoft.com/store/detail/krita/9N6X57ZGRW96?hl=en-us&gl=US 04:55 MTDiscord if i assume they take the max, $1.50 us, still leaves you with $8.50 us per purchase 04:55 celeron55 we kind of need a minetest store team whose only task would be to spam MT into every crappy store there is every time a release is made. it's not something developers should have to think about 04:56 MTDiscord id volunteer 04:56 MTDiscord however, i think that the things pointed out in the issue should be resolved first 04:56 celeron55 and every store listing should say downloading directly from minetest.net is recommended and free 04:57 MTDiscord or you could just release it for free ? 04:58 celeron55 basically use the stores as advertisements for minetest.net 05:01 celeron55 well, devs probably aren't very interested in supporting crappy platforms that have crappy stores so the store team needs someone who can hack each release into working at the various stores and their platforms 06:51 air can shadows be toggled inside the game? 08:28 mrkubax10 air: you can use this mod https://content.minetest.net/packages/ROllerozxa/enable_shadows/ 08:30 air I meant a command or key to toggle them in the client 08:31 air I can type "/set enable_dynamic_shadows true/false" but I think the shaders need to be reloaded for it to work 08:32 mrkubax10 ah, then I'm not sure 09:08 AliasStillTaken Can someone help me un-riddle a couple of log messages? I get these even during login: 2022-08-16 09:07:23: VERBOSE[ConnectionSend]: con(4/1)RE-SENDING timed-out RELIABLE to 2001:9e8:339b:900:8484:e846:2f5a:db79(t/o=0.1): count=1, channel=0, seqnum=4132 09:08 AliasStillTaken That's my IP v6 address 09:16 celeron55 that means your or the server's network connection (or something in between, or the server itself) is dropping some packets. it's normal to a slight extent as it's the way programs detect available bandwidth, but if there's a lot then it could be a network issue 09:18 AliasStillTaken That would be odd - it's a 1 GBit connection serverside, which according to serverstats is hardly at its limit. On my ide its only 100 Mbit, but still enough bandwidth available. 09:19 AliasStillTaken Ofc real life distance doesn't always say something about network distance, but ping to the machine is 18 ms 09:19 AliasStillTaken And I live about 100 km away :D 09:20 Ingar that's less than 12 parsec! 09:21 AliasStillTaken What bothers me most is that my server has odd episodes of chatlag. With only 22 people on it, they feel their chat gets delayed by more than a minute occasionally 09:21 AliasStillTaken Sometimes, 40 people happily dig and fight and there's no lag whatsoever 09:22 AliasStillTaken 2022-08-16 08:53:21: VERBOSE[ConnectionSend]: con(4/1)Possible packet stall to peer id: 5401 queued_commands=59436 09:22 AliasStillTaken I was suggested to add prometheus support, I did that on the testserver. (same machine, only port 30001 instead) 09:23 celeron55 have you done the obvious and started logging the round trip time and packet loss to the server using ping 09:23 celeron55 and looked at what that looks like when there are problems 09:24 AliasStillTaken yes. ping is fine. server step is fine. the chatbridge that runs on the same machine gets the message way before we do in the client 09:24 celeron55 something is obviously eating the packets so i hope you're friends with the relevant network admins 09:24 AliasStillTaken It's odd: I send a mesasge. I can see in the log its there. Chatbridge hands it out. Only my (and the clients of *some* others) have a delay 09:25 celeron55 :D 09:25 AliasStillTaken It's a dedicated machine, the only clog we found was the MT server sending packets to clients 09:27 AliasStillTaken But that's an area I'm not familiar with. How does the server decide when to send a packet? On each entity movement? Or does it compile info each serverstep into one packet to send to a client? 09:28 celeron55 it sends separate packets for different things. as you can see the log message, it's re-sending a packet which means it didn't get an ACK from the client for a previous attempt 09:29 celeron55 i.e. it already tried, and the client didn't respond (i assume this is the server's log) 09:29 AliasStillTaken yes, its verbose log on the server 09:30 celeron55 this can of course happen also if the packets get to the client but the responses don't get to the server 09:31 AliasStillTaken I can watch clients from log. Even through those 1 minute lag episodes, whenever I do something on the client, it goes to the log immediately. 09:31 AliasStillTaken So, the connection from client to server is ok, right? The result is processed, if its a chatmesage, the bridge hands it out 09:32 AliasStillTaken But when the server wants to tell my client "hey, there's someone chatting, this is the mesage!" it doesn't reach the client 09:32 celeron55 that does sound like the problem is outgoing packets at the server, or incoming packets at the client 09:33 AliasStillTaken Is there some hardcoded limit? 09:33 AliasStillTaken It also says 2022-08-16 09:30:00: VERBOSE[ConnectionSend]: con(4/1)Ran out of sequence numbers! 09:35 celeron55 the sequence numbers are a 16 bit integer which rolls over from 65535 to 0 09:35 celeron55 if i recall correctly 09:36 celeron55 so maybe there are 65535 packets queued? seems like a lot 09:36 AliasStillTaken If they roll over, under what circumstances would they run out then? 09:36 celeron55 well you can't reuse a number before the packet has been acknowledged by the client 09:36 AliasStillTaken The server is not entirely lightweight :D I can read out any values that you want me to, but I wouldn't know which 09:37 proller ... still not switched to enet or other working proto.... ;) 09:37 celeron55 is there a chance it's just trying to send impossible large amounts of data to the client and MT's poor little network stack can't handle it? 09:38 AliasStillTaken How can I gauge this? 09:41 BuckarooBanzai last time i had issues like this i tested the connection with iperf3 (https://github.com/esnet/iperf) and udp always had some losses, about 10% iirc from my residential connection to a vps a couple 100km away 09:41 celeron55 the client logs a "Client packetcounter" list every 30 seconds 09:42 BuckarooBanzai but the engine did a good job re-transmitting those, never had too much network-lag problems 09:42 celeron55 you could take a look what that's showing 09:42 celeron55 the server doesn't have anything similar i think 09:42 AliasStillTaken I assume I should enable verbose log on the client, too? 09:42 AliasStillTaken Then what would I grep for? 09:43 celeron55 it's at the info log level 09:43 celeron55 that exact string, "Client packetcounter". the list comes after that as one line per packet type 09:43 celeron55 each line starting with "cmd" 09:47 AliasStillTaken https://pastebin.com/HTUXZ8Ha 09:48 AliasStillTaken A while later: 2022-08-16 11:47:37: INFO[Main]: cmd 67 (TOCLIENT_DETACHED_INVENTORY) count 10218 09:48 AliasStillTaken Are those "amount of poackets" ? 09:49 celeron55 67 is the packet type in its numeric form, 10218 is the number of packets since the previous print 09:49 celeron55 as a baseline that looks fine enough 09:49 celeron55 well, 10218 of TOCLIENT_DETACHED_INVENTORY seems like a lot 09:50 celeron55 but not like anything that would break it 09:50 celeron55 or 09:50 celeron55 well, depending on the number of players, it might, maybe? no idea 09:51 AliasStillTaken THis recoridng is not from a chatlag episode. They happen "randomly", so I'd need to wait for one of those and see if the numbers are different 09:51 celeron55 but i'd be looking if i could find more like 100 000 of some packet type 09:52 celeron55 if you do, then you run into the problem that the server doesn't collect statistics about how much packets each mod is causing to be sent 09:52 AliasStillTaken Ok, but if the server for some reason cannot send the packages out, then they wouldn't appear on the client anyways, right? 09:53 celeron55 well 09:53 celeron55 better just get the data first and then think 09:53 sfan5 !c 10218 / 30 09:53 MinetestBot 340.6 09:54 sfan5 as a rough number - expect for movement-related packets - more than 10 per second of any packet type is too much 09:54 sfan5 except* 09:54 celeron55 yes it's clearly a lot of players being active or a mod doing something it probably shouldn't 09:55 celeron55 but it shouldn't break the protocol 09:55 AliasStillTaken so this detached inventory (largest amount i've seen so far was 12553) is not so good? 09:55 celeron55 let's say there's a 50% chance you'll see a ridiculous number in that when the problem occurs 09:58 AliasStillTaken Alright. Anything else I could record/look at/read out when the next episode arrives? 09:58 sfan5 the entire network traffic with wireshark 09:59 AliasStillTaken The problem is not wiresharking the data. The problem is making sense of what I gathered 10:00 AliasStillTaken Regarding prometheus: What value would I look at there? 10:00 AliasStillTaken Unfortunately I only enabled it on the testserver yet 10:00 sfan5 not sure if it has something relevant out of the box 10:01 sfan5 re wireshark: the dissector can help here but you can also give the capture to me and I can interpret it for you 10:02 AliasStillTaken to add prometheus to the main server, I need to recompile anyways. If there's stuff we can add to get more relvant readings, I'd like to test them oin the tesserver first 10:05 AliasStillTaken lol. We just hit another chatlag episode, but the packet numbers went down 10:06 AliasStillTaken https://pastebin.com/kHUWBiSK 10:07 AliasStillTaken In this paste, the lag started between 2022-08-16 12:04:08 and 2022-08-16 12:04:38 10:09 AliasStillTaken Na packets in the 100k range. 10:09 MTDiscord Chatlag may not be related to packet problems 10:10 MTDiscord I can imagine that a bad mod causes lags in one case and sends a lot of packets in another 10:10 AliasStillTaken It's not only chatlag. All actions, inventory, fight, anything that is sent from server to client is delayed 10:10 MTDiscord Yeah, a classic server lag. Smth blocked the main thread. 10:11 MTDiscord It could be a network problem too tho 10:12 AliasStillTaken server is on a 1 Gbit connection, all other network related back and forth runs nicely 10:17 MTDiscord Doesn't prometheus have smth like server lag metric? You should look into that. 10:25 AliasStillTaken You mean minetest_core_latency ? 10:29 AliasStillTaken If I'm not terribly mistaken, those are the metrics I can see via prometheus: https://pastebin.com/M6Tx61BE 10:29 AliasStillTaken (disregard the values of the paste, they're from another test instance) 10:36 AliasStillTaken perf top gives me con::ReliablePacketBuffer::insert as one of the toppermost functions 10:37 AliasStillTaken (along with server::ActiveObjectMgr::getObjectsInsideRadius) 10:43 MTDiscord yes minetest_core_latency 10:46 AliasStillTaken That would be the same as the dtime value ingame, right? On the main server I set server step to 0.044, which is not affected by the chatlag episodes 11:00 MTDiscord chatlag doesn't affect dtime? 11:00 AliasStillTaken No. When I send a message ingame during chatlag, the chat bridge immedialte yrelay it to discord 11:01 MTDiscord but other players don't see it? 11:01 AliasStillTaken That tells me client to server is ok, server lua is ok, server to localhost http is ok, but server to clients is not ok 11:01 AliasStillTaken I don't see it myself until the server says I can have the message :D 11:02 AliasStillTaken That's how I measure delay between discord message and ingame message 11:04 MTDiscord sus 11:05 MTDiscord Then the server fails to send packets indeed 11:08 MTDiscord Probably the server thinks that's enough for me and drops all packets in the queue 11:09 MTDiscord That's why packet number counters went down 11:10 AliasStillTaken You mean client side? Yes, we came to the same conclusion 11:11 AliasStillTaken For some reason server says "I'd like to send out this bazillion of packets, but ... I don't" 11:12 AliasStillTaken dedicated_server_step = 0.044 11:12 AliasStillTaken <- Is that too fast maybe? 11:13 celeron55 i wouldn't set it that fast, there's really no benefit 11:13 AliasStillTaken Actually there is, people felt it was much more responsive 11:13 AliasStillTaken What's the minimum reasonable value? 11:14 celeron55 well, anything, as long as you know your game can take it 11:14 MTDiscord Is there anything in server logs during the chatlag about how many packets it dropped? 11:14 AliasStillTaken Same goes for max_packets_per_iteration? 11:16 celeron55 never played with that setting, i don't actually even know what "iteration" means 11:16 AliasStillTaken Since the lua thread is fine during chatlag episodes, the dtime during normal gameplay hardly ever increases beyond 0.1, only during heavy monster invasion events we get ticks of 0.2 or 0.3, I'd say the machine can take it. If you disagree and say better set it to x, I'll do 11:19 MTDiscord It can be that a mod changes smth every globalstep and it causes the server to send packets. So the less dedicated_server_step, the more packets. 11:20 AliasStillTaken Yes, ofc. But the quesiton is, why can't the server send out those packets? 11:21 appguru The server probably sends them out but they fail to reach the clients 11:21 appguru So the server tries to resend and the queue fills up 11:22 AliasStillTaken This would indicate a problem on clients to receive that many packets. I need to compare wireshark of the server and wireshark of the client 11:22 MTDiscord The queue is per client. They fail to reach all the clients? 11:23 AliasStillTaken At least some of them. But during those episodes, at least half of the population claims they feel lag 11:23 AliasStillTaken Also, we have messages like this: 2022-08-16 08:53:21: VERBOSE[ConnectionSend]: con(4/1)Possible packet stall to peer id: 5401 queued_commands=59436 11:24 AliasStillTaken Tehre is no way to match peer id to a playername? 11:24 celeron55 looks like max_packets_per_iteration divided by player count is the maximum number of commands to try to send per ConnectionSendThread loop iteration, and that loop doesn't run at a consistent rate but does a 50ms sleep between iterations except when it's waken up by the engine wanting to send a packet. so 50ms when nothing is sent but waiting for timeouts, otherwise no delay 11:24 celeron55 command is an MT network stack concept, it translates into one or more packets depending on how large the payload is 11:26 definitelya savilli: Yes, some of the authors of the mods may assume you are just hosting a server with lightly modded MTG, and at this scale the server just can't keep up. 11:26 celeron55 AliasStillTaken: grep the log for INFO[Server]: Server: New connection: "singleplayer" from 127.0.0.1 (peer_id=2) 11:27 AliasStillTaken Ok, thanks :) 11:32 AliasStillTaken Could that 50ms send the network thread into a sleep-wake cycle, when server step is lower than 50 ms? 11:33 celeron55 ? 11:33 celeron55 your problem is definitely not there 11:57 AliasStillTaken Someone tried to log in to the server, but supplied and empty password. I have a single attempt from this IP address in the log. However, this prompted a lot of those : 2022-08-16 11:53:40: VERBOSE[ConnectionSend]: con(4/1)RE-SENDING timed-out RELIABLE to ::ffff:HIS_IP_ADDRESS(t/o=0.708): count=3, channel=2, seqnum=24876 12:19 BuckarooBanzai AliasStillTaken: did you take a look at this: https://github.com/minetest/minetest/issues/10088#issuecomment-647723155 ? 12:23 BuckarooBanzai (re: the "timed-out RELIABLE" part) 12:25 AliasStillTaken Thanks, didn't see that yet 12:26 AliasStillTaken I'm still trying to wrap my head around wireshark atm 12:47 MinetestBot 02[git] 04rubenwardy -> 03minetest/minetest: Add package title to ContentDB error messages (#12705) 13c4ffe63 https://github.com/minetest/minetest/commit/c4ffe630f106712d5c655e642cfca10096efba3a (152022-08-16T12:46:11Z) 12:55 sfan5 AliasStillTaken: do you have the a pcap of the traffic during before/the lag 12:55 sfan5 eh messed up the word order 12:55 sfan5 I think you still get it 12:56 AliasStillTaken Yes :D I still experiment with how to generate a pcap file 12:56 sfan5 if you have the packets in wireshark you can just click save 12:57 AliasStillTaken That's what I struggle with. If you have a simple recipe, I could certainly put it to work. 12:59 AliasStillTaken Another drawback: Ever since the serverrestart, there was no chatlag episode -.- 13:01 sfan5 you're struggling with the "have packets in wireshark" or the "save it" part? 13:02 AliasStillTaken With the CLI of wireshark atm 13:02 sfan5 tshark? 13:04 AliasStillTaken Yes. Plus, some other questions around it. When I start logging, will that disconnect all the players? 13:07 BuckarooBanzai AliasStillTaken: nope, this _should_ not affect anyone, it is usually just `tshark -i -w ` (this will capture _all_ the traffic though, including non-encrypted stuff you may have running) 13:08 BuckarooBanzai this should work better: `tshark -i -w "port 30000"` 13:08 AliasStillTaken Will that not only capture incoming traffic? 13:09 BuckarooBanzai it captures everything on that interface with source- or destination-port 30000 13:15 AliasStillTaken Sounds like you're familiar with wireshark a bit. How can I make it that I capture the episodes? Is there a way to rotate the capture file ? 13:16 AliasStillTaken Like "the following 10 minutes everything goes to THIS file, the next 10 minutes everything goes to THAT file" ? 13:16 AliasStillTaken I mean, sure, I could let it run continuously, but that would give us gigabytes of captures, right? 13:17 AliasStillTaken Earlier today, the chatlag episodes were frequent, but after a serverrestart it takes a bit until the episodes start again 13:17 BuckarooBanzai don't let that thing run continuously, it has some memory-leak issues :) 13:17 BuckarooBanzai i usually just run it when something happens, otherwise you have TB's worth of captures and no idea where to start :/ 13:19 AliasStillTaken That means I might miss what happens immediately before and the first few seconds of the event 13:20 BuckarooBanzai there's a stop-condition parameter, let me check the --help quick... 13:20 sfan5 -b ..., --ring-buffer 13:20 sfan5 duration:NUM - switch to next file after NUM secs 13:20 BuckarooBanzai `-a duration:3600` to run it for an hour for example 13:21 BuckarooBanzai sfan5: you are too far below... 13:21 BuckarooBanzai ah, sorry, my bad 13:21 BuckarooBanzai no, this is exactly your use case :) 13:21 sfan5 -b does more than -a depending on what AliasStillTaken needs 13:22 sfan5 you also definitely want to run with -n and -M to fix the memory "leaks" 13:22 BuckarooBanzai yeah 13:22 sfan5 this would perhaps be easier with tcpdump 13:29 AliasStillTaken What's a nice value for -M? about 1 Million? 13:30 sfan5 dunno, maybe 5000 13:31 AliasStillTaken Alright. That's what I cobbled together: tshark -i eno1 -w /root/administration156.pcap -b duration:180 -n -M 5000 "port 30000" 13:33 sfan5 looks good 13:37 AliasStillTaken Ok ... now comes the fun part: How do I filter out all chat and private messages? 13:38 sfan5 if you have the minetest dissector installed in wireshark you can filter out all chat packets 13:38 AliasStillTaken How do I do that? 13:39 AliasStillTaken Sorry to bother you about such things, I have very little knowledge in ... a lot of parts of this adventure :D 13:47 sfan5 you copy util/wireshark/minetest.lua to ~/.wireshark/plugins/ and use the following filter 13:47 sfan5 !(minetest.server.command == 0x002f) 13:48 sfan5 its not perfect since it can only take the first of a split packet into account but should be good enough unless your users PM eachother entire novels 13:48 sfan5 you can also exclude auth information with types 0x51 0x52 and 0x60 13:51 sfan5 (those aren't cleartext anyway) 13:53 AliasStillTaken Hm, I don't have a .wireshark directory ... 14:19 MinetestBot 02[git] 04celeron55 -> 03minetest/minetest: Log sockets into tracestream instead of dstream (#12701) 133f67215 https://github.com/minetest/minetest/commit/3f67215df9dbc16f3bfe1ffc5c2582a308532914 (152022-08-16T14:18:11Z) 15:16 sfan5 you can create those two obviously 15:17 MTDiscord wireshark...you can check minetest packets? 15:28 MTDiscord Everyone can 16:14 diceLibrarian they're usp 16:14 diceLibrarian *udp 19:00 Guest2931 hi