Minetest logo

IRC log for #minetest, 2022-08-16

| Channels | #minetest index | Today | | Google Search | Plaintext

All times shown according to UTC.

Time Nick Message
00:00 AliasStillTaken joined #minetest
01:01 smk joined #minetest
01:02 Pope_Rigby Is there a way to see which mod an item comes from?
01:11 sabino joined #minetest
01:11 MinetestBot sabino: Aug-06 06:46 UTC <BuckarooBanzai> which version of the `pipeworks` mod are you running? And, about the wrench: which fork of `technic` are you using? afaik in the https://github.com/mt-mods/technic fork the wrench is almost completely rewritten and should work for almost everything chest-like
01:12 sabino hello! there is a version of minetest on microsoft store. Is the official?
01:12 sabino https://www.microsoft.com/store/productId/9NZ5P9FHDG1M
01:12 MTDiscord <Jonathon> no
01:12 MTDiscord <Warr1024> It wouldn't be 4 releases behind if it were official
01:13 MTDiscord <Warr1024> Also it would almost certainly not cost money
01:15 sabino yeah, it was that I think. But the people shouldn't sell a project if they dont has a copyright
01:20 Noisytoot joined #minetest
02:00 beanzilla joined #minetest
02:03 sparky4 joined #minetest
02:06 stevet2 joined #minetest
02:15 stevet joined #minetest
02:40 diceLibrarian joined #minetest
03:26 Verticen joined #minetest
04:00 MTDiscord joined #minetest
04:29 air joined #minetest
04:38 specing joined #minetest
04:47 celeron55 yeah that's ridiculous. feels like M$ is trying to gouge my eyes out
04:49 celeron55 i wonder how easy it is to upload something there
04:50 celeron55 i think it could be worth it to me for the low price of $1/copy
04:50 MTDiscord <Jonathon> krita, some other projects charge on the ms store, steam, for people who want the convince/want to donate to the project
04:51 MTDiscord <Jonathon> that specific listing has been reported multiple times
04:51 celeron55 i really don't want to endorse that store though, or any other store
04:51 celeron55 so maybe 10$ just to keep people away
04:51 MTDiscord <Jonathon> relevant https://github.com/minetest/minetest/issues/12605
04:54 MTDiscord <Jonathon> krita charges $15 us, so $10 wouldnt be that bad https://apps.microsoft.com/store/detail/krita/9N6X57ZGRW96?hl=en-us&amp;gl=US
04:55 MTDiscord <Jonathon> if i assume they take the max, $1.50 us, still leaves you with $8.50 us per purchase
04:55 celeron55 we kind of need a minetest store team whose only task would be to spam MT into every crappy store there is every time a release is made. it's not something developers should have to think about
04:56 MTDiscord <Jonathon> id volunteer
04:56 MTDiscord <Jonathon> however, i think that the things pointed out in the issue should be resolved first
04:56 celeron55 and every store listing should say downloading directly from minetest.net is recommended and free
04:57 MTDiscord <Jonathon> or you could just release it for free ?
04:58 celeron55 basically use the stores as advertisements for minetest.net
05:01 celeron55 well, devs probably aren't very interested in supporting crappy platforms that have crappy stores so the store team needs someone who can hack each release into working at the various stores and their platforms
05:05 calcul0n_ joined #minetest
05:13 TomTom joined #minetest
06:17 Lesha_Vel joined #minetest
06:28 definitelya joined #minetest
06:51 air can shadows be toggled inside the game?
07:05 mrkubax10 joined #minetest
07:39 mrkubax10 joined #minetest
08:05 sometalgoo joined #minetest
08:24 Verticen joined #minetest
08:28 mrkubax10 air: you can use this mod https://content.minetest.net/packages/ROllerozxa/enable_shadows/
08:30 air I meant a command or key to toggle them in the client
08:31 air I can type "/set enable_dynamic_shadows true/false" but I think the shaders need to be reloaded for it to work
08:32 mrkubax10 ah, then I'm not sure
08:52 proller joined #minetest
08:59 lemonzest joined #minetest
09:08 AliasStillTaken Can someone help me un-riddle a couple of log messages? I get these even during login: 2022-08-16 09:07:23: VERBOSE[ConnectionSend]: con(4/1)RE-SENDING timed-out RELIABLE to 2001:9e8:339b:900:8484:e846:2f5a:db79(t/o=0.1): count=1, channel=0, seqnum=4132
09:08 AliasStillTaken That's my IP v6 address
09:16 celeron55 that means your or the server's network connection (or something in between, or the server itself) is dropping some packets. it's normal to a slight extent as it's the way programs detect available bandwidth, but if there's a lot then it could be a network issue
09:18 AliasStillTaken That would be odd - it's a 1 GBit connection serverside, which according to serverstats is hardly at its limit. On my ide its only 100 Mbit, but still enough bandwidth available.
09:19 AliasStillTaken Ofc real life distance doesn't always say something about network distance, but ping to the machine is 18 ms
09:19 AliasStillTaken And I live about 100 km away :D
09:20 Ingar that's less than 12 parsec!
09:21 AliasStillTaken What bothers me most is that my server has odd episodes of chatlag. With only 22 people on it, they feel their chat gets delayed by more than a minute occasionally
09:21 AliasStillTaken Sometimes, 40 people happily dig and fight and there's no lag whatsoever
09:22 AliasStillTaken 2022-08-16 08:53:21: VERBOSE[ConnectionSend]: con(4/1)Possible packet stall to peer id: 5401 queued_commands=59436
09:22 AliasStillTaken I was suggested to add prometheus support, I did that on the testserver. (same machine, only port 30001 instead)
09:23 celeron55 have you done the obvious and started logging the round trip time and packet loss to the server using ping
09:23 celeron55 and looked at what that looks like when there are problems
09:24 AliasStillTaken yes. ping is fine. server step is fine. the chatbridge that runs on the same machine gets the message way before we do in the client
09:24 celeron55 something is obviously eating the packets so i hope you're friends with the relevant network admins
09:24 AliasStillTaken It's odd: I send a mesasge. I can see in the log its there. Chatbridge hands it out. Only my (and the clients of *some* others) have a delay
09:25 celeron55 :D
09:25 AliasStillTaken It's a dedicated machine, the only clog we found was the MT server sending packets to clients
09:27 AliasStillTaken But that's an area I'm not familiar with. How does the server decide when to send a packet? On each entity movement? Or does it compile info each serverstep into one packet to send to a client?
09:28 celeron55 it sends separate packets for different things. as you can see the log message, it's re-sending a packet which means it didn't get an ACK from the client for a previous attempt
09:29 celeron55 i.e. it already tried, and the client didn't respond (i assume this is the server's log)
09:29 AliasStillTaken yes, its verbose log on the server
09:30 celeron55 this can of course happen also if the packets get to the client but the responses don't get to the server
09:31 AliasStillTaken I can watch clients from log. Even through those 1 minute lag episodes, whenever I do something on the client, it goes to the log immediately.
09:31 AliasStillTaken So, the connection from client to server is ok, right? The result is processed, if its a chatmesage, the bridge hands it out
09:32 AliasStillTaken But when the server wants to tell my client "hey, there's someone chatting, this is the mesage!" it doesn't reach the client
09:32 celeron55 that does sound like the problem is outgoing packets at the server, or incoming packets at the client
09:33 AliasStillTaken Is there some hardcoded limit?
09:33 AliasStillTaken It also says 2022-08-16 09:30:00: VERBOSE[ConnectionSend]: con(4/1)Ran out of sequence numbers!
09:35 celeron55 the sequence numbers are a 16 bit integer which rolls over from 65535 to 0
09:35 celeron55 if i recall correctly
09:36 celeron55 so maybe there are 65535 packets queued? seems like a lot
09:36 AliasStillTaken If they roll over, under what circumstances would they run out then?
09:36 celeron55 well you can't reuse a number before the packet has been acknowledged by the client
09:36 AliasStillTaken The server is not entirely lightweight :D I can read out any values that you want me to, but I wouldn't know which
09:37 proller ... still not switched to enet or other working proto.... ;)
09:37 celeron55 is there a chance it's just trying to send impossible large amounts of data to the client and MT's poor little network stack can't handle it?
09:38 AliasStillTaken How can I gauge this?
09:41 BuckarooBanzai last time i had issues like this i tested the connection with iperf3 (https://github.com/esnet/iperf) and udp always had some losses, about 10% iirc from my residential connection to a vps a couple 100km away
09:41 celeron55 the client logs a "Client packetcounter" list every 30 seconds
09:42 BuckarooBanzai but the engine did a good job re-transmitting those, never had too much network-lag problems
09:42 celeron55 you could take a look what that's showing
09:42 celeron55 the server doesn't have anything similar i think
09:42 AliasStillTaken I assume I should enable verbose log on the client, too?
09:42 AliasStillTaken Then what would I grep for?
09:43 celeron55 it's at the info log level
09:43 celeron55 that exact string, "Client packetcounter". the list comes after that as one line per packet type
09:43 celeron55 each line starting with "cmd"
09:47 AliasStillTaken https://pastebin.com/HTUXZ8Ha
09:48 AliasStillTaken A while later: 2022-08-16 11:47:37: INFO[Main]: cmd 67 (TOCLIENT_DETACHED_INVENTORY) count 10218
09:48 AliasStillTaken Are those "amount of poackets" ?
09:49 celeron55 67 is the packet type in its numeric form, 10218 is the number of packets since the previous print
09:49 celeron55 as a baseline that looks fine enough
09:49 celeron55 well, 10218 of TOCLIENT_DETACHED_INVENTORY seems like a lot
09:50 celeron55 but not like anything that would break it
09:50 celeron55 or
09:50 celeron55 well, depending on the number of players, it might, maybe? no idea
09:51 AliasStillTaken THis recoridng is not from a chatlag episode. They happen "randomly", so I'd need to wait for one of those and see if the numbers are different
09:51 celeron55 but i'd be looking if i could find more like 100 000 of some packet type
09:52 celeron55 if you do, then you run into the problem that the server doesn't collect statistics about how much packets each mod is causing to be sent
09:52 AliasStillTaken Ok, but if the server for some reason cannot send the packages out, then they wouldn't appear on the client anyways, right?
09:53 celeron55 well
09:53 celeron55 better just get the data first and then think
09:53 sfan5 !c 10218 / 30
09:53 MinetestBot 340.6
09:54 sfan5 as a rough number - expect for movement-related packets - more than 10 per second of any packet type is too much
09:54 sfan5 except*
09:54 celeron55 yes it's clearly a lot of players being active or a mod doing something it probably shouldn't
09:55 celeron55 but it shouldn't break the protocol
09:55 AliasStillTaken so this detached inventory (largest amount i've seen so far was 12553) is not so good?
09:55 celeron55 let's say there's a 50% chance you'll see a ridiculous number in that when the problem occurs
09:57 Flabb joined #minetest
09:58 AliasStillTaken Alright. Anything else I could record/look at/read out when the next episode arrives?
09:58 sfan5 the entire network traffic with wireshark
09:59 AliasStillTaken The problem is not wiresharking the data. The problem is making sense of what I gathered
10:00 AliasStillTaken Regarding prometheus: What value would I look at there?
10:00 AliasStillTaken Unfortunately I only enabled it on the testserver yet
10:00 sfan5 not sure if it has something relevant out of the box
10:01 sfan5 re wireshark: the dissector can help here but you can also give the capture to me and I can interpret it for you
10:02 AliasStillTaken to add prometheus to the main server, I need to recompile anyways. If there's stuff we can add to get more relvant readings, I'd like to test them oin the tesserver first
10:05 AliasStillTaken lol. We just hit another chatlag episode, but the packet numbers went down
10:06 AliasStillTaken https://pastebin.com/kHUWBiSK
10:07 AliasStillTaken In this paste, the lag started between 2022-08-16 12:04:08 and 2022-08-16 12:04:38
10:09 AliasStillTaken Na packets in the 100k range.
10:09 MTDiscord <savilli> Chatlag may not be related to packet problems
10:10 MTDiscord <savilli> I can imagine that a bad mod causes lags in one case and sends a lot of packets in another
10:10 AliasStillTaken It's not only chatlag. All actions, inventory, fight, anything that is sent from server to client is delayed
10:10 MTDiscord <savilli> Yeah, a classic server lag. Smth blocked the main thread.
10:11 MTDiscord <savilli> It could be a network problem too tho
10:12 AliasStillTaken server is on a 1 Gbit connection, all other network related back and forth runs nicely
10:17 MTDiscord <savilli> Doesn't prometheus have smth like server lag metric? You should look into that.
10:25 AliasStillTaken You mean minetest_core_latency ?
10:29 AliasStillTaken If I'm not terribly mistaken, those are the metrics I can see via prometheus: https://pastebin.com/M6Tx61BE
10:29 AliasStillTaken (disregard the values of the paste, they're from another test instance)
10:36 AliasStillTaken perf top gives me con::ReliablePacketBuffer::insert as one of the toppermost functions
10:37 AliasStillTaken (along with server::ActiveObjectMgr::getObjectsInsideRadius)
10:43 MTDiscord <savilli> yes minetest_core_latency
10:46 AliasStillTaken That would be the same as the dtime value ingame, right? On the main server I set server step to 0.044, which is not affected by the chatlag episodes
11:00 MTDiscord <savilli> chatlag doesn't affect dtime?
11:00 AliasStillTaken No. When I send a message ingame during chatlag, the chat bridge immedialte yrelay it to discord
11:01 MTDiscord <savilli> but other players don't see it?
11:01 AliasStillTaken That tells me client to server is ok, server lua is ok, server to localhost http is ok, but server to clients is not ok
11:01 AliasStillTaken I don't see it myself until the server says I can have the message :D
11:02 AliasStillTaken That's how I measure delay between discord message and ingame message
11:04 MTDiscord <savilli> sus
11:05 MTDiscord <savilli> Then the server fails to send packets indeed
11:08 MTDiscord <savilli> Probably the server thinks that's enough for me and drops all packets in the queue
11:09 MTDiscord <savilli> That's why packet number counters went down
11:10 AliasStillTaken You mean client side? Yes, we came to the same conclusion
11:11 AliasStillTaken For some reason server says "I'd like to send out this bazillion of packets, but ... I don't"
11:12 AliasStillTaken dedicated_server_step = 0.044
11:12 AliasStillTaken <- Is that too fast maybe?
11:12 mrkubax10 joined #minetest
11:13 celeron55 i wouldn't set it that fast, there's really no benefit
11:13 AliasStillTaken Actually there is, people felt it was much more responsive
11:13 AliasStillTaken What's the minimum reasonable value?
11:14 celeron55 well, anything, as long as you know your game can take it
11:14 MTDiscord <savilli> Is there anything in server logs during the chatlag about how many packets it dropped?
11:14 AliasStillTaken Same goes for max_packets_per_iteration?
11:16 celeron55 never played with that setting, i don't actually even know what "iteration" means
11:16 AliasStillTaken Since the lua thread is fine during chatlag episodes, the dtime during normal gameplay hardly ever increases beyond 0.1, only during heavy monster invasion events we get ticks of 0.2 or 0.3, I'd say the machine can take it. If you disagree and say better set it to x, I'll do
11:17 appguru joined #minetest
11:19 MTDiscord <savilli> It can be that a mod changes smth every globalstep and it causes the server to send packets. So the less dedicated_server_step, the more packets.
11:20 AliasStillTaken Yes, ofc. But the quesiton is, why can't the server send out those packets?
11:21 appguru The server probably sends them out but they fail to reach the clients
11:21 appguru So the server tries to resend and the queue fills up
11:21 mrkubax10 joined #minetest
11:22 AliasStillTaken This would indicate a problem on clients to receive that many packets. I need to compare wireshark of the server and wireshark of the client
11:22 MTDiscord <savilli> The queue is per client. They fail to reach all the clients?
11:23 AliasStillTaken At least some of them. But during those episodes, at least half of the population claims they feel lag
11:23 AliasStillTaken Also, we have messages like this: 2022-08-16 08:53:21: VERBOSE[ConnectionSend]: con(4/1)Possible packet stall to peer id: 5401 queued_commands=59436
11:24 AliasStillTaken Tehre is no way to match peer id to a playername?
11:24 celeron55 looks like max_packets_per_iteration divided by player count is the maximum number of commands to try to send per ConnectionSendThread loop iteration, and that loop doesn't run at a consistent rate but does a 50ms sleep between iterations except when it's waken up by the engine wanting to send a packet. so 50ms when nothing is sent but waiting for timeouts, otherwise no delay
11:24 celeron55 command is an MT network stack concept, it translates into one or more packets depending on how large the payload is
11:26 definitelya savilli: Yes, some of the authors of the mods may assume you are just hosting a server with lightly modded MTG, and at this scale the server just can't keep up.
11:26 celeron55 AliasStillTaken: grep the log for INFO[Server]: Server: New connection: "singleplayer" from 127.0.0.1 (peer_id=2)
11:27 AliasStillTaken Ok, thanks :)
11:32 AliasStillTaken Could that 50ms send the network thread into a sleep-wake cycle, when server step is lower than 50 ms?
11:33 celeron55 ?
11:33 celeron55 your problem is definitely not there
11:37 sometalgoo1 joined #minetest
11:57 AliasStillTaken Someone tried to log in to the server, but supplied and empty password. I have a single attempt from this IP address in the log. However, this prompted a lot of those : 2022-08-16 11:53:40: VERBOSE[ConnectionSend]: con(4/1)RE-SENDING timed-out RELIABLE to ::ffff:HIS_IP_ADDRESS(t/o=0.708): count=3, channel=2, seqnum=24876
12:19 BuckarooBanzai AliasStillTaken: did you take a look at this: https://github.com/minetest/minetest/issues/10088#issuecomment-647723155 ?
12:23 BuckarooBanzai (re: the "timed-out RELIABLE" part)
12:25 AliasStillTaken Thanks, didn't see that yet
12:26 AliasStillTaken I'm still trying to wrap my head around wireshark atm
12:30 orwell96 joined #minetest
12:47 MinetestBot [git] rubenwardy -> minetest/minetest: Add package title to ContentDB error messages (#12705) c4ffe63 https://github.com/minetest/minetest/commit/c4ffe630f106712d5c655e642cfca10096efba3a (2022-08-16T12:46:11Z)
12:50 proller joined #minetest
12:55 sfan5 AliasStillTaken: do you have the a pcap of the traffic during before/the lag
12:55 sfan5 eh messed up the word order
12:55 sfan5 I think you still get it
12:56 AliasStillTaken Yes :D I still experiment with how to generate a pcap file
12:56 sfan5 if you have the packets in wireshark you can just click save
12:57 AliasStillTaken That's what I struggle with. If you have a simple recipe, I could certainly put it to work.
12:59 AliasStillTaken Another drawback: Ever since the serverrestart, there was no chatlag episode -.-
13:01 sfan5 you're struggling with the "have packets in wireshark" or the "save it" part?
13:02 AliasStillTaken With the CLI of wireshark atm
13:02 sfan5 tshark?
13:04 AliasStillTaken Yes. Plus, some other questions around it. When I start logging, will that disconnect all the players?
13:07 BuckarooBanzai AliasStillTaken: nope, this _should_ not affect anyone, it is usually just `tshark -i <interface-name> -w <outputfile>` (this will capture _all_ the traffic though, including non-encrypted stuff you may have running)
13:08 BuckarooBanzai this should work better: `tshark -i <interface-name> -w <outputfile> "port 30000"`
13:08 AliasStillTaken Will that not only capture incoming traffic?
13:09 BuckarooBanzai it captures everything on that interface with source- or destination-port 30000
13:14 Fixer joined #minetest
13:15 AliasStillTaken Sounds like you're familiar with wireshark a bit. How can I make it that I capture the episodes? Is there a way to rotate the capture file ?
13:16 AliasStillTaken Like "the following 10 minutes everything goes to THIS file, the next 10 minutes everything goes to THAT file" ?
13:16 AliasStillTaken I mean, sure, I could let it run continuously, but that would give us gigabytes of captures, right?
13:17 AliasStillTaken Earlier today, the chatlag episodes were frequent, but after a serverrestart it takes a bit until the episodes start again
13:17 BuckarooBanzai don't let that thing run continuously, it has some memory-leak issues :)
13:17 BuckarooBanzai i usually just run it when something happens, otherwise you have TB's worth of captures and no idea where to start :/
13:19 AliasStillTaken That means I might miss what happens immediately before and the first few seconds of the event
13:20 BuckarooBanzai there's a stop-condition parameter, let me check the --help quick...
13:20 sfan5 -b <ringbuffer opt.> ..., --ring-buffer <ringbuffer opt.>
13:20 sfan5 duration:NUM - switch to next file after NUM secs
13:20 BuckarooBanzai `-a duration:3600` to run it for an hour for example
13:21 BuckarooBanzai sfan5: you are too far below...
13:21 BuckarooBanzai ah, sorry, my bad
13:21 BuckarooBanzai no, this is exactly your use case :)
13:21 sfan5 -b does more than -a depending on what AliasStillTaken needs
13:22 sfan5 you also definitely want to run with -n and -M <some value here> to fix the memory "leaks"
13:22 BuckarooBanzai yeah
13:22 sfan5 this would perhaps be easier with tcpdump
13:29 AliasStillTaken What's a nice value for -M? about 1 Million?
13:30 sfan5 dunno, maybe 5000
13:31 AliasStillTaken Alright. That's what I cobbled together: tshark -i eno1 -w /root/administration156.pcap -b duration:180 -n -M 5000 "port 30000"
13:33 sfan5 looks good
13:37 AliasStillTaken Ok ... now comes the fun part: How do I filter out all chat and private messages?
13:38 sfan5 if you have the minetest dissector installed in wireshark you can filter out all chat packets
13:38 AliasStillTaken How do I do that?
13:39 AliasStillTaken Sorry to bother you about such things, I have very little knowledge in ... a lot of parts of this adventure :D
13:47 sfan5 you copy util/wireshark/minetest.lua to ~/.wireshark/plugins/ and use the following filter
13:47 sfan5 !(minetest.server.command == 0x002f)
13:48 sfan5 its not perfect since it can only take the first of a split packet into account but should be good enough unless your users PM eachother entire novels
13:48 sfan5 you can also exclude auth information with types 0x51 0x52 and 0x60
13:51 sfan5 (those aren't cleartext anyway)
13:53 AliasStillTaken Hm, I don't have a .wireshark directory ...
14:19 MinetestBot [git] celeron55 -> minetest/minetest: Log sockets into tracestream instead of dstream (#12701) 3f67215 https://github.com/minetest/minetest/commit/3f67215df9dbc16f3bfe1ffc5c2582a308532914 (2022-08-16T14:18:11Z)
14:26 proller joined #minetest
14:40 FreeFull joined #minetest
14:57 FreeFull joined #minetest
15:16 sfan5 you can create those two obviously
15:17 MTDiscord <PrairieWind> wireshark...you can check minetest packets?
15:28 MTDiscord <savilli> Everyone can
16:14 diceLibrarian they're usp
16:14 diceLibrarian *udp
16:36 specing_ joined #minetest
16:50 Talkless joined #minetest
17:31 creation joined #minetest
18:54 appguru joined #minetest
18:54 Verticen joined #minetest
18:59 Guest2931 joined #minetest
19:00 Guest2931 hi
19:22 vampirefrog joined #minetest
19:39 Clygro joined #minetest
19:44 Clygro left #minetest
19:57 YuGiOhJCJ joined #minetest
20:29 YuGiOhJCJ joined #minetest
21:35 Sven_vB joined #minetest
21:46 natewrench joined #minetest
21:50 Sven_vB joined #minetest
21:51 Noisytoot joined #minetest
22:31 Noisytoot joined #minetest
22:33 panwolfram joined #minetest
22:34 proller joined #minetest
22:50 prairiewind[m] joined #minetest
23:09 Noisytoot joined #minetest
23:13 Noisytoot joined #minetest
23:17 Noisytoot joined #minetest
23:34 Noisytoot joined #minetest
23:37 Lesha_Vel joined #minetest
23:43 Noisytoot joined #minetest
23:54 Noisytoot joined #minetest
23:59 AliasAlreadyTake joined #minetest

| Channels | #minetest index | Today | | Google Search | Plaintext