Time Nick Message 14:34 Niklp sfan5: I ported the caverealms mapgen (https://github.com/Archtec-io/caverealms_lite/tree/mglua) to your new async api, really impressive! But the max lag goes up to ~1.5s (even more than w/ "normal" on_generated), is this a problem in my implementation or a mglua issue? 16:50 sfan5 implementation looks ok, would have to profile to see why it still causes lag 16:50 sfan5 if it works correctly any lua mapgen should not be causing more lag than the engine mapgen 17:31 MTDiscord <.niklp> I'll profile it with jitprofiler later 17:41 rubenwardy you'll probably want a C++ profiler. The performance of the Lua shouldn't matter as it's on a different thread 17:50 Niklp I ran /emergeblocks, every 20-30 seconds theres a ~1-2 seconds long server freeze. Which C++ profiler should I use? 18:00 Niklp the freeze comes from map saving, I'll try the default save interval 18:40 sfan5 perf record -z --call-graph dwarf -- ./bin/minetestserver ... 18:40 sfan5 (debug symbols needed) 18:41 sfan5 after that I put the results into https://github.com/KDAB/hotspot 18:53 Niklp Are debug symbols included in RelWithDebInfo? 18:57 MTDiscord Yes 18:58 MTDiscord I can confirm that doing a Release build does not include debug symbols but doing a RelWithDebInfo does. 19:20 Niklp oke how must I "read" the results in hotspot? 19:22 sfan5 find the main thread at the bottom, figure out when it froze and filter for that 19:22 sfan5 then the top graph gives you a good idea what it was busy with 19:23 MTDiscord "every 20-30 seconds theres a ~1-2 seconds long server freeze" ... You wouldn't happen to have a mechanical HDD, would you? 19:24 MTDiscord That is suspiciously close to the issues I had with database activity on HDD. Enabling pragma journal_mode=wal on sqlite mitigated it a bit (spread out the interval). Moving to postgresql got rid of write spikes, but I still had a lot of delay loading mapblocks from disk. The only thing that fully resolved it was upgrading to an SSD. 19:25 MTDiscord HDD wasn't always bad but it got bad pretty consistently any time the disk was busy with anything other than MT. 19:25 Niklp No, I have a fast(TM) nvme SSD. I did some tests, with the default interval (5.3 seconds), there's just a ~400ms freeze 19:26 MTDiscord Hmm, you wouldn't happen to have a machine that's slow in other ways, like CPU, would you? Could be compression or something maybe... Or maybe you just have some really big mapblocks (like lots of meta or ents) 19:27 Niklp I've never experienced a similar issue in the last two years with my pc... 19:30 Niklp The mapblocks are new generated since I test mapgen. There shouldn't be any nodemeta or ents 19:31 celeron55 Niklp: you are using sqlite, right? 19:31 celeron55 (wasn't mentioned yet) 19:32 Niklp yes, it's just a debug server, otherwise I would use postgres 19:33 Niklp (context: I'm using sfan's async lua mapgen PR) 19:33 sfan5 consider using the dummy backend to rule out any disk activity 19:33 Niklp +1 I'll test this 19:33 celeron55 what filesystem? 19:34 celeron55 you could try setting sqlite_synchronous to 1 (possibly permanently) or 0 (for testing) to see if the filesystem is somehow very slow to sync 19:35 celeron55 (it sets PRAGMA synchronous, see https://www.sqlite.org/pragma.html) 19:35 celeron55 on android it defaults to 1 and 1 should be safe enough 19:35 celeron55 on desktop it defaults to 2 19:36 Niklp I'm using btrfs, w/o any weird config 19:36 celeron55 (warr1024's suggestion about wal is also relevant, but i haven't personally played with that) 19:37 celeron55 btrfs is a bit weird but i don't know anything about its sync characteristics 19:39 Niklp max_lag w/ dummy map is 0.53 seconds definitely better than ~1.5 seconds 19:40 MTDiscord Re: WAL, it's basically the "modern" way to run a sqlite database and I would recommend it for ALL non-pathological setups (i.e. unless you NEED a single file only, or are running stuff over nfs or something). I think the only reason it's not the default is because sqlite is really super strict about compatibility or something. 19:41 sfan5 wasn't WAL enabled by default? 19:41 celeron55 i wonder if changing map_compression_level_disk would have an effect 19:41 MTDiscord WAL is not enabled by default that I know of in sqlite in general, and it's definitely not in Minetst. 19:41 sfan5 hmm no 19:41 sfan5 celeron55: better yet move compression out of the main thread 19:42 celeron55 well that's not a simple config change that can be tested right now 19:42 MTDiscord I would actually really like if there were an option in MT to turn it on immediately upon creating ANY sqlite db, so I wouldn't have to go in with scripts and do it post hoc. 19:42 celeron55 unless you're about to write a patch 8) 19:42 MTDiscord Re: compression, I'd try both lowering the compression (in case it's CPU pressure) and possibly raising it (in case it's I/O somehow) 19:43 celeron55 maybe map_compression_level_disk = 2 or something like that for test 19:43 celeron55 but also sqlite_synchronous 19:45 MTDiscord heh, test each combination of config in nested for loops and graph the data ... in principle if you can measure the spikes somehow, like looking at globalstep dtime for 1 minute after the first minute (to ensure the game warms up first) and then calling request_shutdown, you can actually automate testing stuff like this. Really helpful when you have to bisect. 19:50 Niklp map_compression_level_disk = -1 and sqlite_synchronous = 1 keep the maxlag at ~0.7. I'll test this w/o sfan's PR tomorrow 19:50 [MTMatrix] btw, can recreate database with synchronous=0 and other danger options, from raw sql dump (just adding some SQL lines to head of file), and it will be fast as possible, but... can be easy breakable 20:03 celeron55 Niklp: -1 is the default value, it won't change anything 20:04 celeron55 i don't know what the default level actually is but it's 0 to 9, 0 is least compression, so try 0 or 1 20:07 Niklp oops, you're right 20:16 sfan5 zstd actually has levels beyond 9 but it seems we don't map those