Time Nick Message 00:28 MTDiscord Are you sure M_TRIM_THRESHOLD would even help there? According to man, the default value from M_TRIM_THRESHOLD is 128*1024 bytes. In the forum thread, the amount of supposedly freed memory is significantly bigger, and yet the memory is not returned. 07:27 sfan5 libc tuning is out of scope for MT imo 07:32 sfan5 pgimeno: hmmm from the issue report on the LJ repo that wasn't so clear to me 07:35 sfan5 ...however people also said it affects android 07:46 pgimeno relevant to the memory issue: https://stackoverflow.com/questions/27945855/force-free-to-return-malloc-memory-back-to-os 07:47 pgimeno sfan5: yes but it appears to affect Android differently 07:58 pgimeno apparently the problem lies in whether the OS is able to mmap the memory within 128 MB of the critical symbols, and that's what varies with the OS 07:59 pgimeno (128 MB for ARM obviously) 08:06 pgimeno IIUC, the workaround given in the issue (using -Wl,-image_base,) works because it places the library in a virtual address that isn't surrounded by occupied addresses 08:09 pgimeno unfortunately it also has a side effect, which is to prevent ASLR for the LuaJIT library 09:52 sfan5 sounds like a possible short term workaround would be to place vm_exit_handler inside its own section? 11:01 pgimeno not sure how that would help, if you mean by placing them at the end then I doubt it would 14:08 sfan5 the section can be mapped independently with free space around it 14:48 MTDiscord I tested M_TRIM_THRESHOLD and it doesn't do shit. Gigabytes of memory are still not returned. 14:49 MTDiscord The only way to return memory I found is to call malloc_trim 14:49 MTDiscord Probably the same situation as here https://stackoverflow.com/questions/38644578/understanding-glibc-malloc-trimming 15:38 sfan5 the thing with mapblocks is they use an allocation of a nice round 16384 bytes, mmap makes the most sense for these and that should be directly freeable 15:39 sfan5 looks like glibc.malloc.mmap_threshold is 131072 so nevermind 17:51 MTDiscord i see. have we considered switching to a malloc library like jemalloc or tcmalloc? would there be a strong reason against this? 18:00 sfan5 by "switching to it" you mean bundling it by default? 18:01 MTDiscord yes 18:04 MTDiscord As for my ambient light: it has one approval from appgurueu, is waiting for a second approval mandatory from anyone else to be merged? Because like my PR after that approval is in the same situation as before: it is awaiting for unknowingly something and someone, but the feature freeze is approaching 18:06 MTDiscord Moreover it was reviewed already multiple times, so does it make sense to await for reviews still? 18:06 MTDiscord I tested few alternative allocators - tcmalloc and mimalloc also don't return the memory, but jemalloc does. I'm not sure about its performance tho. 18:11 MTDiscord I believe jemalloc has good performance. 18:11 MTDiscord I haven't tested it personally, but that's what I've heard. 18:17 [MTMatrix] andrey2470t: it's neither trivial nor does it fulfill the requirements for the experimental one approval rule, so yes. 18:17 [MTMatrix] considering the amount of discussion that's already happened on the PR, giving it a second review should be relatively easy though. I'll have a look in the evening. 18:21 MTDiscord It is nice to hear, grorp 🙂 Thanks. 18:22 MTDiscord re: memalloc: interesting 🤔 so is this why RAM usage for minetest servers on linux always steadily rises without freeing any until it OOMs? 18:44 Mantar I think that may depend on the game somewhat, I haven't noticed that happening on Exile servers 18:51 sfan5 if it's just the mapblock data then a custom allocator for that would be easy 20:21 MTDiscord In theory, yes. But it should be thread-safe and somehow smart to return the excessive memory. 20:31 MTDiscord assuming it's just the mapblocks feels like it would end in whack-a-mole. after we "fix" the mapblocks using a custom allocator, the same issue with all kinds of other allocations would probably crop up. 20:43 sfan5 well if that fixes 90% of "leaks" then it's probably good enough for a long time 20:43 sfan5 but I agree it's better fixed in the allocator (glibc in this case) 20:46 MTDiscord It should be very low effort to test an allocator such as jemalloc. 20:46 sfan5 savilli already did :) 20:47 MTDiscord Oh, right. 21:09 sfan5 @savilli a fixed size arena allocator is really easy. get a big memory block and keep a bitmap of used blocks. big would mean e.g. 16MB 21:10 sfan5 this would not necessarily perform well but certainly better than what is happening with glibc. and it's guaranteed to free everything when you reach zero blocks 21:10 sfan5 with access to the data structure you could even implement memory compaction so you don't get the worst case of having multiple chunks and all of them 1% used 21:10 sfan5 or just call madvice(MADV_DONTNEED) on those parts yourself 21:12 sfan5 to be clear I would use this for the MapBlock::data array. I reckon this is what is triggering the bad glibc behavior 21:15 sfan5 *or* we could side-step all of this by just calling malloc_trim ourselves 21:19 Desour why does the mem usage grow so much in the first place anyways? 21:19 Desour is it fragmentation? 21:22 sfan5 the server starts unloading data after 30s only, this point is the peak memory usage 21:23 sfan5 system bytes = 1242898432 21:23 sfan5 in use bytes = 1146413296 21:23 sfan5 wait let me graph this 21:29 sfan5 https://0x0.st/X8Qt.png this is with malloc_trim. also the data only starts once the first unload happens (after 30s like I said) 21:29 sfan5 oops, actually malloc_trim doesn't make a difference here 21:29 sfan5 but you can see it in external monitoring (e.g. htop) 21:30 sfan5 without malloc_trim the RSS memory is basically equal to the "system" line 21:33 Desour so server owners can already decrease the server_unload_unused_data_timeout setting 21:39 sfan5 sure but that only works around the fact that RSS never decreases from the peak 21:44 Desour but it might make the peak at least a little lower in some cases 21:57 sfan5 anyway I commented my results in the forum post 22:01 sfan5 http://sprunge.us/8wtD9b?diff pushing in 15m 22:45 MTDiscord I replaced MapBlock::data allocation with mmap and munmap calls and it fixed the problem 🎉 . It means a custom allocator would work too.