Time Nick Message 08:38 sfan5 PRs welcome 15:57 Guest54 given that the filesystem interface supports zip files and the irrlicht media loaders use that filesystem interface AFAIK, would there be any benefit to loading mod assets from zip files? except for saving filesystem space, i mean. 15:58 rubenwardy would prevent needing to extract assets on Android 15:58 Guest54 well, actual mod directories could, of course, still be needed for dynamic asset generation. 15:58 rubenwardy mod directories aren't necessarily always writable 15:58 rubenwardy ie: /usr/share/minetest/mods/ 15:59 Guest54 well, this is a case where doing semgrep on everything in cdb would help to see if anyone does it 16:20 MTDiscord Not having to extract could be pretty nice, actually. Android is one example but probably not the only filesystem/context in which handling lots of little files isn't necessarily that efficient. 16:21 Guest54 many games do it by default. and i have also found that this is the explanation for why they can get away with having ridiculous asset size uncompressed, everything is zipped. 16:21 MTDiscord Contiguity of related data could also be a nice side effect, e.g. you might get better cache hit ratios that affect stuff like HDDs. Also, some assets might actually compress pretty well, e.g. the lua code files, some models... 16:29 MTDiscord windows also sucks at handling small files 16:50 Guest54 Warr1024 counter-intuitively (to me at least), zipping uncompressed bitmaps can work much better than using optipng (and is faster too) – e.g. more than half the textures in mineclone2 (over 1200 out of about 2300) get smaller as a tga.gz than the smallest thing optipng has spit out for me (by average of about 26 bytes per texture, if i am not 16:50 Guest54 mistaken, probably a bit more because of the gzip framing). if you cheat by using A1R5G5B5 colors for the uncompressed bitmap, over 2000 out of 2300 pics are smaller. 16:51 Guest54 in practice, putting them in a zip will prevent each one from consuming a 4k filesystem block 16:52 Guest54 (together, not individually) 16:55 MTDiscord Doesn't seem THAT counterintuitive to me. Most image formats that support built-in compression neglect to compress headers, and on small enough images, and formats that have complex header structure, those can become dominant. 16:56 MTDiscord If MT supported sending and caching assets with external compression, e.g. tga.gz or something, then I'd have an easier time supporting their use in the engine. But as it is, compressing assets on disk at the expense of crappier network sends is probably not a trade-off I'm interested in. 16:56 Guest54 the counterintuitive result for me was mainly that optipng can't beat it, despite wasting a lot of CPU cycles 16:57 MTDiscord Diminishing returns. It's not terribly hard to find simple and small enough images where running zopfli brute-forcing for hours can't beat a pretty naive gzip. Recompressors just try more and more desperate strategies to squeeze out the last few bits, but sometimes there just weren't any hidden opportunities to begin with. 16:58 Guest54 indeed 16:58 Guest54 i was pretty surprised how well it worked for the devtest checkerboard though 16:59 MTDiscord Also, basically everything uses the same deflate compression algorithms these days, so if you can get a certain result with a PNG compressor, then you should be able to get very comparable, or slightly better, with like .bmp.gz as long as the gz engine you're using is as thorough as the PNG recompressor's. You can get the same for zip files too (though it's still per-file of course). 17:00 Guest54 not really, it's a speed-space tradeoff. gimp with default settings for example does produce PNGs that are about as good/bad as throwing raw pixels into deflate. 17:01 Guest54 the reason is probably because that makes saving instant … meanwhile optipng can cut the size in half sometimes, but you don't want to wait 5 seconds on even the fastest machine until it has figured out which knobs to turn. 17:02 Guest54 Warr1024 i think i may have mentioned it already, since i want to improve the handling of generated textures i do care a lot about network transfer size. 17:02 Guest54 and speed, of course 17:04 MTDiscord I played around with adding per-packet lzo to MT, and was a bit disappointed to see like 3% or less improvement, so I didn't push the project further. Granted, that may have been because most of the data dominating the bandwidth usage was mapblock sends, and those were already zstd'd. 17:04 MTDiscord Also, testing against NodeCore was probably not a great idea because I probably should have used content that was more wasteful to begin with 😄 17:04 Guest54 hehe 17:05 Guest54 i wouldn't want to automate it actually 17:07 MTDiscord I would think that MT should be able to make at least reasonable guesses as to what kind of data would be good for compression and what would not. Also, for media, MT could cache the compressed versions in advance, and choose to fall back to the uncompressed version if the compressed version isn't smaller enough to warrant it. 17:08 MTDiscord Being able to turn off automatic compression, and just provide like .obj.gz files or something as assets, which the engine knows to send with compression intact and decompress at load time, might be nice for some of us (more serious game devs and server owners and such) but having sane default behavior for the "average (naive) case" would be nice too 17:09 Guest54 well, you can actually not make a guess and just try it out server-side if the client supports it. 17:14 Guest54 by the way i strongly advise against using BMP. the format is a big mess (and compresses worse than the alternatives) 17:14 MTDiscord You could, though I'd follow the pattern that a lot of web servers use for on-the-fly compression, which includes them often having a list of filetypes that they can identify as being very likely not worth even trying. If we know that, say, PNG, OGG, and JPEG (perhaps with certain size constraints) aren't worth trying, we can keep things pretty simple and still have fast startup times. Also, not every server operator will want to 17:14 MTDiscord keep trying the same files over and over again on every restart, even if we can do it inteligently in the background without delaying startup, because they may e.g. pay for CPU burst or something. 17:15 Guest54 my problem is that you may not want to auto-compress stuff based on the contents, not on file type in particular. 17:16 Guest54 e.g. some random noise texture is not very compressible 17:17 Guest54 given the amount of useless sha1 hashing going on during asset load i have a suspicion few people would notice on-the-fly compression ;) 17:22 MTDiscord If you would want to inform the engine what it should try and not try to compress, that'd be fine too. But in the absence of info provided by the dev, the engine will need to be able to make the decision based on some sane default heuristic. "Try to compress it all" is not a terrible idea but I could see some issues with it at least, while guessing by file type is a pretty decent mitigation. 17:23 Guest54 well, i do agree with exclusions by filetype – you won't get png, vorbis or jpeg content much smaller anyways – it's already compressed (even if it is compressed badly) 17:25 Guest54 btw, xmaps does already store deflated bitmaps in item meta (so the maps work in world downloads) 17:25 Guest54 i suggest to do the same for anything in nodecore (but ig it's a #minetest topic, sorry) 17:57 Guest54 given that zstd has negative levels up to -7 or so, why is the default compression level in builtin/settingtypes.txt limited to -1? 21:24 Guest5 we know where your workplace is, andrew 21:24 Guest5 don't forget the condom when you get fucked hard in the ass 21:30 MTDiscord imagine sinking this low 21:48 rubenwardy Pathetic little man 22:19 MTDiscord uhh 22:19 MTDiscord yeesh 22:24 MTDiscord What just happened 22:24 MTDiscord Was context deleted? 22:31 MTDiscord not yet, see Guest5's message above 22:31 MTDiscord or don't, it's probably better to not engage 22:34 Guest54 indeed, engaging is a mistake: i asked guest5 for an explanation and got only a “you know the problem“ and “who are you?”. i should have known better. :( 22:35 MTDiscord kilibith 22:35 MTDiscord likely