Time Nick Message 00:01 rubenwardy #2338 can be closed 00:02 nrzkt can you close the terrorist in my country also ? 00:02 nrzkt :( 01:13 gregorycu How are we all? 01:17 gregorycu Fucking hell, don't look at the news 01:19 kahrl_ I already did :( 01:29 gregorycu kahrl_: Are you capable of profiling and wouldn't mind assisting me? 01:30 kahrl_ I'm not great at profiling 01:31 kahrl_ I mean, I can run callgrind, but it runs so slow I can't do anything 01:33 gregorycu Man, really need a better profiling tool for the nix 01:34 gregorycu I can do this myself 01:34 gregorycu Time for a virtual machine :) 01:43 gregorycu So much to do, so little time 01:47 gregorycu https://github.com/minetest/minetest/pull/3348 01:48 gregorycu celeron55: Forgive me if I am wrong, but are you able to update your objects to the PR, unless I am mistaken and your objections still stands? 01:53 gregorycu For the record, we probably shouldn't be decompressing zips in the main game loop 01:58 gregorycu Here he is 01:59 est31 hi gregorycu! 01:59 est31 I am here to announce a new patch 01:59 gregorycu Hooray! 01:59 gregorycu What does it do? 01:59 est31 it removes a hack 01:59 est31 and cleanup 01:59 gregorycu In what area? 02:00 est31 src/sound_openal.cpp search for "TODO: Actually load it directly from memory" 02:00 est31 I implement the TODO :) 02:01 gregorycu Nice 02:01 gregorycu Tell you what's funny 02:01 gregorycu I will be working on something similar(ish) soon 02:02 gregorycu And that loading mods from .zip files 02:15 est31 #3367 02:15 gregorycu At the moment it looks like all zlib serialisation happens on the main thread 02:15 gregorycu deserialisation 02:15 est31 https://github.com/minetest/minetest/pull/3367 02:15 est31 !title 02:15 ShadowBot est31: Error: The command "title" is available in the Format and Web plugins. Please specify the plugin whose command you wish to call by using its name as a command before "title". 02:15 gregorycu Which means spikes during gameplay 02:15 gregorycu Is there any hard in moving the deserialisation to the transport thread? 02:16 est31 that would make sense, agreed 02:16 est31 but it would require a change in structure 02:16 est31 and question is whether the new structure is as nice as current one 02:17 est31 right now there is deserialisation of data + acting on data intermixed 02:17 est31 so if we did what you propose they had to be separated from each other 02:17 gregorycu Sorry, I mean decompress 02:17 gregorycu Not deserialise 02:18 est31 well, the compression is added at a very high level in the protocol 02:18 est31 and the connection receive thread works on a lower level 02:18 est31 so what I say still applies 02:18 gregorycu True 02:18 gregorycu Fuck 02:19 gregorycu GUI thread really needs to stop doing so much work 02:19 est31 even if we added yet another thread ^ tm only for decompression it would still require to separate decompress from deserialisation 02:19 gregorycu I don't like having a gazillion threads 02:20 gregorycu Was someone doing work in this area? 02:20 gregorycu Sending mapblock diffs etc.? 02:20 est31 adding a gazillion threads? 02:20 est31 no 02:21 est31 sending mapblock diffs? not that I can recall 02:21 est31 we do already something like that 02:21 est31 we send single node added / node removed events 02:21 est31 so we can't really improve on this alot 02:21 est31 except what RBA proposes ofc 02:21 est31 doing the same for metadata changes 02:22 est31 he even implemented it, just there are some issues which still aren't resolved 02:22 est31 other way to improve would be that the client actually caches the server map on disk 02:23 est31 protocol could look like that the client sends all the mapblock timestamps to the server, server only responds with newer mapblocks 02:23 est31 responding with diffs wouldn't be prefferable here either, as for that the server would need to know the history 02:23 est31 which it doesn't know 02:23 est31 well it does have rollback 02:24 est31 but thats not always enabled 02:24 est31 and rollback scanning is comparatively slow 02:24 est31 it involves sql query, these stuff is usually very slow if done for each mapblock 02:25 gregorycu I don't think it's too bad 02:25 gregorycu It's just that this stuff happens in the GUI thread 02:25 gregorycu And it causes jitter 02:25 gregorycu Like, makes 1 frame out of 30 to take 10 times longer 02:25 est31 well you are generally right, but it should perhaps be worth to benchmark first 02:26 gregorycu Someone has done this... um 02:26 est31 e.g. have a us counter inside the code that unzips stuff like the mapblock data 02:26 est31 and count it up 02:27 est31 then play for 1 minute with very fast map loading 02:27 est31 and take the value 02:27 est31 the rest is math, to get percent number 02:27 gregorycu What are you trying to determine? 02:28 est31 percentage of time spent in unzip 02:28 gregorycu I can tell you that already 02:28 gregorycu Let me do the calculations 02:30 RealBadAngel hi 02:30 est31 hello 02:31 gregorycu We are looking at 0.43% 02:31 gregorycu Which seems trivial, but you have to remember that's the average for all frames 02:31 gregorycu Not the amount for a "bad frame" 02:31 est31 yeah 02:32 gregorycu I am making the assumption that the bad frames are caused by Client::ProcessData 02:32 gregorycu Which is _probably_ correct 02:33 est31 I guess there are heuristics to find that out as well 02:33 est31 just measure its performance and whole frame perf 02:33 est31 and if whole frame perf is spiking, print the spiking perf and the perf of ProcessData. 02:33 est31 anyway, gotta go 02:33 est31 bye! 02:33 gregorycu Easier said than done... 02:34 gregorycu Non-trivial to isolate frames 02:34 RealBadAngel btw, i saw another voxel game on droid, theres nicely solved placing/removing nodes, there are just two buttons for that "+" "-", like just two mouse buttons 04:11 paramat gregorycu, i not sure if hmmmmm is actually working on the 'liquid spreads on ignore' bug, if you feel like working on it yourself 04:15 paramat i'll merge https://github.com/minetest/minetest/pull/3364 'Mgfractal: Create a choice of 4 mandelbrot formulas' very soon since it's dev work on my mapgen 04:22 paramat now merging 3364 04:29 paramat done 04:46 paramat now merging game 719 game 723 04:55 paramat done 04:58 paramat it's not a big issue, the code varies with the use of commas 04:59 paramat talk in channel don't pm 05:00 paramat i might even fix it myself soon 05:02 paramat anonymous pms from username 'guest****' don't carry much weight =} 05:15 paramat now merging game 727 (trivial fix) 05:19 paramat done 07:21 paramat https://github.com/minetest/minetest/pull/3365 'Decoration API: Allow very low density decorations' is ready for review 07:23 hmmmm looks good to me 07:24 hmmmm well one minor thing, the check for deco_count_f < 1.f in the else if clause is redundant 07:25 paramat oh yeah .. thanks 07:26 paramat it's great how you usually spot something i missed =) 07:47 paramat updated will merge after checks 07:48 hmmmm https://github.com/minetest/minetest/commit/987d6a82f639126d5d8833739499150da286bdda#diff-40af646e2d53a4cf277cca877b995e14R429 07:48 hmmmm this is exactly the kind of thing you should use an enum or constant macros for 07:50 paramat ah i did wonder about my use of an int there 07:50 paramat now merging 07:56 paramat done. yes i'll look into that 08:10 hmmmm yup 08:11 hmmmm https://github.com/kwolekr/minetest/commit/095627968b1b5ae0352b5eb66daf7c76c6be2321 08:24 celeron55 oh gregorucy wants blocks to be serialized in a thread? 08:24 celeron55 well i guess i could do that 08:25 celeron55 i'm guessing that the time spent doing that is pretty small but i haven't checked 08:25 hmmmm ideally these sorts of things would be done in the connection receive handler or something 08:26 hmmmm it's not about the time spent, but rather the jitter caused by suddenly doing zlib decompression (read: unnecessary not-rendering tasks) 08:26 celeron55 it's done there, and the handler operates in the main thread both on the server and the client 08:26 celeron55 i don't think it actually causes jitter though 08:27 celeron55 what causes annoying jitter depending on hardware is pushing meshes to the GPU, in my experience 08:27 celeron55 for what i have seen, zlib is insanely fast; it's not a problem 08:27 celeron55 but i need to profile this a bit 08:28 hmmmm remember you need to do all of this stuff in 16 milliseconds 08:29 celeron55 what time do we consider too long, given a situation where the game is running at 60fps? 08:29 hmmmm I don't know but it simply needs to not add up to 16 milliseconds except under very rare circumstances that are difficult to account for 08:30 hmmmm you have this fantastic client event queue setup that a separate thread could be a producer of 08:30 celeron55 like, maybe it's not a problem if one block takes less than 0.2ms 08:30 hmmmm do you really get one block? 08:31 hmmmm or do you get a couple hundred blocks all of a sudden 08:31 celeron55 no; but a few 08:31 celeron55 the networkings work in such a way that it's practically impossible for the client to receive more than 10 per frame; it's more usual to be 1 per frame 08:33 celeron55 and that is intentional, because mesh generation can't generally be done faster than one per frame because it seems that some graphics hardware really doesn't like it when you create more meshes than that in one frame in the way irrlicht adds them 08:33 hmmmm touche 08:33 hmmmm is this bottleneck an irrlicht problem mostly? 08:35 celeron55 i'm sure with modern opengl that bottleneck wouldn't exist 08:36 celeron55 or... well not sure, but i would guess so 08:37 hmmmm well if you want to be the one to test that be my guest 08:37 celeron55 anyway i'll add some stuff to the built-in profiler and see what the serialization and deserialization numbers look like 08:37 hmmmm it's called mineTEST after all 08:37 celeron55 i have never actually checked how much time the zlib calls are taking 08:42 RealBadAngel celeron55, http://irc.minetest.ru/minetest-dev/2015-11-14#i_4449752 08:44 celeron55 ??? 08:45 RealBadAngel hes talking about time spent unzipping stuff 08:45 celeron55 yes guess what you smartass? i already read all that 08:46 celeron55 that's why i am talking about it 08:58 celeron55 looks like on my machine generally the zlib-compressed mapnode data deserialization takes about 50us, but if i open VanessaE's crazy world, the highest i can see is 144us 08:58 celeron55 considering it's plausible that someone would be using a computer with the same GPU speed and half the CPU speed, that is a bit high 09:13 celeron55 the thing people forget is that there's way more happening in a mapblock deserialization than just a zlib decompress; the zlib decompress is generally about 67% of it, but it's not unusual for it to be as low as 10%... and now i have seen one that took 15ms in total 09:13 celeron55 i don't know what it's doing nor is it actually important, but i mean, stop talking about zlib 09:13 celeron55 it's not the issue here 09:13 celeron55 the whole deserialization is the issue 09:15 celeron55 (that 15ms MapBlock is one on the crazy world; it's probably pure minetest abuse but that doesn't mean it wouldn't lengthen the rendering of a frame by 15ms) 11:02 celeron55 just so that gregorycu knows if he's reading this: i'm not going to modify the basic packet handling framework in the client, so making it asynchronous is totally something that someone can start looking into 11:03 celeron55 (with the main goal of getting deserializations out from the main thread) 11:55 rubenwardy sfan5, ERROR[CurlFetch]: servers.minetest.net/announce not found (HTTP response code said error) (response code 500) 12:04 sfan5 rubenwardy: works fine here 12:26 rubenwardy misspelt my server's address 17:45 est31 I dunno, is it worth a commit to remove #include "filesys.h" from src/sound_openal.cpp ? 17:45 est31 its not needed anymore 17:53 * Krock looks up to celeron55 and wonders what the weather on top of this throne is 17:54 celeron55 my throne has a roof 17:54 celeron55 so it's kind of warm and non-wet 17:55 celeron55 outside the roof it's so dark and depressing that it would kill any non-finnish person 18:02 Krock Break the roof and rise! 18:03 who_wants_some and fix bugs *cough* 18:06 est31 hah 18:06 est31 I compared opus to vorbis now 18:06 est31 and in fact they both produce the same file size 18:07 est31 even though vorbis includes a 3-4kb header codebook 18:07 est31 which opus doesnt 18:07 est31 but at same bitrate of 161 kb/s 18:07 est31 they both need 13 kb for default_break_glass.1 18:07 est31 of mtgame 18:08 est31 but on the long term, I do think we should add opus support 18:09 Calinou yeah, Godot now supports Opus 18:09 Calinou it's the future :) 18:10 est31 it does* 18:10 est31 ? 18:10 est31 wow thats great! 18:10 est31 since early october it seems 18:10 est31 6e87314d83a9beab56fdd115277e230ef683c53d ... 20:30 celeron55 https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-8126 20:30 celeron55 basically every libpng in this world is vulnerable to this 20:30 celeron55 good luck everyone 20:32 celeron55 https://news.ycombinator.com/item?id=10566664 20:32 celeron55 here's some talk about it 20:33 est31 will get into technews I guess 20:33 est31 and get a catchy name 20:33 est31 and logo 20:33 est31 and wiki page 20:34 est31 not in the "secvulnlulz.wikia.com", but in en.wikipedia.org 20:35 celeron55 this isn't really even funny anymore though; libpng is used everywhere and everyone just pulls data from the public internet into it 20:35 celeron55 minetest included 20:36 est31 soo I should do sudo apt-get update && sudo apt-get dist-upgrade then? 20:36 est31 and now is when the fun starts 20:36 est31 all the sec people will look at the code and see its all just bs 20:36 est31 and find tons of other vulns 20:37 est31 or perhaps no 20:37 est31 r 20:37 est31 t* 20:37 celeron55 well 20:37 celeron55 at least on fedora 22 (that i use), there is no new enough version of libpng available 20:38 celeron55 i guess the distros will catch up 20:38 * est31 eww 20:38 celeron55 windows is another story though; everyone bundles their own copy of libpng with their programs and they will never be updated 20:39 * est31 waiting for people to upload malicious png files to pastebrd or imgur and post links here 20:40 est31 but its only denial of service, no? 20:40 est31 hrmm wondering what "unspecified other impact" means 20:44 est31 it says it affects libpng 1.6.18 and previous versions 20:44 est31 I have 1.2.51 20:46 celeron55 the attack vector caused by this is so large that if i was to protect my things against this, i would have to just shut down everything 20:48 celeron55 at least my server has proper security limits in place between services; if one of them is exploited, the others will be fine 20:49 gravgun Oddly enough loading http://www.libpng.org/pub/png/libpng.html fails for me 20:49 celeron55 what's odd about that 20:49 est31 the page is down for me as well 20:49 est31 "server has reset the connection" 20:50 celeron55 obviously everyone tries to open that page right now and it's just a single server 20:50 est31 slashdot effect 20:50 celeron55 with normally almost zero traffic 20:50 gravgun "normally" 20:50 gravgun with relatively frequent spikes 20:51 gravgun i.e. on every CVE published about libpng 20:51 est31 or every new release 20:53 celeron55 but a good question is: where to get libpng 1.6.19 20:55 est31 without browsing too much in the internet, downloading tons of png images 20:55 est31 perhaps use lynx ?? 20:56 who_wants_some Q: any advance on liquid over ignore problem? too much stuck water on servers :( 20:56 celeron55 https://github.com/glennrp/libpng 20:56 celeron55 this seems to be a recent enough mirror of libpng 20:58 est31 btw which version of libpng do you currently have? 20:59 celeron55 my main machines that i mostly care about seem to have 1.6.16 20:59 est31 I have 1.2.51 20:59 est31 and it makes me really wonder, because there is a libpng12 package and I have it 20:59 est31 but there is no libpng16 package 21:00 est31 debian has it neither https://packages.debian.org/search?suite=stable§ion=all&arch=any&searchon=names&keywords=libpng 21:01 celeron55 one of them has 1.2.51; it's a linux mint box with a broken package management system so it's just bound to die of old age some day 21:01 est31 wondering whether the 1.2 branch is affected as well 21:04 est31 yuppy, 1.2.51 has the vuln as well 21:06 est31 and oddly, 1.2.54, released on thursday, has it too 21:06 est31 at least I can't see a check in pngrutil.c like added in this commit https://github.com/glennrp/libpng/commit/0b4e7deab1c736273c2d57a19c1bd996576d93f8 21:09 celeron55 that's not the vulnerability fix 21:09 celeron55 it's after 1.6.19 21:10 celeron55 i guess this is what the CVE is talking about https://github.com/glennrp/libpng/commit/81f44665cce4cb1373f049a76f3904e981b7a766 21:10 celeron55 hmm maybe not 21:11 celeron55 well dunno 21:13 est31 http://www.openwall.com/lists/oss-security/2015/11/12/2 21:13 est31 hrmm seems 1.2.54 is fixed after all 21:13 est31 at least he claims it is 21:22 est31 lol had a kernel panic 21:22 est31 now I am completely hacked :) 21:23 est31 bad guys have root now 21:31 est31 debian only has 1.2.50 21:41 celeron55 one thing is for certain: writing security-critical software in this multiplatform macroassembly is really just fucking stupid 21:42 est31 https://github.com/PistonDevelopers/image-png 21:42 est31 I for one welcome our new rusty overlords 21:43 est31 btw do the windows builds of minetest bundle libpng? 21:43 est31 would it be an idea to update them? 21:43 * est31 wonders about android 21:44 celeron55 of course they bundle libpng; it's not like windows has it built-in 21:45 celeron55 i guess it usually comes directly from irrlicht 21:45 celeron55 irrlicht depends on it, and minetest uses it through irrlicht 21:47 kahrl I suppose one could write a PNG loader based on GDI+ instead of libpng 21:47 celeron55 irrlicht 1.8.3 bundles libpng 1.5.9 21:48 est31 ah so they _bundle_ it... 21:48 est31 not depend 21:48 celeron55 well it makes building irrlicht on windows reasonable 21:49 celeron55 i guess on linux it usually is dynamically linked to system libpng 21:49 celeron55 you can do it on windows too of course 21:49 celeron55 not that anyone does it but anyway 21:50 est31 yea on linux it is linked 21:50 est31 well, irrlicht isn't worse than godot 21:51 est31 godot has 1.5.4 21:51 est31 https://github.com/godotengine/godot/issues/2787 21:53 celeron55 https://github.com/zaki/irrlicht/blob/master/source/Irrlicht/CImageLoaderPNG.cpp#L155 21:53 celeron55 of course somebody could check whether irrlicht does the vulnerable thing or not 21:53 hmmmm hmrm 21:54 hmmmm unless I'm mistaken, this is a DoS at best, no? 21:54 est31 https://news.ycombinator.com/item?id=10567353 21:54 est31 yes 21:54 hmmmm it's serious because of the scope but it's not like anybody's r00ting us 21:54 celeron55 it's DoS for now 21:55 est31 probably libpng will get a bit attention for the next weeks 21:55 hmmmm ermm 21:57 celeron55 i guess it could be more than DoS depending on the application that uses libpng 21:58 celeron55 it seems to me that irrlicht just converts it from a palette to rgb if it isn't rgb, and to 8 bits if it isn't 8 bits 21:59 hmmmm i think we're technically safe here 22:00 celeron55 i don't understand how nobody found this before though 22:00 celeron55 this seems like something you would easily find by just taking a bunch of png images and fuzzing them under a memory checker 22:00 est31 yup that was how it was found 22:00 est31 http://sourceforge.net/p/png-mng/mailman/png-mng-implement/thread/CA%2BPdXcuhLXJ89s6qjOEcm%2B99eWLmPBFcYSwcwJkajkLrNRLTeQ%40mail.gmail.com/#msg34581085 22:01 celeron55 404 22:02 est31 what about http://sourceforge.net/p/png-mng/mailman/message/34577507/ 22:02 celeron55 same 22:02 celeron55 it seems sourceforge is partly down or broken 22:02 hmmmm this sounds like downright negligence 22:03 est31 https://web.archive.org/web/20151114220245/http://sourceforge.net/p/png-mng/mailman/message/34577507/ 22:04 est31 or better, here https://web.archive.org/web/20151114220333/http://sourceforge.net/p/png-mng/mailman/png-mng-implement/thread/CAAoVtZzzx%2BUvXY0ddbY8jSuWE-v9hqGTQoCLZubyf66hF2AMLQ%40mail.gmail.com/ 22:08 celeron55 by the way, godot does something much more delicate: https://github.com/godotengine/godot/blob/master/drivers/png/image_loader_png.cpp 22:08 celeron55 if est31 is interested 22:11 celeron55 i actually very much like what irrlicht does when compared to that 22:11 celeron55 irrlicht is like "fuck all these special features, just now give me the RGBA8 and we're done here"