Minetest logo

IRC log for #minetest-dev, 2015-11-14

| Channels | #minetest-dev index | Today | | Google Search | Plaintext

All times shown according to UTC.

Time Nick Message
00:01 rubenwardy #2338 can be closed
00:02 nrzkt can you close the terrorist in my country also ?
00:02 nrzkt :(
00:25 gregorycu joined #minetest-dev
00:31 Icedream joined #minetest-dev
01:06 H-H-H joined #minetest-dev
01:13 gregorycu How are we all?
01:13 Icedream joined #minetest-dev
01:17 gregorycu Fucking hell, don't look at the news
01:19 kahrl_ I already did :(
01:29 gregorycu kahrl_: Are you capable of profiling and wouldn't mind assisting me?
01:30 kahrl_ I'm not great at profiling
01:31 kahrl_ I mean, I can run callgrind, but it runs so slow I can't do anything
01:33 gregorycu Man, really need a better profiling tool for the nix
01:34 gregorycu I can do this myself
01:34 gregorycu Time for a virtual machine :)
01:43 gregorycu So much to do, so little time
01:47 gregorycu https://github.com/minetest/minetest/pull/3348
01:48 Miner_48er joined #minetest-dev
01:48 gregorycu celeron55: Forgive me if I am wrong, but are you able to update your objects to the PR, unless I am mistaken and your objections still stands?
01:53 gregorycu For the record, we probably shouldn't be decompressing zips in the main game loop
01:58 est31 joined #minetest-dev
01:58 gregorycu Here he is
01:59 est31 hi gregorycu!
01:59 est31 I am here to announce a new patch
01:59 gregorycu Hooray!
01:59 gregorycu What does it do?
01:59 est31 it removes a hack
01:59 est31 and cleanup
01:59 gregorycu In what area?
02:00 est31 src/sound_openal.cpp search for "TODO: Actually load it directly from memory"
02:00 est31 I implement the TODO :)
02:01 gregorycu Nice
02:01 gregorycu Tell you what's funny
02:01 gregorycu I will be working on something similar(ish) soon
02:02 gregorycu And that loading mods from .zip files
02:04 cib0 joined #minetest-dev
02:05 crazyR_ joined #minetest-dev
02:15 est31 #3367
02:15 gregorycu At the moment it looks like all zlib serialisation happens on the main thread
02:15 gregorycu deserialisation
02:15 est31 https://github.com/minetest/minetest/pull/3367
02:15 est31 !title
02:15 ShadowBot est31: Error: The command "title" is available in the Format and Web plugins.  Please specify the plugin whose command you wish to call by using its name as a command before "title".
02:15 gregorycu Which means spikes during gameplay
02:15 gregorycu Is there any hard in moving the deserialisation to the transport thread?
02:16 est31 that would make sense, agreed
02:16 est31 but it would require a change in structure
02:16 est31 and question is whether the new structure is as nice as current one
02:17 est31 right now there is deserialisation of data + acting on data intermixed
02:17 est31 so if we did what you propose they had to be separated from each other
02:17 gregorycu Sorry, I mean decompress
02:17 gregorycu Not deserialise
02:18 est31 well, the compression is added at a very high level in the protocol
02:18 est31 and the connection receive thread works on a lower level
02:18 est31 so what I say still applies
02:18 gregorycu True
02:18 gregorycu Fuck
02:19 gregorycu GUI thread really needs to stop doing so much work
02:19 est31 even if we added yet another thread ^ tm only for decompression it would still require to separate decompress from deserialisation
02:19 gregorycu I don't like having a gazillion threads
02:20 gregorycu Was someone doing work in this area?
02:20 gregorycu Sending mapblock diffs etc.?
02:20 est31 adding a gazillion threads?
02:20 est31 no
02:21 est31 sending mapblock diffs? not that I can recall
02:21 est31 we do already something like that
02:21 est31 we send single node added / node removed events
02:21 est31 so we can't really improve on this alot
02:21 est31 except what RBA proposes ofc
02:21 est31 doing the same for metadata changes
02:22 est31 he even implemented it, just there are some issues which still aren't resolved
02:22 est31 other way to improve would be that the client actually caches the server map on disk
02:23 est31 protocol could look like that the client sends all the mapblock timestamps to the server, server only responds with newer mapblocks
02:23 est31 responding with diffs wouldn't be prefferable here either, as for that the server would need to know the history
02:23 est31 which it doesn't know
02:23 est31 well it does have rollback
02:24 est31 but thats not always enabled
02:24 est31 and rollback scanning is comparatively slow
02:24 est31 it involves sql query, these stuff is usually very slow if done for each mapblock
02:25 gregorycu I don't think it's too bad
02:25 gregorycu It's just that this stuff happens in the GUI thread
02:25 gregorycu And it causes jitter
02:25 gregorycu Like, makes 1 frame out of 30 to take 10 times longer
02:25 est31 well you are generally right, but it should perhaps be worth to benchmark first
02:26 gregorycu Someone has done this... um
02:26 est31 e.g. have a us counter inside the code that unzips stuff like the mapblock data
02:26 est31 and count it up
02:27 est31 then play for 1 minute with very fast map loading
02:27 est31 and take the value
02:27 est31 the rest is math, to get percent number
02:27 gregorycu What are you trying to determine?
02:28 est31 percentage of time spent in unzip
02:28 gregorycu I can tell you that already
02:28 gregorycu Let me do the calculations
02:30 RealBadAngel joined #minetest-dev
02:30 RealBadAngel hi
02:30 est31 hello
02:31 gregorycu We are looking at 0.43%
02:31 gregorycu Which seems trivial, but you have to remember that's the average for all frames
02:31 gregorycu Not the amount for a "bad frame"
02:31 est31 yeah
02:32 gregorycu I am making the assumption that the bad frames are caused by Client::ProcessData
02:32 gregorycu Which is _probably_ correct
02:33 est31 I guess there are heuristics to find that out as well
02:33 est31 just measure its performance and whole frame perf
02:33 est31 and if whole frame perf is spiking, print the spiking perf and the perf of ProcessData.
02:33 est31 anyway, gotta go
02:33 est31 bye!
02:33 gregorycu Easier said than done...
02:34 gregorycu Non-trivial to isolate frames
02:34 RealBadAngel btw, i saw another voxel game on droid, theres nicely solved placing/removing nodes, there are just two buttons for that "+" "-", like just two mouse buttons
02:54 cmdskp joined #minetest-dev
02:59 cib0 joined #minetest-dev
03:16 Supertanker2 joined #minetest-dev
03:26 Supertanker2 joined #minetest-dev
03:29 cmdskp left #minetest-dev
03:36 Warr1024_ joined #minetest-dev
03:39 Warr10241 joined #minetest-dev
03:40 Warr10242 joined #minetest-dev
03:41 Warr1024 joined #minetest-dev
03:42 Warr1024 joined #minetest-dev
04:01 Supertanker2 joined #minetest-dev
04:04 Miner_48er joined #minetest-dev
04:08 paramat joined #minetest-dev
04:11 paramat gregorycu, i not sure if hmmmmm is actually working on the 'liquid spreads on ignore' bug, if you feel like working on it yourself
04:15 paramat i'll merge https://github.com/minetest/minetest/pull/3364 'Mgfractal: Create a choice of 4 mandelbrot formulas' very soon since it's dev work on my mapgen
04:22 paramat now merging 3364
04:29 paramat done
04:35 Fritigern joined #minetest-dev
04:45 VanessaE joined #minetest-dev
04:46 paramat now merging game 719 game 723
04:55 paramat done
04:58 paramat it's not a big issue, the code varies with the use of commas
04:59 paramat talk in channel don't pm
05:00 paramat i might even fix it myself soon
05:02 paramat anonymous pms from username 'guest****' don't carry much weight =}
05:03 paramat left #minetest-dev
05:08 nore_ joined #minetest-dev
05:11 Garth joined #minetest-dev
05:14 cib_ joined #minetest-dev
05:14 exio4 joined #minetest-dev
05:15 paramat joined #minetest-dev
05:15 paramat now merging game 727 (trivial fix)
05:19 paramat done
05:20 _tutima joined #minetest-dev
05:20 paramat left #minetest-dev
05:49 Brains_ joined #minetest-dev
05:51 janakas joined #minetest-dev
05:53 leat joined #minetest-dev
05:55 pozzoni joined #minetest-dev
06:31 Hunterz joined #minetest-dev
07:02 sofar_ joined #minetest-dev
07:03 nrzkt joined #minetest-dev
07:11 cib0 joined #minetest-dev
07:20 paramat joined #minetest-dev
07:21 paramat https://github.com/minetest/minetest/pull/3365 'Decoration API: Allow very low density decorations' is ready for review
07:23 hmmmm looks good to me
07:24 hmmmm well one minor thing, the check for deco_count_f < 1.f in the else if clause is redundant
07:25 paramat oh yeah .. thanks
07:26 paramat it's great how you usually spot something i missed =)
07:47 paramat updated will merge after checks
07:48 hmmmm https://github.com/minetest/minetest/commi​t/987d6a82f639126d5d8833739499150da286bdda​#diff-40af646e2d53a4cf277cca877b995e14R429
07:48 hmmmm this is exactly the kind of thing you should use an enum or constant macros for
07:50 paramat ah i did wonder about my use of an int there
07:50 paramat now merging
07:52 sofar joined #minetest-dev
07:56 paramat done. yes i'll look into that
07:58 paramat left #minetest-dev
08:10 hmmmm yup
08:11 hmmmm https://github.com/kwolekr/minetest/commit​/095627968b1b5ae0352b5eb66daf7c76c6be2321
08:24 celeron55 oh gregorucy wants blocks to be serialized in a thread?
08:24 celeron55 well i guess i could do that
08:25 celeron55 i'm guessing that the time spent doing that is pretty small but i haven't checked
08:25 hmmmm ideally these sorts of things would be done in the connection receive handler or something
08:26 hmmmm it's not about the time spent, but rather the jitter caused by suddenly doing zlib decompression (read:  unnecessary not-rendering tasks)
08:26 celeron55 it's done there, and the handler operates in the main thread both on the server and the client
08:26 celeron55 i don't think it actually causes jitter though
08:27 celeron55 what causes annoying jitter depending on hardware is pushing meshes to the GPU, in my experience
08:27 celeron55 for what i have seen, zlib is insanely fast; it's not a problem
08:27 celeron55 but i need to profile this a bit
08:28 hmmmm remember you need to do all of this stuff in 16 milliseconds
08:29 celeron55 what time do we consider too long, given a situation where the game is running at 60fps?
08:29 hmmmm I don't know but it simply needs to not add up to 16 milliseconds except under very rare circumstances that are difficult to account for
08:30 hmmmm you have this fantastic client event queue setup that a separate thread could be a producer of
08:30 celeron55 like, maybe it's not a problem if one block takes less than 0.2ms
08:30 hmmmm do you really get one block?
08:31 hmmmm or do you get a couple hundred blocks all of a sudden
08:31 celeron55 no; but a few
08:31 celeron55 the networkings work in such a way that it's practically impossible for the client to receive more than 10 per frame; it's more usual to be 1 per frame
08:33 celeron55 and that is intentional, because mesh generation can't generally be done faster than one per frame because it seems that some graphics hardware really doesn't like it when you create more meshes than that in one frame in the way irrlicht adds them
08:33 hmmmm touche
08:33 hmmmm is this bottleneck an irrlicht problem mostly?
08:35 celeron55 i'm sure with modern opengl that bottleneck wouldn't exist
08:36 celeron55 or... well not sure, but i would guess so
08:37 hmmmm well if you want to be the one to test that be my guest
08:37 celeron55 anyway i'll add some stuff to the built-in profiler and see what the serialization and deserialization numbers look like
08:37 hmmmm it's called mineTEST after all
08:37 celeron55 i have never actually checked how much time the zlib calls are taking
08:42 RealBadAngel celeron55, http://irc.minetest.ru/minet​est-dev/2015-11-14#i_4449752
08:44 celeron55 ???
08:45 RealBadAngel hes talking about time spent unzipping stuff
08:45 celeron55 yes guess what you smartass? i already read all that
08:46 celeron55 that's why i am talking about it
08:46 RealBadAngel was kicked by celeron55: RealBadAngel
08:48 RealBadAngel joined #minetest-dev
08:58 celeron55 looks like on my machine generally the zlib-compressed mapnode data deserialization takes about 50us, but if i open VanessaE's crazy world, the highest i can see is 144us
08:58 celeron55 considering it's plausible that someone would be using a computer with the same GPU speed and half the CPU speed, that is a bit high
09:13 celeron55 the thing people forget is that there's way more happening in a mapblock deserialization than just a zlib decompress; the zlib decompress is generally about 67% of it, but it's not unusual for it to be as low as 10%... and now i have seen one that took 15ms in total
09:13 celeron55 i don't know what it's doing nor is it actually important, but i mean, stop talking about zlib
09:13 celeron55 it's not the issue here
09:13 celeron55 the whole deserialization is the issue
09:13 cib joined #minetest-dev
09:15 celeron55 (that 15ms MapBlock is one on the crazy world; it's probably pure minetest abuse but that doesn't mean it wouldn't lengthen the rendering of a frame by 15ms)
09:21 Krock joined #minetest-dev
09:29 ElectronLibre joined #minetest-dev
09:31 Calinou joined #minetest-dev
09:45 Amaz joined #minetest-dev
10:17 ShadowBot` joined #minetest-dev
10:19 rom1504_ joined #minetest-dev
10:21 Calinou joined #minetest-dev
10:28 Wayward_One joined #minetest-dev
10:34 zupoman joined #minetest-dev
10:34 zupoman joined #minetest-dev
11:02 celeron55 just so that gregorycu knows if he's reading this: i'm not going to modify the basic packet handling framework in the client, so making it asynchronous is totally something that someone can start looking into
11:03 celeron55 (with the main goal of getting deserializations out from the main thread)
11:07 who_wants_some joined #minetest-dev
11:18 cib0 joined #minetest-dev
11:45 rubenwardy joined #minetest-dev
11:47 gravgun joined #minetest-dev
11:52 deltib_ joined #minetest-dev
11:55 rubenwardy sfan5, ERROR[CurlFetch]: servers.minetest.net/announce not found (HTTP response code said error) (response code 500)
12:04 sfan5 rubenwardy: works fine here
12:26 rubenwardy misspelt my server's address
13:19 Player2 joined #minetest-dev
14:23 Soni joined #minetest-dev
14:40 hmmmm joined #minetest-dev
16:03 rubenwardy joined #minetest-dev
16:05 zat1 joined #minetest-dev
16:13 Hijiri joined #minetest-dev
16:24 zat1 joined #minetest-dev
16:32 damiel joined #minetest-dev
17:33 Darcidride joined #minetest-dev
17:43 est31 joined #minetest-dev
17:45 est31 I dunno, is it worth a commit to remove #include "filesys.h" from  src/sound_openal.cpp ?
17:45 est31 its not needed anymore
17:47 turtleman_ joined #minetest-dev
17:53 * Krock looks up to celeron55 and wonders what the weather on top of this throne is
17:54 celeron55 my throne has a roof
17:54 celeron55 so it's kind of warm and non-wet
17:55 celeron55 outside the roof it's so dark and depressing that it would kill any non-finnish person
17:56 Darcidride joined #minetest-dev
18:02 Krock Break the roof and rise!
18:03 who_wants_some and fix bugs *cough*
18:06 est31 hah
18:06 est31 I compared opus to vorbis now
18:06 est31 and in fact they both produce the same file size
18:07 est31 even though vorbis includes a 3-4kb header codebook
18:07 est31 which opus doesnt
18:07 est31 but at same bitrate of 161 kb/s
18:07 est31 they both need 13 kb for default_break_glass.1
18:07 est31 of mtgame
18:08 est31 but on the long term, I do think we should add opus support
18:09 Calinou yeah, Godot now supports Opus
18:09 Calinou it's the future :)
18:10 est31 it does*
18:10 est31 ?
18:10 est31 wow thats great!
18:10 est31 since early october it seems
18:10 est31 6e87314d83a9beab56fdd115277e230ef683c53d ...
18:27 Gael-de-Sailly joined #minetest-dev
18:39 rubenwardy joined #minetest-dev
18:43 cib0 joined #minetest-dev
19:16 zupoman joined #minetest-dev
19:18 est31 joined #minetest-dev
19:42 Hunterz joined #minetest-dev
20:30 celeron55 https://web.nvd.nist.gov/view/v​uln/detail?vulnId=CVE-2015-8126
20:30 celeron55 basically every libpng in this world is vulnerable to this
20:30 celeron55 good luck everyone
20:32 celeron55 https://news.ycombinator.com/item?id=10566664
20:32 celeron55 here's some talk about it
20:33 est31 will get into technews I guess
20:33 est31 and get a catchy name
20:33 est31 and logo
20:33 est31 and wiki page
20:34 est31 not in the "secvulnlulz.wikia.com", but in en.wikipedia.org
20:35 celeron55 this isn't really even funny anymore though; libpng is used everywhere and everyone just pulls data from the public internet into it
20:35 celeron55 minetest included
20:36 est31 soo I should do sudo apt-get update && sudo apt-get dist-upgrade  then?
20:36 est31 and now is when the fun starts
20:36 est31 all the sec people will look at the code and see its all just bs
20:36 est31 and find tons of other vulns
20:37 est31 or perhaps no
20:37 est31 r
20:37 est31 t*
20:37 celeron55 well
20:37 celeron55 at least on fedora 22 (that i use), there is no new enough version of libpng available
20:38 celeron55 i guess the distros will catch up
20:38 * est31 eww
20:38 celeron55 windows is another story though; everyone bundles their own copy of libpng with their programs and they will never be updated
20:39 * est31 waiting for people to upload malicious png files to pastebrd or imgur and post links here
20:40 est31 but its only denial of service, no?
20:40 est31 hrmm wondering what "unspecified other impact" means
20:44 cib joined #minetest-dev
20:44 est31 it says it affects libpng 1.6.18 and previous versions
20:44 est31 I have 1.2.51
20:46 celeron55 the attack vector caused by this is so large that if i was to protect my things against this, i would have to just shut down everything
20:48 celeron55 at least my server has proper security limits in place between services; if one of them is exploited, the others will be fine
20:49 gravgun Oddly enough loading http://www.libpng.org/pub/png/libpng.html fails for me
20:49 celeron55 what's odd about that
20:49 est31 the page is down for me as well
20:49 est31 "server has reset the connection"
20:50 celeron55 obviously everyone tries to open that page right now and it's just a single server
20:50 est31 slashdot effect
20:50 celeron55 with normally almost zero traffic
20:50 gravgun "normally"
20:50 gravgun with relatively frequent spikes
20:51 gravgun i.e. on every CVE published about libpng
20:51 est31 or every new release
20:53 celeron55 but a good question is: where to get libpng 1.6.19
20:55 est31 without browsing too much in the internet, downloading tons of png images
20:55 est31 perhaps use lynx ??
20:56 who_wants_some Q: any advance on liquid over ignore problem? too much stuck water on servers :(
20:56 celeron55 https://github.com/glennrp/libpng
20:56 celeron55 this seems to be a recent enough mirror of libpng
20:58 est31 btw which version of libpng do you currently have?
20:59 celeron55 my main machines that i mostly care about seem to have 1.6.16
20:59 est31 I have 1.2.51
20:59 est31 and it makes me really wonder, because there is a libpng12 package and I have it
20:59 est31 but there is no libpng16 package
21:00 est31 debian has it neither https://packages.debian.org/search?suit​e=stable&amp;section=all&amp;arch=any&a​mp;searchon=names&amp;keywords=libpng
21:01 celeron55 one of them has 1.2.51; it's a linux mint box with a broken package management system so it's just bound to die of old age some day
21:01 est31 wondering whether the 1.2 branch is affected as well
21:04 est31 yuppy, 1.2.51 has the vuln as well
21:06 est31 and oddly, 1.2.54, released on thursday, has it too
21:06 est31 at least I can't see a check in pngrutil.c like added in this commit https://github.com/glennrp/libpng/commit/​0b4e7deab1c736273c2d57a19c1bd996576d93f8
21:09 celeron55 that's not the vulnerability fix
21:09 celeron55 it's after 1.6.19
21:10 celeron55 i guess this is what the CVE is talking about https://github.com/glennrp/libpng/commit/​81f44665cce4cb1373f049a76f3904e981b7a766
21:10 celeron55 hmm maybe not
21:11 celeron55 well dunno
21:13 est31 http://www.openwall.com/list​s/oss-security/2015/11/12/2
21:13 est31 hrmm seems 1.2.54 is fixed after all
21:13 est31 at least he claims it is
21:20 est31 joined #minetest-dev
21:21 Amaz joined #minetest-dev
21:22 est31 lol had a kernel panic
21:22 est31 now I am completely hacked :)
21:23 est31 bad guys have root now
21:24 proller joined #minetest-dev
21:31 proller joined #minetest-dev
21:31 est31 debian only has 1.2.50
21:41 celeron55 one thing is for certain: writing security-critical software in this multiplatform macroassembly is really just fucking stupid
21:42 est31 https://github.com/PistonDevelopers/image-png
21:42 est31 I for one welcome our new rusty overlords
21:43 est31 btw do the windows builds of minetest bundle libpng?
21:43 est31 would it be an idea to update them?
21:43 * est31 wonders about android
21:44 celeron55 of course they bundle libpng; it's not like windows has it built-in
21:45 celeron55 i guess it usually comes directly from irrlicht
21:45 celeron55 irrlicht depends on it, and minetest uses it through irrlicht
21:47 kahrl I suppose one could write a PNG loader based on GDI+ instead of libpng
21:47 celeron55 irrlicht 1.8.3 bundles libpng 1.5.9
21:48 est31 ah so they _bundle_ it...
21:48 est31 not depend
21:48 celeron55 well it makes building irrlicht on windows reasonable
21:49 celeron55 i guess on linux it usually is dynamically linked to system libpng
21:49 celeron55 you can do it on windows too of course
21:49 celeron55 not that anyone does it but anyway
21:50 est31 yea on linux it is linked
21:50 est31 well, irrlicht isn't worse than godot
21:51 est31 godot has 1.5.4
21:51 est31 https://github.com/godotengine/godot/issues/2787
21:53 celeron55 https://github.com/zaki/irrlicht/blob/maste​r/source/Irrlicht/CImageLoaderPNG.cpp#L155
21:53 celeron55 of course somebody could check whether irrlicht does the vulnerable thing or not
21:53 hmmmm hmrm
21:54 hmmmm unless I'm mistaken, this is a DoS at best, no?
21:54 est31 https://news.ycombinator.com/item?id=10567353
21:54 est31 yes
21:54 hmmmm it's serious because of the scope but it's not like anybody's r00ting us
21:54 celeron55 it's DoS for now
21:55 est31 probably libpng will get a bit attention for the next weeks
21:55 hmmmm ermm
21:57 celeron55 i guess it could be more than DoS depending on the application that uses libpng
21:58 celeron55 it seems to me that irrlicht just converts it from a palette to rgb if it isn't rgb, and to 8 bits if it isn't 8 bits
21:59 hmmmm i think we're technically safe here
22:00 celeron55 i don't understand how nobody found this before though
22:00 celeron55 this seems like something you would easily find by just taking a bunch of png images and fuzzing them under a memory checker
22:00 est31 yup that was how it was found
22:00 est31 http://sourceforge.net/p/png-mng/mailm​an/png-mng-implement/thread/CA%2BPdXcu​hLXJ89s6qjOEcm%2B99eWLmPBFcYSwcwJkajkL​rNRLTeQ%40mail.gmail.com/#msg34581085
22:01 celeron55 404
22:02 est31 what about http://sourceforge.net/p/png-​mng/mailman/message/34577507/
22:02 celeron55 same
22:02 celeron55 it seems sourceforge is partly down or broken
22:02 hmmmm this sounds like downright negligence
22:03 est31 https://web.archive.org/web/201511​14220245/http://sourceforge.net/p/​png-mng/mailman/message/34577507/
22:04 est31 or better, here https://web.archive.org/web/20151114220333/ht​tp://sourceforge.net/p/png-mng/mailman/png-mn​g-implement/thread/CAAoVtZzzx%2BUvXY0ddbY8jSu​WE-v9hqGTQoCLZubyf66hF2AMLQ%40mail.gmail.com/
22:08 celeron55 by the way, godot does something much more delicate: https://github.com/godotengine/godot/blob​/master/drivers/png/image_loader_png.cpp
22:08 celeron55 if est31 is interested
22:11 celeron55 i actually very much like what irrlicht does when compared to that
22:11 celeron55 irrlicht is like "fuck all these special features, just now give me the RGBA8 and we're done here"
22:29 DFeniks joined #minetest-dev
23:57 VanessaE joined #minetest-dev

| Channels | #minetest-dev index | Today | | Google Search | Plaintext