Time  Nick         Message
17:30 iqualfragile please merge that in: https://github.com/celeron55/minetest/pull/426
17:34 celeron55_   ehm
17:34 celeron55_   that will just make huge liquid transform queues when a server runs a long time?
17:35 celeron55_   and they will all be tried every second, or whatever the way that works was
17:35 celeron55_   shouldn't be merged
17:36 celeron55_   (without testing that the queue does not actually grow in a real situation)
17:45 hmmmm        jeepers.... that's insane
17:45 hmmmm        don't do it
17:47 celeron55_   hmmmm: what is the status of your... thing? people are telling all kinds of rumors about it, most of which are quite far from reality 8)
17:47 hmmmm        oh shoot
17:47 hmmmm        it's done
17:47 celeron55_   i might want to write a blog post about the status of many things
17:48 hmmmm        i haven't been doing anything minetest related for about a week, i apologize.
17:48 celeron55_   doesn't matter; can you summarize what it does and what it needs, so that it can be replied to anyone who asks
17:49 hmmmm        sure
17:49 celeron55_   +?
17:49 hmmmm        where would this be posted?
17:49 celeron55_   dunno, maybe c55.me/blog, and if not, then at least everybody in this channel knows it and can inform others properly
17:50 hmmmm        alright
17:50 hmmmm        i should probably mention that not all the changes in my branch are mapgen related; i did take some time out to fix some minor nagging issues along the way
17:54 iqualfragile oh, yeah, i have to admid i did not look at it, i was just annoyed by that problem
17:54 iqualfragile maybee load the needed blocks?
17:54 celeron55_   it will create infinite loadings of blocks
17:55 celeron55_   we have an infinite world; everything non-linear just stops at some point, there's no silver bullet to that
17:55 celeron55_   (dunno if non-linear is the right term though)
17:57 doserj       the proper way would be to store the queue with the block
17:57 doserj       and add it to the real queue when the block is loaded
17:57 doserj       (I guess)
17:58 celeron55_   i think it would work if water would stop, without spreading, when it touches unloaded mapblocks, and when a mapblock is loaded, it would "activate" all water touching it to make it move if it is about to move
17:58 celeron55_   s/water/liquid/
17:59 celeron55_   if somebody does that, it will be merged
17:59 doserj       you need more than just touching
17:59 doserj       you need to activate all water in the block itself
17:59 doserj       (the block may be unloaded while water is flowing inside it)
17:59 celeron55_   well that is obvious (dunno if it is done already)
18:01 doserj       I'm pretty sure it doesn't
18:04 doserj       And you have to be careful with your solution that it doesn't blow up when you load an ocean block full of water
18:09 celeron55_   indeed that could be bad
18:10 celeron55_   maybe the queue should be stored then, for the in-block cases - it's actually a different case compared to the one that started this discussion though
21:49 * celeron55_ gives a bit of more visibility to this: http://forum.minetest.net/viewtopic.php?id=4243
22:07 hmmmm        guys
22:07 * VanessaE   raises an eyebrow
22:08 hmmmm        i think we should all start working on minecraft more
22:08 hmmmm        minetest
22:08 VanessaE     I thought we were?>
22:08 hmmmm        we were but not enough
22:08 hmmmm        there's like nothing going on :(
22:08 hmmmm        i have a goal in mind; by friday at the latest we should have master with all the mapgen stuff and leveldb stuff added
22:09 hmmmm        and more than that.. before the last bit of snow melts for me outside, I want to have snow on the blocky ground in a winter biome in minetest
22:10 VanessaE     how much leveldb "stuff"?
22:10 hmmmm        all the leveldb 'stuff'
22:10 VanessaE     just the framework, or the whole backend like oldcoder wants?
22:10 hmmmm        the whole backend
22:10 hmmmm        plus a converter for sqlite3 maps
22:10 hmmmm        oh wow that might be a bit taller of an order
22:10 hmmmm        okay nevermind friday
22:10 VanessaE     sweet
22:10 VanessaE     get with oldcoder - he's already done this
22:10 hmmmm        well
22:11 VanessaE     he runs leveldb on his server 'nexus'
22:11 hmmmm        thexyz is the person i'd really want to talk to
22:11 VanessaE     at least for linux...
22:11 init         leveldb?
22:11 hmmmm        it's an alternative to sqlite3 that runs a whole lot better
22:11 init         what is it? new format for maps faster?
22:11 hmmmm        yes, it makes maps faster :).
22:11 init         ;P
22:11 hmmmm        so here we are with leveldb
22:12 hmmmm        the framework and the backend itself have been tried and proven to work well for a couple months now
22:12 hmmmm        plopping that in should be no problem
22:12 hmmmm        i don't know what it does about conversion of old maps, which is where i'd ask xyz
22:13 hmmmm        but ideally what would be done is all old maps are _converted_ to the new leveldb format
22:13 hmmmm        and then eventually we remove the code supporting loading from and everything from 0.3 sector directory and and 0.4 sqlite3
22:13 sapier       I don't want to offend someone but issues are queuing up really fast and noone doesn't even seem to categorize them
22:14 hmmmm        issues on what
22:14 init         github i think
22:14 sapier       yes github... isn't bugfixing more important than adding new featues?
22:15 hmmmm        a lot of those are enhancements and very non-critical bugs
22:15 sapier       of course but without categorizing it you can't decide wich are critical bugs and which are feature requests
22:15 VanessaE     hmmmm: oldcoder has also gotten quite proficient at migrating sqlite maps -> leveldb btw
22:16 hmmmm        sapier, i know, but just froma glance it really does look like the majority of them are enhacements
22:16 hmmmm        from a*
22:16 hmmmm        vanessa, that's nice, but it's not particularily difficult to do so
22:16 hmmmm        the plan is to have it completely automated
22:17 VanessaE     hmmmm: just trying to save some time.
22:17 init         i wonder
22:17 hmmmm        user upgrades minetest, copies over map, starts it up, says "hey your map format is out of date, to use this version you need to update it!"
22:17 VanessaE     init: I already know, and I'm not sure I like what I know ;-)
22:17 init         why not a "big upgrade" between adding the new mapgen / new format
22:17 VanessaE     (as opposed to just wondering ;) )
22:17 hmmmm        and then it asks for a confirmation if you'd like to overwrite the file or move it to another or something
22:17 VanessaE     oh yes
22:17 VanessaE     +1
22:18 sapier       there are at least some important bugs ... entity attachement is buggy, collision handling too, entity client/server out of sync and of course the famouse emerge thread bug is still there
22:18 hmmmm        hm
22:18 VanessaE     leveldb should be enough to bump the version to 0.4.5.
22:18 sapier       I know none of that bugs is easy to solve and won't give lots of merits too ... but they are really anoying
22:18 hmmmm        the emerge bug thing.... i'll take a better look at that once my current work is done
22:18 hmmmm        collision handling is totally not my area
22:18 hmmmm        i have no idea what's going on there or even what source file it resides in
22:20 sapier       for this special one I've already written a changeset. But it needs discussion if this way of fixing is accepted or not
22:21 VanessaE     hmmmm: regarding emergethread, one thing I've noticed is that the more map content there is (even if it's nothing but a bunch of idle blocks), the sooner the emergethread bug manifests.
22:21 VanessaE     is that of any use?
22:21 VanessaE     at least it *seems* to be that way
22:21 hmmmm        we already know what the problem is exactly, just not a good way to solve it
22:21 hmmmm        i have an idea that i'd like to try out, but after i get my other stuff in
22:21 sapier       I had a thought about this ... did I understand correct that the real problem is some client not receiving data fast enough?
22:22 hmmmm        no
22:22 hmmmm        the real problem is that the emerge queue fills up too much and it keeps getting enumerated every tick in the serverthread by getnextblocks
22:23 hmmmm        sendblocks holds an envlock which the emerge thread waits on
22:23 sapier       why?
22:23 hmmmm        so when the emergethread finally gets a chance to run it only clears a couple hundred blocks from the queue and more keep getting added on
22:24 sapier       yes but why is sendblocks holding that lock?
22:25 hmmmm        because it reads from the environment and it needs to be in a consistent state
22:25 sapier       so the real problem is environment being a non shareable resource
22:26 hmmmm        i would have to take a better look but i feel like SendBlocks has too broad of an envlock
22:26 hmmmm        really envlock is too big
22:27 hmmmm        it would be really great if we sit down someday and actually take a look at what resources are being used at what times and make some smaller granularity locks for things within the environment
22:27 sapier       I agree
22:28 sapier       I haven't had a look if this idea might work but what about block range locking of env?
22:28 VanessaE     [01-16 17:28] <Menche> could they fix the bug where you have to jiggle the mouse to open a formspec after just closing one?
22:28 VanessaE     for the logs ^^^
22:28 sapier       jiggle?
22:28 hmmmm        sapier, that too
22:28 sapier       what exactly happens there?
22:29 VanessaE     sapier: if you open a chest and then close it, you can't hit any other key to do anything, even move.  you have to jiggle the mouse a little to reset keyboard focus back onto the world.
22:29 VanessaE     or something like that.
22:30 sapier       might be combinable with on_close callback
22:30 VanessaE     (I did not want to steer the conversation this direction, that was supposed to just be for the logs)
22:31 sapier       I think as there's a easy workaround for this bug it's not quite high priority
22:31 hmmmm        in the meantime i think it might be a good idea for my code to actually ship, the emergethread getting done with its stuff quicker would be a way to alleviate the problem (but not actually fix the consumer starvation)
22:31 sapier       related to emerge thread is a problem in on_generated callback
22:31 hmmmm        oh god....
22:31 hmmmm        let's hear it
22:32 sapier       some ppl use on_generated to heavyly modify world
22:32 VanessaE     oh boy.
22:32 sapier       but behaviour of setnode is not intended to do that at all
22:32 VanessaE     jeez, speaking of emergethread, my server just went down hard :-/
22:33 hmmmm        hrmm
22:33 hmmmm        there are also other remedies for fixing that
22:33 sapier       any node modified by setnode will touch at least 6 other nodes queuing them to transform queue in worst case and doing light calculations
22:33 hmmmm        like make the queue count for each peer an O(1) lookup so it spends the same amount of time in sendblocks
22:34 hmmmm        and also my work that i'm planning to do with the emergethread would fix that as well, that is, do a trylock when blitting the new blocks back to the environment
22:34 hmmmm        instead of waiting
22:34 sapier       that's not a solution hmmmm it will improve situation but won't fix it
22:34 hmmmm        what, the trylock?
22:35 hmmmm        all of these are things that should have been done in the first place
22:35 sapier       not the separate queuing for each client
22:35 sapier       still it should be done for other reasons
22:40 sapier       puhhh ... I've just had a look at sendblocks function ... where I work we do call functions like that "infinite loop"
22:43 sapier       those loops don't have a worst case limit
22:43 VanessaE     sapier: do you think you can fix it?
22:44 sapier       you're kidding ;-) to fix bugs like that you need to reconsider what you're doing and what you really need to be done
22:45 VanessaE     ohheh
22:45 sapier       I think hmmmm has already some interesting ideas
22:45 VanessaE     agreed, for as much as I can understand it anyway.
22:46 sapier       hmmmm I still don't understand why envlock is taken for whole sendblocks function
22:47 sapier       is consistency between different clients really required?
23:09 sapier       VanessaE could you have a look if this change reduces emerge thread problems? https://github.com/sapier/minetest/commit/81098c8e112c24bd503473a4b5b5eb26b388d8f2
23:09 VanessaE     sure, lemme pull that in real quick
23:10 sapier       I don't think it'll fix it completely but as far as I understand that code it most likely won't harm
23:11 VanessaE     building it now...
23:11 sapier       still real problem is the way block updates are done
23:12 VanessaE     ok, it's installed.
23:12 sapier       and we need some testcase to decide if a change improves or makes things worse ... as I don't think there's a single fix to make it instantly run
23:12 init         VanessaE: how can you "reproduce" the bug?
23:13 VanessaE     init:  just have a really detailed, heavily-loaded map
23:13 VanessaE     e.g. tons of buildings, or lots of conifers, jungle trees, etc.
23:13 VanessaE     fly around for a while
23:13 sapier       :-) not qute a testcase
23:13 init         my netbook can't handle that, so i can't try :P
23:13 sapier       you can't even decide "fixed" with it
23:14 VanessaE     sapier: I don't see any improvement... :(
23:15 sapier       I thought so ... env lock is huge spliting it wasn't quite promising ... 2big locks are still to big
23:16 VanessaE     it was worth a try
23:17 sapier       so this bug obviously needs mor investigation maybe whole mechanism needs to be redesigned ... but maybe some small adjustments are enough ... don't quite know by now
23:26 sapier       VanessaE do you run linux?
23:26 VanessaE     sapier: yep
23:28 sapier       hmm ... I have to check something maybe there is a quick solution for this problem
23:29 VanessaE     oh?
23:30 sapier       linux uses fifo as default scheduler ... this results in not changing from a busy thread to a waiting thread without being forced
23:30 sapier       this isn't as critical in multicore environments ... how many cores do you have?
23:33 VanessaE     6 cores.
23:33 VanessaE     and I think it uses a different kind of scheduler these days, "Completely Fair" they call it.
23:33 sapier       hmmm ... ok that doesn't comply with my guess
23:33 sapier       process scheduling but not pthread thread scheduling
23:34 VanessaE     oh right
23:48 sapier       https://github.com/sapier/minetest/commit/3861bf3d86689d85cf3b08324dfea98b160326fc
23:48 sapier       VanessaE probably adding this improves behaviour ... but switching from fifo to rr may show bugs not beeing visible before
23:50 sapier       I remember you had a link to that world triggering the problem yesterday? can you give it to me so I can test myself?
23:50 VanessaE     sure, world:  http://digitalaudioconcepts.com/vanessa/hobbies/minetest/My_World_server.tar.bz2   game:  http://digitalaudioconcepts.com/vanessa/hobbies/minetest/vanessa_game.tar.bz2
23:51 VanessaE     (you'll need the game or the world will be like 50% unknown blocks ;-)
23:52 sapier       127MB? .... I think that'll take some time
23:53 sapier       that commit changes two things, SCHED_RR for all threads and lift emerge thread above all other threads
23:54 VanessaE     127MB is nothin' compared to redcrab's server.  His world file is in the 5GB range now :-)
23:55 sapier       :-)
23:55 VanessaE     also, the map:  http://digitalaudioconcepts.com/vanessa/hobbies/minetest/images/my_world_map.png
23:55 VanessaE     so you can see where the dense areas are.
23:56 sapier       do you have some coordinates? :-)
23:56 VanessaE     not really, just fly around and eventually it'll barf :-)
23:58 VanessaE     the spawn is somewhere around 130,25,100 or so
23:58 sapier       is this bug occuring singleplayer too?
23:58 VanessaE     -138,24,100 to be exact.
23:59 VanessaE     the only thing I've seen singleplayer is the map sometimes loads slowly, but I never play in that mode anymore
23:59 VanessaE     I just use it for the occasional test of a mod