Time Nick Message 03:51 VanessaE sapier: So far, no issues with your pull. I think it's good to merge. 09:08 sapier Ok preparing network fixes for merge, I'm gonna merge the mutex queue in same step but as separate commit as it's required too 09:09 sapier mutex fixes have been part of 1090 so they're tested too 09:14 kahrl those names (pop_frontNoEx) are a really weird mix of snake_case and camelCase 09:14 VanessaE is THAT what you call this_sort_of_naming ? 09:14 VanessaE (I'm rather partial to that rather than camelCase) 09:15 kahrl even wikipedia calls it that! https://en.wikipedia.org/wiki/Snake_case 09:16 VanessaE I never knew that. 09:20 sapier that naming was already used somewhere else within minetest for no exception functions so I reused it 09:21 sapier I didn't want to introduce a new naming 09:21 kahrl maybe pop_front_no_ex 09:22 kahrl or get rid of the exception-throwing variant completely 09:22 kahrl (and call the new version pop_front) 09:23 sapier was suggested before but as you didn't mention this to be a problem for more then 3 weeks now (mutex changes are there for so long) noone tested sideeffects of a change like that 09:23 sapier I principle I think you're right 09:23 kahrl iirc someone complained about the names before 09:24 kahrl don't remember for sure though 09:24 sapier I remember only the suggestion about removing the non exceptioning functions, but I'm not 100% sure too 09:26 sapier but as I said, mutex fixes already had some side effects I'm almost sure replacing all exceptioning variants by non exceptionings will result in problems 09:26 kahrl right 09:27 kahrl anyway I won't block those changes just because of the names 09:28 sapier thanks ... there's a lot of cleanup to do in server removing usage of exceptions as return values is one of the things 09:29 sapier basically it's abused as some weird variant of cross stack goto :-) 09:30 sapier ok lets make more people test those changes mergeing now 11:02 proller workaround_window_size max_packets_per_iteration - WHYYY? 11:02 proller nobody never will change it 11:03 proller network settings must have good hardcoded defaults 11:03 sapier if you don't know what this is for you don't have to change it ;-) 11:03 proller nobody know except you 11:04 sapier there's no way to hardcode settings for the variaty of our usecases 11:04 VanessaE what's good for someone in rural backwoods Russia (hi xyz ;) ) won't work well for someone in inner-city New York 11:04 sapier we've got adsl bandwith up to 100mbit ... no sane defaults will ever match this 11:04 proller okay, who can change it, for what and to what ? 11:05 sapier window size is supposed to be removed once old clients die out ... so about 2016 ;-) 11:06 sapier I would've set it to 1 in order to fix the old clients reliability issue ... but this doesn't work for high latency connections so 5 is a tradeoff between client bug occurance and speed 11:06 proller 1989 u32 datasize = m_max_packet_size * 2; // Double it just to be safe 11:06 proller --- 11:06 sapier but you're right, I shoud've added a sentence about it, I'm gonna fix it 11:06 sapier that's one of the rare parts I haven't changed proller ;-P 11:06 proller increase recv buffer now!!! 11:07 sapier for what reason? 11:07 proller to 1500 or 65535 11:07 proller to increase packet size in future 11:07 sapier we can't increase packet size in a compatible way so why do it? 11:08 sapier and don't forget this is not the final solution 11:08 proller you can increase this bufeer now, and +2 versions increase sending packet 11:09 proller increasing very needed, because average block larger than current 500 bytes 11:09 proller but less than 1000 11:09 sapier we had a discussion about this some days ago as I suggested increasing it to 1200 too 11:10 proller recv buffer can be any, better to 10000-65000 11:11 proller it not broke anything, but helps in future 11:11 sapier basicaly the result was for ipv4 networks biggest reliable packet size is 576 bytes, if we assume we have ipv6 infrastructure everywhere we might be able to increase to 1360 11:12 proller WHAT 11:12 sapier proller if you suggested this yesterday it'd be already in 11:13 proller you can send 60k packet via udp 11:13 sapier ipv4 only guarantees to send datagrams with 576 bytes any other packet is legal to be dropped without notification 11:13 proller udp and notification ?? 11:14 sapier of course most network components will support 14xx today 11:14 sapier yes udp does notifications e.g. port unreachable 11:15 proller inside game packets stream?? 11:15 sapier I guess you won't get a port unreachable in middle of communication ;-) maybe at end ... but due to inet packet structure you might receive delayed packets later 11:15 proller can you increase it now to 64 without 2 days talking about nothing? 11:16 sapier not now 11:16 proller okay, in 5 mins ;) 11:16 proller and 11:16 proller - MeshUpdateResult r = m_mesh_update_thread.m_queue_out.pop_front(); 11:16 proller + MeshUpdateResult r = m_mesh_update_thread.m_queue_out.pop_frontNoEx(); 11:16 proller WAT 11:16 sapier no on next protocol fixes 11:17 proller +1 for removing NoEx and making all pop_front no-exeption 11:17 sapier +1 too but will cause a lot of new issues 11:17 proller or add Ex version if very needed 11:20 proller when you will make next protocol fixes? 11:21 sapier don't know depends on what's gonna happen next days 11:53 proller -std::list ReliablePacketBuffer::getTimedOuts(float timeout) 11:53 proller +std::list ReliablePacketBuffer::getTimedOuts(float timeout, 11:53 proller + unsigned int max_packets) 11:53 proller wat 11:53 proller you have 4sp tab? 11:55 VanessaE blame c55 11:56 darkrose blame the gnomes! 11:57 VanessaE hey darkrose 11:57 darkrose hi 11:58 VanessaE darkrose: well seeing as how gnome sucks anyways, I guess it's safe to blame it. 11:58 VanessaE wait, what? :) 11:58 xyz codestyle! 11:59 xyz http://i.imgur.com/3B8cOTT.png 11:59 xyz to illustrate the problem 12:01 sapier iirc we use 4 space tabs as default 12:01 proller sapier, and your diff (as diff) is WTF 12:01 proller everyone use 8 12:02 Exio4 8 sucks 12:02 sapier no you use 8 12:02 Exio4 but what is the point of having tabs if it needs to be fixed 12:02 xyz you aren't supposed to do this with tabs 12:02 xyz yup, Exio4 is right 12:02 VanessaE no, minetest code style uses 4. 12:02 VanessaE it's weird, but that's how it is. 12:04 VanessaE interesting, it isn't mentioned in the code style guidelines anymore. 12:04 sapier I'm gonna change it to two tabs only next time I touch it ... and I always wonder why it's not possible to mention those things PRIOR merge was done, it'd be way less work to fix it then 12:05 Exio4 just use spaces 12:05 sapier no I use the two tabs that seem to be used everywhere else in this file 12:06 sapier code style guidelines feel to be changed once a month the last half year so they're starting to be useless 12:06 VanessaE must have been in the old dev wiki/site/thingy that it was mentioned 12:07 VanessaE http://c55.me/minetest2/wiki/doku.php?id=contrib"Tabs are tabs, and they are visually 4 spaces in width" 12:07 VanessaE oops. 12:08 VanessaE well there it is/was. 12:08 proller щлфн 12:08 proller okay 12:08 VanessaE when was this requirement dropped? 12:09 sapier guess someone who demanded tabs to be 8 spaces dropped it ;-) 12:12 Exio4 tabs should be tabs, not X spaces 12:12 Exio4 something is wrong when talking about them in that way 12:13 sapier code is skrewd up on using a different tab width no matter what you do, it's only about HOW MUCH it's skrewed up 12:14 sapier if you don't believe, set tab width to 32 and you'll see it ;-) 12:14 Exio4 that is an extreme 12:14 Exio4 if you changed to six-spaces tabs, and the code looks fucked up, can anyone tell me what is the whole point of using tabs? 12:14 sapier it's meant as an extreme example ;-) 12:15 Exio4 whatever 12:15 sapier I could say size reduction ... but I guess those 3/7/5 bytes wont really make a difference 12:18 Exio4 my .git directory is about 1.4G 12:18 Exio4 would it be 1.5 if we were using spaces? 12:19 sapier as git uses compression and spaces can be compressed quite good I don't think so 12:23 VanessaE maybe because it takes less time to tap the tab key to indent, than to hold the space bar? 12:23 VanessaE one tab and *click*, there you are 12:23 VanessaE it's about convenience while coding, not about storage 12:25 xyz so we're using tabs instead of spaces because it takes less, erm, space? 12:25 xyz that's pretty autistic, i like it 12:26 Exio4 haha 12:26 Exio4 VanessaE: proper editors let you "place spaces" when you press the tab key 12:26 Exio4 and also remove them by a single backspace if wanted 12:28 sapier xyz "I could say" but I didn't say ;-) 12:28 xyz right 12:28 xyz is this something to argue about? 13:03 proller фпфшт сщкувгьзы щт ыргевщцт 13:03 proller again coredumps on shutdown 13:12 proller sapier, your changes still x3 slower than in freeminer.. 13:12 proller but in FM only ~5 lines tuned.. 13:17 sapier I didn't do a performance fix 13:17 sapier I fixed protocol reliability 13:18 sapier current implementation had major flaws resulting in "reliable" communication not beeing reliable at all 13:18 proller okay 13:18 sapier if there are performance improvements they're result of cleanup 13:19 sapier what test did you do to get that 3 times slower? 13:19 sapier what freeminer variant enet or udp? 13:20 proller udp 13:20 sapier and what did you test? 13:20 proller transfer 13:20 proller enet must be 10x faster 13:20 sapier media transfer? 13:20 proller yes 13:21 sapier curl or non curl? 13:21 proller non of course 13:22 sapier hmm you should do a sha sum check on transfered media with freemine 13:22 sapier r 13:23 sapier old implementation did sometimes corrupt files on download due to assembling wrong packets in a split packet 13:35 proller LOG(a) 13:35 xyz yea but you know 13:35 proller NEED MORE LOCKS!!! 13:35 xyz client checks hashes of all files 13:36 sapier after reception too? 13:36 VanessaE proller: http://onepeggenius.com/wp-content/uploads/2009/07/IMG_5775.jpg 13:36 VanessaE here ya go. 13:36 VanessaE wait, wrong kind of lox.. ;) 13:36 xyz sapier: yeah, like, all the time 13:36 xyz wait 13:37 xyz what do you mean? when else would it check hashes? 13:37 sapier if I recognize correctly files in cache arend hashed any longer as their filename should be the correct hash 13:38 sapier so once a file is there it's just assumed to be correct 13:39 sapier I've seen quite some broken files in cache, they never have been replaced by correct ones so I assume they aren't checked 13:39 xyz maybe, I only checked receiving code, to get into cache it should match sha1 13:40 xyz so your argument is invalid this time, try something else 13:43 proller if (current_bytes_transfered > (window_size*512/2)) 13:43 proller why 512 hardcoded 13:43 sapier proller read back history discussion with hmmmm about 1 or two weeks ago or ask him 13:44 sapier xyz you're right, the files are checked prior storage ... so they most likely have been result of debugging while downloading 13:45 sapier but doesn't change the fact that communication in old implementation wasn't reliable or did you ever manage to transfer 150mb of texture data ? ;-) 13:46 sapier does anyone understand how curl support is supposed to work? 13:48 sapier yet it's interesting if how freeminer achieves to be three times faster ;-) even enet is only 30% faster for me 13:58 proller JMutexAutoLock internal(m_internal_mutex); 13:58 proller current_packet_too_late++; 13:58 proller sapier, can you never use JMutex* , 13:58 proller ? 13:59 proller never more 14:06 sapier of course I can but why should I? 14:06 sapier there's one code location where I had to use it for good reason but in general the autolock avoids forgetting to unlock something 14:07 proller can you stop locking every line? 14:08 sapier no I can't stop locking variables that are accessed by multiple threads as their result may be invalid 14:08 xyz sapier: for some reason your enet felt slower to me than freeminer's enet, i.e. in FM I've managed to get instantly connected to localhost, while in your branch it took a second or two or three 14:09 proller okay, then add more locks to be more stable 14:09 sapier xyz that's been a bug in that implementation it's fixed 14:09 xyz ah, i should check once again 14:09 xyz what branch? 14:09 xyz you have quite a lot 14:09 sapier once I merged it up to enet yes 14:11 sapier proller I don't say everything has to be locked, if you find a lock where you're absolutely sure it's not required we can remove it. I usually add a lock if I have reason to believe a variable is accessed by multiple threads. If something in implementation changes and this reason is gone, it's absolutely possible a now useless lock remains in code. 14:13 sapier actually the less locks the better ... but removing a lock and risiking data corruption isn't an option too 14:16 xyz sapier: so what branch should I use? 14:16 xyz for your enet with all fixes 14:16 sapier it's noit there for testing yet 14:18 sapier I'm looking for reason why curl seems to slow down with new mutexqueues 14:18 xyz ah, alright 14:27 proller ConnectionReceiveThread ConnectionSendThread for WAT??? 14:35 proller sapier, maybe you fix one rare bug, but you make connection much more shitty than was 14:35 proller now you must add tcp to it to increase shitness 14:36 VanessaE proller: you know, you are of course free to help improve the situation. 14:36 VanessaE (without trying to shoehorn weather et al. into the system) 14:36 proller can you show exact lines which solves reliable bug ? 14:37 proller VanessaE, yes, enet. 14:37 VanessaE enet may or may not be the solution. 14:42 proller adding 25 locks is perfect solution! 14:43 proller sapier, when you first time start use threads ? 14:49 Exio4 what do you mean with that 14:49 Exio4 that english is worse than mine 14:57 sapier proller I haven't read a single constructive suggestion from you since you started freeminer the only thing you seem to do since then is throwing in unprooven accusations and unprooven success storys about freeminer. If you have a real, at least somehow specific describet problem with vanilla minetest head I'm gonna investigat it but I'll ignore anything else starting NOW 14:58 xyz hey no need to get so upset over simple things like this 15:00 sapier It's not this simple thing I'm anoyed to waste time for comments that don't seem to be valid at least lots of times ... e.g. 3 times faster ... what does this tell 3seconds instead of one where you have accuracy of 2s? 15:00 proller TOP SECRET: http://paste.org.ru/?7oc144 15:01 proller old minetest have ~ 250k stable transfer 15:01 xyz yes, I can agree to that, comparing two broken protocols isn't fair; that's why I'm waiting for your enet implementation 15:02 sapier 250k? you're doing something wrong I get 1.2mb stable on localhost 15:02 proller can you show exact lines which solves reliable bug ? 15:02 sapier of course not because it's been a design flaw 15:03 proller but now we have x3 more design flaw's [unprooven] 15:03 sapier old implementation did asume 65k packet Id's are enough to never receive a packet from last iteration 15:04 sapier ol tell me about them 15:04 proller threads, locks 15:05 sapier lol ... you're great proller :-) 15:10 proller you add 1200 lines to connection.cpp to solve imagined problem 15:11 sapier actually it'd been only 800 if I didn't include design corrections for tcp/enet in this version ... but this wouldn't matter you'd call anything bad 15:13 proller +400 .h 15:19 sapier yes I'm evil I added 200 lines of rtt and jitter statistic definition to header ;-P 15:23 sapier did anyone else have issues with curl download through proxy server? 15:54 sapier kahrl do you have any idea where to look for a 200ms delay in get requests beeing sent by httpfetch thread? 15:57 sapier requests are sent up to max parallelity then I see a 200ms delay 16:01 sapier ttps://github.com/minetest/minetest/commit/3aa28bc7a2014dcefd9f24517ff32223bec32a35 as of printfing I see a sequence of one 1ms sleep followd by two (sometimes three or more) 100ms sleeps 16:01 sapier https://github.com/minetest/minetest/commit/3aa28bc7a2014dcefd9f24517ff32223bec32a35 16:13 ShadowNinja kahrl: https://github.com/minetest/minetest/commit/b03135548bbd9bcb73d14b067b96ec913404dd5f breaks announcement for me. Any ideas why? Config: http://ix.io/9Ne 16:14 ShadowNinja kahrl: It doesn't send any TCP packets on port 80 at all. 16:21 sapier https://gist.github.com/sapier/8357314 can someone check if I'm really right about this? 16:34 proller - T pop_front(u32 wait_time_max_ms=0) 16:34 proller + T pop_front(u32 wait_time_max_ms) 16:34 proller why? 16:35 sapier because the default value wasn't necessary as I had to touch all files where it was used anyway 16:37 sapier and if remember correctly timeout 0 would've meant something completely different for semaphore based mutex queue 16:37 sapier that's been the reason why I had to touch those locations 16:37 sapier and additionally it was wrong before too as timeout 0 actually was timeout 10ms 16:39 sapier hope I don't mix up this with another change as you didn't post file and line of change 17:57 sapier https://gist.github.com/sapier/8357314 can someone check if I'm really right about this? ANY comments? 18:12 kahrl sapier: uhh yeah... how did this work before? 18:13 VanessaE kahrl: when I applied that change, it increased my rate from <1.2 Mbps to 1.5-3 Mbps. Add curl_parallel_limit = 32 to my minetest.conf and I got >28 Mbps. 18:13 VanessaE (tested several times) 18:13 sapier it waited till all transmissions have been completed 18:13 sapier then you get -1 ... but I should've failed on windows 18:14 sapier merging it now 18:15 sapier done 18:17 kahrl ShadowNinja: can you try it again with the fix sapier just merged? 18:20 kahrl VanessaE: that change = changing "==" to "!="? or the original commit that added the "if"? 18:21 VanessaE kahrl: changing == to != 18:21 kahrl ah 18:21 kahrl yeah, it should increase performance definitely 18:21 VanessaE and, 18:21 VanessaE since we seem to advise using nginx 18:22 VanessaE and clearly nginx can handle 32 simultaneous transfers at once with its default settings 18:22 VanessaE perhaps you should up it from the default of 8 18:22 VanessaE that is, the minetest client setting 18:23 kahrl not sure if setting aggressive defaults is a good idea 18:23 VanessaE well, idk if 32 is what I'd call aggressive, really 18:23 VanessaE based on my measurements, 18:23 sapier but there's still might be a race condition, if transmission is finished prior reading the fd's, at least if curl uses it's own thread to handle communication 18:23 kahrl if you open 32 simultaneous tcp transfers to one server, you effectively disable congestion control 18:23 VanessaE that's only about 64 kB per transfer request 18:24 sapier in this special case we would wait 100ms instead of processing the incoming data 18:25 kahrl sapier: I'm pretty sure curl doesn't use its own thread 18:25 kahrl if it did, it would probably export some kind of simple API to do async requests 18:26 sapier ok so this case propably can't happen 18:26 kahrl but all we get is curl_easy_perform and curl_multi_perform 18:26 sapier so the only thing to think about might be the 100ms timeout beeing a little bit too big 18:31 kahrl sapier: there is also the main thread that checks for completion of requests only every so often 18:32 sapier so probably that doesn't make a difference 18:33 sapier well case we choose to use tcp as successor of our current protocol curl download might be less important ... despite of beeing able to download from a dedicated server 18:34 VanessaE fwiw, I can max out at around 9 Mbps over pure UDP alone now with sapier's latest code that's in master 18:35 VanessaE that's perhaps two orders of magnitude faster than it used to be? 18:50 proller i got 8-9mbps transfer for single player in fm.. 20:22 ShadowNinja kahrl: Still nothing. 23:32 iqualfragile ok, minetest realy needs a primitive for "on" and "off" 23:38 sapier https://github.com/sapier/minetest/tree/network_addon_tcp_2 tcp version (does anyone know how to reduce stall time of a tcp socket on linux?) 23:40 sapier https://github.com/sapier/minetest/tree/network_addon_enet_4 enet variant enet doesn't seem to provide reliable rtt and jitter information help is welcome