Time Nick Message 05:18 hmmmm i tuned up the new mapgen's lua biome definitions and i am seeing that i really can get satisfactory results in terms of biome distribution 05:18 hmmmm i can break up what problems i have into three major things: 05:18 hmmmm 1). height transition from biome to biome - looking at how minecraft does that, but i'm working on other things at the moment 05:19 VanessaE hmmmm: http://pastebin.com/8KRxHRdc <-- note the commented-out stuff. When I did this, the server un-broke and seems to have stayed that way. 05:19 hmmmm 2). biome borders along with the shapes themselves are quite uninteresting and very round: can solve this by adding more octaves and then filling in the centers with a quick algorithm i came up with 05:19 VanessaE height transition: use a double-declining average. 05:20 VanessaE well perhaps not, that'll probably lead to weird shapes when extended to 2d/3d. 05:20 hmmmm and 3). lack of 'neat' features like the hollowed out areas in cliffs and what not; i believe that minecraft does what i'm thinking of, it carves caves out above ground 05:20 hmmmm ah 05:21 hmmmm so let's see what the default values were for those 05:24 hmmmm can't be the first four, your custom active_block_range was 1 more so that's a potential, not the custom save interval, might be server_unloade_data_timeout increase, not sure about the congestion control 05:24 hmmmm let me check out 0.4.4 05:25 hmmmm chances are it has to do with the congestion control 05:25 VanessaE possibly. 05:26 hmmmm I guess you can re-enable your other custom settings 05:26 VanessaE regarding biome borders - which would be faster, adding another octave or two, or adding a few "layers" of sinuses? 05:26 hmmmm sinuses? 05:26 hmmmm hrmm? 05:28 VanessaE sure 05:28 VanessaE well, we called them "plasmas" in the old days 05:28 hmmmm oh 05:28 VanessaE basically you draw a sine wave across the x axis, with the amplitude mapped to colors, then do the same along the Y axis, with the same sort of mapping 05:29 VanessaE then blend the colors at each crossing point 05:29 hmmmm yea 05:29 hmmmm adding octaves here would be better 05:29 VanessaE ok 05:29 hmmmm the speed of the noise isn't really a problem 05:29 hmmmm the reason why the number of octaves is low for the biome group selector is so you don't have too much "complexity" inside 05:30 VanessaE hrm 05:30 hmmmm so it looks coherent and natural 05:30 hmmmm but I can get rid of that problem by doing a pseudo-flood-fill algorithm 05:30 VanessaE what about a second layer of perlin, but it only gets applied to the transitions? 05:30 hmmmm so i get the neat, rough border outline, but none of the junk in the middle 05:30 hmmmm that's the first thing i considered 05:30 VanessaE i.e. you check for block A being near block B for each block you're about to generate, and if they're close enough, apply that second perlin layer 05:30 hmmmm but i have no idea how i'd actually do that 05:31 VanessaE if within your check radius, block A = block B, you don't apply the second layer of perlin 05:31 VanessaE problem is that could be CPU-intensive unless your radius is quite small 05:31 hmmmm i'd have to check if neighbor nodes are a different biome 05:31 VanessaE yes exactly. 05:31 hmmmm and not just for the nearest neighbor 05:32 VanessaE hrm, right 05:32 hmmmm in order to make the transition more than just +- 1 block 05:32 hmmmm it'd have to be like a lot more 05:32 hmmmm which would turn out to be, if you wanted to do a somewhat interesting border, 2^6 checks per node 05:33 hmmmm which is implausible 05:33 hmmmm something better can probably be done cheaper 05:33 hmmmm and that sort of blends in with the problem with the height smoothing 05:33 VanessaE well wait.. 05:33 VanessaE 2^6 checks isn't right 05:33 hmmmm er 05:34 VanessaE you don't have to check the whole area, just at the radius 05:34 hmmmm whoops 05:34 hmmmm yeah that's true 05:34 VanessaE so if you've got a radius of 10, that's about 32 nodes to check 05:34 VanessaE still, it's probably sub-optimal anyway 05:35 hmmmm so let's say you're checking a node right at the border 05:35 hmmmm and you see that at some point in the radius there's a different biome 05:35 hmmmm you go and do your transition 05:35 hmmmm but then 05:35 hmmmm let's say a node back from that one 05:35 hmmmm it does the radius check and the biome border is still in there 05:36 hmmmm so then it'd add another layer of the same transition 05:36 hmmmm aha 05:36 hmmmm there is potential for interesting stuff here 05:36 VanessaE ah good 05:36 hmmmm great idea 8) 05:36 VanessaE and if your check radius is less than the wavelength of your initial sinuses, you can be reasonably sure the biome edges won't cut sharply back and forth 05:37 hmmmm of course 05:37 VanessaE well, half the wavelength anyway 05:38 VanessaE for the height transitions, double-declining average is what a friend called a piece of interpolating code I wrote years ago. 05:38 VanessaE as I recall, the formula was simple, something like {average of last two samples} - average/2 05:39 VanessaE or something ridiculously simple like that 05:39 VanessaE it took the sample data from horrible square waves to something that looked like it was passed through a small capacitor. 05:39 VanessaE if you could extend that idea to two dimensions and average in all four directions, it might give you what you want 05:40 hmmmm perhaps 05:40 hmmmm alright, i am going to try that out 05:40 VanessaE ok 05:42 VanessaE hope it works :D 05:44 VanessaE for the 'hollowed out' areas, how about simple white noise compared against some value to "thin it out", then chop the result to one-bit wide, use that as a collection of one-node seeds, then hollow out around those seeds randomly? 05:45 VanessaE maybe draw "paths" between nearby seeds and hollow out along those paths 05:46 hmmmm actually that sounds like a bastardized version of 3d perlin noise 05:46 VanessaE heh 05:46 VanessaE I didn't think of it like that 05:46 hmmmm perlin noise is, after all, just points of white noise that's been interpolated 05:46 VanessaE huh. good point 05:47 VanessaE well in that case...MOAR PERLIN! 05:47 VanessaE :-) 05:47 hmmmm well 05:47 hmmmm i'll do that later 05:47 VanessaE right 05:51 hmmmm i'll first try out a radius of 3 with square borders 05:51 hmmmm square, because it won't really matter when the height reduction of all the neighboring nodes overlap 05:51 VanessaE ok 05:53 hmmmm so 20 checks 05:55 hmmmm actually 24 05:55 hmmmm + 20 05:55 VanessaE radius 3 = 3 from the center, so that's an edge length of 5 05:55 VanessaE so 4*4+4 = 20. 05:56 hmmmm i guess it depends on if i want to modify the height of all the nodes in the radius 05:56 VanessaE right 05:56 hmmmm do i? 05:56 hmmmm hrm 05:56 hmmmm well i'm either going to be writing to or reading from a mixture of a total of 24 + 20 nodes 05:56 VanessaE well in my routine every byte passing through the filter did get modified, but that was unavoidable in that routine 05:56 hmmmm so i'd rather have them all reading 05:56 VanessaE in a 2d filter, probably nw 05:56 VanessaE not* 05:58 hmmmm this is probably going to kill performance :/ 05:58 VanessaE hrm 05:58 VanessaE Nakedfury pointed out a possibility some days ago 05:59 hmmmm if i add more radius 05:59 VanessaE what was it.. 05:59 hmmmm to make it 4 05:59 hmmmm that'd be 32 + 24 + 20 05:59 VanessaE you know it occurs to me.. 06:00 VanessaE that double-declining average....if you extend it to 2d you get something not unlike Floyd Steinberg error distribution 06:01 hmmmm it's almost exactly like it actually 06:01 hmmmm except the diffusion coefficients are all 1/2 06:02 VanessaE right 06:03 VanessaE that's assuming I'm remembering my old formula 06:03 VanessaE correctly. 06:03 hmmmm i looked up double declining average, it's widely used in accounting for deprecation 06:03 hmmmm depreciation rather 06:03 VanessaE hah! I never knew 06:03 VanessaE this was something like 17 years ago that I wrote that bit of code, I'm surprised I remember it at all :-) 06:05 hmmmm you know 06:05 VanessaE yes? 06:06 hmmmm i can get away with the just a radius 06:06 hmmmm not all the blocks in between 06:06 VanessaE right 06:06 VanessaE that's what I was suggesting 06:06 hmmmm what i do is check if there's the biome difference, if so, make the pixel opposite to the one(s) that are outside of the biome lower 06:06 VanessaE because you're gonna re-check those blocks anyway 06:06 hmmmm but 06:06 hmmmm by how much... 06:07 hmmmm nevermind 06:08 hmmmm that way i'd be able to cover all the pixels without extraneous checking, but the problem would then be that i don't know by how much to lower them 06:08 hmmmm there must be some better way to go about this all 06:08 hmmmm i'll keep thinking 06:14 VanessaE well you could plan out an internal map of say 2-3x your check radius, which would keep a record of nearby blocks that have already been tested 06:14 VanessaE that would be pretty quick to consult 06:40 VanessaE bbl, time to sleep. 15:12 dysoco Hello, I'd like to provide a Spanish translation for Minetest, I just need to create a po/es/minetest.po file and complete it, or do I need to "plug" anything into the core? 16:37 VanessaE hmmmm: still getting server lockups, but now they're just the random ones as before and not the "won't load the map after a cold start" like I was getting yesterday. 16:37 VanessaE so the abridged settings helped but did not solve the problem. 16:38 hmmmm hrm' 16:39 hmmmm has anyone else reported these lockups? 16:39 hmmmm if not, what makes your configuration unique? 16:39 VanessaE Oldcoder had the same problem on his Server Nexus as he calls it, and I have seen it at least one other place as well (don't remember where) 16:39 thexyz i had similar lockup once when i was using settings from this topic http://minetest.net/forum/viewtopic.php?id=1825 16:40 thexyz server just was at 100% cpu right after startup 16:40 VanessaE Some of the slowness could of course be ordinary mods lagging it, but 100% CPU for several minutes on end? not likely to be a mod then. 16:41 hmmmm man 16:41 hmmmm this is whack 16:41 VanessaE what> 16:41 VanessaE ? 16:41 hmmmm basically the only way i'm going to be able to see what the problem is, is to run a server of my own just like you guys 16:42 thexyz or ask VanessaE for ssh access 16:42 VanessaE hmmmm: world and game files are available on my forum thread 16:42 hmmmm yeah or that 16:42 VanessaE (or yes, I could also enable ssh if you wanna log in and poke around) 16:42 hmmmm hmmmm 16:44 hmmmm well the first thing i would do would be to compile a debug version and run it in gdb 16:44 hmmmm but that means performance will be just.... nothing 16:44 hmmmm what kind of cpu does this server have 16:45 hmmmm for me it runs pretty okay with the debug settings (but i have the best of the newest stuff) 16:45 VanessaE AMD Phenom II X6 1055T at 2.6 GHz, 4GB RAM, 60 GB OCZ Vertex2 SSD, HD6870 video, 30M/3Mbps cablemodem connection 16:46 hmmmm it might be okay 17:00 thexyz /* 17:00 thexyz Insert to container 17:00 thexyz */ 17:00 thexyz m_sectors[p2d] = sector; 17:00 thexyz should pointless comments like this ^ one be kept? 17:03 celeron55 >but that means performance will be just.... nothing 17:03 celeron55 a debug build in gdb eats performance roughly a zero amount 17:04 celeron55 it's the difference between -O3 of the release and -O1 of the debug build (or what is it even? -O2?) (you can make an -O3 debug build if you want, but then gdb has less access to variable contents and such) 17:04 celeron55 anyway, no practical difference 17:25 hmmmm yeah, that's what i was referring to 17:25 hmmmm minetest with -O1 on my old machine just wouldn't run 17:25 hmmmm maybe at like 5fps if i was lucky 19:52 hmmmm okay 19:53 hmmmm it appears that the problem is that the emergethread keeps getting stuck on either envlock or conlock (more ofter the former) 19:53 hmmmm while ServerThread is stuck in peerItemCount() 19:53 hmmmm core::list::find is taking a ridiculous amount of time for some reason 19:54 hmmmm basically the problem is: 19:54 hmmmm emergethread can't do anything if sendblocks is being ridiculous 19:54 hmmmm which it is 19:56 hmmmm core::map::find i meant 19:58 thexyz i'm working at replacing core:: containers with stl ones atm, that should fix this issue, i guess 19:58 hmmmm well wait we don't know the actual reason why it's getting stuck 19:59 hmmmm it might not be irrlicht's fault 19:59 thexyz oh 20:00 hmmmm this is what i was looking at http://pastebin.com/gBfbmyzt 20:29 hmmmm okay, i found the problem 20:30 VanessaE \o/ 20:30 hmmmm the number of iterations peerItemCount does climbs up into the stratosphere while being called a lot 20:30 hmmmm and obviously this takes lots of cpu time 20:30 hmmmm all while EmergeThread is being stuck on the envlock or conlock 20:33 hmmmm the root of the problem is that BlockEmergeQueue fills up too fast 20:33 hmmmm and never gets processed 20:34 hmmmm BlockEmergeQueue gets its items removed when EmergeThread pops from the queue, but it can never do that, because it's stuck' 20:34 hmmmm on the envlock 20:35 hmmmm so there are two solutions here: 20:35 hmmmm 1). REALLY limit the scope of envlock and conlock in SendBlocks 20:36 hmmmm 2). simply drop blocks that would've been added to BlockEmergeQueue 20:36 hmmmm if it's above a certain amount (let's say 500 is a good number; when it gets stuck and the block requests keep piling up, it grows to about 50000) 20:37 hmmmm now #1 is tough, of course 20:37 hmmmm #2 would cause clients to never get some blocks 20:38 hmmmm so what we can do to solve this issue is still up in the air, open to suggestions 20:38 hmmmm it's a classic consumer starvation problem 21:04 doserj can you keep track of the peerItemCount seperately, so you don't have to compute it? 21:09 hmmmm yes 21:10 hmmmm i think the way it currently works is pretty shoddy anyway 21:13 doserj also, peerItemCount seems to be only used to be compared with max_emerge 21:13 doserj so you don't have to iterate through all of it 21:13 doserj (if you are lucky) 21:14 doserj But that point is moot if you get rid of the need to compute it in the first place 21:16 hmmmm yeah, we'll store the queued blocks with the peer.. that way we'll only have to use a list of u8s instead of u16 + u8s, so we'll be saving some memory too 21:16 hmmmm and we can just compare to list.size() 21:17 hmmmm which would fix peerItemCount()'s tendancy to get slower as the number of requests increases, thus preventing EmergeThread from consuming enough of the queue 21:17 hmmmm but that's not solving the problem at its core 21:18 hmmmm i think the way to solve the core problem is to simply not add the block to the queue in the first place if the count is too high 21:18 hmmmm so we'd put the check in addBlock 21:19 doserj drop it completely? 21:23 doserj and std::list::size() can still be linear (core::list::size() seems to be costant, though) 21:34 hmmmm that's the thing... does the client create a dummy block while it waits for the block content from the server? 21:34 hmmmm maybe we can have it destroy the dummy block after a certain amount of time so it can try again 21:35 hmmmm or what's setBlockNotSent? 21:35 hmmmm list::size() being linear isn't really a problem because we'd be solving the actual problem by not adding it to the queue 21:36 hmmmm most sane list implementations would have a constant timed size() though 22:57 RealBadAngel hi all 22:58 RealBadAngel hmmmm, are you familiar with matrix4 manipulations? 22:59 RealBadAngel i got stuck because 1.7.x irrlicht doesnt have functions required by me, 1.8 does 23:24 hmmmm what seems to be the problem 23:58 RealBadAngel hmmmm, meanwhile i just rewrote needed functions 23:58 RealBadAngel :) 23:58 RealBadAngel irrlicht 1.7 is missing set rotation matrix from axis and angle 23:59 RealBadAngel and to transpose rotation matrix to vector 23:59 RealBadAngel tried to use quaternions but i dont know nothin bout them 23:59 hmmmm you are using quaternions