Time  Nick       Message
01:00 paramat    hmmmm perhaps we can provide a 3D biomemap for use in on-generated? easy to create this in generateBiomes()
01:02 paramat    not necessarily a replacement for the 2D biomemap
03:29 hmmmm      3d biomemap
03:29 hmmmm      that's really big you know
03:30 hmmmm      the mapgen already uses an abnormally large amount of memory and noise in lua moreso
03:30 hmmmm      i know we live in the era where 16gb of ram is the norm but plz
03:30 hmmmm      what would the use case for a 3d biomemap be anyway
03:31 hmmmm      i'm still trying to figure out what a sane interface to the biomegen from lua would look like
04:01 paramat    yeah it's big, just a random suggestion
04:04 paramat    but if you're planning to store 3D biome data that's also big
15:17 asl97      I got a quick stupid question, would it be consider a problem if minetest created a 2GB map file (on windows only) when creating a world?
15:28 asl97      i am gonna assume yes
15:33 Krock      asl97, right after it was initialized?
15:34 asl97      from the code, it should be doing that afaik
15:34 Krock      a 2GB map?
15:34 asl97      it > the lmdb backend
15:34 Krock      ah well then, no idea
15:34 asl97      a 2GB map
15:35 Krock      with sqlite and leveldb it doesn't allocade so much space
15:35 asl97      i am guessing it probably ain't gonna get merge if it does that
15:35 Krock      s/ad/at/
15:36 asl97      the 2GB is the default size that is in the pull request
15:36 asl97      the default size in the lmdb code is just 10mb afaik
15:36 Krock      2 GB is way too much
15:37 asl97      https://github.com/minetest/minetest/pull/4206/files#diff-8dbe0b201a90e7f52c7665af4b1bcf26R52
15:37 asl97      you can leave your comment there
15:40 Krock      thanks. Commented
16:00 hmmmm      heh!  i suppose this is the way you can tell that an open source project has reached maturity
16:00 hmmmm      you keep getting pull requests to add <new DB #541>
16:00 ShadowBot  https://github.com/minetest/minetest/issues/541 -- Add minetest.register_on_craft(player, craftitem)
16:01 Krock      somewhen there are too many backends
16:02 asl97      sqlite is slow, leveldb get corrupted
16:02 hmmmm      yeah there are configurable settings for these things though
16:02 hmmmm      you know that right
16:02 asl97      redis is limited by ram and i am sure most consider it hard to setup
16:05 asl97      having more backend seem to be the answer that people can come up with xD
16:30 Krock      leveldb_corrupt_factor = 20
16:35 asl97      the minetest wiki call leveldb as reliable,  more reliable than sqlite3 from how I read it,  maybe that should be change.
16:35 asl97      there is also the part where it says it can use more than 4 GB of space which doesn't seem to make much sense.
16:53 Calinou    <Krock> 2 GB is way too much
16:53 Calinou    on Android it is
16:53 Calinou    on other systems, it's hardly an issue
16:53 Calinou    with drives usually having at least 500 GB
16:53 Krock      still, 2 GB for a raw new world?
16:54 Krock      creating some of those and you've already at 10 GB
16:54 asl97      Calinou: afaik, android has sparse file
16:54 Krock      10 GB for actually almost no world data
16:54 asl97      same with linux and other
16:55 nrzkt      pgsql full generated world : 12GB
16:57 OldCoder   What would a native MT DB format be like?
16:57 OldCoder   One optimized for the game?
16:57 OldCoder   s/format/system/
16:57 OldCoder   ^ But 1000% exportable # requirement
16:58 Krock      one which can be extended with new features, if required
16:58 OldCoder   ^ Simple; there is a permanent header which indicates release level and/or bit flags for features
16:59 Krock      yeah
16:59 OldCoder   ^ Forward and backwards compatible
17:00 OldCoder   Fault tolerant; low corruption
17:00 OldCoder   ^ Desirable
17:00 OldCoder   Sensible, fast, efficient compression based on knowledge of the types of data
17:01 OldCoder   3D optimized for loading
17:01 Krock      good memory management
17:01 OldCoder   Very low storage for air or vacuum blocks
17:02 OldCoder   ^ This should be queued and discussed further
17:03 OldCoder   Friendly to editing tools
17:03 OldCoder   and mapping tools
17:07 est31      OldCoder 	What would a native MT DB format be like?
17:07 est31      I dont think this is a good idea
17:07 est31      we shouldn't do the job of databases
17:08 est31      db's are a solved problem, no need to re-do it
17:08 est31      we will just do it worse, hardly there is a chance we will do better
17:09 OldCoder   est31, Krock and I disagree. Let's see where consensus goes. We've all experienced problems with different types of DBs.
17:09 OldCoder   Native might be an option to address them.
17:09 OldCoder   No rush to settle it
17:09 est31      what problems
17:09 OldCoder   Just an interesting thought at this point
17:09 nrzkt      OldCoder, except for clients, for dedicated postgresql has no issue.
17:09 * OldCoder will review PostgreSQL then
17:09 OldCoder   What problems are there with clients?
17:09 OldCoder   For PostgreSQL
17:10 est31      OldCoder 	Very low storage for air or vacuum blocks <------ I think a mapblock full of air:air takes about 82 bytes
17:10 nrzkt      clients cannot use postgresql, like redis, because it's a dedicated service
17:10 nrzkt      est31, yes
17:10 OldCoder   est31, not if it is impure
17:10 * OldCoder has written a world compression tool... he may use knowledge gained there to write of native options
17:10 OldCoder   We got 12GB worlds that could be stored as 1GB
17:11 OldCoder   ^ One huge plus of a native format!
17:11 asl97      OldCoder: you don't mean the database do you... you mean something like the ordering of the x,y,z?  that kind of DB format right?
17:11 OldCoder   And speed improvements go along with that
17:11 OldCoder   asl97, reviewing
17:11 nrzkt      using lz4 for compression permit to fix some performance bottleneck when saving/loading nodes
17:11 est31      I don't think a native format will improve anything in this direction
17:11 nrzkt      gzip is slow and consume some CPU. lz4 will fix this problem
17:12 OldCoder   asl97, no, I mean the fundamental format. But not arguing for it today; just considering. And nrzkt what I have in mind will beat that 20 to 1.
17:12 * OldCoder has actually written this already in a sense *and* tested it
17:12 est31      OldCoder, you can get 99% of speedups by changing the mapblock serialisation format
17:12 est31      no need for completely native format
17:12 OldCoder   est31, I'll need to study this. Thank you.
17:12 OldCoder   But
17:12 Krock      btw, you can use the backend "dummy" to not waste any disk space </shitcomment>
17:13 * OldCoder *has* experimented with compression that has ranged up to 80 to 1
17:13 est31      Well in the optimal case we had a pixel perfect mapgen
17:13 est31      and just stored the diff to the mapgen
17:13 OldCoder   Ha
17:13 OldCoder   How well did that work out?
17:13 asl97      Krock: the dummy backend is good and is being use on one public server
17:13 est31      but good luck convincing hmmmm and paramat to make the mapgen pixel perfect
17:13 Krock      then one does some param changes and everything messes up
17:13 Krock      asl97, yeah. for testing purposes
17:14 hmmmm      that is not possible unless you can save a copy of every single mod that ever touches the mapgen
17:14 asl97      Krock: not, it's being use for the ctf server
17:14 OldCoder   Fractal mapgens could produce very very small maps
17:14 OldCoder   As in 6 bytes :-)
17:14 OldCoder   Hm
17:14 Krock      asl97, that's a very small area, so keeping it in RAM is the best option there
17:14 asl97      it's a great backend
17:15 Krock      OldCoder, yeah, when you set the scale 1:30000
17:16 * OldCoder is intrigued by the idea of different approaches; will study this further
17:16 Krock      yet another study research
17:16 OldCoder   Map size *can* be reduced dramatically
17:16 OldCoder   And Krock your point is not clear
17:17 * OldCoder does not know what "yet another study research" is referring to
17:17 OldCoder   Allow lossy storage, a JPEG for worlds, and the results are amazing
17:17 Krock      OldCoder, with this I mean there are sometimes studies that contradict another stufy
17:17 Krock      *study
17:17 asl97      lmdb is a great database but i don't think it suit minetest because of the way it work, lmdb is meant mostly for read workload
17:17 est31      hmmmm, the alternative is to not store anything except some kind of hash that changes everytime a mapgen mod's behaviour changes
17:18 Krock      lol. jpeg for world
17:18 OldCoder   Krock, that is a pretty general statement. asl97 I have read-only worlds.
17:18 OldCoder   Krock, I have done it
17:18 est31      In fact worlds are three dimensional
17:18 * OldCoder has gotten 12GB worlds down to about 250MB. That level is pretty lossy.
17:18 est31      so rather use videos :)
17:18 OldCoder   s/JPEG/MJPEG/ then
17:19 est31      hmmmm, and then make the engine decline to start if the hash of the mods doesnt match the hash from the db
17:19 Krock      compress - eh- enlarge it with the AVI format
17:19 est31      hmmmm, and also add a migration tool to migrate from older mods to newer mods
17:19 hmmmm      are you willing to work on that?
17:19 est31      no :)
17:20 est31      Krock, OldCoder, I've done an experiment with lossless vp9 I think
17:20 Krock      I think the modders should renew the mods themselves when they're broken
17:20 Krock      est31, how lovely. How many frames per second did that movie have? :P
17:20 Krock      how about watching it with 3d-glasses?
17:20 est31      no movie
17:20 est31      just trying to compress mapblocks via vp9
17:21 est31      thats it
17:21 Krock      no, seriously. How big was the size difference?
17:21 est31      and decompress
17:21 est31      idk, dont remembe
17:21 est31      r
17:21 Krock      too bad.. would be amazing to make a chart with the different compression methods
17:21 asl97      OldCoder: then you might be glad to know that lmdb is faster than leveldb and it doesn't corrupt easily
17:21 asl97      i quote: LMDB was designed from the start to resist data loss in the face of system and application crashes.
17:22 asl97      it doesn't use compression though so it would probably be as big as sqlite world but that means low cpu usage
17:23 est31      Krock, https://github.com/est31/minetest/commit/97c030f1fbe1d4d64c638946e21713764b71aeca if you want to try yourself
17:23 est31      but bear in mind that vp9 was designed for video streaming
17:23 est31      meaning that people encode a video once, then store it on disk
17:23 Krock      est31, should I use the # commmented line instead of "~/src/libvpx"?
17:23 est31      after that it doesnt get changed
17:24 est31      Krock, I think so
17:24 Krock      well, it's made to encode and decode the stuff
17:25 est31      yeah and lossless
17:25 est31      the problem is speed however
17:26 est31      but its really experimental
17:27 est31      e.g. it doesnt use it yet for storing it to the map
17:27 nrzkt      asl97, if lmdb doesn't use compression world will go to hundreds of gigabytes !
17:27 est31      but just makes some benchmarks etc
17:27 Krock      ah I see
17:27 nrzkt      est31, why not use lz4 ? it's very light and comrpss very well
17:27 Krock      or just use LZMA?
17:27 nrzkt      i'm using it into my pgsql backend it's a very good performance backend
17:27 nrzkt      LZMA is slower than lz4
17:27 asl97      nrzkt: sqlite doesn't use compression ether afaik
17:28 Krock      but it could /maybe/ compress it batter
17:28 est31      nrzkt, which compression do you mean
17:28 est31      mapblock compression?
17:28 Krock      aren't we using zlib to compress the map data a bit?
17:28 est31      or compressing compressed already
17:28 est31      Krock, correct
17:28 est31      each mapblock gets compressed by zlib
17:28 Krock      ah. then my memory isn't lost yet
17:28 est31      but as nrzkt started talking about postgres I started wondering
17:29 est31      what does mapblock compression have to do with the db backend choice
17:29 asl97      est31: leveldb has compression at the database level, i was comparing that to lmdb (and sqlite)
17:30 asl97      iirc, leveldb map are smaller than sqlite map
17:33 paramat    http://i.imgur.com/mTsbrlF.png making dungeons more customisable
17:36 hmmmm      you know instead of making things more customizable i wonder if our time is better spent on making new stuff
17:36 hmmmm      in any case i'm gonna create a new dungeon type that's based off of voronoi cells
17:38 Krock      \o/ more dungeons
17:41 nrzkt      est31, yes i talked about mapblock compression
17:50 hmmmm      i have heard of some block games doing in-memory compression
17:51 hmmmm      they do RLE on vertical columns on a per-mapblock basis and gets some huge wins with that setup
17:51 hmmmm      (since the majority of the time, nodes are checked in vertical succession due to reasons)
18:06 Fixer      paramat: nice, underground dungeon city...
18:20 Fixer      paramat: or even crazy subgame idea, make underground dungeon city with loot, hunger and stuff, mobs inside, your goal is to reach the surface
18:24 Fixer      paramat: or opposite, as in rpgs... your goal is to descent into dungeon and blablabla
18:53 paramat    a second dungeon type would be good
19:17 hmmmm      also you know what, since i've cancelled that mapgen settings simplification (there simply wasn't significant enough of an advantage) i think i'd like to unify mapgen settings
19:18 hmmmm      also now that I think about it I sorta think the current way mapgen specific params work is dumb
19:19 hmmmm      i'm gonna turn it into a union
19:19 hmmmm      i mean this is literally what a union is for
19:19 est31      unions are not typesafe :(
19:20 est31      I mean the usual use case is struct { int which; union {a, b, c} stuff; }
19:20 est31      but yeah thats what unions are for
19:21 hmmmm      don't get what you mean
19:21 est31      its not important
19:21 hmmmm      how is that any different from the current way we use C++ polymorphic structures?
19:22 est31      not much
19:22 est31      just that c++ polymorphic structures only work with pointers
19:22 hmmmm      yup
19:22 est31      while unions store the stuff directly
19:22 est31      and take more storage
19:22 hmmmm      doing an alloc/free for this params structure is definitely not necessary
19:23 est31      because they use max of all sizeof()*s
19:23 est31      s/*/s/
19:23 est31      err
19:23 est31      s/*/'/
19:23 hmmmm      yeah but the difference between the sizes isn't all that much
19:23 hmmmm      i'd rather not allocate heap unless it's necessary
19:23 hmmmm      I mean more heap
19:23 est31      its gonna be stored in the heap one way or another
19:23 est31      but unions mean less indirection
19:25 hmmmm      so basically you don't like unions because it's not as size-efficient as possible
19:25 hmmmm      but don't forget with the added indirection you're adding at least 8 bytes for the pointer and then however much the heap header is (probably 16 bytes?) so a total of 24 more
19:25 Fritigern  "so basically you don't like unicorns "
19:26 Fritigern  ;-)
19:26 hmmmm      i say 16 because most libcs implement heaps with a flink and blink
20:29 Ambistic   There is no mob in Minetest ?
20:29 KaadmY     Ambistic: not in vanilla
20:30 KaadmY     there's some mods you can use to add mobs though
21:16 est31      nrzkt, btw is the 0.4.14 android apk uploaded already
21:19 paramat    #4216 WIP
21:19 ShadowBot  https://github.com/minetest/minetest/issues/4216 -- Dungeons: Generalise. Add new parameters by paramat
21:42 OldCoder   est31, nrzkt looks like the build.gradle file expects Android API 23 but API is updated to 24...
21:42 OldCoder   Would one change just the two associated settings in that file?
21:43 est31      no
21:43 est31      OldCoder, just download api v23
21:43 OldCoder   Link?
21:43 est31      OldCoder, its part of the sdk
21:43 nrzkt      OldCoder, not a problem
21:43 est31      there is an updater tool isndie the sdk
21:44 OldCoder   That needs a GUI
21:44 est31      it has one
21:44 * OldCoder is working with a headless server
21:44 est31      ah
21:44 OldCoder   No GUI; just CLI
21:44 est31      well there is a CLI variant of it
21:44 OldCoder   So, do not update to API 24?
21:44 est31      in fact its the same command
21:44 OldCoder   Hm? Go on
21:44 est31      OldCoder, google for it, i dont know myself
21:45 OldCoder   Not following, but thanks. Bottom line is, this has been tested with API 23 and will not build presently with API 24.
21:46 nrzkt      the API version is not a problem for us.
21:49 OldCoder   nrzkt, produces error on attempt to build apk if it does not match
21:49 OldCoder   gradle level error, that is
21:55 est31      OldCoder, can you paste the error somewhere
21:55 OldCoder   est31, sure, once I nail it down... may be able to post a fix instead
21:55 * OldCoder is working on it
21:56 est31      OldCoder, http://stackoverflow.com/a/4682241
21:56 OldCoder   R
21:56 est31      but I doubt that we should adopt the policy of updating each time a new sdk gets released
21:56 est31      its not how android building works
21:56 OldCoder   Proceeding
21:58 OldCoder   Huh. That gets you all of the APIs.
21:59 est31      there is a --filter subcommand
21:59 nrzkt      OldCoder, it's just due to your NDK
21:59 est31      --filter android-26
21:59 OldCoder   Thank both of you
21:59 est31      maan
21:59 OldCoder   What does the -26 signify? Did you mean -23 ?
21:59 est31      I think I had a headless version on the wiki
21:59 est31      yeah
21:59 est31      but shadowninja removed it
22:00 est31      because "better"
22:00 OldCoder   Heh
22:01 OldCoder   Handy command
22:02 * OldCoder tries make -j128
22:03 * KaadmY   stares at OldCoder
22:03 * KaadmY   cannot belive this
22:03 OldCoder   y not?
22:03 OldCoder   It seems happy
22:03 KaadmY     dare you to try make -j
22:03 OldCoder   OK
22:03 KaadmY     see ya when your machine reboots
22:03 est31      lol
22:03 OldCoder   make -j hooray
22:04 OldCoder   RAM usage is a little higher
22:04 KaadmY     :\
22:04 OldCoder   Ah, managed to peg all of the cores
22:04 KaadmY     darnit
22:04 OldCoder   No, it is not crashed
22:07 OldCoder   Managed to use 100% of RAM. Into swap now.
22:08 KaadmY     mah
22:08 KaadmY     meh*
22:08 KaadmY     make -j just crashes my laptop
22:08 KaadmY     like immediately
22:11 OldCoder   It's past the difficult part, low load now
22:11 OldCoder   This is not a laptop :-)
22:11 OldCoder   RAM is about 56GB counting swap
22:11 OldCoder   > failed to find Build Tools revision 23.0.3
22:11 OldCoder   Hm
22:21 hmmmm      i'm not sure if this is really a minetest development topic
22:21 OldCoder   It's a build topic, but fair enough
22:21 OldCoder   Can discuss it in -project... but are compiles
22:21 OldCoder   not core dev? Or is it the question of compiles and links for a specific platform?
22:22 OldCoder   I.e. if you'd like Android builds to work easily for people... is that core dev?
22:22 hmmmm      using a lot of cores to try compiling minetest is development of the core?
22:23 hmmmm      different kind of cores I guess
22:23 * OldCoder thinks the build will work
22:24 OldCoder   We need better Android knowledge
22:34 est31      idk
22:35 est31      mtgame stuff gets discussed here as well
22:35 est31      so why not discussing this too
22:35 est31      but getting minetest build is borderline offtopic
22:35 est31      its not about "what should we change in the git repo"
22:35 est31      but more about "what should i do after I've cloned it"
22:36 hmmmm      once upon a time, mtdev used to be a super serious channel where people got banned often
22:37 * KaadmY   is vewy serious
22:37 * KaadmY   bans some peeps
22:40 OldCoder   hmmmm, too serious for the good of the project. Seriously. But FYI I have now built MT for Android and will write notes for others.
22:40 * OldCoder can now contribute Android builds
22:40 OldCoder   It isn't as easy as the wiki and Wayward_One's notes suggest
22:41 OldCoder   One more question for now... what is an aligned vs. unaligned APK?
22:41 OldCoder   People can answer in -project