Minetest logo

IRC log for #minetest-dev, 2016-06-12

| Channels | #minetest-dev index | Today | | Google Search | Plaintext

All times shown according to UTC.

Time Nick Message
00:04 Taoki joined #minetest-dev
00:09 Miner_48er joined #minetest-dev
00:35 Void7 joined #minetest-dev
00:40 STHGOM joined #minetest-dev
00:55 Tmanyo joined #minetest-dev
01:00 paramat hmmmm perhaps we can provide a 3D biomemap for use in on-generated? easy to create this in generateBiomes()
01:02 paramat not necessarily a replacement for the 2D biomemap
01:25 Puka joined #minetest-dev
02:28 Zeitgeist_ joined #minetest-dev
02:34 Tmanyo joined #minetest-dev
02:57 Miner_48er joined #minetest-dev
02:58 Miner_48er joined #minetest-dev
03:01 Miner_48er joined #minetest-dev
03:18 paramat joined #minetest-dev
03:29 hmmmm 3d biomemap
03:29 hmmmm that's really big you know
03:30 hmmmm the mapgen already uses an abnormally large amount of memory and noise in lua moreso
03:30 hmmmm i know we live in the era where 16gb of ram is the norm but plz
03:30 hmmmm what would the use case for a 3d biomemap be anyway
03:31 hmmmm i'm still trying to figure out what a sane interface to the biomegen from lua would look like
04:01 paramat yeah it's big, just a random suggestion
04:04 paramat but if you're planning to store 3D biome data that's also big
04:35 paramat left #minetest-dev
05:55 Hunterz joined #minetest-dev
06:18 Hijiri joined #minetest-dev
06:18 Hijiri joined #minetest-dev
06:32 jin_xi joined #minetest-dev
06:40 lisac joined #minetest-dev
06:49 Zeno` joined #minetest-dev
07:38 edgrey joined #minetest-dev
07:43 Krock joined #minetest-dev
08:02 edgrey joined #minetest-dev
08:12 Gael-de-Sailly joined #minetest-dev
08:50 davisonio joined #minetest-dev
09:05 edgrey joined #minetest-dev
09:31 Thomas-S joined #minetest-dev
10:41 Megal_ joined #minetest-dev
10:52 Fixer joined #minetest-dev
11:00 Ambistic joined #minetest-dev
11:00 Ambistic left #minetest-dev
11:01 Ambistic joined #minetest-dev
11:11 Puka joined #minetest-dev
11:20 davisonio joined #minetest-dev
11:26 damiel_ joined #minetest-dev
12:07 davisonio joined #minetest-dev
12:36 nrzkt joined #minetest-dev
13:00 davisonio joined #minetest-dev
13:16 exoplanet joined #minetest-dev
14:04 hmmmm joined #minetest-dev
14:12 Void7 joined #minetest-dev
14:27 rubenwardy joined #minetest-dev
14:58 lisac joined #minetest-dev
15:14 asl97 joined #minetest-dev
15:17 asl97 I got a quick stupid question, would it be consider a problem if minetest created a 2GB map file (on windows only) when creating a world?
15:28 asl97 i am gonna assume yes
15:33 Krock asl97, right after it was initialized?
15:34 asl97 from the code, it should be doing that afaik
15:34 Krock a 2GB map?
15:34 asl97 it > the lmdb backend
15:34 Krock ah well then, no idea
15:34 asl97 a 2GB map
15:35 Krock with sqlite and leveldb it doesn't allocade so much space
15:35 asl97 i am guessing it probably ain't gonna get merge if it does that
15:35 Krock s/ad/at/
15:36 asl97 the 2GB is the default size that is in the pull request
15:36 asl97 the default size in the lmdb code is just 10mb afaik
15:36 Krock 2 GB is way too much
15:37 asl97 https://github.com/minetest/minetest/pull/4206/​files#diff-8dbe0b201a90e7f52c7665af4b1bcf26R52
15:37 asl97 you can leave your comment there
15:40 Krock thanks. Commented
15:58 paramat joined #minetest-dev
15:59 xunto joined #minetest-dev
16:00 hmmmm heh!  i suppose this is the way you can tell that an open source project has reached maturity
16:00 hmmmm you keep getting pull requests to add <new DB #541>
16:00 ShadowBot https://github.com/minetest/minetest/issues/541 -- Add minetest.register_on_craft(player, craftitem)
16:01 Krock somewhen there are too many backends
16:02 asl97 sqlite is slow, leveldb get corrupted
16:02 hmmmm yeah there are configurable settings for these things though
16:02 hmmmm you know that right
16:02 asl97 redis is limited by ram and i am sure most consider it hard to setup
16:05 asl97 having more backend seem to be the answer that people can come up with xD
16:15 KaadmY joined #minetest-dev
16:16 damiel_ joined #minetest-dev
16:30 Krock leveldb_corrupt_factor = 20
16:32 xunto joined #minetest-dev
16:35 asl97 the minetest wiki call leveldb as reliable,  more reliable than sqlite3 from how I read it,  maybe that should be change.
16:35 asl97 there is also the part where it says it can use more than 4 GB of space which doesn't seem to make much sense.
16:47 Void7 joined #minetest-dev
16:52 Miner_48er joined #minetest-dev
16:53 Calinou <Krock> 2 GB is way too much
16:53 Calinou on Android it is
16:53 Calinou on other systems, it's hardly an issue
16:53 Calinou with drives usually having at least 500 GB
16:53 Krock still, 2 GB for a raw new world?
16:54 Krock creating some of those and you've already at 10 GB
16:54 asl97 Calinou: afaik, android has sparse file
16:54 Krock 10 GB for actually almost no world data
16:54 asl97 same with linux and other
16:55 davisonio joined #minetest-dev
16:55 nrzkt pgsql full generated world : 12GB
16:57 OldCoder What would a native MT DB format be like?
16:57 OldCoder One optimized for the game?
16:57 OldCoder s/format/system/
16:57 OldCoder ^ But 1000% exportable # requirement
16:58 Krock one which can be extended with new features, if required
16:58 OldCoder ^ Simple; there is a permanent header which indicates release level and/or bit flags for features
16:59 Krock yeah
16:59 OldCoder ^ Forward and backwards compatible
17:00 OldCoder Fault tolerant; low corruption
17:00 OldCoder ^ Desirable
17:00 OldCoder Sensible, fast, efficient compression based on knowledge of the types of data
17:01 OldCoder 3D optimized for loading
17:01 Krock good memory management
17:01 OldCoder Very low storage for air or vacuum blocks
17:02 OldCoder ^ This should be queued and discussed further
17:03 OldCoder Friendly to editing tools
17:03 OldCoder and mapping tools
17:07 est31 joined #minetest-dev
17:07 est31 OldCoder What would a native MT DB format be like?
17:07 est31 I dont think this is a good idea
17:07 est31 we shouldn't do the job of databases
17:08 est31 db's are a solved problem, no need to re-do it
17:08 est31 we will just do it worse, hardly there is a chance we will do better
17:09 OldCoder est31, Krock and I disagree. Let's see where consensus goes. We've all experienced problems with different types of DBs.
17:09 OldCoder Native might be an option to address them.
17:09 OldCoder No rush to settle it
17:09 est31 what problems
17:09 OldCoder Just an interesting thought at this point
17:09 nrzkt OldCoder, except for clients, for dedicated postgresql has no issue.
17:09 * OldCoder will review PostgreSQL then
17:09 OldCoder What problems are there with clients?
17:09 OldCoder For PostgreSQL
17:10 est31 OldCoder Very low storage for air or vacuum blocks <------ I think a mapblock full of air:air takes about 82 bytes
17:10 nrzkt clients cannot use postgresql, like redis, because it's a dedicated service
17:10 nrzkt est31, yes
17:10 OldCoder est31, not if it is impure
17:10 * OldCoder has written a world compression tool... he may use knowledge gained there to write of native options
17:10 OldCoder We got 12GB worlds that could be stored as 1GB
17:11 OldCoder ^ One huge plus of a native format!
17:11 asl97 OldCoder: you don't mean the database do you... you mean something like the ordering of the x,y,z?  that kind of DB format right?
17:11 OldCoder And speed improvements go along with that
17:11 OldCoder asl97, reviewing
17:11 nrzkt using lz4 for compression permit to fix some performance bottleneck when saving/loading nodes
17:11 est31 I don't think a native format will improve anything in this direction
17:11 nrzkt gzip is slow and consume some CPU. lz4 will fix this problem
17:12 OldCoder asl97, no, I mean the fundamental format. But not arguing for it today; just considering. And nrzkt what I have in mind will beat that 20 to 1.
17:12 * OldCoder has actually written this already in a sense *and* tested it
17:12 est31 OldCoder, you can get 99% of speedups by changing the mapblock serialisation format
17:12 est31 no need for completely native format
17:12 OldCoder est31, I'll need to study this. Thank you.
17:12 OldCoder But
17:12 Krock btw, you can use the backend "dummy" to not waste any disk space </shitcomment>
17:13 * OldCoder *has* experimented with compression that has ranged up to 80 to 1
17:13 est31 Well in the optimal case we had a pixel perfect mapgen
17:13 est31 and just stored the diff to the mapgen
17:13 OldCoder Ha
17:13 OldCoder How well did that work out?
17:13 asl97 Krock: the dummy backend is good and is being use on one public server
17:13 est31 but good luck convincing hmmmm and paramat to make the mapgen pixel perfect
17:13 Krock then one does some param changes and everything messes up
17:13 Krock asl97, yeah. for testing purposes
17:14 hmmmm that is not possible unless you can save a copy of every single mod that ever touches the mapgen
17:14 asl97 Krock: not, it's being use for the ctf server
17:14 OldCoder Fractal mapgens could produce very very small maps
17:14 OldCoder As in 6 bytes :-)
17:14 OldCoder Hm
17:14 Krock asl97, that's a very small area, so keeping it in RAM is the best option there
17:14 asl97 it's a great backend
17:15 Krock OldCoder, yeah, when you set the scale 1:30000
17:16 * OldCoder is intrigued by the idea of different approaches; will study this further
17:16 Krock yet another study research
17:16 OldCoder Map size *can* be reduced dramatically
17:16 OldCoder And Krock your point is not clear
17:17 * OldCoder does not know what "yet another study research" is referring to
17:17 OldCoder Allow lossy storage, a JPEG for worlds, and the results are amazing
17:17 Krock OldCoder, with this I mean there are sometimes studies that contradict another stufy
17:17 Krock *study
17:17 asl97 lmdb is a great database but i don't think it suit minetest because of the way it work, lmdb is meant mostly for read workload
17:17 est31 hmmmm, the alternative is to not store anything except some kind of hash that changes everytime a mapgen mod's behaviour changes
17:18 Krock lol. jpeg for world
17:18 OldCoder Krock, that is a pretty general statement. asl97 I have read-only worlds.
17:18 OldCoder Krock, I have done it
17:18 est31 In fact worlds are three dimensional
17:18 * OldCoder has gotten 12GB worlds down to about 250MB. That level is pretty lossy.
17:18 est31 so rather use videos :)
17:18 OldCoder s/JPEG/MJPEG/ then
17:19 est31 hmmmm, and then make the engine decline to start if the hash of the mods doesnt match the hash from the db
17:19 Krock compress - eh- enlarge it with the AVI format
17:19 est31 hmmmm, and also add a migration tool to migrate from older mods to newer mods
17:19 hmmmm are you willing to work on that?
17:19 est31 no :)
17:20 est31 Krock, OldCoder, I've done an experiment with lossless vp9 I think
17:20 Krock I think the modders should renew the mods themselves when they're broken
17:20 Krock est31, how lovely. How many frames per second did that movie have? :P
17:20 Krock how about watching it with 3d-glasses?
17:20 est31 no movie
17:20 est31 just trying to compress mapblocks via vp9
17:21 est31 thats it
17:21 Krock no, seriously. How big was the size difference?
17:21 est31 and decompress
17:21 est31 idk, dont remembe
17:21 est31 r
17:21 Krock too bad.. would be amazing to make a chart with the different compression methods
17:21 asl97 OldCoder: then you might be glad to know that lmdb is faster than leveldb and it doesn't corrupt easily
17:21 asl97 i quote: LMDB was designed from the start to resist data loss in the face of system and application crashes.
17:22 asl97 it doesn't use compression though so it would probably be as big as sqlite world but that means low cpu usage
17:23 est31 Krock, https://github.com/est31/minetest/commit/​97c030f1fbe1d4d64c638946e21713764b71aeca if you want to try yourself
17:23 est31 but bear in mind that vp9 was designed for video streaming
17:23 est31 meaning that people encode a video once, then store it on disk
17:23 Krock est31, should I use the # commmented line instead of "~/src/libvpx"?
17:23 est31 after that it doesnt get changed
17:24 est31 Krock, I think so
17:24 Krock well, it's made to encode and decode the stuff
17:25 est31 yeah and lossless
17:25 est31 the problem is speed however
17:26 est31 but its really experimental
17:27 est31 e.g. it doesnt use it yet for storing it to the map
17:27 nrzkt asl97, if lmdb doesn't use compression world will go to hundreds of gigabytes !
17:27 est31 but just makes some benchmarks etc
17:27 Krock ah I see
17:27 nrzkt est31, why not use lz4 ? it's very light and comrpss very well
17:27 Krock or just use LZMA?
17:27 nrzkt i'm using it into my pgsql backend it's a very good performance backend
17:27 nrzkt LZMA is slower than lz4
17:27 asl97 nrzkt: sqlite doesn't use compression ether afaik
17:28 Krock but it could /maybe/ compress it batter
17:28 est31 nrzkt, which compression do you mean
17:28 est31 mapblock compression?
17:28 Krock aren't we using zlib to compress the map data a bit?
17:28 est31 or compressing compressed already
17:28 est31 Krock, correct
17:28 est31 each mapblock gets compressed by zlib
17:28 Krock ah. then my memory isn't lost yet
17:28 est31 but as nrzkt started talking about postgres I started wondering
17:29 est31 what does mapblock compression have to do with the db backend choice
17:29 asl97 est31: leveldb has compression at the database level, i was comparing that to lmdb (and sqlite)
17:30 asl97 iirc, leveldb map are smaller than sqlite map
17:33 davisonio joined #minetest-dev
17:33 paramat http://i.imgur.com/mTsbrlF.png making dungeons more customisable
17:36 hmmmm you know instead of making things more customizable i wonder if our time is better spent on making new stuff
17:36 hmmmm in any case i'm gonna create a new dungeon type that's based off of voronoi cells
17:38 Krock \o/ more dungeons
17:41 nrzkt est31, yes i talked about mapblock compression
17:50 hmmmm i have heard of some block games doing in-memory compression
17:51 hmmmm they do RLE on vertical columns on a per-mapblock basis and gets some huge wins with that setup
17:51 hmmmm (since the majority of the time, nodes are checked in vertical succession due to reasons)
18:05 Tmanyo joined #minetest-dev
18:06 Fixer paramat: nice, underground dungeon city...
18:20 Fixer paramat: or even crazy subgame idea, make underground dungeon city with loot, hunger and stuff, mobs inside, your goal is to reach the surface
18:24 Fixer paramat: or opposite, as in rpgs... your goal is to descent into dungeon and blablabla
18:35 edgrey joined #minetest-dev
18:53 paramat a second dungeon type would be good
19:17 hmmmm also you know what, since i've cancelled that mapgen settings simplification (there simply wasn't significant enough of an advantage) i think i'd like to unify mapgen settings
19:18 hmmmm also now that I think about it I sorta think the current way mapgen specific params work is dumb
19:19 hmmmm i'm gonna turn it into a union
19:19 hmmmm i mean this is literally what a union is for
19:19 est31 unions are not typesafe :(
19:20 est31 I mean the usual use case is struct { int which; union {a, b, c} stuff; }
19:20 est31 but yeah thats what unions are for
19:21 hmmmm don't get what you mean
19:21 est31 its not important
19:21 hmmmm how is that any different from the current way we use C++ polymorphic structures?
19:22 est31 not much
19:22 est31 just that c++ polymorphic structures only work with pointers
19:22 hmmmm yup
19:22 est31 while unions store the stuff directly
19:22 est31 and take more storage
19:22 hmmmm doing an alloc/free for this params structure is definitely not necessary
19:23 est31 because they use max of all sizeof()*s
19:23 est31 s/*/s/
19:23 est31 err
19:23 est31 s/*/'/
19:23 hmmmm yeah but the difference between the sizes isn't all that much
19:23 hmmmm i'd rather not allocate heap unless it's necessary
19:23 hmmmm I mean more heap
19:23 est31 its gonna be stored in the heap one way or another
19:23 est31 but unions mean less indirection
19:25 hmmmm so basically you don't like unions because it's not as size-efficient as possible
19:25 hmmmm but don't forget with the added indirection you're adding at least 8 bytes for the pointer and then however much the heap header is (probably 16 bytes?) so a total of 24 more
19:25 Fritigern "so basically you don't like unicorns "
19:26 Fritigern ;-)
19:26 hmmmm i say 16 because most libcs implement heaps with a flink and blink
19:50 davisonio joined #minetest-dev
19:52 Void7 joined #minetest-dev
20:03 edgrey joined #minetest-dev
20:03 Amaz joined #minetest-dev
20:12 rubenwardy joined #minetest-dev
20:29 Ambistic There is no mob in Minetest ?
20:29 KaadmY Ambistic: not in vanilla
20:30 KaadmY there's some mods you can use to add mobs though
20:40 AnotherBrick joined #minetest-dev
21:16 est31 nrzkt, btw is the 0.4.14 android apk uploaded already
21:17 davisonio joined #minetest-dev
21:19 paramat #4216 WIP
21:19 ShadowBot https://github.com/minetest/minetest/issues/4216 -- Dungeons: Generalise. Add new parameters by paramat
21:42 OldCoder est31, nrzkt looks like the build.gradle file expects Android API 23 but API is updated to 24...
21:42 OldCoder Would one change just the two associated settings in that file?
21:43 est31 no
21:43 est31 OldCoder, just download api v23
21:43 OldCoder Link?
21:43 est31 OldCoder, its part of the sdk
21:43 nrzkt OldCoder, not a problem
21:43 est31 there is an updater tool isndie the sdk
21:44 OldCoder That needs a GUI
21:44 est31 it has one
21:44 * OldCoder is working with a headless server
21:44 est31 ah
21:44 OldCoder No GUI; just CLI
21:44 est31 well there is a CLI variant of it
21:44 OldCoder So, do not update to API 24?
21:44 est31 in fact its the same command
21:44 OldCoder Hm? Go on
21:44 est31 OldCoder, google for it, i dont know myself
21:45 OldCoder Not following, but thanks. Bottom line is, this has been tested with API 23 and will not build presently with API 24.
21:46 nrzkt the API version is not a problem for us.
21:49 OldCoder nrzkt, produces error on attempt to build apk if it does not match
21:49 OldCoder gradle level error, that is
21:55 est31 OldCoder, can you paste the error somewhere
21:55 OldCoder est31, sure, once I nail it down... may be able to post a fix instead
21:55 * OldCoder is working on it
21:56 est31 OldCoder, http://stackoverflow.com/a/4682241
21:56 OldCoder R
21:56 est31 but I doubt that we should adopt the policy of updating each time a new sdk gets released
21:56 est31 its not how android building works
21:56 OldCoder Proceeding
21:58 OldCoder Huh. That gets you all of the APIs.
21:59 est31 there is a --filter subcommand
21:59 nrzkt OldCoder, it's just due to your NDK
21:59 est31 --filter android-26
21:59 OldCoder Thank both of you
21:59 est31 maan
21:59 OldCoder What does the -26 signify? Did you mean -23 ?
21:59 est31 I think I had a headless version on the wiki
21:59 est31 yeah
21:59 est31 but shadowninja removed it
22:00 est31 because "better"
22:00 OldCoder Heh
22:01 OldCoder Handy command
22:02 * OldCoder tries make -j128
22:03 * KaadmY stares at OldCoder
22:03 * KaadmY cannot belive this
22:03 OldCoder y not?
22:03 OldCoder It seems happy
22:03 KaadmY dare you to try make -j
22:03 OldCoder OK
22:03 KaadmY see ya when your machine reboots
22:03 est31 lol
22:03 OldCoder make -j hooray
22:04 OldCoder RAM usage is a little higher
22:04 KaadmY :\
22:04 OldCoder Ah, managed to peg all of the cores
22:04 KaadmY darnit
22:04 OldCoder No, it is not crashed
22:07 OldCoder Managed to use 100% of RAM. Into swap now.
22:08 KaadmY mah
22:08 KaadmY meh*
22:08 KaadmY make -j just crashes my laptop
22:08 KaadmY like immediately
22:11 OldCoder It's past the difficult part, low load now
22:11 OldCoder This is not a laptop :-)
22:11 OldCoder RAM is about 56GB counting swap
22:11 OldCoder > failed to find Build Tools revision 23.0.3
22:11 OldCoder Hm
22:21 hmmmm i'm not sure if this is really a minetest development topic
22:21 OldCoder It's a build topic, but fair enough
22:21 OldCoder Can discuss it in -project... but are compiles
22:21 OldCoder not core dev? Or is it the question of compiles and links for a specific platform?
22:22 OldCoder I.e. if you'd like Android builds to work easily for people... is that core dev?
22:22 hmmmm using a lot of cores to try compiling minetest is development of the core?
22:23 hmmmm different kind of cores I guess
22:23 * OldCoder thinks the build will work
22:24 OldCoder We need better Android knowledge
22:34 est31 idk
22:35 est31 mtgame stuff gets discussed here as well
22:35 est31 so why not discussing this too
22:35 est31 but getting minetest build is borderline offtopic
22:35 est31 its not about "what should we change in the git repo"
22:35 est31 but more about "what should i do after I've cloned it"
22:36 hmmmm once upon a time, mtdev used to be a super serious channel where people got banned often
22:37 * KaadmY is vewy serious
22:37 * KaadmY bans some peeps
22:40 OldCoder hmmmm, too serious for the good of the project. Seriously. But FYI I have now built MT for Android and will write notes for others.
22:40 * OldCoder can now contribute Android builds
22:40 OldCoder It isn't as easy as the wiki and Wayward_One's notes suggest
22:41 OldCoder One more question for now... what is an aligned vs. unaligned APK?
22:41 OldCoder People can answer in -project
23:14 Ambistic joined #minetest-dev
23:52 Puka joined #minetest-dev

| Channels | #minetest-dev index | Today | | Google Search | Plaintext