Minetest logo

IRC log for #minetest, 2022-02-03

| Channels | #minetest index | Today | | Google Search | Plaintext

All times shown according to UTC.

Time Nick Message
00:09 v-rob joined #minetest
00:28 AliasAlreadyTake joined #minetest
01:51 v-rob joined #minetest
02:01 Jason232_ joined #minetest
02:10 delta23 joined #minetest
02:13 olliy joined #minetest
02:15 Verticen joined #minetest
02:54 Hawk777 joined #minetest
02:55 erlehmann assuming i have binary data, like the output of minetest.compress() and want to save that in node meta. is it safe?
02:55 erlehmann is it smart? probably not
02:55 erlehmann oh, i want to save it in item meta too hehe
02:58 erlehmann just minetest.serialize? i wonder
03:16 frostsnow joined #minetest
03:28 queria^clone joined #minetest
03:31 jingkaimori joined #minetest
03:32 jingkaimori Can this issue be closed? https://github.com/minetest/minetest/issues/3374
03:34 queria^clone joined #minetest
03:36 MTDiscord <Jonathon> how about explaining why?
03:36 MTDiscord <Jonathon> and env_meta.txt still randomly gets corrupted
03:38 jingkaimori now mod storage is migrate to database
03:40 MTDiscord <Jonathon> sure, but mods dont have db access, which is ultimately what is wanted by a lot of devs
03:43 MTDiscord <Jonathon> which sorta puts https://github.com/minetest/minetest/issues/10371 and it as duplicates
03:46 Hawk777 joined #minetest
03:46 MTDiscord <Jonathon> ban mods are the ones already starting to role there own solutions, see sban, verbanna. other mods such as some afcm? and others are starting to use them. key point of contention being that they require insequire env and external setup
03:47 v-rob joined #minetest
03:47 MTDiscord <Jonathon> thanks @rubenwardy
04:07 oliverwood joined #minetest
04:09 Jason232 joined #minetest
04:11 erlehmann joined #minetest
04:17 erlehmann joined #minetest
04:28 RenMinWanSui joined #minetest
05:00 MTDiscord joined #minetest
05:06 v-rob joined #minetest
05:32 Hawk777 joined #minetest
06:01 riff-IRC joined #minetest
06:12 mmmax joined #minetest
06:20 Jason232 joined #minetest
07:03 SwissalpS left #minetest
07:07 v-rob joined #minetest
07:09 grouinos joined #minetest
07:18 specing_ joined #minetest
07:26 Gustavo6046 joined #minetest
07:28 search_social joined #minetest
07:28 mazes_83 joined #minetest
07:28 Lunatrius joined #minetest
07:28 luk3yx joined #minetest
07:28 BuckarooBanzai joined #minetest
07:28 bwarden joined #minetest
07:28 erstazi joined #minetest
07:28 asdflkj_sh joined #minetest
07:28 Krock joined #minetest
07:28 dabbill joined #minetest
07:28 syl joined #minetest
07:28 kevinsan_ joined #minetest
07:28 Peppy joined #minetest
07:28 Norkle joined #minetest
07:28 lumidify joined #minetest
07:46 calcul0n_ joined #minetest
07:48 appguru joined #minetest
08:15 MinetestBot [git] bell07 -> minetest/minetest_game: player_api - fix compatibility code 584a6a2 https://github.com/minetest/minetest_game/commit/584a6a21029dcac694efdf12b2a94ece06186f85 (2022-02-03T08:10:53Z)
08:15 MinetestBot [git] appgurueu -> minetest/minetest_game: player_api - fix compatibility code a840ac0 https://github.com/minetest/minetest_game/commit/a840ac0a4d3db935b20ce6c6259795fbe263fefe (2022-02-03T08:15:14Z)
08:17 sfan5 twice the same commit?
08:18 sfan5 ah, the second is a merge commit
08:18 sfan5 how did that happen
08:31 Boingo joined #minetest
09:02 Peppy joined #minetest
09:08 Jason232_ joined #minetest
09:22 mazes_80 joined #minetest
10:04 grouinos joined #minetest
10:14 Fixer joined #minetest
10:36 HuguesRoss joined #minetest
10:45 MinetestBot [git] Desour -> minetest/minetest: Add more documentation for the list[] fs element (#11979) d387e9b https://github.com/minetest/minetest/commit/d387e9b6d39d9689b79da21fa263c93d2b26e840 (2022-02-03T10:43:28Z)
10:45 MinetestBot [git] TurkeyMcMac -> minetest/minetest: Clean up ClientInterface locking 1c73902 https://github.com/minetest/minetest/commit/1c73902005bb5c7a40be5571bff9c232d8c69536 (2022-02-03T10:43:51Z)
10:46 mazes_80 about log: why complaining about depends.txt or description.txt when mod.conf is ab
10:46 mazes_80 about log: why complaining about depends.txt or description.txt when mod.conf is there ?
10:48 mazes_80 i think the case warning should be emitted is when no mod.conf, aren't previous files ignored when mod.conf exists
10:54 GNUHacker joined #minetest
11:06 Jason232 joined #minetest
11:14 mazes_80 leveldb --recompress: for one of my local games I use leveldb. Size was 1.6G, I tried to recompress -> 2.6G
11:14 mazes_80 after converting to sqlite3 and back to leveldb -> 1.4G
11:31 Markow joined #minetest
11:47 olliy joined #minetest
11:53 SOMBRIO joined #minetest
12:14 proller joined #minetest
12:21 definitelya joined #minetest
12:22 tech_exorcist joined #minetest
13:15 olliy1or joined #minetest
13:17 olliy1or joined #minetest
13:20 appguru joined #minetest
14:14 olliy joined #minetest
14:45 kamdard_ joined #minetest
15:49 oliverwood joined #minetest
15:56 Gustavo6046 the stone axe in MineClone 5 is a bit overpowered, either that or the monsters are a bit too low health
15:56 Gustavo6046 I don't mean this as in "it should be more accurate to Minecraft!!111!1one", I mean more like, balance
15:57 Gustavo6046 Not to imply that minecraft is anyhow any sort of benchmark of balance to begin with lol
15:58 Gustavo6046 but two good critical hits with a stone axe seems a bit too easy, doesn't even give time for a skeleton to shoot back at you, which would punish you for meleeing him so early game.
15:59 grouinos joined #minetest
16:08 ShadowNinja joined #minetest
16:12 ShadowBot joined #minetest
16:23 JimBer110 joined #minetest
16:35 Fixer joined #minetest
16:37 jluc joined #minetest
16:50 erlehmann Gustavo6046 axe has high damage, but is slow. try fighting an ice bear with it.
16:59 Gustavo6046 yeah, that's the case in minecraft too, but I think it feels a bit too high damage on mineclone
17:13 erlehmann Gustavo6046 i doubt mineclone2 development is going anywhere tbh, but you can file a bug on mineclonia if you have evidence (e.g. youtube videos, wiki entries) that the damage should be lower for balance reasons
17:13 v-rob joined #minetest
17:13 Gustavo6046 ah
17:13 erlehmann and mineclone5 will take literally any patch that does not crash the game (and sometimes even ones that do), so you can try there as well
17:13 Gustavo6046 dang
17:14 Gustavo6046 that's two extremes
17:14 erlehmann there is a reason we are not all developing the same project ^^
17:14 tech_exorcist joined #minetest
17:15 GNUHacker left #minetest
17:31 definitelya r.i.p Mineclone 2 - supplanted by Memeclone
17:31 MTDiscord <Sublayer plank> hahaha, memeclone is funny
17:31 definitelya xD
17:31 definitelya It's bad...
17:37 Taoki joined #minetest
17:39 erlehmann memeclone has gender fluid
17:39 erlehmann it is left by genderman
17:39 erlehmann hehehehe
17:39 erlehmann i hope fleckenstein develops it further
17:39 erlehmann it is just too silly
17:41 definitelya Scariest man after Slenderpants.
17:42 definitelya But yeah that makes two I guess. All for it.
17:55 erlehmann appguru do you have an example of compressed data being serialized/deserialized for item meta?
17:55 erlehmann output of minetest.compress() i mean
17:55 appguru I don't
17:55 erlehmann then i use base64 instead i guess
17:55 MTDiscord <Jonathon> never use minetest.deserialize with big data
17:56 erlehmann this is like 10k maximum data
17:56 appguru is it for ender chests?
17:57 erlehmann no, it is for pixel arrays in maps. i figured that if i save the pixel data in the item, it works in world downloads
17:57 erlehmann and also you can re-encode it later
17:57 appguru I'm still convinced that the best way to store large item-related data would be files or a proper database
17:57 appguru unfortunately there are no proper callbacks for when an item goes missing
17:57 erlehmann oh, they are stored in files server-side
17:57 appguru and when are they dropped?
17:57 erlehmann never lol
17:58 erlehmann so i want the files to be disposable
17:58 erlehmann i.e. when the file does not exist, code encodes the pixel to a texture and sends dynamic media
17:58 erlehmann seriously, i know this makes the item heavier, but i see no other way to make world dls work
17:58 erlehmann also this would be a good test case for the anon5 proxy thing
17:59 erlehmann cora told me that anon5 proxy is just cleverer about minetest protocol than minetest itself hehe
17:59 erlehmann i feared it might break map dls
17:59 Talkless joined #minetest
18:01 MTDiscord <Warr1024> If you use zlib compression and then base64 on plaintext, it can still often come out more compact than the original text, even if it's just an ascii->ascii transformation in the end
18:01 MTDiscord <Warr1024> What is the "anon5 proxy thing"?
18:02 erlehmann https://github.com/OysterityAnarchy/mt-netopt-proxy
18:03 erlehmann i do not understand the code, but i was told it makes for much less inventory traffic
18:06 erlehmann i have a question: when exactly is mod security blocking my file writes and when not?
18:06 erlehmann i have one mod calling a function from another mod writing files and that is not blocked
18:06 erlehmann but when i do it in the first mod itself, mod security blocks it
18:06 erlehmann in 5.4
18:06 appguru mods can write to their own folders
18:06 appguru to do so, they must obtain the handles at load time AFAIK
18:07 appguru mods can only read from the folders of other (world)mods
18:07 erlehmann so i guess i somehow did not give the first mod that handle. but how does the handle look like?
18:07 appguru the handle is the return value of io.open
18:08 erlehmann local f = assert(io.open(filename, "wb"))
18:08 erlehmann f:write(self.data)
18:08 erlehmann f:close()
18:08 appguru f is the handle
18:08 erlehmann yeah, but it is not a handle to the folder
18:08 erlehmann the thing is, i do not understand when i can write new textures at runtime
18:08 appguru no, the handles are always to files which must be within the mod folder
18:08 appguru erlehmann: it might be impossible
18:09 erlehmann oh but this is in  world folder
18:09 erlehmann appguru it is evidently possible
18:09 erlehmann tga_encoder does it, for example
18:09 appguru you should be able to write almost anywhere in the world folder, except for the worldmods
18:09 erlehmann then maybe i get the path wrong
18:09 appguru I write textures in epidermis to, and I write them to the world folder as well
18:09 rubenwardy erlehmann: you can only write to your mods folder whilst your mod is loading
18:09 appguru I was saying it might be impossible to write to the mod folder at runtime
18:10 rubenwardy once the game has loaded, all mod folders are read only
18:10 erlehmann i want to write to the world folder though
18:10 rubenwardy well, you can have handles open at load time and then write later
18:10 rubenwardy you can write to the world folder at any time
18:10 appguru not if the file names are only known at runtime though
18:10 rubenwardy providing it's not in worldmods or minetest.conf
18:10 appguru or if you don't want to open 1000s of handles in preparation xD
18:10 rubenwardy both are protected for security reasons
18:11 erlehmann i guess i got the filepath wrong
18:11 rubenwardy yeah
18:11 rubenwardy use minetest.get_worldpath() .. "/filepath.txt"
18:11 rubenwardy (MT converts / to \ on windows correctly)
18:11 MTDiscord <luatic> is that documented anywhere?
18:13 rubenwardy don't know, but should be
18:13 MTDiscord <Jonathon> no
18:13 MTDiscord <Warr1024> Haha, crap, I got all excited but I guess it looks like that proxy wouldn't help me much.
18:14 erlehmann Warr1024 what is the matter with it?
18:15 erlehmann can i maybe not write to existing files o.0?
18:28 MTDiscord <Warr1024> I mean that proxy sounds like it mainly solves a problem I don't have.
18:28 MTDiscord <Warr1024> Mainly what I've been seeing are object properties and block sends.
18:28 MTDiscord <Warr1024> I assume block sends I can't do much about, but I know object properties are pretty hardcore-inefficient.
18:37 jfred joined #minetest
18:43 erlehmann rubenwardy appguru i had indeed used the wrong path
18:43 erlehmann 2022-02-03 19:43:30: [Server]: uncompressing texture
18:43 erlehmann segmentation violation
18:43 erlehmann nice lol
19:06 Verticen joined #minetest
19:07 v-rob joined #minetest
19:10 lagash joined #minetest
19:12 ghoti Er, newbie lua question here... I know how I can read data from a file with io.input(), but how do I read the array stored in xban.db so that I can do things with it? It there some soft of "eval()" that I haven't found yet?
19:15 MTDiscord <Warr1024> if it's like return { stuff } then you'd use minetest.deserialize
19:16 MTDiscord <Warr1024> if it's like {"stuff":"more stuff"} then that's JSON and you'd look for the JSON-reading/parsing functions, I don't remember what they're called.
19:19 specing_ joined #minetest
19:19 erlehmann local function load_json_file(name)
19:19 erlehmann local file = assert(io.open(modpath .. "/" .. name .. ".json", "r"))
19:20 erlehmann local data = minetest.parse_json(file:read())
19:20 erlehmann file:close()
19:20 erlehmann return data
19:20 erlehmann end
19:20 erlehmann ghoti, does that help you?
19:20 MTDiscord <Jonathon> dont use xban without patching its data lose
19:20 MTDiscord <Jonathon> *loss
19:21 MTDiscord <Jonathon> xban should use json to store its data so it doesnt lose it, but krock closed that issue as fixed when it really isnt
19:21 MTDiscord <Jonathon> areas, and (mt-mods) travelnets used to have this issue but have been fixed
19:23 ghoti erlehmann: it is surely helpful information, excalt that the file appears to be in lua array format rather than JSON.  `return { .. [234] = { ["names"] = { "foo" = true, ...`
19:23 ghoti This is xban.db
19:23 erlehmann well then listen to jonathon instead of me ig
19:23 erlehmann i know nothing about xban db format
19:23 erlehmann on the data louse
19:24 MTDiscord <Jonathon> minetest.deserialize(xbandb)
19:24 MTDiscord <Jonathon> but you should really patch it to use json
19:24 MTDiscord <Jonathon> unless you like all your bans going away every so often
19:24 ghoti Jonathon, thanks, I am only looking to analyze my daily snapshots of xban.db using a standalone tool, the results do not need to be complete. Just dealing with the basic of lua still.
19:24 ghoti JSON would be much better, yes.
19:25 ghoti Jonathon, where would I look for the patch? Should I search closed issues on github?
19:26 MTDiscord <Warr1024> If you're capturing daily snaps, make sure to keep some old copies around in case the aforementioned data loose happens so you can recover from it.
19:26 ghoti I started keeping daily snapshots when I realized I was losing old data. :/
19:27 MTDiscord <Jonathon> anyways, you know xban has a api right?
19:27 MTDiscord <Jonathon> xban.db gets you it
19:27 MTDiscord <Warr1024> Maybe they should consider switching to mod_storage now that it's actually using a database backend...
19:27 ghoti MTDiscord: is that API available to a external scripts? (Sorry, I am very new to lua.)
19:28 MTDiscord <Jonathon> external as in external mods? or something outside of minetest
19:28 ghoti oops, s/MTDiscord/Jonathon/
19:28 ghoti Jonathon, external as in, a munin or nagios plugin?
19:28 ghoti Outside of minetest entirely.
19:28 MTDiscord <Jonathon> ah, "a standalone tool"
19:29 ghoti yep.
19:29 MTDiscord <Jonathon> then you would need to reimplement minetest.deserialize outside minetest
19:29 ghoti JSON seems like a way easier option. :)
19:29 MTDiscord <Jonathon> even more reason you should patch it to use json, just import any json library you want
19:29 ghoti Or, require()?
19:29 ghoti Yes, looking for the patch now..
19:30 MTDiscord <Jonathon> its not public
19:30 MTDiscord <Warr1024> Lua has loadstring/dofile that I think can be used to deserialize minetest.serialize()d data; it's basically just lua code.  Be absolutely sure you trust any data that you're attempting to decode though, and/or run your scripts as sandboxed as you reasonably can.
19:30 MTDiscord <Jonathon> https://github.com/minetest-mods/xban2/issues/19 here is the issue that krock claims fixes it, which is literally just a clean up command to make it smaller, which eventually it will lose its mind anyways
19:31 MTDiscord <Jonathon> https://github.com/minetest-mods/areas/commit/a9c05f0e387ddcbc196d263e6338d27eb286b04e areas fix, https://github.com/mt-mods/travelnet/commit/d8c0c9d42318126b1759f589a38d093c9e3cc389 travelnet fix
19:31 MTDiscord <Warr1024> Any format that involves serializing and deserializing the entire database to/from a string is gonna run into SOME problem eventually though, so switching over to JSON just changes it from an outright crash to a slower degradation over time.
19:32 MTDiscord <Jonathon> . s/crash/data loss and you would be correct
19:32 MTDiscord <Jonathon> but minetest is yet to provide proper db access, so sban, verbanna, etc have to turn of mod sec and use external dbs
19:33 MTDiscord <Warr1024> when mod_storage was just a flat text file, there was no solution for this other than either doing your own custom multi-file database, or suffering external dependencies and complex installation.  I'm hopeful that db-driven mod_storage fixes it, but tbh haven't tested it yet.
19:33 ghoti And minetest doesn't use posgresql transactions, does it? I get the impression that postgres support is basically "swap in postgres commands instead od sqlite ones" mostly?
19:33 MTDiscord <Warr1024> mod_storage won't give you a full-blown DB, but it should hopefully at least give you a competent key/value store, upon which you can implement DB-ish things.
19:33 MTDiscord <Jonathon> the problem with mod storage is searching it
19:34 MTDiscord <Warr1024> yeah, you'd have to maintain an index manually
19:34 MTDiscord <Warr1024> not impossible, but also not pretty
19:34 MTDiscord <Jonathon> but yeah, now that its sql backed, its probably the most secure storage available for mt mods within the sand box
19:34 MTDiscord <Warr1024> Also, extra irony points if you implement a table store on top of a key value store that's now already on top of a table store :-D
19:35 MTDiscord <Jonathon> well, loading from mod storage into a table gives you faster lookup
19:35 MTDiscord <Warr1024> most SQL databases offer some ACID guarantees at least within individual queries, so postgresql and sqlite should have comparable safety guarantees
19:35 MTDiscord <Jonathon> the problem with giving mods access to pure sql is how do you handle db migration/differences in queries
19:35 MTDiscord <Warr1024> considering that MT saves all its data fully async and offers mod authors no durability guarantees, this is all about the same anyway.
19:36 MTDiscord <Jonathon> course the easy option is give mods only sqlite3 access since you always have to have it
19:36 rubenwardy Minetest deserialize is just Lua loadstring
19:36 MTDiscord <Warr1024> MT doesn't guarantee that your back-end is SQL either.  You can use redis, and mongodb wouldn't be a stretch.
19:36 rubenwardy But with sandboxing
19:38 MTDiscord <Jonathon> using Minetest.deserialize is like randomly waving a shotgun around and firing, then hoping you didnt get shot
19:38 MTDiscord <Warr1024> I briefly looked into making an external pluggable backend system by just emulating the subset of the Redis protocol that MT uses and pretending to be a Redis server for it.  That would allow me to e.g. record changes into a sqlite db but also separate out the WAL checkpointing, so I would retain having the most universal portable format, but not have to pay the cost in lagspikes.
19:38 MTDiscord <Warr1024> minetest.deserialize isn't exactly terrible, we just have better tools than it in every way but are stuck with it for legacy reasons.
19:39 MTDiscord <Jonathon> basically
19:39 MTDiscord <Jonathon> if you really want legacy, look at minetest.compress
19:40 MTDiscord <Warr1024> minetest.compress ... looks fine to me?  Or at least about what I'd expect for a simplified zlib API with room to expand that nobody ever actually availed themselves of.
19:40 erlehmann it is used
19:41 erlehmann zlib is in png and … mts ig?
19:41 MTDiscord <Jonathon> minetest.compress(table.concat(table, ""))
19:41 MTDiscord <Jonathon> ^dont use that please
19:41 ghoti Warr1024, I see regular crashes when the server tries to create a user that already exits,
19:41 MTDiscord <Warr1024> Wat?  That doesn't even make sense.  It's not the compress that doesn't make sense there, it's the concat
19:42 MTDiscord <Warr1024> like there's no local table = table.unconcat(string, "") or something XD
19:42 ghoti 'ERROR:  duplicate key value violates unique constraint "auth_name_key"'
19:42 MTDiscord <Jonathon> yeah, ive seen it and ran away screaming
19:43 MTDiscord <Warr1024> concat/split is a can of worms because whatever you use as the separator string, you have to find some way to guard against that appearing INSIDE the strings that you're concatting.
19:43 MTDiscord <Jonathon> warr, i know, i already said it was terrible
19:43 MTDiscord <Warr1024> Then of course you can only actually join a single-layer flat array with integer keys this way ... basically you'll have just reinvented the mess that is texturemod escaping.
19:44 MTDiscord <Warr1024> Yeah, terrible just like texturemod escaping XD
19:44 MTDiscord <Warr1024> ghoti, I know of no reason why MT would be attempting to insert duplicate users into the database instead of just looking up and finding the existing one.
19:44 MTDiscord <Warr1024> MT's auth system is not really my area of expertise though
19:45 MTDiscord <Warr1024> IIRC if you modify the auth data directly on the back-end, or I suppose maybe share auth data between worlds (which might itself not be a sane thing to attempt) then you might have to do some kind of /auth_reload kind of thing to refresh the MT-side cache.
19:47 ghoti Warr, I wondered if perhaps it could be as simple as someone double-clicking a Register button in the client gui, and there being insufficient error handling on the server. It sure feels like an error I should not see.
19:47 rubenwardy ghoti: what database backend?
19:47 rubenwardy It might be due to case sensitive bugs
19:47 ghoti postgresql. It has happened with both minetest and multicraft servers.
19:48 ghoti I have seen 56 of these since the start of the year. :)
19:48 rubenwardy Yeah, there's an issue for that
19:49 MTDiscord <Warr1024> Haha, good thing I heard about this before I decided to convert my auth backend to postgresql 😅
19:50 MTDiscord <Jonathon> anyways, tldr minetest data storage, always make backups because its guaranteed to eventually lose it
19:51 ghoti Warr, I still prefer having everything in postgres, so it is easier to include in dumps.
19:51 MTDiscord <Warr1024> Oh, also, since DB experts are around atm: if I, say, update and blank out the password hash column in auth.sqlite, then use a vacuum command, since that basically dumps and rewrites the database from scratch, that should mean that there's no residual password hash data left in the file that it writes out, right?
19:52 MTDiscord <Jonathon> >experts leaves the room
19:52 MTDiscord <Warr1024> ghoti, I really like using postgres for running the server, and I also back up the postgres data files themselves, but I like using sqlite as a "transfer format" for MT worlds because of its universality and portability.
19:52 MTDiscord <Warr1024> So my public backups are actually a sync from postgresql to sqlite.
19:53 ghoti Warr, I hear ya, that is a valid use. But is there a way to convert things back from postgres to sqlite for transport?
19:53 MTDiscord <Warr1024> Also means that verifying my backups are just a matter of downloading the image and trying to run it in SP, I don't have to setup a local postgresql server.
19:53 MTDiscord <Jonathon> minetest --migrate
19:53 ghoti I do a daily pg_dump, but I have been told that doing this while the server is running guarantees a corrupt dump. I just dont know if it is corrupt enough for me to worry...
19:53 MTDiscord <Warr1024> --migrate works okay, but it recreates the sqlite db each time, which means it can be organized differently, meaning you get less deduping in backups
19:54 ghoti Jonathan, oh, --migrate goes both ways? That is good news indeed.
19:54 MTDiscord <Warr1024> pg_dump really looks like it should at least give you consistency and row-level atomicity, though, which is about as good as MT itself guarantees anyway.
19:54 MTDiscord <Warr1024> Migrate goes every direction, assuming, of course, that the source db isn't already corrupted.
19:54 ghoti The fellow who warned me about this is very familiar with minetest internals but not so familiar with postgres. :)
19:55 MTDiscord <Jonathon> pg_dump should give you valid db
19:55 MTDiscord <Jonathon> pretty sure its what buckaroo uses
19:55 MTDiscord <Warr1024> What I know WILL give you a risk of corruption is trying to zip up a sqlite db while it's open by MT.
19:55 ghoti Jonathan, "valid" as a db, sure. But valid as a world?
19:55 MTDiscord <Jonathon> now dont be the idiot who cp -r a db while minetest is running
19:56 MTDiscord <Jonathon> i believe\ so
19:56 ghoti confident++
19:56 MTDiscord <Jonathon> but also test first when it comes to data
19:56 ghoti thanks
19:56 MTDiscord <Jonathon> better to check your backup before needing it and finding out its broken
19:56 MTDiscord <Jonathon> totally not talking from experience there
19:56 ghoti I would expect there to be some loss when it comes to minetest's async updates, which is to be expected.
19:57 MTDiscord <Warr1024> For my snapshots, I pull all the world files via rsync, and then do some basic validation of them.  The players and mapblock data are pulled separately out of postgresql, and I actually use pg_dump, pipe it through a short perl script to convert it into a sqlite script, and then pipe that into sqlite to apply it to the backup.
19:57 ghoti Warr, you pull in sqlite3 files via rsync _while the server is running_?
19:57 MTDiscord <Warr1024> I pull the old backup snapshot into a staging dir, update it there, validate it, and if it all succeeds, I do an rsync --delay-updates to deploy it back into my backup share.
19:58 MTDiscord <Warr1024> The only sqlite3 file in my world is auth.sqlite, which I know isn't corrupted because if it were then the command to scrub out password hashes in the public copy would fail.
19:58 MTDiscord <Warr1024> My published auth.sqlite has passwords removed so that people can't offline-brute-force attempt against other peoples' accounts, and only auth.sqlite.pgp has the passwords in it.
19:59 MTDiscord <Warr1024> https://nodecore.mine.nu/world/
19:59 MTDiscord <Warr1024> oh, I guess it's a .asc file
19:59 ghoti I store everything I can in postgres, and pg_dump daily. The dump gets stored for a week, but each one is also imported into an alternate server in case I want to get up and running quickly after a failure of the primary server.
20:00 ghoti What I would really like to get going is database replication.
20:00 MTDiscord <Jonathon> doesnt postgresql support backing up to another instance on a different server
20:00 appguru ghoti: Serialization is incorrectly demonized here. JSON is inferior to minetest.serialize in general. There is one special case if you happen to (mis)use serialization for large amounts of data where a crash will mean a server running LuaJIT won't be able to deserialize it anymore.
20:01 MTDiscord <Warr1024> I basically dump from pgsql to sqlite every 2 hours or so.  I tend to pull backup images about every 8 hours, which includes both the raw and exported data, and I commit backup snaps to borgbackup every ~24 hours, and then thin out the images over time.
20:01 MTDiscord <Warr1024> Also some players pull backups too, though I don't know their schedules.
20:02 MTDiscord <Warr1024> appguru: JSON is probably at least as good as Lua serialization for things that can be represented in JSON. Lua serialization supports more structures, but still is missing some (like ItemStacks) that you'd think should be supported.
20:02 ghoti appguru: Are you suggesting that I would in general be better off not converting various pieces to store data in JSON rather than serialized lua?
20:02 appguru The real issue here is in fact that xban chooses to store large amounts of k/v data as a serialized file instead of a proper DB structure.
20:03 appguru I'm saying for the "small" data sizes serialization to Lua is intended for, you can and should use it.
20:03 MTDiscord <Warr1024> To be fair for most of xban's life, the options provided by MT itself have been pretty bad.
20:03 MTDiscord <Jonathon> xban is in minetest-mods which is basically dead anyways
20:03 MTDiscord <Warr1024> mod_storage in an actual proper DB is also new enough in 5.5, and 5.5 itself new enough, that it might be unwise to rely on people having access to "the good stuff" for now.
20:03 appguru Warr1024: Which is why (one of) modlib's luon serializers has support for ItemStacks  :D
20:04 ghoti appguru: I would be perfectly happy with a small tool to convert lua serialized data to json for analysys by my external tools rather than having to write stuff in lua.. I am much more comfortable in other platforms.
20:04 MTDiscord <Jonathon> bloatlib shills now heh
20:05 appguru such a tool is trivial to write in Lua hehe
20:05 ghoti :)
20:05 appguru anyways, MT's JSON implementation is IMO trash
20:05 appguru good enough for simple JSON though
20:06 ghoti appguru: does the data stored in xban.db count as "simple"?
20:06 appguru for more details, have a look at https://github.com/minetest/minetest_docs/pull/13
20:06 appguru ghoti: yes, you could migrate it to JSON
20:06 appguru I'm pretty sure writing will be roughly half as fast though
20:09 MTDiscord <Warr1024> Really would be nice if the Lua language would just have included a "native" serialization format.  One of the reasons why JSON is basically unquestioningly the format of note in JS is because of its explicit support in the language itself.
20:09 TomTom joined #minetest
20:09 MTDiscord <Warr1024> It started its life actually as about the equivalent of MT's lua serialization, i.e. something designed to be eval'd, and ended up growing strictness and a non-eval parser along the way.
20:10 MTDiscord <Jonathon> good luck with that given lua 5.1 is eol
20:10 MTDiscord <luatic> https://www.npmjs.com/package/luon 😁
20:10 proller joined #minetest
20:10 MTDiscord <Warr1024> Lua is a bit lispy though in that they provide you with very flexible tools to solve a problem, and then more or less refuse to be prescriptive about how, and thus that tends to encourage solution fragmentation.
20:11 MTDiscord <Warr1024> Is luon part of the lua language, or just something built on top of it
20:11 MTDiscord <luatic> Luon is like JSON for JS, except it's not built on Lua but on JS
20:11 MTDiscord <Warr1024> wait whytf is that on npm?
20:11 MTDiscord <luatic> hey, I wanted to practice some JS
20:12 MTDiscord <Warr1024> oh, I guess if lua can attempt to write JSON, then JS attempting to read Luon is not all that different. ¯_(ツ)_/¯
20:13 MTDiscord <luatic> that's came out a lot shorter though: https://github.com/appgurueu/modlib/blob/master/json.lua
20:15 sfan5 is that a json parser?
20:17 ghoti I found this.. https://github.com/agladysh/lua2json .. but I don't seem to be able to run it.
20:18 MTDiscord <Warr1024> Heh, in cdbrelease, I just remembered that I wanted to have JS call a lua function that returns a table ... so I write a tiny JSON serializer for the output so it can be parsed on the JS side.
20:36 v-rob joined #minetest
20:38 oliverwood joined #minetest
20:39 jordan4ibanez joined #minetest
21:14 v-rob joined #minetest
21:17 Jason232 joined #minetest
21:18 v_rob joined #minetest
21:20 DarkSide joined #minetest
21:25 lagash Why not use MsgPack? https://github.com/kengonakajima/lua-msgpack-native
21:26 vrob joined #minetest
21:29 mazes_80 joined #minetest
21:41 vrob joined #minetest
21:59 v-rob joined #minetest
22:16 v-rob joined #minetest
22:29 proller joined #minetest
22:58 Jason232 joined #minetest
23:01 Verticen joined #minetest
23:20 Markow joined #minetest
23:32 jordan4ibanez joined #minetest
23:37 erlehmann ghoti, you need to do file.read("a") to read all of the file sorry.
23:38 erlehmann otherwise you only get one line
23:39 jordan4ibanez Fun fact: His name is pronounced "fish"
23:39 Peppy joined #minetest
23:47 erlehmann i do know that
23:47 MTDiscord <Warr1024> ffici is another fun one
23:48 MTDiscord <Warr1024> that one only comes from one word.
23:48 v-rob joined #minetest
23:56 Markow joined #minetest

| Channels | #minetest index | Today | | Google Search | Plaintext