Minetest logo

IRC log for #minetest-hub, 2021-06-24

| Channels | #minetest-hub index | Today | | Google Search | Plaintext

All times shown according to UTC.

Time Nick Message
00:01 Alias joined #minetest-hub
06:08 CWz joined #minetest-hub
09:09 entuland joined #minetest-hub
09:37 calcul0n joined #minetest-hub
10:27 calcul0n_ joined #minetest-hub
12:11 CWz joined #minetest-hub
14:56 TenPlus1 joined #minetest-hub
14:56 MinetestBot TenPlus1: Jun-17 23:41 UTC <rubenwardy> please may you upload your nssm fork to CDB?
14:56 TenPlus1 hi folks
15:08 Fixer joined #minetest-hub
15:09 TenPlus1 hi fixer
15:09 Fixer henlo!
15:09 TenPlus1 o/
15:11 entuland joined #minetest-hub
15:11 TenPlus1 hi entu
15:19 TenPlus1 Would it be possible to lock the map database and do a backup while server is still running ?
15:22 MTDiscord <Jonathon> Prostgresql will let you do this, not sqlite tho
15:22 TenPlus1 hi Jonathon
15:23 TenPlus1 shame, I really think that minetest needs a way for server owners to be able to back-up and maybe even compress the map in some ways while things are running
15:24 MTDiscord <Jonathon> You can, it just depends on your map db
15:24 TenPlus1 I mean for sql and leveldb one's also ;)
15:25 MTDiscord <Jonathon> More of the db backends problem than minetests
15:25 TenPlus1 yeh, was hopeful :)
15:27 MTDiscord <Jonathon> @Warr1024 didnt you have a script for backing up and compressing while mintest was still running?
15:28 MTDiscord <Warr1024> My method is based on doing a backend migration from postgresql to sqlite, since a second instance of MT can read-only access the running instance's pgsql database consistently.  You might miss or only sporadically capture the last several minutes' worth of changes but it's at least internally consistent.
15:29 MTDiscord <Warr1024> One drawback to that is that the migration process recreates the sqlite database each time, and you'll end up with the file internal organization changing between versions, so if you try to keep rotating backups with any kind of intra-file deduping, this approach itself is relatively inefficient.
15:30 TenPlus1 I did have a .sh script before that backed up a map and let you remove areas that weren't being used, but it was so slow and only works on unused maps
15:30 MTDiscord <Warr1024> To work around that, I migrate to sqlite, then load a copy of the past backup's sqlite file, find and merge changes from the new database to the copy, and then use that copy-merged version, since it started with all the unchanged stuff in its original organization.
15:30 TenPlus1 seems a lot of work
15:30 TenPlus1 another reason it would be handy to have included in the engine itself in some form or fashion
15:31 MTDiscord <Warr1024> I mean, the script does the work, not me, so it doesn't bother me.
15:31 TenPlus1 you got a link I can see ?
15:31 MTDiscord <Warr1024> Yes, it would be nice if MT offered some kind of "flush all current changes to disk, and then stop writing new changes so I can take a snapshot until I tell you to start writing again" kind of command, like Certain Other Game Systems have...
15:31 TenPlus1 exactly :)))
15:32 MTDiscord <Warr1024> Gimme a few, gotta make sure the gist edition I have is the latest unbuggy version.
15:32 TenPlus1 okies :P
15:33 MTDiscord <Warr1024> It's a pain in the ass but it allows me to have my server down only for seconds at a time to do updates, instead of potentially minutes to capture snapshots of arbitrarily-large database files.  I've seen other people use versioning filesystems to achieve a similar result using sqlite but I really don't want to have to rely on something OS-provided, since I want any solution I use to be portable across platforms.
15:33 TenPlus1 thing is, our server mapfile is 95gb in size
15:35 MinetestBot joined #minetest-hub
15:36 TenPlus1 o/ wb
15:36 VanessaE joined #minetest-hub
15:36 TenPlus1 yay, they returneth
15:36 MTDiscord <Warr1024> https://gist.github.com/Warr1024/ae760d8d330cb06ef00a048dda89e251
15:36 MinetestBot joined #minetest-hub
15:36 TenPlus1 Thanks warr, checking now
15:37 VanessaE joined #minetest-hub
15:37 MTDiscord <Warr1024> If you're using postgres, then you can use a whole separate machine to run this process if you want, as long as you can access the database server, and you could use ssh or wireguard or whatever to tunnel in if necessary.
15:37 bigfoot547 joined #minetest-hub
15:38 TenPlus1 like I mentioned, the only issue us that the mapfile is 95gb in size, so pretty dang huge to be copying around without somehow shrinking
15:38 Krock joined #minetest-hub
15:38 TenPlus1 wb krock
15:38 MTDiscord <Warr1024> by "mapfile" do you mean sqlite, or do you mean you're on postgres and that's the postgres size?
15:38 TenPlus1 sqlite
15:39 MTDiscord <Warr1024> if you're on sqlite, I guess rsync avoids the "copying" part of that ... but then, you still have to take the server offline to capture a consistent snapshot and you still have to READ 95G
15:39 TenPlus1 yeh, will take some time
15:40 MTDiscord <Warr1024> something like btrfs snapshots could allow you to quickly make a snapshot and then resume the server while you process the snap in background ... but as previously stated, I don't like OS-level approaches like that
15:40 TenPlus1 it's all so fiddly
15:40 MTDiscord <Warr1024> for one thing I deploy in docker for easy portability and there's afaik no sane way to control kernel-level functions like mounting from inside docker, and even if I could, it'd still hurt portability...
15:48 TenPlus1 back laters :)
15:48 TenPlus1 thx for help warr
15:55 celeron55 18:30:04 <@TenPlus1> I did have a .sh script before that backed up a map and let you remove areas that weren't being used, but it was so slow and only works on unused maps
15:56 celeron55 i'd like to look at the logic used for removing areas that weren't used
16:06 MTDiscord <Jonathon> Buckaroo banzai has mapcleaner
16:07 MTDiscord <Jonathon> https://github.com/BuckarooBanzay/mapcleaner
16:10 MTDiscord <Warr1024> ah, okay, it's based on protection
16:10 MTDiscord <Warr1024> I was wondering
16:10 MTDiscord <Warr1024> I actually have a mod that collects statistics at the mapblock level about what has been done in that mapblock, including stuff like dig/place counts, how long players have been standing in there, moving, idle, etc.
16:11 MTDiscord <Warr1024> I only need to get around to writing a script to analyze that.
16:11 MTDiscord <Warr1024> mapcleaner wouldn't work for me because I don't use protection mods in general, and the lack of protection for an area wouldn't indicate that it was unused anyway.
16:25 ShadowNinja For sqlite you can put it in wal mode and then run `sqlite3 map.sqlite ".backup backup.sqlite"` without taking down or hanging the server.  Just make sure backup.sqlite doesn't exist or it will try to do an update which will take way longer.
16:26 wsor4035 joined #minetest-hub
16:27 ShadowNinja You can then use a deduplicating backup program like restic or borg to keep a history of backups without too much extra space.
16:28 ShadowNinja It's definitely not going to be fast on a 96GiB DB though, it'll probably be like an hour.
16:36 MTDiscord <Warr1024> "do an update which will take way longer" also sounds like the thing that makes deduplication actually reasonably work...
16:41 ShadowNinja I have a 3.2 GiB db in a restic repo with a few snapshots using this method that only takes up 3.3GiB.  restic doesn't do compression.
16:44 Lone_Wolf joined #minetest-hub
17:22 CWz joined #minetest-hub
17:22 celeron55 do we have any backup recommendations published anywhere?
17:22 celeron55 the sqlite wal mode is a nice tip
17:23 celeron55 basically it makes it completely unnecessary to add any engine support for creating a backup at runtime
17:37 MTDiscord <Warr1024> If we do have it published, it's clearly not doing a great job because I've heard questions about backups asked a lot and it seems like there are a ton of different competing strategies.
20:41 Thomas-S joined #minetest-hub
20:41 Thomas-S joined #minetest-hub
20:49 Lone_Wolf joined #minetest-hub
22:21 ShadowNinja joined #minetest-hub
22:21 ShadowBot joined #minetest-hub
22:21 m42uko joined #minetest-hub
22:21 Thomas-S joined #minetest-hub
22:21 VanessaE joined #minetest-hub
22:21 entuland joined #minetest-hub
22:21 Fixer joined #minetest-hub
22:21 Sokomine joined #minetest-hub
22:21 Jordach joined #minetest-hub
22:21 MTDiscord joined #minetest-hub
22:21 Evergreen joined #minetest-hub
22:22 Krock joined #minetest-hub
22:22 MinetestBot joined #minetest-hub
22:22 calcul0n_ joined #minetest-hub
22:22 | joined #minetest-hub
22:22 Alias joined #minetest-hub
22:22 Peppy joined #minetest-hub
22:22 rubenwardy joined #minetest-hub
22:22 BuckarooBanzai joined #minetest-hub
22:22 clavi joined #minetest-hub
22:22 x_ joined #minetest-hub
22:22 wsor4035 joined #minetest-hub
22:22 olliy joined #minetest-hub
22:23 bwarden joined #minetest-hub
22:23 nore joined #minetest-hub
22:23 lumidify joined #minetest-hub
22:23 sfan5 joined #minetest-hub
22:23 ssieb joined #minetest-hub
22:23 Lone_Wolf[m] joined #minetest-hub
22:23 greeter joined #minetest-hub
22:23 Shara joined #minetest-hub
22:24 luk3yx joined #minetest-hub
22:36 Alias joined #minetest-hub
22:37 T4im joined #minetest-hub
23:59 AliasAlreadyTake joined #minetest-hub

| Channels | #minetest-hub index | Today | | Google Search | Plaintext