Time Nick Message 14:56 MinetestBot TenPlus1: Jun-17 23:41 UTC please may you upload your nssm fork to CDB? 14:56 TenPlus1 hi folks 15:09 TenPlus1 hi fixer 15:09 Fixer henlo! 15:09 TenPlus1 o/ 15:11 TenPlus1 hi entu 15:19 TenPlus1 Would it be possible to lock the map database and do a backup while server is still running ? 15:22 MTDiscord Prostgresql will let you do this, not sqlite tho 15:22 TenPlus1 hi Jonathon 15:23 TenPlus1 shame, I really think that minetest needs a way for server owners to be able to back-up and maybe even compress the map in some ways while things are running 15:24 MTDiscord You can, it just depends on your map db 15:24 TenPlus1 I mean for sql and leveldb one's also ;) 15:25 MTDiscord More of the db backends problem than minetests 15:25 TenPlus1 yeh, was hopeful :) 15:27 MTDiscord @Warr1024 didnt you have a script for backing up and compressing while mintest was still running? 15:28 MTDiscord My method is based on doing a backend migration from postgresql to sqlite, since a second instance of MT can read-only access the running instance's pgsql database consistently. You might miss or only sporadically capture the last several minutes' worth of changes but it's at least internally consistent. 15:29 MTDiscord One drawback to that is that the migration process recreates the sqlite database each time, and you'll end up with the file internal organization changing between versions, so if you try to keep rotating backups with any kind of intra-file deduping, this approach itself is relatively inefficient. 15:30 TenPlus1 I did have a .sh script before that backed up a map and let you remove areas that weren't being used, but it was so slow and only works on unused maps 15:30 MTDiscord To work around that, I migrate to sqlite, then load a copy of the past backup's sqlite file, find and merge changes from the new database to the copy, and then use that copy-merged version, since it started with all the unchanged stuff in its original organization. 15:30 TenPlus1 seems a lot of work 15:30 TenPlus1 another reason it would be handy to have included in the engine itself in some form or fashion 15:31 MTDiscord I mean, the script does the work, not me, so it doesn't bother me. 15:31 TenPlus1 you got a link I can see ? 15:31 MTDiscord Yes, it would be nice if MT offered some kind of "flush all current changes to disk, and then stop writing new changes so I can take a snapshot until I tell you to start writing again" kind of command, like Certain Other Game Systems have... 15:31 TenPlus1 exactly :))) 15:32 MTDiscord Gimme a few, gotta make sure the gist edition I have is the latest unbuggy version. 15:32 TenPlus1 okies :P 15:33 MTDiscord It's a pain in the ass but it allows me to have my server down only for seconds at a time to do updates, instead of potentially minutes to capture snapshots of arbitrarily-large database files. I've seen other people use versioning filesystems to achieve a similar result using sqlite but I really don't want to have to rely on something OS-provided, since I want any solution I use to be portable across platforms. 15:33 TenPlus1 thing is, our server mapfile is 95gb in size 15:36 TenPlus1 o/ wb 15:36 TenPlus1 yay, they returneth 15:36 MTDiscord https://gist.github.com/Warr1024/ae760d8d330cb06ef00a048dda89e251 15:36 TenPlus1 Thanks warr, checking now 15:37 MTDiscord If you're using postgres, then you can use a whole separate machine to run this process if you want, as long as you can access the database server, and you could use ssh or wireguard or whatever to tunnel in if necessary. 15:38 TenPlus1 like I mentioned, the only issue us that the mapfile is 95gb in size, so pretty dang huge to be copying around without somehow shrinking 15:38 TenPlus1 wb krock 15:38 MTDiscord by "mapfile" do you mean sqlite, or do you mean you're on postgres and that's the postgres size? 15:38 TenPlus1 sqlite 15:39 MTDiscord if you're on sqlite, I guess rsync avoids the "copying" part of that ... but then, you still have to take the server offline to capture a consistent snapshot and you still have to READ 95G 15:39 TenPlus1 yeh, will take some time 15:40 MTDiscord something like btrfs snapshots could allow you to quickly make a snapshot and then resume the server while you process the snap in background ... but as previously stated, I don't like OS-level approaches like that 15:40 TenPlus1 it's all so fiddly 15:40 MTDiscord for one thing I deploy in docker for easy portability and there's afaik no sane way to control kernel-level functions like mounting from inside docker, and even if I could, it'd still hurt portability... 15:48 TenPlus1 back laters :) 15:48 TenPlus1 thx for help warr 15:55 celeron55 18:30:04 <@TenPlus1> I did have a .sh script before that backed up a map and let you remove areas that weren't being used, but it was so slow and only works on unused maps 15:56 celeron55 i'd like to look at the logic used for removing areas that weren't used 16:06 MTDiscord Buckaroo banzai has mapcleaner 16:07 MTDiscord https://github.com/BuckarooBanzay/mapcleaner 16:10 MTDiscord ah, okay, it's based on protection 16:10 MTDiscord I was wondering 16:10 MTDiscord I actually have a mod that collects statistics at the mapblock level about what has been done in that mapblock, including stuff like dig/place counts, how long players have been standing in there, moving, idle, etc. 16:11 MTDiscord I only need to get around to writing a script to analyze that. 16:11 MTDiscord mapcleaner wouldn't work for me because I don't use protection mods in general, and the lack of protection for an area wouldn't indicate that it was unused anyway. 16:25 ShadowNinja For sqlite you can put it in wal mode and then run `sqlite3 map.sqlite ".backup backup.sqlite"` without taking down or hanging the server. Just make sure backup.sqlite doesn't exist or it will try to do an update which will take way longer. 16:27 ShadowNinja You can then use a deduplicating backup program like restic or borg to keep a history of backups without too much extra space. 16:28 ShadowNinja It's definitely not going to be fast on a 96GiB DB though, it'll probably be like an hour. 16:36 MTDiscord "do an update which will take way longer" also sounds like the thing that makes deduplication actually reasonably work... 16:41 ShadowNinja I have a 3.2 GiB db in a restic repo with a few snapshots using this method that only takes up 3.3GiB. restic doesn't do compression. 17:22 celeron55 do we have any backup recommendations published anywhere? 17:22 celeron55 the sqlite wal mode is a nice tip 17:23 celeron55 basically it makes it completely unnecessary to add any engine support for creating a backup at runtime 17:37 MTDiscord If we do have it published, it's clearly not doing a great job because I've heard questions about backups asked a lot and it seems like there are a ton of different competing strategies.