| Time |
Nick |
Message |
| 00:01 |
|
Alias joined #minetest-hub |
| 06:08 |
|
CWz joined #minetest-hub |
| 09:09 |
|
entuland joined #minetest-hub |
| 09:37 |
|
calcul0n joined #minetest-hub |
| 10:27 |
|
calcul0n_ joined #minetest-hub |
| 12:11 |
|
CWz joined #minetest-hub |
| 14:56 |
|
TenPlus1 joined #minetest-hub |
| 14:56 |
MinetestBot |
TenPlus1: Jun-17 23:41 UTC <rubenwardy> please may you upload your nssm fork to CDB? |
| 14:56 |
TenPlus1 |
hi folks |
| 15:08 |
|
Fixer joined #minetest-hub |
| 15:09 |
TenPlus1 |
hi fixer |
| 15:09 |
Fixer |
henlo! |
| 15:09 |
TenPlus1 |
o/ |
| 15:11 |
|
entuland joined #minetest-hub |
| 15:11 |
TenPlus1 |
hi entu |
| 15:19 |
TenPlus1 |
Would it be possible to lock the map database and do a backup while server is still running ? |
| 15:22 |
MTDiscord |
<Jonathon> Prostgresql will let you do this, not sqlite tho |
| 15:22 |
TenPlus1 |
hi Jonathon |
| 15:23 |
TenPlus1 |
shame, I really think that minetest needs a way for server owners to be able to back-up and maybe even compress the map in some ways while things are running |
| 15:24 |
MTDiscord |
<Jonathon> You can, it just depends on your map db |
| 15:24 |
TenPlus1 |
I mean for sql and leveldb one's also ;) |
| 15:25 |
MTDiscord |
<Jonathon> More of the db backends problem than minetests |
| 15:25 |
TenPlus1 |
yeh, was hopeful :) |
| 15:27 |
MTDiscord |
<Jonathon> @Warr1024 didnt you have a script for backing up and compressing while mintest was still running? |
| 15:28 |
MTDiscord |
<Warr1024> My method is based on doing a backend migration from postgresql to sqlite, since a second instance of MT can read-only access the running instance's pgsql database consistently. You might miss or only sporadically capture the last several minutes' worth of changes but it's at least internally consistent. |
| 15:29 |
MTDiscord |
<Warr1024> One drawback to that is that the migration process recreates the sqlite database each time, and you'll end up with the file internal organization changing between versions, so if you try to keep rotating backups with any kind of intra-file deduping, this approach itself is relatively inefficient. |
| 15:30 |
TenPlus1 |
I did have a .sh script before that backed up a map and let you remove areas that weren't being used, but it was so slow and only works on unused maps |
| 15:30 |
MTDiscord |
<Warr1024> To work around that, I migrate to sqlite, then load a copy of the past backup's sqlite file, find and merge changes from the new database to the copy, and then use that copy-merged version, since it started with all the unchanged stuff in its original organization. |
| 15:30 |
TenPlus1 |
seems a lot of work |
| 15:30 |
TenPlus1 |
another reason it would be handy to have included in the engine itself in some form or fashion |
| 15:31 |
MTDiscord |
<Warr1024> I mean, the script does the work, not me, so it doesn't bother me. |
| 15:31 |
TenPlus1 |
you got a link I can see ? |
| 15:31 |
MTDiscord |
<Warr1024> Yes, it would be nice if MT offered some kind of "flush all current changes to disk, and then stop writing new changes so I can take a snapshot until I tell you to start writing again" kind of command, like Certain Other Game Systems have... |
| 15:31 |
TenPlus1 |
exactly :))) |
| 15:32 |
MTDiscord |
<Warr1024> Gimme a few, gotta make sure the gist edition I have is the latest unbuggy version. |
| 15:32 |
TenPlus1 |
okies :P |
| 15:33 |
MTDiscord |
<Warr1024> It's a pain in the ass but it allows me to have my server down only for seconds at a time to do updates, instead of potentially minutes to capture snapshots of arbitrarily-large database files. I've seen other people use versioning filesystems to achieve a similar result using sqlite but I really don't want to have to rely on something OS-provided, since I want any solution I use to be portable across platforms. |
| 15:33 |
TenPlus1 |
thing is, our server mapfile is 95gb in size |
| 15:35 |
|
MinetestBot joined #minetest-hub |
| 15:36 |
TenPlus1 |
o/ wb |
| 15:36 |
|
VanessaE joined #minetest-hub |
| 15:36 |
TenPlus1 |
yay, they returneth |
| 15:36 |
MTDiscord |
<Warr1024> https://gist.github.com/Warr1024/ae760d8d330cb06ef00a048dda89e251 |
| 15:36 |
|
MinetestBot joined #minetest-hub |
| 15:36 |
TenPlus1 |
Thanks warr, checking now |
| 15:37 |
|
VanessaE joined #minetest-hub |
| 15:37 |
MTDiscord |
<Warr1024> If you're using postgres, then you can use a whole separate machine to run this process if you want, as long as you can access the database server, and you could use ssh or wireguard or whatever to tunnel in if necessary. |
| 15:37 |
|
bigfoot547 joined #minetest-hub |
| 15:38 |
TenPlus1 |
like I mentioned, the only issue us that the mapfile is 95gb in size, so pretty dang huge to be copying around without somehow shrinking |
| 15:38 |
|
Krock joined #minetest-hub |
| 15:38 |
TenPlus1 |
wb krock |
| 15:38 |
MTDiscord |
<Warr1024> by "mapfile" do you mean sqlite, or do you mean you're on postgres and that's the postgres size? |
| 15:38 |
TenPlus1 |
sqlite |
| 15:39 |
MTDiscord |
<Warr1024> if you're on sqlite, I guess rsync avoids the "copying" part of that ... but then, you still have to take the server offline to capture a consistent snapshot and you still have to READ 95G |
| 15:39 |
TenPlus1 |
yeh, will take some time |
| 15:40 |
MTDiscord |
<Warr1024> something like btrfs snapshots could allow you to quickly make a snapshot and then resume the server while you process the snap in background ... but as previously stated, I don't like OS-level approaches like that |
| 15:40 |
TenPlus1 |
it's all so fiddly |
| 15:40 |
MTDiscord |
<Warr1024> for one thing I deploy in docker for easy portability and there's afaik no sane way to control kernel-level functions like mounting from inside docker, and even if I could, it'd still hurt portability... |
| 15:48 |
TenPlus1 |
back laters :) |
| 15:48 |
TenPlus1 |
thx for help warr |
| 15:55 |
celeron55 |
18:30:04 <@TenPlus1> I did have a .sh script before that backed up a map and let you remove areas that weren't being used, but it was so slow and only works on unused maps |
| 15:56 |
celeron55 |
i'd like to look at the logic used for removing areas that weren't used |
| 16:06 |
MTDiscord |
<Jonathon> Buckaroo banzai has mapcleaner |
| 16:07 |
MTDiscord |
<Jonathon> https://github.com/BuckarooBanzay/mapcleaner |
| 16:10 |
MTDiscord |
<Warr1024> ah, okay, it's based on protection |
| 16:10 |
MTDiscord |
<Warr1024> I was wondering |
| 16:10 |
MTDiscord |
<Warr1024> I actually have a mod that collects statistics at the mapblock level about what has been done in that mapblock, including stuff like dig/place counts, how long players have been standing in there, moving, idle, etc. |
| 16:11 |
MTDiscord |
<Warr1024> I only need to get around to writing a script to analyze that. |
| 16:11 |
MTDiscord |
<Warr1024> mapcleaner wouldn't work for me because I don't use protection mods in general, and the lack of protection for an area wouldn't indicate that it was unused anyway. |
| 16:25 |
ShadowNinja |
For sqlite you can put it in wal mode and then run `sqlite3 map.sqlite ".backup backup.sqlite"` without taking down or hanging the server. Just make sure backup.sqlite doesn't exist or it will try to do an update which will take way longer. |
| 16:26 |
|
wsor4035 joined #minetest-hub |
| 16:27 |
ShadowNinja |
You can then use a deduplicating backup program like restic or borg to keep a history of backups without too much extra space. |
| 16:28 |
ShadowNinja |
It's definitely not going to be fast on a 96GiB DB though, it'll probably be like an hour. |
| 16:36 |
MTDiscord |
<Warr1024> "do an update which will take way longer" also sounds like the thing that makes deduplication actually reasonably work... |
| 16:41 |
ShadowNinja |
I have a 3.2 GiB db in a restic repo with a few snapshots using this method that only takes up 3.3GiB. restic doesn't do compression. |
| 16:44 |
|
Lone_Wolf joined #minetest-hub |
| 17:22 |
|
CWz joined #minetest-hub |
| 17:22 |
celeron55 |
do we have any backup recommendations published anywhere? |
| 17:22 |
celeron55 |
the sqlite wal mode is a nice tip |
| 17:23 |
celeron55 |
basically it makes it completely unnecessary to add any engine support for creating a backup at runtime |
| 17:37 |
MTDiscord |
<Warr1024> If we do have it published, it's clearly not doing a great job because I've heard questions about backups asked a lot and it seems like there are a ton of different competing strategies. |
| 20:41 |
|
Thomas-S joined #minetest-hub |
| 20:41 |
|
Thomas-S joined #minetest-hub |
| 20:49 |
|
Lone_Wolf joined #minetest-hub |
| 22:21 |
|
ShadowNinja joined #minetest-hub |
| 22:21 |
|
ShadowBot joined #minetest-hub |
| 22:21 |
|
m42uko joined #minetest-hub |
| 22:21 |
|
Thomas-S joined #minetest-hub |
| 22:21 |
|
VanessaE joined #minetest-hub |
| 22:21 |
|
entuland joined #minetest-hub |
| 22:21 |
|
Fixer joined #minetest-hub |
| 22:21 |
|
Sokomine joined #minetest-hub |
| 22:21 |
|
Jordach joined #minetest-hub |
| 22:21 |
|
MTDiscord joined #minetest-hub |
| 22:21 |
|
Evergreen joined #minetest-hub |
| 22:22 |
|
Krock joined #minetest-hub |
| 22:22 |
|
MinetestBot joined #minetest-hub |
| 22:22 |
|
calcul0n_ joined #minetest-hub |
| 22:22 |
|
| joined #minetest-hub |
| 22:22 |
|
Alias joined #minetest-hub |
| 22:22 |
|
Peppy joined #minetest-hub |
| 22:22 |
|
rubenwardy joined #minetest-hub |
| 22:22 |
|
BuckarooBanzai joined #minetest-hub |
| 22:22 |
|
clavi joined #minetest-hub |
| 22:22 |
|
x_ joined #minetest-hub |
| 22:22 |
|
wsor4035 joined #minetest-hub |
| 22:22 |
|
olliy joined #minetest-hub |
| 22:23 |
|
bwarden joined #minetest-hub |
| 22:23 |
|
nore joined #minetest-hub |
| 22:23 |
|
lumidify joined #minetest-hub |
| 22:23 |
|
sfan5 joined #minetest-hub |
| 22:23 |
|
ssieb joined #minetest-hub |
| 22:23 |
|
Lone_Wolf[m] joined #minetest-hub |
| 22:23 |
|
greeter joined #minetest-hub |
| 22:23 |
|
Shara joined #minetest-hub |
| 22:24 |
|
luk3yx joined #minetest-hub |
| 22:36 |
|
Alias joined #minetest-hub |
| 22:37 |
|
T4im joined #minetest-hub |
| 23:59 |
|
AliasAlreadyTake joined #minetest-hub |
