Time Nick Message 11:59 MTDiscord https://github.com/minetest/minetest/issues/11449 11:59 MTDiscord Sounds really important especially for old servers with lots of generated areas 12:03 MTDiscord This can't be a non-duplicate. Certain other blocky-sandbox games have had comparable features since years before I came to MT roughly 7 years ago so I just can't imagine it hasn't been brought up before. 12:05 MTDiscord 7+ years ago? And wasnt added.. i think thats a ver important feature 12:08 MTDiscord There are workarounds, and of course workarounds do tend to kill momentum on fixing a fundamental issue. 12:09 MTDiscord I actually switched to postgres almost purely because I was tired of having to take the server down for several minutes for a backup ... but with postgres I don't even need any integration with the server to do backups, I can pull backups directly off the database without MT even knowing. 12:11 MTDiscord Sounds useful, however, you said you dont want to take it down for minutes.. as mentioned in the issue map is 95gb that means it will need 6-7 hours to back up 12:13 MTDiscord NVMe time :-D ... but seriously at that stage you can't just make a dumb copy of the file, you need some kind of incremental scheme. That's why the idea of freezing saves, copying the file, and then resuming saves, is a bad idea. Don't include the file copy in the feature, make the external software responsible for figuring out what it wants to do with the quiescent database. MC got the design right in that regard. 12:14 MTDiscord If I had a 95G world I would certainly at least want to rsync it to a separate machine instead of copying it locally, so that I could resume saving the original while I recompressed the copy on a separate machine, and so I'm not limited by the IO bandwidth of a single disk device. 12:15 MTDiscord Even more likely I might try to hack in some timestamp columns into the database so I can detect changes per-row and extract only modifications directly from the original DB without having to scan the entire file... 12:15 MTDiscord Anyway the point is to provide tools, don't dictate process. 12:15 MTDiscord And investigate cleaning the map to shrink the size 12:15 MTDiscord It would make it 100 times easier if there wa- ^(he said it..) 12:15 MTDiscord Map cleanup isn't necessarily something that a "save suspend for backup" command would do in itself though. You'd need a separate one to invalidate the in-memory stuff. 12:16 MTDiscord a way to unload every mapblock that was not touched 12:16 MTDiscord Mapcleaner already exists 12:17 MTDiscord The ability to make any kind of modifications to the on-disk data be honored safely by an already-running MT instance is probably a harder problem than this, so I would want it to be out of scope for the first PR to attempt this because I'd want to start with something simpler, easier to review, and easier to merge quickly. We can always add ambition later after a little success... 12:20 MTDiscord well, everything about this will be cool, little/big success.. The best way for it i think is to unload unused mapblocks then back them up, for addition make these while server is up.. 12:47 MTDiscord I commented and tried to prune it down to something that could actually work. 12:47 MTDiscord Now we just need a PR :-) 12:51 MTDiscord I'd actually like to see background backups of sqlite become more feasible because working around this problem is pretty much the only reason I'm using postgresql, and working around this one issue is pretty much the only compensation I get for the deployment complexity and lack of portability it introduces. 15:34 MTDiscord How long would this command take with a 80 GB map? 15:34 MTDiscord I imagine quite a bit 15:35 MTDiscord Also the map alone is not enough in most cases...it would need to be the entire world folder...mod_storage and other stuff needs to be in sync with the map in many cases 15:36 MTDiscord The map is usually by far the largest part of that though ... but yes, any sane backup includes all the things. 15:36 MTDiscord but they don't necessarily need to be THAT in sync usually 15:36 MTDiscord any decent mod after all has to already take into account the possibility that the world is at least slightly out of sync with denormalized metadata. 15:38 MTDiscord On my server I coded a custom player shop system that uses the ID of a certain shop node on the map...the mod_storage is directly tied to the meta of that node 15:39 MTDiscord Of course the mod won't completely break if that node is suddenly missing but it will display the shop node as valid in its data even though it's gone 15:43 MTDiscord Map saves and mod storage are run async, and as I understand it, in a sudden shutdown scenario, aside from the possibility of outright file corruption, it's still possible that changes made at the same time to map and meta may be durable in one but not the other. 15:44 MTDiscord There's a fair amount of background noise e.g. in NodeCore just running ABMs to make sure NodeTimers are regenerated if lost so that the game doesn't stay indefinitely broken until manual intervention. 15:44 MTDiscord No I actually think mod_storaqge gets written during map_save_interval as well 15:44 MTDiscord So ultimately as long as the backups are at least not significantly less coherent than anything else on disk, they're really good enough for an emergency at least. 15:45 MTDiscord Assuming of course you don't end up with corrupted sqlite files because you captured out-of-order writes. 19:31 BuckarooBanzai my 2 cents on the whole backup thing: i'm doing a daily online-backup from a postgres db for 3 years now and restoring from it every odd week or so for a testing-environment and never had an issue with "stale" or missing mapblocks (or data whatsoever) 19:34 BuckarooBanzai also IMO: the engine shouldn't be burdened with additional complexity if you could solve the backup issue with filesystem-level tools (snapshots for example) 20:42 MTDiscord Yeah, I'm actually starting to lean toward the idea of handling both backup and storage externally. 20:43 MTDiscord Like, I like the simplicity of sqlite, and the compatibility of using it as the native world format, but it occurs to me that redis is supported by MT, and uses a very simple TCP-based protocol, and could easily act as a middleman for a persistent sqlite database 20:43 MTDiscord but the ability to MITM between MT and the db could offer a lot of opportunities for things like backups, or even live offsite replication... 20:44 MTDiscord And also db reading/writing via mods 20:44 MTDiscord MT's db storage is basically just key/value, so implementing a DBMS for it is almost laughably easy given an existing back-end that you can MITM. 20:44 MTDiscord Yeah, mods could be interesting, though granted all I'm really able to do is bypass the file locks and such, but dealing with how MT caches everything in RAM is still a bit of a thang. 20:47 MTDiscord I might putz around with this and see where it goes. At the very least, the ability to quiesce and close the main sqlite database and divert all changes to a holding area for backups, then resume writes and flush the backlog, could be a huge boon, and such a system would be (1) simpler to deploy and less RAM-hungry than pgsql, (2) much more compatible with existing sqlite solutions (easier to offer world downloads), and (3) easier to 20:47 MTDiscord restore/test the backup without setting up a whole db server. 20:47 MTDiscord sqlite is traditionally serverless, but there's no reason one can't whip up a "sqlited" to allow concurrent access, especially with very limited access patterns, e.g. a single key/value store like MT. 20:48 MTDiscord I've found Redis to be a nice server db. Almost as light as sqlite, and not nearly as complex as postgre 20:50 rubenwardy Redis is more of a in-memory caching, the persistence is an after thought 20:51 rubenwardy I have a feeling that they keep everything in memory? Which would make it impractical for 96GB maps 20:51 MTDiscord They do not keep everything in memory 20:51 MTDiscord That wouldnt make much sense 20:53 MTDiscord no, thats the map's size, the server have been running since 2016 and most its times was popular, people exploring places, building stuff etc 20:54 MTDiscord Last time I checked, redis kept everything in RAM, but had the option to also write it back to disk, but it's much more async and less ACID than other databases. It basically got its start as a better memcached redo. 20:54 MTDiscord I think redis is preferred as a backend platform for worlds that are physically smaller but demand faster performance. 20:55 MTDiscord But MT doesn't necessarily USE redis all that much different than any other backend. The game still makes decisions when/what to read/write all about the same 20:55 MTDiscord so if I were to use the redis protocol, there's nothing that says the underlying backend needs to behave like redis actually does or offer the same advantages/disadvantages for MT's sake. 20:56 MTDiscord I just struck on redis as the idea because it has the simplest protocol, being optimized for acting as a shared RAM cache for like web server app farms and such. No TLS, the authentication stuff is all optional, everything is more or less just plain TCP segments. 20:57 MTDiscord Postgres has a fairly complex rich protocol, sqlite and leveldb are embedded and have no network element...