| Time |
Nick |
Message |
| 00:25 |
erlehmann |
minetest_game, the game where water can be on fire https://mister-muffin.de/p/IHtW.png |
| 00:30 |
independent56 |
Can't worldedit act on unloaded blocks? Can you issue a command, and when someone laods that mapblock, worldedit does the command? |
| 00:33 |
MTDiscord |
<Jonathon> mineclonia: the game performs terribly |
| 00:43 |
|
calcul0n_ joined #minetest |
| 00:46 |
erlehmann |
Jonathon could yo be more specific? |
| 00:46 |
erlehmann |
currently the only performance hogs i am aware of are rain/snow and fire |
| 00:46 |
MTDiscord |
<Jonathon> i was being sarcastic |
| 00:47 |
erlehmann |
if you doubt that we improved performance by, like, a lot, compare the packet count to mineclone2 or mineclone5 |
| 00:47 |
erlehmann |
rain is cursed though |
| 00:48 |
MTDiscord |
<Jonathon> weather in general is cursed because players can eventually out run the weather |
| 00:48 |
MTDiscord |
<Jonathon> and using hud overlays would spam to many packets |
| 00:48 |
MTDiscord |
<Jonathon> unless you can use animated images in huds? |
| 00:49 |
MTDiscord |
<Jonathon> that would be a ideal way to doweather |
| 00:57 |
MTDiscord |
<Jonathon> however it appears that does not exist |
| 00:57 |
mazes_81 |
jonadab the autocrafter group: recipe patch is for the pipeworks autocrafter ? |
| 01:00 |
mazes_81 |
I took a look, and will have a try on it |
| 01:03 |
mazes_81 |
still about weather, does the regional_weather/climate_api author chat here ? |
| 01:05 |
MTDiscord |
<Jonathon> if they use the testificate handle here, i dont think so |
| 01:43 |
|
Hawk777 joined #minetest |
| 02:18 |
|
NotReallyASaint joined #minetest |
| 02:27 |
|
Teckla joined #minetest |
| 02:28 |
|
queria^clone joined #minetest |
| 02:30 |
|
queria^clone joined #minetest |
| 02:37 |
|
Extex joined #minetest |
| 02:55 |
|
Elzington joined #minetest |
| 03:04 |
MTDiscord |
<waar> I am bored |
| 04:00 |
|
MTDiscord joined #minetest |
| 04:06 |
|
hanetzer joined #minetest |
| 04:07 |
|
hanetzer joined #minetest |
| 04:14 |
|
hanetzer joined #minetest |
| 04:14 |
|
hanetzer joined #minetest |
| 05:01 |
|
riff-IRC joined #minetest |
| 05:04 |
|
est31 joined #minetest |
| 05:04 |
|
sec^nd joined #minetest |
| 05:57 |
|
Sven_vB joined #minetest |
| 05:59 |
|
Flabb joined #minetest |
| 06:16 |
|
CWz joined #minetest |
| 08:00 |
|
specing_ joined #minetest |
| 08:17 |
|
reumeth joined #minetest |
| 08:20 |
|
jfred joined #minetest |
| 08:23 |
|
longerstaff13 joined #minetest |
| 08:33 |
|
independent56 joined #minetest |
| 08:42 |
|
appguru joined #minetest |
| 09:00 |
|
olliy joined #minetest |
| 09:13 |
|
Tusha joined #minetest |
| 09:33 |
|
Rafi59 joined #minetest |
| 09:33 |
|
absurb joined #minetest |
| 09:38 |
independent56 |
is 111000 "us" equivlent to 111 ms? |
| 09:38 |
independent56 |
(probably microseconds) |
| 09:38 |
sfan5 |
let's see |
| 09:38 |
sfan5 |
!c 11000 / 1000 |
| 09:38 |
MinetestBot |
11.0 |
| 09:39 |
sfan5 |
wait |
| 09:39 |
sfan5 |
forgot an 1 but you can see that yes, it's correct |
| 09:41 |
independent56 |
Cool. So microseconds also follows metric. |
| 09:42 |
independent56 |
None of that silly "a minute is 1/60th of an hour unless your clock is wrong" |
| 09:43 |
|
Talkless joined #minetest |
| 09:46 |
independent56 |
My mother is getting a new computer because the "o" key keeps on spamming, making it unusable for her. Is it worth the pain of installing ubuntu server on a different machine, getting it prepared, moving the Drive, and then using that as my server just to escape the outdated 32 bit decade old webook?? |
| 09:51 |
Tusha |
Well, I would say it depends how good is the "failing" computer compared to what are you using, and I would check if user configs could be ported saving you a great hassle on the setup |
| 09:52 |
sfan5 |
or simplier: swap out the hard disks, zero changes needed in the ideal case |
| 09:53 |
Tusha |
Well, if the destination is using uefi and the old one uses legacy bios, trouble will arise |
| 09:54 |
Tusha |
It also depends if it is a laptop or desktop, as some laptops tends to be picky with linux, sadly |
| 09:56 |
independent56 |
it's a nice laptop, with being 3 years old at most. |
| 09:56 |
independent56 |
Of course, i will be using ubuntu server instead of bodhi because "i no need gui" |
| 09:57 |
Tusha |
In any case, independent56, if you want better feedback, if you can put the specs from the origin and destination computers it could make tech savvy people here (I can't include myself, to be honest) give you the best advices |
| 09:58 |
Tusha |
Also the use-case (what you intend to do with it) could be really helpful |
| 09:59 |
independent56 |
1 gb ram, 300 gb hd at most with the 32 bit. >4 gb ram, ~500 gb ssd for the medion laptop. |
| 10:00 |
independent56 |
And i will use it for the same reasons i do now: web and minetest server hosting. |
| 10:00 |
independent56 |
But i may use it split - the 32 bit hosts web, and the superior lapop can do more complex tasks |
| 10:04 |
Tusha |
With my limited knowledge on Minetest I understand that the ssd should improve the server performance |
| 10:04 |
Tusha |
I would also measure the different energy consumption on heavy load, so the powerbill don't give you a surprise at the month's end |
| 10:05 |
independent56 |
I propobably will have a moral objecitionto using it: it stores my mother's files. |
| 10:05 |
independent56 |
And i guess it only eats as much energy as it being constantly plugged in |
| 10:06 |
Tusha |
I though you were going to do a zero install on it? Or your mother will stil be using it? |
| 10:06 |
Tusha |
still* |
| 10:08 |
|
independent_ joined #minetest |
| 10:10 |
|
independent_ joined #minetest |
| 10:11 |
independent_ |
No, mum will be getting a new laptop. But i will use it with a hard disk from a computer which installed ubuntu server for that purpouse. |
| 10:13 |
Tusha |
So your mother will keep the ssd the computer is actually using then? If not a repartitioning could easily solve the file's problem |
| 10:14 |
independent_ |
I will object - putting sensitive files on a hard drive accessible to the open web is ahuge risk. partitioned or not. |
| 10:14 |
independent_ |
I will use a spare hdd |
| 10:15 |
|
grouinos joined #minetest |
| 10:16 |
Tusha |
I was talking about deleting existing partitions and creating new ones, giving slow format (although not sure if that helps with data erasing on ssds) |
| 10:16 |
grouinos |
fuckoff asshole |
| 10:16 |
grouinos |
and suck my big dick |
| 10:18 |
Tusha |
I was going to recommend using dban, but seems it doesn't destroy data securely on ssds |
| 10:19 |
independent_ |
I do not wish to destroy any data. I have a horrible reputation for data loss, losing sentimental data biannually |
| 10:20 |
independent_ |
I will make two copies of the data,but the mainamount is to be preserved on my mother'scomputer. |
| 10:20 |
independent_ |
*the ssd drava for the computer |
| 10:21 |
Tusha |
Ok so if I understand well the ssd the computer is using now will be extracted and used on your mother's new PC? |
| 10:21 |
grouinos |
you can put yours sentimental data in your ass, more secure |
| 10:22 |
grouinos |
and put your motherboard too |
| 10:23 |
independent_ |
Tusha, i was thinkingit would just sit around, and we couldcopy files. The computer isn'teven fromthe same company. There is bound tobe cross-compatibility issues |
| 10:24 |
independent_ |
\whois independent_ |
| 10:24 |
independent_ |
Wait wrong slash direction |
| 10:26 |
grouinos |
fuckoff asshole |
| 10:26 |
sfan5 |
independent_: the manufacturer doesn't matter as much as you think |
| 10:26 |
|
grouinos was kicked by MinetestBot: grouinos |
| 10:26 |
Tusha |
Well, in that case that would be the sensible option, although it could work out-the-box if you just plug in it in the new laptop, if it is easy to do so (some laptops are a pita to do anthing to them) |
| 10:27 |
sfan5 |
moving from a 32-bit to a 64-bit system should obviously mean a reinstall but other than that Linux installations are very portable |
| 10:27 |
Tusha |
yeah as sfan5 said, if both laptops have uefi it could work, although I'm not sure if the are using secure boot, which is also a pita to work with |
| 10:27 |
independent_ |
Cool, but i would like a fresh install of windows, so we can take the minimumm amount, and then let mum grow the amount by their own. |
| 10:27 |
sfan5 |
oh, windows |
| 10:27 |
sfan5 |
I suggest reinstalling that |
| 10:28 |
Tusha |
I was afraid of that |
| 10:28 |
independent_ |
Mum uses windows and it's what she was grown up with. I do not wish to switch her to another OS. it would be too much work |
| 10:28 |
Tusha |
Yeah, if you have an usb hdd adaptor, just copy files to the new computer and then use the old ssd on the "failing" pc |
| 10:28 |
independent_ |
But she is using a spare linux mintcomputer, and she oes just fine. |
| 10:29 |
Tusha |
Or use an usb stick to copy all the "sensible" data to the new one |
| 10:29 |
independent_ |
Tusha, The thing is, whenever i try to copyfiles, some inevitably get missed. I want to ensure mum dosent get that onosecond. |
| 10:30 |
Tusha |
Uhm, how big the space used in the ssd? |
| 10:30 |
|
grouinos joined #minetest |
| 10:30 |
independent_ |
No idea. It's at the repair shop, who said they couldn't repair the ssd |
| 10:30 |
grouinos |
you can put your ssd in your ass |
| 10:30 |
|
grouinos was kicked by MinetestBot: grouinos |
| 10:31 |
Tusha |
But the computer was failing besides the o key? I mean, it could boot? |
| 10:32 |
independent_ |
Yeah, it could boot. It could do anything. But it can't do anything. Logging in is impossible. daily functions are. because the keyboard is screaming "oooooooooooo" |
| 10:33 |
independent_ |
She can't write an email. She can't open files (as typing will open upthe search prompt). |
| 10:33 |
Tusha |
I see, but that doesn't have anything to do about repairing the ssd, as the store said, there is nothing to repair there to start with |
| 10:33 |
sfan5 |
I'm pretty sure you can stop it from doing that if you clean the keyboard somehow |
| 10:34 |
independent_ |
I meant that the store said "they can't repair the computer". The ssd is fine. Sorry for the mistake. |
| 10:34 |
Tusha |
yeah, probably a good vacuum cleaner could help with it. If you have the laptop model chances are there are a few videos on youtube about how to open it |
| 10:34 |
Tusha |
Oh ok, that makes sense independent_ don't worry :) |
| 10:35 |
independent_ |
But she is getting a new computer. hopefully the placebo effectcan make her more productive or something. |
| 10:36 |
independent_ |
And there will at least be a spare computer. |
| 10:36 |
Tusha |
It depends for what she uses the computer, if she is using MS Office, sadly getting outside of windows will impact her, if she just uses it for web-browsing and web applications, a transition to a linux distro with win-like desktop distribution could be easy |
| 10:37 |
independent_ |
She does a lot. She has to downlaod bank things and view/edit them. She is fine with the linux mint computer i gave her, but i can't see things staying like that. Windows is the only thing she is even close to being fluent in. |
| 10:38 |
independent_ |
The spare computer is my sister's. It's HDD was failing, so i gave it a strong ssd and installed linux mint. I find it impossible to create a windows boot disk without a windows OS. I will get her windows hen i can. |
| 10:39 |
Tusha |
Uhm, so maybe Excel is beign used, I guess LibreOffice Calc is one of the nearest feture-wise replacements for ms office, but sadly it isn't enough |
| 10:42 |
Tusha |
If you have an spare usb drive I think you can create a win10 installation from linux |
| 10:43 |
independent_ |
I did try. Using balenaetcher, it wouldn |
| 10:43 |
independent_ |
't work |
| 10:44 |
Tusha |
I would suggest rufus, it hasn't failed me yet, at least on relative new computers |
| 10:45 |
Tusha |
Also there is a itsfoss guide about win10 boot creation using ventoy |
| 10:46 |
|
calcul0n_ joined #minetest |
| 10:47 |
Tusha |
https://itsfoss.com/bootable-windows-usb-linux/ |
| 10:48 |
Tusha |
I would try rufus first, and if it doesn't work then give a try to that tutorial, since ventoy seems overcomplicated for what it does |
| 10:48 |
independent_ |
intresting, willhave a look when the computer isn't being used for family-critical stuff and mum has her computer with all files. |
| 10:48 |
|
independent_ joined #minetest |
| 10:54 |
Tusha |
If you want to backup data with a 90% chance of not missing anything, "x:/Users" contains all the user data, and I would copy it using a dedicated program or on linux to avoid permissions issues |
| 10:55 |
Tusha |
If you want to keep the entire filesystem, then a ddrescue backup (which allows to omit non-used sectors) could be the safest bet, although re-mounting the image isn't completly straightfoward though |
| 10:58 |
independent_ |
hmm |
| 10:58 |
independent_ |
Maybe i just give her the SSD and say "copy on demand" |
| 11:00 |
Tusha |
As long as you have a ssd usb adapter, yeah that could be the safest bet to not be blamed for anything |
| 11:01 |
Tusha |
Although you will need to buy a new ssd for the new laptop, if you plan on using the old one too |
| 11:05 |
|
independent_ joined #minetest |
| 11:07 |
|
proller joined #minetest |
| 11:14 |
Tusha |
Anyone knows if using btrfs can extend an hdd life compared to default config ext4? |
| 11:26 |
|
sec^nd joined #minetest |
| 11:28 |
erlehmann |
ported the meshnode mod to mcl2-derived mods and got the elusive half-bed as a reward https://mister-muffin.de/p/lGoG.png |
| 11:39 |
|
est31 joined #minetest |
| 11:58 |
Tusha |
Is possible to change the first-person tool animation? I haven't seen anything about it so far, and even weapon-oriented games/mods still use the same animation when the weapon is used (so far checked capture the flag and the weapons mod) |
| 12:01 |
sfan5 |
nope, there was a PR for it once I think |
| 12:01 |
Tusha |
So sfan5, it is implemented in core then? |
| 12:02 |
sfan5 |
the animation is hardcoded client-side yes |
| 12:03 |
Tusha |
Dang, well better know about it that trying to find how to change it and not fiding anything, thanks :) |
| 12:04 |
|
CWz joined #minetest |
| 12:04 |
Tusha |
Also I guess support for changing textures inside a tool isn't supported too? At least I saw that the compass just changes the holded object instead of updating just the texture |
| 12:06 |
sfan5 |
indeed |
| 12:14 |
|
est31 joined #minetest |
| 12:21 |
erlehmann |
https://git.minetest.land/Mineclonia/mcl_meshnode/pulls/10 |
| 12:31 |
|
kamdard joined #minetest |
| 12:33 |
MTDiscord |
<Noodlemire> "Is possible to change the first-person tool animation?" There is a roundabout way to accomplish this. You can disable the first person using "player:hud_set_flags({wielditem = false})" and then attack an item entity to the model's hand, set to be visible in first person and to match the player's wielded item in an on_step. A bit unwieldy and I'm not fully sure if a publicly released mod can do this yet? But the end result is that |
| 12:33 |
MTDiscord |
changing the first person animation is as easy as changing the third person animation, now that they're one and the same. An empty hand will be invisible in first person by default, but I'll let you decide if that's a big deal or not. |
| 12:43 |
|
mdk joined #minetest |
| 12:47 |
|
mdk joined #minetest |
| 12:49 |
|
Verticen joined #minetest |
| 12:55 |
|
mdk joined #minetest |
| 12:55 |
|
mdk joined #minetest |
| 13:01 |
|
TaokiLaptop joined #minetest |
| 13:05 |
Tusha |
Noodlemire I could explore that, fixing the invisible empty hand could be accomplished by creating an item that resembles the hand itself, seems like a complete roundabound fix to a problem that shouldn't exists, I guess :P |
| 13:07 |
|
rymiel joined #minetest |
| 13:13 |
|
Fixer joined #minetest |
| 13:18 |
MTDiscord |
<Noodlemire> Roundabout solutions to problems that shouldn't exist are the best solutions. :) |
| 13:19 |
independent_ |
What is the worldedit command to replace anything that isn't "air", "advtrains:dtrack_st", "default:torch", and "default:stone" with "default:stone"? |
| 13:32 |
erlehmann |
regarding hands |
| 13:33 |
erlehmann |
cheat clients can help you to lend someone a hand |
| 13:33 |
erlehmann |
literally |
| 13:33 |
|
proller joined #minetest |
| 13:33 |
erlehmann |
i have a chest will illegal items somewhere and one of it is a player hand |
| 13:33 |
erlehmann |
will → with |
| 14:04 |
specing |
lol |
| 14:22 |
|
appguru joined #minetest |
| 14:32 |
Tusha |
This is a bit offtopic but I will ask, I have a forgotten ubuntu server machine which I intend to use as a relay server of sorts, and was thinking on setting some kind of bouncer for irc. I reduced the list to TheLounge and soju, which would be the recommended one? |
| 14:34 |
sfan5 |
only ever heard of the former, supposedly it's nice |
| 14:40 |
specing |
ZNC? |
| 14:40 |
specing |
Tusha: https://matrix.to/#/#minetest:libera.chat?via=libera.chat |
| 14:41 |
specing |
substitute channel name as you wish, they all work |
| 14:41 |
|
Elzington_ joined #minetest |
| 14:41 |
Tusha |
Yeah But I wanted something direct, without vector getting in the way, I don't completly trust them |
| 14:42 |
Tusha |
I though theloungue is a full featured hosted client which can also act as a bouncer |
| 14:42 |
Tusha |
I though on ZNC, but reading about bouncers seems soju is more lightweight |
| 14:44 |
Tusha |
https://git.sr.ht/~emersion/soju https://github.com/emersion/soju |
| 14:44 |
sfan5 |
looking at which is more lightweight does not make sense at this scale |
| 14:45 |
sfan5 |
znc consumes 10MB of RAM and barely any CPU on my server |
| 14:46 |
Tusha |
I think it was more how it is configured-wise than resource-wise, but since I don't know any in depth I'm just repeating someone else opinion |
| 14:47 |
Tusha |
Just for reference, here's where I read it: https://old.reddit.com/r/irc/comments/lh12hu/which_is_the_best_irc_bouncer/ |
| 14:51 |
|
Extex joined #minetest |
| 14:56 |
Tusha |
How much resources thelouge uses? I'm more concerned about the CPU, as the PC has 8GB RAM but it is an intel AoC (or however is is said, can't recall the acronym) and as far as I know it will be the bottleneck |
| 15:11 |
jonadab |
mazes_81: The autocrafter is a pipeworks component, yes. |
| 15:12 |
jonadab |
Though theoretically it could be extracted and made a separate mod; the _core_ autocrafter functionality would be useful even if pipeworks were not installed. |
| 15:12 |
jonadab |
Being able to pipe things in and out makes it much MORE useful though. |
| 15:14 |
* jonadab |
has a love-hate relationship with pipeworks. It's just so darned useful, you put up with the lag it causes, and the need to constantly check for broken tubes. |
| 15:32 |
|
longerstaff13 joined #minetest |
| 15:33 |
|
behalebabo joined #minetest |
| 15:36 |
|
Desour joined #minetest |
| 15:47 |
|
proller joined #minetest |
| 15:47 |
|
kamdard joined #minetest |
| 15:57 |
|
independent_ joined #minetest |
| 16:00 |
|
kamdard joined #minetest |
| 16:01 |
|
independent_ joined #minetest |
| 16:04 |
|
Wuzzy joined #minetest |
| 16:18 |
|
independent_ joined #minetest |
| 16:25 |
|
garywhite joined #minetest |
| 16:25 |
|
garywhite joined #minetest |
| 17:24 |
|
independent_ joined #minetest |
| 17:27 |
erlehmann |
so i have a voxelmanip that takes a bunch of node from HERE and puts them THERE. problem: they are chests, they spawn chest entities via LBM. |
| 17:27 |
|
independent_ joined #minetest |
| 17:27 |
erlehmann |
is there some way to let the voxelmanip run the “run at every load” LBM upon placing the node maybe? |
| 17:29 |
MTDiscord |
<Warr1024> you can call the func yourself directly after you've committed the vmanip data to the map |
| 17:37 |
|
est joined #minetest |
| 17:40 |
independent_ |
https://imgur.com/a/3yuFeAq What do you think of this monstorus viaduct (i built this as a preview for other builders to get an idea) |
| 17:41 |
erlehmann |
i must admit i do not understad the SCCSM push |
| 17:41 |
erlehmann |
understand i mean |
| 17:42 |
erlehmann |
i have consistently seen in the wild that people who want to execute arbitrary code on some random computers that are not theirs produce code that is horrible in both performance and security time after time |
| 17:42 |
erlehmann |
i mean, if huge corporations can't get their javascript right with millions of dollars … |
| 17:43 |
MTDiscord |
<Jordach> Consider |
| 17:43 |
MTDiscord |
<Jordach> Every fucking client server video game |
| 17:45 |
MTDiscord |
<GreenXenith> By the way, erlehmann, I've seen you doing a lot of stuff with meshnode. It's super outdated and the author is gone, though. I have a mod that does the node-as-entity thing that is much more up to date and complete https://github.com/GreenXenith/lvae/ (yes, this is a shameless plug). It's just an API mod, so theres no ships or anything-- that will come in a later mod. I figured you might be interested in taking a look, maybe you could |
| 17:45 |
MTDiscord |
write your own ship mod, or port meshnode's ship-making system. |
| 17:46 |
|
CWz joined #minetest |
| 17:47 |
erlehmann |
GreenXenith nice. is there any way i can use your stuff today? |
| 17:48 |
erlehmann |
Jordach in fact, some of the best games do not ever send executable code, but highly structured data |
| 17:48 |
erlehmann |
Jordach regardless, popularity is a useless metric to evaluate maintenance & performance & security |
| 17:49 |
MTDiscord |
<GreenXenith> What do you mean by "use it today"? Of course you can download and use it? It is public after all... If you mean using it for ships, again, you'd need to make your own ship constructor mod using it or port meshnode's construction system. |
| 17:49 |
erlehmann |
GreenXenith is there any mod that uses that API? |
| 17:49 |
MTDiscord |
<GreenXenith> There is not, as far as I know |
| 17:50 |
MTDiscord |
<GreenXenith> I havent advertised it much |
| 17:50 |
MTDiscord |
<GreenXenith> (at all, until now) |
| 17:50 |
erlehmann |
please do |
| 17:50 |
erlehmann |
sadly, i have little energy. i will continue improving meshnode for now. |
| 17:51 |
MTDiscord |
<GreenXenith> That's fair; I'll let you know when I have a complete replacement for meshnode. |
| 17:52 |
erlehmann |
GreenXenith are stairs and slabs supported? |
| 17:52 |
erlehmann |
also panes |
| 17:52 |
erlehmann |
and fences |
| 17:52 |
|
Desour joined #minetest |
| 17:52 |
erlehmann |
-- NOTE: Most of this file is a mess and is likely broken. Could use a rewrite. |
| 17:52 |
erlehmann |
lol |
| 17:53 |
erlehmann |
i like the ASCII art here https://github.com/GreenXenith/lvae/blob/master/nodebox.lua |
| 17:53 |
|
Desour joined #minetest |
| 17:54 |
MTDiscord |
<GreenXenith> Every node except liquid and plantlike_rooted is supported |
| 17:56 |
MTDiscord |
<GreenXenith> Nodeboxes are iffy |
| 17:56 |
MTDiscord |
<GreenXenith> So fences, panes, and stairs should in theory work, but in practice they may break |
| 17:57 |
MTDiscord |
<GreenXenith> (Oh, also glasslike_framed is not implemented) |
| 17:59 |
erlehmann |
what is it with the liquid filled glasslikes |
| 18:00 |
MTDiscord |
<GreenXenith> Ah yes, I forgot glasslike_liquidlevel was a thing |
| 18:01 |
MTDiscord |
<GreenXenith> Lacks support for glasslike_framed/liquidlevel and plantlike_rooted. Fortunately the glasslikes are not commonly used. |
| 18:04 |
|
Sven_vB joined #minetest |
| 18:12 |
Tusha |
Uhm, could the twitch bot let us see the others' teams creations, so far we can see only an small fraction of the winner |
| 18:14 |
erlehmann |
i have seen https://github.com/minetest/minetest/pull/11567 and while i have not tested it, i want to say that i have seen “lets use sqlite instead of the file system” a few times now and strongly advise against storing blobs in a database. have implemented a build system (redo) and want to note that usually approaches relying on atime/ctime/mtime are much simpler and faster. benchmarking is really important h |
| 18:14 |
erlehmann |
ere – because of course you can have a lot of small files, in which a database wins. |
| 18:14 |
Desour |
lol, twitch bridge to irc |
| 18:15 |
Desour |
#11567 |
| 18:15 |
ShadowBot |
https://github.com/minetest/minetest/issues/11567 -- Use an sqlite database for the media cache by Desour |
| 18:15 |
Desour |
media is a lot of small files, actually |
| 18:15 |
erlehmann |
for example, apenwarr redo uses sqlite “for speed” (i.e. to cheat on benchmarks), as it is possible to beat it with a shell script using the filesystem as a k/v store. |
| 18:16 |
erlehmann |
Desour, with *a lot* i mean “you may run out of inodes” |
| 18:16 |
erlehmann |
but regardless, robust benchmarking with real-world data is important |
| 18:17 |
Desour |
well, I have done some benchmarking. and the loading from media cache, and updating new files, does not seem to be a bottleneck |
| 18:17 |
erlehmann |
Desour, if i wish to benchmark your approach, what to run? |
| 18:17 |
Tusha |
Sorry that was in the wrong channel, my apologies :P |
| 18:18 |
Desour |
erlehmann: start the game and watch the terminal |
| 18:18 |
Desour |
and look into the code to see what it benchmarks |
| 18:18 |
Desour |
and fill your db. there's a script in the title comment |
| 18:19 |
erlehmann |
so how do i figure out if it is faster or slower than the file approach *on my machine*? |
| 18:20 |
Desour |
I've also added some timers to (nearly) master. see link at https://github.com/minetest/minetest/pull/11567#issuecomment-904933673 |
| 18:20 |
Desour |
but note that there's some things that I still have to do to improve performance. ie. the indexes |
| 18:20 |
erlehmann |
nice |
| 18:21 |
erlehmann |
Desour as the db fill script exists, is it executed automatically to populate the db? |
| 18:22 |
Desour |
no |
| 18:22 |
erlehmann |
why not |
| 18:22 |
Desour |
1) running the lua script from c++ would require installing the dependencies (luafilesystem, lsqlite3) |
| 18:23 |
Desour |
so, it'd have to be done in C++ |
| 18:23 |
erlehmann |
also, will the caching backend be configurable? i.e. file or db? |
| 18:23 |
Desour |
2) there's no real benefit. it fills automatically anyways |
| 18:23 |
erlehmann |
ah ok |
| 18:23 |
Desour |
no |
| 18:26 |
erlehmann |
Desour wait if you remove the old caching are you invalidating everyones cache once it is changed? o.0 |
| 18:27 |
Desour |
no, it just stays there |
| 18:27 |
erlehmann |
so it still gets loaded? |
| 18:27 |
erlehmann |
but not saved? |
| 18:27 |
Desour |
I thought ppl might have some nostalgic feelings about their cache |
| 18:27 |
Desour |
it's just there and not used |
| 18:28 |
erlehmann |
so if i joined a server that has a TON of mods like meseclams where it can take several minutes to connect |
| 18:28 |
|
Extex joined #minetest |
| 18:28 |
erlehmann |
then the change would mean ALL of that is downloaded again? |
| 18:28 |
Desour |
!mod meseclams |
| 18:28 |
MinetestBot |
Desour: There are no results for this query :( |
| 18:28 |
Desour |
yes |
| 18:29 |
erlehmann |
weelllllll … |
| 18:29 |
Desour |
new players would have to download it too |
| 18:30 |
erlehmann |
on the test machine i am typing the cache is 431M (as reported by du), on another machine on which i play more it is 943M (as reported by du) |
| 18:32 |
erlehmann |
lets say an average cache is 500M. there are currently around 500 players reported by “minetest-servers clients_list |wc -l” |
| 18:32 |
erlehmann |
have i mentioned i am on a metered connection? |
| 18:32 |
Desour |
they prolly don't have to redownload the whole cache |
| 18:33 |
erlehmann |
if you want to improve the caching system i think it is imperative to import the previous cache |
| 18:33 |
Desour |
ppl like you can still import the previous cache with a script |
| 18:33 |
erlehmann |
“ppl like you” |
| 18:33 |
erlehmann |
what the fuck |
| 18:33 |
MTDiscord |
<Warr1024> Is 500M really a reasonable cache size? I clear mine on every reboot, but I don't think it gets above like 5M... |
| 18:34 |
Desour |
mine is also around 450M I think |
| 18:34 |
erlehmann |
look most users are not technical and i know ppl who have worse connections and worse computers than my T60 and still run minetest |
| 18:35 |
MTDiscord |
<Warr1024> Do you play a bunch of different games/servers, or do they just have huge media or something? |
| 18:35 |
erlehmann |
not importing the existing cache to me seems as misguided as the “yes our website loads 10MB of javascript at startup, but after tht it is blindingly fast” |
| 18:35 |
MTDiscord |
<IhrFussel> The cache folder of people who don't ever clear it can become super big...my server alone adds 70-80 MB of media |
| 18:36 |
erlehmann |
Warr1024 i play on like 3 servers regularly, my cache folder of >900M represents prob 8 or 9 servers |
| 18:36 |
Desour |
if coredevs want me to add code to populate the new cache with the old stuff, I will do that of course. but I hardly doubt that's they consider it worth adding more code complexity, just to make ppl wait once shorter |
| 18:37 |
erlehmann |
well the code is already really complex and i don't really get for what gain if not performance |
| 18:37 |
|
longerstaff13 joined #minetest |
| 18:37 |
MTDiscord |
<IhrFussel> But it's not just once...it's once per server they play on |
| 18:37 |
MTDiscord |
<Warr1024> Haha, okay, last I checked my server, it was 2MB, and I don't bother worrying about keeping the cache around ... but I've also had some users on shitty satellite internet who have all sorts of trouble joining, including media... |
| 18:37 |
erlehmann |
usually on a shitty connection, once you get the media, it is workable |
| 18:37 |
MTDiscord |
<Warr1024> Cache is shared across servers if there's a hash collision, isn't it? |
| 18:38 |
Desour |
IhrFussel: yes, but it accumulates to a constant (the cache size) ;) |
| 18:38 |
erlehmann |
i play on 3 different servers since they have different mods ofc (mineclonia and meseclams mostly) |
| 18:38 |
erlehmann |
i mean |
| 18:38 |
erlehmann |
mesecraft |
| 18:38 |
erlehmann |
and a bit of mineclone5 |
| 18:38 |
Desour |
warr: yes |
| 18:38 |
Tusha |
My cache, which I haven't cleaned after playing on 3 servers and tested several mods and games, adds up to 910MB |
| 18:39 |
erlehmann |
“if coredevs want me to add code to populate the new cache with the old stuff” – considering that the purpose of a cache is to AVOID new downloads, i hope someone requires that! |
| 18:39 |
Tusha |
Also I would add that current in-game ContentDB doesn't like crappy connections |
| 18:39 |
Desour |
request it in the PR and see what the coredevs say |
| 18:40 |
erlehmann |
or running out of inodes lol |
| 18:40 |
Desour |
>the purpose of a cache is to AVOID new downloads |
| 18:41 |
Desour |
it's purpose is to avoid frequent re-downloads |
| 18:41 |
erlehmann |
if you don't agree that it is at least counterproductive to invalidate everyones cache at once i don't see what you are up to |
| 18:42 |
MTDiscord |
<IhrFussel> Many players live in countries that limit the traffic per month or you have to charge up to get another GB of traffic...and losing the entire cache in those cases sounds horrible |
| 18:42 |
MTDiscord |
<Warr1024> The most sensible way to manage a cache would be to keep track of how much value is being gotten out of each object in it (e.g. how much data has been avoided downloading due to a cache hit) over a recent time period, and evict items offering the lowest value if the cache is ever too large. |
| 18:42 |
erlehmann |
i am actually on a connection that gets throttled after a few GB |
| 18:42 |
Desour |
the benefits of the PR are stated in the PR: cache cleaning (and the possibility to store media zlib-compressed, for the future) |
| 18:43 |
erlehmann |
for what reason was cache cleaning based on atime/ctime/mtime not considered? i mean, i can do that in a shell script in the background. |
| 18:43 |
erlehmann |
in fact, i happen to be an expert on the topic of doing exactly that, which is why i chime in. |
| 18:44 |
MTDiscord |
<Warr1024> The benefits of zlibbing media, where most of the bulkiest stuff like sound and images is already compressed, sounds a bit dubious to me? |
| 18:44 |
erlehmann |
lol yes |
| 18:44 |
erlehmann |
what Warr1024 says |
| 18:45 |
MTDiscord |
<Warr1024> There are some things like obj meshes or locale files that could benefit a lot from zlib, but in that case it would make more sense to just support .obj.gz or .tr.gz files directly as media on the server-side, or have the server zlib them upon loading, so they're transferred compressed as well, and only need to be unpacked into memory for actual use... |
| 18:45 |
MTDiscord |
<Warr1024> But either way if we added zlib to any media, it would make more sense to be selective about the types it applies to. |
| 18:45 |
MTDiscord |
<Warr1024> AND not limit the application just to the cache... |
| 18:45 |
Desour |
Warr102: yes, but approximations of that should be fine too. also note that this approach would mean that big files are favored (because they save more data), but they also take up more space |
| 18:46 |
erlehmann |
so what is the problem with existing cache eviction techniques based on the file system? |
| 18:46 |
Desour |
>for what reason was cache cleaning based on atime/ctime/mtime not considered? |
| 18:47 |
Desour |
multiple reasons. ie. databases are useful and easy. and I don't have to fear to run into synchronization issues |
| 18:47 |
MTDiscord |
<Warr1024> Yeah, probably one of the reasons why we don't have a lot of compression features is because much of the media that would benefit the most from compression (e.g. obj, tr) are already pretty small and don't take up as much of the media space as, say, ogg files. |
| 18:47 |
erlehmann |
> databases are useful and easy |
| 18:47 |
erlehmann |
lol no |
| 18:47 |
erlehmann |
useful yes, easy not at all |
| 18:47 |
Krock |
btw, there's a file limit for FAT32-formatted SD-cards (for example) |
| 18:47 |
Krock |
hence in favour for database-based or aggressive deletion policy |
| 18:47 |
Desour |
>The benefits of zlibbing media, where most of the bulkiest stuff like sound and images is already compressed, sounds a bit dubious to me? |
| 18:48 |
Krock |
compression is futile on most assets. there's no need for that |
| 18:48 |
MTDiscord |
<Warr1024> Plain files have some transparency advantages, esp. for storing binary blobs ... but then, MT also moved its map format away from file-per-mapblock and into databases due to filesystem non-idealities.... |
| 18:48 |
erlehmann |
Krock considering that minetest cdb stops working once inodes run out, i am in favor of detecting this condition |
| 18:48 |
Desour |
many of our png images are very small. and there are some media files that might still be compressed very well, ie. translation files, .obj, sscsm .lua |
| 18:49 |
erlehmann |
Warr1024 the map *always* consists of many small things though |
| 18:49 |
erlehmann |
and those are much smaller than media typically (with the exception of inventories and books) |
| 18:50 |
Desour |
>useful yes, easy not at all |
| 18:51 |
Desour |
if you do more than just writing unstructured data, and also want some of the db features, databases are actually relatively easy. you just need to know how to use them (ie. know sql) |
| 18:51 |
Desour |
>transparency advantages |
| 18:51 |
Desour |
what do you mean with transparency? |
| 18:52 |
erlehmann |
i know enough sql to shoot myself in the foot not too often (i.e. in production) |
| 18:52 |
erlehmann |
Desour the database blobs are opaque to almost everything. filesystems are not. |
| 18:53 |
Desour |
that's just negating the word |
| 18:53 |
MTDiscord |
<Warr1024> The filesystem is essentially a universal database designed for storing blobs and which every program on the system that handles storable data can interact with. Specialized databases require you to use only tools that are specifically compatible with it. |
| 18:53 |
Desour |
if something is transparent, you can't see it |
| 18:54 |
Desour |
Warr1024: ahh, I see. |
| 18:54 |
MTDiscord |
<Warr1024> Transparency isn't an all or nothing thing. Things can be more transparent if there are more tools that allow access to it in different ways, and more opaque if you're more constrained in how you access them. |
| 18:54 |
erlehmann |
and also transparency means the backing storage does not matter |
| 18:54 |
erlehmann |
like which filesystem you have does not matter to unix tools |
| 18:54 |
MTDiscord |
<Warr1024> SQLite, for example, is a database within a file, so it must be by nature at least as opaque as the filesystem it's in. |
| 18:55 |
Desour |
do we have many programs or so, that operate on the meida cache? |
| 18:55 |
MTDiscord |
<Warr1024> The downside to the filesystem is that the places where you can store metadata are limited, so e.g. you can keep track of recency of use via atime, but not frequency of use. That might still be adequate to make a useful cache eviction tool anyway, though. |
| 18:55 |
erlehmann |
i ca do |
| 18:55 |
erlehmann |
find cache/media -type f |xargs file --mime |cut -d' ' -f2 |sort |uniq -c |
| 18:56 |
erlehmann |
Warr1024 redo-sh proves that it is possible to be faster than sqlite if you just rely on the filesystem |
| 18:56 |
MTDiscord |
<Warr1024> We don't have programs for operating on the filesystem cache because they can be so trivial to make that people who do may not even think about distributing them :-) |
| 18:56 |
erlehmann |
331 application/octet-stream; |
| 18:56 |
erlehmann |
1255 audio/ogg; |
| 18:56 |
erlehmann |
61 image/jpeg; |
| 18:56 |
erlehmann |
36833 image/png; |
| 18:56 |
erlehmann |
1123 text/plain; |
| 18:56 |
erlehmann |
that is my 431MB cache contents |
| 18:56 |
erlehmann |
now, Desour what is the equivalent of “find cache/media -type f |xargs file --mime |cut -d' ' -f2 |sort |uniq -c” for your database? |
| 18:57 |
Desour |
wait a moment |
| 18:57 |
Tusha |
Well, I guess that a SQL query could do that, not sure exactly which is needed for SQlite though |
| 18:58 |
MTDiscord |
<Warr1024> I could frankly see a use-case for storing the cache content as files, but having a small, optional metadata-only database that could be used to better inform cache eviction tools. I'm not sure if overall a bunch of "could" and "optional" is enough to justify the extra work to add one though. |
| 18:58 |
erlehmann |
Warr1024 is exactly right, cache eviction with unix shell is a oneliner |
| 18:59 |
erlehmann |
Warr1024 that is the approach that apenwarr redo takes … so far it performs worse than every tool i have seen which stores the metadata in the filesystem as key/value store |
| 19:00 |
MTDiscord |
<Warr1024> Yep, it's a heavy trade-ff |
| 19:00 |
erlehmann |
the thing is, you don't easily get enough metadata to justify it for any cache eviction strategy i know of |
| 19:00 |
MTDiscord |
<Warr1024> you'd need to have a very strong need to do very accurate cache eviction to warrant doing that. |
| 19:01 |
MTDiscord |
<Warr1024> Cache eviction is ultimately guesswork at some level though no matter how sophisticated your statistics are |
| 19:01 |
erlehmann |
i actually wrote my own build system because i wanted a very accurate up-to-date calculation (no unnecessary rebuilds, no skipping rebuilding if it is necessary) |
| 19:02 |
MTDiscord |
<Warr1024> Caching with a rich/complex database is an interesting academic exercise but it might be overengineered for what we've got in MT. It might be useful as a research tool to help us find out if what we assume we know about how caches behave in MT matches reality, but we probably don't have enough hard evidence to suggest it's a good idea for production use. |
| 19:02 |
erlehmann |
regarding the 36833 PNG images in my cache, they are already zlib compressed most likely |
| 19:03 |
MTDiscord |
<Warr1024> PNG files with no compression are I think technically possible according to the spec but I've never seen one in the wild :-) |
| 19:03 |
erlehmann |
just in case anyone wonders how to delete files with unix tools |
| 19:03 |
Desour |
is there something like `file` that takes the file data and not a path? |
| 19:03 |
erlehmann |
somethig like this: find /path/to/files -type f -mtime +30 -exec rm {} \; |
| 19:04 |
erlehmann |
Desour pipe it to stdin |
| 19:04 |
MTDiscord |
<Warr1024> file can read stdin |
| 19:04 |
Desour |
ok |
| 19:04 |
erlehmann |
; printf '<!DOCTYPE html>' |file --brief --mime - |
| 19:04 |
erlehmann |
text/html; charset=us-ascii |
| 19:04 |
erlehmann |
# exited 0 0 |
| 19:09 |
erlehmann |
Desour is writing the SQL taking time or running it? |
| 19:09 |
erlehmann |
i mean i have no idea how much more complex it is in SQL |
| 19:09 |
Desour |
writing the bash is taking time |
| 19:16 |
erlehmann |
Desour but i already posted it: “find cache/media -type f |xargs file --mime |cut -d' ' -f2 |sort |uniq -c” |
| 19:16 |
Desour |
yeah, I'm not good at bash |
| 19:16 |
erlehmann |
well, bash is very complex, try to keep to bourne shell (sh) |
| 19:16 |
erlehmann |
i can help |
| 19:17 |
erlehmann |
want to show what you have? |
| 19:17 |
erlehmann |
Desour may it be that databases are easier for you precisely bc you can't do the shell stuff and can do the db stuff? |
| 19:18 |
Desour |
tbh, I don't do db stuff often. but shell stuff neither |
| 19:18 |
erlehmann |
oh |
| 19:18 |
erlehmann |
well then i just wait until you show something and i'll tell you if it can be improved |
| 19:18 |
erlehmann |
(if i see something) |
| 19:22 |
Desour |
I want to do something like `<some stuff> | xargs echo "<thing that came from some stuff> bla"` |
| 19:22 |
Desour |
how do I do that? |
| 19:24 |
Desour |
ahh, -I |
| 19:24 |
erlehmann |
eh |
| 19:24 |
erlehmann |
are you *sure* you are doing the right thing? |
| 19:24 |
erlehmann |
Desour use printf it is portable, echo is not |
| 19:26 |
Desour |
I'm currently trying this: sqlite3 media_db.sqlite "select sha1 from files limit 10;" | xargs -d '\n' -I '{}' `sqlite3 media_db.sqlite "select data from files where sha1='{}';" | file --mime /dev/stdin` |
| 19:26 |
Desour |
but files reading from stdin is complicated |
| 19:27 |
Desour |
all the shell stuff is made for paths, not for raw data, that's my main problem I think |
| 19:32 |
Tusha |
Doesn't sqlite has ways to extract the blob info you want instead of the raw data itself? |
| 19:33 |
Desour |
what do u mean with blob info? |
| 19:33 |
Tusha |
Like, size, mime tipe, and that stuff, although I could just have missunderstood what you are trying to accomplish |
| 19:34 |
MTDiscord |
<Warr1024> The SQLite command line tool is not really useful for extracting individual items out into files or separate streams |
| 19:34 |
MTDiscord |
<Warr1024> Piping them to file to discover type based on file magic is actually sort of a worst-case for SQLite in particular. |
| 19:35 |
MTDiscord |
<Warr1024> The idea is that you have a database that lets you define more or less arbitrary metadata and handle it reasonably efficiently, so it only makes sense to use really if you're actually adding metadata. |
| 19:37 |
Desour |
Tusha: I can get the length of a blob, but not the mime, afaik. see also https://www.sqlite.org/lang_corefunc.html |
| 19:37 |
Tusha |
Just fiddling with DB Browser for SQlite and seeing if there is an internal way of checking that |
| 19:38 |
Tusha |
I see, somehow the db reader detects I saved a .png, but it may be an extension inside the program itself |
| 19:38 |
MTDiscord |
<Warr1024> More useful would be to get information that only SQLite could actually store, like how frequently the file has been accessed over a certain time window. To do that, though, you'd have to also actually store that data. |
| 19:38 |
Desour |
yeah, reading stuff that was never stored is quite impossible |
| 19:39 |
MTDiscord |
<luatic> Desour: Well, most file formats have a 4-byte magic number / header at the beginning usually. |
| 19:39 |
Desour |
Tusha: the PNG stuff that you see in the data is the filedata. the file format has some magic bytes in the beginning, to allow things such as `file` to get the file type |
| 19:39 |
MTDiscord |
<Warr1024> Shell script's advantage here is that it's very easy to extend with utilities and scripts, so it can just use file to infer the type at the time the script is run. SQLite doesn't let you add extensions like that nearly as easily. |
| 19:40 |
erlehmann |
Desour i have an opinion on the “it is easier to do stuff using a database” regarding this and many similar tasks (like cache eviction) |
| 19:41 |
MTDiscord |
<Warr1024> For an MT specific cache you could probably just add the magic bytes to a table and join on that ... but there are ambiguous cases (locale and obj files both look like text) and recreating the work of the file utility in SQL is just too much work to prove a point :-) |
| 19:41 |
MTDiscord |
<luatic> Well well |
| 19:41 |
MTDiscord |
<luatic> Databases are noice and abstract away all the FS bewlshit |
| 19:41 |
erlehmann |
look once you are starting stuff like “join on magic bytes” you are just putting lipstick on a pig |
| 19:41 |
erlehmann |
luatic databases ARE nice but you need to have a problem that fits the solution |
| 19:42 |
MTDiscord |
<luatic> Databases are always the solution |
| 19:42 |
erlehmann |
and i have my doubts that the caching problem is even in the same ballpark |
| 19:42 |
MTDiscord |
<luatic> Say, checking for old files |
| 19:42 |
erlehmann |
yeah, if you have a really big hammer you can feel like you can nail the whole world |
| 19:42 |
erlehmann |
if you get what i mean |
| 19:42 |
erlehmann |
(i mean exactly what i wrote) |
| 19:42 |
MTDiscord |
<luatic> SQLite lets you have an index, so this is actually logarithmic |
| 19:42 |
MTDiscord |
<luatic> (this is what nrz tells in a comment) |
| 19:42 |
MTDiscord |
<luatic> A filesystem would require linear cache |
| 19:42 |
MTDiscord |
<luatic> Which is just ugh |
| 19:42 |
erlehmann |
the point is you nneed to have A LOT of records to make this outperform the filesystem |
| 19:42 |
MTDiscord |
<luatic> search* |
| 19:43 |
MTDiscord |
<luatic> No, databases outperform poor filesystems all the time |
| 19:43 |
erlehmann |
years ago i did live maintenance on a database with hundreds of thousands of records. you usually don't want *that* in files. |
| 19:43 |
erlehmann |
and yes it was an emergency |
| 19:44 |
erlehmann |
i was assured i would not be fired |
| 19:44 |
erlehmann |
if it went haywire |
| 19:44 |
erlehmann |
(it worked out) |
| 19:44 |
MTDiscord |
<luatic> Well, nowadays, I'm not much of a fan of "throw SQL at it" anymore. But I'm still a fan of databases. |
| 19:44 |
erlehmann |
luatic you always need benchmarks. there are so many tools that make sense if you have problems the size of facebook or google or amazon. |
| 19:45 |
MTDiscord |
<luatic> SQL is where most bloat comes from anyways, which is why simple k/v DBs are so popular nowadays |
| 19:45 |
erlehmann |
and then there is the COST |
| 19:45 |
erlehmann |
the filesystem is a simple k/v store |
| 19:45 |
erlehmann |
with a cache |
| 19:45 |
MTDiscord |
<luatic> yes |
| 19:45 |
MTDiscord |
<luatic> but without proper indexing |
| 19:46 |
MTDiscord |
<luatic> it only indexes on filenames, usually, and even for that, it doesn't even sort them |
| 19:46 |
MTDiscord |
<luatic> it's also different from platform to platform |
| 19:47 |
erlehmann |
> it only indexes on filenames, usually |
| 19:47 |
erlehmann |
oh lol |
| 19:47 |
erlehmann |
have you read the COST paper? |
| 19:51 |
MTDiscord |
<Warr1024> Databases in general are a good solution to a lot of data storage problems. There are a lot of specialty databases that are good for specific sub-problems. For instance, for storing a lot of blob data in essentially a tree-structured key/value store with a bit of metadata, filesystems are a pretty nice kind of database for that application. |
| 19:53 |
Desour |
I give up. xargs is too difficult for me |
| 19:54 |
MTDiscord |
<Warr1024> I'm actually curious if there's any quantitative empirical evidence for the SQLite approach? Like, are there measurable savings or improvements it makes? |
| 19:55 |
MTDiscord |
<Warr1024> In theory these kind of things can just be better in some cases and worse in others, and it would be nice to know e.g. if there's an actual crossover point. |
| 19:55 |
Desour |
(but yes, I see the transparency issue that you were talking about) |
| 19:56 |
Desour |
btw., other backends could be added later, of course |
| 19:57 |
|
specing_ joined #minetest |
| 19:58 |
erlehmann |
the “in theory” thing has the problem IMO that it leads to the kind of stuff that apenwarr redo is |
| 19:59 |
erlehmann |
as i said, using sqlite “for performance” has to be justified with benchmarks |
| 19:59 |
erlehmann |
and tbh, if your stuff performs about the same i would gladly take the battle-tested less complex solution that allows me to write my own cache eviction strategy in one line of shell script |
| 19:59 |
erlehmann |
i volunteer even to compete if its just about clearing up the cache |
| 20:00 |
erlehmann |
like what approach can handle the problem faster |
| 20:00 |
erlehmann |
(for the same algorithm) |
| 20:01 |
Desour |
the reason why I chose sqlite was not performance |
| 20:01 |
erlehmann |
well it obv can't be ease of use if the “count the mime types” task is any hint of future usability |
| 20:01 |
Desour |
(at least not the top reason, I hoped for not making anything worse, of course) |
| 20:01 |
erlehmann |
what is the top reason though? |
| 20:02 |
erlehmann |
the zlib everything thing is a bad idea, we already know that |
| 20:02 |
celeron55_ |
i'd like to see a cache eviction implementation for the files without using a database, but before that i'd like to hear how you'd do it as the filesystem, given all the platforms we support, doesn't give you that much metadata |
| 20:02 |
Desour |
the problem with the count the mime types is that I'm too unexperienced with the shell. I couldn't have written the one liner you presented before |
| 20:03 |
erlehmann |
then do it in pure sql? |
| 20:04 |
sfan5 |
celeron55_: atime and mtime are POSIX but I don't think it's wise to rely on them (esp. atime, which may be disabled) |
| 20:04 |
erlehmann |
i agree that atime is not a very good measurement |
| 20:04 |
celeron55_ |
we all know the fs based cache works well and has worked well for 9 years already, and the only thing missing is eviction |
| 20:04 |
Desour |
that's like saying "do it in pure VHDL". sql isn't made to operate with things such as filetypes. (and shell couldn't do it either without the tools you've installed (ie. files)) |
| 20:05 |
celeron55_ |
the question is, what's the best solution from this starting point |
| 20:05 |
erlehmann |
i think the best solution is trying to figure out 1. ideas for when to evict 2. what strategies to use |
| 20:06 |
erlehmann |
i already pointed out one point of when to evict the cache, when you run out of inodes. granted, that's an extreme case. |
| 20:06 |
erlehmann |
lets say you do it based on cache directory size, then the naive strategy is to get rid of files that have not been touched longest first. |
| 20:07 |
erlehmann |
no additional state would need to be maintained for that naive solution |
| 20:08 |
sfan5 |
bad strategy, there are likely files that are very old (because every server has them for a long time) yet are often used |
| 20:09 |
erlehmann |
i meant “not used” with “have not been touched” |
| 20:09 |
sfan5 |
well how do you determine that from file metadata |
| 20:10 |
erlehmann |
if atime is not a useful concept, one can just update ctime by slightly tickling the file |
| 20:12 |
sfan5 |
workable idea |
| 20:12 |
erlehmann |
celeron55_ anything except windows that is not POSIX stat(2) compatible? |
| 20:12 |
sfan5 |
that merely reading the files will cause disk writes is too bad but ¯\_(ツ)_/¯ |
| 20:12 |
erlehmann |
technically, having atime does that already |
| 20:13 |
erlehmann |
and noatime disables it lol |
| 20:13 |
erlehmann |
how about |
| 20:13 |
erlehmann |
using atime and whoever does explicitly use noatime can enjoy their breakage (lots of stuff breaks then AFAIK) |
| 20:14 |
celeron55_ |
one option is to use the files, and add sqlite on the side just for the metadata |
| 20:14 |
celeron55_ |
it's kind of silly, but so is storing files in sqlite |
| 20:14 |
erlehmann |
yeah but the thing is, if you do that, you can also just save the metadata in the filesystem |
| 20:15 |
erlehmann |
in redo, i just have filename.md5sum and filename.ctime for freshness checks |
| 20:15 |
erlehmann |
its ugly but it works out |
| 20:15 |
celeron55_ |
but you really need more metadata than just the timestamp |
| 20:15 |
erlehmann |
well what do you need? |
| 20:15 |
erlehmann |
doing that kind of stuff works really well actually, considering i use the read builtin from bourne shell and some implementations make that read 1 byte at a time |
| 20:16 |
erlehmann |
like i was surprised it as so fast |
| 20:16 |
celeron55_ |
https://github.com/minetest/minetest/pull/11567#issuecomment-904888258 |
| 20:16 |
erlehmann |
but compared to even loading sqlite into memory, ofc it is |
| 20:16 |
|
reumeth joined #minetest |
| 20:17 |
erlehmann |
celeron55_ my suggested naive expiration strategy is different than the one you suggested there. it is just “to stay under a threshold, delete the files where the access data is the oldest”. this does not have the expiration problem you pose at all! |
| 20:17 |
erlehmann |
access data → access date |
| 20:18 |
erlehmann |
celeron55_ what problem does the two-stage expiration solve that my naive approach does not? |
| 20:18 |
erlehmann |
i.e. in which situations is it actually better? |
| 20:19 |
celeron55_ |
"stay under a threshold" probably works fine too, as long as the threshold is wisely set |
| 20:19 |
celeron55_ |
so that would indeed work with the filesystem |
| 20:20 |
erlehmann |
yeah but what made you come up with the two-stage expiration? there *must* be a motivation for that |
| 20:20 |
celeron55_ |
we just have to calculate what the maximum reasonable cache size is, given how many servers and games a user normally plays and what the average data size per server or game is |
| 20:20 |
erlehmann |
ppl don't just pull this out of thin air |
| 20:21 |
celeron55_ |
what's the comment missing, it already has my reasoning |
| 20:22 |
erlehmann |
ok if it does not get any more sophisticated than that, i guess that is it then. |
| 20:22 |
erlehmann |
i mean it is a simple strategy, maybe i was expecting a complicated explanation ^^ |
| 20:22 |
|
proller joined #minetest |
| 20:22 |
|
Extex joined #minetest |
| 20:22 |
erlehmann |
but you delivered a simple one |
| 20:22 |
erlehmann |
so ig i thought there was more to it and there is not |
| 20:23 |
erlehmann |
thanks |
| 20:23 |
celeron55_ |
lol |
| 20:23 |
celeron55_ |
anyway, if someone can come up with a reason why erlehmann's proposal wouldn't work, please do |
| 20:26 |
celeron55_ |
of course if there's a weird corner case or obscure platform where it doesn't work, they can just disable the cache; in practice cache eviction can be done manually or never as everyone indeed has done so far |
| 20:26 |
erlehmann |
yeah that is the beauty of it, it literally can't get worse lol |
| 20:27 |
erlehmann |
i must say i have rarely cared for the cache, is it a recent thing that ppl care? are they playing minetest on 4GB sdcards on raspis or something? |
| 20:27 |
Desour |
I don't care much how big my media cache folder is (as long as it's not tooo big). and I don't want to keep metadata of servers where I didn't play for ages. if the max cache size is too small, it will get useless, though. therefore I prefer a time-based strategy. |
| 20:28 |
celeron55_ |
i'd guesstimate the cache threshold size could be 500 MB or something like that. on the machines MT is playable, that size barely counts but it already covers tens of servers or games |
| 20:28 |
erlehmann |
well, that is not a problem, you can have both |
| 20:29 |
erlehmann |
celeron55_ 500MB is most likely too low. let me check something. |
| 20:30 |
celeron55_ |
also, there could be multiple thresholds, like 1 year @ 500 MB and 1 month @ 2 GB, or whatever |
| 20:34 |
|
independent__ joined #minetest |
| 20:34 |
erlehmann |
yeah |
| 20:34 |
erlehmann |
it's fairly simple |
| 20:34 |
erlehmann |
you can also have multiple strategies at the same time |
| 20:35 |
erlehmann |
let me check something |
| 20:35 |
erlehmann |
i renamed my cache dir to cache2 |
| 20:35 |
erlehmann |
and made a new empty one |
| 20:35 |
erlehmann |
i will now connect to a few popular servers and a few others i play on and report back |
| 20:37 |
|
independent56 joined #minetest |
| 20:40 |
erlehmann |
server 1: your-land.de:30000 cache size 86M |
| 20:41 |
erlehmann |
server 2: rubenwardy ctf. not accepting new players. |
| 20:44 |
erlehmann |
server 3: edgy1.net (catlandia, not a dating server) cache size 126M |
| 20:44 |
erlehmann |
and yes i am testing on a machine that has not a metered connection lol |
| 20:44 |
erlehmann |
(i.e. not the one i am writing from) |
| 20:47 |
erlehmann |
server 4: kay27.ignorelist.com (the mineclone5 test server) cache size 165M |
| 20:49 |
Desour |
are errorstream and co. thread-safe? |
| 20:51 |
erlehmann |
server 5: kay27.ignorelist.com (mineclone2) cache size still 165M |
| 20:51 |
sfan5 |
I don't think so but does it really matter? |
| 20:53 |
Desour |
I don't know if it matters |
| 20:56 |
erlehmann |
server 6: kay27.ignorelist.com (mineclone5 creative) cache size 167M |
| 20:57 |
|
Elzington joined #minetest |
| 20:59 |
|
luk3yx joined #minetest |
| 20:59 |
|
kevinsan_ joined #minetest |
| 21:00 |
|
beanzilla_ joined #minetest |
| 21:01 |
Desour |
erlehmann: are you always deleting your whole cache or are you checking how many files there are with a new enough timestamp? |
| 21:01 |
Desour |
(if you have atime) |
| 21:01 |
erlehmann |
i am not deleting any files right now, i am looking what happens if i visit a buch of servers |
| 21:01 |
erlehmann |
a bunch |
| 21:02 |
erlehmann |
server 7: clam.minetest.land:48140 (mineclonia with minecraft-like textures) cache size 201M (note: loading this stuff took a *long* time) |
| 21:08 |
erlehmann |
server 8: mesecraft.net:30000 (mesecraft) cache size 241M |
| 21:09 |
erlehmann |
server 9: minetest.foo-projects.org:30394 (inside the box) cache size 272M |
| 21:10 |
erlehmann |
server 10: nodecore.mine.nu:30000 (nodecore) cache size 274M |
| 21:11 |
erlehmann |
server 11: skyblock.telesight.nl (telesight skyblock) cache size 276M |
| 21:12 |
MTDiscord |
<Warr1024> it would be interesting to see those as sort of like a venn diagram or something to understand the probability distribution of media sizes for a "first" server vs subsequent ones, and how much of an impact cross-server reuse has |
| 21:13 |
erlehmann |
server 12: server.minetest.one:30001 (braincraft) cache size 307M |
| 21:15 |
erlehmann |
server 13: nico.multicraft.world:40779 (survivalgo) cache size 309M |
| 21:15 |
erlehmann |
server 14: i start exile locally cache size 315M |
| 21:16 |
erlehmann |
server 15: i start repixture locally cache size 320M |
| 21:16 |
Tusha |
The only method to compare actual cache with sqlite based cache in terms of speed, although it is a bit crude and black boxy, would be to use 2 clients on the same machine with clear data, and measure first logging load and second login load on the same server, on several servers. Maybe a cache size check could be a plus, but it depends what we want to know in the first place |
| 21:17 |
erlehmann |
so i stop now |
| 21:17 |
Tusha |
load times* |
| 21:17 |
erlehmann |
what i think i gathred |
| 21:17 |
erlehmann |
1. singleplayer games affect the cache. i had not thought of that. |
| 21:18 |
erlehmann |
2. the cache makes a HUGE difference. erroneous cache eviction must not be allowed! |
| 21:18 |
erlehmann |
3. mineclone2/mineclone5/mineclonia … all the same. unless someone uses a different texture pack. |
| 21:19 |
Desour |
media cache loading is no bottleneck, you might be able to achive 100ms less waiting times on servers where there are masses of media, but you still have to wait >0.5s. |
| 21:19 |
erlehmann |
4. by visiting a few servers and having a few singleplayer games you can easily get to 500M |
| 21:19 |
erlehmann |
loading from the cache is not a bottleneck probably, but erroneously evicting it makes everything so much slower |
| 21:20 |
|
Verticen joined #minetest |
| 21:20 |
Desour |
it just makes first joins slow again |
| 21:20 |
erlehmann |
i gather you have never waited minutes for your friend to come and play |
| 21:20 |
Tusha |
I tend to have the same load speeds on servers before/after caching, at least it feels like the same time, as I haven't measured it |
| 21:20 |
erlehmann |
bc they have to dl the media |
| 21:21 |
Tusha |
I have a good speed, yet it takes about 20 minutes to get into the server |
| 21:21 |
Desour |
dl? |
| 21:21 |
erlehmann |
if anyone would ask me, i would set the cache eviction threshold at about 1G disk space. |
| 21:21 |
Tusha |
Internet speed that is |
| 21:21 |
erlehmann |
download |
| 21:21 |
Desour |
I don't want 1G of media if half of it is unused |
| 21:22 |
erlehmann |
nah because it will be used |
| 21:22 |
erlehmann |
i just figured that out |
| 21:22 |
Desour |
it will not |
| 21:23 |
erlehmann |
look any worlds folder is *easily* easily bigger than that if you play for a few months |
| 21:23 |
erlehmann |
way bigger |
| 21:23 |
Desour |
I join at servers, play some days, and never come back for months or years => cache for that server not used |
| 21:23 |
erlehmann |
oh i think you misunderstand how the caching works |
| 21:24 |
Desour |
pls explain what I'm musinderstanding |
| 21:24 |
erlehmann |
as you can see in my notes above, if the same asset is used locally or on another server, you profit from it |
| 21:24 |
Desour |
not if it's a server with some texture pach, ie. fussels's server |
| 21:24 |
erlehmann |
like, all the mineclone2/mineclone5/mineclonia stuff was only one single blob except for clamity using a different texture pack and music |
| 21:24 |
Desour |
pack* |
| 21:25 |
erlehmann |
as i said, you can combine strategies, like: “prune oldest files when 1G is hit and also prune all files older than a year” |
| 21:26 |
erlehmann |
and by older, i mean “not used in” |
| 21:26 |
erlehmann |
and by oldest i mean “not used for the longest time” |
| 21:26 |
erlehmann |
Desour, would that satisfy you? |
| 21:28 |
Desour |
I do like the twice-expiration though |
| 21:28 |
Desour |
but, a combination allowing both would of course satisfy me |
| 21:29 |
erlehmann |
i think the twice-expiration is a weird idea that probably performs in non-obvious ways |
| 21:29 |
Desour |
(if the threshold is configurable, it's basically possible to just turn it off) |
| 21:29 |
erlehmann |
like i sometimes do not play minetest for a while, like months |
| 21:29 |
erlehmann |
or do not connect to a server for a while |
| 21:30 |
Desour |
it's not that weird imo. I'm not sure where, but I think I've seen a similar thing elsewhere already |
| 21:30 |
erlehmann |
well the thing is, i like the “solution fits in head” algorithms approach |
| 21:31 |
Desour |
yes, and it does fit in the head |
| 21:31 |
erlehmann |
mechanically yes. but i can't really reason about it. |
| 21:31 |
erlehmann |
i mean, i can, if i put effort in it |
| 21:32 |
erlehmann |
but compared to “take out the trash if the cache is 1G” it is harder |
| 21:32 |
|
behalebabo joined #minetest |
| 21:34 |
erlehmann |
in general, for size based eviction the threshold must be comfortably high with safety margin |
| 21:35 |
erlehmann |
bc otherwise you can connect to 5 servers and play 3 different singleplayer games and if say, the cache was 250M then it is basically evicted all the time |
| 21:35 |
erlehmann |
which would undeniably suck |
| 21:37 |
Desour |
well, you can easily get some guarantees from this size-too-big approach: the upper size limit. but from the twice-expire time-based approach you can also get some (more complicated to formulate) guarantees: unused files are removed after some time if you were active in this time |
| 21:38 |
Desour |
(I prefer the guaranteed deletion of old stuff) |
| 21:42 |
erlehmann |
as i told you, just run all of the strategies lol |
| 21:42 |
Desour |
btw. about the compression thing: using the file type for this has the downside that the client can't decide to compress the file independently from the server |
| 21:42 |
erlehmann |
look if you recompress stuff you don't gain much space |
| 21:42 |
erlehmann |
and you lose the caching |
| 21:43 |
Desour |
the sha1 digests in the db are meant to be over the uncompressed data |
| 21:44 |
Desour |
recompressing obviously doesn't make much sense. but old servers send ie. translation files uncompressed |
| 21:45 |
erlehmann |
i am not trying to be mean here … but i do have the suspicion that you are trying to solve a problem that is not really big enough to warrant attention with the recompression approach |
| 21:46 |
erlehmann |
meanwhile, i should probably run “optipng -o7 -strip all” on mineclonia |
| 21:46 |
Desour |
yeah, I don't have a strong feeling about the compression thing. it doesn't hurt though |
| 21:47 |
erlehmann |
it does |
| 21:47 |
erlehmann |
code is a liability |
| 21:47 |
erlehmann |
or to say it more brazen: deleted code is debugged code |
| 21:47 |
erlehmann |
the best code is code not written |
| 21:48 |
erlehmann |
brazenly |
| 21:48 |
erlehmann |
it's an adverb |
| 21:48 |
Desour |
yes, but it costs nearly no code |
| 21:48 |
erlehmann |
it costs code, CPU time and RAM |
| 21:49 |
erlehmann |
the space/time tradeoff is not good |
| 21:49 |
erlehmann |
you are better off convincing mod authors to compress their stuff well |
| 21:50 |
erlehmann |
disk-space and traffic savings-wise |
| 21:50 |
Tusha |
I don't think experimenting is a bad thing, prototypes are always neccesary to advance |
| 21:50 |
Tusha |
Also there is the experience gain in the process |
| 21:51 |
erlehmann |
we have a workable model of caching resources already, it is in web browsers |
| 21:51 |
erlehmann |
25 years of experience |
| 21:51 |
erlehmann |
i posit it makes sense to read the existing literature on that *first* and then figure out if anything is missing |
| 21:52 |
erlehmann |
except, of course, if it is a thing of having enlightenment |
| 21:52 |
Tusha |
Well I was talking about the experience the dev gets from trying it, even if it isn't the best approach |
| 21:52 |
erlehmann |
i wrote a quine without cheating |
| 21:52 |
Tusha |
lol |
| 21:52 |
erlehmann |
here |
| 21:52 |
erlehmann |
#!/usr/bin/rc |
| 21:52 |
erlehmann |
p='''';q='#!/usr/bin/rc |
| 21:52 |
erlehmann |
p=%s;q=%s;printf $q $p$p$p$p $p$q$p |
| 21:52 |
erlehmann |
';printf $q $p$p$p$p $p$q$p |
| 21:52 |
erlehmann |
the rc shell is a funny thing |
| 21:54 |
Tusha |
Just trying to gasp zsh over sh or bash, and why it could be an alternative, never heard about rc shell |
| 21:54 |
erlehmann |
the rc shell is the standard shell in plan9 and solves a bunch of things that always bugged me in sh/bash/zsh |
| 21:54 |
erlehmann |
it has a very small grammar too |
| 21:55 |
erlehmann |
i suggest to try it if you are looking for something that has very little historical warts |
| 21:55 |
erlehmann |
(plan9 design was mostly sound … and incompatible with other stuff) |
| 21:56 |
|
est joined #minetest |
| 21:57 |
Tusha |
Anything is better than powershell, at least for me xD... |
| 22:04 |
|
independent56 joined #minetest |
| 22:23 |
|
olliy joined #minetest |
| 22:46 |
|
Verticen joined #minetest |
| 22:52 |
|
erlehmann joined #minetest |
| 23:02 |
|
est joined #minetest |
| 23:26 |
|
independent56 joined #minetest |
| 23:29 |
|
search_social joined #minetest |
| 23:30 |
|
independent56 joined #minetest |
| 23:43 |
|
illwieckz joined #minetest |
