| Time |
Nick |
Message |
| 00:41 |
|
Grandolf joined #minetest-hub |
| 01:14 |
|
Grandolf_ joined #minetest-hub |
| 01:22 |
|
Grandolf joined #minetest-hub |
| 01:28 |
|
Grandolf_ joined #minetest-hub |
| 01:45 |
|
Grandolf joined #minetest-hub |
| 01:45 |
|
Grandolf joined #minetest-hub |
| 01:59 |
|
paramat joined #minetest-hub |
| 02:58 |
|
lhofhansl left #minetest-hub |
| 03:04 |
|
ThomasMonroe joined #minetest-hub |
| 03:15 |
|
CalebDavis joined #minetest-hub |
| 03:23 |
|
bwarden joined #minetest-hub |
| 03:34 |
|
paramat left #minetest-hub |
| 04:02 |
|
bwarden joined #minetest-hub |
| 05:30 |
|
lhofhansl joined #minetest-hub |
| 05:30 |
|
lhofhansl left #minetest-hub |
| 05:30 |
|
lhofhansl joined #minetest-hub |
| 06:24 |
|
ChainLynxMobile joined #minetest-hub |
| 07:01 |
|
CWz joined #minetest-hub |
| 07:07 |
|
Dargod joined #minetest-hub |
| 07:36 |
|
Dargod joined #minetest-hub |
| 08:10 |
|
nore joined #minetest-hub |
| 08:19 |
CWz |
just did something stupid. i made a texturepack for myself that replaces steel axe,pick and shovel with bongs. newbs are now stoners |
| 08:25 |
|
coyote1 joined #minetest-hub |
| 08:51 |
|
nerzhul joined #minetest-hub |
| 09:57 |
|
coyote2 joined #minetest-hub |
| 10:45 |
|
Fixer joined #minetest-hub |
| 10:58 |
|
IhrFussel joined #minetest-hub |
| 11:41 |
|
Megaf joined #minetest-hub |
| 11:41 |
Megaf |
Heya |
| 11:41 |
Megaf |
Just installed ircloud here again |
| 11:41 |
Megaf |
Let's see if it's any good |
| 11:49 |
IhrFussel |
Okay guys I ran my modified mod profiler for 3 days now and *not even once* I saw 1 lag-entry that could come even close to the max_lag the server reports when a few more players play...the highest I saw was 720 ms caused by mod zombies but that was very rare...most "lag mods" caused 3-5 ms lag which would never explain 2+ secs max_lag |
| 11:51 |
sfan5 |
the lag can also be caused inside C++ code |
| 11:52 |
IhrFussel |
How can the engine lag for 2 seconds outside of Lua? |
| 11:54 |
Megaf |
Speaking of mods, do we have a tool that will load the whole map, like clear objects does, and remove all unknown objects? |
| 12:17 |
IhrFussel |
99.9% certain it's not the mods now... I disabled all my mods (expect areas + MTG) and the max_lag peaks at 1.7 secs with 3 players |
| 12:18 |
IhrFussel |
except* |
| 12:22 |
shivajiva |
how many players are registered in the auth file? |
| 12:26 |
shivajiva |
if you have 30k of players registered then you are searching 30k of players for privilege validation for 3 players and it's several times a second, continuously whilst they are logged in... |
| 12:27 |
shivajiva |
this adds lag despite tables being relatively fast in lua |
| 12:27 |
Megaf |
Oh |
| 12:27 |
Megaf |
How do we clean the Auth file? |
| 12:27 |
shivajiva |
ofc sauth mod makes it reasonable by only caching the players logged in |
| 12:29 |
shivajiva |
Megaf you need to compare the entries in auth against player files and remove entries from the auth that didn't make it to the player file stage |
| 12:29 |
shivajiva |
and then you can do a time based cull |
| 12:30 |
Megaf |
My server sure have several thousands of players registered |
| 12:30 |
shivajiva |
or shift to sauth and player db and use a little query to fix the orphaned entries and never suffer auth lag again |
| 12:31 |
shivajiva |
skyblocks goes down to 0.1 lag with sauth running, used to be 5 seconds |
| 12:32 |
shivajiva |
>200k of entries in the auth file was bogging it down |
| 12:32 |
shivajiva |
I haven't even purged the orphans to get that kind of response |
| 12:32 |
shivajiva |
:P |
| 12:33 |
shivajiva |
also, use sban and stop loading all the banned players into memory |
| 12:33 |
|
lisac joined #minetest-hub |
| 12:34 |
shivajiva |
correction, all the players it's ever seen! |
| 12:35 |
shivajiva |
xban2 file normally corrupts at around the 6MB size |
| 12:36 |
|
Jordach joined #minetest-hub |
| 12:36 |
shivajiva |
people use databases when they are available to prevent corruption! |
| 12:38 |
shivajiva |
I took the knowledge gained from running a server for 18 months and made 2 mods that fix the biggest issues a server owner faces... |
| 12:40 |
* shivajiva |
packs his soapbox up and saunters off |
| 12:44 |
shivajiva |
~950 privilege checks for a player just to get into the game... |
| 12:44 |
IhrFussel |
shivajiva, I guess it's time to try sauth |
| 12:44 |
shivajiva |
pls ;) |
| 12:46 |
Jordach |
shivajiva, check your privilege |
| 12:46 |
Jordach |
Kappa |
| 12:46 |
shivajiva |
? |
| 12:46 |
Jordach |
being sarcastic about SJWs :^) |
| 12:46 |
shivajiva |
lol :P |
| 12:46 |
IhrFussel |
I installed lsqlite3 via luarocks and moved sauth-master to worldmods/sauth ... is that all I need? |
| 12:47 |
shivajiva |
yes that should be all that is required |
| 12:48 |
shivajiva |
if the record set is big it would bbe better to do it locally and copy the db to the server though |
| 12:48 |
shivajiva |
>30k of entries forces a dump file you have to import |
| 12:49 |
shivajiva |
trivial still but if you don't play with db's it may seem a little daunting |
| 12:49 |
shivajiva |
and it will require the installation of sqlite3 for the import |
| 12:49 |
shivajiva |
no biggie |
| 12:51 |
shivajiva |
if anyone wants me to do the conversion just pm me with a link to the file and I'll process it and link you back |
| 12:51 |
shivajiva |
80MB import takes less than a minute |
| 12:52 |
shivajiva |
thank goodness for bulk import statements :) |
| 13:00 |
|
coyote1 joined #minetest-hub |
| 13:21 |
IhrFussel |
shivajiva, sauth just created a new DB with only 1 entry.. I see no import file either |
| 13:26 |
shivajiva |
hmm how many entries are in the auth file? |
| 13:27 |
IhrFussel |
I have a auth.sql file (7.7 MB) and a sauth.sqlite (4.9 KB) ... the auth.txt has been deleted (by the mod?) but it had ~ 18,000 entries |
| 13:27 |
shivajiva |
auth.txt was renamed to auth.txt.bak |
| 13:28 |
IhrFussel |
Ok yeah that happened so it's correct? |
| 13:28 |
shivajiva |
so it should have imported directly with only 18k of entries |
| 13:28 |
IhrFussel |
Well first time I started the server it shutdown itself after 10 secs |
| 13:28 |
Fixer |
mt engine: 28 notification, mtg: 0 |
| 13:29 |
shivajiva |
the sql file is auth.sql, that needs importing fussel |
| 13:30 |
shivajiva |
hnce the small db and the large import file |
| 13:30 |
shivajiva |
+e |
| 13:30 |
IhrFussel |
shivajiva, but the auth.sql already exists and it's several MB so I guess it worked? |
| 13:30 |
IhrFussel |
Ohh .sql is the import file? |
| 13:30 |
shivajiva |
yea |
| 13:30 |
shivajiva |
sauth.sqlite is the db |
| 13:31 |
IhrFussel |
Ok what's the command to import it now? |
| 13:31 |
shivajiva |
have you installed sqlite3? |
| 13:32 |
shivajiva |
once you have the commands are:- |
| 13:32 |
shivajiva |
sqlite3 |
| 13:33 |
IhrFussel |
Yes...so just sqlite3 sauth.sqlite3 < auth.sql ? |
| 13:33 |
IhrFussel |
sqlite* |
| 13:33 |
shivajiva |
that will suffice, personally I open the db and read the auth.sql |
| 13:34 |
shivajiva |
you should delete the sauth.sqlite file first |
| 13:34 |
shivajiva |
otherwise you will get a duplicate entry for yourself |
| 13:35 |
IhrFussel |
That took 2 seconds...is that right? o.o |
| 13:35 |
shivajiva |
how big is the sqlite file now? |
| 13:35 |
IhrFussel |
8.7 MB |
| 13:35 |
shivajiva |
perfect :) |
| 13:37 |
shivajiva |
only the sban import reduces the data size because of the layout used by xban2 for readable saving |
| 13:39 |
shivajiva |
sqlite is pretty fast, that's why I chose it. Tests showed <5ms access time for a record pulled from the db and it's cached after that till the player leaves |
| 13:40 |
IhrFussel |
shivajiva, unfortunately that's not the source of the problem but I'll leave it like that anyway...max_lag just jumped to 1.4 secs with me ALONE |
| 13:41 |
shivajiva |
okay but loading has an effect that needs time to settle, more importantly will be the lag figure in 5 mins |
| 13:42 |
shivajiva |
when the server start event is a distant reflection on the max_lag figure |
| 13:43 |
shivajiva |
despite the stated value does it feel more responsive? |
| 13:44 |
IhrFussel |
I guess the best way to check is switching privs? |
| 13:47 |
shivajiva |
I noticed teleporting around was quicker and obviously I have a gamehub mod running that instantly became more responsive in handling events |
| 13:48 |
shivajiva |
sauth runs on Darklands and dropped the lag from 7s down to 1.5s |
| 13:51 |
shivajiva |
it runs both sauth and sban though so it recovered about 66MB of lua memory |
| 13:51 |
shivajiva |
from the change over |
| 13:54 |
shivajiva |
heh your server is heavy on the media |
| 13:54 |
IhrFussel |
shivajiva, the lag only goes up when I move quickly on the map...when I idle for a bit the max_lag goes <0.5 |
| 13:55 |
|
coyote1 joined #minetest-hub |
| 13:57 |
|
Calinou joined #minetest-hub |
| 14:09 |
|
coyote1 left #minetest-hub |
| 14:25 |
|
ThomasMonroe joined #minetest-hub |
| 14:41 |
shivajiva |
IhrFussel: grant invokes core.get_auth_handler().get_auth(grantname) which does use the registered handler so it's not that |
| 14:41 |
IhrFussel |
shivajiva, I use minetest.player_exists now is that safe? |
| 14:42 |
|
oOChainLynxOo joined #minetest-hub |
| 14:42 |
IhrFussel |
shivajiva, 0.4.16 stable chatcommands.lua uses core.auth_table |
| 14:43 |
shivajiva |
player_exists should work fine |
| 14:43 |
|
CWz joined #minetest-hub |
| 14:43 |
IhrFussel |
Or maybe not...maybe my chatcommand.lua is older than 0.4.16 |
| 14:44 |
shivajiva |
yea I'm in 0.4-stable and chatcommands.lua is using core.get_auth_handler().get_auth(grantname) for grant |
| 14:45 |
|
CalebDavis joined #minetest-hub |
| 14:47 |
shivajiva |
shows no use of core.auth_table in that file |
| 14:48 |
IhrFussel |
minetest.auth_table seems to be very rarely used by mods...only areas used it (changed now) |
| 14:49 |
shivajiva |
only other example of it was in the border mod |
| 14:51 |
|
CWz joined #minetest-hub |
| 14:53 |
shivajiva |
I'll do some testing but I'm sure I checked /grant, maybe you modified your version? |
| 14:57 |
|
Jordach joined #minetest-hub |
| 14:59 |
shivajiva |
did you grep for auth_table to see where it's used on your server? |
| 14:59 |
IhrFussel |
"maybe my chatcommand.lua is older than 0.4.16" |
| 15:00 |
IhrFussel |
I probably didn't use the one from 0.4.16 cause I modified quite a bit in it |
| 15:00 |
shivajiva |
yea something isn't quite right about your setup in terms of what's expected at that revision |
| 15:01 |
rubenwardy |
it's generally a bad idea to modify default |
| 15:01 |
shivajiva |
^ |
| 15:01 |
rubenwardy |
baring in mind it's possible to override chat commands anyway... |
| 15:01 |
shivajiva |
^^ |
| 15:02 |
rubenwardy |
minetest.registered_chatcommands["name"].func = function(...) end |
| 15:02 |
rubenwardy |
ops |
| 15:02 |
rubenwardy |
*modify builtin |
| 15:03 |
shivajiva |
were in chatcommands.lua so we both knew what section you were referring to ruben ;) |
| 15:04 |
rubenwardy |
:D |
| 15:04 |
IhrFussel |
I modified a bit than just the commands...I modified the overall chat message handling since I use an external bot and want its commands to be accepted |
| 15:04 |
IhrFussel |
A bit more* |
| 15:04 |
rubenwardy |
what does that mean? |
| 15:05 |
IhrFussel |
I use a bash bot for my server and want its commands to be valid ingame |
| 15:05 |
rubenwardy |
what |
| 15:05 |
rubenwardy |
you run bash commands in game? |
| 15:05 |
shivajiva |
chat handling is setup so you can terminate or pass forward, that covers your needs for all chat commands |
| 15:05 |
IhrFussel |
No the bot runs in the background and listens to the ingame chat log and executes functions based on commands it finds |
| 15:06 |
rubenwardy |
what kind of commands? |
| 15:06 |
rubenwardy |
this sounds awfully a lot like a remote code execution vulnerability |
| 15:06 |
shivajiva |
join/part/level etc |
| 15:07 |
IhrFussel |
Lots...it has 150+ commands ... no players cannot just execute code, the bot will add the actual command/program by itself |
| 15:07 |
|
behaleba- joined #minetest-hub |
| 15:08 |
IhrFussel |
Like when a player enters something the bot executes cat something for example or if [[ -f something ]] |
| 15:08 |
shivajiva |
what you are doing can be done by intercepting the chat stream and passing it on after you have checked it against your list of commands |
| 15:09 |
shivajiva |
unless they alter the behaviour ofc |
| 15:09 |
IhrFussel |
Lua is painfully slow to type and lots of the bot commands need to check for file names/contents... that is just easier to write in bash $(cat $file) [[ -f $file ]] etc |
| 15:10 |
rubenwardy |
errr... ssh? |
| 15:11 |
|
atorian37 joined #minetest-hub |
| 15:16 |
rubenwardy |
btw, IhrFussel, I saw in the log you're suffering from performance issues. As well installing sauth, I recommend doing the following: |
| 15:16 |
rubenwardy |
cd worlds/world |
| 15:16 |
rubenwardy |
sudo inotifywait -e modify -e attrib -e move -e create -e delete -m -r . |
| 15:16 |
rubenwardy |
it tells you which files are being modified as they happen |
| 15:17 |
rubenwardy |
so you can see if it's due to file i/o the server is freezing |
| 15:17 |
rubenwardy |
(you can also tell if a server is freezing from the packets on f5 btw) |
| 15:17 |
rubenwardy |
but sauth helps a lot for server |
| 15:17 |
rubenwardy |
*servers |
| 15:18 |
rubenwardy |
oh, you'll need to do sudo apt install inotify-tools or smth |
| 15:30 |
|
sniper570 joined #minetest-hub |
| 15:52 |
shivajiva |
IhrFussel: is your disk HDD or SHD or SSD? |
| 15:53 |
IhrFussel |
rubenwardy, with grep -v sqlite it's basically just debug.txt that is modified a lot...there are some other small files here and there every 30-60 secs |
| 15:53 |
rubenwardy |
are you using the players sqlite DB? |
| 15:54 |
rubenwardy |
I think there's a bug with it |
| 15:54 |
rubenwardy |
it was causing massive lockups on my server |
| 15:54 |
shivajiva |
:O |
| 15:54 |
rubenwardy |
which it shouldn't with 15 players online |
| 15:54 |
rubenwardy |
like, updating 15 records shouldn't take very long aha |
| 15:55 |
rubenwardy |
I made an issue |
| 15:55 |
IhrFussel |
Nope still using files and shivajiva it's a regular HDD ... those lag spikes are not normal... with 3 players spiking to 3+ secs once per minute or so...something is completely wrong there |
| 15:55 |
rubenwardy |
would be code to have it independently verified |
| 15:57 |
shivajiva |
wonder if we can persuade someone to verify it who has reasonable traffic |
| 16:00 |
shivajiva |
trouble I have with this is skyblocks sits at 0.1 lag with 3 players in and for over an hour, it's nothing but a vanilla stable-0.4 with about 30 mods |
| 16:01 |
rubenwardy |
my server has been having block loading issues recently, I reckon it's due to some max packet send though |
| 16:01 |
rubenwardy |
MT is no where near my max CPU or network |
| 16:02 |
shivajiva |
same for Fussel, mt is taking hardly any resources 8% CPU 20% memory and it's still spiking up to 3s |
| 16:03 |
rubenwardy |
odd |
| 16:03 |
rubenwardy |
what about hard drive usage? |
| 16:03 |
shivajiva |
that's why he's obsessed with tracing it |
| 16:03 |
IhrFussel |
max_lag 3.6 secs with 11 players 28% load, 13% CPU and 16% RAM |
| 16:03 |
rubenwardy |
also, are map blocks read asynchronously? |
| 16:04 |
shivajiva |
you'd hope so |
| 16:04 |
rubenwardy |
there's emerge threads, so yeah, I'd hope so |
| 16:04 |
rubenwardy |
not gone into that area yet |
| 16:05 |
IhrFussel |
Is max_packets_per_iteration = 512 maybe too low? |
| 16:07 |
rubenwardy |
is that a server cap? |
| 16:07 |
IhrFussel |
I set server_unload_unused_data_timeout to 900 in hope that the server would cache mapblocks better |
| 16:07 |
IhrFussel |
Nope I just saw that somewhere and applies it |
| 16:08 |
IhrFussel |
I guess I will have to create a new test world with all mods and let it run for a bit to see if it still spikes...if no then the map DB is the problem |
| 16:08 |
shivajiva |
skyblocks was running with max_simultaneous_block_sends_server_total = 10000 |
| 16:12 |
shivajiva |
when moving fast as the chunks load it produces a little stutter but nothing that would spoil the game play |
| 16:12 |
Fixer |
hmm, what is this youtube insanity with 1280*738? resolution? scales like shit, everything is blurry, wth? |
| 16:12 |
IhrFussel |
Or maybe I had the map_save_interval too high? (60) |
| 16:13 |
shivajiva |
what's the default for the save interval? |
| 16:14 |
IhrFussel |
15 I think |
| 16:18 |
IhrFussel |
Maybe if one sets it too high it writes too many mapblocks at once |
| 16:18 |
shivajiva |
that could stack it up as you are potentially writing x4 times slower, with x4 the data to save, wonder if they are done as a series of single transactions |
| 16:19 |
shivajiva |
you're just trying to drag me into the core code :P |
| 16:21 |
IhrFussel |
I set it to 20...will see if it's better after restart |
| 16:21 |
shivajiva |
excellent :) |
| 16:32 |
Calinou |
hi |
| 16:34 |
shivajiva |
Hi Calinou |
| 16:42 |
|
IhrFussel joined #minetest-hub |
| 16:44 |
|
ThomasMonroe joined #minetest-hub |
| 17:27 |
|
paramat joined #minetest-hub |
| 17:53 |
|
Darcidride_ joined #minetest-hub |
| 17:55 |
|
nerzhul joined #minetest-hub |
| 18:00 |
IhrFussel |
inotifywait shows 20-30 MODIFY map.sqlite at once every few secs...is that normal? |
| 18:06 |
IhrFussel |
paramat, sauth requires lsqlite though...do you plan to bundle that? |
| 18:06 |
rubenwardy |
yes |
| 18:06 |
rubenwardy |
I wonder if you can compile it in |
| 18:06 |
rubenwardy |
as a Lua module |
| 18:06 |
|
Fixer joined #minetest-hub |
| 18:06 |
rubenwardy |
but yeah, it needs to work without Lua rocks or any system hackery |
| 18:07 |
IhrFussel |
Yes regarding my question or lsqlite? |
| 18:07 |
IhrFussel |
My first question* |
| 18:08 |
rubenwardy |
lsqlite |
| 18:08 |
rubenwardy |
regarding sqlite, that's not great. It may be part of the libsqlite rather than what we're doing |
| 18:08 |
rubenwardy |
ie: async staggered writes |
| 18:10 |
IhrFussel |
It also creates map.sqlite-journal a lot ... too many requests? |
| 18:11 |
|
Darcidride_ joined #minetest-hub |
| 18:11 |
rubenwardy |
same happens with players.sqlite, but suspiciously often |
| 18:13 |
IhrFussel |
Here the last 500 or so entries https://pastebin.com/Cvs6ftSn |
| 18:14 |
IhrFussel |
Okay more like last 1,900 xP ... I also added the date |
| 18:15 |
rubenwardy |
maybe try postgresql? You can always go back |
| 18:15 |
rubenwardy |
make sure to keep a backup of the sqlite too |
| 18:16 |
rubenwardy |
actually, does the migrate work all ways? Thought it does |
| 18:16 |
rubenwardy |
nerzhul would be able to answer :P |
| 18:22 |
nerzhul |
use postgresql for a good long term server relational database yes |
| 18:23 |
IhrFussel |
Will that reduce the DB reads/writes? Do I need to recompile MT for postgresql? |
| 18:23 |
rubenwardy |
I'd to investigate this and do benchmarks at some point |
| 18:24 |
nerzhul |
IhrFussel, postgresql will do less write but better write too, and MT doesn't care about those and is faster because it's not done by itself |
| 18:24 |
rubenwardy |
my intuition is that a key-value database like LevelDB or Mongo would be better for our inherently non-relational data, but that doesn't seem to be the case. |
| 18:24 |
nerzhul |
rubenwardy, postgresql on my machine can go 1M transactions per second, try to do that with sqlite :p |
| 18:24 |
rubenwardy |
so I |
| 18:24 |
nerzhul |
mongo is shit and we have relational data :) |
| 18:24 |
rubenwardy |
maps aren't though |
| 18:24 |
nerzhul |
binaryblob is just a field |
| 18:25 |
rubenwardy |
it's a key - position hash - and a value - binary blob |
| 18:25 |
nerzhul |
do you want to query inside the mapblock ? no ? then nosql is useless |
| 18:25 |
rubenwardy |
no relation ships |
| 18:25 |
nerzhul |
relation is the primary key: postition |
| 18:25 |
IhrFussel |
Does postgresql require a certain compile flag or does it just work when I enter the info into world.mt ? |
| 18:25 |
rubenwardy |
IhrFussel, you need to compile with postgresql-dev installed on the system |
| 18:25 |
rubenwardy |
no flags |
| 18:25 |
nerzhul |
IhrFussel, regular postgresql, just tune shared_buffer value to 512MB or 1GB if you have 2GB ram |
| 18:25 |
rubenwardy |
just need to have the header install |
| 18:25 |
rubenwardy |
+ed |
| 18:26 |
nerzhul |
and use PostgreSQL 9.5 at least, 10 is prefered |
| 18:27 |
IhrFussel |
rubenwardy, wait...I install postgresql-dev (name of ppa?) then I compile the engine again with the EXACT same flags as usual and it will just work? |
| 18:28 |
rubenwardy |
yes, but you also need to do a migration |
| 18:28 |
rubenwardy |
I'm not sure how |
| 18:28 |
rubenwardy |
nerzhul will be able to give more detail |
| 18:28 |
rubenwardy |
but you'll need postgresql login details somewhere |
| 18:30 |
IhrFussel |
Too many packages https://pastebin.com/SdGCFeD1 |
| 18:31 |
rubenwardy |
ha |
| 18:31 |
rubenwardy |
nerzhul? |
| 18:31 |
rubenwardy |
:D |
| 18:31 |
rubenwardy |
I'm guessing postgresql-9.5-dev, but it'd be nice to have 10 |
| 18:32 |
rubenwardy |
what that isn't a thing |
| 18:32 |
rubenwardy |
heh |
| 18:32 |
Calinou |
you can probably get a PPA or something, to get latest PostgreSQL on Linux |
| 18:36 |
nerzhul |
there is apt.postgresql.org official repository |
| 18:39 |
IhrFussel |
Now there's even more packages https://pastebin.com/WY2dYphU |
| 18:41 |
Calinou |
there's a v10 now :) |
| 18:41 |
IhrFussel |
But which do I need? server-dev? |
| 18:42 |
IhrFussel |
10 doesn't even have a server-dev |
| 18:47 |
Calinou |
IhrFussel: postgresql-server-dev-10 |
| 18:47 |
Calinou |
it's the other say around :) |
| 18:47 |
Calinou |
way* |
| 18:48 |
IhrFussel |
Is that the one I need?? |
| 18:48 |
Calinou |
IhrFussel: probably :) |
| 18:51 |
IhrFussel |
I'm asking #postgresql ... maybe they know |
| 18:56 |
|
Aerozoic joined #minetest-hub |
| 19:00 |
paramat |
rubenwardy could this CSM mod be disruptive to servers? https://forum.minetest.net/viewtopic.php?f=53&t=18800 it does say 'kicks you if you are trying that "something" online' but that kick could easily be removed |
| 19:04 |
Fixer |
*,,,,,* |
| 19:05 |
Shara |
Sounds like a way to spy on contents of containers you are not meant to see into |
| 19:06 |
sofar |
he kicks the player if they use it outside of singleplayer mode |
| 19:06 |
sofar |
lol |
| 19:06 |
Shara |
Yay for CSM? |
| 19:07 |
sofar |
even without that kick, this is fairly harmless |
| 19:07 |
Fixer |
paramat: is not it is pretty much the same as chest inspector? or worse? |
| 19:07 |
sofar |
it's a generic version of chest inspector, yes |
| 19:07 |
Shara |
Harmless maybe, but still unwanted as far as I'm concerned |
| 19:08 |
Shara |
What if you set up a game wher ea player can select one chest to open for a reward? |
| 19:08 |
sofar |
ha |
| 19:08 |
sofar |
just make a chest that doesn't store the content in nodemeta |
| 19:09 |
Shara |
Shouldn't need to make workarounds like that though |
| 19:09 |
Shara |
If chest content was meant to be seen by everyone it should just be visible to everyone |
| 19:09 |
sofar |
it essentially is visible to everyone |
| 19:10 |
sofar |
you can mark fields in nodemeta as private and then this csm mod is defeated |
| 19:10 |
sofar |
that mechanism exists |
| 19:10 |
Shara |
The ability to do such things has fewer and fewer barrier though. That's the issue. |
| 19:10 |
shivajiva |
+1 for chest contents stored in db only retrievable by the owner |
| 19:12 |
paramat |
well see my reply, the kick code can be removed, and this would be fairly useless in singleplayer, plus it's marked as a 'cheat' |
| 19:14 |
Shara |
There is clearly no use for it in singleplayer unless you have copied a map from somewhere |
| 19:14 |
paramat |
doesn't seem harmless |
| 19:14 |
Shara |
Even then, you'd have protection_bypass |
| 19:14 |
Shara |
So this would maybe only help with mods having containers that don't respect that, at best |
| 19:15 |
paramat |
looks like a server cheat mod carefully created to get around the rules |
| 19:15 |
Shara |
Yup |
| 19:16 |
shivajiva |
just a thought but shouldn't we be heavily biased towards db storage, not just for the map but everything. Seems there must be some issue with that scenario given we are so far down the line |
| 19:17 |
IhrFussel |
Okay I pretty much can confirm now that the ingame lags are caused by the DB... whenever the engine modifies map.sqlite it lags 1-2 secs right now |
| 19:18 |
shivajiva |
probably not noticeable on SDD |
| 19:18 |
shivajiva |
try levelDb ? |
| 19:18 |
Shara |
leveldb didn't help at all on Dark lands |
| 19:19 |
Shara |
I'm still pretty sure it made things worse |
| 19:20 |
shivajiva |
could you quantify in what way it affected the performance or was it just a further deterioration? |
| 19:21 |
shivajiva |
*SSD |
| 19:21 |
Shara |
Well, it didn't seem to make anything better |
| 19:21 |
IhrFussel |
shivajiva, it didn't always happen...the DB isn't small so maybe that takes longer? |
| 19:22 |
IhrFussel |
map.sqlite is 17 GB now |
| 19:22 |
shivajiva |
wow |
| 19:22 |
Shara |
Fussel, that's comparable to RC in size |
| 19:22 |
Shara |
I have an SSD though |
| 19:24 |
IhrFussel |
That minetest modifies the DB 20+ times in a row seems very inefficient to me |
| 19:24 |
shivajiva |
I'm disadvantaged by skyblocks as it keeps the map size down and for whatever reason leveldb performs better than sqlite. The map db is only 1.8GB |
| 19:25 |
sofar |
17gb map.sqlite... heh |
| 19:25 |
sofar |
itb's is 30mb lol |
| 19:25 |
shivajiva |
lol |
| 19:25 |
IhrFussel |
I will try server_map_save_interval with 120 and see if the lag goes down more |
| 19:25 |
shivajiva |
wonder what the access time is on a 17 GB sqlite db |
| 19:26 |
Shara |
I think RC is around 16GB now |
| 19:26 |
sofar |
longer delays between writes may make things worse |
| 19:26 |
|
Krock joined #minetest-hub |
| 19:26 |
Shara |
But RC doesn't really have any real lag issues |
| 19:26 |
sofar |
but you may have hit an sqlite scaling problem |
| 19:27 |
|
Megaf joined #minetest-hub |
| 19:27 |
Shara |
I think DL is around the same size, but using leveldb. I do get some lag spikes there if a lot of people join at the same time sometimes, but since adding sauth it's been pretty good |
| 19:30 |
IhrFussel |
time sqlite3 .minetest/worlds/oldserver/map.sqlite "SELECT * from blocks WHERE ROWID='1'" << this query takes 37 ms |
| 19:32 |
shivajiva |
as sofar noted, writing is where it may impact the most |
| 19:34 |
sofar |
37ms is probably 30ms for the exec() overhead already, so it's almost negligeable time |
| 19:34 |
sofar |
you want to try adding a block and see how long that takes |
| 19:34 |
IhrFussel |
sofar, inotifywait displays 20-50 entries within the SAME second though when the server saves the map |
| 19:35 |
sofar |
sure, that's to be expected. With a 17gb file it has to touch various journal blocks. |
| 19:35 |
sofar |
the larger the map is, the more it has to do that too, I bet |
| 19:36 |
sofar |
20-50 isn't even shocking |
| 19:36 |
IhrFussel |
I see lots of journal CREATE MODIFY and DELETE |
| 19:36 |
sofar |
yup |
| 19:36 |
IhrFussel |
Would that explain 1+ sec lag? Sometimes even 5 secs |
| 19:37 |
sofar |
do you have a precise log of those 20-50 entries? with timestamps? |
| 19:37 |
IhrFussel |
https://pastebin.com/Cvs6ftSn |
| 19:39 |
sofar |
not enough time resolution, I need like millisecond time stamps |
| 19:39 |
shivajiva |
can't see from that what if any timeframes there are in between each transaction |
| 19:39 |
IhrFussel |
No problem |
| 19:39 |
IhrFussel |
One sec |
| 19:42 |
Calinou |
<IhrFussel> map.sqlite is 17 GB now |
| 19:42 |
Calinou |
reminds me, weren't there issues with map.sqlite being over 4 GB? |
| 19:44 |
IhrFussel |
sofar, https://pastebin.com/xEYUpAF2 |
| 19:45 |
Shara |
Calinou: Red Cat doesn't seem to mind it. It's been over 4 GB for approaching two years :P |
| 19:47 |
IhrFussel |
MODIFY means write correct? |
| 19:48 |
paramat |
surely a long server_map_save_interval means that when it does it has a huge amount to do, causing freezes? |
| 19:48 |
shivajiva |
MODIFY suggests either update or insert |
| 19:49 |
shivajiva |
that was my take on it paramat |
| 19:49 |
paramat |
the default is 5.3 so that may be fairly optimal |
| 19:49 |
paramat |
120 is huge |
| 19:50 |
sofar |
well it seems to be taking 400+ms in some cases so that's awfully close to 5.3ms (only 1 order off) |
| 19:50 |
shivajiva |
the tuned edge is in keeping it low enough to not interfere with the core processes so the data is minimal |
| 19:50 |
sofar |
5.3s |
| 19:51 |
paramat |
IhrFussel what's your current save interval? |
| 19:51 |
paramat |
or previous |
| 19:52 |
shivajiva |
I think he's tried 60/20/120 now |
| 19:52 |
IhrFussel |
I will try 5.3 then ... paramat I had it at 60 for 2 months or so...now it's at 20...will try 5.3 after restart |
| 19:53 |
Shara |
I recall being told by someone that high was fine, and actually use 60 myself. |
| 19:53 |
Shara |
No issues from it. |
| 19:53 |
Shara |
So if anyone can confirm it should actually be lower, it would be good to know |
| 19:53 |
IhrFussel |
Could server_unload_unused_data_timeout = 900 be the problem? Maybe the longer the server remembers blocks the more it stores per save_interval? |
| 19:54 |
Shara |
RC has that much lower, so maybe |
| 19:54 |
IhrFussel |
I doubt that it's just the save_interval but we'll see in a minute |
| 19:54 |
Shara |
server_unload_unused_data_timeout = 28 here |
| 19:55 |
Shara |
But I won't pretend I fully understand these settings |
| 19:55 |
Shara |
I think it was Ten who advised me to keep that one low back when I had some problems. |
| 19:56 |
shivajiva |
why are these settings not documented? |
| 19:56 |
Shara |
This one says: |
| 19:56 |
Shara |
# How much the server will wait before unloading unused mapblocks. |
| 19:56 |
Shara |
# Higher value is smoother, but will use more RAM. |
| 19:56 |
Shara |
So it kind of is |
| 19:56 |
shivajiva |
to the depth of how to utilise them |
| 19:57 |
Shara |
It would require someone who fully understands them to be willing to write more about them :) |
| 19:58 |
sfan5 |
what's so hard to understand about that setting? (honest question) |
| 19:58 |
shivajiva |
nothing at all about that one |
| 19:58 |
Shara |
sfan5: Boundaries. What would count as too low/too high and so on |
| 19:59 |
sfan5 |
hmm right |
| 19:59 |
shivajiva |
it's obvious that keeping the block loaded increases the chance of it becoming available if the player turns back or another player enters it |
| 19:59 |
Shara |
Yup, but keeping it loading too long can be an issue because then it's a risk of wasted effort |
| 19:59 |
Shara |
But knowing what the right balance is can be really difficult |
| 20:00 |
shivajiva |
specifically the map save was what I was interested in at this point |
| 20:02 |
|
Mr-Pardison joined #minetest-hub |
| 20:02 |
IhrFussel |
I see no real difference...except the game now shows 20-50 MODIFY entries per 5 seconds |
| 20:02 |
Shara |
Hi Mr-Pardison :) |
| 20:03 |
Mr-Pardison |
Hello Shara. |
| 20:03 |
shivajiva |
2 years down the line this stuff is a lot more transparent but even so if information can be added to what we have to make it clearer in terms of boundaries as Shara suggested |
| 20:03 |
IhrFussel |
I wonder why it even has to store that many mapblocks at once...we are 4 players right now and I doubt that 50 mapblocks have been modified |
| 20:03 |
Shara |
It's probably a case of info overload for many people when trying to work out which of all the possible options they need as well. |
| 20:04 |
shivajiva |
^ |
| 20:04 |
sfan5 |
IhrFussel: one modify operation != one mapblock write |
| 20:04 |
paramat |
set server_unload_unused_data_timeout to the default or close, 900 is huge compared to the default |
| 20:04 |
IhrFussel |
default is 300 right? |
| 20:05 |
paramat |
not sure, thout it was around 30 |
| 20:05 |
Shara |
server_unload_unused_data_timeout (Unload unused server data) int 29 |
| 20:05 |
Shara |
So 29? |
| 20:05 |
sfan5 |
300 still sounds pretty high, try something in the range of [90, 120] if you'd like an increased value |
| 20:06 |
IhrFussel |
But that likely won't affect the DB problem...I'll try |
| 20:15 |
IhrFussel |
It looked like it helped... 3 players @ 0.6 secs max_lag but then I ran a bit and 1.7 secs |
| 20:21 |
IhrFussel |
I just added -e access to inotifywait and some accesses take 100 ms |
| 20:24 |
IhrFussel |
I'm pretty sure ACCESS time is the problem |
| 20:29 |
Megaf |
Folks, #Minetest-Mods is it closed for good or actually just for invited? |
| 20:29 |
Mr-Pardison |
invite only. |
| 20:29 |
Mr-Pardison |
just checked. |
| 20:30 |
* Megaf |
quietly roads at Shara |
| 20:30 |
Megaf |
Mr-Pardison: Channel is empty, not even ChanServ is there |
| 20:30 |
Mr-Pardison |
huh. |
| 20:30 |
Megaf |
I do believe the channel is closed for good |
| 20:31 |
* Shara |
wonders if that means a road dropped on her |
| 20:31 |
Megaf |
when you join, Chanserv joins and kicks you |
| 20:31 |
Mr-Pardison |
yeah. |
| 20:31 |
Megaf |
Shara: I eman |
| 20:31 |
Shara |
I still don't get why that channel is closed |
| 20:31 |
* Megaf |
quietly roars at Shara |
| 20:31 |
Shara |
Even if it would not be hugely busy, is there any harm in opening that channel? |
| 20:31 |
Megaf |
Shara: Yep, I think that channel should be open and we should be there... |
| 20:32 |
Mr-Pardison |
Now that I'm speaking, mind if I bring up this: https://github.com/minetest-mods/technic/issues/375 ? |
| 20:32 |
Shara |
I'm quite happy to talk about mods in here as well, but some people seem to want a dedicated place for it, so it may as well be an official one |
| 20:34 |
Megaf |
sfan5: celeron55: Hi gentlemen, a word on Minetest-Mods please? |
| 20:34 |
sfan5 |
i'm not involved in that |
| 20:35 |
sfan5 |
however i don't see the need for yet another channel to talk about something minetest-related |
| 20:35 |
Megaf |
Chanserv says otherwise |
| 20:35 |
Megaf |
Information on #minetest-mods: |
| 20:35 |
Megaf |
Founder : celeron55, sfan5 |
| 20:35 |
Shara |
sfan5: we've got people making their own channels for it instead. |
| 20:36 |
Shara |
DOes it do any harm if it's not busy? |
| 20:36 |
sfan5 |
Megaf: huh good to know that i'm marked as founder of that |
| 20:36 |
Megaf |
I'm neutral regarding #Minetest-Mod actually |
| 20:36 |
Megaf |
I think it's nice having it, although not a problem not having it |
| 20:37 |
Shara |
Like I said - I'm totally happy to talk about mods here, but I don't understand the resistance to letting people use it |
| 20:37 |
sfan5 |
there is no need to split the community into even more channels IMO |
| 20:37 |
sfan5 |
you can use #minetest for that |
| 20:37 |
Shara |
Mr-Pardison: You will probably need to check who maintains technic and ask them |
| 20:38 |
Shara |
sfan5: they'll split themselves then. |
| 20:38 |
sfan5 |
i haven't heard of anyone wanting a separate place to talk about mods |
| 20:38 |
Shara |
check the forum |
| 20:38 |
Megaf |
I don't fully get the need for this very channel tho |
| 20:38 |
Shara |
wuzzy already made a seperate channel |
| 20:38 |
Megaf |
IMHO all we need is #Minetest and #Minetest-OffTopic |
| 20:39 |
Shara |
Two devs (if I recall) said reoppening -mods was an option instead, yet it sits there closed |
| 20:39 |
Megaf |
I don't even think there's a need for #Minetest-Dev |
| 20:39 |
Shara |
Hah |
| 20:39 |
Megaf |
Everything regarding Minetest could be spoken at #Minetest and other things in the offtopic channel |
| 20:40 |
Megaf |
maybe we need #Minetest-Hub (where admins and mod creators join) -Dev, for Dev and Minetest for off topic? |
| 20:40 |
Megaf |
meh, whatever |
| 20:40 |
Megaf |
hah |
| 20:41 |
sfan5 |
#minetest is for everything minetest related without strict offtopic rules |
| 20:42 |
Shara |
sfan5: https://forum.minetest.net/viewtopic.php?f=47&t=18510&p=294799#p294799 |
| 20:43 |
* Megaf |
do /join #Minetest-Mod-Dev |
| 20:44 |
Shara |
Both paramat and rubenwardy mentioned the possibility of reopening the channel |
| 20:45 |
Megaf |
#Minetest #Minetest-Dev #Minetest-Mod-Dev #Minetest-Servers #Minetest-OffTopic |
| 20:45 |
Megaf |
perfect |
| 20:45 |
sfan5 |
Shara: does he detail his reasoning for a separate channel anywhere? |
| 20:45 |
Shara |
Don't know, but I've come across others asking for such a channel too |
| 20:45 |
Shara |
Some people simple want a dedicated space for things. |
| 20:46 |
Shara |
I don't mind so much for myself, but in reality the channel would either not get used much (in which case it does no harm) or it would get used (which means we would have people using an official channel instead of other random ones they made because the team refused to reopen this one) |
| 20:51 |
Megaf |
Information on #minetest-servers: |
| 20:51 |
Megaf |
Founder : jordach |
| 20:51 |
Megaf |
Registered : Jun 03 22:11:44 2012 (5y 23w 2d ago) |
| 20:52 |
Megaf |
Jordach ... |
| 20:53 |
shivajiva |
I've been in -mod-dev a while and nothing is spoken there so I'm not sure what the purpose is, -hub seems like a good place to chat about many aspects of minetest, keeps the devs informed about perceptions and attempts to be mature in the content and language. |
| 20:56 |
Megaf |
I think for our current community size more channels is not really needed, -hub takes care of almost anything not dev related |
| 20:57 |
Megaf |
but as our community grows and our project gets more complex, the larger number of specific channel will be welcomed |
| 20:57 |
IhrFussel |
From #sqlite "Are you using the INT column in the predicate to read out the BLOB in the access query? If so, have you verified it's indexed?" I told them one column in the B is a BLOB the other an INT (maybe INT was wrong though) |
| 20:58 |
IhrFussel |
DB* |
| 21:00 |
paramat |
meh wuzzy should use #minetest for modding discussion, it already is mostly that anyway |
| 21:00 |
Megaf |
we talk alot about modding here |
| 21:01 |
sfan5 |
yes, #minetest is dead enough already, just use that |
| 21:01 |
paramat |
plus #minetest is logged which is useful, and is fairly quiet |
| 21:20 |
paramat |
it seems that if modding discussion moves to a new unofficial channel, that #minetest would then be dead, despite being an official and logged channel, while the discussion is then at an unofficial unlogged channel. makes no sense, better to consider #minetest the modding channel, which it roughly already is |
| 21:23 |
Shara |
Is the purpose for channels (other than -hub) listed anywhere? |
| 21:24 |
Shara |
Could fix it if so maybe. |
| 21:26 |
paramat |
erm |
| 21:27 |
paramat |
https://www.minetest.net/irc/ could be improved |
| 21:28 |
paramat |
i think there's more need for an offtopic channel, but that can be unofficial, i hope someone sets this up :] |
| 21:29 |
* VanessaE |
peeks in just long enough to spam the channel... https://www.thingiverse.com/thing:2640546 (Pipeworks models in 3d printable form :D ) |
| 21:29 |
sfan5 |
offtopic is not forbidden in #minetest |
| 21:29 |
Shara |
paramat: looks like red-001 has registered -offtopic |
| 21:31 |
paramat |
excessive offtopic is, and it has been excessive over the last few months, especially in -hub |
| 21:32 |
Shara |
-hub isn't the same as #minetest |
| 21:32 |
paramat |
3D printed MT =) |
| 21:33 |
paramat |
yes, but th rues for all IRC channels are "Off-topic is allowed, but excessive use of it is disallowed. |
| 21:33 |
paramat |
oops |
| 21:33 |
paramat |
*the rules |
| 21:33 |
paramat |
rightly so too |
| 21:41 |
VanessaE |
enjoy. I'm off. |
| 22:08 |
|
benrob0329 joined #minetest-hub |
| 23:04 |
|
Fixer_ joined #minetest-hub |
