Minetest logo

IRC log for #minetest-hub, 2017-11-13

| Channels | #minetest-hub index | Today | | Google Search | Plaintext

All times shown according to UTC.

Time Nick Message
00:41 Grandolf joined #minetest-hub
01:14 Grandolf_ joined #minetest-hub
01:22 Grandolf joined #minetest-hub
01:28 Grandolf_ joined #minetest-hub
01:45 Grandolf joined #minetest-hub
01:45 Grandolf joined #minetest-hub
01:59 paramat joined #minetest-hub
02:58 lhofhansl left #minetest-hub
03:04 ThomasMonroe joined #minetest-hub
03:15 CalebDavis joined #minetest-hub
03:23 bwarden joined #minetest-hub
03:34 paramat left #minetest-hub
04:02 bwarden joined #minetest-hub
05:30 lhofhansl joined #minetest-hub
05:30 lhofhansl left #minetest-hub
05:30 lhofhansl joined #minetest-hub
06:24 ChainLynxMobile joined #minetest-hub
07:01 CWz joined #minetest-hub
07:07 Dargod joined #minetest-hub
07:36 Dargod joined #minetest-hub
08:10 nore joined #minetest-hub
08:19 CWz just did something stupid. i made a texturepack for myself that replaces steel axe,pick and shovel with bongs. newbs are now stoners
08:25 coyote1 joined #minetest-hub
08:51 nerzhul joined #minetest-hub
09:57 coyote2 joined #minetest-hub
10:45 Fixer joined #minetest-hub
10:58 IhrFussel joined #minetest-hub
11:41 Megaf joined #minetest-hub
11:41 Megaf Heya
11:41 Megaf Just installed ircloud here again
11:41 Megaf Let's see if it's any good
11:49 IhrFussel Okay guys I ran my modified mod profiler for 3 days now and *not even once* I saw 1 lag-entry that could come even close to the max_lag the server reports when a few more players play...the highest I saw was 720 ms caused by mod zombies but that was very rare...most "lag mods" caused 3-5 ms lag which would never explain 2+ secs max_lag
11:51 sfan5 the lag can also be caused inside C++ code
11:52 IhrFussel How can the engine lag for 2 seconds outside of Lua?
11:54 Megaf Speaking of mods, do we have a tool that will load the whole map, like clear objects does, and remove all unknown objects?
12:17 IhrFussel 99.9% certain it's not the mods now... I disabled all my mods (expect areas + MTG) and the max_lag peaks at 1.7 secs with 3 players
12:18 IhrFussel except*
12:22 shivajiva how many players are registered in the auth file?
12:26 shivajiva if you have 30k of players registered then you are searching 30k of players for privilege validation for 3 players and it's several times a second, continuously whilst they are logged in...
12:27 shivajiva this adds lag despite tables being relatively fast in lua
12:27 Megaf Oh
12:27 Megaf How do we clean the Auth file?
12:27 shivajiva ofc sauth mod makes it reasonable by only caching the players logged in
12:29 shivajiva Megaf you need to compare the entries in auth against player files and remove entries from the auth that didn't make it to the player file stage
12:29 shivajiva and then you can do a time based cull
12:30 Megaf My server sure have several thousands of players registered
12:30 shivajiva or shift to sauth and player db and use a little query to fix the orphaned entries and never suffer auth lag again
12:31 shivajiva skyblocks goes down to 0.1 lag with sauth running, used to be 5 seconds
12:32 shivajiva >200k of entries in the auth file was bogging it down
12:32 shivajiva I haven't even purged the orphans to get that kind of response
12:32 shivajiva :P
12:33 shivajiva also, use sban and stop loading all the banned players into memory
12:33 lisac joined #minetest-hub
12:34 shivajiva correction, all the players it's ever seen!
12:35 shivajiva xban2 file normally corrupts at around the 6MB size
12:36 Jordach joined #minetest-hub
12:36 shivajiva people use databases when they are available to prevent corruption!
12:38 shivajiva I took the knowledge gained from running a server for 18 months and made 2 mods that fix the biggest issues a server owner faces...
12:40 * shivajiva packs his soapbox up and saunters off
12:44 shivajiva ~950 privilege checks for a player just to get into the game...
12:44 IhrFussel shivajiva, I guess it's time to try sauth
12:44 shivajiva pls ;)
12:46 Jordach shivajiva, check your privilege
12:46 Jordach Kappa
12:46 shivajiva ?
12:46 Jordach being sarcastic about SJWs :^)
12:46 shivajiva lol :P
12:46 IhrFussel I installed lsqlite3 via luarocks and moved sauth-master to worldmods/sauth ... is that all I need?
12:47 shivajiva yes that should be all that is required
12:48 shivajiva if the record set is big it would bbe better to do it locally and copy the db to the server though
12:48 shivajiva >30k of entries forces a dump file you have to import
12:49 shivajiva trivial still but if you don't play with db's it may seem a little daunting
12:49 shivajiva and it will require the installation of sqlite3 for the import
12:49 shivajiva no biggie
12:51 shivajiva if anyone wants me to do the conversion just pm me with a link to the file and I'll process it and link you back
12:51 shivajiva 80MB import takes less than a minute
12:52 shivajiva thank goodness for bulk import statements :)
13:00 coyote1 joined #minetest-hub
13:21 IhrFussel shivajiva, sauth just created a new DB with only 1 entry.. I see no import file either
13:26 shivajiva hmm how many entries are in the auth file?
13:27 IhrFussel I have a auth.sql file (7.7 MB) and a sauth.sqlite (4.9 KB) ... the auth.txt has been deleted (by the mod?) but it had ~ 18,000 entries
13:27 shivajiva auth.txt was renamed to auth.txt.bak
13:28 IhrFussel Ok yeah that happened so it's correct?
13:28 shivajiva so it should have imported directly with only 18k of entries
13:28 IhrFussel Well first time I started the server it shutdown itself after 10 secs
13:28 Fixer mt engine: 28 notification, mtg: 0
13:29 shivajiva the sql file is auth.sql, that needs importing fussel
13:30 shivajiva hnce the small db and the large import file
13:30 shivajiva +e
13:30 IhrFussel shivajiva, but the auth.sql already exists and it's several MB so I guess it worked?
13:30 IhrFussel Ohh .sql is the import file?
13:30 shivajiva yea
13:30 shivajiva sauth.sqlite is the db
13:31 IhrFussel Ok what's the command to import it now?
13:31 shivajiva have you installed sqlite3?
13:32 shivajiva once you have the commands are:-
13:32 shivajiva sqlite3
13:33 IhrFussel Yes...so just sqlite3 sauth.sqlite3 < auth.sql ?
13:33 IhrFussel sqlite*
13:33 shivajiva that will suffice, personally I open the db and read the auth.sql
13:34 shivajiva you should delete the sauth.sqlite file first
13:34 shivajiva otherwise you will get a duplicate entry for yourself
13:35 IhrFussel That took 2 seconds...is that right? o.o
13:35 shivajiva how big is the sqlite file now?
13:35 IhrFussel 8.7 MB
13:35 shivajiva perfect :)
13:37 shivajiva only the sban import reduces the data size because of the layout used by xban2 for readable saving
13:39 shivajiva sqlite is pretty fast, that's why I chose it. Tests showed <5ms access time for a record pulled from the db and it's cached after that till the player leaves
13:40 IhrFussel shivajiva, unfortunately that's not the source of the problem but I'll leave it like that anyway...max_lag just jumped to 1.4 secs with me ALONE
13:41 shivajiva okay but loading has an effect that needs time to settle, more importantly will be the lag figure in 5 mins
13:42 shivajiva when the server start event is a distant reflection on the max_lag figure
13:43 shivajiva despite the stated value does it feel more responsive?
13:44 IhrFussel I guess the best way to check is switching privs?
13:47 shivajiva I noticed teleporting around was quicker and obviously I have a gamehub mod running that instantly became more responsive in handling events
13:48 shivajiva sauth runs on Darklands and dropped the lag from 7s down to 1.5s
13:51 shivajiva it runs both sauth and sban though so it recovered about 66MB of lua memory
13:51 shivajiva from the change over
13:54 shivajiva heh your server is heavy on the media
13:54 IhrFussel shivajiva, the lag only goes up when I move quickly on the map...when I idle for a bit the max_lag goes <0.5
13:55 coyote1 joined #minetest-hub
13:57 Calinou joined #minetest-hub
14:09 coyote1 left #minetest-hub
14:25 ThomasMonroe joined #minetest-hub
14:41 shivajiva IhrFussel: grant invokes core.get_auth_handler().get_auth(grantname) which does use the registered handler so it's not that
14:41 IhrFussel shivajiva, I use minetest.player_exists now is that safe?
14:42 oOChainLynxOo joined #minetest-hub
14:42 IhrFussel shivajiva, 0.4.16 stable chatcommands.lua uses core.auth_table
14:43 shivajiva player_exists should work fine
14:43 CWz joined #minetest-hub
14:43 IhrFussel Or maybe not...maybe my chatcommand.lua is older than 0.4.16
14:44 shivajiva yea I'm in 0.4-stable and chatcommands.lua is using core.get_auth_handler().get_auth(grantname) for grant
14:45 CalebDavis joined #minetest-hub
14:47 shivajiva shows no use of core.auth_table in that file
14:48 IhrFussel minetest.auth_table seems to be very rarely used by mods...only areas used it (changed now)
14:49 shivajiva only other example of it was in the border mod
14:51 CWz joined #minetest-hub
14:53 shivajiva I'll do some testing but I'm sure I checked /grant, maybe you modified your version?
14:57 Jordach joined #minetest-hub
14:59 shivajiva did you grep for auth_table to see where it's used on your server?
14:59 IhrFussel "maybe my chatcommand.lua is older than 0.4.16"
15:00 IhrFussel I probably didn't use the one from 0.4.16 cause I modified quite a bit in it
15:00 shivajiva yea something isn't quite right about your setup in terms of what's expected at that revision
15:01 rubenwardy it's generally a bad idea to modify default
15:01 shivajiva ^
15:01 rubenwardy baring in mind it's possible to override chat commands anyway...
15:01 shivajiva ^^
15:02 rubenwardy minetest.registered_chatcommands["name"].func = function(...)  end
15:02 rubenwardy ops
15:02 rubenwardy *modify builtin
15:03 shivajiva were in chatcommands.lua so we both knew what section you were referring to ruben ;)
15:04 rubenwardy :D
15:04 IhrFussel I modified a bit than just the commands...I modified the overall chat message handling since I use an external bot and want its commands to be accepted
15:04 IhrFussel A bit more*
15:04 rubenwardy what does that mean?
15:05 IhrFussel I use a bash bot for my server and want its commands to be valid ingame
15:05 rubenwardy what
15:05 rubenwardy you run bash commands in game?
15:05 shivajiva chat handling is setup so you can terminate or pass forward, that covers your needs for all chat commands
15:05 IhrFussel No the bot runs in the background and listens to the ingame chat log and executes functions based on commands it finds
15:06 rubenwardy what kind of commands?
15:06 rubenwardy this sounds awfully a lot like a remote code execution vulnerability
15:06 shivajiva join/part/level etc
15:07 IhrFussel Lots...it has 150+ commands ... no players cannot just execute code, the bot will add the actual command/program by itself
15:07 behaleba- joined #minetest-hub
15:08 IhrFussel Like when a player enters something the bot executes cat something for example or if [[ -f something ]]
15:08 shivajiva what you are doing can be done by intercepting the chat stream and passing it on after you have checked it against your list of commands
15:09 shivajiva unless they alter the behaviour ofc
15:09 IhrFussel Lua is painfully slow to type and lots of the bot commands need to check for file names/contents... that is just easier to write in bash $(cat $file) [[ -f $file ]] etc
15:10 rubenwardy errr... ssh?
15:11 atorian37 joined #minetest-hub
15:16 rubenwardy btw, IhrFussel, I saw in the log you're suffering from performance issues. As well installing sauth, I recommend doing the following:
15:16 rubenwardy cd worlds/world
15:16 rubenwardy sudo inotifywait -e modify -e attrib -e move -e create -e delete -m  -r .
15:16 rubenwardy it tells you which files are being modified as they happen
15:17 rubenwardy so you can see if it's due to file i/o the server is freezing
15:17 rubenwardy (you can also tell if a server is freezing from the packets on f5 btw)
15:17 rubenwardy but sauth helps a lot for server
15:17 rubenwardy *servers
15:18 rubenwardy oh, you'll need to do   sudo apt install inotify-tools    or smth
15:30 sniper570 joined #minetest-hub
15:52 shivajiva IhrFussel: is your disk HDD or SHD or SSD?
15:53 IhrFussel rubenwardy, with grep -v sqlite it's basically just debug.txt that is modified a lot...there are some other small files here and there every 30-60 secs
15:53 rubenwardy are you using the players sqlite DB?
15:54 rubenwardy I think there's a bug with it
15:54 rubenwardy it was causing massive lockups on my server
15:54 shivajiva :O
15:54 rubenwardy which it shouldn't with 15 players online
15:54 rubenwardy like, updating 15 records shouldn't take very long aha
15:55 rubenwardy I made an issue
15:55 IhrFussel Nope still using files and shivajiva it's a regular HDD ... those lag spikes are not normal... with 3 players spiking to 3+ secs once per minute or so...something is completely wrong there
15:55 rubenwardy would be code to have it independently verified
15:57 shivajiva wonder if we can persuade someone to verify it who has reasonable traffic
16:00 shivajiva trouble I have with this is skyblocks sits at 0.1 lag with 3 players in and for over an hour, it's nothing but a vanilla stable-0.4 with about 30 mods
16:01 rubenwardy my server has been having block loading issues recently, I reckon it's due to some max packet send though
16:01 rubenwardy MT is no where near my max CPU or network
16:02 shivajiva same for Fussel, mt is taking hardly any resources 8% CPU 20% memory and it's still spiking up to 3s
16:03 rubenwardy odd
16:03 rubenwardy what about hard drive usage?
16:03 shivajiva that's why he's obsessed with tracing it
16:03 IhrFussel max_lag 3.6 secs with 11 players 28% load, 13% CPU and 16% RAM
16:03 rubenwardy also, are map blocks read asynchronously?
16:04 shivajiva you'd hope so
16:04 rubenwardy there's emerge threads, so yeah, I'd hope so
16:04 rubenwardy not gone into that area yet
16:05 IhrFussel Is max_packets_per_iteration = 512 maybe too low?
16:07 rubenwardy is that a server cap?
16:07 IhrFussel I set server_unload_unused_data_timeout to 900 in hope that the server would cache mapblocks better
16:07 IhrFussel Nope I just saw that somewhere and applies it
16:08 IhrFussel I guess I will have to create a new test world with all mods and let it run for a bit to see if it still spikes...if no then the map DB is the problem
16:08 shivajiva skyblocks was running with max_simultaneous_block_sends_server_total = 10000
16:12 shivajiva when moving fast as the chunks load it produces a little stutter but nothing that would spoil the game play
16:12 Fixer hmm, what is this youtube insanity with 1280*738? resolution? scales like shit, everything is blurry, wth?
16:12 IhrFussel Or maybe I had the map_save_interval too high? (60)
16:13 shivajiva what's the default for the save interval?
16:14 IhrFussel 15 I think
16:18 IhrFussel Maybe if one sets it too high it writes too many mapblocks at once
16:18 shivajiva that could stack it up as you are potentially writing x4 times slower, with x4 the data to save, wonder if they are done as a series of single transactions
16:19 shivajiva you're just trying to drag me into the core code :P
16:21 IhrFussel I set it to 20...will see if it's better after restart
16:21 shivajiva excellent :)
16:32 Calinou hi
16:34 shivajiva Hi Calinou
16:42 IhrFussel joined #minetest-hub
16:44 ThomasMonroe joined #minetest-hub
17:27 paramat joined #minetest-hub
17:53 Darcidride_ joined #minetest-hub
17:55 nerzhul joined #minetest-hub
18:00 IhrFussel inotifywait shows 20-30 MODIFY map.sqlite at once every few secs...is that normal?
18:06 IhrFussel paramat, sauth requires lsqlite though...do you plan to bundle that?
18:06 rubenwardy yes
18:06 rubenwardy I wonder if you can compile it in
18:06 rubenwardy as a Lua module
18:06 Fixer joined #minetest-hub
18:06 rubenwardy but yeah, it needs to work without Lua rocks or any system hackery
18:07 IhrFussel Yes regarding my question or lsqlite?
18:07 IhrFussel My first question*
18:08 rubenwardy lsqlite
18:08 rubenwardy regarding sqlite, that's not great. It may be part of the libsqlite rather than what we're doing
18:08 rubenwardy ie: async staggered writes
18:10 IhrFussel It also creates map.sqlite-journal a lot ... too many requests?
18:11 Darcidride_ joined #minetest-hub
18:11 rubenwardy same happens with players.sqlite, but suspiciously often
18:13 IhrFussel Here the last 500 or so entries https://pastebin.com/Cvs6ftSn
18:14 IhrFussel Okay more like last 1,900 xP ... I also added the date
18:15 rubenwardy maybe try postgresql? You can always go back
18:15 rubenwardy make sure to keep a backup of the sqlite too
18:16 rubenwardy actually, does the migrate work all ways? Thought it does
18:16 rubenwardy nerzhul would be able to answer :P
18:22 nerzhul use postgresql for a good long term server relational database yes
18:23 IhrFussel Will that reduce the DB reads/writes? Do I need to recompile MT for postgresql?
18:23 rubenwardy I'd to investigate this and do benchmarks at some point
18:24 nerzhul IhrFussel, postgresql will do less write but better write too, and MT doesn't care about those and is faster because it's not done by itself
18:24 rubenwardy my intuition is that a key-value database like LevelDB or Mongo would be better for our inherently non-relational data, but that doesn't seem to be the case.
18:24 nerzhul rubenwardy, postgresql on my machine can go 1M transactions per second, try to do that with sqlite :p
18:24 rubenwardy so I
18:24 nerzhul mongo is shit and we have relational data :)
18:24 rubenwardy maps aren't though
18:24 nerzhul binaryblob is just a field
18:25 rubenwardy it's a key - position hash - and a value - binary blob
18:25 nerzhul do you want to query inside the mapblock ? no ? then nosql is useless
18:25 rubenwardy no relation ships
18:25 nerzhul relation is the primary key: postition
18:25 IhrFussel Does postgresql require a certain compile flag or does it just work when I enter the info into world.mt ?
18:25 rubenwardy IhrFussel, you need to compile with postgresql-dev installed on the system
18:25 rubenwardy no flags
18:25 nerzhul IhrFussel, regular postgresql, just tune shared_buffer value to 512MB or 1GB if you have 2GB ram
18:25 rubenwardy just need to have the header install
18:25 rubenwardy +ed
18:26 nerzhul and use PostgreSQL 9.5 at least, 10 is prefered
18:27 IhrFussel rubenwardy, wait...I install postgresql-dev (name of ppa?) then I compile the engine again with the EXACT same flags as usual and it will just work?
18:28 rubenwardy yes, but you also need to do a migration
18:28 rubenwardy I'm not sure how
18:28 rubenwardy nerzhul will be able to give more detail
18:28 rubenwardy but you'll need postgresql login details somewhere
18:30 IhrFussel Too many packages https://pastebin.com/SdGCFeD1
18:31 rubenwardy ha
18:31 rubenwardy nerzhul?
18:31 rubenwardy :D
18:31 rubenwardy I'm guessing postgresql-9.5-dev, but it'd be nice to have 10
18:32 rubenwardy what that isn't a thing
18:32 rubenwardy heh
18:32 Calinou you can probably get a PPA or something, to get latest PostgreSQL on Linux
18:36 nerzhul there is apt.postgresql.org official repository
18:39 IhrFussel Now there's even more packages https://pastebin.com/WY2dYphU
18:41 Calinou there's a v10 now :)
18:41 IhrFussel But which do I need? server-dev?
18:42 IhrFussel 10 doesn't even have a server-dev
18:47 Calinou IhrFussel: postgresql-server-dev-10
18:47 Calinou it's the other say around :)
18:47 Calinou way*
18:48 IhrFussel Is that the one I need??
18:48 Calinou IhrFussel: probably :)
18:51 IhrFussel I'm asking #postgresql ... maybe they know
18:56 Aerozoic joined #minetest-hub
19:00 paramat rubenwardy could this CSM mod be disruptive to servers? https://forum.minetest.net/viewtopic.php?f=53&amp;t=18800 it does say 'kicks you if you are trying that "something" online' but that kick could easily be removed
19:04 Fixer *,,,,,*
19:05 Shara Sounds like a way to spy on contents of containers you are not meant to see into
19:06 sofar he kicks the player if they use it outside of singleplayer mode
19:06 sofar lol
19:06 Shara Yay for CSM?
19:07 sofar even without that kick, this is fairly harmless
19:07 Fixer paramat: is not it is pretty much the same as chest inspector? or worse?
19:07 sofar it's a generic version of chest inspector, yes
19:07 Shara Harmless maybe, but still unwanted as far as I'm concerned
19:08 Shara What if you set up a game wher ea player can select one chest to open for a reward?
19:08 sofar ha
19:08 sofar just make a chest that doesn't store the content in nodemeta
19:09 Shara Shouldn't need to make workarounds like that though
19:09 Shara If chest content was meant to be seen by everyone it should just be visible to everyone
19:09 sofar it essentially is visible to everyone
19:10 sofar you can mark fields in nodemeta as private and then this csm mod is defeated
19:10 sofar that mechanism exists
19:10 Shara The ability to do such things has fewer and fewer barrier though. That's the issue.
19:10 shivajiva +1 for chest contents stored in db only retrievable by the owner
19:12 paramat well see my reply, the kick code can be removed, and this would be fairly useless in singleplayer, plus it's marked as a 'cheat'
19:14 Shara There is clearly no use for it in singleplayer unless you have copied a map from somewhere
19:14 paramat doesn't seem harmless
19:14 Shara Even then, you'd have protection_bypass
19:14 Shara So this would maybe only help with mods having containers that don't respect that, at best
19:15 paramat looks like a server cheat mod carefully created to get around the rules
19:15 Shara Yup
19:16 shivajiva just a thought but shouldn't we be heavily biased towards db storage, not just for the map but everything. Seems there must be some issue with that scenario given we are so far down the line
19:17 IhrFussel Okay I pretty much can confirm now that the ingame lags are caused by the DB... whenever the engine modifies map.sqlite it lags 1-2 secs right now
19:18 shivajiva probably not noticeable on SDD
19:18 shivajiva try levelDb ?
19:18 Shara leveldb didn't help at all on Dark lands
19:19 Shara I'm still pretty sure it made things worse
19:20 shivajiva could you quantify in what way it affected the performance or was it just a further deterioration?
19:21 shivajiva *SSD
19:21 Shara Well, it didn't seem to make anything better
19:21 IhrFussel shivajiva, it didn't always happen...the DB isn't small so maybe that takes longer?
19:22 IhrFussel map.sqlite is 17 GB now
19:22 shivajiva wow
19:22 Shara Fussel, that's comparable to RC in size
19:22 Shara I have an SSD though
19:24 IhrFussel That minetest modifies the DB 20+ times in a row seems very inefficient to me
19:24 shivajiva I'm disadvantaged by skyblocks as it keeps the map size down and for whatever reason leveldb performs better than sqlite. The map db is only 1.8GB
19:25 sofar 17gb map.sqlite... heh
19:25 sofar itb's is 30mb lol
19:25 shivajiva lol
19:25 IhrFussel I will try server_map_save_interval with 120 and see if the lag goes down more
19:25 shivajiva wonder what the access time is on a 17 GB sqlite db
19:26 Shara I think RC is around 16GB now
19:26 sofar longer delays between writes may make things worse
19:26 Krock joined #minetest-hub
19:26 Shara But RC doesn't really have any real lag issues
19:26 sofar but you may have hit an sqlite scaling problem
19:27 Megaf joined #minetest-hub
19:27 Shara I think DL is around the same size, but using leveldb. I do get some lag spikes there if a lot of people join at the same time sometimes, but since adding sauth it's been pretty good
19:30 IhrFussel time sqlite3 .minetest/worlds/oldserver/map.sqlite "SELECT * from blocks WHERE ROWID='1'" << this query takes 37 ms
19:32 shivajiva as sofar noted, writing is where it may impact the most
19:34 sofar 37ms is probably 30ms for the exec() overhead already, so it's almost negligeable time
19:34 sofar you want to try adding a block and see how long that takes
19:34 IhrFussel sofar, inotifywait displays 20-50 entries within the SAME second though when the server saves the map
19:35 sofar sure, that's to be expected. With a 17gb file it has to touch various journal blocks.
19:35 sofar the larger the map is, the more it has to do that too, I bet
19:36 sofar 20-50 isn't even shocking
19:36 IhrFussel I see lots of journal CREATE MODIFY and DELETE
19:36 sofar yup
19:36 IhrFussel Would that explain 1+ sec lag? Sometimes even 5 secs
19:37 sofar do you have a precise log of those 20-50 entries? with timestamps?
19:37 IhrFussel https://pastebin.com/Cvs6ftSn
19:39 sofar not enough time resolution, I need like millisecond time stamps
19:39 shivajiva can't see from that what if any timeframes there are in between each transaction
19:39 IhrFussel No problem
19:39 IhrFussel One sec
19:42 Calinou <IhrFussel> map.sqlite is 17 GB now
19:42 Calinou reminds me, weren't there issues with map.sqlite being over 4 GB?
19:44 IhrFussel sofar, https://pastebin.com/xEYUpAF2
19:45 Shara Calinou: Red Cat doesn't seem to mind it. It's been over 4 GB for approaching two years :P
19:47 IhrFussel MODIFY means write correct?
19:48 paramat surely a long server_map_save_interval means that when it does it has a huge amount to do, causing freezes?
19:48 shivajiva MODIFY suggests either update or insert
19:49 shivajiva that was my take on it paramat
19:49 paramat the default is 5.3 so that may be fairly optimal
19:49 paramat 120 is huge
19:50 sofar well it seems to be taking 400+ms in some cases so that's awfully close to 5.3ms (only 1 order off)
19:50 shivajiva the tuned edge is in keeping it low enough to not interfere with the core processes so the data is minimal
19:50 sofar 5.3s
19:51 paramat IhrFussel what's your current save interval?
19:51 paramat or previous
19:52 shivajiva I think he's tried 60/20/120 now
19:52 IhrFussel I will try 5.3 then ... paramat I had it at 60 for 2 months or so...now it's at 20...will try 5.3 after restart
19:53 Shara I recall being told by someone that high was fine, and actually use 60 myself.
19:53 Shara No issues from it.
19:53 Shara So if anyone can confirm it should actually be lower, it would be good to know
19:53 IhrFussel Could server_unload_unused_data_timeout = 900 be the problem? Maybe the longer the server remembers blocks the more it stores per save_interval?
19:54 Shara RC has that much lower, so maybe
19:54 IhrFussel I doubt that it's just the save_interval but we'll see in a minute
19:54 Shara server_unload_unused_data_timeout = 28 here
19:55 Shara But I won't pretend I fully understand these settings
19:55 Shara I think it was Ten who advised me to keep that one low back when I had some problems.
19:56 shivajiva why are these settings not documented?
19:56 Shara This one says:
19:56 Shara #    How much the server will wait before unloading unused mapblocks.
19:56 Shara #    Higher value is smoother, but will use more RAM.
19:56 Shara So it kind of is
19:56 shivajiva to the depth of how to utilise them
19:57 Shara It would require someone who fully understands them to be willing to write more about them :)
19:58 sfan5 what's so hard to understand about that setting? (honest question)
19:58 shivajiva nothing at all about that one
19:58 Shara sfan5: Boundaries. What would count as too low/too high and so on
19:59 sfan5 hmm right
19:59 shivajiva it's obvious that keeping the block loaded increases the chance of it becoming available if the player turns back or another player enters it
19:59 Shara Yup, but keeping it loading too long can be an issue because then it's a risk of wasted effort
19:59 Shara But knowing what the right balance is can be really difficult
20:00 shivajiva specifically the map save was what I was interested in at this point
20:02 Mr-Pardison joined #minetest-hub
20:02 IhrFussel I see no real difference...except the game now shows 20-50 MODIFY entries per 5 seconds
20:02 Shara Hi Mr-Pardison :)
20:03 Mr-Pardison Hello Shara.
20:03 shivajiva 2 years down the line this stuff is a lot more transparent but even so if information can be added to what we have to make it clearer in terms of boundaries as Shara suggested
20:03 IhrFussel I wonder why it even has to store that many mapblocks at once...we are 4 players right now and I doubt that 50 mapblocks have been modified
20:03 Shara It's probably a case of info overload for many people when trying to work out which of all the possible options they need as well.
20:04 shivajiva ^
20:04 sfan5 IhrFussel: one modify operation != one mapblock write
20:04 paramat set server_unload_unused_data_timeout to the default or close, 900 is huge compared to the default
20:04 IhrFussel default is 300 right?
20:05 paramat not sure, thout it was around 30
20:05 Shara server_unload_unused_data_timeout (Unload unused server data) int 29
20:05 Shara So 29?
20:05 sfan5 300 still sounds pretty high, try something in the range of [90, 120] if you'd like an increased value
20:06 IhrFussel But that likely won't affect the DB problem...I'll try
20:15 IhrFussel It looked like it helped... 3 players @ 0.6 secs max_lag but then I ran a bit and 1.7 secs
20:21 IhrFussel I just added -e access to inotifywait and some accesses take 100 ms
20:24 IhrFussel I'm pretty sure ACCESS time is the problem
20:29 Megaf Folks, #Minetest-Mods is it closed for good or actually just for invited?
20:29 Mr-Pardison invite only.
20:29 Mr-Pardison just checked.
20:30 * Megaf quietly roads at Shara
20:30 Megaf Mr-Pardison: Channel is empty, not even ChanServ is there
20:30 Mr-Pardison huh.
20:30 Megaf I do believe the channel is closed for good
20:31 * Shara wonders if that means a road dropped on her
20:31 Megaf when you join, Chanserv joins and kicks you
20:31 Mr-Pardison yeah.
20:31 Megaf Shara: I eman
20:31 Shara I still don't get why that channel is closed
20:31 * Megaf quietly roars at Shara
20:31 Shara Even if it would not be hugely busy, is there any harm in opening that channel?
20:31 Megaf Shara: Yep, I think that channel should be open and we should be there...
20:32 Mr-Pardison Now that I'm speaking, mind if I bring up this: https://github.com/minetest-mods/technic/issues/375 ?
20:32 Shara I'm quite happy to talk about mods in here as well, but some people seem to want a dedicated place for it, so it may as well be an official one
20:34 Megaf sfan5: celeron55: Hi gentlemen, a word on Minetest-Mods please?
20:34 sfan5 i'm not involved in that
20:35 sfan5 however i don't see the need for yet another channel to talk about something minetest-related
20:35 Megaf Chanserv says otherwise
20:35 Megaf Information on #minetest-mods:
20:35 Megaf Founder    : celeron55, sfan5
20:35 Shara sfan5: we've got people making their own channels for it instead.
20:36 Shara DOes it do any harm if it's not busy?
20:36 sfan5 Megaf: huh good to know that i'm marked as founder of that
20:36 Megaf I'm neutral regarding #Minetest-Mod actually
20:36 Megaf I think it's nice having it, although not a problem not having it
20:37 Shara Like I said - I'm totally happy to talk about mods here, but I don't understand the resistance to letting people use it
20:37 sfan5 there is no need to split the community into even more channels IMO
20:37 sfan5 you can use #minetest for that
20:37 Shara Mr-Pardison: You will probably need to check who maintains technic and ask them
20:38 Shara sfan5: they'll split themselves then.
20:38 sfan5 i haven't heard of anyone wanting a separate place to talk about mods
20:38 Shara check the forum
20:38 Megaf I don't fully get the need for this very channel tho
20:38 Shara wuzzy already made a seperate channel
20:38 Megaf IMHO all we need is #Minetest and #Minetest-OffTopic
20:39 Shara Two devs (if I recall) said reoppening -mods was an option instead, yet it sits there closed
20:39 Megaf I don't even think there's a need for #Minetest-Dev
20:39 Shara Hah
20:39 Megaf Everything regarding Minetest could be spoken at #Minetest and other things in the offtopic channel
20:40 Megaf maybe we need #Minetest-Hub (where admins and mod creators join) -Dev, for Dev and Minetest for off topic?
20:40 Megaf meh, whatever
20:40 Megaf hah
20:41 sfan5 #minetest is for everything minetest related without strict offtopic rules
20:42 Shara sfan5: https://forum.minetest.net/viewtopic.php?f=47&amp;t=18510&amp;p=294799#p294799
20:43 * Megaf do /join #Minetest-Mod-Dev
20:44 Shara Both paramat and rubenwardy mentioned the possibility of reopening the channel
20:45 Megaf #Minetest #Minetest-Dev #Minetest-Mod-Dev #Minetest-Servers #Minetest-OffTopic
20:45 Megaf perfect
20:45 sfan5 Shara: does he detail his reasoning for a separate channel anywhere?
20:45 Shara Don't know, but I've come across others asking for such a channel too
20:45 Shara Some people simple want a dedicated space for things.
20:46 Shara I don't mind so much for myself, but in reality the channel would either not get used much (in which case it does no harm) or it would get used (which means we would have people using an official channel instead of other random ones they made because the team refused to reopen this one)
20:51 Megaf Information on #minetest-servers:
20:51 Megaf Founder    : jordach
20:51 Megaf Registered : Jun 03 22:11:44 2012 (5y 23w 2d ago)
20:52 Megaf Jordach ...
20:53 shivajiva I've been in -mod-dev a while and nothing is spoken there so I'm not sure what the purpose is, -hub seems like a good place to chat about many aspects of minetest, keeps the devs informed about perceptions and attempts to be mature in the content and language.
20:56 Megaf I think for our current community size more channels is not really needed, -hub takes care of almost anything not dev related
20:57 Megaf but as our community grows and our project gets more complex, the larger number of specific channel will be welcomed
20:57 IhrFussel From #sqlite "Are you using the INT column in the predicate to read out the BLOB in the access query? If so, have you verified it's indexed?" I told them one column in the B is a BLOB the other an INT (maybe INT was wrong though)
20:58 IhrFussel DB*
21:00 paramat meh wuzzy should use #minetest for modding discussion, it already is mostly that anyway
21:00 Megaf we talk alot about modding here
21:01 sfan5 yes, #minetest is dead enough already, just use that
21:01 paramat plus #minetest is logged which is useful, and is fairly quiet
21:20 paramat it seems that if modding discussion moves to a new unofficial channel, that #minetest would then be dead, despite being an official and logged channel, while the discussion is then at an unofficial unlogged channel. makes no sense, better to consider #minetest the modding channel, which it roughly already is
21:23 Shara Is the purpose for channels (other than -hub) listed anywhere?
21:24 Shara Could fix it if so maybe.
21:26 paramat erm
21:27 paramat https://www.minetest.net/irc/ could be improved
21:28 paramat i think there's more need for an offtopic channel, but that can be unofficial, i hope someone sets this up :]
21:29 * VanessaE peeks in just long enough to spam the channel... https://www.thingiverse.com/thing:2640546  (Pipeworks models in 3d printable form :D )
21:29 sfan5 offtopic is not forbidden in #minetest
21:29 Shara paramat: looks like red-001 has registered -offtopic
21:31 paramat excessive offtopic is, and it has been excessive over the last few months, especially in -hub
21:32 Shara -hub isn't the same as #minetest
21:32 paramat 3D printed MT =)
21:33 paramat yes, but th rues for all IRC channels are "Off-topic is allowed, but excessive use of it is disallowed.
21:33 paramat oops
21:33 paramat *the rules
21:33 paramat rightly so too
21:41 VanessaE enjoy.  I'm off.
22:08 benrob0329 joined #minetest-hub
23:04 Fixer_ joined #minetest-hub

| Channels | #minetest-hub index | Today | | Google Search | Plaintext