| Time |
Nick |
Message |
| 00:05 |
|
est31 joined #minetest-dev |
| 00:06 |
|
Puka joined #minetest-dev |
| 00:19 |
|
Void7 joined #minetest-dev |
| 00:36 |
|
kaeza joined #minetest-dev |
| 01:40 |
hmmmmm |
hey paramat |
| 01:40 |
paramat |
hi |
| 01:40 |
hmmmmm |
was just wondering what you thought about removing generateNoise() |
| 01:41 |
hmmmmm |
and splitting all the noise generation calls into each method where they're used |
| 01:41 |
hmmmmm |
if i do this, it is a bit less organized, but now you're able to use each generateThing() method independent of eachother |
| 01:42 |
hmmmmm |
but there are methods that use the same noise, so i'd have to add a little mechanism to tell if a certain noise has been generated for that position yet or not |
| 01:43 |
hmmmmm |
what's your opinion on this? |
| 01:43 |
paramat |
you mean splitting the calculateNoise() function in the mapgen cpp file? |
| 01:43 |
hmmmmm |
yup |
| 01:44 |
hmmmmm |
so move out all the terrain noise generations to generateTerrain() and so on |
| 01:44 |
hmmmmm |
and eliminate calculateNoise() entirely |
| 01:44 |
paramat |
cave noise is already moved out of that for optimisation |
| 01:44 |
paramat |
hmm |
| 01:44 |
hmmmmm |
yeah that's another advantage, it's easier to optimize for cases where it's not going to be used |
| 01:45 |
paramat |
at one point mgv7 had mountain noise calculation moved into generateMountainTerrain() for optimisation too |
| 01:45 |
paramat |
sounds good to me |
| 01:46 |
paramat |
i guess this helps with making mapgen more modular |
| 01:47 |
hmmmmm |
yeah but less organized |
| 01:47 |
hmmmmm |
i like grouping similar tasks all in one place |
| 01:47 |
hmmmmm |
also what do you think of getting rid of MapgenParams, BiomeParams, etc. |
| 01:47 |
hmmmmm |
just pass around a Settings instead to everything |
| 01:49 |
paramat |
"oh i remember why i did it this way now" < not an obstacle anymore? |
| 01:50 |
paramat |
reading your posts on this in logs |
| 01:52 |
paramat |
well as far as i understand it, seems good, as long as we still have per-mapgen biome noise parameter defaults, and still have dungeon params (per mapgen too) |
| 01:53 |
paramat |
of course i don't much understand the settings architecture |
| 02:08 |
paramat |
you mention it's simpler, so sounds good to me |
| 02:14 |
|
Miner_48er joined #minetest-dev |
| 02:35 |
|
Void7 joined #minetest-dev |
| 02:44 |
|
ssieb joined #minetest-dev |
| 03:13 |
|
stormchaser3000 joined #minetest-dev |
| 03:13 |
|
Void7 joined #minetest-dev |
| 03:17 |
|
Zeno` joined #minetest-dev |
| 03:25 |
hmmmmm |
paramat, here's what I'm thinking: |
| 03:26 |
hmmmmm |
we start out with g_settings |
| 03:26 |
hmmmmm |
it gets updated with the game-specific settings |
| 03:26 |
hmmmmm |
the map_meta.txt is a configuration file that creates its own Settings |
| 03:28 |
hmmmmm |
so then we merge the map_meta.txt *into* a copy of g_settings |
| 03:28 |
hmmmmm |
this creates our world-overridden Settings that we'll have to somehow trim the non-mapgen settings from |
| 03:29 |
hmmmmm |
then we'll have a mapgen_settings api that modifies this mapgen Settings that works with the same interface as the regular global settings |
| 03:30 |
hmmmmm |
and you'll be able to discriminate on biome settings by mapgen by checking "mg_name" inside the script initialization |
| 03:32 |
hmmmmm |
the new settings API should be minimal in comparison to the ridiculous amount of code that goes into mapgen configuration currently |
| 03:32 |
hmmmmm |
configuration is a very mundane task, there's no reason why it has to be so expansive and complicated |
| 03:40 |
paramat |
this settings API is used in a mod? |
| 03:42 |
paramat |
hmmmmm can the default biome noiseparams still be set different for each mapgen? doing that in a mod doesn't seem right |
| 03:43 |
hmmmmm |
why not, paramat? |
| 03:44 |
paramat |
erm |
| 03:45 |
paramat |
ok, just seems strange. noiseparams seem something to be set in c++ code before any mods run |
| 03:45 |
hmmmmm |
the defaults are set |
| 03:46 |
paramat |
oh of course, default is a mod |
| 03:47 |
paramat |
it would be done in mapgen.lua |
| 03:47 |
paramat |
i'm sure it'll seem fine once i understand it :) |
| 03:51 |
hmmmmm |
well the reason why i'm asking you is to make sure everybody who has anything to do with the mapgen is on board with this |
| 03:51 |
paramat |
ok so biome noiseparams are set to a single default, then made per-mapgen by the settings API |
| 03:51 |
hmmmmm |
right, in mapgen.lua you'd have a bunch of if mapgen_name == "v6" then minetest.mapgen_settings_set("np_biome_humidity", { ... }); |
| 03:52 |
paramat |
likewise with dungeonparams i guess |
| 03:52 |
hmmmmm |
if we were to really do it the way you want, do you realize how many different settings there would be? |
| 03:52 |
paramat |
i don't heh |
| 03:52 |
hmmmmm |
you'd need to have a different set of settings for every single combination of mapgen, biomegen, cavegen, etc. |
| 03:53 |
hmmmmm |
as if those settings weren't enough the way it is |
| 03:53 |
hmmmmm |
there are simply too many mapgen settings as it is right now |
| 03:53 |
hmmmmm |
and modifying it through the config file is incredibly unwieldy and i don't think it should be in the config file to begin with |
| 03:54 |
hmmmmm |
the only reason why it's there now is for legacy purposes |
| 03:54 |
hmmmmm |
people who know how to modify the noise params are able to write mods |
| 03:54 |
paramat |
caveparams also work like this? |
| 03:54 |
hmmmmm |
yes |
| 03:54 |
paramat |
ok |
| 03:54 |
hmmmmm |
so this is: general mapgen settings, specific mapgen settings, cave settings, dungeon settings, biome settings |
| 03:55 |
hmmmmm |
and of course there are different cave settings for each specific cave generation type |
| 03:55 |
hmmmmm |
same with biomes, dungeons, etc. |
| 03:55 |
hmmmmm |
there are simply too many permutations to deal with individually |
| 03:55 |
paramat |
so only base terrain noiseparams remain per-mapgen from the beginning? |
| 03:55 |
hmmmmm |
not quite |
| 03:56 |
hmmmmm |
for mapgen v6 the trees and apple noises are still v6-specific for example |
| 03:56 |
hmmmmm |
also part of the refactor is making it easy, no, trivial to add features to mapgens |
| 03:57 |
hmmmmm |
so i'm going to add the unified biome system to v6 |
| 03:57 |
paramat |
as an option |
| 03:57 |
hmmmmm |
i haven't decided if v6 biomes will be its own BiomeGen or what |
| 03:57 |
hmmmmm |
or how to expose the option |
| 03:59 |
paramat |
i considered writing a 'mgv62' which would be mgv6 terrain with the new biome system, looks like you might make this possible |
| 03:59 |
hmmmmm |
please stop making so many mapgens |
| 03:59 |
hmmmmm |
consider ways of adding options or modifying existing interfaces |
| 03:59 |
hmmmmm |
i'm sorry to say, but if you did make a pr for mgv62 i'd have to -1 it |
| 03:59 |
paramat |
heh that's what i mean, i won't need to do this now |
| 04:00 |
hmmmmm |
this is getting ridic |
| 04:03 |
hmmmmm |
i think you were a bit too ctrl+v ctrl+c happy |
| 04:03 |
hmmmmm |
i'm having trouble dealing with how to handle reverse compatibility when it comes to mgXX_np_cave1 and so on |
| 04:04 |
hmmmmm |
i have many options.. but all of them suck |
| 04:04 |
hmmmmm |
whatever happened to the old days when you'd just be able to type in a seed and get a map |
| 04:12 |
paramat |
it was scary when you were absent, good to see it being organised. afk for noms |
| 05:07 |
|
Hunterz joined #minetest-dev |
| 05:18 |
|
paramat joined #minetest-dev |
| 05:24 |
paramat |
hmmmmm to be clear, everything you have proposed seems good to me, i'm on board and have fastened my seatbelt |
| 05:26 |
hmmmmm |
the real takeaway from using Settings instead is that adding a new mapgen param is as simple as writing float foobar_amount = settings->getFloat("mg_blah_foobar_amount"); |
| 05:26 |
hmmmmm |
no additional work required, you get everything else - the lua modability, the config setting, for free |
| 06:06 |
|
paramat left #minetest-dev |
| 06:12 |
|
swaaws joined #minetest-dev |
| 06:20 |
|
red-001 joined #minetest-dev |
| 06:40 |
|
Thomas-S joined #minetest-dev |
| 06:41 |
Thomas-S |
Hello! The PR minetest_game#1099 got updated according to your wishes. |
| 06:49 |
|
troller joined #minetest-dev |
| 06:55 |
|
Obani joined #minetest-dev |
| 06:55 |
|
Obani left #minetest-dev |
| 07:09 |
|
torgdor joined #minetest-dev |
| 07:56 |
|
davisonio joined #minetest-dev |
| 08:16 |
|
Amaz joined #minetest-dev |
| 08:29 |
|
damiel joined #minetest-dev |
| 08:35 |
|
jin_xi joined #minetest-dev |
| 09:28 |
|
Megaf joined #minetest-dev |
| 09:32 |
|
Puka_ joined #minetest-dev |
| 09:44 |
|
davisonio joined #minetest-dev |
| 10:35 |
Zeno` |
#4141 is a bit mysterious |
| 10:35 |
ShadowBot |
Zeno`: Error: Delimiter not found in "An error has occurred and has been logged. Please contact this bot's administrator for more information." |
| 10:35 |
Zeno` |
#4142 |
| 10:35 |
ShadowBot |
Zeno`: Error: Delimiter not found in "An error has occurred and has been logged. Please contact this bot's administrator for more information." |
| 10:36 |
Zeno` |
bot broken :( |
| 10:38 |
|
davisonio joined #minetest-dev |
| 10:46 |
|
Zeno` joined #minetest-dev |
| 11:09 |
|
Fixer joined #minetest-dev |
| 11:11 |
|
davisonio joined #minetest-dev |
| 11:19 |
|
DI3HARD139 joined #minetest-dev |
| 11:19 |
|
Icedream joined #minetest-dev |
| 11:22 |
Zeno` |
I have no idea how to test https://github.com/minetest/minetest/pull/4145 |
| 11:22 |
Zeno` |
But it's gotta be right... anyone using irrlicht 1.8.1 ? |
| 11:23 |
Zeno` |
brb |
| 11:24 |
|
Zeno` joined #minetest-dev |
| 11:29 |
|
davisonio joined #minetest-dev |
| 11:33 |
|
rubenwardy joined #minetest-dev |
| 11:54 |
nrzkt |
nore ? |
| 11:55 |
nore |
yes? |
| 12:00 |
nrzkt |
can you help me to finish https://github.com/minetest/minetest/pull/4124 ? if you can look at last kwoelr comments to help me to finish fixing, and maybe answer to my idea to add an extra field |
| 12:01 |
nrzkt |
thanks :) |
| 12:01 |
|
davisonio joined #minetest-dev |
| 12:02 |
* nore |
looks |
| 12:05 |
|
Darcidride joined #minetest-dev |
| 12:11 |
nore |
nrzkt: IIRC, the main problem is that you use sizeof(int) |
| 12:11 |
nore |
where you need instead the *stored size* of an int |
| 12:12 |
nore |
which I don't know how to get, sorry :/ |
| 12:12 |
nrzkt |
nore i think this is a problem. I can set it to 4 to be sure it's always 4 |
| 12:14 |
nore |
it should be better |
| 12:21 |
nrzkt |
nore i'm doing it and renaming val(ues) to arg(s) |
| 12:21 |
nore |
ok, good :) |
| 12:22 |
|
DFeniks joined #minetest-dev |
| 12:33 |
nrzkt |
nore updated |
| 12:35 |
Zeno` |
you updated nore?! :-o |
| 12:35 |
|
davisonio joined #minetest-dev |
| 12:35 |
nore |
:D |
| 12:35 |
nrzkt |
Zeno`: lol |
| 12:35 |
nrzkt |
Zeno`: can you help on the linked PR, i asked for a question about a new field and i want the last things to be fixed |
| 12:35 |
* nore |
is happy as his script that changes the clock's color when he is highlighted works |
| 12:36 |
nore |
also, your suggestion looks like a good idea to me |
| 12:37 |
nore |
and I can't help you with the other issues hmmmm talked about, it's beyond me :/ |
| 12:38 |
Zeno` |
what's the sizeof(int) issue? |
| 12:38 |
nrzkt |
nore thanks |
| 12:38 |
nrzkt |
i replaced sizeof(int) by 4 |
| 12:39 |
Zeno` |
where? |
| 12:39 |
nrzkt |
but honnestly i don't care about amstrad cpc with MT |
| 12:39 |
nrzkt |
in the argLen arrays |
| 12:42 |
nrzkt |
which machine set int to other thing than 4 nowadays |
| 12:46 |
Zeno` |
not sure |
| 12:47 |
Zeno` |
I mean I'm not sure about the size thing |
| 12:47 |
sfan5 |
why replace sizeof(int) by 4? |
| 12:47 |
sfan5 |
is it faster somehow? |
| 12:47 |
Zeno` |
it couldn't be for speed because sizeof is evaluated at compile time |
| 12:48 |
nrzkt |
sfan5: because hmmmm said sizeof(int) could move between architectures... |
| 12:48 |
nrzkt |
he said: No, no, no. I don't get what it is you changed here but this is all wrong. |
| 12:48 |
nrzkt |
pos.X and so on are not integers, and they do not have the same size as integers. This is an out-of-bounds memory access bug. And same with saveBlock. |
| 12:48 |
Zeno` |
nrzkt, but now we have a different problem... an architecture where sizeof(int) != 4 |
| 12:49 |
nrzkt |
oh no |
| 12:49 |
nrzkt |
i saw the problem in fact |
| 12:49 |
nore |
sfan5: the problem is that this number should be the same on all versions of mt |
| 12:49 |
nrzkt |
i'm stupid, it's a v3s16 |
| 12:49 |
nore |
so that the db is consistent between versions |
| 12:49 |
nrzkt |
s16 memory access into a int |
| 12:49 |
nrzkt |
i should convert s16 values to int and then do the proper thing |
| 12:49 |
nrzkt |
sorry for the stupid fix :) |
| 12:49 |
sfan5 |
please tell me you're not using the bytes of the v3s16 as a database key |
| 12:50 |
nrzkt |
no |
| 12:50 |
nrzkt |
i split into X Y Z |
| 12:50 |
nore |
sfan5: btw, that reminds me we *really* should fix that sql backend into x/y/z someday |
| 12:50 |
sfan5 |
but you're using the bytes of those? |
| 12:50 |
nrzkt |
triple field primary key, i have 10M mapblocks on my server and it response < 0.1 ms |
| 12:50 |
nrzkt |
in fact i find the error, i'm reading a S16 values and said we read 4 bytes |
| 12:50 |
nrzkt |
it's the mistake |
| 12:51 |
nrzkt |
i didn't see it |
| 12:51 |
|
stormchaser3000 joined #minetest-dev |
| 12:51 |
sfan5 |
are you using the bytes of the position as keys or not? |
| 12:51 |
nrzkt |
look at the scheme in the db creation you will have the answer |
| 12:52 |
sfan5 |
so you won't tell me nice |
| 12:52 |
sfan5 |
using &pos.x, &pos.y, &pos.z as database indices is not portable |
| 12:53 |
nrzkt |
sfan5: in fact that's not the problem pointed by hmmm, i understand it now |
| 12:54 |
sfan5 |
ah it's an int |
| 12:54 |
nrzkt |
s16 = 2 bytes, int = 4 bytes that was the mistake |
| 12:54 |
nrzkt |
i'm fixing it properly |
| 12:54 |
sfan5 |
&pos.X is not 4 bytes long though |
| 12:54 |
sfan5 |
yeah |
| 12:54 |
sfan5 |
can't you use indices of type int2? |
| 12:55 |
nore |
sfan5: what's int2? |
| 12:55 |
nore |
2-bytes long int? |
| 12:55 |
nrzkt |
i can but i think it's not a good idea if we want to change our map size to 4 bytes radius |
| 12:55 |
sfan5 |
nore: short |
| 12:55 |
nrzkt |
shortint yes |
| 12:55 |
sfan5 |
or int16_t or whatever you want to call it |
| 12:55 |
nore |
sfan5: yeah, I have the same opinion as nrzkt |
| 12:55 |
|
davisonio joined #minetest-dev |
| 12:55 |
nore |
it's better to keep them 4 bytes long |
| 12:56 |
nore |
it doesn't take much memory |
| 12:56 |
nore |
and it's more future-proof |
| 12:56 |
nrzkt |
no, the many memory per record is entiteis and after the block itself |
| 12:56 |
sfan5 |
changing the maximum map size is probably not going to happen any time soon but whatever |
| 12:57 |
|
STHGOM joined #minetest-dev |
| 12:57 |
nore |
nrzkt: yep, 6 bytes is ridiculous |
| 12:57 |
nore |
sfan5: I know, I was the one who made the infinite_map branch |
| 12:57 |
nrzkt |
sfan5: no , but beeing ready on a new backend is a good idea :) |
| 12:57 |
nore |
(which, btw, had more issues that just map saving) |
| 12:57 |
nrzkt |
nore i hope you use bigint :p (8 bytes) |
| 12:57 |
nore |
nrzkt: well, I abstracted v3s16 into v3pos |
| 12:58 |
nore |
but the problem is that it affected network too |
| 12:58 |
VanessaE |
why not make every change you can toward the idea of "infinite" maps, without actually "enabling" the feature? |
| 12:58 |
nrzkt |
yes nore that was the big problem |
| 12:58 |
|
Hunterz joined #minetest-dev |
| 12:58 |
nore |
VanessaE: that should be done someday |
| 12:58 |
nore |
but it breaks many many PRs |
| 12:58 |
VanessaE |
so? |
| 12:58 |
VanessaE |
that'll happen anyway no matter when it's done |
| 12:58 |
nrzkt |
VanessaE: the big problem is the protocol, maybe use a v3s32 on the next release with a new packet could be good but compat is very difficult |
| 12:59 |
nore |
but anyway, I always think about it when coding something new |
| 12:59 |
nore |
nrzkt: well, I think it should internally be changed where possible |
| 12:59 |
VanessaE |
that's what I mean. |
| 12:59 |
nore |
just, v3s16 to v3pos |
| 12:59 |
VanessaE |
change it everywhere you can without breaking compat yet. |
| 12:59 |
nore |
so that sometime in the future, it can work better |
| 12:59 |
nrzkt |
VanessaE: in fact this will do some spaghettis :p |
| 13:00 |
nrzkt |
many conversions from one to one |
| 13:00 |
VanessaE |
perhaps |
| 13:00 |
nore |
(even if we add a static_assert(sizeof(pos) == 2) in network functions) |
| 13:00 |
nore |
and we change that later |
| 13:00 |
nore |
it will need to be done sometime anyway |
| 13:00 |
nore |
before servers run out of space :D |
| 13:01 |
nrzkt |
i use lz4 instead of zlib for mapblocks on my fork this decrease greatly the compression load |
| 13:01 |
nrzkt |
i didn't use it on network |
| 13:02 |
nrzkt |
and this have a nice compression ratio without any load problem |
| 13:03 |
nrzkt |
Zeno`, nore, sfan5: i updated the pgsql PR to fix really the int problem :p |
| 13:08 |
Zeno` |
grr.. ARRLEN! |
| 13:08 |
Zeno` |
(or whatever it's called) |
| 13:08 |
Zeno` |
maybe ARR_LEN |
| 13:09 |
|
Taoki joined #minetest-dev |
| 13:12 |
nrzkt |
Zeno`: what do you think about the PR now ? |
| 13:15 |
Zeno` |
scary |
| 13:16 |
Zeno` |
https://github.com/minetest/minetest/pull/4124#discussion-diff-64037707R179 |
| 13:16 |
Zeno` |
what does the cast do? |
| 13:17 |
nrzkt |
it's in the postgresql api, they use this exact code to properly use the API |
| 13:17 |
Zeno` |
geez |
| 13:18 |
|
Obani joined #minetest-dev |
| 13:18 |
nrzkt |
geez ? |
| 13:18 |
nrzkt |
what is this :p |
| 13:19 |
Zeno` |
jesus |
| 13:19 |
Zeno` |
in the sense I used it, a blaspheme |
| 13:20 |
Zeno` |
I probably should have said "I can't believe they do that shit" |
| 13:21 |
nrzkt |
http://www.postgresql.org/docs/9.5/static/libpq-exec.html |
| 13:21 |
nrzkt |
take that book |
| 13:23 |
Zeno` |
the problem looks like it's more in https://github.com/minetest/minetest/pull/4124/files#diff-1e10a222eed5cccc6aa9da7bdcb3e275R61 |
| 13:23 |
Obani |
is geez really a blaspheme ? |
| 13:24 |
Zeno` |
Obani, these days? Not really |
| 13:25 |
|
Obani left #minetest-dev |
| 13:25 |
Zeno` |
execPrepared() should be fixed I think to use void ** and then cast inside the function |
| 13:25 |
Zeno` |
it's all a bit confusing as it currently is (IMO) |
| 13:26 |
Zeno` |
and probably implementation defined behaviour |
| 13:26 |
|
DonBatman joined #minetest-dev |
| 13:28 |
Zeno` |
yeah |
| 13:30 |
Zeno` |
it doesn't make sense to cast to char ** (effectively) just because postgres wants that (not in the calling functions anyway) |
| 13:32 |
Zeno` |
how is this safe for binary data anyway? |
| 13:32 |
Zeno` |
it doesn't seem to care about endieness for a start |
| 13:32 |
Zeno` |
or am I missing something? |
| 13:33 |
Zeno` |
e.g. isn't this making the database itself non-portable? |
| 13:34 |
Zeno` |
nrzkt ^^^^^^^^ |
| 13:39 |
sfan5 |
Zeno`: i assumed declaring the type "int4" in postgres means that the library decodes the pointer into a number and not just stores the 4 bytes |
| 13:42 |
Zeno` |
perhaps it does |
| 13:42 |
Zeno` |
the values pointed to I assume you mean |
| 13:43 |
Zeno` |
that doesn't address the first issue though |
| 13:43 |
Zeno` |
maybe comments would help |
| 13:43 |
Zeno` |
but there is no way (IMO) that execPrepared() should be wanting char ** and not void ** |
| 13:44 |
sfan5 |
what is "the first issue"? |
| 13:45 |
Zeno` |
execPrepared()'s expected type for params |
| 13:45 |
Zeno` |
here: https://github.com/minetest/minetest/pull/4124/files#diff-1e10a222eed5cccc6aa9da7bdcb3e275R61 |
| 13:47 |
nrzkt |
the ::int4 is a sql cast |
| 13:47 |
nrzkt |
to ensure the type is correct |
| 13:47 |
Zeno` |
there is no reason that I can see that it should be char ** and coercing things to char * when setting up the params array is a) hard to see why it's done in the calling functions; and b) probably not portable |
| 13:48 |
nrzkt |
const char * const *paramValues, |
| 13:48 |
nrzkt |
did you see that in the API Zeno`? |
| 13:48 |
nrzkt |
PGresult *PQexecPrepared(PGconn *conn, |
| 13:48 |
nrzkt |
const char *stmtName, |
| 13:48 |
nrzkt |
int nParams, |
| 13:48 |
nrzkt |
const char * const *paramValues, |
| 13:48 |
nrzkt |
const int *paramLengths, |
| 13:48 |
nrzkt |
const int *paramFormats, |
| 13:48 |
nrzkt |
int resultFormat); |
| 13:48 |
Zeno` |
nrzkt, it doesn't matter... execPrepared is not part of the API |
| 13:48 |
|
Obani joined #minetest-dev |
| 13:48 |
|
Obani left #minetest-dev |
| 13:48 |
nrzkt |
did you look at the called function ? :p |
| 13:48 |
Zeno` |
cast it when needed (in the inline function) |
| 13:48 |
nrzkt |
return checkResults(PQexecPrepared(m_conn, stmtName, nbParams, |
| 13:48 |
nrzkt |
+params, valLen, valFormats, nobinary ? 1 : 0), clear); |
| 13:48 |
Zeno` |
yeah |
| 13:49 |
nrzkt |
why casting to void ** and then recasting to char ** |
| 13:49 |
Zeno` |
because that's where it makes sense? |
| 13:49 |
Zeno` |
you don't cast to void ** |
| 13:49 |
nrzkt |
maybe be we are going out of the pg API design |
| 13:49 |
nrzkt |
it's better to be near the API than hidding the things not beautiful they do |
| 13:49 |
Zeno` |
void * can accept a pointer to ANY type |
| 13:50 |
nrzkt |
i know |
| 13:50 |
nrzkt |
but casting to recast is useless |
| 13:50 |
Zeno` |
there would be no cast in the calling function |
| 13:50 |
|
Obani joined #minetest-dev |
| 13:50 |
|
Obani left #minetest-dev |
| 13:50 |
Zeno` |
it'd be simply const char *args[] = { &x, &y, &z }; for example |
| 13:50 |
Zeno` |
err |
| 13:51 |
Zeno` |
const void *args[] = { &x, &y, &z }; for example |
| 13:51 |
nrzkt |
htonl is requried too, i forget it |
| 13:51 |
Zeno` |
there is no "wtf are these casts to (char*) for" questions raised |
| 13:52 |
nrzkt |
okay then |
| 13:53 |
Zeno` |
also by being void* it gets rid of the undefined behaviour (sizes of different pointers can technically be different) |
| 13:57 |
nrzkt |
i'm changing this |
| 13:59 |
nrzkt |
Zeno`: i pushed, can you look ? |
| 14:00 |
Zeno` |
looks better |
| 14:01 |
Zeno` |
now things like https://github.com/minetest/minetest/pull/4124/files#diff-1ecbc203ee1ec538cd4464f78aeb2ee7R234 can have the casts removed ;) |
| 14:01 |
nrzkt |
oops forgot it |
| 14:01 |
nrzkt |
i do it in other place :p |
| 14:01 |
Zeno` |
heh |
| 14:02 |
Zeno` |
after that change, and from what I understand, it looks good (apart from the style issues hmmmm pointed out) |
| 14:02 |
sfan5 |
huh why is htonl needed? |
| 14:02 |
Zeno` |
and maybe the ARRLEN comment I made |
| 14:05 |
nrzkt |
sfan5: because the library doesn't know the binary data is a int and it's required for network |
| 14:05 |
nrzkt |
https://stackoverflow.com/questions/29802669/inserting-integer-array-with-postgresql-in-c-libpq |
| 14:06 |
sfan5 |
i'd assume that libpq does that |
| 14:06 |
sfan5 |
postgres is weird |
| 14:07 |
sfan5 |
(since it's not minetest doing something with network but libpq doing that) |
| 14:07 |
nrzkt |
libpq is like minetest, it's a generic engine :p |
| 14:16 |
nrzkt |
Zeno`: ARRLEN is a standard macro ? |
| 14:16 |
nrzkt |
err i didn't see no :) |
| 14:17 |
|
Void7 joined #minetest-dev |
| 14:17 |
nrzkt |
i didn't see why i should use ARRLEN there |
| 14:21 |
|
DI3HARD139 joined #minetest-dev |
| 14:23 |
|
Zeno` joined #minetest-dev |
| 14:23 |
Zeno` |
will merge #4145 |
| 14:23 |
ShadowBot |
Zeno`: Error: Delimiter not found in "An error has occurred and has been logged. Please contact this bot's administrator for more information." |
| 14:27 |
nrzkt |
Zeno`: and 4124 ? :D |
| 14:27 |
nrzkt |
is this good for you now ? |
| 14:27 |
Zeno` |
nrzkt, I'm happy with it apart from the style issues :P |
| 14:27 |
nrzkt |
if it's good i will do last tests locally |
| 14:27 |
nrzkt |
can you set your comment on the PR ? :) |
| 14:28 |
nrzkt |
i will test the last changed to ensure all is good and re-test the unescape issue mentionned by hmmm |
| 14:28 |
Zeno` |
ok |
| 14:32 |
Zeno` |
nrzkt, you're really opposed to ARRLEN? |
| 14:33 |
nrzkt |
i'm not opposed i didn't see what is the benefit |
| 14:33 |
nrzkt |
you want to replace sizeof with ARRLEN ? |
| 14:33 |
Zeno` |
no, the 4 |
| 14:34 |
Zeno` |
here https://github.com/minetest/minetest/pull/4124/files#diff-1ecbc203ee1ec538cd4464f78aeb2ee7R200 for example |
| 14:35 |
Zeno` |
and line 214... might be other places |
| 14:35 |
Zeno` |
ARRLEN(args) would probably do |
| 14:40 |
|
hmmmm joined #minetest-dev |
| 14:41 |
nrzkt |
i removed the 4 and set sizeof |
| 14:41 |
|
DI3HARD139 joined #minetest-dev |
| 14:41 |
nrzkt |
oh |
| 14:41 |
nrzkt |
i see |
| 14:41 |
nrzkt |
sorry :) |
| 14:41 |
Zeno` |
heh |
| 14:41 |
nrzkt |
okay for me to replace this thing |
| 14:43 |
Zeno` |
what's PGSQL_V3S16_TO_INT ? |
| 14:43 |
Zeno` |
nevermind |
| 14:43 |
Zeno` |
shouldn't pos be an argument to the macro? :P |
| 14:44 |
nrzkt |
no i generate the code |
| 14:45 |
hmmmm |
nrzkt: I hate to say this but you really suck at C++ |
| 14:45 |
hmmmm |
there are three distinct errors with that single piece of code |
| 14:45 |
hmmmm |
a). Out-of-bounds read |
| 14:46 |
hmmmm |
b). Relying on the size of a variable width type to be a certain number of bytes |
| 14:46 |
nrzkt |
hmmmm: it's now fixed, did you update your window ? |
| 14:46 |
hmmmm |
c). Relying on always having the same endianness |
| 14:47 |
hmmmm |
I don't see anything fixed |
| 14:47 |
Zeno` |
I asked about the endianness earlier... it really needs a comment if the explanation was correct |
| 14:47 |
nrzkt |
conversion from s16 to int + usage and htons to fix endianess |
| 14:48 |
hmmmm |
Oh, this PGSQL_V3S16_TO_INT macro? |
| 14:48 |
hmmmm |
First off, EW, you couldn't come up with any better way to do this? |
| 14:48 |
nrzkt |
what's your proposition ? |
| 14:48 |
hmmmm |
second, there's still the mistake of assuming sizeof(int) == 4 |
| 14:49 |
nrzkt |
int = 4, it's not a 16byte computer |
| 14:49 |
hmmmm |
no, int != 4 |
| 14:49 |
hmmmm |
int is "at least 2" |
| 14:50 |
hmmmm |
and not even depending on the definition of CHAR_BIT |
| 14:50 |
nrzkt |
then what's your suggestion to not hardcode 4 |
| 14:51 |
hmmmm |
use a fixed width type, but the most correct answer here would be to use a type that guarantees at least a certain amount of storage, such as uint_least16_t |
| 14:51 |
|
Hunterz joined #minetest-dev |
| 14:52 |
nrzkt |
i can use uint32_t |
| 14:52 |
hmmmm |
that would cause problems when you go to compile for MSVC |
| 14:52 |
nrzkt |
int32_t * |
| 14:52 |
|
KaadmY joined #minetest-dev |
| 14:52 |
hmmmm |
s32 would be a nice compromise |
| 14:52 |
hmmmm |
but positions are s16s |
| 14:52 |
hmmmm |
why don't you just use s16? |
| 14:53 |
nrzkt |
because i prefer to have now the possibility to extend map to s32 for the future instead of having problems later |
| 14:53 |
|
proller joined #minetest-dev |
| 14:53 |
nrzkt |
it's a new backend we can do it now |
| 14:53 |
hmmmm |
nobody is ever going to change the max size of the map. |
| 14:53 |
hmmmm |
trust me. it's a pipe dream. |
| 14:53 |
nrzkt |
maybe, but that's not a reason |
| 14:53 |
hmmmm |
and it's not even necessary |
| 14:53 |
hmmmm |
how about sqlite and leveldb though? |
| 14:53 |
nrzkt |
i didn't remember about level b but sqlite use this magic shit |
| 14:54 |
hmmmm |
you're not magically saving work by making postgresql use s32s |
| 14:54 |
hmmmm |
you're just using up extra storage space |
| 14:54 |
nrzkt |
lol |
| 14:54 |
nrzkt |
6 bytes vs metadata |
| 14:54 |
nrzkt |
just ridiculous |
| 14:54 |
nrzkt |
we can populate at maximum 11M mapblocks * 6bytes. 66 MB wow, the whole database: 14GB |
| 14:55 |
hmmmm |
sqlite uses "magic shit"? |
| 14:55 |
Zeno` |
umm |
| 14:55 |
Zeno` |
htonl is POSIX |
| 14:55 |
hmmmm |
in any case I can go along with making the fields s32 |
| 14:55 |
hmmmm |
i don't have a huge problem with that |
| 14:55 |
Zeno` |
unless we redefined it somewhere |
| 14:55 |
nrzkt |
okay i change s32 to int |
| 14:56 |
nrzkt |
htons is used un socket.cpp |
| 14:56 |
hmmmm |
no, change int to s32 |
| 14:56 |
nrzkt |
sorry : yes :) |
| 14:56 |
nrzkt |
i inverted the case :p |
| 14:56 |
hmmmm |
however i do have a huge problem with assuming a non-fixed-width type will be exactly a certain number of bytes |
| 14:56 |
nrzkt |
void UDPSocket::Bind(Address addr) |
| 14:56 |
nrzkt |
address.sin6_port = htons(addr.getPort()); |
| 14:56 |
hmmmm |
this is not a design choice or something |
| 14:56 |
hmmmm |
this is a BUG |
| 14:57 |
hmmmm |
Zeno`: We rely on other POSIX-isms like the famous function-pointers-as-void-pointers |
| 14:57 |
hmmmm |
I'm okay with being dependent upon POSIX |
| 14:57 |
Zeno` |
ok |
| 14:57 |
nrzkt |
hmmmm: we are using htons into socket.cpp on our listener |
| 14:57 |
nrzkt |
it's portable |
| 14:58 |
Zeno` |
well, so long as the functions are implemented :P |
| 14:58 |
Zeno` |
everything breaks if they're not I guess |
| 14:59 |
hmmmm |
the only place I'd see that as a potential issue is mingw, like they did with strtok_r |
| 14:59 |
Zeno` |
if we don't currently have an issue then there probably is no issue |
| 14:59 |
Zeno` |
well, there /is/ no issue |
| 15:00 |
nrzkt |
Zeno`, hmmmm: the pr was updated with ARRLEN + s32 |
| 15:01 |
hmmmm |
nrzkt: I'm not too happy with using a macro to declare variables like x y and z |
| 15:01 |
nrzkt |
i can reset those to function if you prefer, it's only to have the same way everywhere if we add functions |
| 15:02 |
hmmmm |
Zeno`, what do you think? |
| 15:03 |
Zeno` |
I'm not sure in this context. TBH I have (had?) a similar issue with the macro assuming that the places where it's used assumes that a parameter is named pos |
| 15:04 |
hmmmm |
I think it'd make a lot more sense if it were more than 3 times |
| 15:04 |
nrzkt |
then, removal, not removal ? |
| 15:04 |
hmmmm |
like for the script api functions, where you have 500 entry points, totally makes sense |
| 15:04 |
hmmmm |
here, i'm not sure the decrease in readability is worth it |
| 15:05 |
Zeno` |
I don't mind it too bad because it's just in that one file |
| 15:05 |
Zeno` |
it's not in a header or anything... |
| 15:05 |
hmmmm |
Yeah, at least there's that |
| 15:05 |
nrzkt |
it's why it's there :p |
| 15:06 |
hmmmm |
I'd rather it be less misleading though |
| 15:06 |
hmmmm |
PGSQL_V3S16_TO_INT makes it sound like.. |
| 15:06 |
hmmmm |
why is a macro necessary anyway |
| 15:06 |
nrzkt |
difficult to descript this operation without writing a book :p |
| 15:06 |
nrzkt |
but okay, i remove it |
| 15:06 |
nrzkt |
no problem |
| 15:06 |
hmmmm |
v3s32 = v3s32(pos.X, pos.Y, pos.Z); |
| 15:07 |
hmmmm |
errm |
| 15:07 |
hmmmm |
v3s32 pos32(pos.X, pos.Y, pos.Z); |
| 15:07 |
hmmmm |
that's one line |
| 15:08 |
hmmmm |
if you were to keep it as a macro though, I would at least rename PGSQL_V3S16_TO_INT to PGSQL_MAKE_COORD_VARS_FROM_POS or something |
| 15:08 |
nrzkt |
too late, removed |
| 15:08 |
hmmmm |
the former sounds like it's a conversion macro whereas the latter implies you're making something (in this case, new variables) |
| 15:09 |
hmmmm |
so, about this |
| 15:09 |
hmmmm |
nrzkt, have you actually *tried* testing postgresql speed with the single 64 bit position hash? |
| 15:10 |
hmmmm |
I'm willing to bet it would be a bit faster |
| 15:10 |
nrzkt |
no |
| 15:10 |
hmmmm |
why not? |
| 15:10 |
hmmmm |
what are the numbers? |
| 15:10 |
nrzkt |
it's definitively not recommended by postgresql |
| 15:10 |
hmmmm |
where does postgresql say that? |
| 15:10 |
nrzkt |
the primary key is composed with 3 int |
| 15:10 |
nrzkt |
it's a very simple key |
| 15:10 |
hmmmm |
sure it's simple |
| 15:10 |
hmmmm |
i just wanted to know if you tested it |
| 15:11 |
hmmmm |
satisfy my curiosity here |
| 15:11 |
nrzkt |
i didn't tested it and i didn't agree with this key design it's just a pain for server owners who wants to read easily their database to do things with it |
| 15:12 |
hmmmm |
you could be missing out on performance in loading |
| 15:12 |
nrzkt |
no |
| 15:12 |
hmmmm |
how do you know? you said, by your own admission, that you never tested it |
| 15:12 |
nrzkt |
i'm using this backend (except i'm using char conversion for coordinates using itos) since 1 year with 11M mapblocks in database (the maximum mapblocks generated by a map) and there is no fetching issue |
| 15:13 |
hmmmm |
and? |
| 15:13 |
nrzkt |
the keys are not varchar and text |
| 15:13 |
|
red-001 joined #minetest-dev |
| 15:13 |
hmmmm |
i never said there was a problem with loading performance |
| 15:13 |
hmmmm |
i'm saying that, if the point of postgresql is speed, why are you missing out on at least trying what could be an extremely simple optimization? |
| 15:14 |
nrzkt |
it's a stupid optimization |
| 15:14 |
hmmmm |
because right now it sounds to me like your primary motivation for this PR is to add support for postgresql because that's your favorite database |
| 15:15 |
nrzkt |
postgresql is not sqlite. it have very very good indexes, memory pools, optimizations etc. |
| 15:15 |
hmmmm |
irrelevant |
| 15:15 |
nrzkt |
maybe but i will not do it. I'm against this stupid method |
| 15:15 |
hmmmm |
but I just don't get it |
| 15:16 |
nrzkt |
i don't care, we only need 2 core dev to approve |
| 15:16 |
nrzkt |
you are free to set a -1 |
| 15:16 |
hmmmm |
not if you have a -1 |
| 15:16 |
nrzkt |
then we will be three to agree that the user experience is better than you big head |
| 15:17 |
hmmmm |
so because you don't like a certain design that might actually be faster, you are not even willing to entertain the possibility of using it |
| 15:18 |
hmmmm |
therefore we can conclude that to the author of this PR, performance is not the primary concern for adding PostgreSQL |
| 15:18 |
hmmmm |
so if it's not about performance, what is the benefit of PostgreSQL...? |
| 15:18 |
nore |
hmmmm: combining a position into an int is not a good idea |
| 15:18 |
hmmmm |
because I am highly opposed to adding support for 50000 different kinds of Foobars to minetest just because everybody has their own favorite Foobar |
| 15:19 |
nrzkt |
postgresql is slighter better for huge maps than redis |
| 15:19 |
nore |
because it makes the format impossible to extend |
| 15:19 |
nrzkt |
redis is just the worst backend we have |
| 15:19 |
nrzkt |
postgresql permit user to create websites to interact with it |
| 15:20 |
nrzkt |
redis with mapsize > physical RAM = shit |
| 15:20 |
nore |
I.e. you can't extend the size without changing the format |
| 15:20 |
nrzkt |
exact |
| 15:20 |
hmmmm |
so you're not satisfied with redis's default caching policies |
| 15:20 |
hmmmm |
it sounds to me like it might possibly be a configuration issue |
| 15:21 |
nrzkt |
did you test MT with redis 3.0 ? |
| 15:21 |
hmmmm |
.. |
| 15:21 |
hmmmm |
no i haven't and i'm not going to |
| 15:21 |
nrzkt |
redis 2.8 no problem, 3.0 is a big problem |
| 15:21 |
nrzkt |
redis load the whole map in memory and then, bye bye :p |
| 15:21 |
nrzkt |
it seem you are only a developper, not a user in fact |
| 15:21 |
nrzkt |
think about the user needs, not your big head |
| 15:21 |
hmmmm |
if you're willing to put forth the time and effort to configuring an advanced database for use with MT, you should also be willing to put in research into what each of the options do |
| 15:21 |
nrzkt |
postgresql fetch for 1000 mapblocks is < 1ms |
| 15:22 |
hmmmm |
nrzkt: a string is 10cm long |
| 15:22 |
nrzkt |
i don't know if you put string on your head :p |
| 15:23 |
hmmmm |
nore: also that argument is bunk because you can just as easily upgrade a map by running a conversion utility like there exists for 0.3.x -> 0.4.x |
| 15:23 |
nrzkt |
hmmmm: your argument is stupid because you want to complicate the user's life if they want to interact with the database backend and with the current representation no conversion is needed. |
| 15:23 |
hmmmm |
why would the user want to interact with a database |
| 15:24 |
|
lisac joined #minetest-dev |
| 15:24 |
nrzkt |
i have plans to move some files to databases, for example players |
| 15:24 |
hmmmm |
absolutely not |
| 15:24 |
nrzkt |
this will permit server owners to interact with the game using a website |
| 15:24 |
hmmmm |
i -5000 that |
| 15:24 |
nrzkt |
and do REAL gaming, with REAL community |
| 15:24 |
hmmmm |
the whole point of text files is to be modifiable by the end user |
| 15:24 |
nrzkt |
they can unify authentication, create inventory modifiers from a website, or see the items from pages, etc |
| 15:25 |
nrzkt |
the end user can do a query in database to modify the real data |
| 15:25 |
nrzkt |
update player_attribute set hp=20 where id=(select id from player where name = 'toto'); |
| 15:25 |
hmmmm |
nrzkt, just the same way you're highly opposed to using a coordinate hash for the key in the postgresql backend, i am highly opposed to this idea as well |
| 15:26 |
nrzkt |
you are against progress, sorry you are just the minetest Donald Trump |
| 15:26 |
nrzkt |
go vote for him, remove 80% of the americans from your country :p |
| 15:26 |
hmmmm |
besides, you say it's "for the players" but you're the only nerd here who wants to modify players by SQL instead of editing a text file |
| 15:26 |
nrzkt |
i'm not the only user to need this |
| 15:26 |
hmmmm |
literally nobody else wants to write sql queries |
| 15:26 |
nrzkt |
lol |
| 15:27 |
hmmmm |
also that's the progressivist fallacy |
| 15:27 |
nrzkt |
i think you didn't see how many servers have a website |
| 15:27 |
nrzkt |
or maybe you are working for minecraft and didn't want MT to win some gaming parts |
| 15:28 |
hmmmm |
I would be okay with keeping database copies of the other files as long as it's only an option in any case |
| 15:28 |
hmmmm |
and one that is disabled by default |
| 15:28 |
nrzkt |
it could be an option yes |
| 15:28 |
hmmmm |
it should be an option |
| 15:28 |
nrzkt |
but i plan to have a migration mode. |
| 15:28 |
nrzkt |
when you get 10k files in a folder it's just a performance shit for your FS |
| 15:28 |
hmmmm |
you can't force 99% of the users to interact with a pgsql prompt if they want to modify a player file |
| 15:28 |
nrzkt |
you inode is very big and the file fetching is por |
| 15:28 |
nrzkt |
poor* |
| 15:29 |
hmmmm |
this simply goes against the original design ideals of minetest |
| 15:29 |
nrzkt |
design can change |
| 15:29 |
hmmmm |
i don't approve of automatically doing anything to anybody's files |
| 15:29 |
nrzkt |
and can be enhanced for players |
| 15:29 |
hmmmm |
sorry |
| 15:29 |
hmmmm |
you're not 'for the players' if you think you know what's best for them |
| 15:29 |
hmmmm |
option only |
| 15:29 |
nrzkt |
in fact, SQLITE fetching for players is very fast compared to read and write a file on the FS |
| 15:30 |
Calinou |
hmmmm_thinking_he_is_a_player.webm.mp4.mkv.ogv |
| 15:30 |
hmmmm |
and that's why the map was moved from a set of files to sqlite... |
| 15:30 |
hmmmm |
player files on the other hand |
| 15:30 |
hmmmm |
there's not really any good argument that can be made there |
| 15:30 |
Calinou |
there are GUI front-ends to PostgreSQL by the way, I use Adminer here. |
| 15:31 |
Calinou |
https://www.adminer.org/ |
| 15:31 |
hmmmm |
maybe for extremely large servers and people who want to modify things by writing raw SQL queries in a web interface |
| 15:31 |
hmmmm |
but don't take away peoples' options to just modify the damn text file |
| 15:31 |
hmmmm |
nrzkt, you have some big ideas that just suck |
| 15:31 |
hmmmm |
the network changes were one of them |
| 15:31 |
nrzkt |
hmmmm: you have a big head that just suck :p |
| 15:31 |
Calinou |
our current network protocol sucks, let's admit it |
| 15:31 |
hmmmm |
i had to spend a lot of time and effort mopping up your bugs from that screwup |
| 15:32 |
Calinou |
it's incredibly laggy and intolerant to network faults |
| 15:32 |
nrzkt |
our current protocol is just a stupid fucking shit |
| 15:32 |
Calinou |
1999 games had better networking |
| 15:32 |
nrzkt |
go to enet |
| 15:32 |
hmmmm |
and it wasn't even the network that you modified, it was network serialization |
| 15:32 |
hmmmm |
it has nothing to do with the networking code |
| 15:32 |
Calinou |
nrzkt: ENet does not support IPv6, IMO it's not a solution… |
| 15:32 |
Calinou |
(except some fork of it) |
| 15:32 |
hmmmm |
ENet is not a solution anyway because it doesn't offer any real advantages over TCP |
| 15:32 |
Calinou |
(which has no guarantee of being maintained) |
| 15:32 |
nrzkt |
Calinou: i studied different libraries and i didn't see good gaming libraries |
| 15:32 |
Calinou |
and yes, IPv6 *is* being used in the real world |
| 15:32 |
nrzkt |
with IPv6 support |
| 15:33 |
Calinou |
hmmmm: TCP is slow for games. never use it. |
| 15:33 |
hmmmm |
Calinou: And ENet is faster than TCP? |
| 15:33 |
Calinou |
yes, by far |
| 15:33 |
nrzkt |
hmmmm: yes |
| 15:33 |
hmmmm |
Calinou: Because I actually did research on the topic |
| 15:33 |
Calinou |
reliable UDP will always be faster than TCP |
| 15:33 |
Calinou |
TCP is the Python of network protocols. |
| 15:33 |
Calinou |
UDP is the C. |
| 15:33 |
hmmmm |
ENet is not faster than TCP because it blocks on reordering packets |
| 15:33 |
nrzkt |
Calinou: depends of the implementation |
| 15:33 |
Calinou |
and reliable UDP is the PyPy. |
| 15:33 |
|
Void7 joined #minetest-dev |
| 15:33 |
Calinou |
(:D) |
| 15:33 |
hmmmm |
I asked about this in #enet and the author admitted I was correct, but was very defensive about it |
| 15:34 |
nrzkt |
i tested both TCP and Enet on my fork, and i didn't see a real difference between them. The ony problem with pure TCP was the non reliable packets which are too slow |
| 15:34 |
hmmmm |
In fact the only thing I can see visibly different from TCP is that it has the concept of channels (which MT already has) |
| 15:34 |
celeron55 |
oh, it's this discussion again; i should remind people that the current protocol was meant to be a temporary solution before a final assesment on enet would be done and enet would likely be taken into use |
| 15:34 |
hmmmm |
Big NO to enet |
| 15:34 |
hmmmm |
fuck that shit |
| 15:34 |
Calinou |
we currently support IPv6 (and have done since 2013) btw :/ |
| 15:35 |
Calinou |
my Quake 3 client can do IPv6, why can't a modern game like Minetest do it |
| 15:35 |
nrzkt |
Calinou, hmmmm: you can take enet, fix the wrong things and integrated it with core |
| 15:35 |
hmmmm |
nrzkt: In my experience I've found that fixing a third-party turd ends up requiring more work than fixing up what we already know the problems of |
| 15:35 |
hmmmm |
trust me, I've done it before.. |
| 15:36 |
nrzkt |
i know you know all and you are a fucking god and others suck :) |
| 15:36 |
kahrl |
and then some update comes along with security fixes and someone has to figure out a way to merge the upstream changes with ours |
| 15:36 |
hmmmm |
instead of saying that why don't you attack the substance of my claims |
| 15:37 |
sfan5 |
nrzkt: what's wrong with redis 3 and minetest? |
| 15:38 |
nrzkt |
sfan5: before i switch to postgresql 1 year ago (that's why i created this backend) redis takes more and more RAM when v3.0 comes, and at the end, redis take the whole server ram and stuck it. |
| 15:38 |
nrzkt |
because my map growed to 14GB |
| 15:38 |
sfan5 |
yes that's because thats what redis does |
| 15:38 |
Calinou |
yes, Redis is not suited for everything |
| 15:38 |
Calinou |
which is why we ought to support a client-server DBMS as well that's still SQL |
| 15:38 |
nrzkt |
it's why postgresql is best for big maps, you have a shared buffer, you tune it to the best memory you have and it's as better as redis |
| 15:40 |
nrzkt |
my shared buffer is set to 2GB (for a 10GB storage backend) it's sufficient to load a mapblock < 0.1mss and 1000mapblocks in 1ms |
| 15:40 |
hmmmm |
nrzkt: I'm not saying I am against having composite primary key, I am simply asking if you've tested the performance difference |
| 15:40 |
nrzkt |
i talked with my DBA at work |
| 15:40 |
hmmmm |
nrzkt: If the performance difference is negligible then nobody would feel bad about not doing that |
| 15:40 |
nrzkt |
to be sure |
| 15:41 |
hmmmm |
so it's about going forward with design decisions and having no regrets |
| 15:41 |
hmmmm |
investigating all the possibilities first |
| 15:41 |
nrzkt |
the performance will be negligible because it's CPU friendly types for PG, the extra usage was only 4 bytes and there is no join with other tables |
| 15:41 |
hmmmm |
that's guesswork |
| 15:41 |
hmmmm |
you have no evidence for that |
| 15:42 |
nrzkt |
the problem could occur if we have many join with other tables because pg will replicate the indexes for position for each table. But we only have 1 |
| 15:45 |
|
rubenwardy joined #minetest-dev |
| 15:46 |
nrzkt |
hmmmm: give me the calcul you want to generate the key from 4 int to 1 int8 ? |
| 15:46 |
nrzkt |
int64 |
| 15:47 |
nrzkt |
i will duplicate my production table with 11M records with this and doing a benchmark |
| 15:48 |
hmmmm |
nrzkt, Database::getBlockAsInteger() |
| 15:49 |
nrzkt |
it's a int64 ? |
| 15:49 |
hmmmm |
yes |
| 15:49 |
nrzkt |
ok will look at this, but didn't get time to do this now, maybe tomorrow |
| 15:49 |
hmmmm |
:| |
| 15:50 |
|
DI3HARD139 joined #minetest-dev |
| 15:54 |
nrzkt |
the v3s32 is a good idea yes, why not, but i need to do the htonl |
| 15:54 |
hmmmm |
oh you're right |
| 15:55 |
hmmmm |
that won't save on lines |
| 15:55 |
hmmmm |
ok nevermind that idea then |
| 15:58 |
* Zeno` |
pushes away the debris covering his bomb shelter, slowly emerges and peers through the dust |
| 16:01 |
nrzkt |
transaction type: Custom query |
| 16:01 |
nrzkt |
scaling factor: 1 |
| 16:01 |
nrzkt |
query mode: simple |
| 16:01 |
nrzkt |
number of clients: 10 |
| 16:01 |
nrzkt |
number of threads: 1 |
| 16:01 |
nrzkt |
duration: 60 s |
| 16:01 |
nrzkt |
number of transactions actually processed: 1321121 |
| 16:01 |
nrzkt |
latency average: 0.454 ms |
| 16:01 |
nrzkt |
tps = 22017.011141 (including connections establishing) |
| 16:01 |
nrzkt |
tps = 22018.197392 (excluding connections establishing) |
| 16:01 |
nrzkt |
this is with the triple key |
| 16:01 |
nrzkt |
10 servers connected to the same database |
| 16:02 |
nrzkt |
query: select posx,posy,posz from blocks WHERE posx=(SELECT RANDOM(-1800,1800)::int) AND posy=(SELECT RANDOM(-1800,1800)::int) AND posz=(SELECT RANDOM(-1800,1800)::int); |
| 16:02 |
nrzkt |
1.3M mapblocks in 1min this is quite good |
| 16:03 |
nrzkt |
pgbench -S -f test.sql -c 10 -t 60 epixel |
| 16:03 |
nrzkt |
that was the command used |
| 16:10 |
|
PilzAdam joined #minetest-dev |
| 16:10 |
nrzkt |
hmmmm: is this posx+posy*4096+posz*16777216 ? |
| 16:10 |
nrzkt |
did you agree with this calcul ? |
| 16:10 |
hmmmm |
yes |
| 16:10 |
nrzkt |
hmmm strange then |
| 16:10 |
nrzkt |
pos | bigint | not null |
| 16:11 |
nrzkt |
ERROR: integer out of range |
| 16:11 |
hmmmm |
did you make it unsigned |
| 16:11 |
hmmmm |
it's supposed to be signed |
| 16:11 |
nrzkt |
we have negative coordinates |
| 16:12 |
nrzkt |
in my db i see -1800 -> 1800 at least |
| 16:13 |
nrzkt |
wtf then |
| 16:14 |
hmmmm |
how are you doing this |
| 16:15 |
nrzkt |
a single sql query, but it seems it's not a 64b int |
| 16:15 |
nrzkt |
SELECT (posx+posy*4096+posz*16777216)::int8, block_data, b_nodes, b_nodes_metas, b_time FROM blocks |
| 16:15 |
hmmmm |
errm |
| 16:15 |
hmmmm |
::int8 means an 8 byte int, not 8 bit, right? |
| 16:15 |
nrzkt |
remove the cast for the first try |
| 16:15 |
nrzkt |
yes yes |
| 16:15 |
nrzkt |
it's the last try, i tried to cast it |
| 16:16 |
hmmmm |
and in this copy of your database you converted all the blocks from x, y, z format to the hash format? |
| 16:16 |
nrzkt |
in fact i'm trying to doing the SQL query to do it :p |
| 16:16 |
nrzkt |
insert into block_bigint (pos, blockdata, b_nodes, b_nodes_meta, b_time) SELECT (posx+posy*4096+posz*16777216)::int8, block_data, b_nodes, b_nodes_metas, b_time FROM blocks; |
| 16:17 |
nrzkt |
it's the exact query to do it |
| 16:17 |
nrzkt |
except the integer problem |
| 16:17 |
nrzkt |
i have an idea |
| 16:17 |
hmmmm |
well that's your problem |
| 16:17 |
hmmmm |
"insert into block_bigint" |
| 16:17 |
hmmmm |
but |
| 16:17 |
nrzkt |
okay this is working |
| 16:17 |
nrzkt |
i found |
| 16:17 |
hmmmm |
your select statement is selecting from blocks |
| 16:17 |
hmmmm |
not block_bigint |
| 16:17 |
nrzkt |
no no |
| 16:17 |
nrzkt |
this is the good query |
| 16:17 |
nrzkt |
i need to cast posx,y,z to int8 before doing the calc |
| 16:18 |
hmmmm |
oh, yeah, duh |
| 16:18 |
hmmmm |
otherwise they'd overflow and not give you the correct result |
| 16:18 |
nrzkt |
postgresql prevent it :p |
| 16:19 |
nrzkt |
okay conversion in progress, could be long because there is 11M entry and 14GB datas to change |
| 16:20 |
nrzkt |
the pg transaction will be huge but interesting |
| 16:20 |
nrzkt |
i go back home, will test it this evening or tomorrow |
| 16:20 |
hmmmm |
ok |
| 16:21 |
hmmmm |
thanks, by the way |
| 16:21 |
nrzkt |
no problem |
| 16:22 |
nrzkt |
will fix the last mentoinned issues when i will do the last bench |
| 16:29 |
|
Obani joined #minetest-dev |
| 16:30 |
|
Obani left #minetest-dev |
| 16:32 |
|
DI3HARD139 joined #minetest-dev |
| 16:49 |
|
davisonio joined #minetest-dev |
| 16:50 |
|
davisonio joined #minetest-dev |
| 16:53 |
|
Krock joined #minetest-dev |
| 17:04 |
|
torgdor joined #minetest-dev |
| 17:08 |
|
naltamur joined #minetest-dev |
| 17:08 |
naltamur |
Hello |
| 17:08 |
naltamur |
Can anyone zip all build files? |
| 17:08 |
naltamur |
and can anyone send me them? |
| 17:10 |
sfan5 |
which build files |
| 17:11 |
naltamur |
build files of minetest |
| 17:11 |
Krock |
what platform? win, mac, linux, android? |
| 17:11 |
naltamur |
includes libs and minetest source |
| 17:11 |
naltamur |
Windows |
| 17:11 |
Krock |
I have the most important win32 libraries uploaded if you need them |
| 17:11 |
Krock |
no idea if they work with 64-bit too.. I doubt it |
| 17:12 |
sfan5 |
https://kitsunemimi.pw/mtlibs-2015_01-win32.zip |
| 17:12 |
sfan5 |
alternatively run the buildbot script and abort it midway |
| 17:12 |
sfan5 |
then you'll also have all those libs |
| 17:12 |
naltamur |
sfan5: thanks |
| 17:12 |
Krock |
https://www.dropbox.com/s/r4u925x60ol6q1q/Minetest_MSVC_Win32_libraries_compiled.7z |
| 17:12 |
naltamur |
krock: thanks |
| 17:12 |
Krock |
mine were compiled with M$ visual C++, the one from sfan5 with MinGW |
| 17:13 |
naltamur |
thanks guys! |
| 17:13 |
naltamur |
minetest is faster than minecraft |
| 17:13 |
Krock |
good luck with your mission |
| 17:14 |
Krock |
compiling on windows isn't easy when you do it the first time |
| 17:15 |
naltamur |
okay but I want To try |
| 17:16 |
naltamur |
:) |
| 17:17 |
naltamur |
Can I delete my post on forum? |
| 17:19 |
Krock |
nope.avi |
| 17:19 |
naltamur |
:*) |
| 17:20 |
Krock |
but you can edit them and use later <-- if you want to be a environmentalist |
| 17:21 |
naltamur |
Okay |
| 17:22 |
naltamur |
:) |
| 17:39 |
|
nrzkt joined #minetest-dev |
| 17:41 |
|
ssieb joined #minetest-dev |
| 17:51 |
|
ElectronLibre joined #minetest-dev |
| 17:53 |
|
Void7 joined #minetest-dev |
| 18:05 |
|
Megal joined #minetest-dev |
| 18:08 |
|
davisonio joined #minetest-dev |
| 18:14 |
|
rubywarden joined #minetest-dev |
| 18:24 |
|
DI3HARD139 joined #minetest-dev |
| 18:28 |
nrzkt |
hmmmm, look at my last comment on github for the bench in MT conditions |
| 18:31 |
hmmmm |
strange |
| 18:31 |
hmmmm |
well whatever then |
| 18:31 |
hmmmm |
it's good to know |
| 18:32 |
kahrl |
what's the query for the single-key benchmark? |
| 18:41 |
|
Void7 joined #minetest-dev |
| 18:49 |
nrzkt |
select blockdata from block_bigint WHERE pos=(SELECT RANDOM(-1800::int8*4096::int8*4096::int8,1800::int8*4096::int8*4096::int8)::int8); |
| 18:49 |
nrzkt |
and select block_data from blocks WHERE posx=(SELECT RANDOM(-1800,1800)::int) AND posy=(SELECT RANDOM(-1800,1800)::int) AND posz=(SELECT RANDOM(-1800,1800)::int); |
| 18:49 |
nrzkt |
in multi key |
| 18:49 |
nrzkt |
it's better to pull the block_data :) |
| 18:51 |
hmmmm |
hmm |
| 18:51 |
hmmmm |
just FWIW, neither queries really approximate the minetest use case |
| 18:51 |
hmmmm |
blocks are usually loaded close to each other, not randomly |
| 18:52 |
hmmmm |
how much of an impact that has on your results is anyone's guess |
| 18:57 |
rubywarden |
Surely sqlite uses some sort of hash map to find records with a composite primary key? Why is a single key faster? |
| 18:57 |
rubywarden |
well, s/faster/significantly faster/ |
| 18:57 |
rubywarden |
a good hash map would have max 10 blocks in a cell |
| 18:58 |
hmmmm |
because software is never as highly optimized as you assume |
| 18:58 |
hmmmm |
disappointing truth |
| 19:00 |
|
davisonio joined #minetest-dev |
| 19:03 |
|
Krock joined #minetest-dev |
| 19:06 |
nrzkt |
hmmmm, i know yes, i don't know the real impact but it depend on the mapblock generated and stored in database. I think for most server it doesn't have a huge impact because the whole map isn't generated |
| 19:06 |
|
Void7 joined #minetest-dev |
| 19:09 |
hmmmm |
by the way nrzkt did you ever try using binary data instead of escaped text |
| 19:09 |
hmmmm |
i was wondering how that ended up working out |
| 19:09 |
hmmmm |
according to the documentation you shouldn't *need* to use PQunescapeBytea |
| 19:10 |
hmmmm |
i'm not saying it's a bottleneck, but things could be a bit faster without it |
| 19:12 |
|
BrandonReese joined #minetest-dev |
| 19:20 |
nrzkt |
hmmmm, i will try it a little bit later, i know the current way works well, but i didn't tested without it atm since my old devel 1 year ago |
| 19:21 |
hmmmm |
look I don't mean to be a pain in the ass but I've seen it 10000 times before, when code gets added as-is with promises to fix it later or make it better or whatever, 9 out of 10 times it doesn't happen |
| 19:21 |
|
DI3HARD139 joined #minetest-dev |
| 19:22 |
hmmmm |
i'm plenty guilty of this too |
| 19:22 |
nrzkt |
yeah, i didn't mean i will not do it for this pr, just not atm :p |
| 19:24 |
|
ElectronLibre joined #minetest-dev |
| 19:28 |
|
Void7 joined #minetest-dev |
| 19:35 |
|
naltamur joined #minetest-dev |
| 19:35 |
naltamur |
hello |
| 19:41 |
Krock |
wb |
| 19:44 |
naltamur |
I m compiling minetest with vs |
| 19:45 |
naltamur |
I hope no errors |
| 19:51 |
Krock |
that's what we all hope |
| 19:53 |
naltamur |
:( |
| 19:53 |
naltamur |
*:) |
| 19:54 |
sofar |
I got stuck on zlibwapi.dll |
| 19:54 |
sofar |
yesterday night |
| 19:55 |
sofar |
I thin that's the last dependency, and then I have a msvc build |
| 19:55 |
Krock |
you must use the wapi to link the stuff but then use zlib1.dll to run Minetest |
| 19:55 |
Krock |
kinda strange but works |
| 19:55 |
sofar |
problem is that I don't have the zlibwapi.dll and I don't know how to build it |
| 19:56 |
sofar |
docs are ... shoddy and old |
| 19:56 |
sfan5 |
i built those by manually editing zlib's makefile |
| 19:56 |
|
Player_2 joined #minetest-dev |
| 19:56 |
sofar |
also cmake entirely messes up the sqlite3_include_dirs |
| 19:56 |
sofar |
I have to manually edit the vcxproj files |
| 19:57 |
Krock |
well, I used the cmakelists.txt and it built it (with "/p:Configuration=Release") into zlib-1.2.8\contrib\vstudio\vc10\x86\ZlibDllRelease |
| 19:57 |
sofar |
it ends up writing ;SQLITE3_INCLUDE_DIRS.lib; |
| 19:57 |
sofar |
hmm, will check tonight again |
| 19:58 |
Krock |
there are also other project files, like zlibvc.sln which should build that dll |
| 19:58 |
Krock |
but I'm not 100% sure how I did that last year |
| 20:01 |
naltamur |
is libluajit.a the lua library? |
| 20:01 |
sfan5 |
yes |
| 20:01 |
naltamur |
okay tank |
| 20:01 |
naltamur |
thanks |
| 20:04 |
|
MillersMan joined #minetest-dev |
| 20:10 |
|
rubywarden joined #minetest-dev |
| 20:33 |
|
STHGOM joined #minetest-dev |
| 20:42 |
|
Anchakor__ joined #minetest-dev |
| 20:42 |
|
ElectronLibre joined #minetest-dev |
| 20:43 |
|
torgdor joined #minetest-dev |
| 20:45 |
nrzkt |
hmmmm, the unescape is mandatory |
| 20:45 |
nrzkt |
i don't find better thing else the mapblock is wronly read |
| 20:46 |
nrzkt |
using std::string(PQgetvalue(results, 0, 0), PQgetlength(results, 0, 0)) trigger the following error due to non unescaped string: 2016-05-20 22:39:48: ERROR[Main]: "ERROR: MapBlock format not supported" |
| 20:47 |
nrzkt |
it's the only method i found to properly read our datas |
| 20:47 |
|
KaadmY joined #minetest-dev |
| 20:49 |
|
paramat joined #minetest-dev |
| 20:51 |
paramat |
nore PilzAdam sfan5 please can anyone review/approve https://github.com/minetest/minetest_game/pull/1099 Change Vessels Textures ? |
| 20:51 |
sfan5 |
lgtm |
| 20:52 |
paramat |
thanks |
| 20:59 |
paramat |
i'll merge that and game 1094 later |
| 21:06 |
paramat |
even if a larger world is possible, it's not worth the effort and disruption, and is not even needed, please work on more important and useful stuff |
| 21:11 |
Fixer |
egh, feels like some server entered into unstability phase again |
| 21:12 |
Fixer |
new crashes, lock ups, rage quits |
| 21:12 |
sofar |
lol, rage quits |
| 21:12 |
|
red-001 joined #minetest-dev |
| 21:14 |
Fixer |
maikerumines servers are down, locking up again (even on server with few mods, also max_lag went up a lot by his words), Xanadu guys say it crashes more often and even lockups sometimes, strange |
| 21:15 |
Fixer |
evermzah may also has some lock up issue, but i'm not sure, need to ask him |
| 21:17 |
Fixer |
on amhi |
| 21:17 |
Fixer |
this server lock up issue is worrying me, it was reported long ago, but still elusive and nobody provided more data on it |
| 21:19 |
Fixer |
MillersMan: any results on that flow gap problem? it is only biggest problem i see with those flow prs.. (at least for me) |
| 21:22 |
paramat |
not much of a problem if the gap closes up again |
| 21:24 |
Fixer |
paramat: it closes nicely, but cpu load will be higher obviously (because some parts will reflow again) |
| 21:25 |
paramat |
i see |
| 21:25 |
Fixer |
gap closes, it is 1 mapblock gap iirc |
| 21:25 |
Fixer |
on some very big and talls flowsd |
| 21:25 |
Fixer |
tall* flows |
| 21:33 |
|
DI3HARD139 joined #minetest-dev |
| 21:57 |
|
troller joined #minetest-dev |
| 22:03 |
|
Megaf joined #minetest-dev |
| 22:05 |
|
kaeza joined #minetest-dev |
| 22:13 |
|
diemartin joined #minetest-dev |
| 22:36 |
|
Puka joined #minetest-dev |
| 22:44 |
hmmmm |
nrzkt, maybe your problem is "nobinary = true" in execPrepared() |
| 22:56 |
|
kaeza joined #minetest-dev |
| 23:02 |
|
proller joined #minetest-dev |
| 23:30 |
|
torgdor joined #minetest-dev |
| 23:31 |
|
Void7 joined #minetest-dev |
| 23:31 |
|
Megal joined #minetest-dev |
| 23:37 |
cheapie |
I should have a PR coming soon to improve /rollback performance significantly. |
| 23:37 |
cheapie |
Same general idea as the one I made a while back improving /rollback_check. |
| 23:38 |
cheapie |
I'm just currently testing it to make sure that everything still works properly with it, and measuring the improvement. |
| 23:43 |
|
est31 joined #minetest-dev |
| 23:45 |
* cheapie |
waves to est31 |
| 23:45 |
cheapie |
est31: I'm doing more work on the rollback DB stuff. |
| 23:45 |
est31 |
hello cheapie |
| 23:45 |
cheapie |
Hi. |
| 23:45 |
est31 |
still reading the log of this channel |
| 23:45 |
est31 |
today was a busy day |
| 23:45 |
est31 |
and lots of discussion |
| 23:46 |
est31 |
lots of heated discussion to be precise |
| 23:46 |
* cheapie |
scrolls up |
| 23:46 |
est31 |
Database::getBlockAsInteger is shit |
| 23:46 |
est31 |
it does not use z indexes |
| 23:46 |
cheapie |
Not surprising :P |
| 23:46 |
Fixer |
hai |
| 23:47 |
est31 |
If we use our own id's we should use z indexes |
| 23:47 |
cheapie |
est31: The stuff I'm talking about would be the 3 lines right before you joined. |
| 23:47 |
est31 |
hehe I started from the other side |
| 23:47 |
Fixer |
est31: about that network bug, does it lock up server (increasing lag) or just refusing to connect people? |
| 23:50 |
est31 |
it only refuses to connect people |
| 23:50 |
est31 |
the server should have minimal lag only |
| 23:51 |
Fixer |
ok |
