Time Nick Message 00:33 erlehmann pgimeno do you think this helps development? https://github.com/minetest/minetest/issues/11631#issuecomment-947170934 00:34 erlehmann i ask because all i did was figure out that biome generation is more broken than i thought 00:34 erlehmann but i have no solution :( 00:34 MTDiscord erlehmann, I think people are looking for solutions, not more issues, haha 00:35 erlehmann so i guess biome generation depends on optimization level and compiler version and possibly CPU architecture 00:35 erlehmann i mean, the mapgen is not reproducible anyway, but was it ever intended to be? is it a goal? 00:35 erlehmann if it is not a goal to get the same map with the same seed it is only a minor issue 00:36 erlehmann if it should be reproducible though, i'd suggest to make it a major one 00:36 MTDiscord Apparently mapgen stuff that breaks reproducibility isn't wanted... so I spy a conflict of interest here. 00:36 erlehmann this could explain some strange artifacts we have encountered 00:37 erlehmann josiah_wi where is the conflict of interest, it is already not reproducible 00:37 MTDiscord That's what I meant. If we can start PRing fixes for stuff that's another matter. I still need to write unit tests for my PR to fix the noise UB. 00:38 MTDiscord And #11656 is higher priority. 00:38 ShadowBot https://github.com/minetest/minetest/issues/11656 -- Add variable to use existing IrrlichtMt build by JosiahWI 00:38 erlehmann yeah so for the noise UB i want to talk to you 00:38 erlehmann the thing is, there is probably one major configuration 00:38 MTDiscord Oops that's the issues; meant to link the PR. 00:39 erlehmann i would assume that many ppl have servers on x86_64 using gcc of a specific version etc. pp. 00:39 erlehmann which *might* have predictable results 00:40 MTDiscord It isn't guaranteed be predictable if the configuration changes. But it seems to generally be predictable, hence why it's low priority. 00:40 erlehmann josiah_wi how are you approaching the unit tests – there are a limited amount of u16, so we can test them all? 00:40 erlehmann “it seems to generally be predictable”, uh no, it is not 00:41 erlehmann i have had encountered maps that generated slightly different on the same machine, same binary 00:41 erlehmann i never managed to reproduce it 00:41 erlehmann but now it makes sense 00:41 MTDiscord Well the core devs aren't freaking out so it's probably ok to be patient. 00:42 erlehmann if they do not freak out then ig the map should only be largely similar, not exactly 00:42 erlehmann but i would prefer exact 00:42 erlehmann josiah_wi can i help you in any way with the noise UB fix? 00:42 MTDiscord I was going to try unit tests that would test edge cases and check against a precomputed result. 00:43 MTDiscord If you want to write unit tests I recommend reading the PR (don't have link; I can find it later) which discusses the tests. 00:43 erlehmann i was not joking when i meant there are a limited amount of u16 or float or whatever 00:43 erlehmann ok! 00:43 erlehmann thank you for your effort 00:43 erlehmann oh btw 00:48 erlehmann i think if you use ubsan or a debugger it might change the values you observe ;) 00:49 pgimeno well, spamming 5 tests in 5 different posts is not helpful. People who subscribe to issues will receive as many emails as messages you post. They won't be happy about that. 00:52 erlehmann oh sorry 00:53 erlehmann it was not intended to annoy 00:53 MTDiscord As much as I want to get into doing some C++ stuff, I've found a niche for myself in the build system and I trust myself there more because I have a little experience now and got critique. 00:54 MTDiscord Fixing the CMake deps is something I believe I can do better than fixing UB. 00:55 MTDiscord And if anyone wants to help with that, I'll at least have some process in mind for how to move forward with that. xD 00:58 erlehmann i believe that you can only fix the cmake deps though if you eliminate all non-existence dependencies, as make can not handle them. 00:59 erlehmann embedded devs often do that, for the most part by ensuring there are no fallback paths for any includes 01:00 erlehmann josiah_wi do i need to elaborate? 01:02 erlehmann https://stackoverflow.com/questions/16875664/preventing-cmake-from-finding-installed-libraries-instead-of-local-libraries 01:02 erlehmann > By default, find_path (or find_library, etc.) first checks for files in the system standard locations, before searching in the values provided in PATH. That's why "installed" libraries are always found first on Linux (but not on Windows, that doesn't have standard locations for installed libraries). 01:03 erlehmann i believe that option helps, i have not tested it 01:04 MTDiscord We have variables for setting explicit library paths. That won't fall back to system paths, though. 01:05 erlehmann well if you have no fallback paths ever in the build, you are good 01:05 MTDiscord Now that you mention it I still disable system paths for IRRLICHTMT_BUILD_DIR. Thank you for bringing that up. 01:06 MTDiscord I should* 01:06 erlehmann but any fallback path will be an issue 01:06 erlehmann because the not-found-headers or not-found-libraries are non-existence dependencies 01:06 MTDiscord CMake is designed to have fallback paths. Working against the design is usually messy, so there better be a good reason. 01:07 erlehmann ok, i explain: cmake looks for a header or library at location A, location B, location C, in that order 01:07 MTDiscord erlehmann, the fix for that is to make them required deps or not link them if they aren't found. 01:07 erlehmann it finds the thing at location C 01:08 erlehmann now you invoke make, build, everything works. C is the path of the dependency. 01:08 erlehmann but A and B are paths of non-existence dependencies. if in the future a header or library shows up there, the binary needs to be rebuilt. 01:08 erlehmann otherwise you get a different result in an incremental build than a clean build 01:08 erlehmann which would be at least a bit weird 01:09 MTDiscord Correct. And CMake will not change the dependency path after it's been found. 01:09 erlehmann long story short, last i checked cmake could not handle this case and it would be a waste of time to make it handle it. 01:09 erlehmann because make can not handle it AND architecture-wise can not be fixed to handle it. 01:10 MTDiscord CMake deals in absolutes. Paths passed to make should be absolute paths. If they aren't that's a bug. 01:10 erlehmann yeah but that means that the compiler also needs to know that 01:11 MTDiscord The compiler is given the paths explicitly when it's invoked. You can observe this by building in verbose mode. 01:11 erlehmann non-trivial C++ projects usually have hundreds of thousands of non-existence dependencies, which usually means that the build system looks at absolute paths, but the compiler, preprocessor, etc. have fallback paths 01:11 erlehmann good! 01:12 MTDiscord As I said, CMake uses absolute paths. Library paths are passed to the compiler with options. Now I'll bet the compiler has fallbacks. 01:12 erlehmann josiah_wi i gather this means that headers are only ever included from the location given to cmake? 01:12 erlehmann otherwise there is a problem 01:13 MTDiscord I am not sure what the search order is for the compilers. 01:13 erlehmann i have made the build scripts for the game liberation circuit years ago, i actually straced the compiler there 01:13 erlehmann so now comes the funny part 01:14 MTDiscord What exactly is the issue that arises due to these fallbacks? 01:14 erlehmann if you have *any* kind of fallback path mechanism in your build process *anywhere*, the only build systems that can build it reliably are the ones which produce a list of deps and ne deps as side effect of the build. all other ones will sometimes genrate a binary not corresponding to the source, usually without ppl noticing. 01:14 erlehmann it is very funny to show ppl this issue in their own projects 01:15 MTDiscord I don't understand the issue yet. 01:15 erlehmann they first go “oh, that is never an issue”, then “maybe sometimes” 01:15 erlehmann wait i have a writeup 01:17 MTDiscord It seems like you shouldn't care which library binary is used as long as it works and doesn't introduce bugs. 01:17 erlehmann josiah_wi https://web.archive.org/web/20170326021857/http://news.dieweltistgarnichtso.net/posts/redo-gcc-automatic-dependencies.html 01:17 MTDiscord By that logic fallbacks are an excellent idea, and if you need a specific version of the library you specify that. 01:17 erlehmann all the dotted lines in the graphic represent stuff that make can not handle, in a simple hello world 01:19 MTDiscord Ah, I see now. 01:19 erlehmann you will probably go from “oh this is not an issue” to “oof this is everywhere” now, within several minutes or hours 01:20 erlehmann but ppl who work on build systems usually try to solve this 01:21 erlehmann just don't try to fix cmake or make, please. it will waste several hours of your life to figure out that you can not fix them and keep existing behaviour. 01:22 MTDiscord CMake does work this way, but I wouldn't say it's an issue. Accidentally including the wrong library is a programmer mistake, not a build system bug. 01:22 erlehmann it is though 01:22 erlehmann since the build system can not represent this relation 01:22 MTDiscord In fact I'd argue the current behavior of CMake is desirable. 01:22 erlehmann only if you have no fallback paths! 01:23 erlehmann in the compiler 01:23 MTDiscord I don't understand. 01:23 MTDiscord IrrlichtMt.so should always be included from the path you passed to CMake. 01:23 MTDiscord If it's included from anywhere else, that's a bug, 01:23 erlehmann yes i think for libraries this might work 01:23 erlehmann but for headers it will certainly not 01:24 erlehmann and with libraries it will be a programmer bug i guess? 01:24 MTDiscord Headers should be included from the directory CMake determines, and nowhere else. 01:24 erlehmann yes! 01:24 erlehmann i agree 01:24 erlehmann but you need to verify that somehow 01:25 MTDiscord CMake might not error if headers get deleted; I'm not sure how that would go. 01:25 erlehmann so the dot thingy, i do not know how to make it with cmake. if you find it, please tell me. 01:25 erlehmann if a header is deleted that is not a problem 01:25 erlehmann after all that is a dependency, the build will fail 01:25 MTDiscord If CMake does not error out then fallbacks will be a problem. 01:26 MTDiscord Otherwise I believe everything should be fine assuming proper precedence of compiler options. 01:26 erlehmann the error class i want to highlight here is not rebuilding all the parts you should rebuild. 01:26 erlehmann i hope it is so 01:26 erlehmann now that you are aware of this and explained to me all is well i feel better :) 01:26 MTDiscord CMake won't rebuild if you create a dependency it already found on a different path. 01:27 erlehmann yeah that is a big problem if the compiler, preprocessor etc. uses that thing then 01:27 MTDiscord There might be an option to change that but if there is I do not think it should be changed. 01:27 erlehmann as i said, if you use cmake or make, the only way to be sure is to rebuild everything from scratch every time 01:28 erlehmann in fact, debian does that and has tools to check reproducibility 01:28 MTDiscord If you change the library path you have to mirror that in CMake. And that can be done without regenerating cache. 01:29 MTDiscord Although in my PR I explicitly said to rm the cache and rebuild if you change the IrrlichtMt path. 01:29 erlehmann yes, rather rebuild too much than too little! 01:29 erlehmann building too much only costs time 01:29 erlehmann building to little gets you the wrong binary 01:30 MTDiscord CMake shouldn't rebuild too little. 01:30 MTDiscord From the compile times this should be obvious hehe. 01:30 erlehmann have you tried my git timestamp thingy? 01:31 MTDiscord I have not. 01:31 MTDiscord If CMake is rebuilding too little that's probably a bug in our CMakeListse 01:32 MTDiscord And that's what I'm asking about. 01:33 erlehmann josiah_wi i will tell you once i encounter it again 01:35 MTDiscord There's probably way more important changes that can be made to the CMake, such as organizing dependencies with targets to improve build system quality. 08:53 sfan5 rubenwardy: might be that google wants you to use app bundles these days (they're deprecating support for that too in Nov. IIRC) 08:53 sfan5 you can manually do it using apksigner still 13:20 rubenwardy This Android update is likely to be a bit risky 13:20 rubenwardy I wonder if there should be an option to copy rather than move, and if there should be a confirmation dialog 13:20 rubenwardy If you uninstall the app now, it'll delete any saved worlds 13:22 sfan5 https://developer.android.com/guide/topics/manifest/application-element#fragileuserdata btw 13:23 rubenwardy oooh 13:25 sfan5 also can't the user use their file manager to interact with /sdcard/Android/data/net.minetest.minetest? 13:25 sfan5 the ZIP thing would ofc be useful but is not strictly necessary 13:26 rubenwardy hm, I can't see MT in the Apps section of my file manager 13:26 rubenwardy (pixel 4a) 13:26 rubenwardy oh it's not alphabetically sorted 13:27 rubenwardy that opens up app info, rather than the files 13:28 rubenwardy I can't see a way using Pixel's Files app 13:28 rubenwardy it might be possible using adb, which is not very user friendly but neither was using the file manager 13:29 rubenwardy InternalStorage > Android/data is empty for me 13:30 sfan5 let me reinstall minetest and check 13:30 MTDiscord I'm able to access /sdcard/Android/data/net.minetest.minetest/ with total commander, I just need to do a minor slight of hand with permissions which it shows how to do 13:32 sfan5 huh yeah it shows as empty, must be some permission thing 13:37 rubenwardy Should we start using app bundles? I'm not familiar with them 13:37 rubenwardy hmm, we still need apks for github releases and testing 13:38 sfan5 you can build both 13:38 rubenwardy yeah, makes sense 13:39 sfan5 I remember reading that Google wanted to force bundles for existing apps too but can't find a timeline 13:39 rubenwardy https://developer.android.com/distribute/play-policies 13:39 rubenwardy only appears to be for new apps 13:39 rubenwardy but still worth doing 13:40 rubenwardy maybe we won't need to +2 for each release with this? If it's clever enough to create 32/64 as needed 13:50 rubenwardy OHHH 13:50 rubenwardy the signed APKs are going to a different directory 13:52 rubenwardy https://github.com/rubenwardy/minetest/releases/tag/5.4.2-rc1 13:52 rubenwardy (eh, "rc1" maybe isn't right, as it's not an official release candidate) 13:53 sfan5 I suggest '5.4.2-android' 13:53 rubenwardy for the actual tag? sure 13:53 sfan5 yea 13:54 rubenwardy probably won't break anything 13:55 rubenwardy hmm, I'll probably need to bump the release properly rather than updating the code manually 14:03 rubenwardy Loading is noticably faster on 5.4.2 14:03 rubenwardy he says, as the bar stops moving 14:04 rubenwardy errr.. 14:04 rubenwardy my camera is upside down 14:04 sfan5 what 14:05 rubenwardy https://i.rubenwardy.com/qZyy5.jpeg 14:06 sfan5 ? 14:06 rubenwardy singleplayer is the same 14:06 rubenwardy it's definitely worked before 14:06 sfan5 maybe you backported too much 14:07 MTDiscord its a feature ™️ 14:10 rubenwardy grr 14:11 rubenwardy my phone is unable to open .conf files 14:12 MTDiscord Total Commander has a built-in editor that I assume should be able to do that. 14:12 rubenwardy Android Studio has a file manager I can use 14:12 rubenwardy is annoying though 14:12 rubenwardy > Important note: This app does NOT contain any ads. However, it contains a link "Add plugins (download)" in the home folder. This is treated as an ad by the Play Store because it links to our other apps (plugins). 14:12 rubenwardy que 14:13 rubenwardy disabling undersampling fixes it 14:15 sfan5 not a new bug then 14:27 rubenwardy ok, it looks fine to me 14:28 rubenwardy what do you think about adding a warning dialog? Is it necessary? 14:28 rubenwardy Any changes now increase the risk of issues, and we won't be able to release updates for a while after 1 nov 14:29 rubenwardy warning dialog probably isn't necessary if we can find ways to extract the worlds currently (ie: using Total Commander / adb) 14:29 rubenwardy and with the uninstall hasFragileUserData 14:29 sfan5 yeah 14:30 rubenwardy adb pull /sdcard/Android/data/net.minetest.minetest/files/Minetest/worlds 14:30 rubenwardy simple as that 14:30 rubenwardy well, plus setting up adb and developer mode 15:33 erlehmann disabling undersampling fixes it 15:33 erlehmann not a new bug then 15:33 erlehmann is this only on android? 15:35 sfan5 so far nobody has reproduced it on desktop 16:20 ShadowNinja Have you tried with an OGLES2 build on desktop? 16:22 sfan5 I have 16:50 MTDiscord Are you sure it’s not corresponding to the upright position of the phone 16:51 MTDiscord For which it would be in landscape 18:42 MTDiscord Do I need both ogles 1 and 2 to compile on Android? 18:43 sfan5 yes, but why are you asking? 18:43 MTDiscord Because I'm doing something stupid. 18:44 MTDiscord I'm trying to compile Minetest with Termux out of curiosity. 18:48 MTDiscord Minetest from the app store works so I must have GLES 2? CMake can only find OGLES1, ouch. 18:48 MTDiscord Actually it didn't find either. 18:49 sfan5 do you want to build it for the system inside termux (which doesn't even run graphical apps) or for Android as an APK? 18:49 sfan5 the former makes no sense and the latter is not possible 18:51 MTDiscord I was trying the former, and I can't find a gles library so I guess it won't even work. 18:52 MTDiscord I could compile the headless server, I suppose. xD 18:53 MTDiscord I actually tried to compile a headless minetest server in termux for funsies a while back, but it kept thinking I wanted to compile it as a native library for an APK and would throw errors 18:54 erlehmann sfan5 do the ntfs extraction problems you encountered on windows go away if you disable 8dot3 names? https://www.neowin.net/forum/topic/1366224-tweak-win-10-ntfs-by-disabling-8dot3-and-stripping-8dot3-huge-per-gain/ 18:54 erlehmann sfan5 i think windows makes a short DOS name for every file by default, so i hope this helps you 18:54 sfan5 I don't know but I doubt it, number of files per directory isn't the issue 18:55 erlehmann the 8dot3thing is reported to make windows file handling slower at about 1000 files or so 18:55 erlehmann no idea if that is still the case 18:56 MTDiscord @ROllerozxa Alright, I'm doing this. I think I can tell CMake I'm cross compiling for Linux or something. 18:57 MTDiscord yeah, that would probably be the best way of going about it 18:58 erlehmann sfan5, btw i appreciate the link to https://www.sqlite.org/fasterthanfs.html – but what i meant with benchmarking was pitting the actual file handling code vs db blob handling code. the article even mentions that itself, you have to measure the actual code you have, not their test cases. 18:59 erlehmann i bet a lot of the cases where someone was like “nice, i'll just use a relational db” were relying on those benchmarks but had different usage of the storage backend or different things than only tiny blobs 19:00 erlehmann sfan5 celeron55 regarding the UB in the noise, is “same seed should generate same map regardless of compiler version or optimization level or CPU architecture” a hard or soft goal of minetest? 19:01 erlehmann hard as in, important 19:01 erlehmann soft as in, well, nice to have 19:01 sfan5 didn't someone already figure out a way to get an identical calculation without UB? 19:03 erlehmann i mainly asked bc of the negative-float-into-unsigned-int test case i unintentionally spammed everyone with 19:03 erlehmann because there is no “identical calculation” unless you say this compiler, this optimization level, this platform, this architecture 19:04 erlehmann it would be good to know what the canonical minetest binaries are compiled with to figure out what the most common behaviour is 19:04 MTDiscord Compilation started. ETA: a very long time 19:04 MTDiscord nvmnd it still thisks it's Android. 19:04 erlehmann if there is such a thing as a canonical minetest binary 19:05 erlehmann the mapgen UB might explain basically all artifacts i have seen btw – stone map blocks hanging in the sky, jarring discontinuities in biomes, maybe even the 1 block high “stone clouds” 19:06 erlehmann appguru do you have an idea if the view bobbing affects head bone with playerplus? bc if so, that is packet spam 19:06 erlehmann cora already found that the player bones seem to never really come to rest 19:06 erlehmann there is some small movement 19:07 erlehmann even without idle anim or so 19:07 MTDiscord Bobbing is entirely clientside. I'm pretty sure playerplus (assuming it's a mod) doesn't implement it at all. 19:09 erlehmann luatic so it does not send the view direction ig? 19:23 MTDiscord The client does send the view direction 19:39 sfan5 merging #11429, #11695 in 10m 19:39 ShadowBot https://github.com/minetest/minetest/issues/11429 -- Add no_texture.png as fallback for unspecified textures by Wuzzy2 19:39 ShadowBot https://github.com/minetest/minetest/issues/11695 -- Fix view bobbing not resetting when resting by appgurueu