[00:32] *** Stiletto has quit IRC () [00:53] *** Stiletto has joined #archiveteam-ot [01:25] *** Sanqui has quit IRC (Ping timeout: 745 seconds) [01:28] *** Sanqui has joined #archiveteam-ot [01:39] *** Raccoon` has quit IRC (Remote host closed the connection) [01:50] *** killsushi has joined #archiveteam-ot [03:55] *** BlueMax has joined #archiveteam-ot [04:01] *** nataraj has joined #archiveteam-ot [04:52] *** killsushi has quit IRC (Read error: Operation timed out) [05:01] *** killsushi has joined #archiveteam-ot [05:32] *** deevious has joined #archiveteam-ot [06:41] *** kiska has quit IRC (Remote host closed the connection) [06:41] *** Flashfire has quit IRC (Remote host closed the connection) [06:42] *** kiska has joined #archiveteam-ot [06:42] *** Fusl__ sets mode: +o kiska [06:42] *** Fusl sets mode: +o kiska [06:42] *** Fusl_ sets mode: +o kiska [06:42] *** Flashfire has joined #archiveteam-ot [08:14] *** nataraj has quit IRC (Read error: Operation timed out) [08:25] *** Quirk has quit IRC (Ping timeout: 246 seconds) [08:27] *** Quirk has joined #archiveteam-ot [08:43] *** Raccoon has joined #archiveteam-ot [08:49] *** Quirk has quit IRC (Read error: Operation timed out) [08:50] *** Quirk has joined #archiveteam-ot [09:14] *** Quirk8 has joined #archiveteam-ot [09:15] *** Quirk has quit IRC (Ping timeout: 246 seconds) [10:10] *** BlueMax has quit IRC (Quit: Leaving) [10:26] *** Quirk8 has quit IRC (Ping timeout: 246 seconds) [10:26] *** Quirk8 has joined #archiveteam-ot [10:33] *** nataraj has joined #archiveteam-ot [12:44] right, considering finally moving off of irccloud - what's the FOTM irc client now? [12:50] The lounge is my choice but you'll need quite a bit of memory for it [12:51] fotm? [12:51] *** muxwayer has joined #archiveteam-ot [12:51] Cause once every 2-3 days the oom killer kills it, or something does cause when I look in journalctl all it tells me is killed [12:51] Flavour of the month [12:52] i use WeeChat in tmux on my dedicated server [12:52] and WeeChat handles quite a bit of buffers [12:52] I might see how kiwi irc is and might switch to that if it doesn't crash like every 2-3 days [12:56] kiska: trying to avoid browser based [12:56] may go weechat [12:57] *** superkuh has quit IRC (Read error: Operation timed out) [12:58] irssi in tmux for me <3 [13:09] *** superkuh has joined #archiveteam-ot [13:10] any plugins you use JAA? in terms of quality of life type stuff [13:13] Yeah, I'm using a couple scripts. adv_windowlist and trackbar-soliton are probably the most important ones for the interface side. The former adds a window list, the latter inserts a line between stuff you've seen before and new messages in each window. [13:14] I also have keyboard shortcuts for 80 windows. Above 80, it gets annoying. [13:14] 118 open at the moment [13:16] Regarding background scripts, I have an auto-opper for my backup client modified from a script called "autoop.pl", autowhois_simple, scripts to deal with idling and CTCP messages, and an anti-spam kickbanner. [13:24] the google get parameters are always encoded. what are they doing? [13:25] I don't understand the encoding. Looking like base64 but not [13:32] which google get parameters [13:33] the search and I think they do it on youtube maybe [13:56] for example there is this parameter: ved=2ahUKEwim5aDevtzbAhXKgrwKHZlICNwQ_AUoAXoECAEQAw&biw=752 [14:02] *** muxwayer has quit IRC (Remote host closed the connection) [14:09] I kind of want something to merge the big three into one [14:09] IRC, SLACK, Discord [14:11] it's hard to get slack to play nice though, because it's the only one with actual threads [14:11] Irccloud handles that OK [14:11] ah I thought you meant get them all in the same place [14:11] i.e say something in slack -> puts it into an irc channel [14:12] Oh no, that was be insane [14:12] I'm in like 30+ slacks, 50+ Discord servers and Unknown amounts of IRC [14:13] Discord "servers"? [14:14] Do you have a Facebook server too? [14:14] nit picking much? [14:14] No. [14:15] That company deserves to be called out on it's bullshit. It's a proprietary remote administration tool who sells it's users. [14:16] Who the fuck doesn't sell its users nowadays, NO ONE on the internet can be trusted with user data [14:16] Slack is even worse but at least people acknowledge it. Corporate stuff will always suck, that's just the nature of it. [14:16] It's called "guilds" in Discord's internal API. "Servers" is only used by marketing. [14:16] every single step of the chain sells its datat [14:16] jrwr, whataboutism. [14:16] Or so I heard. [14:17] There are options that don't. And they all rely on not being centralized and proprietary. [14:17] It's pretty simple. [14:18] and throws easy of use out the window in the process, I support open standards when I can, but there are reasons why these services exist [14:18] most of these honestly rely on network effect and "being first" much more than ease of use [14:19] even if ease of use was a factor in the initial adoption [14:19] come for the ease of use, stay for the friendships and formed social relationships [14:20] Yeah. Someone wanted to make money but they couldn't figure out how to do profit off an open protocol (IRC) so they made inferior centralized versions with emoticons. [14:20] Also, a free bouncer! Wow. [14:20] Oh, it's entirely possible to EEE IRC, come on [14:21] IRC has the same problem Web 1.0 forums have: they can't stand their own ground easily in the smartphone era [14:21] In defense of discord, the permissions system is nice, it makes it pretty simple to run a large public discord for a community [14:21] dear lord, come off it superkuh [14:22] As a co admin of a 2.5k-user Discord, I found the permissions system rather lacking [14:22] Same kind of arguments I got from people about facebook circa 2009. [14:22] sufficient, but far from great [14:22] and IRCs is better? [14:22] can always write my own ircd [14:22] and you can always write your own Discord bot [14:22] I'm not disagreeing with you re: data collection and all that shit [14:22] there are things you cannot achieve with a Discord bot [14:22] They send a tracking request for every single thing you do in their client. Clicked on someone's profile, clicked on a channel, clicked on a server, etc. The URL was named /track before but they renamed it to "/events" recently (but it's still a POST with no response). [14:22] but as least within the walls of this channel, I expect that people that USE discord are aware of what's going on and have somewhat agreed to it [14:23] Also their desktop client is literally a remote administration toolkit, it has full access to FS (electron app) and it loads every script from their servers. [14:23] nobody forces you to use the desktop client, I know I don't [14:23] They can just add something like require('fs').readFileSync(process.env.HOME + '/.ssh/id_rsa').toString() and send this to their servers, and you won't even notice that (since it doesn't require an update on client because the client is just a browser with full permissions that loads obfuscated code from their servers every time you launch it). [14:23] I'm not a fan of Discord at all, but [14:23] cool, so could literally a ton of other things [14:23] they got popular for a reason, not a conspiracy; somehow, nobody is doing anything productive about it [14:23] Other things, not written in JS and downloaded every start. [14:24] IRC will outlive Discord just like it outlived the 90s/00s era of messengers, but at what cost? [14:24] the truth of the matter is IRC does not offer the quality-of-life things that people want. discord filled that void PERFECTLY and gained traction [14:24] If you're not a fan then the solution is simple as well. Don't personally use it. [14:24] and replace it with what? [14:24] if i'm running a project which requires reach, i need to use the solution that gives me said reach [14:24] i only have a say when reach is not a necessity [14:25] Reach? Does that mean you want to make money? [14:25] not really [14:25] those two are not equivalent [14:25] What does it mean? [14:25] superkuh: I have discord open on my PC right now, same as my IRC client. Are you not worried the chat logs of this conversation are being uploaded to (tencent/whoever), along with your IP? [14:25] say, if i'm trying to compete with a big project, i am already on a losing position [14:25] I agree on that front with asie, reach != money [14:25] do what you can to avoid it, doesn't matter [14:25] Sounds like corporate BS. [14:25] so i need to play every card i can [14:25] Kaz, not at all. Go ahead. [14:26] there we go then, it doesn't matter [14:26] or if i'm trying to do something that requires lots of people to join in [14:26] archiving is actually a good example [14:26] i'm going to use the least-friction platform possible [14:26] ^ [14:26] and in AD 2019, that platform is Discord (unless you just need them to enter for a short time, then IRC webchat bridges are tolerable) [14:26] Get on facebook then. [14:26] Oh, wait, do you have objections to that? [14:26] facebook is not low-friction when compared to discord [14:27] Which is what plenty of people do! [14:27] However, Discord has better reach in niche gaming/tech communities than Facebook. [14:27] that and twitter [14:27] god, the amount of infosec nerds on twitter is amazing [14:27] yeah [14:28] Frankly, I deleted my Facebook account on 2011 and Twitter in 2017; Reddit and Discord are next on the chopping block, but the reasons are not of financial politics - I'm not joining the fediverse either. [14:28] And space science people too. It's a shame. [14:28] I just came to the conclusion that social media is unhealthy [14:28] I have a set of perl scripts for viewing twitter. [14:28] Generates RSS feeds for my native client. [14:28] ok, that gets a chuckle from me [14:28] HOWEVER, I can't deny its value in, well, reaching people. Facebook or Twitter or Discord. [14:29] I REFUSE TO USE IT, STEALING MY DATA [14:29] *** Hani111 has joined #archiveteam-ot [14:29] but I still want to consume data from there [14:29] Oh, are we doing straw man now? [14:29] I'd consider it 'stirring the pot' more than anything else [14:30] You sure do resort to calling people names don't you, must not IRC much to handle this light convo [14:30] "I'm $discorduser and I know it's wrong but it's so easy I don't care. The only thing that matters is network effect and reach, I don't care about myself I just want to sell things or spread my ideology." [14:30] I already said: I'm planning to get rid of the account. [14:30] Also stop contributing to multiple FOSS projects which require me to use Discord. [14:30] That's legit. [14:31] Do it. [14:31] whoa dude [14:31] sknebel: isn't that what you are doing right now, trying to "spread my ideology." [14:31] I deleted my Twitter after a year or two of planning to do so [14:31] shit, wrong pring [14:31] now the account is owned by a Russian bot; https://twitter.com/asiekierka [14:31] plenty of FOSS projects on github, owned by microsoft [14:31] can't use that [14:31] !! [14:31] Oh god [14:31] I do have a self-hosted Gitea instance, but it's kind of unmaintained [14:31] Ya, Github, there is mass panic when shit goes kaboom over there [14:31] So Discord is probably coming too. I just need to phase out everything I do that necessiates Discord, you know. [14:32] Mostly contributing to the Minecraft modding scene. [14:32] I don't trust my ISP, disconnecting now, bye guys! [14:32] Just tunnel, Kaz. [14:32] But also my works in the ZZT and MegaZeux preservation scenes, as well as a local anime convention. [14:32] It's what I do. [14:32] Comcast does MITM attacks to inject javascript. [14:32] cool, so you put your trust in whoever is at the other end of the tunnel [14:32] I use many providers and switch between them. [14:32] very cool, fbi must be in awe [14:33] Nah, they didn't seem very impressed when they raided me in 2010. [14:33] actually all my traffic goes through Fusl at the moment [14:33] wikileaks, etc, as was the style at the time. [14:33] * asie gets out the popcorn [14:33] she probably has my facebook password [14:33] Don't forget about the middle men, I know a few carriers that sniff all the traffic for resale [14:33] It's raining outside. I'll get wet anyway. Might as well go out naked. [14:34] superkuh: you gotta get more tunnels man [14:34] /s [14:34] go deeper [14:34] You're just baiting each other at this point, I think [14:34] 👍 [14:34] wait this isn't discord(!!) [14:34] True. You didn't have to pay for that unicode. [14:34] *** Guest99 has joined #archiveteam-ot [14:35] * jrwr looks at superkuh's hostmask [14:35] wrong [14:35] Is that comcast I see [14:35] Also, superkuh, about reach: what's wrong with making money? Just curious. [14:35] irccloud raid my bank account every year [14:35] jrwr, that's what I just said a few lines back. [14:35] Because you seem to have wanted to jump on me on that alone [14:35] 'i tunnel everywhere' except irc, obviously [14:35] Yeah, my ISP doesn't mitm irc. [14:35] Only http. [14:35] cute, mine doesn't mitm anything [14:36] as far as you know of course, (I know TW does all plain text protocols) [14:36] asie, I generally dislike 'making money' in this context because it means you have to care about friction, ease of use, reach, etc and all those things leading you towards the natural centralized solutions of commerce. [14:36] just because they're not injecting, doesn't mean it's not being sold [14:36] superkuh: commie [14:37] what [14:37] And devs brining their bad corporate habits home when they code personal probjects. [14:37] Like single page applications. [14:37] superkuh: The problem is that the opposite of making money is not making money [14:37] -b [14:37] and that only gets you so far [14:37] Just gotta keep it separate from personal life. [14:37] Sure, you gotta use slack at work. But if you use it at home? Pretty lame. [14:38] that more or less necessiates working for someone else though [14:38] which is a centralized solution of employment [14:38] ... that's a good point. [14:38] *** Hani has quit IRC (Ping timeout: 745 seconds) [14:38] * Kaz looks at slack client, open on desktop [14:38] shit. [14:38] *** deevious has quit IRC (Ping timeout: 252 seconds) [14:38] if you work for yourself, at least in the initial stages, you MUST worry about all of those things, and generally don't separate it from personal life [14:38] later, when you're estabilished, you can probably work out work-life balance [14:38] and perhaps take an odd stance or two [14:38] but you need to pick your battles, at all times [14:40] for instance, i could choose between battling the centralization of social communication, or battling the dangerously low contribution rate of certain projects i love [14:40] i can only choose one of those - as i said, those projects congregate on Discord [14:40] i cannot choose both, because they are inherently contradictory [14:41] and both matter to me, but ultimately, one matters more [14:42] one way you can compromise without full "betrayal" on the "IRC/Discord issue" is to refuse setting up bridges between the two [14:42] it doesn't mean much in the long term, but in the worst case your IRC channel dies out and that's what happens to bridged IRC channels anyway, almost always [14:43] because if users have to choose between less convienence and more convienence, eventually they choose convienence [14:43] That the thing, you keep talking about people other than yourself. [14:43] Because I don't matter, as a person. [14:43] Only you matter. [14:43] What only matters is the ripples I effect in the world. [14:43] Because thats what really matters overall, the network affect [14:43] You are finite, but the ripples of your influence are eternal. [14:43] The only changes you can make our your own behavior. [14:44] Correct, but you need to play your cards wisely. If everyone sees you as a weird person, nobody will take your stances seriously. [14:44] Obviously I'm okay with this. [14:44] *** Hani111 has quit IRC (Ping timeout: 745 seconds) [14:44] Then I think we have nothing to discuss. [14:45] On this subject. Because it's a fundamental difference in life philosophy [14:45] Yep. Except that discord doesn't have 'servers'. [14:45] *** deevious has joined #archiveteam-ot [14:45] well that's wrong, there's definitely servers somewhere hosting it [14:45] What he means is that when a person gets a 'server' from Discord, they have no rights to it that would come from the usual rental of a 'server'. [14:45] So does facebook. But you don't have a facebook server. And you don't have a discord server. [14:46] Which is correct, and I despise Discord's marketing choice here [14:46] but it was I think done to appease TeamSpeak users in particular [14:46] Yes, to poach them with lies. [14:46] asie: I know - I'm just stirring [14:46] Not really, just use their terminology. [14:46] superkuh: you're so woke you should go on Ellen [14:46] Thanks bud. [14:46] Most users on Discord don't really care what it's called, and I'm sure they don't believe they own a Discord "server" [14:47] asie what does your philoshophy recommend if your audience want to infect you with evilcorp's evilproject X eg. discord, facebook, etc., and you personally want to send them to hell but decide you roll because you doesn't want to look weird? do you tell them in any way your values personally not compatible with using evilcorp or you hide the fact? [14:48] superkuh: but server is the term they use on discord [14:48] them=evilcorp [14:48] what's the problem [14:48] Guest99: You decide what matters to you more. [14:48] *** Hani has joined #archiveteam-ot [14:48] I tried an experimental approach with running both a Discord and an IRC channel for a certain large project, with the caveat that core developers would stick to the IRC. [14:48] This worked well for a few months, but convienence won. [14:48] asie: I mean you either join or you don't but do you tell them your values in either case [14:49] I think most regulars on said Discord server are *very* well aware I'm opposed to Discord. [14:49] I'll put it that way. [14:49] okay [14:49] I also vetoed getting the server partnered/verified. [14:49] Even though we meet all the requirements and are decently large (2.5k users) [14:50] The IRC is unfortunately mostly dead, but in this community all IRC channels are mostly dead [14:50] We do however get an occasional poster who goes to the IRC first. [14:50] This is for a project where reach was crucial. I have other projects in the community which do not require reach - those are IRC-exclusive. [14:50] (Also, those are personal projects.) [14:51] (So, like, it is not relevant to me if people use them or not, or discuss them or not - as such, I feel empowered to pick what works best for myself.) [14:51] But then, in said community (Minecraft modding) it's irrelevant as the key software distribution [14:51] is owned by Twitch/Amazon, the game is owned by Microsoft, and the key discussion places are owned by either FANDOM or Reddit. [14:52] how do you communicate to others your values are opposed to say discord but that you are ready to compromise yourself for the project or whatever it is you tell them [14:52] Guest99: Essentially, others told me that an unofficial Discord server would be spun up if I didn't give my blessing [14:52] Probably what he just told us, Guest99? [14:52] Which would have... everyone but me [14:52] Which is terrible for the project, as its leader being unaware of the de-facto (not de-iure) community's ongoings is not a good idea at such an early stage. [14:53] superkuh good point [14:53] Again, this is a community in which IRC died out totally about a year or so before we were having this discussion. [14:53] For a project which necessiated reach, it was compromise or death [14:53] (or, rather, compromise or compromise) [14:54] (I mean, even Microsoft themselves has a partnered Discord for the game now.) [14:55] Anyhow. Pick your battles, and prefer the ones where you have a chance of making a meaningful difference. [14:56] well there is probably no point making points to people who doesn't listen [14:56] Which is why I'm considering dumping Minecraft modding in general. An argument can be made that in the corporate-embraced modding era, it has become a bit... exploitative. [14:57] do you prevent rotting your values while inside compromise? lot of people fear if they give in they will be overtaken after a while completely [14:58] There's an idealistic and pragmatic approach, I feel. [14:58] You can try to prioritize holding your ground, or you can try to prioritize a kind of... min-maxing. [14:58] But those are complex topics, like, really. [14:59] Way too complex for a Friday afternoon. [14:59] I am sorry [15:00] To sum it up: if you sacrifice yourself in an empty forest, did you really make a sound? [15:00] But superkuh's point is valid. If you care first and foremost about YOUR ground and YOUR values, compromise is a terrible idea [15:01] On the other hand, I don't feel like I'm some kind of value-less person. I doubt myself all the time. [15:02] And I can't really know what would have happened otherwise, say, if I explicitly forbade making a "sanctioned" Discord guild for the group reach-necessiating project [15:02] well, I mean, you would probably have a hard time telling a vegetarian that if he wants to join your work he HAS to eat a steak before every occasion [15:02] I can't KNOW. I can only GUESS. [15:02] And I can make what I think is the best decision given the data and goals. [15:02] But I have a lot of doubts. I mentioned the "modding as labor exploitation" angle above, as that I think pertains most to the subject at hand, but... [15:03] (ops: am I being too -ot for -ot at this point?) [15:03] well if the mods are non-free then it is [15:04] one could always argue perhaps, if it is this much fun cleaning the streets and building the skyscrapers, is it really work? [15:04] If the mods are free but they link to a proprietary codebase, were they really ever free? [15:04] There's no API. All client-side Minecraft mods link directly to a decompiled, community-deobfuscated version of the game. [15:05] (The server side has an abstraction layer.) [15:05] And as they use very intimate engine details, it is likely quite literally impossible to make a "free as in freedom" way to run those mods. [15:06] (I mean "patching specific bytecode locations in the game" intimate) [15:06] I don't understand how a deobfuscated version proprietary [15:07] How would it not be? [15:07] The game is under an EULA, and it is proprietary. [15:07] It sounds like a few steps away from an open engine rewrite [15:07] The community has an "unofficial" (in very big quotation marks) toolkit to deobfuscate and decompile it. [15:08] But the underlying code is identical. Just with names changed and comments added. [15:08] Guest99: Reimplementing something as free software without violating the original copyright is very difficult. [15:08] Look into cleanroom software engineering. [15:08] https://en.wikipedia.org/wiki/Clean_room_design [15:10] And, as I said, Minecraft mods rely on something which is much closer to a 1:1 of the original engine than "read all the asset files in properly" [15:10] I guess the regulations got stricter in the last 40 years [15:10] I really liked Tekkit (now Tekkit Classic) at one point but it seems like the modding community has gone to hell [15:10] What do you mean by "gone to hell"? [15:11] I did some research into a modern version of those mods and most seem to be abandoned [15:11] That's incorrect, though. [15:11] Most of these mods are either still maintained or cloned. [15:11] And the average modpack's amount of content has gone up tenfold since then. [15:11] Guest99: Around copyright? No, not really. Those laws haven't been brought up to date with current technologies in a long time in most places. [15:12] If you take someone's song and re-record it, you're violating their copyright because it's a derivative work. Same thing in software reimplementations. [15:19] *** Guest99 has quit IRC (Quit: Ping timeout (120 seconds)) [15:41] I lost track of the MC modding community years ago, did the Bukkit team ever get back up after all the source code started getting DMCA'd by that one guy asie ?? [15:41] That was a slightly more nuanced story [15:41] To keep it short: key Bukkit devs resigned, Mojang suddenly announced they technically own the rights to the Bukkit branding after hiring its key creators [15:42] next day, entire Bukkit team resigns + DMCA is thrown [15:42] I vaugly remember at the time reading a lot about it but yeah it was a while ago [15:42] there was this project called Spigot which was a performance-centric fork of Bukkit [15:42] Yeah I remember finding out Mojang had secretly bought Bukkit [15:42] after a month in limbo and consulting some lawyers they realized the GPL is only voided, in this case, on binary redistribution [15:42] so they made BuildTools, which decompiles the Minecraft server, patches it and builds Spigot on your home computer [15:43] they never redistribute linked binaries, so they feel they are "in the clear" [15:43] I remember them having some system where you had to apply their patches over a jar which was fun [15:43] neither do you, you only use them, and the GPL doesn't trip on usage [15:43] now it's automated [15:43] Yeah buildtools, I had a jenkins server automating that that I'd hand access out to on teamspeak back in the day haha [15:43] who doesn't? :D [15:43] Oh is Spigot still going then ? [15:43] yeah [15:43] anyhow, i'll get back to you in a bit [15:44] Oh that's cool, interesting to know it's all still alive [16:12] *** nataraj has quit IRC (Read error: Operation timed out) [16:14] back [16:19] *** DogsRNice has joined #archiveteam-ot [16:27] Anyway funny thing. Sydney trains uses slack + irc bridge. The irc channel/internal server is dead [16:29] most IRC channels die when they get bridges to something more convienent [16:29] i think i've only ever seen one community in which a Discord server was set up then largely ignored and eventually *that* is the one which died out [16:30] but that had no bridge [16:30] Saw it go the opposite way on espernet/freenode #cyberpunk. They were both bridged then a discord bridge was added but the discord died off. [16:31] I think there was discussion from sydney trains internal about usingt discord as a secondary server [16:32] Er, not esper. efnet. [16:35] I do believe that Sydney trains is forking out ~100k per year for its 11k employees for slack [16:36] AT has a discord [16:36] And I think discord was quoting ~25k for the same amount for Sydney trains [16:36] jrwr: Yeah its kinda dead xD [16:37] of users* [17:37] *** killsushi has quit IRC (Quit: Leaving) [17:57] *** Hani has quit IRC (Remote host closed the connection) [18:04] Good. I hope it stays that way. [18:16] *** Hani has joined #archiveteam-ot [18:31] *** godane has quit IRC (Leaving.) [18:38] *** godane has joined #archiveteam-ot [20:07] Hi, I just popped in here to see if anyone has any suggestions for this. I am looking for a hierarchical database format similar in simplicity and function as .json or .plist files, but that can query, add, remove, and edit data via command line (must be scriptable as this is being used in a shell script) and can also support large amounts of data in them (e.g. 250MB). Optionally I would like it to deny read and write p [20:07] ermissions to the database file while it is being read or written (i.e. if one process of the database tool is currently reading the DB or writing to it, prevent any other processes of that tool from doing the same until the first process is done). If that last part isn't supported, that's fine since I could probably deny access other ways in my script. [20:10] TLDR; looking for a hierarchical DB format [20:16] *** tuluu has quit IRC (Ping timeout: 258 seconds) [20:22] generally when people want some weird database thing they actually just need to learn postgresql and relational database design [20:24] if an application has a large number of different entities RDF things like Datomic might make sense [20:27] but if you really want a hierarchical thing for good reasons I would just dump it into a bunch of keys in rocksdb or sqlite [20:41] ivan_: What about .yaml? [20:42] I've never used it but maybe that might work okay? [20:42] It really needs to be hierarchical [20:43] no, yaml is not a database [20:43] another option is using the filesystem [20:43] Right, but could that be a good .json/.plist replacement? It's hierarchical [20:43] Forgot to mention, using .xml was another idea of mine but I didn't go through with it [20:44] you requirement included adding/removing/editing data in 250MB databases [20:44] I guess if you want to rewrite 250MB each time you could use text formats [20:44] I originally used the file system, but then things just got confusing, and I need strings with `:` in them which macOS's kernel doesn't support [20:44] use Linux? :-) [20:45] or a ： [20:45] Ooof, rewriting, I forgot about that. Yeah, I really really prefer to not have to rewrite the entire db if I have to [20:46] The whole : replacement thing has given me such a headache, I'd rather replace them with ` - ` in filename and have a text string tell what the content is _really_ called. [20:46] Yeah I could use the other two : lookalike characters, but idk [20:47] how did you get so much hierarchical data [20:47] YouTube archival :/ [20:47] oh, you want a relational database [20:47] lol, kernels that restrict filenames. [20:47] Well, I'll PM you an example of what I have (in .json format) [20:48] what do you think youtube uses to store their data [20:48] Linux disallows exactly two characters in filenames: slash and NUL. [20:50] JAA: I use `/` more often than `:` in my filenames [20:50] macOS restricts `:` and ASCII control codes [20:50] doesn't macOS convert / to something on disk [20:51] HFS+ actually supports any character in filenames according to Wikipedia? [20:51] "characters shown as slashes in Finder are shown as colons in shells, and vice versa" [20:52] JAA: HFS+ and APFS do allow them, yes, but macOS does not [20:52] Ah [20:52] And I know I said kernel, oops [20:52] writing archiving software for macOS is a dead end once you expand beyond your Apple computers [20:52] Didn't mean kernel [20:56] Some files in macOS have resource forks that are lost on other platforms but that's somewhat out of the scope of what I am using this database for [20:57] It is to keep track of changes between each YT channel grab [20:58] NTFS has ADS and Linux has xattrs [21:00] NTFS also has Alt-Streams [21:01] storing YouTube on filesystems doesn't work so well for the channels with 200K videos [21:02] also some video titles start with . so remember to patch youtube-dl to not do that :/ [21:02] If video titles begin with . that's no problem for me, macOS would just hide the file (which I keep all files visible anyway so it doesn't bother me). [21:03] Windows doesn't care about dot files [21:03] Not sure there are any channels with 200K videos though [21:03] In Windows a file cannot end with a . [21:03] once you share your files with other systems, they will disagree about the importance of the dot [21:03] the idiotic gtk file dialog will hide them, for example [21:03] and since 2016, most large channels have had to delete easily half of their videos because they contain some off color remark or such [21:04] uh-huh [21:04] Isn't there that channel full of generated videos which has 600k or something? [21:04] I use a 99% macOS environment so cross-platform compatibility is of low importance to me regarding filenames [21:04] Forgot the name. [21:04] even AVE has deleted several dozen of his, due to his silly remarks about his wife -- "women". [21:04] somebody mirroed them all on mega.nz [21:05] Ah, Webdriver Torso, right. [21:05] https://www.youtube.com/playlist?list=UUsLiV4WJfkTEHH0b9PmRklw&disable_polymer=1 624,767 videos [21:05] Flashfire's favorite [21:06] *** nataraj has joined #archiveteam-ot [21:06] sweet jesus [21:08] So, anyway, sqlite3. I thought I saw in a video somewhere that sql databases _were_ hierarchical but that that is now depreciated [21:09] Why exactly do you need a hierarchical DB anyway? [21:11] https://github.com/omarroth/invidious/tree/master/config/sql [21:12] heh they're not big on validation [21:12] Or maybe I should ask: why exactly do you *think you* need a hierarchical DB? [21:12] *** SynMonger has joined #archiveteam-ot [21:14] *** SynMonger has quit IRC (Client Quit) [21:15] *** SynMonger has joined #archiveteam-ot [21:25] I bet something like Webdriver Torso is a useful channel to use as a cancellation test. If any spam / censor filters or DMCA pings come back with any of those videos, then the filter or script has to be scrapped and the request rejected. [21:27] lol, as if YouTube does any checking on copyright claims. [21:31] *** Raccoon has quit IRC (Ping timeout: 258 seconds) [21:32] *** Raccoon has joined #archiveteam-ot [21:59] *** nataraj has quit IRC (Read error: Operation timed out) [22:11] *** Raccoon` has joined #archiveteam-ot [22:12] *** Raccoon has quit IRC (Read error: Connection reset by peer) [22:15] *** Raccoon` has quit IRC (Remote host closed the connection) [22:15] *** Raccoon has joined #archiveteam-ot [23:32] *** xit2 has quit IRC (Remote host closed the connection) [23:49] *** BlueMax has joined #archiveteam-ot